Working with Shapefiles and GeoJSON

Shapefiles and GeoJSON remain the foundational vector formats for geospatial workflows. Whether you are assembling a local data pipeline, preparing assets for web mapping, or standardizing inputs for spatial analysis, mastering how to read, validate, and export these formats is essential. This guide sits within the broader Fundamentals of Python GIS curriculum and focuses on production-ready techniques that minimize friction while maintaining data integrity.

Preparing Your Geospatial Environment #

Before executing file I/O operations, your Python environment requires a stable geospatial stack. Silent failures during reads or writes frequently stem from missing compiled libraries, mismatched GDAL bindings, or incorrect system paths. For a reliable baseline configuration that resolves binary dependencies and manages environment variables, follow the procedures in Setting Up Geospatial Environments.

Once configured, geopandas serves as the primary interface. It abstracts lower-level libraries like fiona, shapely, and pyproj into a consistent, DataFrame-like API that handles both legacy and modern formats seamlessly.

Architecture and Format Trade-offs #

The ESRI Shapefile format has functioned as an industry standard since the early 1990s. Despite its name, it is not a single file but a mandatory collection of at least three components: .shp (geometry storage), .shx (spatial index), and .dbf (attribute table). Optional companions like .prj (projection metadata) and .cpg (character encoding) are frequently required for complete interoperability. Its legacy architecture imposes hard constraints: attribute field names are restricted to 10 characters, numeric precision is limited, and the format lacks native support for modern geometry types like compound curves or true 3D coordinates. The official GDAL Shapefile driver documentation outlines these technical boundaries in detail.

GeoJSON, formalized under RFC 7946, is a single, human-readable JSON file that maps directly to web standards. It handles nested properties gracefully, supports all standard OGC geometry types, and requires no sidecar files. When evaluating storage options, understanding how these structures interact with Coordinate Reference Systems is critical. Shapefiles embed projection metadata in a separate .prj file, while GeoJSON strictly mandates WGS 84 (EPSG:4326) for geographic coordinates, requiring explicit transformation for distance or area calculations. The choice between them typically hinges on workflow context: shapefiles dominate legacy desktop GIS and government data portals, while GeoJSON powers web APIs, interactive mapping libraries, and lightweight data exchange.

Reading, Inspecting, and Converting Data #

Loading and converting between these formats follows a predictable, reproducible pattern. The objective is to ingest data into memory, validate its spatial and tabular structure, and export it with explicit driver parameters.

flowchart LR
    A["read_file(input)"] --> B[Inspect structure & CRS]
    B --> C{Valid geometries<br/>& attributes?}
    C -->|warnings| D[Filter / fix]
    C -->|ok| E{Output format?}
    D --> E
    E -->|.shp| F["to_file(ESRI Shapefile)"]
    E -->|.geojson| G["to_file(GeoJSON)"]

import geopandas as gpd
import os
from pathlib import Path

def process_vector_data(input_path: str, output_path: str) -> None:
    """Read, inspect, validate, and export vector data."""

    # 1. Load data (GeoPandas auto-detects format from extension)
    gdf = gpd.read_file(input_path)

    # 2. Inspect core structure
    print(f"Records: {len(gdf)} | Attributes: {len(gdf.columns) - 1}")
    print(f"Active CRS: {gdf.crs}")
    print(f"Geometry Type: {gdf.geom_type.unique()}")

    # 3. Basic validation checks
    if gdf.isna().any().any():
        print("Warning: Missing attribute values detected.")
    if gdf.geometry.is_empty.any():
        print("Warning: Empty geometries found. Consider filtering.")

    # 4. Export with explicit parameters
    if output_path.endswith(".shp"):
        gdf.to_file(output_path, driver="ESRI Shapefile")
    elif output_path.endswith(".geojson"):
        gdf.to_file(output_path, driver="GeoJSON")
    else:
        raise ValueError("Unsupported output format. Use .shp or .geojson")

# Example execution
# process_vector_data("data/municipal_boundaries.shp", "output/boundaries.geojson")

Production pipelines rarely handle single files in isolation. When processing batch directories or validating incoming datasets from external partners, implementing automated checks prevents downstream corruption. Refer to the workflow for Automating shapefile validation scripts to integrate schema verification, topology checks, and CRS normalization into your CI/CD processes.

Transitioning to Modern Workflows #

While shapefiles and GeoJSON remain indispensable for interoperability, modern geospatial engineering increasingly favors containerized formats. GeoPackage (.gpkg) consolidates geometry, attributes, projections, and even raster tiles into a single SQLite database, eliminating sidecar file fragmentation and bypassing the 2GB file size limit inherent to legacy formats. As organizations scale toward Enterprise GIS Architecture, migrating to spatial databases and optimized containers becomes necessary for performance and governance.

For teams ready to adopt these next-generation standards, learning how to efficiently read and write spatial databases without loading entire datasets into memory is a critical skill. The techniques covered in Parsing GeoPackage databases efficiently demonstrate how to leverage SQL filtering, chunked processing, and spatial indexing to maintain high throughput in resource-constrained environments.