Parsing OpenStreetMap XML Data with Python

OpenStreetMap distributes its global geographic dataset primarily as XML, a structured text format that captures discrete points, linear features, and complex polygons. While modern web APIs and compressed vector alternatives like GeoJSON offer convenience, raw OSM XML remains the definitive, uncompressed source for comprehensive Spatial Data Processing & Analysis. Parsing this format efficiently requires navigating its nested hierarchy and selecting Python tools that extract meaningful spatial features without overwhelming system memory.

Understanding the OSM XML Structure #

The OSM data model relies on three fundamental primitives: nodes, ways, and relations, which reference one another to build progressively complex features:

flowchart LR
    R["Relation<br/>(multi-polygon, route)"] -->|references| W["Way<br/>(ordered node refs)"]
    W -->|references| N["Node<br/>(lat, lon, tags)"]
    R -->|references| N

A node represents a single geographic coordinate defined by latitude and longitude, frequently annotated with descriptive key-value pairs known as tags. Examples include highway=traffic_signals or amenity=hospital. Ways are ordered sequences of node references that construct linear features like streets and rivers, or closed loops that outline buildings and parks. Relations combine multiple nodes, ways, or other relations to model complex spatial entities, such as multi-polygon administrative boundaries or routing turn restrictions.

Every element carries a globally unique numeric identifier, version metadata, and a flexible tagging schema. Because tags are stored as child <tag> elements rather than XML attributes, parsers must explicitly traverse the document tree to capture them. For a complete breakdown of the schema, refer to the official OSM XML documentation.

Choosing an Iterative Parsing Strategy #

Traditional XML parsers attempt to load entire documents into RAM, which inevitably crashes when handling regional or continental OSM extracts. Python’s built-in xml.etree.ElementTree module offers an iterparse function that processes files sequentially. As detailed in the Python standard library reference, this event-driven generator yields elements only as their closing tags are encountered. This approach enables you to extract data, discard processed nodes from memory, and maintain a consistent RAM footprint regardless of file size.

The following workflow demonstrates how to isolate node coordinates, reconstruct ways into valid geometries, and export the results to a spatial DataFrame.

Step 1: Extract Nodes and Tags #

The first phase isolates node coordinates and their associated metadata. Storing these in a dictionary keyed by node ID enables rapid lookups when assembling linear features later.

import xml.etree.ElementTree as ET

def parse_osm_nodes(filepath):
    nodes = {}
    # iterparse yields (event, element) tuples as closing tags are encountered
    context = ET.iterparse(filepath, events=("end",))

    for event, elem in context:
        if elem.tag == "node":
            node_id = elem.get("id")
            lat = float(elem.get("lat"))
            lon = float(elem.get("lon"))

            # Extract key-value pairs from <tag> children
            tags = {child.get("k"): child.get("v") for child in elem.findall("tag")}
            nodes[node_id] = {"lat": lat, "lon": lon, "tags": tags}

            # Clear processed elements to prevent memory accumulation
            elem.clear()

    return nodes

Step 2: Reconstruct Ways into Geometries #

Nodes alone lack spatial context. To form lines or polygons, you must parse <way> elements, retrieve their ordered node references, and map them back to the coordinate dictionary. Using the Shapely library (a standard Python package for computational geometry), these coordinate sequences convert directly into LineString or Polygon objects.

from shapely.geometry import LineString, Polygon

def parse_osm_ways(filepath, node_lookup):
    geometries = []
    context = ET.iterparse(filepath, events=("end",))

    for event, elem in context:
        if elem.tag == "way":
            way_id = elem.get("id")
            node_refs = [nd.get("ref") for nd in elem.findall("nd")]
            tags = {child.get("k"): child.get("v") for child in elem.findall("tag")}

            # Build coordinate list from node references
            coords = []
            valid = True
            for ref in node_refs:
                if ref in node_lookup:
                    n = node_lookup[ref]
                    coords.append((n["lon"], n["lat"]))
                else:
                    valid = False
                    break

            if valid and len(coords) >= 2:
                # Determine if the way forms a closed polygon
                if coords[0] == coords[-1] and len(coords) >= 3:
                    geom = Polygon(coords)
                else:
                    geom = LineString(coords)

                geometries.append({"id": way_id, "geometry": geom, "tags": tags})

            elem.clear()

    return geometries

Step 3: Load into a Spatial DataFrame #

Once geometries are constructed, GeoPandas provides a seamless bridge to tabular analysis. A GeoDataFrame extends the familiar pandas DataFrame by adding a dedicated geometry column and coordinate reference system (CRS) awareness. Converting your parsed data into this format unlocks spatial indexing, attribute filtering, and export capabilities to standard GIS formats.

import geopandas as gpd

def create_gdf(ways_data):
    if not ways_data:
        return gpd.GeoDataFrame()

    gdf = gpd.GeoDataFrame(
        ways_data,
        geometry="geometry",
        crs="EPSG:4326"  # Standard WGS84 coordinate system used by OSM
    )
    return gdf

# Example execution flow
# nodes = parse_osm_nodes("extract.osm")
# ways = parse_osm_ways("extract.osm", nodes)
# roads_gdf = create_gdf(ways)
# roads_gdf.to_file("parsed_ways.gpkg", driver="GPKG")

Optimizing for Production Workflows #

Parsing OSM XML sequentially is reliable, but large-scale projects demand additional optimizations. Filtering elements during the parsing loop—rather than post-processing—dramatically reduces overhead. For instance, checking tags.get("highway") before building a way ensures only road networks are retained in memory. When preparing datasets for routing or graph construction, this targeted extraction aligns directly with Network Analysis with Python pipelines, where topology and connectivity take precedence over raw attribute volume.

Additionally, applying spatial indexing after loading the data into GeoPandas accelerates subsequent join and overlay operations. By combining iterative parsing with selective filtering, you can reliably process global-scale data on standard hardware without sacrificing analytical precision.