Filtering performance issues

[ianthetechie]

I'm not sure if this is a pyosmium issue or "I'm holding it wrong," but the osmium CLI tool is able to efficiently filter things from a planet PBF in < 20 mins on some reasonable hardware (96GB RAM and 48 CPUs). But when I run layercake on a single layer, it run out of RAM, but it's fine if I pre-filter using what I THINK are the same filters using the osmium CLI 🤔 I've tried changing the index format but that just seemed to stall out at some point.

Have you hit any issues with this @jake-low?

[jake-low]

I've noticed this too, and I suspect that Layercake is doing something naive that makes it very inefficient, but haven't dug into it yet. The build machine I currently use has 80GB ram and 16 cores. I run Layercake on it using the --osmium-idx dense_file_array option, otherwise it runs out of RAM. The build takes over 24 hours with just the two layers we currently have. I'm hoping that this is something that can be fixed with some small change, but if I need to rewrite Layercake in C++ to use libosmium natively instead of via the Python bindings, I guess I'm open to that too.

If you want to dig into this yourself, something that I think would be informative but I haven't tried yet would be to write a test Python program that uses pyosmium to filter an OSM PBF file by some tags, and write out a new one (i.e. exactly what the osmium tag-filter command you're using to pre-filter the data does). Then time that script and see if you get roughly similar performance to the real osmium CLI tool.

Another thing that might be helpful is to run Layercake under python -m cprofile and see where it's spending most of its time.

[ianthetechie]

Ok, good to know I'm not alone in noticing the oddity!

but if I need to rewrite Layercake in C++ to use libosmium natively instead of via the Python bindings, I guess I'm open to that too

😂 Yeah, before I started hacking on Layercake I was considering rolling something similar in Rust...

Theoretically a Python library calling out to C++ should be able to jump out of the whole GIL mess and get roughly the same perf. What I thought the current filter setup did was keep things at the C++ level. Filters which involve Python code definitely don't, but this is clear in the docs. The implication in how this vs the top section are written implies that the built-in filters should be fast. But what I'm observing right now is that osmium tag-filter utilizes multiple cores just fine, while layercake is essentially bound to a single core. On top of that, I don't normally need to change the osmium index settings when running the C++ CLI, which makes it seem like what's happening is the objects are actually being materialized back to Python for filtering (triggering GIL limitations, among other things, though this doesn't explain the RAM usage).

Anyways, I'll take a look at this at some point within the next month or so once it becomes a pain point again.

[jake-low]

I ran some benchmarks comparing osmium tags-filter with a simple Python script that uses pyosmium to do the same thing. The pyosmium script is a lot slower. I'm not claiming that these are fair benchmarks of pyosmium; it's possible that I'm using the library wrong. But maybe if we can figure out why this test script is slower than osmium-tool, we can also figure out how to make Layercake faster.

Here's the osmium-tool command I tested (benchmarked using hyperfine on an M4 MacBook Air with 24GB RAM).

λ hyperfine 'osmium tags-filter washington-latest.osm.pbf w/building --overwrite -o wa-buildings.osm.pbf'
Benchmark 1: osmium tags-filter washington-latest.osm.pbf w/building --overwrite -o wa-buildings.osm.pbf
  Time (mean ± σ):      2.735 s ±  0.093 s    [User: 14.447 s, System: 0.727 s]
  Range (min … max):    2.569 s …  2.865 s    10 runs

Here's the script I compared it to:

#!/usr/bin/env python3
"""
Filter OSM PBF file to include only building ways, using EntityFilter and KeyFilter
(which are implemented in C++) for efficiency.

Usage: python pyosmium_filter_buildings.py INPUT_OSM_PBF OUTPUT_OSM_PBF
"""

import sys
import osmium

input_file = sys.argv[1]
output_file = sys.argv[2]

fp = osmium.FileProcessor(input_file)\
    .with_locations()\
    .with_filter(osmium.filter.KeyFilter("building"))\
    .with_filter(osmium.filter.EntityFilter(osmium.osm.WAY))

with osmium.BackReferenceWriter(output_file, ref_src=input_file, overwrite=True) as writer:
    for obj in fp:
        writer.add(obj)

And here's the benchmark for that script. I ran it under Python 3.13.4 and using pyosmium 4.1.1.

λ hyperfine 'python pyosmium_filter_buildings.py washington-latest.osm.pbf wa-buildings-py.osm.pbf'
Benchmark 1: python pyosmium_filter_buildings.py washington-latest.osm.pbf wa-buildings-py.osm.pbf
  Time (mean ± σ):     113.409 s ±  1.645 s    [User: 125.906 s, System: 1.378 s]
  Range (min … max):   110.969 s … 116.041 s    10 runs

I checked the output files from both scripts, and confirmed that they have the same number of nodes (18260399), ways (2584007), and relations (0), and are roughly the same size (~116MB). So I think the two are producing basically equivalent results.

I also experimented with the script a bit:

• changing the order of the two filters doesn't seem to matter
• removing .with_locations() also seems to have no effect (on runtime or output data)

So in summary, I wasn't able to make a simple Pyosmium script as fast as the osmium tags-filter command, which suggests there's significant (though maybe avoidable?) overhead that occurs even in this simple test case.

[jake-low]

😂 Yeah, before I started hacking on Layercake I was considering rolling something similar in Rust...

I would be quite interested in porting Layercake to Rust, if possible. Last time I looked at the landscape, I wasn't able to find a good crate for reading OSM PBF files. There were a few options (including a pure Rust implementation and a set of bindings for libosmium), but none were feature complete or seemed to be actively maintained. Maybe things have changed since then though. And if not, I'd definitely be open to collaborating on an OSM PBF reader crate; I think it would be a really useful addition to the georust ecosystem.

[ianthetechie]

Yeah, that was my experience too (looked at this ~6 months ago). We have a pretty strong interest in making it happen eventually (not exactly top of my priority list, but it will happen eventually). I'll be sure to keep you updated if I do take a swing at that 😂

[ianthetechie]

Just as a mental brain dump to plant some more seeds, one of the things I've been thinking about is how we could make Layercake layer definitions more..... layer-y and less code-y.

One possible design is to define the layers in SQL. The DuckDB spatial plugin is terribly slow at reading OSM (last I checked a few months ago). It takes minutes just to do a simple query from a South Korea extract (very small). And it doesn't give the data back in a particularly helpful format either: it's low level ways, nodes, and relations. No higher level geometry.

Apache DataFusion is an interesting alternative I've been using quite a bit lately which is far easier to extend with your own UDFs. I've written a handful and am contributing a bit to the GeoDataFusion project too.

The upshot is that if we can make the OSM querying efficient in a Rust library, we have a clear path to building flexible layer definitions in SQL. (I'm actually doing this at work for several other data pipelines and it's fantastic once you have the infrastructure set up). The challenge is building that great OSM library (which IMO should totally be called oxmium 😂).