Ideas for further reducing memory usage

[systemed]

A few possibilities and thoughts:

Don't add ways to waystore unless they're needed by relations

See #292.

Lazy geometry assembly

At present we create our geometries whenever our Lua process asks for an object to be written. This is in linestringCached, polygonCached and multiPolygonCached in osm_lua_processing.cpp.

We could consider creating them lazily, i.e. only when writing each tile. The advantage of this is that we could purge the geometries when memory is getting low; they could subsequently be recreated next time we need them.

One challenge would be calculating the bbox if we haven't generated the geometry yet. A naive approach would simply take the min/max of the node coordinates - not sure whether this would suffice in the case of large geometries.

(This would rule out the approach in #292, because we'd need the waystore to stick around so that we could generate the geometries on demand.)

Use rtree for OutputObjects

Rather than adding each OutputObject to a list for each z14 tile, we could store an rtree of OutputObjects' bounding boxes, and then query the rtree when writing each tile. (This may need some thought for large, irregularly shaped objects. Performance might be an issue too.)

Support osmium add-locations-to-ways

This tool pre-processes the .pbf to add node lon/lat to the ways themselves. As a result, you don't need to read them into a node store. I've added support for this to my OSRM fork and it's fairly straightforward. (Explanation, docs.)

Optimise bitpacking in OutputObject

This isn't too efficient at present (8,8,8,1,4). We could reduce geomType to 2 bits and maybe z_order to 6. Not sure how much of a difference this would make - possibly none if the following variables are aligned to 4-byte boundaries, for example.

Support a "split layer" or "split region" approach

People have reported success splitting the planet into parts, running tilemaker over each one, and merging the resulting mbtiles. Alternatively, filter the planet by thematic layer, run tilemaker and merge. We could provide tooling to support this - it doesn't have to be in the tilemaker executable, it could be a bash/Ruby/Lua/whatever script that invokes tilemaker, osmium etc. as required.

Look at shapefile reading

A good starting point might be looking at the result of reading in shapefiles - it looks to me as if we're using many GB to read the coastlines for the whole world, when the shapefile itself is only around 1GB total.

[kleunen]

I think: Support osmium add-locations-to-ways is a very good idea. Should not be too difficult to implement and can make a difference in memory usage. Although, possibly, the compact mode already gives you the same benefit in performance. But maybe the add-locations-to-ways is a slightly more efficient way than compact mode.

The outputobject and references to the output objects are the most memory consuming. They are not stored in the store. I tried this, but this gives a very big slowdown in performance. You really have to remove 4 bytes to reduce the size of the outputobject further. Because of alignment. The reference count is 4 bytes, so if the outputobjects are stored in an rtree, you can remove this. But you need the bounding box of the geometry, but this can be calculated on demand. The bounding box however may overlap with a lot of tiles the geometry does not actually touch.

Shapefile might definitly be something to look at, i have no idea how much and what is loaded there. And if this is really needed.

Split region works well, it is easy to make a script for this. You could include renumbering the regions and using the compact mode. That way you should be able to generate the planet in 4 regions with 64G.

[systemed]

The outputobject and references to the output objects are the most memory consuming. They are not stored in the store. I tried this, but this gives a very big slowdown in performance. You really have to remove 4 bytes to reduce the size of the outputobject further. Because of alignment.

We might be able to trim this down by using a bitfield for objectid. It's defined as a NodeID == uint64_t, but OSM node IDs are currently just above 2**33.

The other bitfields (minZoom, geomType, layer, z_order, fromShapefile) add up to 23 bits. So we should be able to get all this into 8 bytes total.

[kleunen]

40 bits NodeID + 24 flags seem reasonable.

[systemed]

There's definitely room to manoeuvre with shapefiles. The coastline is 1GB raw, 20GB in-memory. The glaciers are 4MB raw, 6GB in-memory!

[kleunen]

wow. Why are they this big when loaded ?

[systemed]

After some investigation I think the issue is basically an artefact of the spherical Mercator projection we use. This really stretches the latitudes towards the poles - have a look at a typical spherical Mercator map (osm.org, or Google, or anything) to see how.

That means that a coastline or ice-shelf geometry towards the poles ends up covering thousands of z14 spherical Mercator tiles. In the most extreme example, I found one that generates 3.5 million tiles:

Reading .shp ice_shelf
Large bbox - 3515806 tiles (1498*2347: -85.0511,-180 - -77.8627,-147.067)

So we're writing 3.5 million OutputObjects for one object... ouch.

A long-term fix for this would be to use an rtree as above. For the planet, though, a short-term workaround might just be to only generate tiles between (say) 70°N and -60°S - there's not much of interest beyond that unless you're a polar bear or a penguin.

[kleunen]

3.5 million references to one OutputObject, right ?

[systemed]

Yes, sorry - it's basically the same as with OSM data.

[systemed]

Clamping to -60° to 75° reduces the shapefile impact from 26GB to 13GB - definitely an improvement but still too much.

One other possibility we could consider is shipping an pre-prepared .mbtiles with the coastline (and maybe some low-res landuse), which could be then used with the --merge option.

[kleunen]

13GB for the full planet is not unreasonable i would say. You don't really have to clamp, you can set the bounding box when converting the planet now right ? So it is recommended to use: -60 to 75. Much of these generated geometries are stored in the store.

Having a pre-prepared .mbtiles with coastline and landuse can be useful. But it would fit in this approach:

Support a "split layer" or "split region" approach
People have reported success splitting the planet into parts, running tilemaker over each one, and merging the resulting mbtiles. Alternatively, filter the planet by thematic layer, run tilemaker and merge. We could provide tooling to support this - it doesn't have to be in the tilemaker executable, it could be a bash/Ruby/Lua/whatever script that invokes tilemaker, osmium etc. as required.

Coastline/landuse can be a pre-processed thematic layer ?

[kleunen]

BTW, the shapesfile already have an rtree right ? So maybe instead of generating references to the outputobjects in the tileindex, you can retrieve them using the rtree already ?

[systemed]

That's a good point - we do have an (optional) rtree so I could experiment with using that.

It needs a slight adjustment for the --bbox parameter to take effect in this case (#259 was intended for .pbfs which don't have a bbox at all) but that's pretty trivial - I'll do a PR for that.

[kleunen]

Yes, it would be good to experiment with this. I was expecting the memory usage would go down. But I did not expect this would make such a huge difference. But I guess a full planet extract has a huge number of tiles (i think about 320M ?). 320M x the objects which are in the tiles. And all these references are stored in memory. That is a lot of memory.