Planet generation experiences

[systemed]

I've successfully generated an .mbtiles from planet.osm.pbf. 🎉

Using current master plus #292. Planet renumbered first with osmium renumber. Command line:

tilemaker --input planet.osm.pbf --output planet.mbtiles --bbox -180,-60,180,75 \
    --compact --store /media/ssd/store

Execution time c. 37 hours (with the exception of deallocating, see below). This is a 16-threaded HP machine, 2x X5650, 144GB RAM.

Maximum memory usage c. 131GB, plus 267GB store. mbtiles filesize is 65GB.

What needs fixing?

• Deallocating is very slow, first in the .clear() method after reading the pbf, but more significantly at the end of running (where it takes hours). The .pbf is complete and usable but tilemaker is still nominally running after its "Filled the tileset with good things" message.
• Ijsselmeer and the Great Lakes are not showing.
  
• Roundabouts don't show at lower zoom levels, causing a gap in the road. I suspect that they are simplified to 2-point lines and ultimately disappear entirely. We probably need to ensure that closed linestrings are never simplified below 4 points.
  
• Boundary lines are often discontinuous. This is because we don't yet support type=boundary relations, so any way that is a member of a type=boundary relation but not otherwise tagged will get dropped.
  

I suspect we might be able to get the running time down further too and optimise tile sizes, but the above issues are the most important.

[kleunen]

Yes the issue with the ijsselmeer we had before. The default style only renders water from osm up to zoom level 8. But large lakes are visible after zoom level 8. Some even up to zoom level 0.

[kleunen]

The cleanup at the end is not even needed. The data structures inside the mmap file get deallocated and then when all this is done. The complete file gets deleted anyway ..

[systemed]

There's something very wrong at the end - the "Filled the tileset with good things" line is literally the last line of tilemaker.cpp, but it's been hung for 12 hours now since that was output.

Ijsselmeer and the Great Lakes are missing entirely - not even showing at z8+. I'll see if I can work out what's going on there (need to check it wasn't introduced by #292).

[kleunen]

"Filled the tileset with good things" is the last line in tilemaker.cpp, but after this all the destructors are run. I also ran a planet conversion, it did nog hang or take a lot of time to cleanup. I generated the planet by merging 4 extracts of the planet. So probably there is a deadlock now somewhere when cleaning up ?

I do have ijsselmeer in my extract:

[systemed]

That's interesting. I don't get Ijsselmeer at all when generating from netherlands-latest.osm.pbf, either with #292 or current master.

[kleunen]

I am not sure if I now used my adapted style for this.

[systemed]

I've gone through commit-by-commit, and the problem starts with #307 (a9a54f9). Previous versions of the code rendered Ijsselmeer fine. Not sure yet what it might be in #307 causing it.

Edit: within #307, fe5229e is fine so it's something after that. IJsselmeer is read correctly by Lua which writes it to the water layer.

[kleunen]

maybe something with 'Use OsmID to lookup generated geometry'.

that would be the most likely candidate to cause this issue.

[kleunen]

I think i actually see the issue with this commit

Try #317, i have not tested it yet

[systemed]

Changing that line to

osmID = (relationId & OSMID_MASK) | OSMID_RELATION;

does seem to work. We previously had --newWayId because way and relation IDs were sharing the same ID space, but if we're separating them bitwise anyway with OSMID_RELATION and OSMID_MASK, I don't think we need newWayId any more. Does that make sense or have I misunderstood?

[kleunen]

Yes, it does seem to be a collision between the OSMIDs. That is why i added the extra two bits and the | OSMID_RELATION. To make the osm id unique. So yes, that does make sense.

The & OSMID_MASK is just to make sure the id only takes 40 bits max.

[systemed]

Ok - addressed in #318.

[kuwapa]

I wanted to add my results regarding the boundary issue here learned from #322

Boundaries which are tagged as ways and not as relations are not being generated by tilemaker. For example these -
https://www.openstreetmap.org/way/707454735
https://www.openstreetmap.org/way/136107215

Also, disputed borders are not being generated. Like this one -
https://www.openstreetmap.org/way/133267807

[systemed]

Yep. The issue is that tilemaker doesn't currently have any support for type=boundary relations, so anything that's tagged simply as a boundary relation (without the ways being tagged as boundaries) won't come through. I know how I'm going to fix it - the solution will enable other relation types like routes to be handled too - but haven't yet had the time to do so.