Validating Polygons according to Vector Spec 2.0 in PostGIS production chain

[kristfal]

Hey,

the move to Vector spec 2.0 puts more constraints to the polygon type than previously.

What is best practice that custom exported data should follow to adhere to the specification?

Reason for asking is that we're running a postGIS -> node mapnik -> .mbtiles production chain, and have noticed some issues with the mapbox-gl-js earcut branch. Entry data has a tendency to be slightly dirty, but we haven't had any issues with mapbox-gl-native or other tessellation-based renderers previously.

Currently experimenting with:

PostGIS processing: ST_MakeValid, ST_ForceRHR, ST_ForceSFS
Node Mapnik: Bumping versions,
Mbtile: Bumping versions

We're seeing both artifacts constrained within a single tile, but also covering multiple tiles. None of these seem to be affecting gl-native.

[flippmoke]

@kristfal

Short Answer:

You will continue to see problems with any branch of node-mapnik until the v2_spec branch lands into master and a new version is released.

Long Answer:

While it is good practice to use PostGIS to correct geometries, during our experiences of creating a whole lot of vector tiles over time we found issues surrounding many of the processes involved in vector tile creation in mapnik-vector-tile. Many operations can cause what was valid data prior to encoding in a vector for it to become invalid. I would like to introduce you to a few of these problems. After that, there are some existing options in the currently released node-mapnik that will help you create v1 tiles with less artifacts:

Simplification

Putting the entire detail of an original datasource into a vector isn't necessarily ideal because the large size and complexity goes against the intent of creating a vector tile. Therefore, we must simplify prior to putting data into a vector tile quite often. This currently controlled in node-mapnik by the simplify_distance option, unfortunately there are issues with simplification causing invalid polygons. In the example below you can see an exterior ring of a polygon causing the interior ring to extent past it:

Coordinate Transformation

During the coordinate transformation from any coordinate system to the coordinate system for Mapbox Vector Tiles many issues with the data can arise. The first problem is that if an axis is reversed between coordinate systems this can cause the winding order of the tile to be reversed. This is common because WGS 84 is positive up, while, the coordinates in a vector tile are positive down. This is encoder's responsibility handle properly. The latest version of node-mapnik should handle this properly.

Additionally, rounding can cause winding orders to be reversed as shown in this example, where a red polygon in double coordinates is rounded to integer coordinates:

Winding order here is very important because it determines what is an interior ring and what is a new exterior ring. This is critical for earcut and its branch to know properly.

Work around in node-mapnik:

Creating a vector tile with these options in node-mapnik should help currently:

• strictly_simple = true
• simplify_distance (default is 0.0) 6 or 7 is pretty good for a 4096 extent

If your ring order can not be determined (shouldn't be a problem coming from PostGIS) or you are dealing with some crazy input data, these options can be helpful.

• fill_type
• multi_polygon_union
• process_all_rings

[kristfal]

Hey @flippmoke,

thanks for the solid writeup and tips. We managed to solve this (so far) by updating to the latest node-mapnik and removing unnecessary seams in our dataset (our source data is 'tiled' and we did not dissolve borders between similar layers, which seemed to cause some of the artifacts).

Looking forward to seeing v2 out. Also, a question reg documentation, I can't seem to find any reference to the options you mentioned above in the node-mapnik docs. They might be useful for someone seeing these issues.

Thanks again.

[flippmoke]

@kristfal We know quite well that we do not have excellent documentation around some of these things and there are two reasons for it:

1.) Sometimes when we put in a new option or interface we are doing it so we can test things effectively at Mapbox to figure out if the approach is the correct one, in those situations the interface may not last that long or may change. (Yes, I know this is not ideal but sometimes its the best we can do!) In those situations I don't really want to promise others that a specific interface will last.

2.) This is really the biggest reason more often -- we are not providing documentation up to our personal standards. We are working hard to correct this and have put plans in place to make improvements in this area. Everything from more documentation on the specification to more documentation in mapnik and all of the mapnik modules. In the mean time I am available as a limited resource for any information about technical aspects that might not be documented.

[mapsam]

Hey @kristfal just a heads up we have updated the Node Mapnik documentation now that the 2.0 Mapbox Vector Tile specification has been incorporated into version 3.5.0 of Node Mapnik. http://mapnik.org/node-mapnik/documentation/3.5/#mapnik-vectortile

There are still many things to update, but hopefully you will find more information about the properties that were previously missing. Feel free to submit any issues you find at Node Mapnik's issues.