update node-mapnik benching to use rules + run bench

[millzpaugh]

Benching Results

TL;DR
The results are interesting - sometimes node-mapnik’s composite function is faster, sometimes vtcomposite is faster.

Note: The node-mapnik composite function must take a vector tile as a param - so I commented out the first single in-out rule when benching.

node-mapnik

~/mapbox/vtcomposite[master]$ node ./bench/node-mapnik.bench.js --iterations 1000 --concurrency 1

1: two different tiles at the same zoom level, zero buffer ... 1538 runs/s (650ms)
2: two different tiles from different zoom levels (separated by one zoom), zero buffer ... 5 runs/s (196657ms)
3: two different tiles from different zoom levels (separated by more than one zoom), zero buffer ... 5 runs/s (196789ms)
4: tiles completely made of points, overzooming, no properties ... 13699 runs/s (73ms)
5: tiles completely made of points, same zoom, no properties ... 15152 runs/s (66ms)
6: tiles completely made of points, overzoooming, lots of properties ... 13333 runs/s (75ms)
7: tiles completely made of points, same zoom, lots of properties ... 11628 runs/s (86ms)
8: tiles completely made of linestrings, overzooming and lots of properties ... 11905 runs/s (84ms)
9: tiles completely made of polygons, overzooming and lots of properties ... 9709 runs/s (103ms)

vtcomposite

~/mapbox/vtcomposite[master]$ node bench/vtcomposite.bench.js --iterations 1000 --concurrency 1

1: two different tiles at the same zoom level, zero buffer ... 52 runs/s (19156ms)
2: two different tiles from different zoom levels (separated by one zoom), zero buffer ... 43 runs/s (23081ms)
3: two different tiles from different zoom levels (separated by more than one zoom), zero buffer ... 45 runs/s (22329ms)
4: tiles completely made of points, overzooming, no properties ... 2370 runs/s (422ms)
5: tiles completely made of points, same zoom, no properties ... 3731 runs/s (268ms)
6: tiles completely made of points, overzoooming, lots of properties ... 1437 runs/s (696ms)
7: tiles completely made of points, same zoom, lots of properties ... 1135 runs/s (881ms)
8: tiles completely made of linestrings, overzooming and lots of properties ... 612 runs/s (1635ms)
9: tiles completely made of polygons, overzooming and lots of properties ... 130 runs/s (7716ms)

cc @mapsam @artemp

[artemp]

@millzpaugh thanks for putting this together! I'm a bit baffled by results, we need to investigate.

[artemp]

after abb70c5

node-mapnik

 node ./bench/node-mapnik.bench.js --iterations 50 --concurrency 10
1: two different tiles at the same zoom level, zero buffer ... 1613 runs/s (31ms)
2: two different tiles from different zoom levels (separated by one zoom), zero buffer ... 379 runs/s (132ms)
3: two different tiles from different zoom levels (separated by more than one zoom), zero buffer ... 676 runs/s (74ms)
4: tiles completely made of points, overzooming, no properties ... 12500 runs/s (4ms)
5: tiles completely made of points, same zoom, no properties ... 16667 runs/s (3ms)
6: tiles completely made of points, overzoooming, lots of properties ... 10000 runs/s (5ms)
7: tiles completely made of points, same zoom, lots of properties ... 10000 runs/s (5ms)
8: tiles completely made of linestrings, overzooming and lots of properties ... 365 runs/s (137ms)
9: tiles completely made of polygons, overzooming and lots of properties ... 134 runs/s (374ms)

vtcomposite

node ./bench/vtcomposite.bench.js --iterations 50 --concurrency 10

1: two different tiles at the same zoom level, zero buffer ... 77 runs/s (653ms)
2: two different tiles from different zoom levels (separated by one zoom), zero buffer ... 75 runs/s (671ms)
3: two different tiles from different zoom levels (separated by more than one zoom), zero buffer ... 76 runs/s (654ms)
4: tiles completely made of points, overzooming, no properties ... 5000 runs/s (10ms)
5: tiles completely made of points, same zoom, no properties ... 5556 runs/s (9ms)
6: tiles completely made of points, overzoooming, lots of properties ... 2778 runs/s (18ms)
7: tiles completely made of points, same zoom, lots of properties ... 1852 runs/s (27ms)
8: tiles completely made of linestrings, overzooming and lots of properties ... 1667 runs/s (30ms)
9: tiles completely made of polygons, overzooming and lots of properties ... 397 runs/s (126ms)

[millzpaugh]

@artemp I realized that you're setting --iterations 50 --concurrency 10 - I think we'll want to follow vtshaver's benching and use --iterations something --concurrency 1 to ensure we have consistent results. (Or at least agree on a concurrency for everyone to use so we compare 🍏 to 🍏 and not 🍏 to 🍎.)

Also copying your comment about node-mapnik's performance on tests 4 - 7 looking way too fast for reference
@artemp :

I think I understand now why node-mapnik is x100 faster when overzooming two tiles with the same layer ('polygons') than vtcomposite.node-mapnik doesn't do any work in this case and just returns input tile, vtcomposite on the other hand builds output tile from input tile. We need to introduce similar to node-mapnik short-cut logic in vtcomposite. Meanwhile, it would be great to test overzooming on input with different layers e.g polygons-1andpolygons-2to trigger some real work innode-mapnik`

Captured ^ here: #13