implement gzip decompression in vtvalidate

[GretaCB]

Accept both compressed and uncompressed buffers, using gzip-hpp to decompress.

Next Actions

• Add gzip-hpp as dep
• Add check to see if buffer is compressed/decompressed

const char * pointer = data.data();
std::size_t size = data.size();
bool c = gzip::is_compressed(pointer, size);

• Implement decompression

// Decompress returns a std::string and decodes both zlib and gzip
const char * compressed_pointer = compressed_data.data();
std::string decompressed_data = gzip::decompress(compressed_pointer, compressed_data.size());

• Add tests

cc @springmeyer

[springmeyer]

👍 Will be great to have this for best perf, lower memory needs, and best security. I think this should hold however on:

• Getting Add new, lower level API gzip-hpp#22 landed (since that provides an API that I think is needed for best efficiency for node c++ addons)
• Getting gzip support in node-cpp-skel
• Then the node-cpp-skel example can be mirrored here (and other node C++ addons we maintain).

/cc @flippmoke

[GretaCB]

Adding from chat:
Would it make sense for gzip-hpp to have a node wrapper? For example, to replace my mess: https://github.com/mapbox/vtvalidate/blob/master/bin/vtvalidate.js#L27-L52 (edited)

mmmm, perhaps that’s two different conversations

1. node wrapper needed in gzip-hpp?
2. add inflate/deflate functionality to gzip-hpp?

[springmeyer]

Would it make sense for gzip-hpp to have a node wrapper?

I've wondered this too. We did once do this with https://github.com/kkaefer/DEPRECATED-node-zlib, but moved away from using that C++ addon once node core added the built-in zlib module. The question is whether we could provide some benefit in a custom addon above node's built-in zlib.

For example, to replace my mess: https://github.com/mapbox/vtvalidate/blob/master/bin/vtvalidate.js#L27-L52 (edited)

I think that code can be simplified. The zlib.gunzipSync should be able to decompress both gzip coded data and deflate coded data. However, this is an assumption and would need to be tested. If gunzipSync is able to decompress both types of coding then it behaves the same as gzip-hpp, which is able to transparently handle both due to the window_bits value at https://github.com/mapbox/gzip-hpp/blob/9ff8e64c1ff84c58a08cd5b9c9337a388b9d3385/include/gzip/decompress.hpp#L28-L36.

Refs mapbox/gzip-hpp#6 for documenting this subtlety.

node wrapper needed in gzip-hpp?
add inflate/deflate functionality to gzip-hpp?

I think we should stick with the original plan of this ticket and aim to use gzip-hpp inside of vtvalidate (and other vt* modules) in order to:

• have the JS function exposed by the node C++ addon be able to support both gzip-coded buffers, deflate-coded buffers, and uncompressed buffers automatically and transparently
• save on the allocation and garbage collection cost of creating a node.Buffer with zlib.gunzip per the reasoning in https://github.com/mapbox/vt-shaver-cpp/issues/47.

However we should benchmark since it is possible I'm wrong in my reasoning at https://github.com/mapbox/vt-shaver-cpp/issues/47.

[GretaCB]

The zlib.gunzipSync should be able to decompress both gzip coded data and deflate coded data.

Ran into this earlier today. While running both zlib and gzip fixtures through zlib.gunzipSync(), the zlib unit test was throwing with incorrect header check which led me down the road of handling zlib and gzip separately.

have the JS function exposed by the node C++ addon

Which function?

Refs mapbox/gzip-hpp#6 for documenting this subtlety.

I never got to this, adding it to my ToDo 👍

[springmeyer]

Ran into this earlier today. While running both zlib and gzip fixtures through zlib/gunzipSyn(), the zlib unit test was throwing with incorrect header check which led me down the road of handling zlib and gzip separately.

Cool, thanks for clarifying that.

have the JS function exposed by the node C++ addon

Which function?

In this case I'm referring to vtvalidate.isValid.

[springmeyer]

Update:

• This is ready to move
• V1.0.0 of gzip-hpp is ready to be released Tag v1.0.0 / Package in Mason gzip-hpp#12
• Examples of using gzip-hpp with vtzero by decoding in the C++ threadpool are available at https://github.com/mapbox/vt-shaver-cpp/pull/147 and implement gzip decompression in vtquery vtquery#73

Next action:

• Wait for gzip-hpp v1.0.0 to be tagged and available in mason: Tag v1.0.0 / Package in Mason gzip-hpp#12
• Integrate it such that vtvalidate seamlessly accepts either gzip compressed vector tiles or uncompressed vector tiles.
• Release a new version of vtvalidate and integrate into mapbox-tile-copy such that https://github.com/mapbox/mapbox-tile-copy/blob/aec8c0e075a8b7dac803a92a13115fe04b28446c/lib/migration-stream.js#L145-L154 can be removed.

[springmeyer]

Done buy @alliecrevier in #18 🎉