flattenSpec: unified way to access elements of complex data (JSON, Avro, ProtoBuf, Thrift, ...)

[knoguchi]

I would like to be able to ingest ProtoBuf / Avro using flattenSpec.
flattenSpec uses JSON Path but it's generally applicable to any structured data.

I've read the source of ProtoBufInputRowParser, and probably I can add some hack to buildStringKeyMap. But there should be more generic way to implement this feature.

By the way I definitely want to avoid converting the PB to JSON then apply JSON Path. Any hints?

[knoguchi]

I just realized there are PR's already. All of them try to support complex data but the access to the nested elements are all different.

#2354 Protobuf parser: nested messages and fileds, define any message name to parse
#2414 Support pulling values from complex avro data structures
#3418 [Feature] Thrift support for realtime and batch ingestion

There are some discussions whether to use JSON Path or to use native expression e.g. Avro has its own expression.

[du00cs]

Hi, I am from #3418 . All formats above can be converted to json easily, unified way should come with unified intermediate format first. JSON Path may be a good choice.

[knoguchi]

Now, I'm working on adding ETL capability to Druid, and considering Jackson JsonNode as the intermediate object. I'm thinking of adding flattenSpec type "jq" and accept the jq expression in the "expr".

I would like to introduce two modules.

• Jacskon standard binary dataformat backends
• Jackson-jq

One consideration is that we should NOT use the JSON string as the intermediate format. It costs serializing the native object, and then deserializing back to the common object. The Jackson binary backend should help avoiding this problem (needs research). It supports Avro, Protobuf and two other formats out of the box. Unfortunately Thrift support is left untouched for 4 years(!) though.

As for the jq, it's a "Swiss Army knife for JSON". It's widely accepted and proven to be capable for filtering and transforming the JSON. There are two implementations for Java. One is the Jackson-jq which is pure Java, the other is jjq JNA wrapper for libjq. Obviously the latter is compatible with the original jq but JNI/JNA is not very flexible.

[knoguchi]

We can close this issue as the jq support has been merged.