Skip to content

Fix flattening Avro Maps with Utf8 keys#7258

Merged
gianm merged 2 commits intoapache:masterfrom
hardisgroupcom:master-5884
Mar 18, 2019
Merged

Fix flattening Avro Maps with Utf8 keys#7258
gianm merged 2 commits intoapache:masterfrom
hardisgroupcom:master-5884

Conversation

@chrishardis
Copy link
Copy Markdown
Contributor

Fix issue #5884 by @varaga @jon-wei on master branch.
Successfully tested with Confluent 5.1.0 kafka and schema registry

@fjy fjy added this to the 0.15.0 milestone Mar 13, 2019
Copy link
Copy Markdown
Member

@clintropolis clintropolis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution! 👍

It looks like your PR has some style issues and is failing CI

[ERROR] /home/travis/build/apache/incubator-druid/extensions-core/avro-extensions/src/main/java/org/apache/druid/data/input/avro/GenericAvroJsonProvider.java:142:77: '{' at column 77 should be on a new line. [LeftCurly]
[ERROR] /home/travis/build/apache/incubator-druid/extensions-core/avro-extensions/src/main/java/org/apache/druid/data/input/avro/GenericAvroJsonProvider.java:158:50: ',' is not followed by whitespace. [WhitespaceAfter]

Could you also please add a test case for this issue? It looks lgtm otherwise.

@clintropolis
Copy link
Copy Markdown
Member

It would also appear that you have opened a duplicate PR, #7254.



private Object transformAvroMapKey(String keyToTransform, Map avroMapObj) {
Object key = avroMapObj.keySet().iterator().next();
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this work if the map is empty?

It might be more straightforward to just do something like this in getMapValue

...
} else if (o instanceof Map) {
  final Map theMap = (Map) o;
  if (theMap.containsKey(s)) {
    return theMap.get(s);
  } else {
    final Utf8 utf8Key = new Utf8(s);
    return theMap.get(utf8Key);
  }
}
...

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

your suggestion is more elegant, I will commit it. Thks

@varaga
Copy link
Copy Markdown
Contributor

varaga commented Mar 14, 2019

Thanks @chrishardis for taking over the bug and committing it to master.
Would you be able to add some tests to cover the bug if it's feasible.

@chrishardis
Copy link
Copy Markdown
Contributor Author

I spend my morning trying to run a test case, but it's not so easy has using the io.confluent.kafka.serializers.KafkaAvroSerializer without a kafkaproducer is quite an hard job. I test it many times on real Confluent 5.1 Kafka Cluster with Schema Registry.

@clintropolis
Copy link
Copy Markdown
Member

I spend my morning trying to run a test case, but it's not so easy has using the io.confluent.kafka.serializers.KafkaAvroSerializer without a kafkaproducer is quite an hard job. I test it many times on real Confluent 5.1 Kafka Cluster with Schema Registry.

I think this issue would exist for any type of Avro indexing; like if you could create an Avro formatted file with a column that is a map type with utf8 keys, you could test with the Hadoop indexing tests if that would be easier.

That said, this change seems small and harmless enough that I think it would probably be ok without tests, so I will go ahead and approve.

@clintropolis clintropolis changed the title #5884 Avro Maps can't be flattened in Kafka Indexing on master Fix flattening Avro Maps with Utf8 keys Mar 18, 2019
@gianm gianm merged commit b2a940a into apache:master Mar 18, 2019
clintropolis pushed a commit that referenced this pull request Apr 24, 2019
* #5884 Avro Maps can't be flattened in Kafka Indexing on master

* correct typo and reduce code size
@clintropolis clintropolis modified the milestones: 0.15.0, 0.14.1 Apr 24, 2019
gianm pushed a commit to implydata/druid-public that referenced this pull request May 10, 2019
* apache#5884 Avro Maps can't be flattened in Kafka Indexing on master

* correct typo and reduce code size
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants