Skip to content

Conversation

@nicoloboschi
Copy link
Contributor

Motivation

Avro native objects for Arrays and Maps are not supported by the sink.

If you try to use a Java POJO with a java.util.List/java.util.Set field with Avro Schema, you get:

022-04-26T11:56:51.477776276Z Caused by: java.lang.ClassCastException: class org.apache.avro.generic.GenericData$Array cannot be cast to class [Ljava.lang.Object; (org.apache.avro.generic.GenericData$Array is in unnamed module of loader 'app'; [Ljava.lang.Object; is in module java.base of loader 'bootstrap')
2022-04-26T11:56:51.477782027Z 	at org.apache.pulsar.io.elasticsearch.JsonConverter.toJson(JsonConverter.java:90) ~[?:?

For java.util.Map

ERROR org.apache.pulsar.io.elasticsearch.ElasticSearchSink - write error:
2022-04-26T12:03:34.776219894Z java.lang.IllegalArgumentException: Error while converting a value of type class java.util.HashMap to a MAP: java.lang.ClassCastException: class org.apache.avro.util.Utf8 cannot be cast to class java.lang.String (org.apache.avro.util.Utf8 is in unnamed module of loader 'app'; java.lang.String is in module java.base of loader 'bootstrap')

Modifications

  • Handle GenericData.Array instances
  • Handle Map with UUID as entry key
  • no-need-doc

@nicoloboschi
Copy link
Contributor Author

/pulsarbot rerun-failure-checks

@nicoloboschi nicoloboschi requested review from dlg99, eolivelli and hezhangjian and removed request for eolivelli and hezhangjian May 5, 2022 08:09
nicoloboschi added a commit to datastax/pulsar that referenced this pull request May 10, 2022
…icData.Array and Utf8 map keys) (apache#15430)

(cherry picked from commit 3717b9e)
@nicoloboschi
Copy link
Contributor Author

@lhotari @dlg99 @eolivelli PTAL

@nicoloboschi nicoloboschi added this to the 2.11.0 milestone May 23, 2022
@nicoloboschi
Copy link
Contributor Author

/pulsarbot rerun-failure-checks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/connector doc-not-needed Your PR changes do not impact docs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants