Skip to content

[fix][kinesis-sink] Handle Avro collections native types (GenericData.Array and Utf8 map keys) #15432

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
nicoloboschi merged 2 commits into from
May 31, 2022
Merged

[fix][kinesis-sink] Handle Avro collections native types (GenericData.Array and Utf8 map keys) #15432

nicoloboschi merged 2 commits into from
May 31, 2022

Conversation

@nicoloboschi
Copy link
Contributor

@nicoloboschi nicoloboschi commented May 4, 2022

Motivation

Avro native objects for Arrays and Maps are not supported by the sink.

If you try to use a Java POJO with a java.util.List/java.util.Set field with Avro Schema, you get:

022-04-26T11:56:51.477776276Z Caused by: java.lang.ClassCastException: class org.apache.avro.generic.GenericData$Array cannot be cast to class [Ljava.lang.Object; (org.apache.avro.generic.GenericData$Array is in unnamed module of loader 'app'; [Ljava.lang.Object; is in module java.base of loader 'bootstrap')

For java.util.Map

2022-04-26T12:03:34.776219894Z java.lang.IllegalArgumentException: Error while converting a value of type class java.util.HashMap to a MAP: java.lang.ClassCastException: class org.apache.avro.util.Utf8 cannot be cast to class java.lang.String (org.apache.avro.util.Utf8 is in unnamed module of loader 'app'; java.lang.String is in module java.base of loader 'bootstrap')

Modifications

  • Handle GenericData.Array instances
  • Handle Map with UUID as entry key

See the same issue in the ElasticSearch sink #15430

  • no-need-doc

@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label May 4, 2022
nicoloboschi added a commit to datastax/pulsar that referenced this pull request May 10, 2022
@nicoloboschi
[fix][kinesis-sink] Handle Avro collections native types (GenericData…
91ab03c 
….Array and Utf8 map keys) (apache#15432)

* [fix][kinesis-sink] Handle Avro collections native types (GenericData.Array and Utf8 map keys)

* Fix test
@nicoloboschi
Copy link
Contributor Author

nicoloboschi commented May 17, 2022

@dlg99 @cbornet PTAL

@nicoloboschi nicoloboschi added this to the 2.11.0 milestone May 23, 2022
@nicoloboschi nicoloboschi reopened this May 23, 2022
@nicoloboschi
Copy link
Contributor Author

nicoloboschi commented May 23, 2022

/pulsarbot rerun-failure-checks

@nicoloboschi nicoloboschi reopened this May 30, 2022
@nicoloboschi nicoloboschi mentioned this pull request May 30, 2022
Merged
5 tasks
@nicoloboschi
[fix][kinesis-sink] Handle Avro collections native types (GenericData…
1d8a2df 
….Array and Utf8 map keys)

* [fix][kinesis-sink] Handle Avro collections native types (GenericData.Array and Utf8 map keys)

* Fix test
@nicoloboschi
fix import
7444261
@nicoloboschi nicoloboschi force-pushed the kinesis-collections branch from 2b4ce79 to 7444261 Compare May 31, 2022 09:41
@cbornet
Copy link
Contributor

cbornet commented May 31, 2022

LGTM

@cbornet
Copy link
Contributor

cbornet commented May 31, 2022

/pulsarbot rerun-failure-checks

@nicoloboschi nicoloboschi merged commit ba07cb8 into apache:master May 31, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@eolivelli eolivelli eolivelli approved these changes

Assignees

@nicoloboschi nicoloboschi

Labels

area/connector doc-not-needed Your PR changes do not impact docs

Projects

None yet

Milestone

2.11.0

Development

Successfully merging this pull request may close these issues.

3 participants

@nicoloboschi @cbornet @eolivelli