Skip to content

Add check for required avroBytesDecoder property that otherwise causes NPE.#14177

Merged
zachjsh merged 2 commits intoapache:masterfrom
abhishekrb19:fix_avro_sampling_npe
May 3, 2023
Merged

Add check for required avroBytesDecoder property that otherwise causes NPE.#14177
zachjsh merged 2 commits intoapache:masterfrom
abhishekrb19:fix_avro_sampling_npe

Conversation

@abhishekrb19
Copy link
Copy Markdown
Contributor

This PR fixes an NPE that occurs when avroBytesDecoder is missing in the sampling spec. This is the default behavior from the Druid web console when sampling Avro Kafka streams, unless you manually edit the spec to add in the required property.

The NPE stacktrace:

Caused by: java.lang.NullPointerException: Cannot invoke "org.apache.druid.data.input.avro.AvroBytesDecoder.parse(java.nio.ByteBuffer)" because "this.avroBytesDecoder" is null
	at org.apache.druid.data.input.avro.AvroStreamReader.intermediateRowIterator(AvroStreamReader.java:78) ~[?:?]
	at org.apache.druid.data.input.IntermediateRowParsingReader.intermediateRowIteratorWithMetadata(IntermediateRowParsingReader.java:231) ~[druid-processing-2023.05.0-iap-SNAPSHOT.jar:2023.05.0-iap-SNAPSHOT]
	at org.apache.druid.data.input.IntermediateRowParsingReader.sample(IntermediateRowParsingReader.java:124) ~[druid-processing-2023.05.0-iap-SNAPSHOT.jar:2023.05.0-iap-SNAPSHOT]
	at org.apache.druid.data.input.impl.InputEntityIteratingReader.lambda$sample$1(InputEntityIteratingReader.java:98) ~[druid-processing-2023.05.0-iap-SNAPSHOT.jar:2023.05.0-iap-SNAPSHOT]
	at org.apache.druid.java.util.common.parsers.CloseableIterator$2.findNextIteratorIfNecessary(CloseableIterator.java:84) ~[druid-processing-2023.05.0-iap-SNAPSHOT.jar:2023.05.0-iap-SNAPSHOT]
	at org.apache.druid.java.util.common.parsers.CloseableIterator$2.<init>(CloseableIterator.java:69) ~[druid-processing-2023.05.0-iap-SNAPSHOT.jar:2023.05.0-iap-SNAPSHOT]
	at org.apache.druid.java.util.common.parsers.CloseableIterator.flatMap(CloseableIterator.java:67) ~[druid-processing-2023.05.0-iap-SNAPSHOT.jar:2023.05.0-iap-SNAPSHOT]
	at org.apache.druid.data.input.impl.InputEntityIteratingReader.createIterator(InputEntityIteratingReader.java:108) ~[druid-processing-2023.05.0-iap-SNAPSHOT.jar:2023.05.0-iap-SNAPSHOT]
	at org.apache.druid.data.input.impl.InputEntityIteratingReader.sample(InputEntityIteratingReader.java:94) ~[druid-processing-2023.05.0-iap-SNAPSHOT.jar:2023.05.0-iap-SNAPSHOT]
	at org.apache.druid.data.input.impl.TimedShutoffInputSourceReader.sample(TimedShutoffInputSourceReader.java:62) ~[druid-processing-2023.05.0-iap-SNAPSHOT.jar:2023.05.0-iap-SNAPSHOT]
	at org.apache.druid.segment.transform.TransformingInputSourceReader.sample(TransformingInputSourceReader.java:50) ~[druid-processing-2023.05.0-iap-SNAPSHOT.jar:2023.05.0-iap-SNAPSHOT]
	at org.apache.druid.indexing.overlord.sampler.InputSourceSampler.sample(InputSourceSampler.java:129) ~[druid-indexing-service-2023.05.0-iap-SNAPSHOT.jar:2023.05.0-iap-SNAPSHOT]

Screenshot before the fix:
CleanShot 2023-04-26 at 13 42 47@2x

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

Copy link
Copy Markdown
Contributor

@zachjsh zachjsh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@abhishekrb19
Copy link
Copy Markdown
Contributor Author

I see some unrelated test failures:

Error:  Errors: 
Error:  org.apache.druid.client.HttpServerInventoryViewTest.testSyncMonitoring
Error:    Run 1: HttpServerInventoryViewTest.testSyncMonitoring:330 » RejectedExecution Service not started.
Error:    Run 2: HttpServerInventoryViewTest.testSyncMonitoring:330 » RejectedExecution Service not started.
Error:    Run 3: HttpServerInventoryViewTest.testSyncMonitoring:330 » RejectedExecution Service not started.
Error:    Run 4: HttpServerInventoryViewTest.testSyncMonitoring:330 » RejectedExecution Service not started.

They appear to be fixed in #14180

@zachjsh zachjsh merged commit 954f391 into apache:master May 3, 2023
@abhishekrb19 abhishekrb19 deleted the fix_avro_sampling_npe branch May 3, 2023 18:11
@abhishekagarwal87 abhishekagarwal87 added this to the 27.0 milestone Jul 19, 2023
@abhishekagarwal87 abhishekagarwal87 changed the title Fix NPE when sampling Avro streams. Add check for required avroBytesDecoder property that otherwise causes NPE. Jul 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants