Skip to content

Using annotation to distinguish Hadoop Configuration in each module#9013

Merged
jihoonson merged 6 commits intoapache:masterfrom
jihoonson:bind-annotation-configuration
Dec 12, 2019
Merged

Using annotation to distinguish Hadoop Configuration in each module#9013
jihoonson merged 6 commits intoapache:masterfrom
jihoonson:bind-annotation-configuration

Conversation

@jihoonson
Copy link
Copy Markdown
Contributor

@jihoonson jihoonson commented Dec 11, 2019

Fixes #9004.

Description

This PR is based on #9005.

Each extension module needs to properly bind whatever it will use, but sometimes different modules need to bind the same class which will lead to the duplicate injection error. To avoid this problem, each module is supposed to bind different instances. This PR adds new annotations to bind different instances.

Also ParquetReader is changed to use the injected Configuration.

This PR is tested with

  • Native orc ingestion
  • Native paquet ingestion
  • Hadoop text file ingestion
  • Hadoop orc ingestion
  • Hadoop parquet ingestion

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths.
  • added integration tests.
  • been tested in a test Druid cluster.

Copy link
Copy Markdown
Member

@clintropolis clintropolis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm 👍

@jihoonson
Copy link
Copy Markdown
Contributor Author

@himanshug @clintropolis thank you for the review!

@jihoonson jihoonson merged commit 66056b2 into apache:master Dec 12, 2019
@jnaous
Copy link
Copy Markdown
Contributor

jnaous commented Dec 12, 2019

@jihoonson should this be backported to the 0.17 branch.

@jihoonson
Copy link
Copy Markdown
Contributor Author

@jnaous oh yeah, I will do it.

jihoonson added a commit to jihoonson/druid that referenced this pull request Dec 13, 2019
…pache#9013)

* Multibinding for NodeRole

* Fix endpoints

* fix doc

* fix test

* Using annotation to distinguish Hadoop Configuration in each module
jon-wei pushed a commit that referenced this pull request Dec 13, 2019
…9013) (#9024)

* Multibinding for NodeRole

* Fix endpoints

* fix doc

* fix test

* Using annotation to distinguish Hadoop Configuration in each module
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Hadoop task fails because of duplicate injection of Configuration

4 participants