Allow Hadoop dataSource inputSpec to be specified multiple times.#5717
Merged
fjy merged 2 commits intoapache:masterfrom May 3, 2018
Merged
Allow Hadoop dataSource inputSpec to be specified multiple times.#5717fjy merged 2 commits intoapache:masterfrom
fjy merged 2 commits intoapache:masterfrom
Conversation
b-slim
reviewed
May 3, 2018
| for (int i = 0; i < segments.size(); i++) { | ||
| final WindowedDataSegment segment = segments.get(i); | ||
| logger.info( | ||
| "Segment %,d/%,d for dataSource[%s] has identifier[%s], interval[%s]", |
Contributor
Author
There was a problem hiding this comment.
This is replacing an older log message that was also at info level so I thought it made sense to keep it that way.
Contributor
There was a problem hiding this comment.
i found it not very very useful to log all the segments especially if dealing with more than dozen but it is okay.
Contributor
|
👍 |
jihoonson
reviewed
May 3, 2018
| #### `multi` | ||
|
|
||
| This is a composing inputSpec to combine other inputSpecs. This inputSpec is used for delta ingestion. Please note that you can have only one `dataSource` as child of `multi` inputSpec. | ||
| This is a composing inputSpec to combine other inputSpecs. This inputSpec is used for delta ingestion. You can also use a `multi` inputSpec to combine data from multiple dataSources. However, each particular dataSource can only be specified one time. |
Contributor
There was a problem hiding this comment.
each particular dataSource can only be specified one time
Out of curiosity, do we have this restriction before this PR? There should be no issue for production environments, but I wonder why this restriction is needed.
Contributor
Author
There was a problem hiding this comment.
Before this PR, you could only do one dataSource, so yeah we still had this restriction.
It's not really strictly necessary to have this restriction - I guess we could support adding multiple input specs referring to the same dataSource, but it would make the code more complicated and I didn't see a clear use case.
jihoonson
approved these changes
May 3, 2018
sathishsri88
pushed a commit
to sathishs/druid
that referenced
this pull request
May 8, 2018
…ache#5717) * Allow Hadoop dataSource inputSpec to be specified multiple times. * Fix test
gianm
added a commit
to implydata/druid-public
that referenced
this pull request
May 11, 2018
…ache#5717) * Allow Hadoop dataSource inputSpec to be specified multiple times. * Fix test
gianm
added a commit
to gianm/druid
that referenced
this pull request
May 22, 2018
This feature was introduced in apache#5717 but it didn't work in production because this magical rewriter code wasn't also modified. Now, it is.
gianm
added a commit
to implydata/druid-public
that referenced
this pull request
May 23, 2018
…apache#5790) This feature was introduced in apache#5717 but it didn't work in production because this magical rewriter code wasn't also modified. Now, it is.
gianm
added a commit
to implydata/druid-public
that referenced
this pull request
May 23, 2018
…apache#5790) This feature was introduced in apache#5717 but it didn't work in production because this magical rewriter code wasn't also modified. Now, it is.
gianm
added a commit
to implydata/druid-public
that referenced
this pull request
Jun 7, 2018
…ache#5717) * Allow Hadoop dataSource inputSpec to be specified multiple times. * Fix test
jihoonson
pushed a commit
to jihoonson/druid
that referenced
this pull request
Jul 5, 2018
…apache#5790) This feature was introduced in apache#5717 but it didn't work in production because this magical rewriter code wasn't also modified. Now, it is.
fjy
pushed a commit
that referenced
this pull request
Jul 5, 2018
leventov
pushed a commit
to metamx/druid
that referenced
this pull request
Jul 20, 2018
…apache#5790) (apache#5942) This feature was introduced in apache#5717 but it didn't work in production because this magical rewriter code wasn't also modified. Now, it is.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
It's useful for combining two datasources with similar schemas into one.