DruidInputSource can add new dimensions during re-ingestion by suneet-s · Pull Request #9590 · apache/druid

suneet-s · 2020-03-31T01:19:59Z

Description

Add integration tests for using transformSpec during ingestion. The tests test transformSpecs that add metrics and dimensions using both a Druid InputSource and the deprecated ingestSegmentFirehose.

Writing these tests surfaced #9589 #9591 When these issues are fixed, those tests can be re-enabled.

suneet-s · 2020-03-31T01:41:09Z

+            "version" : "v1",
+            "timestamp" : "2013-08-31T00:00:00.000Z",
+            "event" : {
+                "added_count_times_ten" : 9050.0,


typo in expected results

Suggested change

"added_count_times_ten" : 9050.0,

"added_count_times_ten" : 27150.0,

maytasm · 2020-03-31T04:39:41Z

Can you parameterize all the query and index json files? I think it's much harder to keep track of many many many different query and index json files we have for IT. Especially when they are 80-90% the same.

suneet-s · 2020-03-31T05:50:36Z

Can you parameterize all the query and index json files? I think it's much harder to keep track of many many many different query and index json files we have for IT. Especially when they are 80-90% the same.

Across all AbstractITBatchIndexTest or within the new tests I've added in ITTransformTest?

It's hard to parameterize ITTransformTest because it's broken for most combinations. And there are 2 categories of tests. I think properly parameterizing these tests requires a re-think of what the test environment should look like because all the re-indexing tests start by indexing a datasource which slows down the test execution.

maytasm · 2020-03-31T06:46:12Z

Can you parameterize all the query and index json files? I think it's much harder to keep track of many many many different query and index json files we have for IT. Especially when they are 80-90% the same.

Across all AbstractITBatchIndexTest or within the new tests I've added in ITTransformTest?

It's hard to parameterize ITTransformTest because it's broken for most combinations. And there are 2 categories of tests. I think properly parameterizing these tests requires a re-think of what the test environment should look like because all the re-indexing tests start by indexing a datasource which slows down the test execution.

I am thinking of parameterizing the json files themself across the new json files you added

suneet-s · 2020-03-31T14:52:22Z

I am thinking of parameterizing the json files themself across the new json files you added

I think parameterizing the json files is a great idea, but there's a lot of tight coupling between the json files and the base IT classes. I'm not sure where the best place to do this is because of that.

I'd rather do the parameterization in a separate change - when #9591 #9592 or #9589 are fixed

jihoonson

LGTM

jihoonson · 2020-04-01T21:14:26Z

        "filter" : "transform-data.json"
      },
-      "inpuFormat" : {
+      "inputFormat" : {


Good catch!

jihoonson · 2020-04-01T21:16:11Z

+   * @param explicitDimensions sent as part of the re-ingestion InputSource.
+   * @param dimensionsSpec from the provided ingestion spec.
+   * @param timeLineSegments for the datasource that is being read.
+   * @return


nit: please add description for the return value or remove this.

) * WIP integration tests * Add integration test for ingestion with transformSpec * WIP almost working tests * Add ignored tests * checkstyle stuff * remove newPage from index task ingestion spec * more test cleanup * still not quite working * Actually disable the tests * working tests * fix codestyle * dont use junit in integration tests * actually fix the bug * fix checkstyle * bring index tests closer to reindex tests

…9590) (#9606) * DruidInputSource can add new dimensions during re-ingestion (#9590) * WIP integration tests * Add integration test for ingestion with transformSpec * WIP almost working tests * Add ignored tests * checkstyle stuff * remove newPage from index task ingestion spec * more test cleanup * still not quite working * Actually disable the tests * working tests * fix codestyle * dont use junit in integration tests * actually fix the bug * fix checkstyle * bring index tests closer to reindex tests * Remove QUICKSTART_COMPATIBLE

) * WIP integration tests * Add integration test for ingestion with transformSpec * WIP almost working tests * Add ignored tests * checkstyle stuff * remove newPage from index task ingestion spec * more test cleanup * still not quite working * Actually disable the tests * working tests * fix codestyle * dont use junit in integration tests * actually fix the bug * fix checkstyle * bring index tests closer to reindex tests

suneet-s added 2 commits March 30, 2020 17:35

WIP integration tests

6a4ee82

Add integration test for ingestion with transformSpec

045584b

suneet-s changed the title ~~Transform reindex~~ Add integration test for ingestion with transformSpec Mar 31, 2020

suneet-s commented Mar 31, 2020

View reviewed changes

suneet-s added 3 commits March 30, 2020 20:00

WIP almost working tests

46ae13b

Add ignored tests

608e568

checkstyle stuff

0f7dd96

suneet-s changed the title ~~Add integration test for ingestion with transformSpec~~ Add integration tests for ingestion with transformSpec Mar 31, 2020

suneet-s added 4 commits March 30, 2020 20:58

remove newPage from index task ingestion spec

f6f10aa

more test cleanup

013150d

still not quite working

db4dc28

Actually disable the tests

8d2c471

suneet-s commented Mar 31, 2020

View reviewed changes

Comment thread integration-tests/src/test/resources/indexer/wikipedia_index_queries_with_transform.json Outdated

suneet-s added 2 commits March 30, 2020 23:29

working tests

9da1c53

fix codestyle

7afad45

clintropolis added the Area - Testing label Mar 31, 2020

dont use junit in integration tests

65216e3

suneet-s added 3 commits March 31, 2020 14:18

actually fix the bug

fc8fedd

fix checkstyle

53433a2

bring index tests closer to reindex tests

d5ee5ef

suneet-s changed the title ~~Add integration tests for ingestion with transformSpec~~ DruidInputSource can add new dimensions during re-ingestion Mar 31, 2020

maytasm approved these changes Apr 1, 2020

View reviewed changes

jihoonson added the Bug label Apr 1, 2020

jihoonson approved these changes Apr 2, 2020

View reviewed changes

ccaominh approved these changes Apr 3, 2020

View reviewed changes

ccaominh merged commit af3337d into apache:master Apr 3, 2020

suneet-s mentioned this pull request Apr 3, 2020

[0.18.0] DruidInputSource can add new dimensions during re-ingestion (#9590) #9606

Merged

suneet-s deleted the transform-reindex branch April 3, 2020 05:17

jihoonson added this to the 0.18.0 milestone Apr 3, 2020

suneet-s mentioned this pull request Apr 9, 2020

Re-indexing a DruidInputSource with explicit dimensions includes all dimensions from original datasource #9592

Closed

jihoonson mentioned this pull request Apr 19, 2020

[Draft] 0.18.0 release notes #9652

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DruidInputSource can add new dimensions during re-ingestion#9590

DruidInputSource can add new dimensions during re-ingestion#9590
ccaominh merged 15 commits intoapache:masterfrom
suneet-s:transform-reindex

suneet-s commented Mar 31, 2020 •

edited

Loading

Uh oh!

suneet-s Mar 31, 2020

Uh oh!

Uh oh!

maytasm commented Mar 31, 2020

Uh oh!

suneet-s commented Mar 31, 2020

Uh oh!

maytasm commented Mar 31, 2020

Uh oh!

suneet-s commented Mar 31, 2020

Uh oh!

jihoonson left a comment

Uh oh!

jihoonson Apr 1, 2020

Uh oh!

jihoonson Apr 1, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

	"added_count_times_ten" : 9050.0,
	"added_count_times_ten" : 27150.0,

Conversation

suneet-s commented Mar 31, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

suneet-s Mar 31, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

maytasm commented Mar 31, 2020

Uh oh!

suneet-s commented Mar 31, 2020

Uh oh!

maytasm commented Mar 31, 2020

Uh oh!

suneet-s commented Mar 31, 2020

Uh oh!

jihoonson left a comment

Choose a reason for hiding this comment

Uh oh!

jihoonson Apr 1, 2020

Choose a reason for hiding this comment

Uh oh!

jihoonson Apr 1, 2020

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

suneet-s commented Mar 31, 2020 •

edited

Loading