Skip to content

[WIP] druid-hive integration#2880

Closed
navis wants to merge 7 commits intoapache:masterfrom
navis:hive-druid-integration
Closed

[WIP] druid-hive integration#2880
navis wants to merge 7 commits intoapache:masterfrom
navis:hive-druid-integration

Conversation

@navis
Copy link
Copy Markdown
Contributor

@navis navis commented Apr 25, 2016

Based on #2282. This is the first try to integrate druid with hive.

create table <table-name> (<column-schema>...)
STORED BY "io.druid.hive.DruidHiveStorageHandler" 
TBLPROPERTIES (
  "druid.broker.address"="<broker-address>", 
  "druid.datasource"="<datasource>"
);

tested on hive-2.0 with MR/TEZ. needs configuration hive.optimize.index.filter=true (hive-site.xml) and mapreduce.job.user.classpath.first=true (hadoop mapred-site.xml)

@navis navis force-pushed the hive-druid-integration branch from ba464b5 to 25cc89b Compare April 25, 2016 07:44
long maxSize = conf.getLong(CONF_MAX_SPLIT_SIZE, DEFAULT_MAX_SPLIT_SIZE);

if (maxSize > 0) {
Collections.shuffle(segments);
Copy link
Copy Markdown
Contributor

@jaehc jaehc Apr 25, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am just wondering it is necessary to shuffle the segments list?

@xvrl
Copy link
Copy Markdown
Member

xvrl commented Apr 25, 2016

@navis it's a bit confusing to have the same code changes in two different PRs. We're getting comments in both, which makes it hard to track the ones that are addressed or not. Maybe we should close one of the PRs?

@navis
Copy link
Copy Markdown
Contributor Author

navis commented Apr 26, 2016

@xvrl Sorry, I thought WIP in title can make others skip reviewing. I'll close this, for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants