Skip to content

Add metadata indexes to help with segment allocation. #6348

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
fjy merged 2 commits into from
Sep 19, 2018
Merged

Add metadata indexes to help with segment allocation. #6348

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1194c46
Add metadata indexes to help with segment allocation.
gianm Jul 11, 2018
1e16d83
Merge branch 'master' into sql-indexes
gianm Sep 19, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 15 additions & 3 deletions server/src/main/java/org/apache/druid/metadata/SQLMetadataConnector.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,11 +23,11 @@
import com.google.common.base.Supplier;
import com.google.common.base.Throwables;
import com.google.common.collect.ImmutableList;
import org.apache.commons.dbcp2.BasicDataSource;
import org.apache.druid.java.util.common.ISE;
import org.apache.druid.java.util.common.RetryUtils;
import org.apache.druid.java.util.common.StringUtils;
import org.apache.druid.java.util.common.logger.Logger;
import org.apache.commons.dbcp2.BasicDataSource;
import org.skife.jdbi.v2.Batch;
import org.skife.jdbi.v2.DBI;
import org.skife.jdbi.v2.Handle;
Expand Down Expand Up @@ -215,6 +215,11 @@ public void createPendingSegmentsTable(final String tableName)
+ " UNIQUE (sequence_name_prev_id_sha1)\n"
+ ")",
tableName, getPayloadType(), getQuoteString()
),
StringUtils.format(
"CREATE INDEX idx_%1$s_datasource_used_end ON %1$s(dataSource, used, %2$send%2$s)",
Copy link
Copy Markdown
Member

@QiuMM QiuMM Sep 20, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No used field of pendingSegmentsTable . You placed these code in wrong place. Just exchange their location will be good.

tableName,
getQuoteString()
)
)
);
Expand Down Expand Up @@ -259,8 +264,15 @@ public void createSegmentTable(final String tableName)
+ ")",
tableName, getPayloadType(), getQuoteString()
),
StringUtils.format("CREATE INDEX idx_%1$s_datasource ON %1$s(dataSource)", tableName),
StringUtils.format("CREATE INDEX idx_%1$s_used ON %1$s(used)", tableName)
StringUtils.format(
"CREATE INDEX idx_%1$s_datasource_end ON %1$s(dataSource, %2$send%2$s)",
tableName,
getQuoteString()
),
StringUtils.format(
"CREATE INDEX idx_%1$s_datasource_sequence ON %1$s(dataSource, sequence_name)",
Copy link
Copy Markdown
Member

@QiuMM QiuMM Sep 20, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gianm sorry, I did not review these code carefully. There are no sequence_name of segmentTable, pendingSegmentsTable does...

Copy link
Copy Markdown
Contributor Author

@gianm gianm Sep 20, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ouch, you're right. Thanks for the catch- I will fix it. I'm surprised the tests didn't catch it - I thought they ran these create table commands. I will look into that too.

Copy link
Copy Markdown
Contributor Author

@gianm gianm Sep 20, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Raised #6356 to fix this. The PR description also explains why the tests didn't catch this.

tableName
)
)
);
}
Expand Down