Skip to content

Optimize SegmentsCostCache #12419

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
AmatyaAvadhanula wants to merge 13 commits into from
Closed

Optimize SegmentsCostCache #12419

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
522f1df
Add Treap implementation
AmatyaAvadhanula Apr 11, 2022
3ce21a5
Make Treap generic
AmatyaAvadhanula Apr 11, 2022
10f9aec
Segment cost strategy using treap
AmatyaAvadhanula Apr 11, 2022
93c9d0b
Store Intervals instead of DataSegments in Buckets
AmatyaAvadhanula Apr 13, 2022
4575bba
Optimize computation of segment bucket cost
AmatyaAvadhanula Apr 14, 2022
18baa63
Use new cost code and cleanup
AmatyaAvadhanula Apr 14, 2022
65f085a
Refactoring, cleaning, improved V3
AmatyaAvadhanula Apr 15, 2022
7ae1a6a
revert unnecessary changes
AmatyaAvadhanula Apr 15, 2022
45dcc14
Make SegmentsCostCache equivalent to CostBalancerStrategy
AmatyaAvadhanula Apr 16, 2022
aad54e8
Faster equivalent implementation with tests
AmatyaAvadhanula Apr 16, 2022
26a898a
Handle all, adhoc granularities without overflow
AmatyaAvadhanula Apr 17, 2022
918ff9e
Fix checkstyle, remove SegmentsCostCacheV2
AmatyaAvadhanula Apr 17, 2022
9296063
Enable reuse of builder in V3
AmatyaAvadhanula Apr 18, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 6 additions & 10 deletions server/src/main/java/org/apache/druid/server/coordinator/cost/SegmentsCostCache.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,9 +93,10 @@ public class SegmentsCostCache
* The value of 1 day means that cost function of co-locating two segments which have 1 days between their intervals
* is 0.5 of the cost, if the intervals are adjacent. If the distance is 2 days, then 0.25, etc.
*/
private static final double HALF_LIFE_DAYS = 1.0;
private static final double LAMBDA = Math.log(2) / HALF_LIFE_DAYS;
private static final double MILLIS_FACTOR = TimeUnit.DAYS.toMillis(1) / LAMBDA;
private static final double HALF_LIFE_HOURS = 24.0;
private static final double LAMBDA = Math.log(2) / HALF_LIFE_HOURS;
static final double NORMALIZATION_FACTOR = 1 / (LAMBDA * LAMBDA);
private static final double MILLIS_FACTOR = TimeUnit.HOURS.toMillis(1) / LAMBDA;

/**
* LIFE_THRESHOLD is used to avoid calculations for segments that are "far"
Expand Down Expand Up @@ -156,7 +157,7 @@ public double cost(DataSegment segment)
cost += bucket.cost(segment);
}

return cost;
return cost * NORMALIZATION_FACTOR;
}

public static Builder builder()
Expand Down Expand Up @@ -262,17 +263,12 @@ private double addLeftCost(DataSegment dataSegment, double t0, double t1, int in
double leftCost = 0.0;
// add to cost all left-overlapping segments
int leftIndex = index - 1;
while (leftIndex >= 0
&& sortedSegments.get(leftIndex).getInterval().overlaps(dataSegment.getInterval())) {
while (leftIndex >= 0) {
double start = convertStart(sortedSegments.get(leftIndex), interval);
double end = convertEnd(sortedSegments.get(leftIndex), interval);
leftCost += CostBalancerStrategy.intervalCost(end - start, t0 - start, t1 - start);
--leftIndex;
}
// add left-non-overlapping segments
if (leftIndex >= 0) {
leftCost += leftSum[leftIndex] * (FastMath.exp(-t1) - FastMath.exp(-t0));
}
return leftCost;
}

Expand Down
Loading