Help relieve some memory pressure on brokers & coordinator#2751
Help relieve some memory pressure on brokers & coordinator#2751nishantmonu51 wants to merge 1 commit intoapache:masterfrom
Conversation
|
@nishantmonu51 what is the % improvement? |
There was a problem hiding this comment.
I think we can also make Jackson use this by putting @JsonCreator on a method like public static NoneShardSpec instance() { return INSTANCE; }
There was a problem hiding this comment.
done and added a serde test too.
|
@fjy for our setup it is expected to relieve it by around 4-5 %. |
|
👍 |
Help relieve some memory pressure on brokers & master when it has to manage large number of segments. fix tests review comments
|
Do we expect dimensions in the loadSpec to reflect the order of dimensions used for aggregation? Given that dimension order matters it might be worthwhile to preserve the order defined at aggregation time, anyone else have an opinion about this? |
|
@nishantmonu51 how much difference does it make if we don't sort the list, and instead use the dimension order used for indexing? |
|
@nishantmonu51 conflicts |
|
👍 |
|
@fjy my comments haven't been addressed yet |
|
@xvrl I didn't merge :P |
|
had a discussion with @xvrl on this one. |
|
Also, the changes in #2784 help more in reducing the memory pressure. |
|
@nishantmonu51 this PR merged in 0.12.X?,i can't find it in 0.9.1 and 0.10.X |
5 participants
Help relieve some memory pressure on the broker and coordinator.
This has a side effect that DataSegment now keeps the dimension/metric names in sorted way.
From Histogram -
6: 18221074 583074368 com.google.common.collect.RegularImmutableList
8: 9138836 511774816 io.druid.timeline.DataSegment
16: 3258048 52128768 io.druid.timeline.partition.NoneShardSpec