Simplifying dimension merging

[navis]

Currently, dimension merging is processed by two stages. One for dictionary, one for index. If this can be processed by single stage, total processing time could be deceased.

[navis]

Reduced dim conversion time from 11.6 sec to 7.1 sec, with 12 index with 500K rows each.

[binlijin]

What about the performance improvement when use IntBuffer instead of int[]?

[navis]

Don't think it will affect performace (I'll check that tomorrow). Is IntBuffer is better than int[]? Less intuitive for me.

[binlijin]

I just do not know why use IntBuffer，and other people can tell why.

[xvrl]

previously the IntBuffer was allocated off-heap, with this change it will be on-heap. Whether this is better or worse can be debated. Currently we rely on garbage collection to hopefully cleanup our buffers before we run out of memory, but allocating it on-heap may create lots of heap pressure if those arrays are big and long-lived. It might be worth some benchmarks to get a sense of how things will be affected.

[navis]

With direct IntBuffer, it took 8 seconds, not making any notable differences in performance. I prefer int[] but I'm ok with direct buffer. Opinion?

[drcrallen]

Heap pressure is already really bad during the making of index files, and making sure we know how this change impacts heap pressure during that time is important. During the merge and persist phase of realtime tasks, we already have very high CPU usage, enough to where you have to be aware how it impacts query performance. Adding in more heap pressure during that phase should be done with great care.

There are also issues with heap size during the reduce portion of hadoop tasks (or spark batch tasks). So I'm curious if adding more objects (int[]) messes the limit of the number of rows per segment (or if it impacts high cardinality dimensions).

[navis]

Reverted to direct IntBuffer.

[binlijin]

👍

[fjy]

can we have a more descriptive name than counter?

[fjy]

numDimensions or something

[navis]

renamed to numMergeIndex

[fjy]

👍 This looks good to me, but I think someone else familiar with this code needs to do a review

[binlijin]

+1

[binlijin]

can you squash the commits ?
If no one have any question i will merge.

[navis]

@binlijin squashed. thanks.

[fjy]

@binlijin do you mind holding off on merging? I just want 1 more pair of eyes to review this

@xvrl can you take a look?

[binlijin]

@fjy ok

[navis]

@binlijin It's becoming more and more painful to rebase. Do you really have a time to look into this?

[binlijin]

@navis, i am ok with the PR，but fj tell need @xrvl to take a look.

[binlijin]

@navis, i am ok with the PR，but fj tell need @xrvl to take a look.

[fjy]

@navis yes, there's multiple teams working on the same code, which is why we need proposals to coordinate, and all the changes are important

[fjy]

can these be static? also can we define them at the top of the file?

[navis]

remove timer. just used to check the performance.

[fjy]

why not use Guava's Stopwatch?

[navis]

removed this part

[jon-wei]

@navis

Looks good to me, I don't have any additional comments beyond what @fjy has already noted.

Can you rebase? I'll do another review pass on this after.

[navis]

Rebased on master, barely. I'll address comments.

[jon-wei]

Can you rename "WithConversion", e.g. ConvertingIndexSeeker or IndexSeekerWithConversion? Currently it sounds a bit like a boolean parameter and it's not immediately clear that it's a seeker

[navis]

ok

[jon-wei]

👍 looks good to me after addressing the Seeker renaming comments

[fjy]

@himanshug @xvrl do you want to take a look? I think this is getting close to ready and will merge unless there's more comments

[himanshug]

assuming above code is as is moved into method makeRowIterable(..)

[navis]

Yes, made a method to be used in V9Merger.

[himanshug]

👍 pls squash.

[navis]

squashed