Skip to content

reduce bytearray copy to minimal optimize VSizeIndexedWriter#2079

Merged
xvrl merged 1 commit intoapache:masterfrom
binlijin:master
Dec 11, 2015
Merged

reduce bytearray copy to minimal optimize VSizeIndexedWriter#2079
xvrl merged 1 commit intoapache:masterfrom
binlijin:master

Conversation

@binlijin
Copy link
Copy Markdown
Contributor

@binlijin binlijin commented Dec 10, 2015

No description provided.

@binlijin binlijin closed this Dec 10, 2015
@binlijin binlijin reopened this Dec 10, 2015
xvrl
xvrl reviewed Dec 10, 2015
View reviewed changes
Comment thread processing/src/main/java/io/druid/segment/data/VSizeIndexedInts.java
Copy link
Copy Markdown
Member

@xvrl xvrl Dec 10, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we refactor to reduce copy-paste with fromList?

@xvrl
Copy link
Copy Markdown
Member

xvrl commented Dec 10, 2015

@binlijin any numbers on how much this improves performance?

@binlijin
Copy link
Copy Markdown
Contributor Author

binlijin commented Dec 10, 2015

@xvrl , the performance numbers is：
Have 95 dimensions
Before：
round1
2015-12-10T15:49:19,781 INFO [main] io.druid.segment.IndexMerger - outDir[/var/folders/z7/g5zy3kfj7t54y1f074hsqw7h0000gn/T/base8884394285534279712flush/merged/v8-tmp] walked 500,000/500,000 rows in 16,840 millis.
2015-12-10T15:49:40,627 INFO [main] io.druid.segment.IndexMerger - outDir[/var/folders/z7/g5zy3kfj7t54y1f074hsqw7h0000gn/T/base8884394285534279712flush/merged/v8-tmp] completed walk through of 916,822 rows in 37,767 millis.
round2
2015-12-10T15:50:59,429 INFO [main] io.druid.segment.IndexMerger - outDir[/var/folders/z7/g5zy3kfj7t54y1f074hsqw7h0000gn/T/base774856129942668621flush/merged/v8-tmp] walked 500,000/500,000 rows in 16,473 millis.
2015-12-10T15:51:19,962 INFO [main] io.druid.segment.IndexMerger - outDir[/var/folders/z7/g5zy3kfj7t54y1f074hsqw7h0000gn/T/base774856129942668621flush/merged/v8-tmp] completed walk through of 916,822 rows in 37,065 millis.

After：
round1
2015-12-10T15:52:41,049 INFO [main] io.druid.segment.IndexMerger - outDir[/var/folders/z7/g5zy3kfj7t54y1f074hsqw7h0000gn/T/base4028347591001450805flush/merged/v8-tmp] walked 500,000/500,000 rows in 13,857 millis.
2015-12-10T15:52:58,643 INFO [main] io.druid.segment.IndexMerger - outDir[/var/folders/z7/g5zy3kfj7t54y1f074hsqw7h0000gn/T/base4028347591001450805flush/merged/v8-tmp] completed walk through of 916,822 rows in 31,505 millis.
round2
2015-12-10T15:54:14,273 INFO [main] io.druid.segment.IndexMerger - outDir[/var/folders/z7/g5zy3kfj7t54y1f074hsqw7h0000gn/T/base4543915257477602206flush/merged/v8-tmp] walked 500,000/500,000 rows in 14,390 millis.
2015-12-10T15:54:31,960 INFO [main] io.druid.segment.IndexMerger - outDir[/var/folders/z7/g5zy3kfj7t54y1f074hsqw7h0000gn/T/base4543915257477602206flush/merged/v8-tmp] completed walk through of 916,822 rows in 32,169 millis.

@binlijin binlijin closed this Dec 10, 2015
@binlijin binlijin reopened this Dec 10, 2015
@nishantmonu51
Copy link
Copy Markdown
Member

nishantmonu51 commented Dec 11, 2015

👍

xvrl added a commit that referenced this pull request Dec 11, 2015
@xvrl
Merge pull request #2079 from binlijin/master
d531e69 
reduce bytearray copy to minimal optimize VSizeIndexedWriter
@xvrl xvrl merged commit d531e69 into apache:master Dec 11, 2015
@fjy fjy modified the milestone: 0.9.0 Feb 4, 2016
@fjy fjy mentioned this pull request Feb 5, 2016
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

0.9.0

Development

Successfully merging this pull request may close these issues.

4 participants

@binlijin @xvrl @nishantmonu51 @fjy