Skip to content

ARROW-1547: [JAVA] Fix 8x memory over-allocation in BitVector #1109

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
siddharthteotia wants to merge 2 commits into from
Closed

ARROW-1547: [JAVA] Fix 8x memory over-allocation in BitVector #1109

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f3d1234
ARROW-1547: Fix 8x memory over-allocation in BitVector
siddharthteotia Sep 17, 2017
c92164a
addressed review comments
siddharthteotia Sep 19, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions java/vector/src/main/java/org/apache/arrow/vector/BitVector.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ public final class BitVector extends BaseDataValueVector implements FixedWidthVe
private final Mutator mutator = new Mutator();

int valueCount;
private int allocationSizeInBytes = INITIAL_VALUE_ALLOCATION;
private int allocationSizeInBytes = getSizeFromCount(INITIAL_VALUE_ALLOCATION);
Copy link
Contributor

@icexelloss icexelloss Sep 19, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to fix reset() as well?

Copy link
Contributor Author

@siddharthteotia siddharthteotia Sep 19, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes done. Thanks @icexelloss for pointing out.

Copy link
Contributor

@jacques-n jacques-n Sep 19, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM +1

private int allocationMonitor = 0;

public BitVector(String name, BufferAllocator allocator) {
Expand Down Expand Up @@ -175,7 +175,7 @@ public boolean allocateNewSafe() {
@Override
public void reset() {
valueCount = 0;
allocationSizeInBytes = INITIAL_VALUE_ALLOCATION;
allocationSizeInBytes = getSizeFromCount(INITIAL_VALUE_ALLOCATION);
allocationMonitor = 0;
zeroVector();
super.reset();
Expand Down
9 changes: 7 additions & 2 deletions java/vector/src/test/java/org/apache/arrow/vector/TestBufferOwnershipTransfer.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,9 @@ public void testTransferFixedWidth() {
v1.makeTransferPair(v2).transfer();

assertEquals(0, childAllocator1.getAllocatedMemory());
assertEquals(5 * 4096, childAllocator2.getAllocatedMemory());
int expectedBitVector = 512;
int expectedValueVector = 4096*4;
assertEquals(expectedBitVector + expectedValueVector, childAllocator2.getAllocatedMemory());
}

@Test
Expand All @@ -66,7 +68,10 @@ public void testTransferVariableidth() {
v1.makeTransferPair(v2).transfer();

assertEquals(0, childAllocator1.getAllocatedMemory());
int expected = 8 * 4096 + 4 * 4096 + 4096;
int expectedValueVector = 4096*8;
int expectedOffsetVector = 4096*4;
int expectedBitVector = 512;
int expected = expectedBitVector + expectedOffsetVector + expectedValueVector;
assertEquals(expected, childAllocator2.getAllocatedMemory());
}

Expand Down