Skip to content

ARROW-3191: [Java] Make ArrowBuf work with arbitrary underlying memory #4151

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
siddharthteotia wants to merge 10 commits into from
Closed

ARROW-3191: [Java] Make ArrowBuf work with arbitrary underlying memory #4151

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
68fe274
ARROW-3191:[JAVA] WIP for pointing ArrowBuf to arbitrary memory
siddharthteotia Apr 11, 2019
348be8b
Change callers of ArrowBuf APIs to use ReferenceManager interface
siddharthteotia Apr 18, 2019
9e8beb6
Fix test failures, add javadoc and wrapper over ArrowBuf for
siddharthteotia Apr 19, 2019
2fbb2c5
Fix some test failures and rebase
siddharthteotia Apr 23, 2019
0b9d5d2
Fix test failures happening in non-debug mode
siddharthteotia Apr 24, 2019
76096e5
Refactor NettyArrowBuf
siddharthteotia Apr 24, 2019
47a303b
Setting io.netty.tryReflectionSetAccessible to true
siddharthteotia Apr 25, 2019
283916f
Fix integration test issues
siddharthteotia Apr 25, 2019
2fa139c
integration test issues
siddharthteotia May 3, 2019
6bb9cfc
Cleanup
siddharthteotia May 6, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions java/flight/src/main/java/org/apache/arrow/flight/ArrowMessage.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,7 +203,7 @@ private static ArrowMessage frame(BufferAllocator allocator, final InputStream s
case BODY_TAG:
if (body != null) {
// only read last body.
body.release();
body.getReferenceManager().release();
body = null;
}
int size = readRawVarint32(stream);
Expand Down Expand Up @@ -271,7 +271,7 @@ private InputStream asInputStream(BufferAllocator allocator) {
int size = 0;
List<ByteBuf> allBufs = new ArrayList<>();
for (ArrowBuf b : bufs) {
allBufs.add(b);
allBufs.add(b.asNettyBuffer());
size += b.readableBytes();
// [ARROW-4213] These buffers must be aligned to an 8-byte boundary in order to be readable from C++.
if (b.readableBytes() % 8 != 0) {
Expand All @@ -288,7 +288,7 @@ private InputStream asInputStream(BufferAllocator allocator) {
ArrowBuf initialBuf = allocator.buffer(baos.size());
initialBuf.writeBytes(baos.toByteArray());
final CompositeByteBuf bb = new CompositeByteBuf(allocator.getAsByteBufAllocator(), true, bufs.size() + 1,
ImmutableList.<ByteBuf>builder().add(initialBuf).addAll(allBufs).build());
ImmutableList.<ByteBuf>builder().add(initialBuf.asNettyBuffer()).addAll(allBufs).build());
final ByteBufInputStream is = new DrainableByteBufInputStream(bb);
return is;
} catch (Exception ex) {
Expand Down
2 changes: 1 addition & 1 deletion java/gandiva/src/test/java/org/apache/arrow/gandiva/evaluator/BaseEvaluatorTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -278,7 +278,7 @@ void releaseRecordBatch(ArrowRecordBatch recordBatch) {
List<ArrowBuf> buffers = recordBatch.getBuffers();
recordBatch.close();
for (ArrowBuf buf : buffers) {
buf.release();
buf.getReferenceManager().release();
}
}

Expand Down
Loading