[SQL] SPARK-1427 Fix toString for SchemaRDD NativeCommands.#343
Closed
marmbrus wants to merge 1 commit intoapache:masterfrom
Closed
[SQL] SPARK-1427 Fix toString for SchemaRDD NativeCommands.#343marmbrus wants to merge 1 commit intoapache:masterfrom
marmbrus wants to merge 1 commit intoapache:masterfrom
Conversation
Contributor
|
lgtm |
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
Contributor
|
merged. |
andrewor14
pushed a commit
to andrewor14/spark
that referenced
this pull request
Apr 7, 2014
Fix test breaking downstream builds This wasn't detected in the pull-request-builder because it manually sets SPARK_HOME. I'm going to change that (it should't do this) to make it like the other builds.
pdeyhim
pushed a commit
to pdeyhim/spark-1
that referenced
this pull request
Jun 25, 2014
Author: Michael Armbrust <michael@databricks.com> Closes apache#343 from marmbrus/toStringFix and squashes the following commits: 37198fe [Michael Armbrust] Fix toString for SchemaRDD NativeCommands.
mccheah
pushed a commit
to mccheah/spark
that referenced
this pull request
Oct 3, 2018
Move RBackend to member variable
arjunshroff
pushed a commit
to arjunshroff/spark
that referenced
this pull request
Nov 24, 2020
…mapr ssl truststore (apache#343)
LuciferYang
added a commit
that referenced
this pull request
Mar 17, 2026
### What changes were proposed in this pull request? This pr aims to upgrade `arrow-java` from 18.3.0 to 19.0.0. It also fixes a buffer leak in `SparkResult.processResponses()` that **only manifests after this upgrade** and has **no actual impact under Arrow 18.3.0**. The issue is that when a deserialized Arrow batch contains 0 rows, the `ArrowMessage` objects were silently dropped without calling `close()`, and were not stored in `resultMap` (so `SparkResultCloseable.close()` would not release them either). Under Arrow 18.3.0 this was completely harmless — empty batches produced a 0-byte IPC body, which goes through `BaseAllocator.buffer(0)` → `getEmpty()` (a singleton backed by `EmptyReferenceManager` whose `retain()`/`release()` are no-ops and not tracked by the allocator), so no off-heap memory was ever allocated or leaked. However, Arrow 19.0.0 includes [GH-343](apache/arrow-java#343), which correctly serializes offset buffers for empty vectors per the Arrow spec, making the IPC body non-zero. This causes real tracked off-heap buffers to be allocated, and the missing `close()` becomes a real memory leak detectable by `allocator.close()`. Therefore this fix is included as a necessary companion change for the 19.0.0 upgrade. ### Why are the changes needed? The full release note as follows: - https://github.com/apache/arrow-java/releases/tag/v19.0.0 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass GitHub Acitons ### Was this patch authored or co-authored using generative AI tooling? No Closes #54820 from LuciferYang/SPARK-56000. Lead-authored-by: yangjie01 <yangjie01@baidu.com> Co-authored-by: YangJie <yangjie01@baidu.com> Signed-off-by: yangjie01 <yangjie01@baidu.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.