style: Rename BigObject to LargeBinary #4111
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kunwp1
changed the title
Yicong-Huang
approved these changes
Dec 6, 2025
Contributor
Yicong-Huang
left a comment
Yicong-Huang
left a comment
There was a problem hiding this comment.
LGTM! I have also invoked Copilot for review, for this kind of rename PR, AI maybe better reviewer than human.
Contributor
|
It will be good to create an issue, though. |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR performs a comprehensive renaming from BigObject to LargeBinary to align with naming conventions used in systems like Apache Arrow. The change is purely terminological and does not modify any underlying functionality.
- Renamed all classes, types, and variables from BigObject to LargeBinary throughout the codebase
- Updated S3 bucket naming from
texera-big-objectstotexera-large-binaries - Renamed test files and updated all test descriptions to use new terminology
Reviewed changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/FileScanSourceOpExecSpec.scala | Updated test file to use LargeBinary type and updated all test cases and comments |
| common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/FileScanSourceOpExec.scala | Renamed BigObject imports and variables to LargeBinary; updated stream usage |
| common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/FileAttributeType.java | Renamed BIG_OBJECT enum value to LARGE_BINARY |
| common/workflow-core/src/test/scala/org/apache/texera/service/util/LargeBinaryOutputStreamSpec.scala | Comprehensive test file renaming with all class references, variables, and test descriptions updated |
| common/workflow-core/src/test/scala/org/apache/texera/service/util/LargeBinaryManagerSpec.scala | New test file (replaces BigObjectManagerSpec) with all references updated to LargeBinary |
| common/workflow-core/src/test/scala/org/apache/texera/service/util/LargeBinaryInputStreamSpec.scala | Test file renaming with all class references, variables, and bucket names updated |
| common/workflow-core/src/test/scala/org/apache/texera/service/util/BigObjectManagerSpec.scala | File deleted (replaced by LargeBinaryManagerSpec.scala) |
| common/workflow-core/src/test/scala/org/apache/texera/amber/util/IcebergUtilSpec.scala | Updated all test cases to use LargeBinary type and updated field name suffixes |
| common/workflow-core/src/test/scala/org/apache/texera/amber/core/tuple/AttributeTypeUtilsSpec.scala | Updated test cases to use LARGE_BINARY type with appropriate renaming |
| common/workflow-core/src/main/scala/org/apache/texera/service/util/LargeBinaryOutputStream.scala | Renamed class and all references from BigObject to LargeBinary |
| common/workflow-core/src/main/scala/org/apache/texera/service/util/LargeBinaryManager.scala | Renamed manager class and updated bucket name to texera-large-binaries |
| common/workflow-core/src/main/scala/org/apache/texera/service/util/LargeBinaryInputStream.scala | Renamed input stream class with all BigObject references updated |
| common/workflow-core/src/main/scala/org/apache/texera/amber/util/IcebergUtil.scala | Updated field name encoding suffix from __texera_big_obj_ptr to __texera_large_binary_ptr |
| common/workflow-core/src/main/scala/org/apache/texera/amber/core/tuple/LargeBinary.java | Renamed class from BigObject.java with all internal references updated |
| common/workflow-core/src/main/scala/org/apache/texera/amber/core/tuple/AttributeTypeUtils.scala | Updated type parsing to use LARGE_BINARY and LargeBinary class |
| common/workflow-core/src/main/scala/org/apache/texera/amber/core/tuple/AttributeType.java | Renamed enum value from BIG_OBJECT to LARGE_BINARY |
| amber/src/main/scala/org/apache/texera/web/service/WorkflowService.scala | Updated import and manager call to use LargeBinaryManager |
| amber/src/main/scala/org/apache/texera/web/resource/dashboard/user/workflow/WorkflowResource.scala | Updated import and manager call to use LargeBinaryManager |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
common/workflow-core/src/test/scala/org/apache/texera/service/util/LargeBinaryManagerSpec.scala
Outdated
Show resolved
Hide resolved
amber/src/main/scala/org/apache/texera/web/service/WorkflowService.scala
Outdated
Show resolved
Hide resolved
common/workflow-core/src/main/scala/org/apache/texera/amber/util/IcebergUtil.scala
Outdated
Show resolved
Hide resolved
amber/src/main/scala/org/apache/texera/web/service/WorkflowService.scala
Show resolved
Hide resolved
...src/main/scala/org/apache/texera/web/resource/dashboard/user/workflow/WorkflowResource.scala
Show resolved
Hide resolved
…util/LargeBinaryManagerSpec.scala Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Chris <143021053+kunwp1@users.noreply.github.com>
…il/IcebergUtil.scala Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Chris <143021053+kunwp1@users.noreply.github.com>
…vice.scala Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Chris <143021053+kunwp1@users.noreply.github.com>
5 tasks
kunwp1
added a commit
that referenced
this pull request
Dec 15, 2025
<!-- Thanks for sending a pull request (PR)! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: [Contributing to Texera](https://github.com/apache/texera/blob/main/CONTRIBUTING.md) 2. Ensure you have added or run the appropriate tests for your PR 3. If the PR is work in progress, mark it a draft on GitHub. 4. Please write your PR title to summarize what this PR proposes, we are following Conventional Commits style for PR titles as well. 5. Be sure to keep the PR description updated to reflect all changes. --> ### What changes were proposed in this PR? <!-- Please clarify what changes you are proposing. The purpose of this section is to outline the changes. Here are some tips for you: 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, you can clarify why it is a bug. 3. If it is a refactoring, clarify what has been changed. 3. It would be helpful to include a before-and-after comparison using screenshots or GIFs. 4. Please consider writing useful notes for better and faster reviews. --> This PR introduces Python support for the `large_binary` attribute type, enabling Python UDF operators to process data larger than 2 GB. Data is offloaded to MinIO (S3), and the tuple retains only a pointer (URI). This mirrors the existing Java LargeBinary implementation, ensuring cross-language compatibility. (See #4067 for system diagram and #4111 for renaming) ## Key Features ### 1. MinIO/S3 Integration - Utilizes the shared `texera-large-binaries` bucket. - Implements lazy initialization of S3 clients and automatic bucket creation. ### 2. Streaming I/O - **`LargeBinaryOutputStream`:** Writes data to S3 using multipart uploads (64KB chunks) to prevent blocking the main execution. - **`LargeBinaryInputStream`:** Lazily downloads data only when the read operation begins. Implements standard Python `io.IOBase`. ### 3. Tuple & Iceberg Compatibility - `largebinary` instances are automatically serialized to URI strings for Iceberg storage and Arrow tables. - Uses a magic suffix (`__texera_large_binary_ptr`) to distinguish pointers from standard strings. ### 4. Serialization - Pointers are stored as strings with metadata (`texera_type: LARGE_BINARY`). Auto-conversion ensures UDFs always see `largebinary` instances, not raw strings. ## User API Usage ### 1. Creating & Writing (Output) Use `LargeBinaryOutputStream` to stream large data into a new object. ```python from pytexera import largebinary, LargeBinaryOutputStream # Create a new handle large_binary = largebinary() # Stream data to S3 with LargeBinaryOutputStream(large_binary) as out: out.write(my_large_data_bytes) # Supports bytearray, bytes, etc. ``` ### 2. Reading (Input) Use `LargeBinaryInputStream` to read data back. It supports all standard Python stream methods. ```python from pytexera import LargeBinaryInputStream with LargeBinaryInputStream(large_binary) as stream: # Option A: Read everything all_data = stream.read() # Option B: Chunked reading chunk = stream.read(1024) # Option C: Iteration for line in stream: process(line) ``` ## Dependencies - `boto3`: Required for S3 interactions. - `StorageConfig`: Uses existing configuration for endpoints/credentials. ## Future Direction - Support for R UDF Operators - Check #4123 ### Any related issues, documentation, discussions? <!-- Please use this section to link other resources if not mentioned already. 1. If this PR fixes an issue, please include `Fixes #1234`, `Resolves #1234` or `Closes #1234`. If it is only related, simply mention the issue number. 2. If there is design documentation, please add the link. 3. If there is a discussion in the mailing list, please add the link. --> Design: #3787 ### How was this PR tested? <!-- If tests were added, say they were added here. Or simply mention that if the PR is tested with existing test cases. Make sure to include/update test cases that check the changes thoroughly including negative and positive cases if possible. If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future. If tests were not added, please describe why they were not added and/or why it was difficult to add. --> Tested by running this workflow multiple times and check MinIO dashboard to see whether six objects are created and deleted. Specify the file scan operator's property to use any file bigger than 2GB. [Large Binary Python.json](https://github.com/user-attachments/files/24062982/Large.Binary.Python.json) ### Was this PR authored or co-authored using generative AI tooling? <!-- If generative AI tooling has been used in the process of authoring this PR, please include the phrase: 'Generated-by: ' followed by the name of the tool and its version. If no, write 'No'. Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details. --> No. --------- Signed-off-by: Chris <143021053+kunwp1@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this PR?
This PR renames the
BigObjecttype toLargeBinary. The original feature was introduced in #4067, but we decided to adopt theLargeBinaryterminology to align with naming conventions used in other systems (e.g., Arrow).This change is purely a renaming/terminology update and does not modify the underlying functionality.
Any related issues, documentation, discussions?
#4100 (comment)
How was this PR tested?
Run this workflow and check if the workflow runs successfully and see if three objects are created in MinIO console.
Java UDF.json
Was this PR authored or co-authored using generative AI tooling?
No.