(AWS) Docs: List all AWS S3 properties from all language impl.#11321
(AWS) Docs: List all AWS S3 properties from all language impl.#11321hsiang-c wants to merge 2 commits intoapache:mainfrom
Conversation
|
|
||
| | S3 Property | Go | Java | Python | Rust | | ||
| |-------------------------------------------------------------------------------------------|---|---|---|---| | ||
| | [Progressive Multipart Upload](aws-s3-fileio-properties#progressive-multipart-upload) | | v | | | |
There was a problem hiding this comment.
Pyiceberg fileIO implemenation replies on Pyarrow and fsspec (depends on user configuration to choose which one to use in the background in the end). AFAIK, both lib (Pyarrow.fs.S3FileSystem and s3fs.core.S3FileSystem supports multipart upload.)
I haven't checked the other properties in this list, so there might be other things already supported by Python but didn't marked correctly.
There was a problem hiding this comment.
@TiansuYu Thank you for your review and feedback.
You're right about the capability of Python libraries (Pyarrow and fsspec).
At the moment, I only list s3 properties used in the Iceberg implementations (and therefore configurable in Iceberg). I didn't cover all possible s3 properties from the libraries used by Java/Python/Go/Rust.
Take Rust as another example, we can set the storage class in Rust's OpenDAL library, however, Iceberg's Rust config parsing function doesn't recognize it yet. Therefore, I didn't mark it as supported.
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
2 participants
Note to reviewers
aws-s3-fileio-properties.md) and link to it from the originalaws.mdaws-s3-fileio-properties.mdLanguagecolumn for properties that are supported by multiple languages.