refactor(storage): remove the configured_scheme parameter from storage impls#2338
Merged
CTTY merged 2 commits intoapache:mainfrom Apr 17, 2026
Merged
refactor(storage): remove the configured_scheme parameter from storage impls#2338CTTY merged 2 commits intoapache:mainfrom
CTTY merged 2 commits intoapache:mainfrom
Conversation
rchowell
commented
Apr 15, 2026
CTTY
approved these changes
Apr 17, 2026
Collaborator
CTTY
left a comment
CTTY
left a comment
There was a problem hiding this comment.
LGTM! Thanks for the contribution
| // Check prefix of s3 path. | ||
| let prefix = format!("{}://{}/", configured_scheme, op_info.name()); | ||
| // Use the URL scheme in the path for prefix matching. This enables | ||
| // use of S3-compatible storage backends using custom schemes (e.g., `minio://`, `r2://`). |
Collaborator
There was a problem hiding this comment.
this is a good point, maybe OSS storage can use this directly so we don't even need to fallback to OpenDalStorage::OSS?
gbrgr
added a commit
to RelationalAI/iceberg-rust
that referenced
this pull request
Apr 29, 2026
Re-adds the test case that was removed by upstream in apache#2338 when they dropped configured_scheme entirely. Since we kept the scheme validation, this test should exist to cover it. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
gbrgr
added a commit
to RelationalAI/iceberg-rust
that referenced
this pull request
Apr 29, 2026
The scheme check added no real value — Azure accepts any scheme variant against the same endpoint. Upstream removed it in apache#2338 for the same reason. Dropping it eliminates the Option<AzureStorageScheme> workaround in OpenDalStorage::Azdls and simplifies OpenDalStorageFactory::Azdls back to a unit variant. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
toutane
pushed a commit
to DataDog/iceberg-rust
that referenced
this pull request
Apr 30, 2026
…e impls (apache#2338) ## Which issue does this PR close? - Closes apache#2245 - Related apache#2231 ## What changes are included in this PR? - Remove configured_scheme field from OpenDalStorage::{S3,Azdls} - Make S3 storage use the scheme in the file paths, allowing for custom S3-compatible schemes like minio:// - Use `HashMap<Scheme, Arc<OpenDalStorage>>` so aliases share a storage instance - Added new unit tests and removed some now-obsolete scheme-mismatch tests ## Break Change We are removing a struct field from public types, so this would need to be release in 0.10.0 ```rust // Before OpenDalStorageFactory::S3 { configured_scheme: "s3a".to_string(), customized_credential_load: None, } OpenDalStorageFactory::Azdls { configured_scheme: AzureStorageScheme::Abfss, } // After OpenDalStorageFactory::S3 { customized_credential_load: None } OpenDalStorageFactory::Azdls ``` ## Are these changes tested? Beyond the unit tests, I ran these integration tests. ```sh docker compose -f dev/docker-compose.yaml up -d --wait # requires unset on any AWS_ env vars cargo test -p iceberg-integration-tests cargo test -p iceberg-catalog-hms --test hms_catalog_test cargo test -p iceberg-catalog-loader cargo test -p iceberg-storage-opendal --features opendal-s3 --test file_io_s3_test ``` (cherry picked from commit fda82a2)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
configured_schemefrom OpenDalStorage::{S3, Azdls} #2245What changes are included in this PR?
HashMap<Scheme, Arc<OpenDalStorage>>so aliases share a storage instanceBreak Change
We are removing a struct field from public types, so this would need to be release in 0.10.0
Are these changes tested?
Beyond the unit tests, I ran these integration tests.