Use virtual style addressing for non Amazon S3 endpoints. #763

ebabani · 2018-10-17T02:02:12Z

Description

Allow using the dns style (bucket.s3-endpoint) instead or forcing the path style (s3-endpoint/bucket) for non Amazon s3-compatible endpoint. This allows using the sdk with third party providers that only support the dns style for access (eg. Alicloud)

Motivation and Context

Allows using this sdk with any third party s3 providers that support virtual style addressing.

Testing

Screenshots (if appropriate)

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)

Checklist

I have read the CONTRIBUTING document
Local run of mvn install succeeds
My code follows the code style of this project
My change requires a change to the Javadoc documentation
I have updated the Javadoc documentation accordingly
I have read the README document
I have added tests to cover my changes
All new and existing tests passed
A short description of the change has been added to the CHANGELOG

License

I confirm that this pull request can be released under the Apache 2 license

ebabani · 2018-10-17T02:02:41Z

Related issue: #722

ebabani · 2018-10-26T16:43:19Z

@millems Is it possible to get someone to look at this PR?

The travis failure seems unrelated, and I don't know if I have a way to re-run the test.

millems · 2018-10-26T17:42:23Z

Sorry, talking with the others on our end, it looks like we've actually standardized this behavior across the latest SDK versions:

Determining which addressing style to use:

In general, SDK's MUST default to using virtual hosted-style addressing when making S3 API calls that address an S3 bucket. However, SDK's MUST default to path style addressing under the following scenarios:

The S3 bucket being addressed is non-DNS compatible. To determine DNS compatibility of a bucket, read the section on DNS compatibility.

The S3 bucket being addressed has one or more periods (.) in its name. If the request is being sent over HTTP instead of HTTPS, SDK's MAY chose to default to virtual hosted-style addressing despite the period in the bucket name.

The client is explicitly configured to use path style addressing and the request being sent is not to an S3 accelerate endpoint. For more information related to configuration, read the section on client configuration.

You can explicitly enable path-style addressing in the S3 client builder:

S3Client.builder()
        .serviceConfiguration(c -> c.pathStyleAccessEnabled(true))
        .build();

... but in the interest of behavioral parity with the other SDKs, we won't be able to accept this PR.

ebabani · 2018-10-26T17:55:23Z

@millems Totally agree with that behaviour, but at the moment this SDK is NOT using virtual hosted-style addressing for third party endpoints, and there is no way to change that.

https://github.com/aws/aws-sdk-java-v2/blob/master/services/s3/src/main/java/software/amazon/awssdk/services/s3/handlers/EndpointAddressInterceptor.java#L141

This PR addresses that limitation, but does not affect the addressing behaviour above.

millems · 2018-10-26T17:59:03Z

It looks like setting path-style access to enabled should work for third-party endpoints: https://github.com/aws/aws-sdk-java-v2/blob/master/services/s3/src/main/java/software/amazon/awssdk/services/s3/handlers/EndpointAddressInterceptor.java#L63-L69

Is that not the case?

ebabani · 2018-10-26T18:04:42Z

No, the changeToDnsEndpoint function has a check that the endpoint must start with s3, otherwise it doesn't make the change.

https://github.com/aws/aws-sdk-java-v2/blob/master/services/s3/src/main/java/software/amazon/awssdk/services/s3/handlers/EndpointAddressInterceptor.java#L141

if (mutableRequest.host().startsWith("s3")) {

millems · 2018-10-26T19:15:36Z

Ah, I misunderstood the request. For some reason I was thinking it was to use path-style addressing, not virtual style addressing. Reopening. I suppose it could be nice to be able to force virtual-style addressing on. We'll need to talk about whether we want to support this, since it will probably need to be a configuration option, not an implicit behavior.

…6f574698 Pull request: release <- staging/06a4ef5a-7bc3-49f5-a66f-cfe96f574698

millems · 2020-03-31T20:02:59Z

Just commenting to thank you for your patience on this issue. I know we haven't replied in a while. I'm a little nervous about merging this as-is, because it could break people who need path-style for third-party services and are relying on the current behavior.

ebabani · 2020-04-30T19:58:09Z

No worries, I'm good with closing this PR since it's difficult to change the default behaviour. I was able to use a fork for my purposes.

… bucket mechanism and support different uri styles by flags. (#33858) Many domestic cloud vendors are compatible with the s3 protocol. However, early versions of s3 client will only generate path style http requests (aws/aws-sdk-java-v2#763) when encountering endpoints that do not start with s3, while some cloud vendors only support virtual host style http request. Therefore, Doris used `forceVirtualHosted` in `S3URI` to convert it into a virtual hosted path and implemented it through path style. For example: For s3 uri `s3://my-bucket/data/file.txt`, It will eventually be parsed into: - virtualBucket: my-bucket - Bucket: data (bucket must be set, otherwise the s3 client will report an error) Especially this step is particularly tricky because of the limitations of the s3 client. - Key: file.txt The path style mode is used to generate an http request similar to the virtual host by setting the endpoint to virtualBucket + original endpoint, setting the bucket and key. **However, the bucket and key here are inconsistent with the original concepts of s3, but the aws client happens to be able to generate an http request similar to the virtual host through the path style mode.** However, after #30799 we have upgrade the aws sdk version from 2.17.257 to 2.20.131. The current aws s3 client can already generate a virtual host by third party by default style of http request. So in #31111 need to set the path style option, let the s3 client use doris' virtual bucket mechanism to continue working. **Finally, the virtual bucket mechanism is too confusing and tricky, and we no longer need it with the new version of s3 client.** ### Resolution: Rewrite `S3URI` to remove tricky virtual bucket mechanism and support different uri styles by flags. This class represents a fully qualified location in S3 for input/output operations expressed as as URI. #### For AWS S3, URI common styles: - AWS Client Style(Hadoop S3 Style): `s3://my-bucket/path/to/file?versionId=abc123&partNumber=77&partNumber=88` - Virtual Host Style: `https://my-bucket.s3.us-west-1.amazonaws.com/resources/doc.txt?versionId=abc123&partNumber=77&partNumber=88` - Path Style: `https://s3.us-west-1.amazonaws.com/my-bucket/resources/doc.txt?versionId=abc123&partNumber=77&partNumber=88` Regarding the above-mentioned common styles, we can use <code>isPathStyle</code> to control whether to use path style or virtual host style. "Virtual host style" is the currently mainstream and recommended approach to use, so the default value of <code>isPathStyle</code> is false. #### Other Styles: - Virtual Host AWS Client (Hadoop S3) Mixed Style: `s3://my-bucket.s3.us-west-1.amazonaws.com/resources/doc.txt?versionId=abc123&partNumber=77&partNumber=88` - Path AWS Client (Hadoop S3) Mixed Style: `s3://s3.us-west-1.amazonaws.com/my-bucket/resources/doc.txt?versionId=abc123&partNumber=77&partNumber=88` For these two styles, we can use <code>isPathStyle</code> and <code>forceParsingByStandardUri</code> to control whether to use. Virtual Host AWS Client (Hadoop S3) Mixed Style: <code>isPathStyle = false && forceParsingByStandardUri = true</code> Path AWS Client (Hadoop S3) Mixed Style: <code>isPathStyle = true && forceParsingByStandardUri = true</code> When the incoming location is url encoded, the encoded string will be returned. For <code>getKey()</code>, <code>getQueryParams()</code> will return the encoding string

… bucket mechanism and support different uri styles by flags. (apache#33858) Many domestic cloud vendors are compatible with the s3 protocol. However, early versions of s3 client will only generate path style http requests (aws/aws-sdk-java-v2#763) when encountering endpoints that do not start with s3, while some cloud vendors only support virtual host style http request. Therefore, Doris used `forceVirtualHosted` in `S3URI` to convert it into a virtual hosted path and implemented it through path style. For example: For s3 uri `s3://my-bucket/data/file.txt`, It will eventually be parsed into: - virtualBucket: my-bucket - Bucket: data (bucket must be set, otherwise the s3 client will report an error) Especially this step is particularly tricky because of the limitations of the s3 client. - Key: file.txt The path style mode is used to generate an http request similar to the virtual host by setting the endpoint to virtualBucket + original endpoint, setting the bucket and key. **However, the bucket and key here are inconsistent with the original concepts of s3, but the aws client happens to be able to generate an http request similar to the virtual host through the path style mode.** However, after apache#30799 we have upgrade the aws sdk version from 2.17.257 to 2.20.131. The current aws s3 client can already generate a virtual host by third party by default style of http request. So in apache#31111 need to set the path style option, let the s3 client use doris' virtual bucket mechanism to continue working. **Finally, the virtual bucket mechanism is too confusing and tricky, and we no longer need it with the new version of s3 client.** ### Resolution: Rewrite `S3URI` to remove tricky virtual bucket mechanism and support different uri styles by flags. This class represents a fully qualified location in S3 for input/output operations expressed as as URI. #### For AWS S3, URI common styles: - AWS Client Style(Hadoop S3 Style): `s3://my-bucket/path/to/file?versionId=abc123&partNumber=77&partNumber=88` - Virtual Host Style: `https://my-bucket.s3.us-west-1.amazonaws.com/resources/doc.txt?versionId=abc123&partNumber=77&partNumber=88` - Path Style: `https://s3.us-west-1.amazonaws.com/my-bucket/resources/doc.txt?versionId=abc123&partNumber=77&partNumber=88` Regarding the above-mentioned common styles, we can use <code>isPathStyle</code> to control whether to use path style or virtual host style. "Virtual host style" is the currently mainstream and recommended approach to use, so the default value of <code>isPathStyle</code> is false. #### Other Styles: - Virtual Host AWS Client (Hadoop S3) Mixed Style: `s3://my-bucket.s3.us-west-1.amazonaws.com/resources/doc.txt?versionId=abc123&partNumber=77&partNumber=88` - Path AWS Client (Hadoop S3) Mixed Style: `s3://s3.us-west-1.amazonaws.com/my-bucket/resources/doc.txt?versionId=abc123&partNumber=77&partNumber=88` For these two styles, we can use <code>isPathStyle</code> and <code>forceParsingByStandardUri</code> to control whether to use. Virtual Host AWS Client (Hadoop S3) Mixed Style: <code>isPathStyle = false && forceParsingByStandardUri = true</code> Path AWS Client (Hadoop S3) Mixed Style: <code>isPathStyle = true && forceParsingByStandardUri = true</code> When the incoming location is url encoded, the encoded string will be returned. For <code>getKey()</code>, <code>getQueryParams()</code> will return the encoding string

…4.0 (#34371) * [feature](insert)use optional location and add hive regression test (#33153) * [feature](iceberg)The new DDL syntax is added to create iceberg partitioned tables (#33338) support partition by : ``` create table tb1 (c1 string, ts datetime) engine = iceberg partition by (c1, day(ts)) () properties ("a"="b") ``` * [Enhancement](hive-writer) Adjust table sink exchange rebalancer params. (#33397) Issue Number: #31442 Change table sink exchange rebalancer params to node level and adjust these params to improve write performance by better balance. rebalancer params: ``` DEFINE_mInt64(table_sink_partition_write_min_data_processed_rebalance_threshold, "26214400"); // 25MB // Minimum partition data processed to rebalance writers in exchange when partition writing DEFINE_mInt64(table_sink_partition_write_min_partition_data_processed_rebalance_threshold, "15728640"); // 15MB ``` * [feature](profile) add transaction statistics for profile (#33488) 1. commit total time 2. fs operator total time rename file count rename dir count delete dir count 3. add partition total time add partition count 4. update partition total time update partition count like: ``` - Transaction Commit Time: 906ms - FileSystem Operator Time: 833ms - Rename File Count: 4 - Rename Dir Count: 0 - Delete Dir Count: 0 - HMS Add Partition Time: 0ms - HMS Add Partition Count: 0 - HMS Update Partition Time: 68ms - HMS Update Partition Count: 4 ``` * [feature](iceberg) add iceberg transaction implement (#33629) Issue #31442 add iceberg transaction * [feature](insert)support default value when create hive table (#33666) Issue Number: #31442 hive3 support create table with column's default value if use hive3, we can write default value to table * [refactor](filesystem)refactor `filesystem` interface (#33361) 1. Remame`list` to `globList` . The path of this `list` needs to have a wildcard character, and the corresponding hdfs interface is `globStatus`, so the modified name is `globList`. 2. If you only need to view files based on paths, you can use the `listFiles` operation. 3. Merge `listLocatedFiles` function into `listFiles` function. * [opt](meta-cache) refine the meta cache (#33449) 1. Use `caffeine` instead of `guava cache` to get better performace 2. Add a new class `CacheFactory` All (Async)LoadingCache should be built from `CacheFactory` 3. Use separator executor for different caches 1. rowCountRefreshExecutor For row count cache. Row count cache is an async loading cache, and we can ignore the result if cache missing or thread pool is full. So use a separate executor for this cache. 2. commonRefreshExecutor For other caches. Other caches are sync loading cache. But commonRefreshExecutor will be used for async refresh. That is, if cache entry is missing, the cache value will be loaded in caller thread, sychronously. if cache entry need refresh, it will be reloaded in commonRefreshExecutor. 3. fileListingExecutor File listing is a heavy operation, so use a separate executor for it. For fileCache, the refresh operation will still use commonRefreshExecutor to trigger refresh. And fileListingExecutor will be used to list file. 4. Change the refresh and expire logic of caches For most of caches, set `refreshAfterWrite` strategy, so that even if the cache entry is expired, the old entry can still be used while new entry is being loaded. 5. Add new global variable `enable_get_row_count_from_file_list` Default is true, if false, will disable getting row count from file list * [bugfix](hive)delete write path after hive insert (#33798) Issue #31442 1. delete file according query id 2. delete write path after insert * [Enhancement](multi-catalog) Rewrite `S3URI` to remove tricky virtual bucket mechanism and support different uri styles by flags. (#33858) Many domestic cloud vendors are compatible with the s3 protocol. However, early versions of s3 client will only generate path style http requests (aws/aws-sdk-java-v2#763) when encountering endpoints that do not start with s3, while some cloud vendors only support virtual host style http request. Therefore, Doris used `forceVirtualHosted` in `S3URI` to convert it into a virtual hosted path and implemented it through path style. For example: For s3 uri `s3://my-bucket/data/file.txt`, It will eventually be parsed into: - virtualBucket: my-bucket - Bucket: data (bucket must be set, otherwise the s3 client will report an error) Especially this step is particularly tricky because of the limitations of the s3 client. - Key: file.txt The path style mode is used to generate an http request similar to the virtual host by setting the endpoint to virtualBucket + original endpoint, setting the bucket and key. **However, the bucket and key here are inconsistent with the original concepts of s3, but the aws client happens to be able to generate an http request similar to the virtual host through the path style mode.** However, after #30799 we have upgrade the aws sdk version from 2.17.257 to 2.20.131. The current aws s3 client can already generate a virtual host by third party by default style of http request. So in #31111 need to set the path style option, let the s3 client use doris' virtual bucket mechanism to continue working. **Finally, the virtual bucket mechanism is too confusing and tricky, and we no longer need it with the new version of s3 client.** ### Resolution: Rewrite `S3URI` to remove tricky virtual bucket mechanism and support different uri styles by flags. This class represents a fully qualified location in S3 for input/output operations expressed as as URI. #### For AWS S3, URI common styles: - AWS Client Style(Hadoop S3 Style): `s3://my-bucket/path/to/file?versionId=abc123&partNumber=77&partNumber=88` - Virtual Host Style: `https://my-bucket.s3.us-west-1.amazonaws.com/resources/doc.txt?versionId=abc123&partNumber=77&partNumber=88` - Path Style: `https://s3.us-west-1.amazonaws.com/my-bucket/resources/doc.txt?versionId=abc123&partNumber=77&partNumber=88` Regarding the above-mentioned common styles, we can use <code>isPathStyle</code> to control whether to use path style or virtual host style. "Virtual host style" is the currently mainstream and recommended approach to use, so the default value of <code>isPathStyle</code> is false. #### Other Styles: - Virtual Host AWS Client (Hadoop S3) Mixed Style: `s3://my-bucket.s3.us-west-1.amazonaws.com/resources/doc.txt?versionId=abc123&partNumber=77&partNumber=88` - Path AWS Client (Hadoop S3) Mixed Style: `s3://s3.us-west-1.amazonaws.com/my-bucket/resources/doc.txt?versionId=abc123&partNumber=77&partNumber=88` For these two styles, we can use <code>isPathStyle</code> and <code>forceParsingByStandardUri</code> to control whether to use. Virtual Host AWS Client (Hadoop S3) Mixed Style: <code>isPathStyle = false && forceParsingByStandardUri = true</code> Path AWS Client (Hadoop S3) Mixed Style: <code>isPathStyle = true && forceParsingByStandardUri = true</code> When the incoming location is url encoded, the encoded string will be returned. For <code>getKey()</code>, <code>getQueryParams()</code> will return the encoding string * [improvement](hive)add the `queryid` to the temporary file path (#34278) `_temp_<table_name>` to `_temp_<queryid>_<table_name>`. Prevent users from having a table with the name `_temp_<table_name>`. So as to partition temp dir * [feature](Cloud) Load index data into index cache when writing data (#34046) * [Feature](hive-writer) Implements s3 file committer. (#33937) Issue Number: #31442 [Feature] (hive-writer) Implements s3 file committer. S3 committer will start multipart uploading all files on BE side, and then complete multipart upload these files on FE side. If you do not complete multi parts of a file, the file will not be visible. So in this way, the atomicity of a single file can be guaranteed. But it still cannot guarantee the atomicity of multiple files. Because hive committers have best-effort semantics, this shortens the inconsistent time window. ## ChangeList: - Add `used_by_s3_committer` in `FileWriterOptions` on BE side to start multi-part uploading files, then complete multi-part uploading files on FE side. - `cosn://`use s3 client on FE side, because it need to complete multi-part uploading files on FE side. - Add `Status directoryExists(String dir)` and `Status deleteDirectory` in `FileSystem`. --------- Co-authored-by: slothever <18522955+wsjz@users.noreply.github.com> Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com> Co-authored-by: Qi Chen <kaka11.chen@gmail.com> Co-authored-by: AlexYue <yj976240184@gmail.com>

… bucket mechanism and support different uri styles by flags. (apache#33858) Many domestic cloud vendors are compatible with the s3 protocol. However, early versions of s3 client will only generate path style http requests (aws/aws-sdk-java-v2#763) when encountering endpoints that do not start with s3, while some cloud vendors only support virtual host style http request. Therefore, Doris used `forceVirtualHosted` in `S3URI` to convert it into a virtual hosted path and implemented it through path style. For example: For s3 uri `s3://my-bucket/data/file.txt`, It will eventually be parsed into: - virtualBucket: my-bucket - Bucket: data (bucket must be set, otherwise the s3 client will report an error) Especially this step is particularly tricky because of the limitations of the s3 client. - Key: file.txt The path style mode is used to generate an http request similar to the virtual host by setting the endpoint to virtualBucket + original endpoint, setting the bucket and key. **However, the bucket and key here are inconsistent with the original concepts of s3, but the aws client happens to be able to generate an http request similar to the virtual host through the path style mode.** However, after apache#30799 we have upgrade the aws sdk version from 2.17.257 to 2.20.131. The current aws s3 client can already generate a virtual host by third party by default style of http request. So in apache#31111 need to set the path style option, let the s3 client use doris' virtual bucket mechanism to continue working. **Finally, the virtual bucket mechanism is too confusing and tricky, and we no longer need it with the new version of s3 client.** Rewrite `S3URI` to remove tricky virtual bucket mechanism and support different uri styles by flags. This class represents a fully qualified location in S3 for input/output operations expressed as as URI. #### For AWS S3, URI common styles: - AWS Client Style(Hadoop S3 Style): `s3://my-bucket/path/to/file?versionId=abc123&partNumber=77&partNumber=88` - Virtual Host Style: `https://my-bucket.s3.us-west-1.amazonaws.com/resources/doc.txt?versionId=abc123&partNumber=77&partNumber=88` - Path Style: `https://s3.us-west-1.amazonaws.com/my-bucket/resources/doc.txt?versionId=abc123&partNumber=77&partNumber=88` Regarding the above-mentioned common styles, we can use <code>isPathStyle</code> to control whether to use path style or virtual host style. "Virtual host style" is the currently mainstream and recommended approach to use, so the default value of <code>isPathStyle</code> is false. #### Other Styles: - Virtual Host AWS Client (Hadoop S3) Mixed Style: `s3://my-bucket.s3.us-west-1.amazonaws.com/resources/doc.txt?versionId=abc123&partNumber=77&partNumber=88` - Path AWS Client (Hadoop S3) Mixed Style: `s3://s3.us-west-1.amazonaws.com/my-bucket/resources/doc.txt?versionId=abc123&partNumber=77&partNumber=88` For these two styles, we can use <code>isPathStyle</code> and <code>forceParsingByStandardUri</code> to control whether to use. Virtual Host AWS Client (Hadoop S3) Mixed Style: <code>isPathStyle = false && forceParsingByStandardUri = true</code> Path AWS Client (Hadoop S3) Mixed Style: <code>isPathStyle = true && forceParsingByStandardUri = true</code> When the incoming location is url encoded, the encoded string will be returned. For <code>getKey()</code>, <code>getQueryParams()</code> will return the encoding string

Use virtual style for non Amazon S3 endpoints.

5bd462e

ebabani force-pushed the ebabani-s3-virt-addressing branch from 9482943 to 5bd462e Compare October 25, 2018 18:51

millems closed this Oct 26, 2018

millems reopened this Oct 26, 2018

aws-sdk-java-automation added a commit that referenced this pull request Mar 24, 2020

Merge pull request #763 from aws/staging/06a4ef5a-7bc3-49f5-a66f-cfe9…

568135e

…6f574698 Pull request: release <- staging/06a4ef5a-7bc3-49f5-a66f-cfe96f574698

ebabani closed this Apr 30, 2020

yannrouillard mentioned this pull request Oct 16, 2020

S3 virtual-style addressing for non AWS endpoints #722

Closed

kaka11chen mentioned this pull request Apr 18, 2024

[Enhancement](multi-catalog) Rewrite S3URI to remove tricky virtual bucket mechanism and support different uri styles by flags. apache/doris#33858

Merged

w41ter mentioned this pull request Jul 18, 2024

[Enhancement](multi-catalog) Rewrite S3URI to remove tricky virtualbucket mechanism and support different uri styles by flags. apache/doris#38064

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use virtual style addressing for non Amazon S3 endpoints. #763

Use virtual style addressing for non Amazon S3 endpoints. #763

Uh oh!

ebabani commented Oct 17, 2018 •

edited

Loading

Uh oh!

ebabani commented Oct 17, 2018

Uh oh!

ebabani commented Oct 26, 2018 •

edited

Loading

Uh oh!

millems commented Oct 26, 2018 •

edited

Loading

Uh oh!

ebabani commented Oct 26, 2018

Uh oh!

millems commented Oct 26, 2018

Uh oh!

ebabani commented Oct 26, 2018

Uh oh!

millems commented Oct 26, 2018

Uh oh!

millems commented Mar 31, 2020

Uh oh!

ebabani commented Apr 30, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Use virtual style addressing for non Amazon S3 endpoints. #763

Use virtual style addressing for non Amazon S3 endpoints. #763

Uh oh!

Conversation

ebabani commented Oct 17, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation and Context

Testing

Screenshots (if appropriate)

Types of changes

Checklist

License

Uh oh!

ebabani commented Oct 17, 2018

Uh oh!

ebabani commented Oct 26, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

millems commented Oct 26, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ebabani commented Oct 26, 2018

Uh oh!

millems commented Oct 26, 2018

Uh oh!

ebabani commented Oct 26, 2018

Uh oh!

millems commented Oct 26, 2018

Uh oh!

millems commented Mar 31, 2020

Uh oh!

ebabani commented Apr 30, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ebabani commented Oct 17, 2018 •

edited

Loading

ebabani commented Oct 26, 2018 •

edited

Loading

millems commented Oct 26, 2018 •

edited

Loading