Skip to content

[fix](cloud) Fix aws sdk default retry policy not taking effect #43575

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Nov 12, 2024
Merged

[fix](cloud) Fix aws sdk default retry policy not taking effect #43575

dataroaring merged 1 commit into from
Nov 12, 2024

Conversation

@swjtu-zhanglei
Copy link
Contributor

@swjtu-zhanglei swjtu-zhanglei commented Nov 11, 2024 •
edited
Loading

  • AWS SDK default retry policy check response xml code to determine retry or not, but other object storage such as OSS, COS not compatible with AWS S3, so adding a custome retry policy for solving the problem

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

doris-robot commented Nov 11, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@swjtu-zhanglei
Copy link
Contributor Author

swjtu-zhanglei commented Nov 11, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Nov 11, 2024

clang-tidy review says "All clean, LGTM! 👍"

@swjtu-zhanglei
[fix](cloud) Fix aws sdk default retry policy not taking effect
b24df3e 
* AWS SDK default retry policy check response xml `code` to determine
  retry or not, but other object storage such as `OSS`, `COS` not compatible
  with AWS S3, so adding a custome retry policy for solving the problem
@github-actions
Copy link
Contributor

github-actions bot commented Nov 11, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Nov 11, 2024

TeamCity be ut coverage result:
Function Coverage: 37.90% (9867/26035)
Line Coverage: 29.08% (82240/282812)
Region Coverage: 28.24% (42342/149958)
Branch Coverage: 24.78% (21436/86504)
Coverage Report: http://coverage.selectdb-in.cc/coverage/cf0a2a57683801c0ca1a94ba8f4001751411a439_cf0a2a57683801c0ca1a94ba8f4001751411a439/report/index.html

@gavinchou gavinchou added the p0_c label Nov 11, 2024
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 11, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Nov 11, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Nov 11, 2024

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Nov 11, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring
Copy link
Contributor

dataroaring commented Nov 11, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Nov 11, 2024

TPC-H: Total hot run time: 51720 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b24df3eaa9426633ff4e3f7d44602914f4db2677, data reload: false

------ Round 1 ----------------------------------
q1	17569	7562	7290	7290
q2	2581	1325	1286	1286
q3	9947	1176	1151	1151
q4	10237	860	867	860
q5	7601	2951	2945	2945
q6	235	144	144	144
q7	1030	611	593	593
q8	9373	2371	2366	2366
q9	12626	12662	12441	12441
q10	7157	2393	2423	2393
q11	462	257	256	256
q12	430	211	217	211
q13	17793	3019	3023	3019
q14	251	210	212	210
q15	569	507	501	501
q16	648	602	609	602
q17	985	588	559	559
q18	7343	6746	6776	6746
q19	1331	1011	943	943
q20	3062	2843	2810	2810
q21	4006	3126	3030	3030
q22	1407	1364	1384	1364
Total cold run time: 116643 ms
Total hot run time: 51720 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7478	7587	7478	7478
q2	339	247	239	239
q3	3115	3012	2982	2982
q4	2039	1839	1811	1811
q5	5659	5555	5636	5555
q6	227	135	135	135
q7	2215	1763	1798	1763
q8	3430	3592	3534	3534
q9	14631	14791	14565	14565
q10	3647	3586	3555	3555
q11	591	493	493	493
q12	815	614	603	603
q13	13210	3201	3266	3201
q14	319	276	252	252
q15	596	535	554	535
q16	677	661	626	626
q17	1842	1614	1602	1602
q18	8299	7732	7574	7574
q19	1693	1549	1594	1549
q20	2154	1934	1919	1919
q21	5405	5074	5143	5074
q22	633	547	536	536
Total cold run time: 79014 ms
Total hot run time: 65581 ms

@doris-robot
Copy link

doris-robot commented Nov 11, 2024

TeamCity be ut coverage result:
Function Coverage: 37.73% (9815/26012)
Line Coverage: 29.03% (82086/282714)
Region Coverage: 28.15% (42208/149918)
Branch Coverage: 24.74% (21396/86478)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b24df3eaa9426633ff4e3f7d44602914f4db2677_b24df3eaa9426633ff4e3f7d44602914f4db2677/report/index.html

@swjtu-zhanglei
Copy link
Contributor Author

swjtu-zhanglei commented Nov 11, 2024

run p0

@dataroaring dataroaring merged commit 936ad96 into apache:master Nov 12, 2024
github-actions bot pushed a commit that referenced this pull request Nov 12, 2024
@swjtu-zhanglei
[fix](cloud) Fix aws sdk default retry policy not taking effect (#43575)
7f09808 
* AWS SDK default retry policy check response xml `code` to determine
retry or not, but other object storage such as `OSS`, `COS` not
compatible with AWS S3, so adding a custome retry policy for solving the
problem
@github-actions github-actions bot mentioned this pull request Nov 12, 2024
Closed
swjtu-zhanglei added a commit to swjtu-zhanglei/incubator-doris that referenced this pull request Nov 12, 2024
@swjtu-zhanglei
[fix](cloud) Fix aws sdk default retry policy not taking effect (apac…
f735c7d 
…he#43575)

* AWS SDK default retry policy check response xml `code` to determine
retry or not, but other object storage such as `OSS`, `COS` not
compatible with AWS S3, so adding a custome retry policy for solving the
problem
@swjtu-zhanglei swjtu-zhanglei mentioned this pull request Nov 12, 2024
Merged
16 tasks
swjtu-zhanglei added a commit to swjtu-zhanglei/incubator-doris that referenced this pull request Nov 12, 2024
@swjtu-zhanglei
[fix](cloud) Fix aws sdk default retry policy not taking effect (apac…
cc38aef 
…he#43575)

* AWS SDK default retry policy check response xml `code` to determine
retry or not, but other object storage such as `OSS`, `COS` not
compatible with AWS S3, so adding a custome retry policy for solving the
problem
dataroaring pushed a commit that referenced this pull request Nov 12, 2024
@swjtu-zhanglei
[fix](cloud) Fix aws sdk default retry policy not taking effect (#43575…
f97c054 
…) (#43648)

* AWS SDK default retry policy check response xml `code` to determine
retry or not, but other object storage such as `OSS`, `COS` not
compatible with AWS S3, so adding a custome retry policy for solving the
problem
zzzxl1993 pushed a commit to zzzxl1993/doris that referenced this pull request Nov 12, 2024
@swjtu-zhanglei @zzzxl1993
[fix](cloud) Fix aws sdk default retry policy not taking effect (apac…
91b6788 
…he#43575)

* AWS SDK default retry policy check response xml `code` to determine
retry or not, but other object storage such as `OSS`, `COS` not
compatible with AWS S3, so adding a custome retry policy for solving the
problem
924060929 pushed a commit to 924060929/incubator-doris that referenced this pull request Nov 12, 2024
@swjtu-zhanglei @924060929
[fix](cloud) Fix aws sdk default retry policy not taking effect (apac…
28cf07c 
…he#43575)

* AWS SDK default retry policy check response xml `code` to determine
retry or not, but other object storage such as `OSS`, `COS` not
compatible with AWS S3, so adding a custome retry policy for solving the
problem
py023 pushed a commit to py023/doris that referenced this pull request Nov 13, 2024
@swjtu-zhanglei @py023
[fix](cloud) Fix aws sdk default retry policy not taking effect (apac…
61dbddf 
…he#43575)

* AWS SDK default retry policy check response xml `code` to determine
retry or not, but other object storage such as `OSS`, `COS` not
compatible with AWS S3, so adding a custome retry policy for solving the
problem
@gavinchou gavinchou mentioned this pull request Nov 26, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.3-merged p0_c reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@swjtu-zhanglei @doris-robot @dataroaring @gavinchou