Skip to content

HDDS-5246. Wait for ever to obtain CA list which is needed during OM/DN startup #2266

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
mukul1987 merged 10 commits into from
May 20, 2021
Merged

HDDS-5246. Wait for ever to obtain CA list which is needed during OM/DN startup #2266

mukul1987 merged 10 commits into from
May 20, 2021

Conversation

@bharatviswa504
Copy link
Contributor

@bharatviswa504 bharatviswa504 commented May 19, 2021

What changes were proposed in this pull request?

Wait forever to obtain the CA list.
And also made retry wait configurable.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-5246

How was this patch tested?

Tested with docker-compose ozonesecure-ha (with few changes to remove WAITFOR scm3.org)

docker-compose up scm1.org
docker-compose om1
docker-compose datanode1

om1_1        | 2021-05-19 09:02:16,082 [main] INFO utils.RetriableTask: Execution of task getCAList failed, will be retried in 10000 ms
om1_1        | 2021-05-19 09:02:26,102 [main] INFO utils.HAUtils: Expected CA list size 4, where as received CA List size 2.
om1_1        | 2021-05-19 09:02:26,103 [main] INFO utils.RetriableTask: Execution of task getCAList failed, will be retried in 10000 ms
om1_1        | 2021-05-19 09:02:36,147 [main] INFO utils.HAUtils: Expected CA list size 4, where as received CA List size 2.
om1_1        | 2021-05-19 09:02:36,147 [main] INFO utils.RetriableTask: Execution of task getCAList failed, will be retried in 10000 ms
om1_1        | 2021-05-19 09:02:46,198 [main] INFO utils.HAUtils: Expected CA list size 4, where as received CA List size 2.
om1_1        | 2021-05-19 09:02:46,198 [main] INFO utils.RetriableTask: Execution of task getCAList failed, will be retried in 10000 ms
om1_1        | 2021-05-19 09:02:56,222 [main] INFO utils.HAUtils: Expected CA list size 4, where as received CA List size 2.
om1_1        | 2021-05-19 09:02:56,222 [main] INFO utils.RetriableTask: Execution of task getCAList failed, will be retried in 10000 ms
om1_1        | 2021-05-19 09:03:06,245 [main] INFO utils.HAUtils: Expected CA list size 4, where as received CA List size 2.
om1_1        | 2021-05-19 09:03:06,245 [main] INFO utils.RetriableTask: Execution of task getCAList failed, will be retried in 10000 ms
om1_1        | 2021-05-19 09:03:16,263 [main] INFO utils.HAUtils: Expected CA list size 4, where as received CA List size 2.
om1_1        | 2021-05-19 09:03:16,263 [main] INFO utils.RetriableTask: Execution of task getCAList failed, will be retried in 10000 ms
om1_1        | 2021-05-19 09:03:26,296 [main] INFO utils.HAUtils: Expected CA list size 4, where as received CA List size 2.
om1_1        | 2021-05-19 09:03:26,296 [main] INFO utils.RetriableTask: Execution of task getCAList failed, will be retried in 10000 ms
om1_1        | 2021-05-19 09:03:36,333 [main] INFO utils.HAUtils: Expected CA list size 4, where as received CA List size 3.
om1_1        | 2021-05-19 09:03:36,333 [main] INFO utils.RetriableTask: Execution of task getCAList failed, will be retried in 10000 ms
om1_1        | 2021-05-19 09:03:46,376 [main] INFO utils.HAUtils: Expected CA list size 4, where as received CA List size 3.
om1_1        | 2021-05-19 09:03:46,376 [main] INFO utils.RetriableTask: Execution of task getCAList failed, will be retried in 10000 ms

datanode1_1  | 2021-05-19 09:02:50,155 [main] INFO utils.RetriableTask: Execution of task getCAList failed, will be retried in 10000 ms
datanode1_1  | 2021-05-19 09:03:00,179 [main] INFO utils.HAUtils: Expected CA list size 4, where as received CA List size 2.
datanode1_1  | 2021-05-19 09:03:00,179 [main] INFO utils.RetriableTask: Execution of task getCAList failed, will be retried in 10000 ms
datanode1_1  | 2021-05-19 09:03:10,204 [main] INFO utils.HAUtils: Expected CA list size 4, where as received CA List size 2.
datanode1_1  | 2021-05-19 09:03:10,204 [main] INFO utils.RetriableTask: Execution of task getCAList failed, will be retried in 10000 ms
datanode1_1  | 2021-05-19 09:03:20,225 [main] INFO utils.HAUtils: Expected CA list size 4, where as received CA List size 2.
datanode1_1  | 2021-05-19 09:03:20,225 [main] INFO utils.RetriableTask: Execution of task getCAList failed, will be retried in 10000 ms
datanode1_1  | 2021-05-19 09:03:30,266 [main] INFO utils.HAUtils: Expected CA list size 4, where as received CA List size 2.
datanode1_1  | 2021-05-19 09:03:30,266 [main] INFO utils.RetriableTask: Execution of task getCAList failed, will be retried in 10000 ms
datanode1_1  | 2021-05-19 09:03:40,293 [main] INFO utils.HAUtils: Expected CA list size 4, where as received CA List size 3.
datanode1_1  | 2021-05-19 09:03:40,293 [main] INFO utils.RetriableTask: Execution of task getCAList failed, will be retried in 10000 ms
datanode1_1  | 2021-05-19 09:03:50,323 [main] INFO utils.HAUtils: Expected CA list size 4, where as received CA List size 3.
datanode1_1  | 2021-05-19 09:03:50,323 [main] INFO utils.RetriableTask: Execution of task getCAList failed, will be retried in 10000 ms

bshashikant
bshashikant approved these changes May 19, 2021
View reviewed changes
Copy link
Contributor

@bshashikant bshashikant left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good

adoroszlai
adoroszlai requested changes May 19, 2021
View reviewed changes
Copy link
Contributor

@adoroszlai adoroszlai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but I'd like to suggest some some trivial code improvements.

bharatviswa504 and others added 8 commits May 19, 2021 16:04
@bharatviswa504 @adoroszlai
Update hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/uti…
a31faa6 
…ls/HAUtils.java

Co-authored-by: Doroszlai, Attila <6454655+adoroszlai@users.noreply.github.com>
@bharatviswa504 @adoroszlai
Update hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/uti…
f20f025 
…ls/HAUtils.java

Co-authored-by: Doroszlai, Attila <6454655+adoroszlai@users.noreply.github.com>
@bharatviswa504 @adoroszlai
Update hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/uti…
c0d36a2 
…ls/HAUtils.java

Co-authored-by: Doroszlai, Attila <6454655+adoroszlai@users.noreply.github.com>
@bharatviswa504 @adoroszlai
Update hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/uti…
6bc0582 
…ls/HAUtils.java

Co-authored-by: Doroszlai, Attila <6454655+adoroszlai@users.noreply.github.com>
@bharatviswa504 @adoroszlai
Update hadoop-hdds/common/src/main/resources/ozone-default.xml
e3ccdc2 
Co-authored-by: Doroszlai, Attila <6454655+adoroszlai@users.noreply.github.com>
@bharatviswa504 @adoroszlai
Update hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/uti…
d47a7c7 
…ls/HAUtils.java

Co-authored-by: Doroszlai, Attila <6454655+adoroszlai@users.noreply.github.com>
@bharatviswa504 @adoroszlai
Update hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/uti…
80d69db 
…ls/HAUtils.java

Co-authored-by: Doroszlai, Attila <6454655+adoroszlai@users.noreply.github.com>
@bharatviswa504
review comments
fa5f5c7
@bharatviswa504
Copy link
Contributor Author

bharatviswa504 commented May 19, 2021

Thank You @adoroszlai for the thorough review.
Addressed review comments.

@bharatviswa504 bharatviswa504 requested a review from adoroszlai May 19, 2021 11:32
adoroszlai
adoroszlai approved these changes May 19, 2021
View reviewed changes
Copy link
Contributor

@adoroszlai adoroszlai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @bharatviswa504 for updating the patch.

xiaoyuyao
xiaoyuyao reviewed May 19, 2021
View reviewed changes
hadoop-hdds/common/src/main/resources/ozone-default.xml Outdated


<property>
<name>ozone.scm.ca.list.retry.wait.duration</name>
Copy link
Contributor

@xiaoyuyao xiaoyuyao May 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIT: Can we rename ozone.scm.ca.list.retry.wait.duration to ozone.scm.ca.list.retry.interval to be consistent with other retry settings?

Copy link
Contributor Author

@bharatviswa504 bharatviswa504 May 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

xiaoyuyao
xiaoyuyao reviewed May 19, 2021
View reviewed changes
Copy link
Contributor

@xiaoyuyao xiaoyuyao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I just have one minor suggestion added inline...

@bharatviswa504 bharatviswa504 requested a review from xiaoyuyao May 19, 2021 16:26
xiaoyuyao
xiaoyuyao approved these changes May 19, 2021
View reviewed changes
Copy link
Contributor

@xiaoyuyao xiaoyuyao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 pending CI.

@mukul1987 mukul1987 merged commit 08375d7 into apache:master May 20, 2021
bharatviswa504 added a commit to bharatviswa504/hadoop-ozone that referenced this pull request Jul 25, 2021
@bharatviswa504
HDDS-5246. Wait for ever to obtain CA list which is needed during OM/…
999cb83 
…DN startup (apache#2266)

(cherry picked from commit 08375d7)
Change-Id: I2aa46fc70ec37059946da1c69806bc09bbf4f0ae
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@adoroszlai adoroszlai adoroszlai approved these changes

@xiaoyuyao xiaoyuyao xiaoyuyao approved these changes

@bshashikant bshashikant bshashikant approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@bharatviswa504 @adoroszlai @xiaoyuyao @bshashikant @mukul1987