Skip to content

HDDS-6680. Pre-Finalize behaviour for Bucket Layout Feature. #3377

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
JyotinderSingh merged 12 commits into from
May 13, 2022
Merged

HDDS-6680. Pre-Finalize behaviour for Bucket Layout Feature. #3377

JyotinderSingh merged 12 commits into from
May 13, 2022

Conversation

@JyotinderSingh
Copy link
Contributor

@JyotinderSingh JyotinderSingh commented May 2, 2022 •
edited
Loading

What changes were proposed in this pull request?

Add Request Validators for the prefinalization state during the upgrade for the Bucket Layout feature.

OMBucketCreateRequest when asked to create an FSO bucket can lead to the persistence of new metadata that will not be compatible with an older version of the OzoneManager in case of a downgrade.

The behavior in such a case is defined as follows:

  • Only LEGACY buckets can be created in the pre-finalized state.
  • Requests to create FSO/OBS buckets will be denied in the PRE_PROCESS phase itself.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-6680

How was this patch tested?

Added integration tests to validate pre-finalization and post-finalization upgrade behavior.

errose28
errose28 reviewed May 4, 2022
View reviewed changes
Copy link
Contributor

@errose28 errose28 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this @JyotinderSingh. I appreciate all the effort that went in to adding the upgrade acceptance tests, but I think that may be overkill for just pre-finalization changes. The upgrade acceptance tests are meant for changes that affect the whole upgrade/downgrade cycle, including testing the old bits. The pre-finalize phase only happens in the new bits, so it will be much easier to integration test it. The mini ozone cluster builder allows you to create one without datanodes and with a custom layout version. This will give you a pre-finalized cluster which you can run tests on, finalize with the java client, and then run more tests on. This will cover the same changes with much less code and run much faster than the acceptance test.

rakeshadr
rakeshadr reviewed May 11, 2022
View reviewed changes
Copy link
Contributor

@rakeshadr rakeshadr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @JyotinderSingh for the continuous efforts. Added a few comments.

errose28
errose28 reviewed May 11, 2022
View reviewed changes
Copy link
Contributor

@errose28 errose28 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @JyotinderSingh. Mostly LGTM with a few minor comments in addition to what @rakeshadr has already said.

rakeshadr
rakeshadr approved these changes May 12, 2022
View reviewed changes
Copy link
Contributor

@rakeshadr rakeshadr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 LGTM. Thanks @JyotinderSingh for the contribution!

errose28
errose28 approved these changes May 13, 2022
View reviewed changes
Copy link
Contributor

@errose28 errose28 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM thanks @JyotinderSingh.

@JyotinderSingh
Copy link
Contributor Author

JyotinderSingh commented May 13, 2022

Thank you for the feedback and comments @rakeshadr @errose28. Will be merging this shortly.

@JyotinderSingh JyotinderSingh merged commit 6edcc13 into apache:master May 13, 2022
@JyotinderSingh JyotinderSingh deleted the HDDS-6680 branch May 13, 2022 06:53
errose28 added a commit to errose28/ozone that referenced this pull request May 20, 2022
@errose28
Merge branch 'master' into HDDS-5141
18c0251 
* master: (96 commits)
  HDDS-6738. Migrate tests with rules in hdds-server-framework to JUnit5 (apache#3415)
  HDDS-6650. S3MultipartUpload support update bucket usedNamespace. (apache#3404)
  HDDS-6491. Support FSO keys in getExpiredOpenKeys (apache#3226)
  HDDS-6596. EC: Support ListBlock from CoordinatorDN (apache#3410)
  HDDS-6737. Migrate parameterized tests in hdds-server-framework to JUnit5 (apache#3414)
  HDDS-6660: EC: Add the DN side Reconstruction Handler class. (apache#3399)
  HDDS-6750. Migrate simple tests in hdds-server-scm to JUnit5 (apache#3417)
  HDDS-6749. SCM includes itself as peer in addSCM request (apache#3413)
  HDDS-6657. Improve Ozone integrated Ranger configuration instructions (apache#3365)
  HDDS-6742. Audit operation category mismatch (apache#3407)
  HDDS-6748. Intermittent timeout in TestECBlockReconstructedInputStream#testReadDataWithUnbuffer (apache#3416)
  HDDS-6731. Migrate simple tests in hdds-server-framework to JUnit5 (apache#3412)
  HDDS-5919. In kubernetes OM HA has circular dependency on service availability (apache#3185)
  HDDS-6730. Migrate tests in hdds-tools to JUnit5 (apache#3402)
  HDDS-6630. Explicitly remove node after being chosen (apache#3332)
  HDDS-6560. Add general Grafana dashboard (apache#3285)
  HDDS-6704. EC: ReplicationManager - create version of ContainerReplicaCounts applicable to EC (apache#3405)
  HDDS-6680. Pre-Finalize behaviour for Bucket Layout Feature. (apache#3377)
  HDDS-6619. Add freon command to run r/w mix workload using ObjectStore APIs (apache#3383)
  HDDS-6734. ozone admin pipeline list CLI is not backward compatible (apache#3406)
  ...

Conflicts:
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/metadata/SCMMetadataStore.java
hadoop-hdds/interface-server/src/main/proto/SCMRatisProtocol.proto
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/metadata/SCMDBDefinition.java
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/metadata/SCMMetadataStoreImpl.java
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@errose28 errose28 errose28 approved these changes

@rakeshadr rakeshadr rakeshadr approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@JyotinderSingh @rakeshadr @errose28