HDDS-5462. Multi-raft style placement with permutations for offline data generator #2434
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
elek
changed the title
jojochuang
approved these changes
Jul 22, 2021
.../tools/src/main/java/org/apache/hadoop/ozone/freon/containergenerator/GeneratorDatanode.java
Show resolved
Hide resolved
| int parallelPipelines = maxDatanodes / 3; | ||
| int startOffset = (int) ((containerId % parallelPipelines) * 3); | ||
|
|
||
| int pipelineLevel = (int) (containerId / parallelPipelines); |
Contributor
There was a problem hiding this comment.
the variable name pipelineLevel is a little indirect to me. If we can find a better name for it...
Contributor
|
I'll merge this change now if there's no objection. |
errose28
added a commit
to errose28/ozone
that referenced
this pull request
Jul 30, 2021
* master: (48 commits) HDDS-5514. Skip check for UNHEALTHY containers for datanode finalize. (apache#2469) HDDS-5279. OFS mkdir -p does not work when Volume is not pre-created (apache#2412) HDDS-5328. Remove delete container command from admin CLI (apache#2456) HDDS-5382. Increase default container report interval to 60 mins (apache#2363) HDDS-5378 Add APIs to retrieve Namespace Summary from Recon (apache#2417) HDDS-5466. Refactor BlockOutputStream. (apache#2442) HDDS-5465. Delete redundant code when set、add and remove bucket acl (apache#2439) HDDS-5184. Use separate DB profile for Datanodes. (apache#2214) HDDS-5494. Reduce retry in Kubernetes test (apache#2461) HDDS-5414. Data buffers incorrectly filtered for Ozone Insight (apache#2387) HDDS-5450. Avoid refresh pipeline for S3 headObject (apache#2431) HDDS-5500. New k3s version breaks kubernetes test (apache#2464) HDDS-5489. Install OS-specific flekszible (apache#2462) Multi-raft style placement with permutations for offline data generator (apache#2434) HDDS-5484. Intermittent failure in TestReplicationManager#testMovePrerequisites (apache#2454) HDDS-5443 Create and then recreate a bucket with a randomized name (apache#2436) HDDS-5492. Disable failing kubernetes test (apache#2459) HDDS-4330. Bootstrap new OM node (apache#1494) HDDS-5418. Let Recon send reregisterCommand to Datanodes if DatanodeDetails changed (apache#2392) HDDS-5479. s3g bucket list failed when there is non-english key name. (apache#2450) ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
JIRA: https://issues.apache.org/jira/browse/HDDS-5462
What changes were proposed in this pull request?
Freon does have an offline generator test suite to generate quickly local data (rocksdb + container) for load testing.
But today, the datanode generator doesn't support different replication sets for the same node. Containers are generated in a way where the replication sets ("pipelines") are always include the same nodes:
In this case, if
node1is down, onlynode2andnode3can be used as the source of the nodes.With multi-raft option enabled, it's possible to have a placement were more than two nodes are used as source.
This patch uses a very simple permutation based approach to have similar layout but still keep the generation to be predictable.
With the new
--overlapargument, different containers can have slightly different replication set / pipelines:Please replace this section with the link to the Apache JIRA)
How was this patch tested?
Unit test is added for the new placement calculation.