HDDS-7908. Support OM Metadata operation Generator in `Ozone freon` #4251

xichen01 · 2023-02-06T17:37:21Z

What changes were proposed in this pull request?

Add a tool to measure the Ozone om performance in Ozone freon
Support Ozone freon runs for a specified duration of time.
Support real-time display of QPS and MAX QPS for Ozone metadata test
Support display test arguments in the test report

GIF:
a MIXED operation, include CREATE_FILE and LOOKUP_FILE and LIST_STATUS, test duration time is 10s

ozone freon ommg --operation MIXED --ops CREATE_FILE,LOOKUP_FILE,LIST_STATUS --opsnum 5,4,1 -t 10 -n 1000 --runtime 5 --timebase

[root@Linux /root/ozone]% ozone freon ommg
A tool to measure the Ozone om performance
support Operation:
  CREATE_FILE, LOOKUP_FILE, READ_FILE, LIST_STATUS, CREATE_KEY, LOOKUP_KEY, HEAD_KEY, READ_KEY, LIST_KEYS, INFO_BUCKET, INFO_VOLUME, MIXED

Example:
# create 25000 keys, run time 180s
$ bin/ozone freon ommg --operation CREATE_KEY -n 25000 --runtime 180 --timebase

# read 25000 keys, run time 180s
$ bin/ozone freon ommg --operation READ_KEY -n 25000 --runtime 180 --timebase

# 20 threads, list 1000 keys each request, and run time 180s
$ bin/ozone freon ommg --operation LIST_KEYS -t 20 --batch-size 1000 --runtime 180 --timebase

# 10 threads, 1 threads list keys, 5 threads create file, 4 threads list lookup file and run time 180s
$ bin/ozone freon ommg --operation MIXED --ops CREATE_FILE,LOOKUP_FILE,LIST_STATUS --opsNums 5,4,1 -t 10 -n 1000  --runtime 180 --timebase

Note that: you must create a sufficient number of objects before executing read-related tests

[root@Linux /root/ozone]%

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-7908

How was this patch tested?

xichen01 · 2023-02-07T05:17:07Z

The command output example

[root@centos root]$ ozone freon ommg --operation MIXED --ops CREATE_FILE,LOOKUP_FILE,LIST_STATUS --opsnum 5,4,1 -t 10s -n 1000 --runtime 5 --timebase --verbose

 100.00% |██████████████████████████████████|  5/5 Time: 0:00:05|   CREATE_FILE: rate 76 max 88 LIST_STATUS: rate 3 max 12 LOOKUP_FILE: rate 453 max 843
2/7/23 1:15:14 PM ==============================================================

-- Timers ----------------------------------------------------------------------
CREATE_FILE
             count = 265
         mean rate = 57.17 calls/second
     1-minute rate = 0.00 calls/second
     5-minute rate = 0.00 calls/second
    15-minute rate = 0.00 calls/second
               min = 1.16 milliseconds
               max = 710.05 milliseconds
              mean = 73.13 milliseconds
            stddev = 138.07 milliseconds
            median = 5.29 milliseconds
              75% <= 9.02 milliseconds
              95% <= 299.71 milliseconds
              98% <= 307.26 milliseconds
              99% <= 709.11 milliseconds
            99.9% <= 710.05 milliseconds
LIST_STATUS
             count = 22
         mean rate = 4.29 calls/second
     1-minute rate = 4.40 calls/second
     5-minute rate = 4.40 calls/second
    15-minute rate = 4.40 calls/second
               min = 270.81 milliseconds
               max = 487.88 milliseconds
              mean = 383.47 milliseconds
            stddev = 94.26 milliseconds
            median = 327.36 milliseconds
              75% <= 484.02 milliseconds
              95% <= 487.43 milliseconds
              98% <= 487.88 milliseconds
              99% <= 487.88 milliseconds
            99.9% <= 487.88 milliseconds
LOOKUP_FILE
             count = 2308
         mean rate = 497.17 calls/second
     1-minute rate = 0.00 calls/second
     5-minute rate = 0.00 calls/second
    15-minute rate = 0.00 calls/second
               min = 0.33 milliseconds
               max = 298.57 milliseconds
              mean = 8.25 milliseconds
            stddev = 44.11 milliseconds
            median = 0.66 milliseconds
              75% <= 0.85 milliseconds
              95% <= 1.70 milliseconds
              98% <= 252.02 milliseconds
              99% <= 266.08 milliseconds
            99.9% <= 298.50 milliseconds


Total execution time (sec): 6
Failures: 0
Successful executions: 2595

Option:
--number-of-tests=1000
--threads=10
--timebase=true
--runtime=5
--fail-at-end=false
--prefix=tfae0qxzwe
--verbose=true
--volume=vol1
--bucket=bucket1
--size=0
--buffer=4096
--batch-size=1000
--random=false
--operation=LOOKUP_FILE
--ops=CREATE_FILE,LOOKUP_FILE,LIST_STATUS
--opsnum=5,4,1
--ophelp=false
--om-service-id=null
--help=false
--version=false

xichen01 · 2023-02-07T05:26:38Z

There are some features in this PR, should I split these into multiple PRs, the review will be easier.
such as:
this PR:

Add a tool to measure the Ozone om performance in Ozone freon
Support real-time display of QPS and MAX QPS for Ozone metadata test

another PR:

Support Ozone freon runs for a specified duration of time.
Support display test arguments in the test report

jojochuang · 2023-02-07T05:30:41Z

No this is okay. I'll review. Thanks.
One suggestion though: some of the operations under test are not pure metadata operations, such as READ_FILE and READ_KEY. Some of the operations may duplicate existing freon tools.

In any case, I think this is nice too to have. Being able to generate random-like workloads is great.

cc: @DaveTeng0

xichen01 · 2023-02-07T06:22:08Z

No this is okay. I'll review. Thanks. One suggestion though: some of the operations under test are not pure metadata operations, such as READ_FILE and READ_KEY. Some of the operations may duplicate existing freon tools.

In any case, I think this is nice too to have. Being able to generate random-like workloads is great.

cc: @DaveTeng0

By default, this is the operation of pure metadata in the process of reading and writing
By default, READ_FILE and READ_KEY, WRITE_FILE and WRITE_KEY data size is 0, so will not read/write data from DN, and will only access directly with the OM(the OM will access SCM but will not allocate block). This can be understood as the maximum read/write QPS that OM can achieve when there are infinite DNs.

In this freon sub command, the MIXED operation is possible, such as we can test:
Read: Write 7:3
ozone freon ommg --operation MIXED --ops CREATE_FILE,READ_FILE --opsnum 3,7 -t 10 -n 1000 --runtime 180 --timebase
Read: Write 5:5
ozone freon ommg --operation MIXED --ops CREATE_FILE,READ_FILE --opsnum 5,5 -t 10 -n 1000 --runtime 180 --timebase
....

adoroszlai

Thanks @xichen01 for the patch. I'll need to take another look at OmMetadataGenerator, but here is my first round of review for the existing Freon parts.

hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/BaseFreonGenerator.java

adoroszlai · 2023-02-07T10:04:32Z

hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/BaseFreonGenerator.java

+  public long getThreadSequenceId() {
+    return threadSequenceId.get();


Please add javadoc comment explaining purpose of this new sequence ID.

hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/BaseFreonGenerator.java

…in non-interactive environment

adoroszlai · 2023-02-07T17:10:01Z

hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/BaseFreonGenerator.java

+   * Get current Thread sequence ID.
+   * @return Current Thread sequence ID


Thanks for adding the comment, but I still don't think it's clear right away what it is used for. The doc just repeats the method name.

this has been updated

comments addressed

adoroszlai · 2023-02-28T18:01:51Z

Got this for MIXED workload from command help:

IllegalArgumentException
	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:131)
	at org.apache.hadoop.ozone.om.KeyManagerImpl.listStatus(KeyManagerImpl.java:1453)

xichen01 · 2023-03-01T18:27:16Z

Got this for MIXED workload from command help:

IllegalArgumentException
	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:131)
	at org.apache.hadoop.ozone.om.KeyManagerImpl.listStatus(KeyManagerImpl.java:1453)

This has been fixed

Those command can normally execution

ozone freon ommg --operation CREATE_FILE -n 25000 --duration  180s
ozone freon ommg --operation MIXED --ops CREATE_FILE,LOOKUP_FILE,LIST_STATUS --opsnum 5,4,1 -t 10 -n 1000 --duration 180s

adoroszlai · 2023-04-12T14:13:15Z

@duongkame @jojochuang please review

adoroszlai · 2023-04-17T13:02:41Z

Thanks @xichen01 for the patch, @jojochuang for the review.

* master: (440 commits) HDDS-8445. Move PlacementPolicy back to SCM (apache#4588) HDDS-8335. ReplicationManager: EC Mis and Under replication handlers should handle overloaded exceptions (apache#4593) HDDS-8355. Intermittent failure in TestOMRatisSnapshots#testInstallSnapshot (apache#4592) HDDS-8444. Increase timeout of CI build (apache#4586) HDDS-8446. Selective checks: handle change in ci.yaml (apache#4587) HDDS-8440. Ozone Manager crashed with ClassCastException when deleting FSO bucket. (apache#4582) HDDS-7309. Enable by default GRPC between S3G and OM (apache#3820) HDDS-8458. Mark TestBlockDeletion#testBlockDeletion as flaky HDDS-8385. Ozone can't process snapshot when service UID > 2097151 (apache#4580) HDDS-8424: Preserve legacy bucket getKeyInfo behavior (apache#4576) HDDS-8453. Mark TestDirectoryDeletingServiceWithFSO#testDirDeletedTableCleanUpForSnapshot as flaky HDDS-8137. [Snapshot] SnapDiff to use tombstone entries in SST files (apache#4376) HDDS-8270. Measure checkAccess latency for Ozone objects (apache#4467) HDDS-8109. Seperate Ratis and EC MisReplication Handling (apache#4577) HDDS-8429. Checkpoint is not closed properly in OMDBCheckpointServlet (apache#4575) HDDS-8253. Set ozone.metadata.dirs to temporary dir if not defined in S3 Gateway (apache#4455) HDDS-8400. Expose rocksdb last sequence number through metrics (apache#4557) HDDS-8333. ReplicationManager: Allow partial EC reconstruction if insufficient nodes available (apache#4579) HDDS-8147. Introduce latency metrics for S3 Gateway operations (apache#4383) HDDS-7908. Support OM Metadata operation Generator in `Ozone freon` (apache#4251) ...

HDDS-7908. Support OM Metadata operation Generator in Ozone freon

3e4f183

xichen01 force-pushed the HDDS-7908 branch from 49e860a to 3e4f183 Compare February 7, 2023 05:10

adoroszlai previously requested changes Feb 7, 2023

View reviewed changes

xichen01 added 2 commits February 7, 2023 21:28

Rename var; Add javadoc; Add --verbose mode;Disable interactive mode …

90932d7

…in non-interactive environment

fix findbugs

c2ee7bf

adoroszlai reviewed Feb 7, 2023

View reviewed changes

xichen01 and others added 7 commits February 8, 2023 20:41

add java doc for getThreadSequenceId

fd39f0a

Supports a formatted --runtime

baaf71a

format

e55e219

Merge branch 'apache:master' into HDDS-7908

cf5e0cc

Merge branch 'apache:master' into HDDS-7908

6874318

remove --timebase option

bf38bec

change option --runtime to --duration

f678fc5

xichen01 added 2 commits March 1, 2023 23:09

Fix LIST_STATUS issue;Add GET_KEYINFO op;format

7483520

findbugs

e70b291

adoroszlai requested a review from jojochuang March 12, 2023 17:30

adoroszlai requested a review from duongkame March 20, 2023 21:12

Merge branch 'apache:master' into HDDS-7908

07cad10

adoroszlai approved these changes Apr 17, 2023

View reviewed changes

adoroszlai merged commit cffa386 into apache:master Apr 17, 2023

		public long getThreadSequenceId() {
		return threadSequenceId.get();

		* Get current Thread sequence ID.
		* @return Current Thread sequence ID

HDDS-7908. Support OM Metadata operation Generator in Ozone freon #4251

HDDS-7908. Support OM Metadata operation Generator in Ozone freon #4251

Uh oh!

Conversation

xichen01 commented Feb 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

xichen01 commented Feb 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xichen01 commented Feb 7, 2023

Uh oh!

jojochuang commented Feb 7, 2023

Uh oh!

xichen01 commented Feb 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

adoroszlai Feb 7, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

adoroszlai Feb 7, 2023

Choose a reason for hiding this comment

Uh oh!

xichen01 Feb 11, 2023

Choose a reason for hiding this comment

Uh oh!

adoroszlai commented Feb 28, 2023

Uh oh!

xichen01 commented Mar 1, 2023

Uh oh!

adoroszlai commented Apr 12, 2023

Uh oh!

adoroszlai commented Apr 17, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

HDDS-7908. Support OM Metadata operation Generator in `Ozone freon` #4251

HDDS-7908. Support OM Metadata operation Generator in `Ozone freon` #4251

xichen01 commented Feb 6, 2023 •

edited

Loading

xichen01 commented Feb 7, 2023 •

edited

Loading

xichen01 commented Feb 7, 2023 •

edited

Loading