HDDS-1726. Integrate FsPerfTest into Ozone freon performance test suite. #3192

ArafatKhan2198 · 2022-03-14T13:08:47Z

What changes were proposed in this pull request?

Micro benchmarks for Apache HDFS/HCFS perf testing. The package includes the following TEST

WriteFileThroughputBenchmark.java :- A multi-threaded write IO test that allows tweaking write parameters like replication, IO size, number of threads etc. A single thread should be able to saturate the network or single disk bandwidth (whichever is lower).

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-1726?jql=text%20~%20%22FsPerfTest%22

How was this patch tested?

Since this patch is just a new feature for Ozone Freon, this won't have significant affect on the functioning of Ozone. The system that I have been using uses M1 ARM-64 architecture due to which I am not able to test out the functionality on my local cluster, hence I would need code-review so as to identify potential bugs if any.
@siddhantsangwan @jojochuang

...op-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/WriteFileThroughputBenchmark.java

hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/ContentGenerator.java

...op-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/WriteFileThroughputBenchmark.java

jojochuang · 2022-03-17T05:25:16Z

Re-trigger the test.

adoroszlai · 2022-03-17T06:17:21Z

Please check test results before retriggering.

Acceptance test is failing due to:

Option name '-p' is used by both field String org.apache.hadoop.ozone.freon.WriteFileThroughputBenchmark.rootPath and field String org.apache.hadoop.ozone.freon.BaseFreonGenerator.prefix

adoroszlai

Thanks @ArafatCloudera for working on this.

Ozone already has a file generator for Freon. Would it be possible to extend HadoopFsGenerator with additional parameters as necessary (e.g. flush/sync) instead of duplicating it?

ozone/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/HadoopFsGenerator.java

Line 43 in 4454e6b

public class HadoopFsGenerator extends BaseFreonGenerator

...op-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/WriteFileThroughputBenchmark.java

adoroszlai · 2022-03-17T06:33:07Z

...op-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/WriteFileThroughputBenchmark.java

+        // Enforcing throttle delay
+        final long ioEndTimeNs = (isThrottled ? System.nanoTime() : 0);
+        enforceThrottle(ioEndTimeNs - ioStartTimeNs, expectedIoTimeNs);


Can you please explain the need for throttling in a performance test?

If it is really necessary, shouldn't it happen outside of the timer.time() call?

The original FsPerfTest tool was created to measure both throughput and latency. The throttling option would be more relevant in the context of latency. We can drop the throttling for now as this PR does not support measuring latency. But later, throttling can be added back when latency metrics are added.

Link to the FsPerfTest tool :-https://github.com/arp7/FsPerfTest/blob/master/src/main/java/net/arp7/FsPerfTest/WriteFile.java

...op-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/WriteFileThroughputBenchmark.java

adoroszlai · 2022-03-17T06:49:18Z

hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/ContentGenerator.java

+    if (outputStream instanceof FSDataOutputStream) {
+      if (hSync) {
+        ((FSDataOutputStream) outputStream).hsync();
+      } else if (hFlush) {
+        ((FSDataOutputStream) outputStream).hflush();
+      }
+    }


It would be great if we could avoid this runtime check/cast by using inheritance and/or generics.

If we try to implement a check using inheritance then there would be a lot more code that will be added, hence I believe we could leave out this one.

jojochuang

Other than the hflush/hsync inheritance issue, anything else?

@ArafatKhan2198 on a second thought, we can add a method

public void write(FSDataOutputStream outputStream) {}

and call flushOrSync() in this method.

adoroszlai

Ozone already has a file generator for Freon. Would it be possible to extend HadoopFsGenerator with additional parameters as necessary (e.g. flush/sync) instead of duplicating it?

I'm still curious about the above.

adoroszlai · 2022-04-13T06:54:06Z

...op-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/WriteFileThroughputBenchmark.java

+    // Choosing which flag is to be set
+    boolean flush = false;
+    boolean sync = false;
+    Flags type = Flags.valueOf(flag);


We can avoid the need for this by changing flag from String to Flags. picocli supports enums.

adoroszlai · 2022-04-13T06:55:32Z

...op-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/WriteFileThroughputBenchmark.java

+    switch (type) {
+    case hSync:
+      sync = true;
+      break;
+    case hFlush:
+      flush = true;
+      break;
+    case None:
+      break;


Content generator also has the problem of mutually exclusive boolean flags. Can we pass type to ContentGenerator and let it use it directly in flushOrSync?

adoroszlai · 2022-04-13T06:56:35Z

...op-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/WriteFileThroughputBenchmark.java

+      defaultValue = "1")
+  private short replication;
+
+  @Option(names = {"--flags"},


flags is a rather vague name. I think flush-mode or something similar would be better.

adoroszlai · 2022-04-13T06:57:02Z

...op-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/WriteFileThroughputBenchmark.java

+
+  @Option(names = {"--flags"},
+      description = "Optionally issue hSync or hFlush after every write" +
+          "Cannot be used with hflush",


This line is leftover from previous version (with booleans).

adoroszlai · 2022-07-03T12:49:43Z

/pending

github-actions

Marking this issue as un-mergeable as requested.

Please use /ready comment when it's resolved.

Please note that the PR will be closed after 21 days of inactivity from now. (But can be re-opened anytime later...)

/pending

github-actions · 2022-07-25T00:17:31Z

Thank you very much for the patch. I am closing this PR temporarily as there was no activity recently and it is waiting for response from its author.

It doesn't mean that this PR is not important or ignored: feel free to reopen the PR at any time.

It only means that attention of committers is not required. We prefer to keep the review queue clean. This ensures PRs in need of review are more visible, which results in faster feedback for all PRs.

If you need ANY help to finish this PR, please contact the community on the mailing list or the slack channel."

kerneltime · 2023-01-30T04:59:37Z

@ArafatKhan2198 @adoroszlai @jojochuang This still seems like a good feature for Freon. Do you think we should revisit this PR?

Mohammad Arafat Khan and others added 8 commits March 13, 2022 11:21

Changes made

4883c1b

Made a few changes

628593c

Added the new class path to Freon.class

677d2cb

HDDS-1726.Integrate FsPerfTest into Ozone freon performance test suite.

ad2e962

HDDS-1726.Integrate FsPerfTest into Ozone freon performance test suite.

245e177

HDDS-1726. Integrate FsPerfTest into Ozone freon performance test suite.

54efc3e

Added a period to the first sentence

065bd2c

Merge branch 'apache:master' into HDDS-1726

345a7c1

jojochuang reviewed Mar 15, 2022

View reviewed changes

Mohammad Arafat Khan added 2 commits March 15, 2022 10:57

Replaced print message with LOG message at line:123

54629df

Removed the try block at :128 so as to reuse filesystem object

a843806

aryangupta1998 reviewed Mar 15, 2022

View reviewed changes

...op-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/WriteFileThroughputBenchmark.java Outdated Show resolved Hide resolved

Mohammad Arafat Khan added 3 commits March 15, 2022 18:41

Implemented the changes suggested in code review

33deba7

Added Apache License

3b57f6d

Fixed the style checks

b11756d

ArafatKhan2198 closed this Mar 17, 2022

ArafatKhan2198 reopened this Mar 17, 2022

retrigger checks

2714f4d

adoroszlai requested changes Mar 17, 2022

View reviewed changes

Mohammad Arafat Khan added 6 commits March 21, 2022 13:00

Removed two mutually exclusive flags with a single 3-way option.

f590806

Improved the desciription of throttle option

3001c9b

Improved the desciription of throttle option

529cad7

Made minor changes

6a36916

Changed the problem with the flags

5f9f9ba

Made changes for Code Review

d37ff9d

jojochuang reviewed Apr 13, 2022

View reviewed changes

adoroszlai reviewed Apr 13, 2022

View reviewed changes

github-actions bot requested changes Jul 3, 2022

View reviewed changes

github-actions bot added the pending label Jul 3, 2022

github-actions bot closed this Jul 25, 2022

HDDS-1726. Integrate FsPerfTest into Ozone freon performance test suite. #3192

HDDS-1726. Integrate FsPerfTest into Ozone freon performance test suite. #3192

Uh oh!

Conversation

ArafatKhan2198 commented Mar 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jojochuang commented Mar 17, 2022

Uh oh!

adoroszlai commented Mar 17, 2022

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jojochuang left a comment

Choose a reason for hiding this comment

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adoroszlai commented Jul 3, 2022

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jul 25, 2022

Uh oh!

kerneltime commented Jan 30, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ArafatKhan2198 commented Mar 14, 2022 •

edited

Loading