HDDS-10442. [hsync] Add a Freon tool to measure client to DataNode round-trip latency

[jojochuang]

What changes were proposed in this pull request?

Add a new freon tool similar to ozone freon om-echo that can be used to benchmark client to DataNode route-trip latency.

Usage:
ozone freon dn-echo --container-id=1 -n 100000 -t 32 --payload-req=0 --payload-resp=1024

This tool requires SCM superuser privilege.

It sends an echo request and receives the response to DataNodes associated with container specified with --containerID, repeat 100000 times, using 32 threads. The request has a 0KB payload and the response has 1024KB payload.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-10442

How was this patch tested?

Two unit tests, also tested in a real cluster.

[kerneltime]

cc @tanvipenumudy @duongkame

[kerneltime]

Probably an additional capability can be a delay in response, so the Datanode will wait for the configured duration before sending a response.

[jojochuang]

I was able to do 11k echos per second using one thread, no payload in request/response:

sudo -u hdfs ozone freon dn-echo --containerID=1 -n 100000

         count = 100000
     mean rate = 11068.82 calls/second
 1-minute rate = 8345.00 calls/second
 5-minute rate = 8345.00 calls/second
15-minute rate = 8345.00 calls/second
           min = 0.30 milliseconds
           max = 17.54 milliseconds
          mean = 0.81 milliseconds
        stddev = 0.91 milliseconds
        median = 0.59 milliseconds
          75% <= 0.83 milliseconds
          95% <= 1.67 milliseconds
          98% <= 2.59 milliseconds
          99% <= 4.34 milliseconds
        99.9% <= 9.80 milliseconds

8 threads:

sudo -u hdfs ozone freon dn-echo --containerID=1 -n 100000 -t 8

         count = 100000
     mean rate = 12448.57 calls/second
 1-minute rate = 13015.80 calls/second
 5-minute rate = 13015.80 calls/second
15-minute rate = 13015.80 calls/second
           min = 0.17 milliseconds
           max = 14.01 milliseconds
          mean = 0.53 milliseconds
        stddev = 0.69 milliseconds
        median = 0.41 milliseconds
          75% <= 0.50 milliseconds
          95% <= 1.07 milliseconds
          98% <= 1.94 milliseconds
          99% <= 3.02 milliseconds
        99.9% <= 9.83 milliseconds

32 threads, one MB each response:

sudo -u hdfs ozone freon dn-echo --containerID=1 -n 100000 -t 32 --payload-req=0 --payload-resp=1024

         count = 100000
     mean rate = 11059.79 calls/second
 1-minute rate = 10148.60 calls/second
 5-minute rate = 10148.60 calls/second
15-minute rate = 10148.60 calls/second
           min = 1.02 milliseconds
           max = 18.49 milliseconds
          mean = 2.70 milliseconds
        stddev = 1.81 milliseconds
        median = 2.12 milliseconds
          75% <= 2.56 milliseconds
          95% <= 7.32 milliseconds
          98% <= 8.53 milliseconds
          99% <= 9.14 milliseconds
        99.9% <= 12.66 milliseconds

32 threads, 10MB each response:

sudo -u hdfs ozone freon dn-echo --containerID=1 -n 100000 -t 32 --payload-req=0 --payload-resp=10240

         count = 100000
     mean rate = 7665.10 calls/second
 1-minute rate = 7479.75 calls/second
 5-minute rate = 7395.86 calls/second
15-minute rate = 7381.33 calls/second
           min = 1.32 milliseconds
           max = 24.93 milliseconds
          mean = 3.69 milliseconds
        stddev = 2.25 milliseconds
        median = 3.02 milliseconds
          75% <= 3.60 milliseconds
          95% <= 9.23 milliseconds
          98% <= 11.00 milliseconds
          99% <= 12.31 milliseconds
        99.9% <= 24.93 milliseconds

[tanvipenumudy]

Thank you @jojochuang for the patch, please find a small comment.

[tanvipenumudy]

I think we can reuse the methods under org.apache.hadoop.ozone.common.PayloadUtils for calculating the max payload size here.

[errose28]

nit. Let's keep all flags kebab case for consistency.

[ashishkumar50]

@jojochuang Thanks for the patch, Overall change LGTM.
Left some minor suggestions also PR usage description can be updated with --container-id.

[ashishkumar50]

Suggested change

	* @return GetSmallFileResponseProto
	* @return EchoResponseProto

[ashishkumar50]

Move containerInfo inside call() method

[ashishkumar50]

@jojochuang Thanks for updating patch, change mostly LGTM.

[ashishkumar50]

We can ignore sleeping if sleepTimeMs is 0.

[ashishkumar50]

We can initialize with 0 or is it mandatory to input?

[ashishkumar50]

Change LGTM.

[kerneltime]

The change looks good. We should add a robot test for this to avoid silent breakage.

[kerneltime]

Why have a max size? This is a test tool and it is ok to measure the error handling rate when the data payload is more than the container size.

[jojochuang]

removing the max size.

[kerneltime]

The change looks good to me, just a minor nit, I am ok with freon testing limits for max size or going over it.

[jojochuang]

There's a problem with block token in secure mode. Looking into that now.