trying to understand gpcnet output

[antonl321]

Hello,

I got the table from below after running network_test.

I have two questions:

1. What is the meaning of Avg(Worst) column
2. How is possible to have for Multiple Allreduce the 99% and 99.9% percentile values outside the min-max range?

Kind regards,

Lucian Anton

Network Tests v1.3
  Test with 14320 MPI ranks (1790 nodes)

  Legend
   RR = random ring communication pattern
   Nat = natural ring communication pattern
   Lat = latency
   BW = bandwidth
   BW+Sync = bandwidth with barrier
+------------------------------------------------------------------------------------------------------------------------------------------+
|                                                          Isolated Network Tests                                                          |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
|                            Name |          Min |          Max |          Avg |   Avg(Worst) |          99% |        99.9% |        Units |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
|          RR Two-sided Lat (8 B) |          1.2 |         22.2 |          1.5 |          4.7 |          3.6 |          5.1 |         usec |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
|                RR Get Lat (8 B) |          1.3 |         22.3 |          1.9 |          3.7 |          2.2 |          3.6 |         usec |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
|      RR Two-sided BW (131072 B) |        549.7 |       3015.1 |       1199.2 |        764.5 |        460.4 |        335.0 |   MiB/s/rank |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
|            RR Put BW (131072 B) |          7.4 |      22134.8 |       2598.8 |          7.4 |          0.9 |          0.9 |   MiB/s/rank |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
| RR Two-sided BW+Sync (131072 B) |        336.2 |       2031.9 |        916.5 |        769.7 |        335.5 |        186.9 |   MiB/s/rank |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
|     Nat Two-sided BW (131072 B) |        650.0 |       4913.7 |       1899.5 |       1124.1 |       1142.5 |        883.4 |   MiB/s/rank |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
|        Multiple Allreduce (8 B) |         37.3 |         78.3 |         45.5 |         78.3 |        113.3 |        999.9 |         usec |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
|      Multiple Alltoall (4096 B) |        838.9 |       1003.9 |        901.6 |        838.9 |        479.3 |        186.3 |   MiB/s/rank |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+