Performance Regression or Improvement: Pytorch image classification on 50k images of size 224 x 224 with resnet 152 with Tesla T4 GPU:mean_inference_batch_latency_micro_secs

[AnandInguva]

Performance change found in the
test: Pytorch image classification on 50k images of size 224 x 224 with resnet 152 with Tesla T4 GPU:apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks for the metric: mean_inference_batch_latency_micro_secs.

For more information on how to triage the alerts, please look at
Triage performance alert issues section of the README.

timestamp: Mon May 8 20:21:00 2023, metric_value: 351051.74592690443
timestamp: Sun May 7 20:09:42 2023, metric_value: 373425.0474158369
timestamp: Sat May 6 20:12:19 2023, metric_value: 242029.10829268294
timestamp: Thu May 4 18:12:51 2023, metric_value: 361223.52659574465
timestamp: Wed May 3 18:22:09 2023, metric_value: 328118.64653425216
timestamp: Tue May 2 18:14:51 2023, metric_value: 424903.4859154929
timestamp: Mon May 1 18:14:12 2023, metric_value: 387124.80579584773
timestamp: Sun Apr 30 18:09:14 2023, metric_value: 306414.3679245283
timestamp: Sat Apr 29 18:14:43 2023, metric_value: 380149.34958979033
timestamp: Fri Apr 28 18:11:05 2023, metric_value: 358877.55455659993
timestamp: Thu Apr 27 18:19:19 2023, metric_value: 403779.09769008664 <---- Anomaly
timestamp: Wed Apr 26 18:34:08 2023, metric_value: 71642.43726750085
timestamp: Tue Apr 25 18:36:30 2023, metric_value: 78412.06520314548
timestamp: Mon Apr 24 18:33:33 2023, metric_value: 72853.56342609324
timestamp: Sun Apr 23 18:29:02 2023, metric_value: 79737.66775956284
timestamp: Sat Apr 22 18:32:37 2023, metric_value: 83166.35733538699
timestamp: Fri Apr 21 18:53:30 2023, metric_value: 72210.17876039304
timestamp: Wed Apr 19 18:37:05 2023, metric_value: 79859.03033666297
timestamp: Tue Apr 18 18:54:55 2023, metric_value: 72710.99006387508
timestamp: Mon Apr 17 18:41:13 2023, metric_value: 72510.55364642994
timestamp: Sun Apr 16 18:25:01 2023, metric_value: 66393.77158671587

[AnandInguva]

Seems like this is a performance improvement

[tvalentyn]

this looks like a regression, not improvement?

[tvalentyn]

Can see a bump on the graph too: http://104.154.241.245/d/ZpS8Uf44z/python-ml-runinference-benchmarks?orgId=1&from=now-60d&to=now

[tvalentyn]

noting that according to Grafana, the bump happens on Apr 26, while in the description of this bug it's next day possibly timezone difference.

[tvalentyn]

1a51ba5 might be related

[tvalentyn]

cc: @damccorm @AnandInguva

[damccorm]

@kerrydc

[AnandInguva]

[AnandInguva]

I am testing if this commit 1a51ba5 led to the regression.

Test PR: #27071

[AnandInguva]

I looked at the worker logs of the job. When there was the performance improvement around March 15th, 2023, I looked at the worker logs and found out dataflow was installing torch==2.0.0 but now on the latest experiments, I see it started to use torch=1.12.0. Because of this, the graph went back again.

@kerrydc @tvalentyn @damccorm it is not a regression on the Beam.