Unittest failures seen in test_dnnl

[DickJC123]

Description

https://jenkins.mxnet-ci.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-20965/2/pipeline/ (test_conv_subgraph.py::test_pos_conv_act_add[True-gelu-True-data_shape1])
https://jenkins.mxnet-ci.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-20965/3/pipeline/288 (test_conv_subgraph.py::test_pos_conv_act_add[True-leakyrelu-True-data_shape1])
https://jenkins.mxnet-ci.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-20965/4/pipeline/289 (test_fc_subgraph.py::test_fc_transpose[mxnet.ndarray-auto-False-data_shape2])

[DickJC123]

https://jenkins.mxnet-ci.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-20965/5/pipeline/288/ test_fc_subgraph.py::test_fc_transpose[mxnet.numpy-int8-True-data_shape2]

The unix-cpu Python3:oneDNN-CPU and Python3:oneDNN-MKL-CPU jobs together seem to have a very low passing rate (1 in 5 runs passed by my experience in PR-20965).

Not sure if this is new behavior, but if so, could it be related to the recent onednn upgrade @bartekkuncer?

[bartekkuncer]

Hi @DickJC123, I will look into this.

[bartekkuncer]

Hi @DickJC123,

1. https://jenkins.mxnet-ci.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-20965/5/pipeline/288/ test_fc_subgraph.py::test_fc_transpose[mxnet.numpy-int8-True-data_shape2]

• Not related to oneDNN upgrade, @anko-intel analyzed it and here is PR with the fix [BUGFIX] Fix subgraph tests #20969

2. https://jenkins.mxnet-ci.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-20965/2/pipeline/ (test_conv_subgraph.py::test_pos_conv_act_add[True-gelu-True-data_shape1])
   AND
   https://jenkins.mxnet-ci.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-20965/3/pipeline/288 (test_conv_subgraph.py::test_pos_conv_act_add[True-leakyrelu-True-data_shape1])

• These two suffer from reading data from a neither updated nor zeroed register during convolution with the number of input channels lower than 4 and blocked weights (tests/python/dnnl/subgraphs/test_conv_subgraph.py::test_pos_conv_...[...-data_shape1] fail with oneDNN v2.4+ #20826). The full fix will arrive in v2.6 of oneDNN, therefore temporarily we will force plain format (at least on this axis) to make the convolution work properly and the test to pass PR: [BUGFIX] Fix tests/python/dnnl/subgraphs/test_conv_subgraph.py #20970.

[DickJC123]

Thank you very much for this thorough response! I'll look forward to smoother CI pipelines runs once these fixes are merged.

On an unrelated point, I notice the dnnl tests push out the complete Python3-onednn-cpu job runtime to 2hr, 15min. For example, there are these tests:

[2022-03-17T23:13:35.468Z] 900.98s call     tests/python/dnnl/test_quantization_dnnl.py::test_quantize_gluon_with_forward
[2022-03-17T23:13:35.468Z] 404.96s call     tests/python/dnnl/test_dnnl.py::test_Deconvolution
[2022-03-17T23:13:35.468Z] 232.17s call     tests/python/dnnl/test_dnnl.py::test_convolution
[2022-03-17T23:13:35.468Z] 220.28s call     tests/python/dnnl/test_dnnl.py::test_pooling

Is it possible to trim down the runtime of these tests, or perhaps run the tests with xdist on multiple workers (each with fewer cores presumably)?

[bartekkuncer]

@anko-intel

[bartekkuncer]

I do not know about xdist, but will look into these tests and try to speed them up.

[bartekkuncer]

@DickJC123 All PRs regarding failing tests have been merged.

[bgawrych]

Closing as original issue seems to be solved - please create new issue to track runtime if necessary