[BUGFIX][1.8.x] Temporary fix for RNN with oneDNN seg faults/core dumps #19308

bgawrych · 2020-10-07T09:02:23Z

Description

This fix is workaround for problem with oneDNN memory descriptors comparison (size calculations) which causes segmentation faults in operator destructor - reported issue #19022.
Fix for this issue will be delivered with oneDNN 1.7 - however we would like to fix this issue in 1.8 version as well

Checklist

Essentials

PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage
Code is well-documented

Changes

Memory descriptor comparison in RNN operator

mxnet-bot · 2020-10-07T09:02:28Z

Hey @bgawrych , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:

To trigger all jobs: @mxnet-bot run ci [all]
To trigger specific jobs: @mxnet-bot run ci [job1, job2]

CI supported jobs: [centos-gpu, centos-cpu, windows-cpu, sanity, website, windows-gpu, clang, unix-gpu, unix-cpu, miscellaneous, edge]

Note:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.

bgawrych · 2020-10-07T13:05:31Z

@mxnet-bot run ci [windows-cpu, unix-cpu]

mxnet-bot · 2020-10-07T13:05:38Z

Jenkins CI successfully triggered : [windows-cpu, unix-cpu]

bgawrych · 2020-10-07T17:58:22Z

@mxnet-bot run ci [windows-cpu, unix-gpu]

mxnet-bot · 2020-10-07T17:58:29Z

Jenkins CI successfully triggered : [windows-cpu, unix-gpu]

bgawrych · 2020-10-07T20:14:19Z

@mxnet-bot run ci [unix-gpu]

mxnet-bot · 2020-10-07T20:14:25Z

Jenkins CI successfully triggered : [unix-gpu]

bgawrych · 2020-10-08T06:26:45Z

Hey @szha @samskalicky Do you know what's wrong with unix-gpu CI on 1.8 branch ?
This test is not related to my change and it's failing

[2020-10-07T22:09:54.644Z] 
[2020-10-07T22:09:54.644Z] ======================================================================
[2020-10-07T22:09:54.644Z] FAIL: test_operator_gpu.test_np_mixed_precision_binary_funcs
[2020-10-07T22:09:54.644Z] ----------------------------------------------------------------------
[2020-10-07T22:09:54.644Z] Traceback (most recent call last):
[2020-10-07T22:09:54.644Z]   File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest
[2020-10-07T22:09:54.644Z]     self.test(*self.arg)
[2020-10-07T22:09:54.644Z]   File "/usr/local/lib/python3.5/dist-packages/nose/util.py", line 620, in newfunc
[2020-10-07T22:09:54.644Z]     return func(*arg, **kw)
[2020-10-07T22:09:54.644Z]   File "/work/mxnet/tests/python/gpu/../unittest/common.py", line 218, in test_new
[2020-10-07T22:09:54.644Z]     orig_test(*args, **kwargs)
[2020-10-07T22:09:54.644Z]   File "/work/mxnet/python/mxnet/util.py", line 297, in _with_np_shape
[2020-10-07T22:09:54.644Z]     return func(*args, **kwargs)
[2020-10-07T22:09:54.644Z]   File "/work/mxnet/python/mxnet/util.py", line 481, in _with_np_array
[2020-10-07T22:09:54.644Z]     return func(*args, **kwargs)
[2020-10-07T22:09:54.644Z]   File "/work/mxnet/tests/python/gpu/../unittest/test_numpy_op.py", line 2607, in test_np_mixed_precision_binary_funcs
[2020-10-07T22:09:54.644Z]     check_mixed_precision_binary_func(func, low, high, lshape, rshape, lgrad, rgrad, type1, type2)
[2020-10-07T22:09:54.644Z]   File "/work/mxnet/tests/python/gpu/../unittest/test_numpy_op.py", line 2550, in check_mixed_precision_binary_func
[2020-10-07T22:09:54.644Z]     use_broadcast=False, equal_nan=True)
[2020-10-07T22:09:54.644Z]   File "/work/mxnet/python/mxnet/test_utils.py", line 749, in assert_almost_equal
[2020-10-07T22:09:54.644Z]     raise AssertionError(msg)
[2020-10-07T22:09:54.644Z] AssertionError: 
[2020-10-07T22:09:54.644Z] Items are not equal:
[2020-10-07T22:09:54.644Z] Error 1.732774 exceeds tolerance rtol=1.000000e-02, atol=1.000000e-03 (mismatch 16.666667%).
[2020-10-07T22:09:54.644Z] Location of maximum error: (1, 1), a=0.00978292, b=0.01171875
[2020-10-07T22:09:54.644Z]  ACTUAL: array([[0.77931417, 0.14845479, 0.22072042],
[2020-10-07T22:09:54.644Z]        [1.27150167, 0.00978292, 1.49220479]])
[2020-10-07T22:09:54.644Z]  DESIRED: array([[0.78125   , 0.15039062, 0.22265625],
[2020-10-07T22:09:54.644Z]        [1.2734375 , 0.01171875, 1.49414062]])
[2020-10-07T22:09:54.645Z] -------------------- >> begin captured stdout << ---------------------
[2020-10-07T22:09:54.645Z] 
[2020-10-07T22:09:54.645Z] *** Maximum errors for vector of size 6:  rtol=0.01, atol=0.001
[2020-10-07T22:09:54.645Z] 
[2020-10-07T22:09:54.645Z]   1: Error 1.732774  Location of error: (1, 1), a=0.00978292, b=0.01171875
[2020-10-07T22:09:54.645Z] 
[2020-10-07T22:09:54.645Z] --------------------- >> end captured stdout << ----------------------
[2020-10-07T22:09:54.645Z] -------------------- >> begin captured logging << --------------------
[2020-10-07T22:09:54.645Z] common: WARNING: Error seen with seeded test, use MXNET_TEST_SEED=694743356 to reproduce.
[2020-10-07T22:09:54.645Z] --------------------- >> end captured logging << ---------------------
[2020-10-07T22:09:54.645Z] 
[2020-10-07T22:09:54.645Z] ----------------------------------------------------------------------

bgawrych · 2020-10-08T06:26:57Z

@mxnet-bot run ci [unix-gpu]

mxnet-bot · 2020-10-08T06:27:01Z

Jenkins CI successfully triggered : [unix-gpu]

bgawrych · 2020-10-08T08:55:07Z

@anko-intel @pengzhao-intel @TaoLv

anko-intel · 2020-10-08T10:46:23Z

LGTM

pengzhao-intel · 2020-10-08T11:06:28Z

@ciyongch @zixuanweeei please help take a review.

pengzhao-intel · 2020-10-08T11:08:41Z

@anko-intel what's the timeline of 1.8? Is it possible to release a new patch version for 1.7?

anko-intel · 2020-10-08T12:22:15Z

@anko-intel what's the timeline of 1.8? Is it possible to release a new patch version for 1.7?

According to https://cwiki.apache.org/confluence/display/MXNET/1.8.0+Release+Plan+and+Status November the 1st will be the date of the official release. I think we are unable to prepare 1.7 patch version earlier.

ciyongch · 2020-10-10T01:38:23Z

src/operator/nn/mkldnn/mkldnn_rnn.cc

  }
 }

+// Bug in oneDNN >= 1.6  in memory descriptor comparision operators.


typo? oneDNN <= 1.6 here?

right, done

ciyongch · 2020-10-10T01:41:32Z

src/operator/nn/mkldnn/mkldnn_rnn.cc

+// for specific dims and strides in descriptors == operator can return `true`
+// but get_size() function will return different size
+// TODO(bgawrych): Remove with oneDNN 1.7 upgrade
+bool CheckMemDescEquality(const mkldnn::memory::desc &left, const mkldnn::memory::desc &right) {


better to make this function as static inline.

bgawrych · 2020-10-27T07:34:52Z

@ciyongch @pengzhao-intel @samskalicky Can we merge this change?

pengzhao-intel

LGTM and merging now.

pengzhao-intel · 2020-10-27T07:54:13Z

Please file an issue to track the oneDNN upgrade and remove the temp solution. @bgawrych

samskalicky · 2020-10-27T15:59:41Z

Thanks @bgawrych, can you also backport this PR to the v1.x branch so it stays in sync?

pengzhao-intel added the MKLDNN label Oct 8, 2020

ciyongch reviewed Oct 10, 2020

View reviewed changes

Bartlomiej Gawrych added 3 commits October 22, 2020 08:30

[1.8.x] Temporary fix for RNN with oneDNN seg faults/core dumps

f704efc

fix sanity

0312229

Fix typo and make compare function static inline

ca81a99

bgawrych force-pushed the rnn18xfix branch from 4538149 to ca81a99 Compare October 22, 2020 06:42

lanking520 added pr-awaiting-testing PR is reviewed and waiting CI build and test pr-awaiting-review PR is waiting for code review and removed pr-awaiting-testing PR is reviewed and waiting CI build and test labels Oct 22, 2020

szha requested a review from samskalicky October 22, 2020 15:51

pengzhao-intel approved these changes Oct 27, 2020

View reviewed changes

pengzhao-intel merged commit 7c86f48 into apache:v1.8.x Oct 27, 2020

This was referenced Oct 30, 2020

Remove temporary fix for RNN sporadic segfaults #19451

Merged

[1.x] Add test case for oneDNN RNN #19464

Merged

Add test case for oneDNN RNN #19465

Merged

[BUGFIX][1.8.x] Temporary fix for RNN with oneDNN seg faults/core dumps #19308

[BUGFIX][1.8.x] Temporary fix for RNN with oneDNN seg faults/core dumps #19308

Uh oh!

Conversation

bgawrych commented Oct 7, 2020

Description

Checklist

Essentials

Changes

Uh oh!

mxnet-bot commented Oct 7, 2020

Uh oh!

bgawrych commented Oct 7, 2020

Uh oh!

mxnet-bot commented Oct 7, 2020

Uh oh!

bgawrych commented Oct 7, 2020

Uh oh!

mxnet-bot commented Oct 7, 2020

Uh oh!

bgawrych commented Oct 7, 2020

Uh oh!

mxnet-bot commented Oct 7, 2020

Uh oh!

bgawrych commented Oct 8, 2020

Uh oh!

bgawrych commented Oct 8, 2020

Uh oh!

mxnet-bot commented Oct 8, 2020

Uh oh!

bgawrych commented Oct 8, 2020

Uh oh!

anko-intel commented Oct 8, 2020

Uh oh!

pengzhao-intel commented Oct 8, 2020

Uh oh!

pengzhao-intel commented Oct 8, 2020

Uh oh!

anko-intel commented Oct 8, 2020

Uh oh!

ciyongch Oct 10, 2020

Choose a reason for hiding this comment

Uh oh!

bgawrych Oct 22, 2020

Choose a reason for hiding this comment

Uh oh!

ciyongch Oct 10, 2020

Choose a reason for hiding this comment

Uh oh!

bgawrych Oct 22, 2020

Choose a reason for hiding this comment

Uh oh!

bgawrych commented Oct 27, 2020

Uh oh!

pengzhao-intel left a comment

Choose a reason for hiding this comment

Uh oh!

pengzhao-intel commented Oct 27, 2020

Uh oh!

samskalicky commented Oct 27, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants