fix model compression bug of nan output#1575
fix model compression bug of nan output#1575wanghan-iapcm merged 1 commit intodeepmodeling:develfrom
Conversation
njzjz
left a comment
There was a problem hiding this comment.
I approve this PR. In addition, could you explain which situation will cause the bug, which is even not covered by the UT?
This problem was caused by a previous optimization, which reuse the GPU registers to reduce the global memory access time. However, within that implementation, when the |
see #1274 . |
This PR should also fix #1444 .