[ONNX] Add converter for FastGelu from Microsoft onnxruntime contrib opset #13119
Conversation
|
Thanks for contributing to TVM! Please refer to the contributing guidelines https://tvm.apache.org/docs/contribute/ for useful information and tips. Please request code reviews from Reviewers by @-ing them in a comment.
Generated by tvm-bot |
|
Hello @jwfromm! |
areusch
added
needs-triage
| @pytest.mark.parametrize("op_name", ["Gelu", "FastGelu"], scope="session") | ||
| @tvm.testing.parametrize_targets | ||
| def test_gelu(target, dev): | ||
| def test_gelu(target, dev, op_name): |
There was a problem hiding this comment.
Let's also extend the tests to check accuracy for fp16 datatype.
| const1 = _expr.const(math.sqrt(2 / math.pi), dtype=const_dtype) | ||
| const2 = _expr.const(0.044715 * math.sqrt(2 / math.pi), dtype=const_dtype) |
There was a problem hiding this comment.
I took a look at the ONNX documentation. And they use hard-coded constants in their implementation. Let's check that for fp16 we will have the same accuracy with the current implementation as ONNX.
python/tvm/relay/frontend/onnx.py
Outdated
| const1 = _expr.const(0.7978845608028654, dtype=const_dtype) # sqrt(2.0 / PI) | ||
| const2 = _expr.const(0.0356774081363001, dtype=const_dtype) # 0.044715 * sqrt(2.0 / PI) |
There was a problem hiding this comment.
Let's return explicit calculation of these values if it is not affect on the accuracy
|
Hello @AndrewZhaoLuo! |
|
@tvm-bot rerun |
…opset (apache#13119) * add converter for FastGelu from Microsoft onnxruntime contrib opset * integrate FastGelu into test system for ONNX converters * code review fixes * returned constant calculation
…opset (apache#13119) * add converter for FastGelu from Microsoft onnxruntime contrib opset * integrate FastGelu into test system for ONNX converters * code review fixes * returned constant calculation
5 participants
ONNX Runtime supports
FastGeluin itscontrib opset(documentation and code). Implementation of this operator refers to an approximation for original GELU (article).Previously, converter for
Geluwas added to TVM ONNX frontend, however, there is no such converter forFastGelu.This PR adds this converter to ONNX frontend.