[Bug][FQ2I] Failed to run FakeQuantizationToInteger on QDQ ONNX model

[Nullkooland]

Actual behavior

When I try to apply FakeQuantizationToInteger pass on a Relay IRModule imported from an ONNX model with QDQ quantization representation, it failed with this error message:

  ......
        at .../tvm/include/tvm/runtime/packed_func.h:1646
  1: tvm::relay::qnn::QnnBroadcastRel(tvm::runtime::Array<tvm::Type, void> const&, int, tvm::Attrs const&, tvm::TypeReporter const&)
        at .../tvm/src/relay/qnn/op/op_common.h:310
  0: tvm::relay::qnn::IsScalarType(tvm::Type const&, tvm::runtime::DataType const&)
        at .../tvm/src/relay/qnn/op/../utils.h:225
  File ".../tvm/src/relay/analysis/type_solver.cc", line 624
TVMError: 
---------------------------------------------------------------
An error occurred during the execution of TVM.
For more information, please see: https://tvm.apache.org/docs/errors.html
---------------------------------------------------------------
  Check failed: (false) is false: [16:42:40] .../tvm/src/relay/qnn/op/../utils.h:225: 
---------------------------------------------------------------
An error occurred during the execution of TVM.
For more information, please see: https://tvm.apache.org/docs/errors.html
---------------------------------------------------------------

  Check failed: tensor_type->shape.size() == 0 (1 vs. 0) : 

Environment

TVM v0.10.0

Steps to reproduce

Here is the code to reproduce the BUG:

model_onnx = onnx.load(onnx_path)
ir_mod, _ = relay.frontend.from_onnx(
    model_onnx,
    shape={"input": (1, 3, 224, 224)},
    freeze_params=True
)

passes = tvm.transform.Sequential([
    relay.transform.InferType(),
    relay.transform.FakeQuantizationToInteger(),
])
ir_mod = passes(ir_mod)

Here is the quantized ONNX model with QDQ representation.
regnet_x_400mf_qint8.onnx.tar.gz

Triage

Please refer to the list of label tags here to find the relevant tags and add them below in a bullet format (example below).

• relay:qnn
• frontend:onnx

[masahi]

Ok reproduced, I'll take a look.

BTW, this is an interesting model. How did you quantize it?

[Nullkooland]

Ok reproduced, I'll take a look.

BTW, this is an interesting model. How did you quantize it?

The model is quantized in PyTorch using Torch.FX, then exported using PyTorch's ONNX exporter. (However, currently the ONNX exporter only has limited support for quantized model, some Ops like quantized Concat are not supported yet, and the PyTorch team are working on a universal quant handler for ONNX exporter)

The exported model follows the QDQ pattern, one of the quantization representation used by ONNXRuntime's quantizer

[masahi]

I made some progress on debugging this issue. There is a small problem in the way the model is quantized. The scale and zp of the weight and bias are tensors (of shape 1), so Relay thinks this is per-channel quantization. But the output channel size and the size of scale doesn't match (32 vs 1 below). This leads to huge number of type errors from your model. So can you update your model so that scale and zp of weights and bias are stored as scalars like how it's done in the activation, if you are doing per-tensor quantization?

[masahi]

In the mean time, here is the branch that makes FQ2I work for your model https://github.com/apache/tvm/compare/main...masahi:fq2i-dense-add-fix?expand=1

I've added a hack to the ONNX importer to convert size-1 scale and zp tensors in your model to scalars. But as I said, you should update the way these params are stored in your model.

[Nullkooland]

In the mean time, here is the branch that makes FQ2I work for your model https://github.com/apache/tvm/compare/main...masahi:fq2i-dense-add-fix?expand=1

I've added a hack to the ONNX importer to convert size-1 scale and zp tensors in your model to scalars. But as I said, you should update the way these params are stored in your model.

I found that using rank-1 and size-1 tensor to represent per-tensor scale and zp is also the behavior of ONNXRuntime's quantizer, so is this something TVM's ONNX frontend should take into consideration? Since user has no control over how these quantizer are implemented.

[masahi]

I'd say it is a problem of the ORT quantizer. For activation qparams, they are saved as scalars. They should fix this inconsistency.

[Nullkooland]

I'd say it is a problem of the ORT quantizer. For activation qparams, they are saved as scalars. They should fix this inconsistency.

OK, I will open an issue there