[QNN] [RFC] QNN Dialect -- Prequantize Models

[anijain2305]

We are proposing a new dialect named QNN, that introduces a quantized version of existing relay operators. The goal is to support the models that have been pre-quantized in the framework.

Some important notes about QNN dialect are

• QNN operators are lowered to existing Relay operators to ensure that we can reuse Relay infrastructure.
• Code resides in new directory. Python files are in python/relay/qnn and CPP files are in src/relay/qnn.
• QNN, like any other dialect, introduces new Relay passes. These passes only deal with QNN ops (like lowering of QNN ops to existing Relay ops). For any generic optimization, we rely on existing Relay passes.

We can use this thread to discuss various open questions. Some of these questions can be

1. Code organization, namespaces, API discussion.
2. QNN operator lowering - Infrastructure, correct sequence of Relay operations etc.
3. Ways to efficiently add new operators with minimal engineering efforts.
4. Requirements (if any) of new generic Relay passes to achieve good performance.
5. Any new bugs that arise as we start testing integer computations more thoroughly.

The idea of QNN dialect was a result of discussion at Issue #2351. Thanks @tqchen @FrozenGene @jackwish @jnorwood @shoubhik for the discussions.

First few PRs for the QNN dialect

• Requantize operator - [QNN] Requantize operator #3531
• Quantize and Dequantize operator - [RFC][Quantization] Designing and lowering of quantized ops #3512

[tqchen]

also cc @ajtulloch @ZihengJiang @vinx13 @eqy @jroesch , @anijain2305 can you please list the API proposals and the reference APIs(in tflite etcs?) Then we can try to get everyone's thoughts on these specific API designs

[anijain2305]

Let's start with just Requantize to keep it focussed. I would suggest to add +1 if one likes the API, to show that one agrees.

QNN proposal

def requantize(data,
               input_scale,
               input_zero_point,
               output_scale,
               output_zero_point,
               rounding="AWAY_FROM_ZERO",
               out_dtype="int8"):
    r"""Requantized operator.
     The requantize operator converts one quantized tensor representation to
    another quantized tensor representation. For the output tensor, we are
    provided with output scale and zero point. The computation is as follows
     Q_output = zp_output +  (scale_input)/(scale_ouptut) * (Q_input - zp_input)
     Parameters
    ----------
    data : tvm.relay.Expr
        The input data to the operator.
     input_scale: float
           The quantization scale for the input tensor.
     input_zero_point: int
           The zero point of the input tensor.
     output_scale: float
           The quantization scale for the output tensor.
     output_zero_point: int
           The zero point of the output tensor.
     rounding : string, optional
        Defines the rounding direction when the value is midway between two
        representable values.
     out_dtype : str, optional
        Specifies the output data type for mixed precision conv2d.
     Returns
    -------
    result : tvm.relay.Expr
        The computed result.
    """

TF Requantize

Arguments:

scope: A Scope object
input_min: The float value that the minimum quantized input value represents.
input_max: The float value that the maximum quantized input value represents.
requested_output_min: The float value that the minimum quantized output value represents.
requested_output_max: The float value that the maximum quantized output value represents.
out_type: The type of the output. Should be a lower bit depth than Tinput.

• The min/max in TF are represented by scale and zero_point in QNN
• rounding in QNN proposal is to give a choice between standard rounding (away_from_zero) and round_towards_zero, presenting a performance-accuracy knob.

[anijain2305]

@FrozenGene @tqchen @u99127 Can you please approve the above API, so that we can discuss/move to next discussion. I have so many things to discuss :)

[FrozenGene]

@anijain2305 Let me look at it afternoon of today or evening.

[tqchen]

@anijain2305 to increase the throughput, can you also list other APIs, one per post and we use lazy consensus, wait for a week to see people's feedback then summarize.

[jnorwood]

There are a couple of things in this gemmlowp quantization example, which you should perhaps consider how it could be supported in your requantize api.

1. ability to specify that range should be extended to include a zero value.
2. ability to specify that the range should be nudged so that exact zero is an integer value.
   https://github.com/google/gemmlowp/blob/master/doc/quantization_example.cc

[anijain2305]

@jnorwood Thanks for the comment. Both good points. I will keep those abilities, though, outside of the scope of requantize op.
Another function (not necessarily a Relay operator) can take min/max, a config (like nudge that zero is exactly representable) and generates scale and zero point as per the given choice. As of now, I am thinking of them as utility functions.

[anijain2305]

QNN Conv2D operator

Tensorflow

tf.nn.quantized_conv2d(
    input,
    filter,
    min_input,
    max_input,
    min_filter,
    max_filter,
    strides,
    padding,
    out_type=tf.dtypes.qint32,
    dilations=[1, 1, 1, 1],
    name=None
)

MxNet

mxnet.symbol.contrib.quantized_conv(
data=None, 
weight=None,
bias=None,
min_data=None,
max_data=None,
min_weight=None,
max_weight=None, 
min_bias=None, 
max_bias=None, 
kernel=_Null, 
stride=_Null, 
dilate=_Null, 
pad=_Null, 
num_filter=_Null, 
num_group=_Null, 
workspace=_Null, 
no_bias=_Null, 
cudnn_tune=_Null, 
cudnn_off=_Null, 
layout=_Null, 
name=None, 
attr=None, 
out=None, **kwargs)

TFLite

inline void Conv(
const ConvParams& params,
const RuntimeShape& input_shape,                 
const uint8* input_data, 
const RuntimeShape& filter_shape,                 
const uint8* filter_data, 
const RuntimeShape& bias_shape,                 
const int32* bias_data, 
const RuntimeShape& output_shape,                 
uint8* output_data, 
const RuntimeShape& im2col_shape,                 
uint8* im2col_data, void* cpu_backend_context) {


Here ConvParams are

struct ConvParams {
  PaddingType padding_type;
  PaddingValues padding_values;
  // TODO(starka): This was just "stride", so check that width+height is OK.
  int16 stride_width;
  int16 stride_height;
  int16 dilation_width_factor;
  int16 dilation_height_factor;
  // uint8 inference params.
  // TODO(b/65838351): Use smaller types if appropriate.
  int32 input_offset;
  int32 weights_offset;
  int32 output_offset;
  int32 output_multiplier;
  int output_shift;
  // uint8, etc, activation params.
  int32 quantized_activation_min;
  int32 quantized_activation_max;
  // float activation params.
  float float_activation_min;
  float float_activation_max;
};

QNNPACK

enum qnnp_status qnnp_create_convolution2d_nhwc_q8(
    uint32_t input_padding_top,
    uint32_t input_padding_right,
    uint32_t input_padding_bottom,
    uint32_t input_padding_left,
    uint32_t kernel_height,
    uint32_t kernel_width,
    uint32_t subsampling_height,
    uint32_t subsampling_width,
    uint32_t dilation_height,
    uint32_t dilation_width,
    uint32_t groups,
    size_t group_input_channels,
    size_t group_output_channels,
    uint8_t input_zero_point,
    float input_scale,
    uint8_t kernel_zero_point,
    float kernel_scale,
    const uint8_t* kernel,
    const int32_t* bias,
    uint8_t output_zero_point,
    float output_scale,
    uint8_t output_min,
    uint8_t output_max,
    uint32_t flags,
    qnnp_operator_t* convolution_out)

---

QNN Proposal

def conv2d(data,
           weight,
           input_zero_point,
           kernel_zero_point,
           strides=(1, 1),
           padding=(0, 0),
           dilation=(1, 1),
           groups=1,
           channels=None,
           kernel_size=None,
           data_layout="NCHW",
           kernel_layout="OIHW",
           out_layout="",
           out_dtype="int32"):
    r"""Quantized 2D convolution.

    This operator convolves quantized weight with quantized data. The scale of
    the output quantized tensor is the product of the weight_scale and
    input_scale of the input quantized tensors. The zero point of the output
    quantized tensor is 0. By default, the dtype of output is int32. Please also
    refer to Requantize operator to understand how to scale back the int32
    output to (u)int8.


    Parameters
    ----------
    data : tvm.relay.Expr
        The input data to the operator.

    weight : tvm.relay.Expr
        The weight expressions.

    input_zero_point: int
           The zero point of the data distribution.

    kernel_zero_point: int
           The zero point of the quantized_kernel distribution.

    strides : tuple of int, optional
        The strides of convolution.

    padding : tuple of int, optional
        The padding of convolution on both sides of inputs before convolution.

    dilation : tuple of int, optional
        Specifies the dilation rate to be used for dilated convolution.

    groups : int, optional
        Number of groups for grouped convolution.

    channels : int, optional
        Number of output channels of this convolution.

    kernel_size : tuple of int, optional
        The spatial of the convolution weight.

    data_layout : str, optional
        Layout of the input.

    kernel_layout : str, optional
        Layout of the weight.

    out_layout : str, optional
        Layout of the output, by default, out_layout is the same as data_layout

    out_dtype : str, optional
        Specifies the output data type for mixed precision conv2d.

    Returns
    -------
    result : tvm.relay.Expr
        The computed result.
    """

Key points

• API is very close to TF API.
• Others have bias in the API to perform fused computation. Instead of complicating the Conv2D API, our proposal is to add a bias_add after the qnn.conv2d. Relay fusion will itself fuse the computation.
• TFLite and QNNPACK also have output_scale and output_zero_point. This is again because they want to fuse conv + bias + requantize in one op. We will have all of these 3 ops separately, connected by the framework parser depending on the framework graph. We will rely on Relay fusion.
• TFLite also has output_min_activation/max_Activation because of fused relu. We can easily call a clip operator to do that.

An example of conversion from TFLite to Relay graph will look like this.

[FrozenGene]

@anijain2305 Could we also list the api of TFLite and QNNPACK? I think both should be considered. Because we will parse TFLite model and QNNPACK is an good quantization accelerated library

[jnorwood]

The reference integer implementation of tflite conv2d has optional bias parameters which are pointers to 32 bit signed values, where a null pointer can be used to skip the operation.

Is it the intention to always separate out the bias add operation in tvm apis?

one other comment ... I've seen an implementation of quantized conv2d where the 32 bit signed bias values were preloaded to the accumulator, rather than clearing accumulator prior to the mac loop. I don't recall a discussion of it. I'm wondering if this a common enhancement to reduce overflow/underflow possibility of the signed 32 bit accumulations when a bias is expected?

[anijain2305]

@FrozenGene Updated the Conv2D API. Also, added a diagram explaining how to go from TFLite to Relay operators.

[anijain2305]

@jnorwood Yes, bias is kept outside as a separate operator. But, this can be fused with the qnn.conv2d.

Regarding the accumulation point, if we perform fusion and add the bias in int32 in the accumulator at the end, is it any different than preloading the accumulator? We need to ensure that op is fused i.e. the bias addition happens in the same accumulator where conv2d has just finished.