Integrate Automated QDQ placement tool - Part 3

[willg-nv]

What does this PR do?

Type of change: new feature

Overview: This PR integrates automated QDQ placement tool to ModelOpt. This PR is 3/4 of the change. This PR contains the following changes:

1. Implements QDQAutotuner and Autotuner CLI interface
2. Implements Benchmark to measure E2E time of QDQ models.
3. unit tests for QDQ Autotuner and config.

Part 1: #701
Part 2: #702
Part 3: #703
Part 4: #704

Usage

python -m modelopt.onnx.quantization.autotune --model model.onnx

Testing

Implemented unit tests for QDQAutotuner and Config classes.

Before your PR is "Ready for review"

• Make sure you read and follow Contributor guidelines and your commits are signed.
• Is this change backward compatible?: Yes
• Did you write any new necessary tests?: Yes
• Did you add or update any necessary documentation?: No, document will be in part 4.
• Did you update Changelog?: No. change log will be in part 4.

Additional Information

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[willg-nv]

@vishalpandya1990 could you help me review this PR? thanks!

[vishalpandya1990]

@vishalpandya1990 could you help me review this PR? thanks!

Sorry for the delay. Added Ajinkya for review.

[gcunhase]

Is this conversion function needed or can we insert Q/DQ nodes already at the correct precision?

[gcunhase]

Tried the following test code:

    def test_export_quantized_model(self):
        """Test exporting quantized model with Q/DQ."""
        model = create_simple_conv_model()
        autotuner = QDQAutotuner(model)
        config = self._create_test_config()
        autotuner.initialize(config)

        with open("/tmp/autotuner_model.quant.onnx", "w") as f:  # tempfile.NamedTemporaryFile(suffix=".onnx", delete=False) as f:
            output_path = f.name

        try:
            # Export baseline without Q/DQ insertion
            autotuner.export_onnx(output_path, insert_qdq=True)

            # Verify file was created
            assert os.path.exists(output_path)

            # Verify it's a valid ONNX model
            exported_model = onnx.load(output_path)
            assert exported_model is not None

            # Verify that it contains Q/DQ nodes
            qdq_nodes = [n for n in exported_model.graph.node if n.op_type in ["QuantizeLinear", "DequantizeLinear"]]
            assert qdq_nodes, "Q/DQ nodes not found in quantized model"

            print("✓ QDQAutotuner export quantized model")
        finally:
            print()
            # if os.path.exists(output_path):
            #     os.unlink(output_path)

But the simple Conv->Relu model didn't get quantized. Is this expected?

[modelopt][onnx] - DEBUG - Region 0 (level 0)
[modelopt][onnx] - DEBUG -   → Pattern signature: Conv->Relu
[modelopt][onnx] - DEBUG -   → No scheme available, skipping
[modelopt][onnx] - DEBUG - Matched 0/1 regions, total 0 unique insertion points
[modelopt][onnx] - DEBUG - Inserting 0 Q/DQ pairs into graph
[modelopt][onnx] - DEBUG - Serializing to ONNX format
[modelopt][onnx] - INFO - Exported INT8 model with 0 Q/DQ pairs  → /tmp/autotuner_model.quant.onnx
✓ QDQAutotuner export quantized model

[willg-nv]

I think the above result is expected. because export_onnx(insert_qdq=True) means use the autotune insertion points to insert Q/DQ. Since the regions in autotuner is not tuned, there should be no QDQ node inserted.

[willg-nv]

For model = int8_to_fp8(model), I don't know how to create fp8 QDQ ONNX natively. So I use int8 Q/DQ nodes and converts to fp8.

[gcunhase]

What's the expected scheme_idx for create_simple_conv_model()? Please update this assert accordingly. Thanks.

[gcunhase]

Can we add a test file for workflows.py and potentially benchmark.py?

cli.py can be moved to __main__.py if the plan is for it to be available as a standalone feature and/or for debugging purposes. Unittests might also need to be added for that.

[gcunhase]

The following CLI fails to perform benchmark / quantize the model (this uses TensorRTPyBenchmark):

$ python -m modelopt.onnx.quantization.autotune --onnx_path=conv_relu.onnx

Error:

[modelopt][onnx] - ERROR - Benchmark instance not initialized
[modelopt][onnx] - INFO - Results: 3.73 ms → failed (invalid measurement)

This failure happens because pycuda was not installed. After installing that dependency, no error is thrown but the model is not quantized.

• @ajrasane should we create another optional_dep in setup.py with autotune's dependencies?

[gcunhase]

If --use_trtexec is used, autotune does not fail but also doesn't generate a quantized model.

This is due to Latency being used as a measurement instead of GPU Compute Time.

[ajrasane]

If it is just pycuda, we can probably just include this in the modelopt onnx dependencies. But if we have more dependencies, then it would be better to create a new section in setup.py with autotune dependencies.

[gcunhase]

@willg-nv how should we approach the tensorrt / trtexec requirements for autotune? Are we just adding a disclaimer for the user in the README or adding that in setup.py?

[gcunhase]

Can we add a test file for workflows.py and potentially benchmark.py?

cli.py can be moved to __main__.py if the plan is for it to be available as a standalone feature and/or for debugging purposes. Unittests might also need to be added for that.

Suggestion for test_workflows.py: test_workflows.py