[TEST] test_roi_align in ONNX frontend tests taking 5+ mins of CI time

[denise-k]

Context

CI is taking a while (see #8552 and build#1384), and I am tracking short-term fixes to alleviate this

Summary of the issue

test_roi_align is pretty well tested at the relay and topi level as well as the frontends. However, the ONNX roi align tests take 7 minutes vs just 7 seconds on pytorch.

Part of this is because there are more tests on onnx with larger shapes.

# onnx (7min runtime)
    verify_roi_align((1, 4, 16, 16), 32, 7, 7, sampling_ratio=0, spatial_scale=1.0)
    verify_roi_align((4, 4, 16, 32), 32, 7, 7, sampling_ratio=0, spatial_scale=1.0)
    verify_roi_align((1, 8, 16, 16), 32, 7, 7, sampling_ratio=0, spatial_scale=1.0)
    verify_roi_align((1, 4, 8, 8), 32, 7, 7, sampling_ratio=0, spatial_scale=1.0)
    verify_roi_align((1, 4, 16, 16), 16, 5, 7, sampling_ratio=0, spatial_scale=1.0)
    verify_roi_align((1, 4, 16, 12), 8, 7, 3, sampling_ratio=0, spatial_scale=1.0)
    verify_roi_align((1, 4, 16, 16), 32, 7, 7, sampling_ratio=0, spatial_scale=0.5)
    verify_roi_align((3, 4, 12, 16), 32, 7, 7, sampling_ratio=0, spatial_scale=1.5)
    verify_roi_align((5, 4, 16, 14), 32, 7, 7, sampling_ratio=1, spatial_scale=1.0)
    verify_roi_align((1, 4, 16, 16), 32, 7, 7, sampling_ratio=2, spatial_scale=1.0)

# pytorch (7sec runtime)
    verify_model(ROIAlign(7), [in_data, in_boxes])
    verify_model(ROIAlign((10, 10), 0.7, 5), [in_data, in_boxes])
    verify_model(ROIAlign(15, 0.9, 3), [in_data, in_boxes])

Perhaps secondarily, the ONNX frontend is just slower at lowering ROI align.

Suggested actions

• Reduce the number of ROI align tests on the ONNX frontend. It's already pretty well covered at each level of the stack
• Minimize the input/output shapes of each ROI align test case

[mbrookhart]

@denise-k Something is weird here. When I run this locally on my machine, it takes 8 seconds, it's not even in the top 50 for slowest tests in the onnx test suite.

[denise-k]

thanks for looking into this, @mbrookhart.

this set of tests is pretty consistently slow on the CI machines:

I'll do a bit more digging in the logs and run this locally myself, to see how exactly the behavior changes from local vs. CI.

[mbrookhart]

I refactored the onnx tests, that went in this morning #8621

On that CI job, I saw this:

I wonder if the difference in times is just because I have more cores on my desktop machine. I'll try running on my laptop instead

[mbrookhart]

16 core AMD desktop:

8.67s call     tests/python/frontend/onnx/test_forward.py::test_roi_align[llvm]
6.86s call     tests/python/frontend/onnx/test_forward.py::test_roi_align[cuda]

6 core Intel laptop

183.00s call     tests/python/frontend/onnx/test_forward.py::test_roi_align[llvm]

There shouldn't be that big of a difference between these two machines, it's 2.5x the cores but 23x the performance

I will profile.

[mbrookhart]

Profile says...LLVM is taking a massive amount of time to compile the kernels for 2 of the unit tests. I realized I have llvm12 on my desktop but llvm10 on my laptop, so I upgraded my laptop to llvm12, rebuilt TVM, and then I get this on my laptop:

12.31s call     tests/python/frontend/onnx/test_forward.py::test_roi_align[llvm]

We should probably bump the version of LLVM we use in CI.

[mbrookhart]

FYI - it's the two tests with non-zero sampling ratio that LLVM 10 chokes on. I don't particularly want to remove those, since there are no other tests against the onnx implementaiton of those feature sets.

[Lunderberg]

Closing, as the CI performance was largely resolved, would need re-timing for further performance.