[microTVM] Generalize depthwise_conv2d schedule

[guberti]

This pull request removes a number of restrictions on the usage of the depthwise_conv2d schedule for microTVM. It:

• Adds support for the int16 input data type
• Adds support for arbitrarily-sized (including asymmetric) kernels
• Allows the int16 version to run on Cortex-M0 and M3 (which do not have SIMD instructions)
  • This is accomplished without performance loss
  • Adds unit tests for these features

It also removes use of the SMLAD instruction, which cannot improve performance on NHWC layouts (discussion about why in the comments below). This results in a slight performance boost for depthwise convolutions where kernel_width * kernel_height is odd, and no change otherwise. It also contains some readability improvements.

Known issue: my call to topi.reshape in depthwise_conv2d.py results in useless C code being generated that unnecessarily duplicates an array. The performance hit from this is small, but it is a gross issue.

[areusch]

cc @Mousius @ekalda @leandron

[guberti]

Why doesn't SMLAD improve performance?

Recall that the SMLAD instruction takes two int16*2 values x1::x2 and y1::y2 and an accumulator z, and computes z += x1 * y1 + x2 * y2. For NHWC layouts however, the relevant x1::x2 values in the input tensor are not next to each other, though. Previously, we used a DSP-specific halfword packing instruction PKHBT to fix this, and then called SMLAD after - two instructions for two multiplies.

However, there is a non-DSP instruction SMLAxy that is present on all Cortex-M cores (see docs). This instruction allows us to only read one 16-bit half of an int32 register while performing multiply accumulates, allowing us to skip the PKHBT instruction. Doing the multiplies this way is just as fast, while being way more versatile and simpler.

This means it is impossible to use SMLAD to speed up our performance for input tensors in NHWC format - nowhere in the input tensor is the relevant data in the correct format, which necessitates the use of at least one instruction to fix this. However, this extra instruction already removes all benefit of SMLAD compared to the non-DSP instruction SMLAxy.

Note that for the NCHW format, SMLAD would be very helpful. We should look into changing the format in the Relay graph, as this would yield a major performance improvement.

[areusch]

thanks @guberti , this looks pretty good. leaving a couple comments. @tkonolige, could you give the AlterOpLayout part a look?

[tkonolige]

You'll need to check that the kernel is a constant and fallback to a different implementation if it is not.

[guberti]

I'm not sure how easy it is to check if the kernel is a constant from python/tvm/relay/op/strategy/arm_cpu.py, but you're right that it is a thing we should check. I've added an assertion, though it is a bit of a stopgap solution.

[tkonolige]

Could you add a message to the assert and a comment about what needs to be done to not make it a stopgap.

[guberti]

Unfortunately, a clean solution is hard, as the strategy function does not have access to the needed information. When conv2d_alter_op is called, inputs[1] (the kernel) has the form:

meta[relay.Constant][0] /* ty=Tensor[(3, 3, 3, 8), int16] */

However when the Relay strategy functions are called, inputs[1] (the kernel) looks like:

Tensor(shape=[3, 3, 8, 1], op.name=placeholder)

Nowhere inside relay/op/strategy do any of the strategy functions check whether the relevant tensors are constant, so there's not much we can do. I've added comments explaining this, but please let me know if you have ideas for how this could be done.

[guberti]

Thanks for the detailed comments @areusch @tkonolige! I've addressed your comments with dee04b1 - please take another look.

[ekalda]

LGTM and interesting discussion about using DSP instructions for depthwise! cc @ashutosh-arm for visibility

[tkonolige]

Thanks @guberti!