[Relay] Enhance OpFusion for Tuple

[masahi]

The current fusion implementation cannot fuse concat with anything before it. For example, upsampling + concat, which is common in image segmentation networks, cannot be fused into a single op. The NNVM v1 fusion algorithm has no problem fusing them.

The problem is that the concat op takes a single TupleNode as input, but the current fusion implementation will not fuse two ops if there is a tuple node between them (code here). I tried to tweak the fusion algorithm by unpacking a tuple node etc, but so far I have no success.

Ideally I want to get rid off TupleNode and let Concat op take a variable number of arguments, just like NNVM v1 concat op does. Is it a good idea? Can Relay handle a op with a variable number of arguments?

cc @tqchen @junrushao1994 @jroesch @vinx13

[tqchen]

This can be fixed, but need a bit more effort on upgrading the current code. I will outline what needs to be done in here, @masahi let me know if you would like to take a stab.

The general idea is to make TupleNode part of dataflow analysis.

So when we construct the IndexedDataFlow graph, we add the tuple node as a node in the dataflow, the relation is injective. So the data-flow graph becomes:

a1 -> TupleNode-> concat
a2 /

Now that the fusion algorithm will then merge a1 with tuple, which in term will merge with concat. Note that if there is a case when a TupleNode stands alone, then we don;t need to generate a sub-function for it.

The implementation of TupleNode is likely going to be the same as the Call

[tqchen]

The introduction of tuple does bring certain burdens to the analysis, but on the other hand, also brings some potential future benefit(e.g. return a tuple of closure and Tensor), so likely we will stick to this and make sure our analysis can handle them

[masahi]

ok, I can work on this.