[RELAY][PASS] General OpFusion. #2090

tqchen · 2018-11-13T06:37:17Z

This is a new version of OpFusion algorithm that is cleaner and general than the one used in NNVMv1.

Method

The main challenge of genenral fusor is to handle possible diamond shape branches, in the following graph, conv2d can be fused to elemwise add.

             conv2d
            /  |  \
           /   |   \
         op    op   op
          \    |    /
           \   |   /
          elemwise add
               |

However, at the point of conv2d we do not necessarily know that all its future path will merge at the elemwise add. The new fusor algorithm applies post-dominator analysis. The immediate post-dominator of a node defined by the closest node where all the future path goes into. In the above case, the elemwise add is the immediate post-dominator of conv2d. The general algorithm is as follows:

Construct a DAG of dataflow graph for dominator analysis
Construct a post-dominator tree which gives immediate post-dominator of each node.
Run fusion algorithm with the given post-dominator information.

Note that, because we run analysis on a DAG, we use a single pass post-dominator
tree construction algorithm via LCA, which is simpler than the full version that handles cycles.

The fusion algorithm traverses from each node and checks if it can be fused to its immediate post dominator. It has to check the following things:

CheckPath: check all the path between a node and its immediate post-dominator
satiesfies the fuse condition.
Note that these intermediate node can already be fused with another nodes, the algorithm
will still run correctly.
CommitFuse: mark all the nodes between source and post-dominator as the same group.
We use an Union-Find data structure to manage the groups.

tqchen · 2018-11-13T06:39:15Z

cc @masahi @junrushao1994 @ZihengJiang @jroesch @zhiics @yzhliu please review

jroesch · 2018-11-13T09:29:48Z

src/relay/pass/fuse_ops.cc

+  class Creator;
+};
+
+// Creator a post donimator tree of the dataflow


typo dominator tree

Creator => Create

jroesch · 2018-11-13T09:33:19Z

src/relay/pass/fuse_ops.cc

+   * \param edge_pattern The edge pattern across all the parents.
+   * \return The least common acenstor of thw two.
+   */
+  static Node* LCA(Node* lhs,


Could we use a more descriptive name?

jroesch · 2018-11-13T09:33:57Z

src/relay/pass/fuse_ops.cc

+  /*!
+   * \brief Check all the node between src and sink satisfies fcond.
+   *
+   * The heck does not include, source and sink.


jroesch · 2018-11-13T09:36:54Z

src/relay/pass/fuse_ops.cc



-Expr FuseOps(const Expr& expr) {
+Expr FuseOps(const Expr& expr, int fuse_opt_level) {


I believe the pass should still take a module for the time being, and in the global var case recursively apply fusion. Our experiments with adding data types and using iterative structures requires that we apply these optimizations to all functions which are reachable from the main one.

I agree, however, I think we could wrap a FuseOps for envs via applies all, or get a list of functions via reachable analysis and then apply this Op.

Since this pass optionally requires the env, I feel having a reachable analysis to pull out the list of functions then do foreach apply is a better approach

jroesch · 2018-11-13T09:38:29Z

src/relay/pass/type_infer.cc

        !new_var->type_annotation.defined());

-    if (!need_update_type && !need_update_var && !need_update_call) return new_e;
+    bool need_update_fn = (


We can punt this conversation to the future, but @MarisaKirisame and I feel like this style of resolver is going to continue to cause problems as we continue. We can chat further in person.

masahi · 2018-11-13T11:59:37Z

src/relay/pass/fuse_ops.cc

+    // The parameters of the function.
+    Array<Var> params;
+    // The arguments to call the parameters.
+    Array<Expr> arguments;


call the parameters ?
Shouldn't it be call the function?

tqchen · 2018-11-13T16:13:51Z

Thanks, @masahi @jroesch @liangfu for the helpful reviews, I have updated the code accordingly.

junrushao · 2018-11-13T16:50:16Z

Could you elaborate the fusing condition?

Update: sorry, I got it in the code. ignore me

junrushao · 2018-11-13T16:56:53Z

So we are able to fuse a residual block into one op in relay, have I got it right?

zhiics · 2018-11-13T17:00:49Z

python/tvm/relay/build_module.py

-        func = ir_pass.fuse_ops(func)
+
+        if cfg.pass_enabled("OpFusion"):
+            func = ir_pass.fuse_ops(func, opt_level=2)


should the opt_level here be the default opt_level instead of 2, though default is 2? Or should it be 1 since level 1 enables fusion?

zhiics · 2018-11-13T17:03:16Z

src/common/arena.h

+   * \return The allocated object.
+   * \note The type T must be simple type, or only contain
+   *  memory allocated from the same arena.
+   *  Otherwise the destructor need to be called explicitly.


zhiics · 2018-11-13T17:17:47Z

src/relay/pass/fuse_ops.cc

+   * \param arena The arena used for node allocation.
+   * \param graph The graph to be analyze.
+   * \return The dominator tree of the graph.
+   * \note This algorithm makes use of the fact that graph is DAG,


Just double check, a node will only have at most one parent in this DAG, right?

A node can have multiple outputs in the DAG. A node will only have one parent in the dominator tree(it is immediate post-dominator)

@tqchen Thanks. I misunderstood. I thought the node in the LCA algorithm was a graph node.

zhiics · 2018-11-13T17:20:28Z

src/relay/pass/fuse_ops.cc

+    Array<Expr> arguments;
+    // Get a new parameter or allocate an old one
+    Var GetOrAllocParam(const Expr& expr, const Type& type) {
+      // run linear scan as most fused group contain only a few inputs.


tqchen · 2018-11-13T21:32:07Z

Thanks @jroesch @masahi @junrushao1994 @zhiics this is merged

masahi · 2018-11-13T22:08:56Z

@junrushao1994 No, all paths between the first conv op at the top and the elemwise add at the bottom need to contain only broadcast or elemwise ops, to be fused into a single op. Resblock has another conv ops on one of the paths, so it cannot be fused into a single op.

masahi · 2018-11-14T00:18:17Z

src/relay/pass/fuse_ops.cc

+      auto* link = arena_->make<LinkNode<IndexedForwardGraph::Edge> >();
+      link->value.node = parent;
+      link->value.pattern = pattern;
+      current->outputs.Push(link);


Just to clarify my confusion, is the name parent a good one here? From the code above, it looks like an edge is pointing to a parent node.
Naturally I think of a parent node as a node being pointed from. To me consumer seems a better name here.

I understand that this parent will indeed become the parent node in the post dom tree representation. But in this forward data flow graph, this parent node seems more like a child node.

masahi · 2018-11-14T00:27:15Z

src/relay/pass/fuse_ops.cc

+   *        The combined edge pattern across all the parents.
+   * \return The least common acenstor of thw two.
+   */
+  static Node* LeastCommonAcenstor(


Acenstor -> Ancestor !!

The comment above needs to be fixed too.

Thanks for pointing this out, sorry this is merged after your comment, this is fixed by #2098

masahi · 2018-11-14T01:12:30Z

src/relay/pass/fuse_ops.cc

  // First we convert all chains of fusable ops into
  // abstracted functions which we mark as primtive
  // then we convert these primtive functions into
  // new operators.


Does comment above still apply?

merrymercy · 2018-12-19T16:11:21Z

@masahi @tqchen

If we have a elemwise -> conv -> elemwise block, the current algorithm only fuses conv with the second elemwise. Why don't we fuse all of them?

tqchen · 2018-12-19T17:49:30Z

This is the restriction of current schedule, which avoids to fuse the input, we can change the rules to do so

[RELAY][PASS] General OpFusion.

ec68519

fix comment lint

3c8ebb2

jroesch reviewed Nov 13, 2018

View reviewed changes

masahi reviewed Nov 13, 2018

View reviewed changes

Fix review comments

e41c6e0

zhiics reviewed Nov 13, 2018

View reviewed changes

fix review comment

ea4ffb1

tqchen merged commit 4369b7f into apache:master Nov 13, 2018

masahi reviewed Nov 14, 2018

View reviewed changes

FrozenGene pushed a commit to FrozenGene/tvm that referenced this pull request Dec 27, 2018

[RELAY][PASS] General OpFusion. (apache#2090)

b4cd00b

ZihengJiang mentioned this pull request Feb 1, 2019

TVM 0.5 Release Note #2448

Closed

wweic pushed a commit to neo-ai/tvm that referenced this pull request Feb 20, 2019

[RELAY][PASS] General OpFusion. (apache#2090)

7f8b2d8

wweic pushed a commit to neo-ai/tvm that referenced this pull request Feb 20, 2019

[RELAY][PASS] General OpFusion. (apache#2090)

991707e



		Expr FuseOps(const Expr& expr) {
		Expr FuseOps(const Expr& expr, int fuse_opt_level) {

[RELAY][PASS] General OpFusion. #2090

[RELAY][PASS] General OpFusion. #2090

Uh oh!

Conversation

tqchen commented Nov 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Method

Uh oh!

tqchen commented Nov 13, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

masahi Nov 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tqchen commented Nov 13, 2018

Uh oh!

junrushao commented Nov 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

junrushao commented Nov 13, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tqchen commented Nov 13, 2018

Uh oh!

masahi commented Nov 13, 2018

Uh oh!

masahi Nov 14, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

masahi Nov 14, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

merrymercy commented Dec 19, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tqchen commented Dec 19, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

tqchen commented Nov 13, 2018 •

edited

Loading

masahi Nov 13, 2018 •

edited

Loading

junrushao commented Nov 13, 2018 •

edited

Loading

masahi Nov 14, 2018 •

edited

Loading

masahi Nov 14, 2018 •

edited

Loading

merrymercy commented Dec 19, 2018 •

edited

Loading