[RELAY][BACKEND] CompileEngine refactor.

[tqchen]

This PR attempts to clean up the interpreter and graph runtime compile by making interface consistent, and organize them into backend modules. It also added support of a CompileEngine that caches previously compiled the code and can be used as the backbone of AOT AND JIT. After this PR, I feel most of the compiler scaffolding is done for the graph compiler support.

• Move interpreter, graph runtime codegen to backend
• Introduce CompileEngine
• refactor Interpreter and graph runtime codegen to use CompileEngine
• change fuse_ops to create function inplace(not add to globalvar)
• Enhanced graph codegen to support tuple input/output case
  • The original graph format can be viewed as always returning a tuple
• Make relay.build and relay.optimize consistent with nnvm.
• Change executor interface to also take in ctx and target
  • Allows interpreter to optionally run gpu code

Notes on some observations I made during the refactor

• The graph runtime codegen need to implement a memory allocator pass to enable reuse of memory.
  • Will leave as a followup
• The current interpreter don't memoize(and it is important not to because single Expr can evaluate to the different value given different input)
  • It will cause explosive computation in pure data flow form.
  • so it is important to run interpreter in after let binding transformation.
  • We will likely need a liveness analysis to eagerly free memory in the interpreter if we hope to use it to run large networks.
• It might be a good time to think about what would heterogeneous device support looks like as per (cc @zhiics)

[tqchen]

cc @jroesch @eqy @MarisaKirisame @ZihengJiang please review

[jroesch]

I will review first thing in the morning.

[jroesch]

Why should the executor code be located in build_module?

[tqchen]

Mainly because the logic of graph build is here, and we can reuse the optimize function

[jroesch]

Overall looks good 👍, I think we should still lift all local functions into global definitions before optimizing, it is a good invariant and allows us to do global transformations (in follow-up passes) without having to rewrite the functions which depend on them. If we are worried about performance we can re-perform inning at the end.

@MarisaKirisame could you take a look at this? it effect us.

[tqchen]

re global function lifting. I did not do them in the fuseops to make sure that the current API do not depend heavily on module, once we introduce the entry_func field into module I think we will be more comfortable using modules, that can come as a followup

[MarisaKirisame]

@jroesch how do we do closure? If all function are global I dont think we even have the concept of closure, as we have no partial application.

[MarisaKirisame]

inputs = list(inputs[0])?

[tqchen]

Good catch, fixed by followup PR

[MarisaKirisame]

Overall pretty good, except that line which I think is incorrect.

[tqchen]

@jroesch @MarisaKirisame please take another look, I have rebased and add support for tuple input(concat)

[MarisaKirisame]

LGTM