[Relay] Port LSTM to Relay for testing #2011

slyubomirsky · 2018-10-27T02:02:42Z

We would like to be able to evaluate Relay's performance on an LSTM, particularly since Relay can directly incorporate control flow. However, I could not find a concise example of an LSTM in NNVM to port over. @merrymercy pointed me to his own prior implementation of an LSTM cell, which is what I ported over.

However, in order to get this to match up with the other examples, I would also need to set up the rest of the network and load in a benchmark; I would appreciate any pointers as to how I could proceed, since I am not familiar with LSTMs. Namely, a pointer to NNVM implementations I could point over would be most helpful for being able to set up comparisons. (@tqchen, @merrymercy)

I would also appreciate any advice on how to best present this example (e.g., references to include).

Edit: Full disclosure, this variant of an LSTM is still an unrolled loop because we haven't merged in planned changes for abstract data types in Relay. Relay can currently handle a loop via recursion but without ADTs, it can't take in an arbitrary-length input list

slyubomirsky · 2018-10-27T02:03:58Z

python/tvm/relay/testing/lstm.py

+    h2h = layers.dense_add_bias(data=inputs, weight=h2h_weight,
+                                bias=h2h_bias, units=num_hidden * 4)
+
+    gates = relay.add(i2h, h2h)


I noticed in the example that the line read simply, i2h + h2h, which was the only place where a + operator was used. There were several elemwise_add calls otherwise. What was the reason for the distinction? Did the + mean to concatenate, rather than add?

I believe it is addition, not concatenation. My guess is the plus operator is used because there is no need to assign a name to gates, but there is for next_c.

I see, thanks

joshpoll · 2018-10-30T05:39:20Z

python/tvm/relay/testing/lstm.py

+from . import layers
+from .init import create_workload
+
+def lstm_cell(inputs, states, i2h_weight, h2h_weight,


See #2010 and #2009. Weights and biases should not be exposed as arguments to this function. Instead, you should end the Python function with

args = relay.ir_pass.free_vars(foo) return relay.Function(args, foo)

where foo is the final node in the network (i.e. what you are currently returning).

Duly noted, will change

joshpoll · 2018-10-30T05:39:51Z

python/tvm/relay/testing/lstm.py

+        The result.
+    """
+
+    i2h = layers.dense_add_bias(data=inputs, weight=i2h_weight,


Weight and bias should be left out. They will be created automatically by dense_add_bias

Interesting

slyubomirsky · 2018-10-30T22:04:10Z

python/tvm/relay/testing/lstm.py

+    states = relay.var("states",
+                       relay.TupleType([
+                           relay.TensorType((batch_size, num_hidden)),
+                           relay.TensorType((batch_size, num_hidden))]))


Are there any other explicit type annotations I could have in this function? It would be nice to annotate the return type on the function too, but I am not sure how to state it (do you have a suggestion, @jroesch?)

slyubomirsky · 2018-10-31T02:21:02Z

Something gives one way or another: Type unification is inadequate (I had to annotate the types because it would not be able to conclude that you can unify tuple(tensor, tensor) and tuple(unknown, unknown) -- there should really be a visitor for unification!) and the error reporting is worse (I know there's a shape mismatch somewhere but not where)

slyubomirsky · 2018-10-31T04:28:52Z

I will do a substantial refactor using the ScopeBuilder to see if that might help anything

slyubomirsky · 2018-10-31T23:07:56Z

Annotated every type and am no longer getting type errors in Relay (this points to some serious shortcomings in type unification and inference), but now it hangs on create_workload, possibly looping infinitely. Would appreciate pointers as to what could be done differently.

slyubomirsky · 2018-11-02T00:38:33Z

python/tvm/relay/testing/lstm.py

+    slice_gates = builder.let(("slice_gates", slice_type),
+                              relay.split(gates,
+                                          indices_or_sections=4,
+                                          axis=1).astuple())


@tqchen The default value for axis in split is zero, but the relation for split rejects an axis of 0. That doesn't seem right -- should the relation be corrected, or the default argument?

Split should be able to support axis=0

Will change it then (very easy)

…dn't)

tqchen · 2018-11-21T17:56:35Z

@slyubomirsky can you update this PR? it is ready for another look?

slyubomirsky · 2018-11-21T20:23:06Z

What updates, exactly, are needed? I addressed previous feedback. I suppose I can potentially reduce the number of explicit type annotations, though, so I will investigate whether that's possible.

tqchen · 2018-11-21T20:40:07Z

No, I just want to make sure if you need further updates, we can merge this in as it is

slyubomirsky · 2018-11-21T22:03:04Z

Ah, in that case I would say it's not waiting on further changes. We can perhaps simplify the Relay implementation in a follow-up PR

slyubomirsky commented Oct 27, 2018

View reviewed changes

joshpoll suggested changes Oct 30, 2018

View reviewed changes

slyubomirsky commented Oct 30, 2018

View reviewed changes

slyubomirsky force-pushed the lstm-relay-port branch from 80ccb97 to dc4b027 Compare October 31, 2018 23:02

slyubomirsky changed the title ~~[Relay][WIP] Port LSTM to Relay for testing~~ [Relay] Port LSTM to Relay for testing Nov 1, 2018

slyubomirsky commented Nov 2, 2018

View reviewed changes

slyubomirsky added 14 commits November 5, 2018 17:37

Port LSTM cell definition to Relay for testing

7d97224

Correct typo

3ca992a

Add docstring of questionable accuracy to LSTM

5dfe114

Refactor LSTM so it returns a Relay function

e206c13

Build up LSTM cells into an RNN and add tests

306cc01

Annotate all types in LSTM in hopes of getting it to typecheck (it di…

e4c626f

…dn't)

Correct the weight and bias shapes

e70089e

Use ScopeBuilder to annotate types at every step in LSTM

058d438

Rename ScopeBuilder to please pylint

37cfed7

Missed one

ae060b8

Rename input variable to be nicer with workload initializer

c63b286

Split should not reject axis of 0

2d4643d

Add axis = 0 regression test for split

c85be8e

Symbolic regression test for split on axis zero

2143e21

slyubomirsky force-pushed the lstm-relay-port branch from 3cbde11 to 2143e21 Compare November 6, 2018 01:51

icemelon added the status: need review label Nov 19, 2018

tqchen merged commit 3464827 into apache:master Nov 21, 2018

tqchen added status: accepted and removed status: need review labels Nov 21, 2018

FrozenGene pushed a commit to FrozenGene/tvm that referenced this pull request Dec 27, 2018

[Relay] Port LSTM to Relay for testing (apache#2011)

b5bb67e

wweic pushed a commit to neo-ai/tvm that referenced this pull request Feb 20, 2019

[Relay] Port LSTM to Relay for testing (apache#2011)

f174134

wweic pushed a commit to neo-ai/tvm that referenced this pull request Feb 20, 2019

[Relay] Port LSTM to Relay for testing (apache#2011)

f0e1685

[Relay] Port LSTM to Relay for testing #2011

[Relay] Port LSTM to Relay for testing #2011

Uh oh!

Conversation

slyubomirsky commented Oct 27, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

slyubomirsky Oct 27, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

slyubomirsky commented Oct 31, 2018

Uh oh!

slyubomirsky commented Oct 31, 2018

Uh oh!

slyubomirsky commented Oct 31, 2018

Uh oh!

slyubomirsky Nov 2, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tqchen commented Nov 21, 2018

Uh oh!

slyubomirsky commented Nov 21, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tqchen commented Nov 21, 2018

Uh oh!

slyubomirsky commented Nov 21, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

slyubomirsky commented Oct 27, 2018 •

edited

Loading

slyubomirsky Oct 27, 2018 •

edited

Loading

slyubomirsky Nov 2, 2018 •

edited

Loading

slyubomirsky commented Nov 21, 2018 •

edited

Loading