[Unity][Hexagon] Allow scalar tensors to have null shape during allocation #14427

ibsidorenko · 2023-03-29T15:08:57Z

Update assert during buffer allocation. Zero dimensional tensors (scalars) can have a null shape.

This PR is attempt to port PR#14376 from tvm/main that enables Hexagon tests with Relax.

This PR implements a flexible register-based VM to execute relax programs with dynamic shape and control flow. Design: https://github.com/tlc-pack/relax/wiki/Relax-VM-Design. Co-Authored-by: Ziheng Jiang <ziheng@apache.org> Co-Authored-by: Ruihang Lai <ruihangl@cs.cmu.edu> Co-Authored-by: Sunghyun Park <49998730+sunggg@users.noreply.github.com> Co-Authored-by: Junru Shao <junrushao1994@gmail.com> Co-Authored-by: Prakalp Srivastava <prakalp@octoml.ai> Co-Authored-by: Yong Wu <yongcale@gmail.com> Co-Authored-by: Steven S. Lyubomirsky <slyubomirsky@octoml.ai> Co-Authored-by: Tianqi Chen <tianqi.tchen@gmail.com> Co-Authored-by: Hongyi Jin <3231950289@qq.com>

* [Unity][IR] First-class StructInfo Relax tracks structural information (such as tensor shape) via `StructInfo` about the values in Relax. * Fix rust build --------- Co-authored-by: Junru Shao <junrushao1994@gmail.com>

…pache#13910) This PR setup a unity specific jenkins with minimum jenkinsfile without sharding and disables most of the tests to reduce overall cost. We can add tests of unty branch by configuring the specific groovy file.

[Unity] Basic StructInfo Analysis and Expr construction. This PR adds struct info analysis and expr support. These are logics to construct the IR node and perform struct info related analysis. Testcases are added to cover the IR node construction and related struct info analysis checks. Co-authored-by: Tianqi Chen <tianqi.tchen@gmail.com> Co-authored-by: Altan Haan <altanh@cs.washington.edu> Co-authored-by: Andrew Liu <andrewlliu@gmail.com> Co-authored-by: Hongyi Jin <3231950289@qq.com> Co-authored-by: Jiawei Liu <jaway.liu@gmail.com> Co-authored-by: Junru Shao <junrushao1994@gmail.com> Co-authored-by: Lesheng Jin <34279105+LeshengJin@users.noreply.github.com> Co-authored-by: masahi <masahi129@gmail.com> Co-authored-by: Prakalp Srivastava <prakalp@octoml.ai> Co-authored-by: Ruihang Lai <ruihangl@cs.cmu.edu> Co-authored-by: Siyuan Feng <Hzfengsy@sjtu.edu.cn> Co-authored-by: Steven S. <Lyubomirsky slyubomirsky@octoml.ai> Co-authored-by: Sunghyun Park <49998730+sunggg@users.noreply.github.com> Co-authored-by: Yixin Dong <ubospica@gmail.com> Co-authored-by: Yong Wu <yongcale@gmail.com> Co-authored-by: Ziheng Jiang <ziheng@apache.org>

This PR adds BlockBuilder: the core data structure to construct Relax AST, and ExprMutator: performs AST mutation for implementing transformation passes. Co-Authored-by: Tianqi Chen <tianqi.tchen@gmail.com> Co-Authored-by: Altan Haan <altanh@cs.washington.edu> Co-Authored-by: Andrew Liu <andrewlliu@gmail.com> Co-Authored-by: Hongyi Jin <3231950289@qq.com> Co-Authored-by: Jiawei Liu <jaway.liu@gmail.com> Co-Authored-by: Junru Shao <junrushao1994@gmail.com> Co-Authored-by: Lesheng Jin <34279105+LeshengJin@users.noreply.github.com> Co-Authored-by: masahi <masahi129@gmail.com> Co-Authored-by: Prakalp Srivastava <prakalp@octoml.ai> Co-Authored-by: Ruihang Lai <ruihangl@cs.cmu.edu> Co-Authored-by: Siyuan Feng <Hzfengsy@sjtu.edu.cn> Co-Authored-by: Steven S. <Lyubomirsky slyubomirsky@octoml.ai> Co-Authored-by: Sunghyun Park <49998730+sunggg@users.noreply.github.com> Co-Authored-by: Yixin Dong <ubospica@gmail.com> Co-Authored-by: Yong Wu <yongcale@gmail.com> Co-Authored-by: Ziheng Jiang <ziheng@apache.org>

This PR adds the TVMScript parser/ir_builder support based on the blockbuilder. Co-authored-by: Ruihang Lai <ruihangl@cs.cmu.edu> Co-authored-by: Junru Shao <junrushao1994@gmail.com> Co-authored-by: Tianqi Chen <tianqi.tchen@gmail.com> Co-authored-by: Yuchen Jin <yuchenj@cs.washington.edu> Co-authored-by: Steven S. Lyubomirsky <slyubomirsky@gmail.com> Co-authored-by: Yong Wu <yongcale@gmail.com>

This PR introduces Relax as a dialect supported by the TVMScript Printer. Some caveats: - Needs to rebase to mainline before merging. - Some tests are skiped because some operators are not upstreamed to the unity branch yet. Co-authored-by: Tianqi Chen <tianqi.tchen@gmail.com> Co-authored-by: Yuchen Jin <yuchenj@cs.washington.edu> Co-authored-by: Steven S. Lyubomirsky <slyubomirsky@gmail.com> Co-authored-by: Yong Wu <yongcale@gmail.com> Co-authored-by: Prakalp Srivastava <prakalp@octoml.ai> Co-authored-by: Sunghyun Park <49998730+sunggg@users.noreply.github.com> Co-authored-by: Ruihang Lai <ruihangl@cs.cmu.edu> Co-authored-by: Hongyi Jin <3231950289@qq.com> Co-authored-by: Bohan Hou <32121147+spectrometerHBH@users.noreply.github.com> Co-authored-by: Siyuan Feng <Hzfengsy@sjtu.edu.cn>

This PR introduces Relax `FunctionPass` and `DataflowBlockPass` API, and the `VMShapeLower` pass to lower the shape expression in Relax to TIR functions and VM shape heap builtin functions. Co-Authored-by: Ziheng Jiang <ziheng@apache.org> Co-Authored-by: Lesheng Jin <34279105+LeshengJin@users.noreply.github.com> Co-Authored-by: Altan Haan <altanh@cs.washington.edu> Co-Authored-by: Junru Shao <junrushao1994@gmail.com> Co-Authored-by: Prakalp Srivastava <prakalp@octoml.ai> Co-Authored-by: Ruihang Lai <ruihangl@cs.cmu.edu> Co-Authored-by: Siyuan Feng <Hzfengsy@sjtu.edu.cn> Co-Authored-by: Steven S. <Lyubomirsky slyubomirsky@octoml.ai> Co-Authored-by: Sunghyun Park <49998730+sunggg@users.noreply.github.com> Co-Authored-by: Tianqi Chen <tianqi.tchen@gmail.com> Co-Authored-by: Yong Wu <yongcale@gmail.com>

This PR introduces the e2e Relax lowering flow (`relax.vm.build`). Tests for each pass in the flow are added. Co-Authored-by: Altan Haan <altanh@cs.washington.edu> Co-Authored-by: Andrew Liu <andrewlliu@gmail.com> Co-Authored-by: Hongyi Jin <3231950289@qq.com> Co-Authored-by: Jiawei Liu <jaway.liu@gmail.com> Co-Authored-by: Junru Shao <junrushao1994@gmail.com> Co-Authored-by: Prakalp Srivastava <prakalp@octoml.ai> Co-Authored-by: Ruihang Lai <ruihangl@cs.cmu.edu> Co-Authored-by: Siyuan Feng <Hzfengsy@sjtu.edu.cn> Co-Authored-by: Steven S. <Lyubomirsky slyubomirsky@octoml.ai> Co-Authored-by: Sunghyun Park <49998730+sunggg@users.noreply.github.com> Co-Authored-by: Tianqi Chen <tianqi.tchen@gmail.com> Co-Authored-by: Yong Wu <yongcale@gmail.com> Co-Authored-by: Ziheng Jiang <ziheng@apache.org>

As we've introduced `arg_sinfo` in CallNode, implicit shape constructor is not widely used in TVMScript. This PR removes the implicit shape since it may cause confusion between shape and tuple.

This PR is about the high-level tensor computation operators in Relax. This PR includes the tensor indexing operators.

This PR is about the high-level tensor computation operators in Relax. This PR includes the set operators. Co-authored-by: Prakalp Srivastava <prakalp@octoml.ai>

This PR is about the high-level tensor computation operators in Relax. This PR includes the image operators.

This PR is about the high-level tensor computation operators in Relax. This PR includes the unary, binary and ternary arithmetic and comparison operators. Co-authored-by: Siyuan Feng <Hzfengsy@sjtu.edu.cn> Co-authored-by: Chaofan Lin <1713833595@qq.com>

This PR is about the high-level tensor computation operators in Relax. This PR includes the statistical operators.

This PR is about the high-level tensor computation operators in Relax. This PR includes the neural network operators.

This PR is about the high-level tensor computation operators in Relax. This PR includes the tensor creation operators.

This PR is about the high-level tensor computation operators in Relax. This PR includes the linear algebra operators. Co-authored-by: Siyuan Fneg <Hzfengsy@sjtu.edu.cn>

This PR is about the high-level tensor computation operators in Relax. This PR includes the search operators.

This PR is about the high-level tensor computation operators in Relax. This PR includes the tensor manipulation operators. Co-authored-by: Prakalp Srivastava <prakalp@octoml.ai>

This PR introduce NestedMsg to robustly handle nested-tuple analysis. Relax support nested tuple structures in the IR. Nested tuple structure is important to support advanced groupings in cases such as gradient calculation and other scenarios. The possible presence of nested tuple does mean that we need to to robustly handle analysis that contains nested tuple structures in a dataflow graph. This PR introduces a NestedMsg<T> class that corresponds to a possibly nested message tuple for a given leaf message class T. We also introduces various helper functions to compose and decompose messages. Co-authored-by: Bohan Hou <32121147+spectrometerHBH@users.noreply.github.com> Co-authored-by: Yixin Dong <ubospica@gmail.com> Co-authored-by: Ruihang Lai <ruihangl@cs.cmu.edu>

[Unity][Pass] Operator fusion passes This PR introduces three passes for operator fusion: 1. AnnotateTIROpPattern: analysis the operator kind from PrimFunc. 2. FuseOps: fuse operators for Relax functions, which adds a new fused relax primitive function. 3. FuseTIR: fuse corresponding TIR PrimFuncs for the fused relax.

[VM] Supporting "compiled" exec mode. This PR adds support of "compiled" mode to the VM. The compiled mode translate the relax function into TIR function and drive it through the TIR function. It is different from the micro AOT codegen, which generate TIR code that targets the micro C runtime environment and useful for resource limited settings with smaller set of features. Both leverages the low-level TIR build that is also shared with TensorIR. The current implementation targets full TVM (VM) runtime, that comes with PackedFunc, object, tuple, closure and all kinds of rich structure support. This also mean that we can leverage the full runtime support to handle things like allocation, dynamic shape, easy plugins and python interaction, which are not available in more limited runtime. The user directly use the same API to load the generated code regardless of compiled mode or bytecode. And just need to change one line ```python ex = relax.vm.build(mod, target, exec_mode="compiled") ``` The simplicity is thanks to the TVM runtime archiecture that allows us to compose things together in objects. The only difference is how the PackedFunc of high-level driving is being provided. In the case of bytecode it is normal interpretation and in the case of compiled mode it is TIR. It is a complete implementation Unit-testcases are added. All codegen build tests are updated to include two exec_modes and have passed locally. Co-authored-by: Junru Shao <junrushao1994@gmail.com>

This PR introduces FoldConstant/BindParam passes. Co-authored-by: Yong Wu <yongcale@gmail.com> Co-Authored-by: Hongyi Jin <3231950289@qq.com> Co-Authored-by: Siyuan Feng <Hzfengsy@sjtu.edu.cn>

…pache#14014) Add TuningAPI and MetaSchedule tuning pass

This PR implements a Relay to Relax translator, which allows us to import Relay workloads to Relax for benchmarking and development purposes (tests and examples are added).

Also include output dtype in simt MathInstruction.

This PR added Relax VM builtin functions to execute with CUDA graph. - vm.builtin.cuda_graph.get_cached_alloc: Allocate and cache storage objects for future vm invocation - vm.builtin.cuda_graph.run_or_capture: Launched captured CUDA graph or capture the CUDA graph using CUDA API and save in the cache The graph rewriting to enable CUDA graph backend will be done in a separate PR.

The file tests/cpp/nested_msg_test.cc may fail to compile if <array> is not included explicitly.

@jinhongyii

…14274) Currently, the BYOC system is based on op-level pattern matching, this PR intends to provide primary support for TIR-level pattern matching based on backend registration and dispatching. For now, it simply matches the first set of for loops in PrimFunc. Co-authored-by: Hongyi Jin (@jinhongyii)

This PR adds support for simple dynamic-shape-aware fusion, which is the first step towards supporting dynamic shapes. The main changes are as follows: - Fix FuncStructInfo in well-formed checks - Renew symbolic var defs in fuse_ops to prevent malformed functions

This PR adds a stop_lift_params op to as a hint to the parameter lifter to stop at that boundary point.

…che#14404) This PR enables relax parser to handle Var with ShapeExpr value occuring in R.Tensor annotations.

* [Unity][Pass] Add pass for CSE within dataflow * Fill in CSE definition and test cases * Missing trailing newline --------- Co-authored-by: Prakalp Srivastava <prakalp@octoml.ai>

…ns (apache#14386) * Support Relax Constants in the QNN TOPI operations

This PR implements Conv1d. Unit tests are provided accordingly.

…pache#14412) This PR exposes the custom scale in `R.nn.attention` and adds its legalize op.

@psrivas2

This is attempt to port PR (tlc-pack/relax#167) (submitted by @psrivas2 and @YuchenJin) from tlc-pack/relax to enable Hexagon tests with Relax VM.

)

…ation Update assert during buffer allocation. Zero dimensional tensors (scalars) can have a null shape. This PR is attempt to port PR#14376 from tvm/main that enables Hexagon tests with Relax.

tvm-bot · 2023-03-29T15:09:01Z

Thanks for contributing to TVM! Please refer to the contributing guidelines https://tvm.apache.org/docs/contribute/ for useful information and tips. Please request code reviews from Reviewers by @-ing them in a comment.

cc @quic-sanirudh _{See #10317 for details}

_{Generated by tvm-bot}

ibsidorenko · 2023-03-29T15:17:29Z

cc @csullivan @farshidsp @janetsc @tqchen

janetsc

Thanks for porting this fix!

farshidsp · 2023-03-29T15:43:21Z

Thanks for this PR. LGTM.

masahi · 2023-03-29T19:13:48Z

Can we wait for the next rebase instead of making the same change?

ibsidorenko · 2023-03-29T19:34:34Z

Can we wait for the next rebase instead of making the same change?

Sure, we can wait If rebase is planned in the near future.

tqchen · 2023-03-30T14:38:46Z

yes, we plan to do it sometime this weekend

tqchen · 2023-04-01T20:42:09Z

should be in now

YuchenJin and others added 30 commits March 20, 2023 10:59

[Unity] Relax expressions and types (apache#13901)

2c7f480

[Unity][IR] First-class StructInfo (apache#13907)

4e659d1

* [Unity][IR] First-class StructInfo Relax tracks structural information (such as tensor shape) via `StructInfo` about the values in Relax. * Fix rust build --------- Co-authored-by: Junru Shao <junrushao1994@gmail.com>

[Unity] Relax VM codegen (apache#13954)

ea6cc94

[Unity][TVMScript] Use explicit R.shape in TVMScript (apache#13979)

75eecf7

As we've introduced `arg_sinfo` in CallNode, implicit shape constructor is not widely used in TVMScript. This PR removes the implicit shape since it may cause confusion between shape and tuple.

[Unity] Relax op: index (apache#13987)

de164d2

This PR is about the high-level tensor computation operators in Relax. This PR includes the tensor indexing operators.

[Unity] Relax op: datatype (apache#13986)

c8a1533

[Unity] Relax op: set (apache#13990)

c7a57ae

This PR is about the high-level tensor computation operators in Relax. This PR includes the set operators. Co-authored-by: Prakalp Srivastava <prakalp@octoml.ai>

[Unity] Relax op: image (apache#13994)

4240920

This PR is about the high-level tensor computation operators in Relax. This PR includes the image operators.

[Unity] Relax op: statistical (apache#13991)

a6a2e84

This PR is about the high-level tensor computation operators in Relax. This PR includes the statistical operators.

[Unity] Relax op: neural networks (apache#13993)

a96f200

This PR is about the high-level tensor computation operators in Relax. This PR includes the neural network operators.

[Unity] Relax op: creation (apache#13984)

4dd591b

This PR is about the high-level tensor computation operators in Relax. This PR includes the tensor creation operators.

[Unity] Relax op: linear algebra (apache#13988)

9694c67

This PR is about the high-level tensor computation operators in Relax. This PR includes the linear algebra operators. Co-authored-by: Siyuan Fneg <Hzfengsy@sjtu.edu.cn>

[Unity] Relax op: search (apache#13992)

7a8765d

This PR is about the high-level tensor computation operators in Relax. This PR includes the search operators.

[Unity] Relax op: manipulation (apache#13989)

4ab73ea

This PR is about the high-level tensor computation operators in Relax. This PR includes the tensor manipulation operators. Co-authored-by: Prakalp Srivastava <prakalp@octoml.ai>

[Unity][Pass] LambdaLift pass (apache#14012)

e78d523

[Unity][Pass] BindParams pass, FoldConstant pass (apache#14016)

63e2402

This PR introduces FoldConstant/BindParam passes. Co-authored-by: Yong Wu <yongcale@gmail.com> Co-Authored-by: Hongyi Jin <3231950289@qq.com> Co-Authored-by: Siyuan Feng <Hzfengsy@sjtu.edu.cn>

[Unity][Pass][TuningAPI] Introduce TuningAPI and MetaSchedule pass (a…

87659ea

…pache#14014) Add TuningAPI and MetaSchedule tuning pass

[Unity] Relay -> Relax translator (apache#14026)

388941a

This PR implements a Relay to Relax translator, which allows us to import Relay workloads to Relax for benchmarking and development purposes (tests and examples are added).

areusch and others added 16 commits March 22, 2023 18:53

[Unity] Also include output dtype in simt MathInstruction (apache#14372)

4e46ad4

Also include output dtype in simt MathInstruction.

[Unity] Add missing #include <array> (apache#14383)

cc03017

The file tests/cpp/nested_msg_test.cc may fail to compile if <array> is not included explicitly.

[Unity][Op] Add stop_lift_params (apache#14368)

eff9a0a

This PR adds a stop_lift_params op to as a hint to the parameter lifter to stop at that boundary point.

[Unity][TVMScript] Fix Shape Var occurrence in Tensor annotation (apa…

d377f69

…che#14404) This PR enables relax parser to handle Var with ShapeExpr value occuring in R.Tensor annotations.

[Unity][Transform] Common Subexpression Elimination (apache#14361)

f4d5964

* [Unity][Pass] Add pass for CSE within dataflow * Fill in CSE definition and test cases * Missing trailing newline --------- Co-authored-by: Prakalp Srivastava <prakalp@octoml.ai>

[Unity][QNN][Hexagon]Support Relax Constants in the QNN TOPI operatio…

d97c43b

…ns (apache#14386) * Support Relax Constants in the QNN TOPI operations

[Unity][Op] Conv1d (apache#14388)

9a3ec23

This PR implements Conv1d. Unit tests are provided accordingly.

[Unity] Fix getting shapes for cutlass BYOC kernels (apache#14411)

cd3e107

[Unity][Op] Expose scale in R.nn.attention and add its legalize op (a…

23146d6

…pache#14412) This PR exposes the custom scale in `R.nn.attention` and adds its legalize op.

[Unity][Hexagon] Enable Relax VM for Hexagon (apache#14415)

854f2e9

This is attempt to port PR (tlc-pack/relax#167) (submitted by @psrivas2 and @YuchenJin) from tlc-pack/relax to enable Hexagon tests with Relax VM.

[Unity][Fix] Copy over module attrs in FuseTIR (apache#14418)

51b29ef

[Unity] Handle extern func calls in static memory planning (apache#14419

465994e

)

[Unity][Hexagon] Allow scalar tensors to have null shape during alloc…

9a2a244

…ation Update assert during buffer allocation. Zero dimensional tensors (scalars) can have a null shape. This PR is attempt to port PR#14376 from tvm/main that enables Hexagon tests with Relax.

janetsc approved these changes Mar 29, 2023

View reviewed changes

tqchen approved these changes Mar 30, 2023

View reviewed changes

tqchen force-pushed the unity branch 2 times, most recently from a425bc7 to 5c8b7af Compare April 1, 2023 20:00

tqchen closed this Apr 1, 2023

ibsidorenko deleted the unity-hexagon-enabling-v2 branch April 3, 2023 06:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Unity][Hexagon] Allow scalar tensors to have null shape during allocation #14427

[Unity][Hexagon] Allow scalar tensors to have null shape during allocation #14427

Uh oh!

ibsidorenko commented Mar 29, 2023

Uh oh!

tvm-bot commented Mar 29, 2023

Uh oh!

ibsidorenko commented Mar 29, 2023

Uh oh!

janetsc left a comment

Uh oh!

farshidsp commented Mar 29, 2023

Uh oh!

masahi commented Mar 29, 2023

Uh oh!

ibsidorenko commented Mar 29, 2023 •

edited

Loading

Uh oh!

tqchen commented Mar 30, 2023

Uh oh!

tqchen commented Apr 1, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

[Unity][Hexagon] Allow scalar tensors to have null shape during allocation #14427

[Unity][Hexagon] Allow scalar tensors to have null shape during allocation #14427

Uh oh!

Conversation

ibsidorenko commented Mar 29, 2023

Uh oh!

tvm-bot commented Mar 29, 2023

Uh oh!

ibsidorenko commented Mar 29, 2023

Uh oh!

janetsc left a comment

Choose a reason for hiding this comment

Uh oh!

farshidsp commented Mar 29, 2023

Uh oh!

masahi commented Mar 29, 2023

Uh oh!

ibsidorenko commented Mar 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tqchen commented Mar 30, 2023

Uh oh!

tqchen commented Apr 1, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

ibsidorenko commented Mar 29, 2023 •

edited

Loading