[BYOC][NNAPI]: Implement basic structure of Android NNAPI BYOC #8076

melsonlai · 2021-05-19T13:03:57Z

This commit implements the basic structure of Android NNAPI codegen with the BYOC mechanism:

Basic graph partition using pattern-based rules
RPC-based graph partition
relay.ext.android_nnapi BYOC codegen

Thanks for contributing to TVM! Please refer to guideline https://tvm.apache.org/docs/contribute/ for useful information and tips. After the pull request is submitted, please request code reviews from Reviewers by @ them in the pull request thread.

melsonlai · 2021-05-19T16:18:43Z

cc @comaniac @zhiics

comaniac

Sorry for the late review. In general as I mentioned in the RFC, this PR is too large for people to review and I believe this is the major reason why it's hard to collect feedbacks. I just reviewed the high-level Python parts and will try to review the rest parts when I got time. Meanwhile, It would be better to remove most non-infra parts such as operators first and send follow-up PRs.

In addition, I'm still not very comfortable with the way of verifying the generated C++ code using text comparison. After the NNAPI compiler is available in the CI, we could just verify if the generated code can be successfully compiled, but we probably cannot verify its run time functionality as other BYOC backends.

cc @zhiics

comaniac · 2021-06-16T23:06:53Z

python/tvm/contrib/target/android_nnapi/relayir_to_nnapi_converter/__init__.py

+    Note
+    ----


Suggested change

Note

----

Notes

-----

comaniac · 2021-06-16T23:10:37Z

python/tvm/contrib/target/android_nnapi/relayir_to_nnapi_converter/__init__.py

+from .converter import Converter
+
+
+def convert_relayir_to_nnapi(func):


I won't call this "convert". Following the naming convention of existing BYOC backends, this is the codegen/compiler of NNAPI, so I'll probably just call it nnapi_compiler. Also, can you register it like the following?

@tvm._ffi.register_func("relay.ext.nnapicompiler") def nnapi_compiler(func):

Meanwhile, the directory name relayir_to_nnapi_converter should also be improved. Maybe just name it compiler or codegen.

Hi, I plan to name it "codegen" if that's OK. (I don't want to collide with the Python built-in function "compile")
As an extra explanation that matters, the external compiler is actually a C++ stub in src/relay/backend/contrib/android_nnapi/codegen.cc, and it's that thing that gets registered as relay.ext.android_nnapi.
The existence of the C++ stub is a historical issue, however, it's not a complex file after all, so would you agree if I completely remove the C++ stub and make all of relay.ext.android_nnapi pythonic?

We actually prefer a reverse way that makes most of the implementations in C++ instead of Python for better performance. However, given that would be a huge effort, it should be fine for this specific backend to be all in Python IMHO.

comaniac · 2021-06-16T23:13:46Z

python/tvm/contrib/target/android_nnapi/relayir_to_nnapi_converter/_export_object/operand.py

+        -------
+        dtype: str
+            dtype of the queried operand
+


remove empty line. (ditto to other functions in all Python files).

comaniac · 2021-06-16T23:14:50Z

python/tvm/contrib/target/android_nnapi/relayir_to_nnapi_converter/converter.py

+    """Converts a Relay IR Function into Android NNAPI C++ class
+
+    Parameters
+    ----------------------


Suggested change

----------------------

----------

comaniac · 2021-06-16T23:15:45Z

python/tvm/contrib/target/android_nnapi/relayir_to_nnapi_converter/converter.py

+
+    """
+
+    DEFAULT_OPTIONS = {


Add more comments to explain each fields, such as name and api_level.

comaniac · 2021-06-16T23:26:40Z

python/tvm/contrib/target/android_nnapi/relayir_to_nnapi_converter/_export_object/operand.py

+        if value_dict["type"] == "memory_ptr":
+            return value_dict["value"]
+        assert False, "Unreachable"


Suggested change

if value_dict["type"] == "memory_ptr":

return value_dict["value"]

assert False, "Unreachable"

assert value_dict["type"] == "memory_ptr":

return value_dict["value"]

So that you don't need to disable inconsistent-return-statements.

comaniac · 2021-06-16T23:26:56Z

python/tvm/contrib/target/android_nnapi/relayir_to_nnapi_converter/_export_object/operand.py

+
+        Returns
+        -------
+        obj:


comaniac · 2021-06-16T23:27:22Z

python/tvm/contrib/target/android_nnapi/relayir_to_nnapi_converter/_export_object/operand.py

+            return None
+        return self._export_obj["constants"][value_dict["value"]]
+
+    def is_FuseCode(self, idx):  # pylint: disable=invalid-name


Suggested change

def is_FuseCode(self, idx): # pylint: disable=invalid-name

def is_fuse_code(self, idx):

comaniac · 2021-06-16T23:28:00Z

python/tvm/contrib/target/android_nnapi/relayir_to_nnapi_converter/_export_object/operand.py

+        if value not in {
+            "ANEURALNETWORKS_FUSED_NONE",
+            "ANEURALNETWORKS_FUSED_RELU",
+            "ANEURALNETWORKS_FUSED_RELU1",
+            "ANEURALNETWORKS_FUSED_RELU6",
+        }:
+            return False
+        return True


Suggested change

if value not in {

"ANEURALNETWORKS_FUSED_NONE",

"ANEURALNETWORKS_FUSED_RELU",

"ANEURALNETWORKS_FUSED_RELU1",

"ANEURALNETWORKS_FUSED_RELU6",

}:

return False

return True

return value in {

"ANEURALNETWORKS_FUSED_NONE",

"ANEURALNETWORKS_FUSED_RELU",

"ANEURALNETWORKS_FUSED_RELU1",

"ANEURALNETWORKS_FUSED_RELU6",

}

comaniac · 2021-06-16T23:28:17Z

python/tvm/contrib/target/android_nnapi/relayir_to_nnapi_converter/converter.py

+
+    def convert(self, func):
+        """Converts a Relay IR Function into Android NNAPI C++ class source code
+        Parameters


add an empty line.

melsonlai · 2021-06-17T13:43:08Z

Hey Cody, thank you for taking time to review this. I really appreciate this.

For the suggestion of removing operator implementations in this PR, I would agree that this would make the PR smaller, but please do take note that the main body of this PR is already infrastructure, so removing operators would not help much.

The tests can be made to test for compilability only instead of text comparing, and I'll do this on the next commit to this branch.

Apart from these, it looks like there's a few coding style/document suggestions, so I'll keep working on the refactor.

This commit implements the basic structure of Android NNAPI codegen with the BYOC mechanism: * Basic graph partition using pattern-based rules * RPC-based graph partition * relay.ext.android_nnapi BYOC codegen

s.a. TVM PR #8076

s.a. PR #8076

for operator testing of Android NNAPI BYOC s.a. PR #8076

all of Android NNAPI BYOC Pythonic s.a. PR #8076

to "Compiler" s.a. PR #8076

s.a. PR #8076

s.a. PR #8076 #8088

costs between TVM and Android NNAPI in the RPC partitioner s.a. PR #8076

masahi · 2022-01-09T23:13:18Z

what's the status of this PR?

comaniac · 2022-01-09T23:36:24Z

Thanks for reminding. I'm closing this PR for now as @melsonlai is no longer working on this RFC. The guy that takes over his work will file a new one later.

FabianSchuetze · 2023-11-05T12:19:21Z

May I ask how this PR is going and with what it has been replaced, @comaniac ?

Is there still interested in implenting and NnApi backend for TVM? If so, what would the next steps have to be for it?

comaniac · 2023-11-05T17:36:41Z

Unfortunately the group of authors worked on this PR was no longer contributing due to their own reason, so AFAIK there's no plan for NNAPI upstreaming ATM. You are welcome to propose one.

melsonlai mentioned this pull request May 20, 2021

[BYOC][NNAPI]: Add testing package to ci_cpu image #8088

Merged

comaniac requested changes Jun 16, 2021

View reviewed changes

melsonlai requested review from MarisaKirisame, ZihengJiang, anijain2305, areusch, jroesch, junrushao, jwfromm, mbrookhart, merrymercy, slyubomirsky, tqchen, vinx13, wweic, yzhliu and zhiics as code owners July 30, 2021 12:56

melsonlai added 11 commits July 31, 2021 22:25

[BYOC][NNAPI]: Implement basic structure of Android NNAPI BYOC

624fa76

This commit implements the basic structure of Android NNAPI codegen with the BYOC mechanism: * Basic graph partition using pattern-based rules * RPC-based graph partition * relay.ext.android_nnapi BYOC codegen

[BYOC][NNAPI]: Refactor doc-strings

91095f7

s.a. TVM PR #8076

[BYOC][NNAPI]: Fix exporting constants to Android NNAPI

8de7b7d

s.a. PR #8076

[BYOC][NNAPI]: Add explanation to codegen options

ab3ce62

s.a. PR #8076

[BYOC][NNAPI]: Coding style fix as per the request of reviewer

eac6d6a

s.a. PR #8076

[BYOC][NNAPI]: Test for compilability instead of text-diff

65d903f

for operator testing of Android NNAPI BYOC s.a. PR #8076

[BYOC][NNAPI]: Remove C++ stub of BYOC compiler and make

56e86f5

all of Android NNAPI BYOC Pythonic s.a. PR #8076

[BYOC][NNAPI]: Rename the codegen component from "Converter"

47a971b

to "Compiler" s.a. PR #8076

[BYOC][NNAPI]: Rename Helper to JSONAnalyzer

6ed2f30

s.a. PR #8076

[BYOC][NNAPI]: Setup testing environment for Android NNAPI

5fb79bc

s.a. PR #8076 #8088

[BYOC][NNAPI]: Use linear function to estimate data movement

2c04669

costs between TVM and Android NNAPI in the RPC partitioner s.a. PR #8076

melsonlai requested a review from comaniac August 9, 2021 03:19

comaniac closed this Jan 9, 2022

rhjdvsgsgks mentioned this pull request Jun 2, 2023

[Feature Request] vulkan backend on android mlc-ai/mlc-llm#285

Open

-    Note
-    ----
+    Notes
+    -----

		from .converter import Converter


		def convert_relayir_to_nnapi(func):

	def is_FuseCode(self, idx): # pylint: disable=invalid-name
	def is_fuse_code(self, idx):

[BYOC][NNAPI]: Implement basic structure of Android NNAPI BYOC #8076

[BYOC][NNAPI]: Implement basic structure of Android NNAPI BYOC #8076

Uh oh!

Conversation

melsonlai commented May 19, 2021

Uh oh!

melsonlai commented May 19, 2021

Uh oh!

comaniac left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

melsonlai commented Jun 17, 2021

Uh oh!

masahi commented Jan 9, 2022

Uh oh!

comaniac commented Jan 9, 2022

Uh oh!

FabianSchuetze commented Nov 5, 2023

Uh oh!

comaniac commented Nov 5, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants