Initial framework of an ethos-u runtime backend #2 by robell · Pull Request #595 · pytorch/executorch

robell · 2023-10-03T22:13:14Z

Replaces PR #501

(will close 501 shortly once comments are reviewed/addressed)

Added shell of runtime Arm Backend for Ethos-U

./backends/arm/cmake/build.sh will pull the appropriate compiler and build the ethos-u driver and delegate and produce the libraries needed to link an application in, which can run a delegated .pte with the revised vela_bin_stream form contained in these patches.

Currently supports .pte files stored in SRAM.

Signed-off-by: Rob Elliott <robert.elliott@arm.com>

- Basic runtime targeting Corstone-300 with U55 - cross compile support with a cmake toolchain (Arm baremetal build) - support for a few models AoT -> TOSA -> Vela -> U55 hardware - dependencies on the ethos-u core driver and cmsis (submodules) Signed-off-by: Rob Elliott <robert.elliott@arm.com>

netlify · 2023-10-03T22:13:19Z

✅ Deploy Preview for resplendent-gnome-14e531 canceled.

Name	Link
🔨 Latest commit	`76a393a`
🔍 Latest deploy log	https://app.netlify.com/sites/resplendent-gnome-14e531/deploys/651faf6cda798d0008a44588

backends/arm/runtime/ArmBackendEthosU.cpp

Signed-off-by: Rob Elliott <robert.elliott@arm.com>

Summary: Sometimes, esp when using the Portable library, Release mode is the only viable option to get a resonable binary size and performance, even to debug stuff. Differential Revision: D49876324 fbshipit-source-id: 501d96304b8e37452a1a58e289d57cfc62ebbcdd

Summary: This will align better with structure proposed in D49714823 Differential Revision: D49899370 fbshipit-source-id: 86d89c8060a6685f84dc186ea38beba4388894db

Summary: This is needed to generate `portable_ops_lib` with select number of ops for quick testing when we can't afford to link all the ops for size reasons. Arm Baremetal builds is one example. Differential Revision: D49899554 fbshipit-source-id: 9d35e7bfd1c1975da640e62493fc744898ad6714

Summary: This is an interesting usecase to demonstrate on ARM Baremetal setup where they currently can't lower SoftMax. One of the only few from MV2. It is useful as a test as well while debugging this flow. Differential Revision: D49899589 fbshipit-source-id: c0307c7452a69eb707429328ba6654b2b660ce23

Differential Revision: https://internalfb.com/D49899371 fbshipit-source-id: 42cf5c8ad445b0405f9c23e3654b6a32fbc61522

Summary: An example script to build and run executor_runner baremetal version on ARM M-class CPUs ``` I executorch:runner.cpp:63] Model PTE file loaded. Size: 960 bytes. I executorch:runner.cpp:69] Model buffer loaded, has 1 methods I executorch:runner.cpp:77] Running method forward I executorch:runner.cpp:94] Setting up planned buffer 0, size 32. I executorch:runner.cpp:109] Method loaded. I executorch:runner.cpp:111] Preparing inputs... I executorch:runner.cpp:113] Input prepared. I executorch:runner.cpp:115] Starting the model execution... I executorch:runner.cpp:120] Model executed successfully. I executorch:runner.cpp:124] 1 outputs: Output[0][0]: 0.500000 Output[0][1]: 0.500000 Output[0][2]: 0.500000 Output[0][3]: 0.500000 ``` Differential Revision: D49900956 fbshipit-source-id: e30571f95abd5311f3ab649fcb1d154b4f5c3310

Signed-off-by: Rob Elliott <robert.elliott@arm.com>

* There is a toolchain/linking issue mixing hard and soft float ABI which is causing the test to fail, but the structure is there and the delegate is registered. Signed-off-by: Rob Elliott <robert.elliott@arm.com>

digantdesai · 2023-10-04T15:36:00Z

.gitmodules

+[submodule "backends/arm/third-party/ethos-u-core-driver"]
+	path = backends/arm/third-party/ethos-u-core-driver
+	url = https://git.mlplatform.org/ml/ethos-u/ethos-u-core-driver.git
+[submodule "backends/arm/third-party/cmsis"]


What is this used for?

These are essentially the driver headers for ethos-u we need to link against.

I've also added a build of the ethos-u driver in executorch cmake (ethosu_core_driver target) but I moved to link against the demo app library build of it and can remove the target.

We still need this small submodule for the header/interface - it's fortunately small, but unfortunately it pulls in CMSIS for a few platform defines which is huge - this is really the area we discussed a build script pulling these in. Given ./run has a checkout of the core_platform and core_software, I'll look at moving to those and removing these submodules.

I've removed most of this, we just have the small ethos-u-core-driver for headers now, and don't build it in tree and cmsis is removed. The driver builds in the driver/app repo and that app links the two together.

as it's only 25 files i suggest we keep the driver as a submodule to keep things simple until there's a more general "selective submodule" approach.

Signed-off-by: Rob Elliott <robert.elliott@arm.com>

digantdesai · 2023-10-04T18:50:16Z

backends/arm/cmake/arm-none-eabi-gcc.cmake

@@ -0,0 +1,90 @@
+# Copyright 2023 Arm Limited and/or its affiliates.


Is this same as - core_platform/cmake/toolchain/arm-none-eabi-gcc.cmake? But will respect cmdline vars?

I'd anticipate further changes and cleaning but it was derived from there, yes.

It has a similar override but for m55 our only current supported target. The way the toolchain file is constructed, some of the general ./configure like step in cmake was invoking it without an appropriate value so the default prevents the -mfpu flag which was the default set on an m4 target messing things up.

* Override the default CPU in cmake which was causing a mixture of FPU and ABI flags to be passed to different compilation stages. * Updated fallback logging implementation in delegate app to fix sporadic crash Signed-off-by: Rob Elliott <robert.elliott@arm.com>

* uses the simple_add model to run through the AoT flow and the various debug options for looking at export. * produces a .pte file for runtime delegation on the ArmBackend for Ethos-U55 platforms Signed-off-by: Rob Elliott <robert.elliott@arm.com>

Signed-off-by: Rob Elliott <robert.elliott@arm.com>

* export the list of offset from AoT floq * appropriately copy inputs and ouputs to/from SRAM Signed-off-by: Rob Elliott <robert.elliott@arm.com>

Signed-off-by: Rob Elliott <robert.elliott@arm.com>

robell · 2023-10-06T13:26:26Z

#659 replaces this

Summary: Cleaned up ethos-u runtime flow, replaces #595 Added shell of runtime Arm Backend for Ethos-U ./backends/arm/cmake/build.sh will pull the appropriate compiler and build the ethos-u driver and delegate and produce the libraries needed to link an application in, which can run a delegated .pte with the revised vela_bin_stream form contained in these patches. Currently supports .pte files stored in SRAM. Pull Request resolved: #659 Reviewed By: cccclai Differential Revision: D50035452 Pulled By: digantdesai fbshipit-source-id: 7b8a23836c92dfa9235decf4c631654fa8fc396a

robell added 2 commits October 3, 2023 21:43

[NOMERGE] stub out tests with codegen issues temporarily

521124b

Signed-off-by: Rob Elliott <robert.elliott@arm.com>

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Oct 3, 2023

larryliu0820 reviewed Oct 3, 2023

View reviewed changes

backends/arm/runtime/ArmBackendEthosU.cpp Outdated Show resolved Hide resolved

robell and others added 10 commits October 4, 2023 08:35

Fixed error messages on runtime init

2083a71

Signed-off-by: Rob Elliott <robert.elliott@arm.com>

lintrunner cleanup

f5cff0b

Signed-off-by: Rob Elliott <robert.elliott@arm.com>

Move arm example dir under backends

918c798

Summary: This will align better with structure proposed in D49714823 Differential Revision: D49899370 fbshipit-source-id: 86d89c8060a6685f84dc186ea38beba4388894db

patches and utils for setting up baremetal stack for cs300

69c132d

Differential Revision: https://internalfb.com/D49899371 fbshipit-source-id: 42cf5c8ad445b0405f9c23e3654b6a32fbc61522

Add ArmBackend to example scripts

afd8080

Signed-off-by: Rob Elliott <robert.elliott@arm.com>

Add delegate test and FVP output

a29523f

* There is a toolchain/linking issue mixing hard and soft float ABI which is causing the test to fail, but the structure is there and the delegate is registered. Signed-off-by: Rob Elliott <robert.elliott@arm.com>

digantdesai reviewed Oct 4, 2023

View reviewed changes

robell added 2 commits October 4, 2023 16:05

align builds on same cmake toolchain

8efd2e3

Signed-off-by: Rob Elliott <robert.elliott@arm.com>

Fix delegate runner patch

52c1c09

digantdesai reviewed Oct 4, 2023

View reviewed changes

robell added 11 commits October 4, 2023 20:09

Generate pte for delegate test on the fly

ec03f3c

Signed-off-by: Rob Elliott <robert.elliott@arm.com>

Added support for variable input output patterns

1ba71d7

Signed-off-by: Rob Elliott <robert.elliott@arm.com>

Handle multiple delegate inputs with SRAM offsets

f29715e

* export the list of offset from AoT floq * appropriately copy inputs and ouputs to/from SRAM Signed-off-by: Rob Elliott <robert.elliott@arm.com>

Add TOSA ref model and Vela dependencies

3b35ff6

Signed-off-by: Rob Elliott <robert.elliott@arm.com>

Cleanup from lintrunner and other bits of tidyup

94c598e

Signed-off-by: Rob Elliott <robert.elliott@arm.com>

Removed ethos u driver build and cmsis dependency

683d428

Signed-off-by: Rob Elliott <robert.elliott@arm.com>

renamed lib ethos_u to executorch_delegate_ethos_u

3292199

Signed-off-by: Rob Elliott <robert.elliott@arm.com>

lintfix

e6ede01

Signed-off-by: Rob Elliott <robert.elliott@arm.com>

tidied delegate_runner output

76a393a

Signed-off-by: Rob Elliott <robert.elliott@arm.com>

This was referenced Oct 6, 2023

Initial framework of an ethos-u runtime backend #501

Closed

Initial framework of an ethos-u runtime backend #3 #659

Closed

robell closed this Oct 9, 2023

robell added the partner: arm For backend delegation, kernels, demo, etc. from the 3rd-party partner, Arm label Jul 4, 2024

Gasoonjia pushed a commit that referenced this pull request Jul 30, 2024

add fast fast16 tests (#595)

aa00167

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial framework of an ethos-u runtime backend #2#595

Initial framework of an ethos-u runtime backend #2#595
robell wants to merge 25 commits intopytorch:mainfrom
robell:fresh_rebase

robell commented Oct 3, 2023 •

edited

Loading

Uh oh!

netlify bot commented Oct 3, 2023 •

edited

Loading

Uh oh!

Uh oh!

digantdesai Oct 4, 2023

Uh oh!

robell Oct 4, 2023

Uh oh!

robell Oct 5, 2023 •

edited

Loading

Uh oh!

digantdesai Oct 4, 2023 •

edited

Loading

Uh oh!

robell Oct 4, 2023

Uh oh!

robell commented Oct 6, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		@@ -0,0 +1,90 @@
		# Copyright 2023 Arm Limited and/or its affiliates.

Conversation

robell commented Oct 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify bot commented Oct 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for resplendent-gnome-14e531 canceled.

Uh oh!

Uh oh!

digantdesai Oct 4, 2023

Choose a reason for hiding this comment

Uh oh!

robell Oct 4, 2023

Choose a reason for hiding this comment

Uh oh!

robell Oct 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

digantdesai Oct 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

robell Oct 4, 2023

Choose a reason for hiding this comment

Uh oh!

robell commented Oct 6, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

robell commented Oct 3, 2023 •

edited

Loading

netlify bot commented Oct 3, 2023 •

edited

Loading

robell Oct 5, 2023 •

edited

Loading

digantdesai Oct 4, 2023 •

edited

Loading