Arm(R) Cortex(R)-M55 CPU and Arm(R) Ethos(TM)-U55 NPU Demo App

[grant-arm]

This PR adds a demo application that uses TVM to run a model on bare metal Arm® Cortex®-M55 CPU and Arm® Ethos™-U55 NPU

This PR is blocked on merging #8795 , #8806, #8811, #8849 and #8854. Since this is built on top of all those PR branches, this contains the accumulation of contents of the all the PRs.

[grant-arm]

PRs #8795 , #8806, #8811, #8849 and #8854 are merged now, so I've updated this PR and set it ready for review.

@areusch @manupa-arm @leandron @Mousius

[manupak]

@Mousius @leandron , could you take a look ?

[leandron]

LGTM

[leandron]

PRs #8795 , #8806, #8811, #8849 and #8854 are merged now, so I've updated this PR and set it ready for review.

@areusch @manupa-arm @leandron @Mousius

I’m happy to merge this, just wanted to leave it here for a few more hours to leave the opportunity for other to have a look.

[Mousius]

Apologies for the late review, I've been through and have some change suggestions, otherwise this is a great demonstration of the power of TVM in embedded 😸

[areusch]

neat! :) would be interested to discuss about perhaps creating apps/microtvm/ethosu e.g. project API impl. also have one question about the Python deps.

[grant-arm]

@Mousius Thanks for all the great comments & suggestions.
I believe I've addressed them all now.

[grant-arm]

@areusch Thanks for the comments.
I've addressed the python dependencies question inline. Hopefully that helps.
Could you please elaborate on how we would use the Project API with the demo?

[manupak]

Lets keep this demo for a user seeking to integrate inference to an already existing project rather than generating a new one. However, that does not preclude us creating another demo using project API for other usecases.

[manupak]

@Mousius PTAL

[Mousius]

@grant-arm I just have one set of suggestion 😸 thanks for working through this with me.

[grant-arm]

Thanks for the suggestions @Mousius . Could you please take a look again.

[Mousius]

Minor nit but I don't think it effects the overall demo so more than happy to see this landed.

[Mousius]

I don't think you need +es here (see: https://github.com/apache/tvm/blob/main/python/tvm/micro/interface_api.py#L59-L66)

Suggested change

	header_file.write(
	"#include <tvmgen_default.h>\n"
	+ f"const size_t {tensor_name}_len = {tensor_data.size};\n"
	+ f'uint8_t {tensor_name}[] __attribute__((section("{section}"), aligned(16))) = "'
	header_file.write(
	"#include <tvmgen_default.h>\n"
	f"const size_t {tensor_name}_len = {tensor_data.size};\n"
	f'uint8_t {tensor_name}[] __attribute__((section("{section}"), aligned(16))) = "'

[manupak]

LGTM

[manupak]

Thanks @grant-arm @Mousius @leandron @areusch . This is merged now.

[areusch]

@grant-arm just to follow-up on my comment, to use the Project API we essentially need to write a microtvm_api_server.py which can use this tree as a template in combination with a Model Library Format file to generate a runnable demo app like this. In general we would mainly use this with the full RPC runtime to trial models, so I think it's a bit of a different effort from this demo app.

[JKANG94]

How to run a model only on bare metal Arm® Cortex®-M55 CPU not Ethos(TM)-U55 NPU ?

[grant-arm]

@JKANG94 Currently we compile for offloading Conv2D & Depthwise Conv to Ethos(TM)-U55 NPU

# Compile model for Arm(R) Cortex(R)-M55 CPU and Ethos(TM)-U55 NPU
tvmc compile --target="ethos-u -accelerator_config=ethos-u55-256, \
    c -runtime=c --link-params -mcpu=cortex-m55 --executor=aot --interface-api=c --unpacked-api=1" \
    --pass-config tir.disable_vectorize=1 ./mobilenet_v1_1.0_224_quant.tflite --output-format=mlf

In order to compile for Arm(R) Cortex(R)-M55 CPU only, we would use:

# Compile model for Arm(R) Cortex(R)-M55 CPU
tvmc compile --target="c -runtime=c --link-params -mcpu=cortex-m55 --executor=aot --interface-api=c --unpacked-api=1" \
    --pass-config tir.disable_vectorize=1 ./mobilenet_v1_1.0_224_quant.tflite --output-format=mlf

Please note though that if you try this with the current demo and MobileNet v1, when you come to building the project, the linker will complain that the ITCM memory region has overflowed. This is due to the generated code not placing params in any specific section and the linker defaulting to placing them in ITCM. We have WIP to address this. (Please refer to the USMP RFC for pinning constant buffers on different memories)

[JKANG94]

I just ran "run_demo.sh", but there was a error " File "/workcode/tvm/tvm/python/tvm/relay/build_module.py", line 371, in build
assert False, "Executor " + executor + " not supported"
AssertionError: Executor aot=True not supported".

[grant-arm]

@JKANG94 thanks for pointing this out. #9218 changes the command line arguments required by tvmc which breaks the existing Arm(R) Cortex(R)-M55 CPU and Arm(R) Ethos(TM)-U55 NPU Demo App. #9323 fixes the demo app. In addition, we plan to enable running the demo app on CI so that we can catch these types of problems in future.

[JKANG94]

@grant-arm Using tvmc, I got operator implementations produced by tvm in codegen folder. I want to deploy these to cortext-m devices standalonely, how to get tvm c runtime static library and tvm c GraphExecutor static library?

[grant-arm]

Hi @JKANG94 , probably the most useful reference is the Makefile in tvm/apps/microtvm/ethosu/.

Since we compile with -executor=aot we are using the AOT Executor and it's not necessary to include the Graph Executor.
The only bits of TVM we need for the runtime are:
$(STANDALONE_CRT_PATH)/src/runtime/crt/memory/stack_allocator.c
$(STANDALONE_CRT_PATH)/src/runtime/crt/common/crt_backend_api.c

[JKANG94]

@grant-arm

1. Why you rewrite the functions "TVMPlatformMemoryAllocate" and "TVMPlatformMemoryFree" in tvm_runtime.h?
2. How to determine the value of WORKSPACE_SIZE?

[grant-arm]

@grant-arm

1. Why you rewrite the functions "TVMPlatformMemoryAllocate" and "TVMPlatformMemoryFree" in tvm_runtime.h?
2. How to determine the value of WORKSPACE_SIZE?

@JKANG94

1. We need to provide the platform implementation that uses the stack allocator. In your particular use-case, if malloc is permitted and desired, then you could use that instead.
2. You can get the value for WORKSPACE_SIZE from metadata.json which is included in the Model Library Format file produced by tvmc. Please see Model Library Format for details . workspace_size_bytes for the main function is the "Number of bytes needed in this workspace by this function and all sub-functions invoked"

Hope that helps.