[Apps] [RFC] Bundled interpreter demonstration

[ajtulloch]

This is an example/RFC for a usecase mentioned at the recent TVM conference - specifically, allowing single-file deployment of TVM models ab initio, without having the predeploy the TVM runtime. This is done by building a single "bundled" shared object, prebaking the graph/params into this shared object, and exposing the user-visible functions to be invoked via dlopen/dlsym.

I'm not sure how generally useful this is, but I thought I'd put it up in case folks have thoughts on better ways to achieve the main goal, etc.

(the README.md is embedded below):

How to Bundle TVM Modules

This folder contains an example on how to bundle a TVM module (with the required
interpreter runtime modules such as runtime::GraphRuntime, the graph JSON, and
the params) into a single, self-contained shared object (bundle.so) which
exposes a C API wrapping the appropriate runtime::GraphRuntime instance.

This is useful for cases where we'd like to avoid deploying the TVM runtime
components to the target host in advance - instead, we simply deploy the bundled
shared-object to the host, which embeds both the model and the runtime
components. The bundle should only depend on libc/libc++.

It also contains an example code to load this shared object and invoke the
packaged TVM model instance.

Type the following command to run the sample code under the current folder,
after building TVM first.

make demo

This will:

• Download the mobilenet0.25 model from the MXNet Gluon Model Zoo
• Compile the model with NNVM
• Build a bundle.so shared object containing the model specification and parameters
• Build a demo executable that dlopen's bundle.so, instantiates the
  contained graph runtime, and invokes the run function on a random input.

[tqchen]

Thanks, @ajtulloch ! please request reviews

[ajtulloch]

cc @yidawang (might be interesting for you folks, possibly along the lines of existing deployment models where you bundle model + interpreter for trees in https://github.com/dmlc/treelite), @tqchen (author of apps/howto_deploy), @merrymercy (could be an interesting deployment model for TVM on constrained mobile devices).

[ajtulloch]

@junrushao1994 this might be relevant to your comments in #2122 (comment) about:

The ability to get rid of a runtime, i.e. pure AOT. Having a heavy runtime is repeating the mistakes of previous generation of deep learning framework. What's more, any extra dependency means extra plain on edge devices. I think it could be great to just dump a binary which allows us to call a subset of computation (which contains no control flow, only dataflow), and let users themselves assemble these binarized when containing branches, loops etc.

[yzhliu]

suggest to print the usage, or describe it in README.

[ajtulloch]

Done, thanks for the suggestion.

[tqchen]

LTGM, I will let @yzhliu moderate this and merge if he thinks it is OK

[yidawang]

LGTM. Thanks for the work!

[yzhliu]

Thanks @ajtulloch @tqchen @yidawang This is now merged :)

[merrymercy]

Should we add test for apps?
We have encountered the cases that some modifications broke the apps. (#2073)

[ajtulloch]

@merrymercy absolutely - I’ll add a test in integration tests invoking this Makefile (ie ensuring “make test” runs and returns exit code 0) as a follow up to this diff?