Add benchmark inference_tract #37
Conversation
|
Benchmark adds an AI subtest to Wasmscore through the additions of mobilenet inferencing with by onnx and tensorflow models. Results show Wasm performing at about 30% of native performance. Wasm performance for onnx is similar to what is seen in sightglass. Sightglass does not implement the tensorflow model inferencing. |
jlb6740
force-pushed
the
add-mobile-net-tract-inferencing
branch
from
9bc64cc to
6157300
Compare
| WORKDIR /usr/src/rust-benchmark | ||
| ENV CARGO_REGISTRIES_CRATES_IO_PROTOCOL=sparse | ||
| RUN (cd mobile_net_v2_onnx; cargo build --release --target wasm32-wasi) | ||
| RUN cp target/wasm32-wasi/release/*benchmark.wasm /benchmark/mobile_net_v2_onnx_benchmark.wasm |
There was a problem hiding this comment.
Where is the onnx model being brought in? I saw it in setup.sh which is called from build.rs but its not included here (similar to how tf frozen model is included in the section below)
There was a problem hiding this comment.
build.rs calls setup.sh
| # Compile mobile_net_v2_tensorflow | ||
| ADD mobile_net_v2_tensorflow rust-benchmark | ||
| WORKDIR /usr/src/rust-benchmark | ||
| ENV CARGO_REGISTRIES_CRATES_IO_PROTOCOL=sparse |
There was a problem hiding this comment.
NIT - repeated Env, first in line 9
build.sh
Outdated
| # Create docker image | ||
| echo "Building ${IMAGE_NAME}-${IMAGE_VERSION} for $ARCH." | ||
| echo "" | ||
| docker build -t ${IMAGE_NAME} --build-arg ARCH=$(uname -m) . | ||
| docker build --no-cache -t ${IMAGE_NAME} --build-arg ARCH=$(uname -m) . |
There was a problem hiding this comment.
To force rebuild every time when certain artifacts change but the dockerfile doesn't. I can remove this.
There was a problem hiding this comment.
Are these *expected files being used anywhere? I was going to comment that perhaps a range of probabilities would be acceptable instead of a precise one, since (maybe) hardware and other differences might change that slightly. But then I can't see this being used anywhere.
There was a problem hiding this comment.
Yes, good spot. By renaming this expected does not get checked. I did that because of slight variations in precision as you suggested. I planned a follow-up PR to do a check based on some supplied script that would do whatever check was needed. Note, the benchmark itself already does check for the the highest scoring match so a correctness check is taking place, it is just that this one would be more specific.
There was a problem hiding this comment.
I've made a few comments, mostly nothing major and pretty much looks good.
I didn't see the onnx model being included in dockerfile, thats the only real one, maybe i missed it.
Another thing - I can't test these changes on my local system right now due to docker being unavailable.
No description provided.