Add dockerfiles for the conda package builds #3344
Conversation
|
Don't merge this, I discovered some problems in the resulting packages. I'll fix those problems and add some automated testing of the resulting packages. |
abergeron
force-pushed
the
cuda_docker
branch
2 times, most recently
from
f43cc4a to
00f7c66
Compare
|
Ok now it seems to be good. There are some major changes that I'll explain after. The main point is that this introduces new Dockerfiles for the build and those have to be cached. I think I can't do that part myself. Once that is done, I will update the Jenkinsfile so that the packages are actually built as part of the tests so we know they are good and don't keep breaking :) As for the changes, the tvm, topi and nnvm python packages are now a single package. This is because the there are some parts of tvm that depend on topi. It doesn't change the way the source is organized, only that there is a single conda package for all of them. I've bumped the conda version to 0.6.0dev1 since I want to make sure that there will not be old topi/nnvm packages left behind to conflict with the new ones. This will fix itself when 0.7.0 comes around. This also enables support for Metal in the OS X builds. |
conda/build_cpu.sh
Outdated
| @@ -0,0 +1,5 @@ | |||
| #!/bin/sh | |||
| conda build tvm-libs | |||
There was a problem hiding this comment.
For robustness it is good practice to include at least:
set -e
set -u
at the top of every .sh script. This will ensure that if any command in the script fails, the error is propagated rather than elided away, it will also ensure that scripts using undefined variables fail.
There was a problem hiding this comment.
Ok, I'll do that.
|
Can I get a review of this? It is done now. @tqchen |
|
THanks, @abergeron , this PR is now merged |
* First shot * Add dockerfile for CPU too * Finish the build infrastructure * Remove extra file * Comment out the Jenkinsfile section since it is not ready * Add missing license headers * Update to newer cudnn that anaconda packaged * Bump the build numbers for the newer cudnn * Bring back the toolchain option with a tweak for cuda * Cache some large packages in the docker and update to llvm 7.0.0 * Merge all the python packages together * First fix for the conda cuda builds (again) * Use the tarball version of cudnn since tvm has trouble detecting the other one * Use llvm 8.0 from the numba packages * Also use llvm 8.0 for the cpu builds * Don't use the anaconda compiler for OS X * Enable Metal on OS X builds * Make sure to detect undefined variables in scripts * Fix build when not using cuda
* First shot * Add dockerfile for CPU too * Finish the build infrastructure * Remove extra file * Comment out the Jenkinsfile section since it is not ready * Add missing license headers * Update to newer cudnn that anaconda packaged * Bump the build numbers for the newer cudnn * Bring back the toolchain option with a tweak for cuda * Cache some large packages in the docker and update to llvm 7.0.0 * Merge all the python packages together * First fix for the conda cuda builds (again) * Use the tarball version of cudnn since tvm has trouble detecting the other one * Use llvm 8.0 from the numba packages * Also use llvm 8.0 for the cpu builds * Don't use the anaconda compiler for OS X * Enable Metal on OS X builds * Make sure to detect undefined variables in scripts * Fix build when not using cuda
* First shot * Add dockerfile for CPU too * Finish the build infrastructure * Remove extra file * Comment out the Jenkinsfile section since it is not ready * Add missing license headers * Update to newer cudnn that anaconda packaged * Bump the build numbers for the newer cudnn * Bring back the toolchain option with a tweak for cuda * Cache some large packages in the docker and update to llvm 7.0.0 * Merge all the python packages together * First fix for the conda cuda builds (again) * Use the tarball version of cudnn since tvm has trouble detecting the other one * Use llvm 8.0 from the numba packages * Also use llvm 8.0 for the cpu builds * Don't use the anaconda compiler for OS X * Enable Metal on OS X builds * Make sure to detect undefined variables in scripts * Fix build when not using cuda
I'm not 100% sure what the procedure to add new docker images is. I figure I need to get the images added to docker hub under tvmai and then use the tags for those images in the Jenkinsfile.
I've tested the commands to ensure that building the docker images and running the builds in them works, but I can't upload the built images to the tvmai docker hub myself.
@tqchen