Speed up Travis build times

[dhermes]

I'm working on a strategy to use tox more selectively with Travis to speed up builds.

tox is great for local dev where the tox environ gets set up once and is re-used but setting up 6 environments (not including logs and dist) is a large cost.

$ ls .tox/
cover/  dist/  docs/  lint/  log/  py26/  py27/  regression/
$ ls .tox/log/
tox-0.log
$ ls .tox/dist/
gcloud-0.02.2.zip

[tseaver]

Hmm, the pylint bit is definitely the most time-consuming part, once we actually get scheduled on a CPU. Once we are running regression tests, they will likely run longer, too.

[dhermes]

@tseaver The tox installs take the most time (about a minute each, times six).

From there, run_pylint is the longest running script.

I'm focusing (for now) on reducing the install time.

Also,

From http://jsatt.com/blog/using-tox-with-travis-ci-to-test-django-apps/

Travis runs it's tasks based on the build matrix of environment variables
and language versions, allowing us to run each testenv asynchronously
and reporting the results for each version separately.
Using TOXENV potentially looks promising.

[tseaver]

Note that the compiling C in the upstream dependencies injects the majority of the time: if they were distributed as wheels (compatible with our platform, of course) then we could nuke most of that. I guess we could build the wheels ourselves and push them to GH pages (or somewhere) as a "private index".

[dhermes]

Interesting. This is compiling the crypto parts?

I stored some times in https://gist.github.com/dhermes/5088b5520bba841a3d0e and setup.py install takes up the bulk.

[tseaver]

Crypto and CFFI:

$ find .tox/py27/ -name "*.so"
.tox/py27/lib/python2.7/site-packages/cryptography/_Cryptography_cffi_36a40ff0x2bad1bae.so
.tox/py27/lib/python2.7/site-packages/cryptography/_Cryptography_cffi_8f86901cxc1767c5a.so
.tox/py27/lib/python2.7/site-packages/cryptography/_Cryptography_cffi_fc665d23x4f158fee.so
.tox/py27/lib/python2.7/site-packages/Crypto/Cipher/_DES.so
.tox/py27/lib/python2.7/site-packages/Crypto/Cipher/_AES.so
.tox/py27/lib/python2.7/site-packages/Crypto/Cipher/_ARC2.so
.tox/py27/lib/python2.7/site-packages/Crypto/Cipher/_DES3.so
.tox/py27/lib/python2.7/site-packages/Crypto/Cipher/_XOR.so
.tox/py27/lib/python2.7/site-packages/Crypto/Cipher/_CAST.so
.tox/py27/lib/python2.7/site-packages/Crypto/Cipher/_ARC4.so
.tox/py27/lib/python2.7/site-packages/Crypto/Cipher/_Blowfish.so
.tox/py27/lib/python2.7/site-packages/Crypto/Util/strxor.so
.tox/py27/lib/python2.7/site-packages/Crypto/Util/_counter.so
.tox/py27/lib/python2.7/site-packages/Crypto/Hash/_SHA512.so
.tox/py27/lib/python2.7/site-packages/Crypto/Hash/_MD4.so
.tox/py27/lib/python2.7/site-packages/Crypto/Hash/_RIPEMD160.so
.tox/py27/lib/python2.7/site-packages/Crypto/Hash/_SHA384.so
.tox/py27/lib/python2.7/site-packages/Crypto/Hash/_SHA256.so
.tox/py27/lib/python2.7/site-packages/Crypto/Hash/_SHA224.so
.tox/py27/lib/python2.7/site-packages/Crypto/Hash/_MD2.so
.tox/py27/lib/python2.7/site-packages/_cffi_backend.so

[dhermes]

"Confirmed" Travis can do builds in parallel:
http://www.dominicrodger.com/tox-and-travis.html
http://docs.travis-ci.com/user/speeding-up-the-build/

Looking into compiling the C stuff.

[dhermes]

I found a project which creates wheel files on Travis and then commit them to a separate branch. I adapated this project to our needs and used the generated wheels in a build.

I was able to cut BUILD times from 7:49 to 3:17 and then finally down to 2:09. (NOTE: This simplication dumps off python2.6 testing and running in parallel with reduced install times via custom wheels may be the way to go.)

[dhermes]

Using 100% pre-built wheels the whole install (our library and test deps) takes 10.92s. Running a secondary pip install --upgrade on all the dependencies (to make sure the pre-built wheels are not out-of-date) takes an additional 5.90s. (Also it takes 2.26s to clone the repo holding our pre-built wheels.)

See:
https://gist.github.com/dhermes/b618c1dcbe0d811865eb
https://travis-ci.org/dhermes/gcloud-python/builds/38970047

@tseaver WDYT of this strategy? To clarify, the strategy is:

• Build wheels on Travis, store in separate repo. These are static and only need to be updated on the order of once a week (but likely less).
• Use our pre-built wheels to pip install --no-index --find-links=LOCAL_WHEELS
• Run pip install --upgrade to make sure we are up-to-date (or get the new version). Since updating wheels is disjoint from the gcloud-python builds.

We can incorporate this strategy into tox and still use the parallel builds or ditch tox and manage dependencies ourselves (seems scary).

[tseaver]

I don't honestly think using Travis to build the wheels is much of a win (as opposed to doing it manually when we notice that a new version is out, and decide to adopt it).

We could choose to build a private index from the set of wheels after building them. See https://gist.github.com/tseaver/dc75c100b70323803bc3 for an example (it predates wheels, and so would need to be updated to scan them).

To re-create the index pages after adding a distribution / wheel (e.g., 'foo-1.2.3.tar.gz') into 'gcloud/production':

$ cd /path/to/indexrepo
$ git pull
$ python make_index.py
$ git add gcloud/production
$ git commit -m "Add foo 1.2.3."

[dhermes]

I am under the impression that using Travis to build the wheels is needed due to potential platform dependencies.

Also, with the setup I describe in
GoogleCloudPlatform/gcloud-python-wheels#1
it will allow new wheels to be built without any manual intervention on our part (which is a positive).

[silvolu]

+1 for no manual intervention.

[dhermes]

@tseaver #308 should wrap this up. Sorry to have ignored the custom index you showed in the gist. It certainly looked useful, but I had already gotten this method working. FWIW I was pleasantly surprised at how close to the default tox build process we were able to get with the wheelhouse solution.