Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix python test failure using virtualenv<20.0.0 #641

Conversation

lucagiovagnoli
Copy link
Member

@lucagiovagnoli lucagiovagnoli commented Feb 13, 2020

Fixes #640 and #642 and improves tests running time from ~24min down to ~15min.

  1. Fixes recent virtualenv==20.0.0 bug that broke Travis builds 3 days ago (ImportError: cannot import name 'ensure_text' pypa/virtualenv#1551).
  2. Runs all python tests with all virtualenvs (for real! We were not running all of py27, py36, py37).
  3. Python 2.7 is officially deprecated. Let me know if you'd rather remove this
  4. Runs tests using Travis stages, so that we can run them in parallel and that we can actually use the correct system python version == tox python version. The problem seems to be that $TRAVIS_PYTHON_VERSION is unset (see Python tests are not run on all versions py27, py36, py37 #642). It looks like other projects on Github are using Travis stages to install the correct python version in each VM that is compatible to a specific tox environment.

One thing to note, unfortunately, due to some python tests relying on these jars: https://github.com/combust/mleap/blob/master/mleap-spark-extension/build.sbt#L9
I am forced to run sbt +compile before python tests in all stages (running in different VMs) and this makes them all slower because they are all compiling the entire project.

@talalryz @lydian @voganrc for review

@lucagiovagnoli lucagiovagnoli force-pushed the luca/fix-virtualenv-python-tests-bug-640 branch 8 times, most recently from 26c028d to b4c9d0e Compare February 13, 2020 03:07
@lucagiovagnoli
Copy link
Member Author

lucagiovagnoli commented Feb 13, 2020

scala tests and py36 passes.
sbt +compile fails in py37 tests. I wonder if that's because I changed the linux distro to xenial (as py37 is only available for xenial: https://travis-ci.community/t/unable-to-download-python-3-7-archive-on-travis-ci/639/2).

I could also try py35 and py38. We're not running on py37 right now anyways.

EDIT: this was because xenial is using java11 by default. Setting it to java8 fixes the problem

@lucagiovagnoli lucagiovagnoli force-pushed the luca/fix-virtualenv-python-tests-bug-640 branch 10 times, most recently from 7d894cf to db2ea14 Compare February 13, 2020 20:25
…ing run with the correct python version - pin virtualenv to be <20.0.0 because of pypa/virtualenv#1551 - use different distro for py37 tests
@lucagiovagnoli lucagiovagnoli force-pushed the luca/fix-virtualenv-python-tests-bug-640 branch from db2ea14 to e977ada Compare February 13, 2020 20:30
Copy link
Contributor

@talalryz talalryz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!!

Thank you for fixing these issues!

@ancasarb ancasarb merged commit 0e83927 into combust:master Feb 19, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Python tests broken by virtualenv upgrade to 20.0.0 on Travis
3 participants