Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MacOS travis-ci build stalls #122

Closed
josh146 opened this issue Jan 23, 2019 · 21 comments · Fixed by #129
Closed

MacOS travis-ci build stalls #122

josh146 opened this issue Jan 23, 2019 · 21 comments · Fixed by #129

Comments

@josh146
Copy link

josh146 commented Jan 23, 2019

I noticed today that our MacOS travis-CI build using cibuildwheel has started stalling at the following point of the cibuildwheel setup:

+ pip install --upgrade setuptools
Collecting setuptools
  Downloading https://files.pythonhosted.org/packages/37/06/754589caf971b0d2d48f151c2586f62902d93dc908e2fd9b9b9f6aa3c9dd/setuptools-40.6.3-py2.py3-none-any.whl (573kB)
Installing collected packages: setuptools
  Found existing installation: setuptools 28.8.0
    Uninstalling setuptools-28.8.0:

No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received
The build has been terminated

This hasn't affected our Windows/Linux builds, and no changes to our devops pipeline has occurred (and even then, only superficial Python commits in our codebase were committed).

This issue happens no matter how many times we restart the build, and seems odd - this step is usually instantaneous on previous Mac cibuildwheel builds.

Since this is a command that is called by cibuildwheel, has there been a recent change that breaks this step?

@YannickJadoul
Copy link
Member

I can reproduce this on my build as well: https://travis-ci.org/YannickJadoul/Parselmouth/jobs/483834720, but I have no clue what's going on.
Let's run a branch with verbosity flags for this command? Unless anyone has better suggestions?

@YannickJadoul
Copy link
Member

Weird, Travis builds of cibuildwheel fail in the exact same way, and -vvv does not provide any more insight: https://travis-ci.org/YannickJadoul/cibuildwheel/jobs/483882146#L3949

@YannickJadoul
Copy link
Member

The problem seems to be with Python 3.4, though, because the Python 2.7 does work. Is this the same for you, @josh146?

@YannickJadoul
Copy link
Member

Installing pip<19 seems to work, so I'm guessing it's a problem with pip and Python 3.4: YannickJadoul@f876231 and https://travis-ci.org/YannickJadoul/cibuildwheel/builds/483890015

Not sure what we can do about it though. This fix seems very ad-hoc and ugly?

This seems to be the same/a similar issue: pypa/pip#6169

@bmatthieu3
Copy link

It stalls too with Python3.5 https://travis-ci.org/cds-astro/cds-healpix-python/jobs/483904177 . Seems to be a problem with pip 19.0.1

@josh146
Copy link
Author

josh146 commented Jan 24, 2019

I'm not using either Python 2.7 or Python 3.4, but I see the issue on Python 3.5

@YannickJadoul
Copy link
Member

YannickJadoul commented Jan 24, 2019

Ah, weird that it did work with 2.7 in those builds, then; but thanks for the confirmation.
If you need a quick patch, YannickJadoul@f876231 will fix it on macOS (though the title of the commit is not correct anymore, after amending the commit); but I assume a fix will be released soon enough by pip?

EDIT: Although, maybe it is an interesting option to provide a release with this small addtion, since all builds using cibuildwheel are now failing. What do you think, @joerick?

@josh146
Copy link
Author

josh146 commented Jan 26, 2019

From further reading, it seems like this bug occurs when upgrading a package that was previously installed via sudo, but is attempting to be upgraded without sudo. Previous versions of pip would silently fail. Maybe this is why it only affects the MacOS builds?

@joerick
Copy link
Contributor

joerick commented Jan 26, 2019 via email

@matkoniecz
Copy link

matkoniecz commented Jan 27, 2019

I noticed the same problem (pip freezing on upgrading without sudo) on Ubuntu PC (16.04), so it seems that missing sudo is the problem.

@YannickJadoul
Copy link
Member

@matkoniecz I'm still failing to reproduce the problem manually; I'm only seeing it on the online build :-(

Is there anything in particular you do to trigger the problem on Ubuntu? Is pip installed with/without sudo, locally/globally? Can I do it for any package?

I've just tried installing cibuildwheel<0.10 with sudo, then updating to the latest cibuildwheel, but that does not seem to trigger it on my Ubuntu 18.04 machine.

(Just wondering, because the issue on the pip repo does not seem to advance a lot. So I'm wondering if I could debug/help reporting more details there.)

@matkoniecz
Copy link

I get stall with pip install --upgrade pillow later sudo pip install --upgrade pillow worked fine.

I will try whatever it is reproducible.

@YannickJadoul
Copy link
Member

Thanks for that! When I do this, I'm getting nice errors, though:

~$ pip3 install --upgrade pillow
Collecting pillow
  Using cached https://files.pythonhosted.org/packages/85/5e/e91792f198bbc5a0d7d3055ad552bc4062942d27eaf75c3e2783cf64eae5/Pillow-5.4.1-cp36-cp36m-manylinux1_x86_64.whl
Installing collected packages: pillow
  Found existing installation: Pillow 5.3.0
    Uninstalling Pillow-5.3.0:
Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: 'ImageDraw.py'
Consider using the `--user` option or check the permissions.

YannickJadoul added a commit to YannickJadoul/cibuildwheel that referenced this issue Jan 27, 2019
@YannickJadoul
Copy link
Member

I believe I've figured out the problem, indeed something with permissions: see pypa/pip#6169

I think this means we should just use sudo, in our case, but I'll have a closer look tomorrow.

Meanwhile, I'd suggest using pip install git+https://github.com/YannickJadoul/cibuildwheel.git@pip-19-stalling-workaround (to install this branch using pip 18: https://github.com/YannickJadoul/cibuildwheel/tree/debug-issue-122)

@joerick Enjoy your holiday! I don't know if I have a strong opinion about pinning them. In general I'd say we haven't had a lot of problems with these packages (and if we do, the rest of the world also has them), and maybe we'd like to be in on new pip features without extra work/releases on cibuildwheel?

@matkoniecz
Copy link

@YannickJadoul Thanks for checking this! Let me know if further testing would be useful.

@josh146
Copy link
Author

josh146 commented Jan 27, 2019

@YannickJadoul amazing! I had stumbled on another GitHub issue that mentioned it was something to do with permissions, but for the life of me couldn't find it again - hence why my second comment was so vague.

@YannickJadoul
Copy link
Member

@josh146 Actually, your comment was the start of my debugging. Otherwise I'd have had absolutely no clue where to start. So thanks for that :-)
And in the end, it seems that adding sudo will indeed be the solution, so you were spot on.

@YannickJadoul
Copy link
Member

@matkoniecz Thanks for that! I'll check this hypotheses tomorrow (this should be testable by changing permissions to the site-packages folder) and I'll let you know if I don't manage to reproduce the issue.

@YannickJadoul
Copy link
Member

Update: a solution is on its way: pypa/pip#6215
I think I will wait to see the exact outcome of that to see what we should do about the issue.

Adding sudo or --user to pip install invocations will work, but since it worked without those before, we might want to see if it will work like that after the pip fix?

YannickJadoul added a commit to YannickJadoul/cibuildwheel that referenced this issue Jan 30, 2019
YannickJadoul added a commit to YannickJadoul/cibuildwheel that referenced this issue Jan 30, 2019
joerick added a commit that referenced this issue Feb 3, 2019
Running 'pip install --upgrade setuptools' with 'sudo' on macOS, fixing #122
@joerick
Copy link
Contributor

joerick commented Feb 3, 2019

Temporary fix in #123, releasing now...

@joerick
Copy link
Contributor

joerick commented Feb 3, 2019

Released as 0.10.1

indygreg added a commit to indygreg/python-zstandard that referenced this issue Feb 6, 2019
This fixes an issue with CI stalling when upgrading setuptools
(pypa/cibuildwheel#122).
mayeut added a commit to mayeut/cibuildwheel that referenced this issue Feb 9, 2019
…OS, fixing pypa#122"

This reverts commit d79ca07.
pip has been fixed in 19.0.2
kutsurak added a commit to kutsurak/MonetDBLite-Python that referenced this issue Feb 13, 2019
kutsurak added a commit to kutsurak/MonetDBLite-Python that referenced this issue Feb 15, 2019
YannickJadoul added a commit to YannickJadoul/cibuildwheel that referenced this issue Mar 10, 2019
YannickJadoul added a commit to YannickJadoul/cibuildwheel that referenced this issue Mar 10, 2019
joerick pushed a commit that referenced this issue Mar 10, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants