PyYAML pip import fails with python3

[hwright]

Using this repository:

$ bazel run :hello
INFO: Found 1 target...
Target //:hello up-to-date:
  bazel-bin/hello
INFO: Elapsed time: 1.910s, Critical Path: 0.03s

INFO: Running command line: bazel-bin/hello
Hello
$

But forcing python 3 gives an error:

$ bazel run --python_path=/usr/local/bin/python3 :hello 
INFO: Found 1 target...
Target //:hello up-to-date:
  bazel-bin/hello
INFO: Elapsed time: 0.174s, Critical Path: 0.01s

INFO: Running command line: bazel-bin/hello
Traceback (most recent call last):
  File "/private/var/tmp/_bazel_hwright/0c1dcd494edf6933a7c645d083d11d41/execroot/foo/bazel-out/darwin_x86_64-fastbuild/bin/hello.runfiles/foo/hello.py", line 1, in <module>
    import yaml
  File "/private/var/tmp/_bazel_hwright/0c1dcd494edf6933a7c645d083d11d41/execroot/foo/bazel-out/darwin_x86_64-fastbuild/bin/hello.runfiles/pypi__PyYAML_3_12/yaml/__init__.py", line 2, in <module>
    from error import *
ModuleNotFoundError: No module named 'error'
ERROR: Non-zero return code '1' from command: Process exited with status 1.
$

This doesn't happen with all packages pulled in through pip, just PyYAML. I believe it might be related to the way that PyYAML handles py2/3 compat: instead of enabling the same code to run under both environments, PyYAML has a separate tree for each version internal to the package, and presumably selectively installs one or the other depending on the request. It doesn't look like Bazel is picking the right tree when running under python3.

[hwright]

It looks like PyYAML selects which tree to install based upon the version of Python that is currently running: https://github.com/yaml/pyyaml/blob/master/setup.py#L287 . Is is possible that Bazel isn't running setup.py under the same version of python specified in --python_path?

[duggelz]

Yes, that's exactly right. Unfortunately this will be hard to fix, since Bazel is in a "transition" state right now regarding how it handles multiple Python interpreters, but we can take a look to see if there is a workaround.

[duggelz]

One workaround that immediately comes to mind is:

1. Use ruamel instead of PyYaml, it's a drop-in replacement (PyYaml was abandoned, ruamel was forked, then PyYaml came back to life, not sure of the current status).

[duggelz]

Ignore my previous workaround # 1, reading comprehension fail.

We have this same issue internally in Blaze, but the specifics and the fix are different.

[hwright]

@duggelz It sounds like there's a dependency on some Bazel work, then. Is there a Bazel issue one could follow for that work? Any ETA, or is it just "this should get done sometime, we don't know when"?

[hwright]

@duggelz I've worked around my local issue by just using ruamel (thanks for the pointer!), so this isn't a pressing problem for me right now.

However, it still feels like we'd want to solve this in the general case, for times when there isn't going to be a workaround.

[hwright]

@duggelz @mattmoor It looks like this might also be related to bazelbuild/bazel#3871 ?

[hwright]

Ping. I'm still interested in helping get this fixed, but need a few pointers.

[joshclimacell]

Ping...

[joshclimacell]

For reference, I implemented a (not-super-awesome, but functioning) workaround for loading Python 3 PyPi packages: joshclimacell@90a70d5

[duggelz]

We need something like a py_toolchain target in Bazel so that it actually knows what version of Python it's invoking. See py_toolchain draft proposal

[jkinkead]

This affects tornado as well. Its setup.py selectively adds dependencies depending on the runtime version of Python detected. The concurrent library is not Python 3 compatible, and breaks the runtime.

[jkinkead]

Also, I'd love to see @joshclimacell's patch in this repo.

[jkinkead]

So, I noticed that there's not only a duplicate issue for this (#62), but there's also a duplicate PR for this one (#82). Can one of the project maintainers (@mattmoor ? @duggelz ?) help get this reconciled?

[limdor]

Any update on this?

[brandjon]

Using this as repro to merge (a rebase of) #85; work toward #249.