automotive demo is not working on a mac

[soonho-tri]

How to reproduce

bazel run //drake/automotive:demo -- --num_dragway_lanes=3 --num_trajectory_car=12

Symptom

Fatal Python error: PyThreadState_Get: no current thread

Related PR/Issues : #6616, #6333

Diagnosis

From https://stackoverflow.com/questions/35640529/what-does-fatal-python-error-pythreadstate-get-no-current-thread-mean :

This problem may occur if the .so file from homebrew's formula links against a different python library than the interpreter you are running.

which python returns /usr/bin/python while homebrew's python is located at /usr/local/opt/python/libexec/bin/python. Note that homebrew only put python2 and python2.7 under /usr/local/bin.

A workaround

From brew info python:

==> Caveats
This formula installs a python2 executable to /usr/local/bin.
If you wish to have this formula's python executable in your PATH then add
the following to ~/.zshrc:
  export PATH="/usr/local/opt/python/libexec/bin:$PATH"

Preferably, we need to find a solution to call the homebrew's python from the demo (without asking for modifying the PATH env variable). If that's not feasible, we need to add an instruction to update PATH in the documentation.

[jamiesnape]

Using the toolchains argument of py_binary seems like a possibility. We would need to examine the generated wrapper scripts that launch Python "binaries" to find the appropriate variable for which to substitute.

[jamiesnape]

The other workaround is to add --python2_path= to a user bazelrc.

[EricCousineau-TRI]

FWIW, I'd be up for just checking which python, and if it is /usr/bin/python, possibly in install_prereqs.sh, ask the user to follow the workaround (either augment $PATH, or use --python2_path) as a warning at the beginning, and incorporate this in the FAQ.
That being said, should we have a check_environment.sh script that could do this sort of thing?

My preference would be to avoid a toolchain, so we don't lock the user into a specific Python binary. (When debugging bindings, I found it useful to (a) use a debug version of Ubuntu-packaged Python and (b) build Python from source in debug mode to crawl through the source a little.)

[jamiesnape]

I wonder whether installing a .bazelrc into the workspace root for Mac users that sets —python2_path may not be such a bad thing. Better than doing something that affects other Bazel (e.g., ~/.bazelrc) or non-Bazel (e.g., PATH) and easy to remove or override.

[jwnimmer-tri]

As another option: we could create an executable file tools/bazel (in bash) that checked if you were on Mac, and if so added --python2_path before shelling out to the system Bazel.

[jamiesnape]

I can see advantages with either, though I have a slight aversion to wrappers. I have some tentative plans to factor out the small number of flags (e.g, —keep_going) we add on CI into a .bazelrc, though I think either approach could be compatible with that.

[EricCousineau-TRI]

My preference would be to have this be system-wide (using the $PATH shadowing), especially if it affects users outside of Bazel, e.g. upon install they get the same error when attempting to import pydrake.
This would avoid the need for a wrapper, and could also be handled in some sort of setup.bash as well.

For the .bazelrc approach, I think the only appropriate solution would be the tools/bazel approach, either injecting --python2_path on the command line or injecting it via a ${workspace}/.bazelrc, which would then conflict with the user's ~/.bazelrc (something we'd have to call out very loudly), and either keep it out of version control to keep from everything being dirty, or just prevent people from committing on it.
(My thoughts on the local ${workspace}/.bazelrc are that it'd be useful only if it's not version controlled, and only if the user explicitly creates that file. Additionally, we'd probably have to inject logic to detect --bazelrc to prevent us from overriding the user's preference at the command-line.)

Another option is to explicitly use #!/usr/bin/env python2 in the hash-bang in Drake shell scripts, which would avoid this as well. The only caveat is that we'd need to encourage users to explicitly choose an interpreter version.
This could pave the way to more explicitly transitioning to python3.

[jamiesnape]

The local .bazelrc would be version controlled, but not placed in the root of the workspace since it is only applicable on Mac, so it would be copied either by mac/install_prereqs.sh or manually. #!/usr/bin/env python2 may bring back problems with SIP (not sure how strictly /usr/bin/env is regulated). The wrapper approach would also cause problems with nuking a user set --python2_path so there is no perfect solution. My feeling is that if you know enough about Bazel to be setting a ~/.bazelrc, you can probably deal with the workspace .bazelrc.

[jamiesnape]

(Note that Homebrew already draws attention to the problem of Python not being in the PATH and offers a solution pretty clearly, so any warning message in install_prereqs.sh is not likely be any more successful)

==> Caveats
This formula installs a python2 executable to /usr/local/bin.
If you wish to have this formula's python executable in your PATH then add
the following to ~/.bash_profile:
  export PATH="/usr/local/opt/python/libexec/bin:$PATH"

[jamiesnape]

We should probably also verify the behavior of having all of ~/.bazelrc, WORKSPACE/.bazelrc, tools/bazel.rc, and command line flags. While detailed and clearly written, the Bazel documentation is also frequently outdated/wrong.

[jwnimmer-tri]

@soonho-tri Is this any better now? Is there anything left to do, or could we close it?

[soonho-tri]

I think we're good now. @jamiesnape , please re-open this if you think not.