wheel: script with multiprocessing doesn't work on Windows

[schlamar]

setup.py

from setuptools import setup

setup(
    version='0.0.1',
    name="blub",
    py_modules=["blub"],
    entry_points={
        'console_scripts': ['blub = blub:main'],
    },
)

blub.py

import multiprocessing

def f():
    pass

def main():
    p = multiprocessing.Process(target=f)
    p.start()
    p.join()
    print 'xxx'

When installing this without wheel, everything is fine:

$ pip install .
$ blub
xxx

Installing this as wheel, the script is broken:

$ pip uninstall blub
$ pip wheel .
$ pip install wheelhouse/*
$ blub
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "c:\Python27\Lib\multiprocessing\forking.py", line 380, in main
    prepare(preparation_data)
  File "c:\Python27\Lib\multiprocessing\forking.py", line 488, in prepare
    assert main_name not in sys.modules, main_name
AssertionError: __main__
xxx

This is probably related to http://bugs.python.org/issue10845.

[ncoghlan]

http://bugs.python.org/issue10845 shouldn't be related to this, as that only affects running with the "-m" switch.

However, a potentially relevant different here is that (if I understand things correctly), when installing from source, setuptools generates the wrapper script/executable, while when installing from a wheel, pip does it.

[pfmoore]

That certainly sounds possible. The wrappers used when installing from a wheel are based on distlib's ScripMaker module, but with a modified template (see https://github.com/pypa/pip/blob/develop/pip/wheel.py#L284). It would be useful to comment out lines 284-305 of pip/wheel.py, and retry the test. If that fixes the problems, pip's modified wrapper is the issue. Otherwise, it's likely to be a distlib issue in ScriptMaker.

[xflr6]

Commenting out this part (line numbers have changed) and reinstalling, the problem still occurs (the produced .exe-file was slightly larger than before). Is the script included inside the .exe?

This may affect more users now that pip builds wheels for sdist packages by default.

[xflr6]

A workaround is to run wheel install-scripts <packagename> after the installation (which creates working .exe + -script.py launchers)

I find it somewhat confusing that there are so many (independent?) code paths for launcher installation.

[schlamar]

This is really annoying, I'm stumbling upon this issue from time to time after upgrades. Is there an option to globally disable wheels (-no-use-wheel)?

[piotr-dobrogost]

Is there an option to globally disable wheels (-no-use-wheel)?

From http://pip.readthedocs.org/en/stable/reference/pip_install/#install-no-binary

--no-binary <format_control>
Do not use binary packages. Can be supplied multiple times, and each time adds to the existing value. Accepts either :all: to disable all binary packages, :none: to empty the set, or one or more package names with commas between them. Note that some packages are tricky to compile and may fail to install when this option is used on them.

[schlamar]

@piotr-dobrogost there is already --no-use-wheel. The question is how can this configured to be used every time without explicitly specifying it.

A pip.ini with

[install]
no-use-wheel = true

doesn't work nor a PIP_NO_USE_WHEEL env variable. Maybe this is an issue on its own?

BTW, --no-binary still installs the wheel so this is no 1:1 replacement for --no-use-wheel so this shouldn't be marked as deprecated!

[schlamar]

OK, looks like the correct way to set this is

[install]
no-use-wheel = false

Or PIP_NO_USE_WHEEL=0.

Weird...

[jenisys]

PIP-VERSION: 7.1.2
SETUPTOOLS-VERSION: 18.4

This problem is not related to wheel-based installations.
This problem exists also when you install a source distribution from a ZIP file.
The problem seems to be caused how the <script>.exe is created in the "Scripts/" sub-directory.
A __main__.py is archived/embedded in the <script>.exe executable. Probably the sys.executable from __main__.py is not resolved correctly when the multiprocessing.forking module is involved.

WORKAROUNDS (alternatives):

1. Install package with easy_install (setuptools) and everything is fine.
2. Put a python script ala <script>.py in the "Scripts/" directory and everything is fine.

EXAMPLE:

1. Checkout Github repository: https://github.com/cgoldberg/multi-mechanize
2. Create a virtual environment and install multi-mechanize from checkout repository.
3. Execute "multimech-run.exe examples"

MAYBE:
multiprocessing.set_executable(...) is needed in scripts that use multiprocessing (on Windows) !?!

• http://stackoverflow.com/questions/15636266/embedded-python-multiprocessing-not-working
• https://docs.python.org/2/library/multiprocessing.html#multiprocessing.set_executable

[ncoghlan]

It's important to keep in mind here that the interaction between
multiprocessing and main on Windows under Python 2.7 is fairly
fundamentally broken in various ways.

This is fixed in Python 3.4+, as the ability to select the "spawn"
execution model meant I could finally reproduce those previously Windows
specific problems on Linux, and PEP 451's improvements to the import system
meant I could fix them: https://bugs.python.org/issue19946

The multiprocessing support in 2.7 is now ~5 years older than that in
Python 3, and it shows.

[schlamar]

Yes, obviously. So an universal Script launcher shouldn't use it. Why does installing a wheel creating a different launcher than setuptools's default?

@jenisys That's probably the case because pip is creating a wheel from sdist packages since a while now. So it's an issue with wheels.

[schlamar]

@ncoghlan I analysed this a little bit. The launcher from distlib/pip does basically python script.exe where script.exe looks like a zip file because it has an embedded __main__.py file. So its like calling PYTHONPATH=script.exe python -m __main__.

This means it is probably related to http://bugs.python.org/issue10845 as I initially thought!

[schlamar]

Report upstream against distlib/simple_launcher: https://bitbucket.org/vinay.sajip/simple_launcher/issues/1/scripts-with-multiprocessing-broken-on

[schlamar]

Will be fixed in Python 2.7.11: http://bugs.python.org/issue10128

[dstufft]

Closing this, this is a bug with Python itself and it appears fixed in 3.x and 2.7.111 so the recommendation is to get a newer Python if you are hitting this.