Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding Python 3.8, Always use nuget to install python on Windows #180

Merged
merged 4 commits into from
Nov 5, 2019
Merged

Adding Python 3.8, Always use nuget to install python on Windows #180

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
58048b4
Remove trailing-spaces from README.md
mayeut Nov 3, 2019
f2ef7a6
Always use nuget to install python on Windows
mayeut Oct 25, 2019
104dcf1
Add python 3.8 support
mayeut Nov 2, 2019
05ab66c
Fix documentation for macOS python 3.8
mayeut Nov 4, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 21 additions & 26 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,13 @@ Python wheels are great. Building them across **Mac, Linux, Windows**, on **mult
What does it do?
----------------

| | macOS 10.6+ | manylinux i686 | manylinux x86_64 | Windows 32bit | Windows 64bit |
|---|---|---|---|---|---|
| Python 2.7 | ✅ | ✅ | ✅ | ✅¹ | ✅¹ |
| Python 3.5 | ✅ | ✅ | ✅ | ✅ | ✅ |
| Python 3.6 | ✅ | ✅ | ✅ | ✅ | ✅ |
| Python 3.7 | ✅ | ✅ | ✅ | ✅ | ✅ |
| | macOS 10.6+ intel | macOS 10.9+ x86_64 | manylinux i686 | manylinux x86_64 | Windows 32bit | Windows 64bit |
|---|---|---|---|---|---|---|
| Python 2.7 | ✅ | | ✅ | ✅ | ✅¹ | ✅¹ |
| Python 3.5 | ✅ | | ✅ | ✅ | ✅ | ✅ |
| Python 3.6 | ✅ | | ✅ | ✅ | ✅ | ✅ |
| Python 3.7 | ✅ | | ✅ | ✅ | ✅ | ✅ |
| Python 3.8 | | ✅ | ✅ | ✅ | ✅ | ✅ |

> ¹ Not supported on Travis

Expand Down Expand Up @@ -76,14 +77,7 @@ jobs:
- job: windows
pool: {vmImage: 'vs2017-win2016'}
steps:
- {task: UsePythonVersion@0, inputs: {versionSpec: '2.7', architecture: x86}}
- {task: UsePythonVersion@0, inputs: {versionSpec: '2.7', architecture: x64}}
- {task: UsePythonVersion@0, inputs: {versionSpec: '3.5', architecture: x86}}
- {task: UsePythonVersion@0, inputs: {versionSpec: '3.5', architecture: x64}}
- {task: UsePythonVersion@0, inputs: {versionSpec: '3.6', architecture: x86}}
- {task: UsePythonVersion@0, inputs: {versionSpec: '3.6', architecture: x64}}
- {task: UsePythonVersion@0, inputs: {versionSpec: '3.7', architecture: x86}}
- {task: UsePythonVersion@0, inputs: {versionSpec: '3.7', architecture: x64}}
- task: UsePythonVersion@0
- script: choco install vcpython27 -f -y
displayName: Install Visual C++ for Python 2.7
- bash: |
Expand Down Expand Up @@ -309,9 +303,9 @@ When both options are specified, both conditions are applied and only builds wit

The format is `python_tag-platform_tag`. The tags are similar but not identical to the ones defined in [PEP 425](https://www.python.org/dev/peps/pep-0425/#details).

Python tags look like `cp27` `cp35` `cp36` `cp37`
Python tags look like `cp27` `cp35` `cp36` `cp37` `cp38`

Platform tags look like `macosx_10_6_intel` `manylinux_x86_64` `manylinux_i686` `win32` `win_amd64`
Platform tags look like `macosx_10_6_intel` `macosx_10_9_x86_64` `manylinux_x86_64` `manylinux_i686` `win32` `win_amd64`

You can also use shell-style globbing syntax (as per `fnmatch`).

Expand All @@ -320,6 +314,7 @@ The list of supported and currently selected build identifiers can be retrieved
Examples:
- Only build on Python 3.6: `CIBW_BUILD`:`cp36-*`
- Skip building on Python 2.7 on the Mac: `CIBW_SKIP`:`cp27-macosx_10_6_intel`
- Skip building on Python 3.8 on the Mac: `CIBW_SKIP`:`cp38-macosx_10_9_x86_64`
- Skip building on Python 2.7 on all platforms: `CIBW_SKIP`:`cp27-*`
- Skip Python 2.7 on Windows: `CIBW_SKIP`:`cp27-win*`
- Skip Python 2.7 on 32-bit Windows: `CIBW_SKIP`:`cp27-win32`
Expand Down Expand Up @@ -352,10 +347,10 @@ You must set this variable to pass variables to Linux builds (since they execute

You can use `$PATH` syntax to insert other variables, or the `$(pwd)` syntax to insert the output of other shell commands.

Example: `CFLAGS="-g -Wall" CXXFLAGS="-Wall"`
Example: `PATH=$PATH:/usr/local/bin`
Example: `BUILD_TIME="$(date)"`
Example: `PIP_EXTRA_INDEX_URL="https://pypi.myorg.com/simple"`
Example: `CFLAGS="-g -Wall" CXXFLAGS="-Wall"`\
Example: `PATH=$PATH:/usr/local/bin`\
Example: `BUILD_TIME="$(date)"`\
Example: `PIP_EXTRA_INDEX_URL="https://pypi.myorg.com/simple"`\

Platform-specific variants also available:
`CIBW_ENVIRONMENT_MACOS` | `CIBW_ENVIRONMENT_WINDOWS` | `CIBW_ENVIRONMENT_LINUX`
Expand All @@ -375,11 +370,11 @@ If dependencies are required to build your wheel (for example if you include a h

The active Python binary can be accessed using `python`, and pip with `pip`; `cibuildwheel` makes sure the right version of Python and pip will be executed. `{project}` can be used as a placeholder for the absolute path to the project's root.

Example: `pip install .`
Example: `pip install pybind11`
Example: `pip install .`\
Example: `pip install pybind11`\
Example: `yum install -y libffi-dev && pip install .`

Platform-specific variants also available:
Platform-specific variants also available:\
`CIBW_BEFORE_BUILD_MACOS` | `CIBW_BEFORE_BUILD_WINDOWS` | `CIBW_BEFORE_BUILD_LINUX`

***
Expand All @@ -397,8 +392,8 @@ Beware to specify a valid Docker image that can be used in the same way as the o

Note that `auditwheel` detects the version of the `manylinux` standard in the Docker image through the `AUDITWHEEL_PLAT` environment variable, as `cibuildwheel` has no way of detecting the correct `--plat` command line argument to pass to `auditwheel` for a custom image. If a Docker image does not correctly set this `AUDITWHEEL_PLAT` environment variable, the `CIBW_ENVIRONMENT` option can be used to do so (e.g., `CIBW_ENVIRONMENT="manylinux2010_$(uname -m)"`).

Example: `manylinux1`
Example: `dockcross/manylinux-x64`
Example: `manylinux1`\
Example: `dockcross/manylinux-x64`\
Example: `dockcross/manylinux-x86`

***
Expand Down Expand Up @@ -426,7 +421,7 @@ Optional.

Space-separated list of dependencies required for running the tests.

Example: `pytest`
Example: `pytest`\
Example: `nose==1.3.7 moto==0.4.31`

Platform-specific variants also available:
Expand Down
17 changes: 5 additions & 12 deletions azure-pipelines.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,31 +1,24 @@
jobs:
- job: linux
pool: {vmImage: 'Ubuntu-16.04'}
steps:
pool: {vmImage: 'Ubuntu-18.04'}
steps:
- task: UsePythonVersion@0
- bash: |
python -m pip install -r requirements-dev.txt
python ./bin/run_tests.py

- job: macos
pool: {vmImage: 'macOS-10.13'}
steps:
steps:
- task: UsePythonVersion@0
- bash: |
python -m pip install -r requirements-dev.txt
python ./bin/run_tests.py

- job: windows
pool: {vmImage: 'vs2017-win2016'}
steps:
- {task: UsePythonVersion@0, inputs: {versionSpec: '2.7', architecture: x86}}
- {task: UsePythonVersion@0, inputs: {versionSpec: '2.7', architecture: x64}}
- {task: UsePythonVersion@0, inputs: {versionSpec: '3.5', architecture: x86}}
- {task: UsePythonVersion@0, inputs: {versionSpec: '3.5', architecture: x64}}
- {task: UsePythonVersion@0, inputs: {versionSpec: '3.6', architecture: x86}}
- {task: UsePythonVersion@0, inputs: {versionSpec: '3.6', architecture: x64}}
- {task: UsePythonVersion@0, inputs: {versionSpec: '3.7', architecture: x86}}
- {task: UsePythonVersion@0, inputs: {versionSpec: '3.7', architecture: x64}}
steps:
- task: UsePythonVersion@0
- script: choco install vcpython27 -f -y
displayName: Install Visual C++ for Python 2.7
- bash: |
Expand Down
2 changes: 2 additions & 0 deletions cibuildwheel/linux.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,13 @@ def get_python_configurations(build_selector):
PythonConfiguration(identifier='cp35-manylinux_x86_64', path='/opt/python/cp35-cp35m'),
PythonConfiguration(identifier='cp36-manylinux_x86_64', path='/opt/python/cp36-cp36m'),
PythonConfiguration(identifier='cp37-manylinux_x86_64', path='/opt/python/cp37-cp37m'),
PythonConfiguration(identifier='cp38-manylinux_x86_64', path='/opt/python/cp38-cp38'),
PythonConfiguration(identifier='cp27-manylinux_i686', path='/opt/python/cp27-cp27m'),
PythonConfiguration(identifier='cp27-manylinux_i686', path='/opt/python/cp27-cp27mu'),
PythonConfiguration(identifier='cp35-manylinux_i686', path='/opt/python/cp35-cp35m'),
PythonConfiguration(identifier='cp36-manylinux_i686', path='/opt/python/cp36-cp36m'),
PythonConfiguration(identifier='cp37-manylinux_i686', path='/opt/python/cp37-cp37m'),
PythonConfiguration(identifier='cp38-manylinux_i686', path='/opt/python/cp38-cp38'),
]

# skip builds as required
Expand Down
1 change: 1 addition & 0 deletions cibuildwheel/macos.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ def get_python_configurations(build_selector):
PythonConfiguration(version='3.5', identifier='cp35-macosx_10_6_intel', url='https://www.python.org/ftp/python/3.5.4/python-3.5.4-macosx10.6.pkg'),
PythonConfiguration(version='3.6', identifier='cp36-macosx_10_6_intel', url='https://www.python.org/ftp/python/3.6.8/python-3.6.8-macosx10.6.pkg'),
PythonConfiguration(version='3.7', identifier='cp37-macosx_10_6_intel', url='https://www.python.org/ftp/python/3.7.5/python-3.7.5-macosx10.6.pkg'),
PythonConfiguration(version='3.8', identifier='cp38-macosx_10_9_x86_64', url='https://www.python.org/ftp/python/3.8.0/python-3.8.0-macosx10.9.pkg'),
mayeut marked this conversation as resolved.
Show resolved Hide resolved
]

# skip builds as required
Expand Down
105 changes: 44 additions & 61 deletions cibuildwheel/windows.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,67 +8,48 @@
except ImportError:
from pipes import quote as shlex_quote

try:
from urllib.request import urlopen
except ImportError:
from urllib2 import urlopen

from .util import prepare_command, get_build_verbosity_extra_flags


IS_RUNNING_ON_AZURE = os.path.exists('C:\\hostedtoolcache')
IS_RUNNING_ON_TRAVIS = os.environ.get('TRAVIS_OS_NAME') == 'windows'


def get_python_path(config):
if IS_RUNNING_ON_AZURE:
# We can't hard-code the paths because on Azure, we don't know which
# bugfix release of Python we are getting so we need to check which
# ones exist. We just use the first one that is found since there should
# only be one.
path_pattern = 'C:\\hostedtoolcache\\windows\\Python\\{version}\\{arch}'.format(
version=config.version.replace('x', '*'),
arch='x86' if config.arch == '32' else 'x64'
)
try:
return glob(path_pattern)[0]
except IndexError:
raise Exception('Could not find a Python install at ' + path_pattern)
elif IS_RUNNING_ON_TRAVIS:
if config.version == "3.4.x":
return config.path
else:
nuget_args = get_nuget_args(config)
return os.path.join(nuget_args[-1], nuget_args[0] + "." + config.nuget_version, "tools")
else:
# Assume we're running on AppVeyor
major, minor = config.version.split('.')[:2]
return 'C:\\Python{major}{minor}{arch}'.format(
major=major,
minor=minor,
arch = '-x64' if config.arch == '64' else ''
)
nuget_args = get_nuget_args(config)
return os.path.join(nuget_args[-1], nuget_args[0] + "." + config.version, "tools")


def get_nuget_args(configuration):
if configuration.nuget_version is None:
return None
python_name = "python" if configuration.version[0] == '3' else "python2"
if configuration.arch == "32":
python_name = python_name + "x86"
return [python_name, "-Version", configuration.nuget_version, "-OutputDirectory", "C:/python"]
return [python_name, "-Version", configuration.version, "-OutputDirectory", "C:/python"]

def get_python_configurations(build_selector):
PythonConfiguration = namedtuple('PythonConfiguration', ['version', 'arch', 'identifier', 'path', "nuget_version"])
PythonConfiguration = namedtuple('PythonConfiguration', ['version', 'arch', 'identifier'])
python_configurations = [
PythonConfiguration(version='2.7.x', arch="32", identifier='cp27-win32', path='C:\\Python27', nuget_version="2.7.16"),
PythonConfiguration(version='2.7.x', arch="64", identifier='cp27-win_amd64', path='C:\\Python27-x64', nuget_version="2.7.16"),
PythonConfiguration(version='3.5.x', arch="32", identifier='cp35-win32', path='C:\\Python35', nuget_version="3.5.4"),
PythonConfiguration(version='3.5.x', arch="64", identifier='cp35-win_amd64', path='C:\\Python35-x64', nuget_version="3.5.4"),
PythonConfiguration(version='3.6.x', arch="32", identifier='cp36-win32', path='C:\\Python36', nuget_version="3.6.8"),
PythonConfiguration(version='3.6.x', arch="64", identifier='cp36-win_amd64', path='C:\\Python36-x64', nuget_version="3.6.8"),
PythonConfiguration(version='3.7.x', arch="32", identifier='cp37-win32', path='C:\\Python37', nuget_version="3.7.4"),
PythonConfiguration(version='3.7.x', arch="64", identifier='cp37-win_amd64', path='C:\\Python37-x64', nuget_version="3.7.4")
PythonConfiguration(version='2.7.17', arch="32", identifier='cp27-win32'),
PythonConfiguration(version='2.7.17', arch="64", identifier='cp27-win_amd64'),
PythonConfiguration(version='3.5.4', arch="32", identifier='cp35-win32'),
PythonConfiguration(version='3.5.4', arch="64", identifier='cp35-win_amd64'),
PythonConfiguration(version='3.6.8', arch="32", identifier='cp36-win32'),
PythonConfiguration(version='3.6.8', arch="64", identifier='cp36-win_amd64'),
PythonConfiguration(version='3.7.5', arch="32", identifier='cp37-win32'),
PythonConfiguration(version='3.7.5', arch="64", identifier='cp37-win_amd64'),
PythonConfiguration(version='3.8.0', arch="32", identifier='cp38-win32'),
PythonConfiguration(version='3.8.0', arch="64", identifier='cp38-win_amd64'),
]

if IS_RUNNING_ON_TRAVIS:
# cannot install VCForPython27.msi which is needed for compiling C software
# try with (and similar): msiexec /i VCForPython27.msi ALLUSERS=1 ACCEPT=YES /passive
python_configurations = [c for c in python_configurations if c.version != '2.7.x']
python_configurations = [c for c in python_configurations if not c.version.startswith('2.7.')]

# skip builds as required
python_configurations = [c for c in python_configurations if build_selector(c.identifier)]
Expand All @@ -78,11 +59,20 @@ def get_python_configurations(build_selector):


def build(project_dir, output_dir, test_command, test_requires, test_extras, before_build, build_verbosity, build_selector, environment):
def simple_shell(args, env=None, cwd=None):
print('+ ' + ' '.join(args))
args = ['cmd', '/E:ON', '/V:ON', '/C'] + args
return subprocess.check_call(' '.join(args), env=env, cwd=cwd)
def download(url, dest):
print('+ Download ' + url + ' to ' + dest)
response = urlopen(url)
try:
with open(dest, 'wb') as file:
file.write(response.read())
finally:
response.close()
if IS_RUNNING_ON_AZURE or IS_RUNNING_ON_TRAVIS:
def shell(args, env=None, cwd=None):
print('+ ' + ' '.join(args))
args = ['cmd', '/E:ON', '/V:ON', '/C'] + args
return subprocess.check_call(' '.join(args), env=env, cwd=cwd)
shell = simple_shell
else:
run_with_env = os.path.abspath(os.path.join(os.path.dirname(__file__), 'resources', 'appveyor_run_with_env.cmd'))

Expand All @@ -98,21 +88,19 @@ def shell(args, env=None, cwd=None):
temp_dir = tempfile.mkdtemp(prefix='cibuildwheel')
built_wheel_dir = os.path.join(temp_dir, 'built_wheel')

if IS_RUNNING_ON_TRAVIS:
# instal nuget as best way for provide python
shell(["choco", "install", "nuget.commandline"])
# get pip fo this installation which not have.
get_pip_url = 'https://bootstrap.pypa.io/get-pip.py'
get_pip_script = 'C:\\get-pip.py'
shell(['curl', '-L', '-o', get_pip_script, get_pip_url])
# install nuget as best way to provide python
nuget = 'C:\\nuget.exe'
download('https://dist.nuget.org/win-x86-commandline/latest/nuget.exe', nuget)
# get pip fo this installation which not have.
get_pip_script = 'C:\\get-pip.py'
download('https://bootstrap.pypa.io/get-pip.py', get_pip_script)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curious: do nuget versions not have pip installed?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've not redone the test, it's merely a rewrite of the travis-ci PR which was doing this.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure we hit the conditional statement lower in the code.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm, two lines and a download. Not sure if it's worth testing whether we can remove the lines.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems pip is not installed

Hmmm, two lines and a download. Not sure if it's worth testing whether we can remove the lines.

Not sure I fully understand this comment. If it only refers to get-pip then no, we can't remove those (now tested).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that's what I meant. That's kind of silly. Why do all these distributions not have pip installed?


python_configurations = get_python_configurations(build_selector)
for config in python_configurations:
config_python_path = get_python_path(config)
if IS_RUNNING_ON_TRAVIS and config.nuget_version is not None and not os.path.exists(config_python_path):
shell(["nuget", "install"] + get_nuget_args(config))
if not os.path.exists(os.path.join(config_python_path, 'Scripts', 'pip.exe')):
shell([os.path.join(config_python_path, 'python.exe'), get_pip_script ])
simple_shell([nuget, "install"] + get_nuget_args(config))
YannickJadoul marked this conversation as resolved.
Show resolved Hide resolved
if not os.path.exists(os.path.join(config_python_path, 'Scripts', 'pip.exe')):
simple_shell([os.path.join(config_python_path, 'python.exe'), get_pip_script ])

# check python & pip exist for this configuration
assert os.path.exists(os.path.join(config_python_path, 'python.exe'))
Expand All @@ -139,12 +127,7 @@ def shell(args, env=None, cwd=None):
shell(['python', '-c', '"import struct; print(struct.calcsize(\'P\') * 8)\"'], env=env)

# prepare the Python environment
if config.version == "3.4.x":
shell(['python', '-m', 'pip', 'install', 'pip==19.1.1'],
env=env)
else:
shell(['python', '-m', 'pip', 'install', '--upgrade', 'pip'],
env=env)
shell(['python', '-m', 'pip', 'install', '--upgrade', 'pip'], env=env)
shell(['pip', 'install', '--upgrade', 'setuptools'], env=env)
shell(['pip', 'install', 'wheel'], env=env)

Expand Down
Loading