Accept Python 3 on Windows #29236

joaocgreis · 2019-08-20T21:44:08Z

This loosely depends on #25878. Doesn't make much sense without it, but doesn't break either.

This allows Python 3 to be used, but only if it is the only Python version installed. #25878 does this for configure, this PR does for vcbuild.

If Python 3 is being used and configure fails, a warning is displayed.

cc @nodejs/platform-windows @nodejs/python @cclauss

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
commit message follows commit guidelines

When looking for Python in the registry, as specified in PEP514, this was not able to handle installations in a path with spaces, like Program Files. This ensures the whole path is used, fixing the issue.

If there is no Python 2 available, use Python 3. This allows to test running configure with Python 3.

nodejs-github-bot · 2019-08-20T22:35:14Z

CI: https://ci.nodejs.org/job/node-test-pull-request/25134/ ✔️

cclauss · 2019-08-21T05:28:16Z

I am not a Windows guy so patience please...
If I go to the Microsoft Windows Store and install Python from there, where will it go?
My sense is that py is always set to the current preferred Python 3 on Windows boxes. Is that a better way to find py3?

joaocgreis · 2019-08-21T12:04:07Z

If I go to the Microsoft Windows Store and install Python from there, where will it go?

Under %LocalAppData%\Microsoft\WindowsApps\, which is already in PATH.

My sense is that py is always set to the current preferred Python 3 on Windows boxes. Is that a better way to find py3?

Possibly, but isn't that mostly for people running multiple versions? Is that worth the complexity for our case? For node-gyp that would make more sense I think, if someone has the time to implement it.

Here, we should make sure an installation of Python with all things default works - that's why the PEP514 detection was added: nodejs/CTC#147 (comment). For someone wanting to customize, it's reasonable to assume they can add Python to the path.

Trott · 2019-08-23T04:19:40Z

@nodejs/python @nodejs/platform-windows @nodejs/build-files This could use some reviews.

It's not clear to me if this should be blocked on #25878 or not. Based on @joaocgreis's comment, I'd say not, but I'd love someone else's opinion (especially @joaocgreis's!).

cclauss · 2019-08-23T07:04:31Z

What happens on a Windows box when this is run? How far does it get and what are the results? What cannot be achieved by doing python3 configure.py? Our current Python 3.6 runs on Travis Linux run for 50 minutes before timing out. I would be interested to see our builds get to some interesting place before we open the experimentation to a broad audience.

As discussed at #29196 (comment) there are several vectors of testing that I believe we should shape up before opening the floodgates with #25878 and #29236.

PYTHON=python3 ; tools/gyp_node.py
PYTHON=python3 ; tools/test.py
python3 configure.py ; make test # current Travis CI run
python3 configure.py ; make install

cclauss · 2019-08-25T19:44:03Z

Explained at 9m30s into
https://pythonbytes.fm/episodes/show/130/python.exe-now-shipping-with-windows-10

cclauss

Given the state of #25789, I think we are ready to land this one.

What are the results of python3.6 configure.py ?

joaocgreis · 2019-08-29T15:23:59Z

I think this can land. Building with Python 3 does not work, but the error is clear and this allows for experimentation.

@cclauss running configure.py directly on Windows is not straightforward because the environment needs to be set up by vcbuild.bat. FWIW this is what happens:

C:\Users\Administrator\Desktop\node>C:\Users\Administrator\AppData\Local\Programs\Python\Python36\python.exe configure.py
�[1m�[31mERROR�[0m: Did not find a new enough assembler, install one or build with
       --openssl-no-asm.
       Please refer to BUILDING.md

Running this together with #25878 is a better option, this is what happens:

C:\Users\Administrator\Desktop\node>git checkout nodejs/node/master
...
C:\Users\Administrator\Desktop\node>git clean -fdx
...
C:\Users\Administrator\Desktop\node>git cherry-pick nodejs/node/pull/25878~4..nodejs/node/pull/25878
...
C:\Users\Administrator\Desktop\node>git cherry-pick nodejs/node/pull/29236~2..nodejs/node/pull/29236
...
C:\Users\Administrator\Desktop\node>vcbuild.bat
Looking for Python
Python found in C:\Users\Administrator\AppData\Local\Programs\Python\Python36\\python.exe
WARNING: Python 3 is not yet fully supported, to avoid issues Python 2 should be installed.
Looking for NASM
Looking for Visual Studio 2017
Found MSVS version 15.0
configure  --dest-cpu=x64
Node configure: Found Python 3.6.8...
�[1m�[32mINFO�[0m: Using floating patch "tools/icu/patches/64/source/common/putil.cpp" from "tools/icu"
�[1m�[32mINFO�[0m: Using floating patch "tools/icu/patches/64/source/i18n/dtptngen.cpp" from "tools/icu"
Traceback (most recent call last):
  File "configure", line 23, in <module>
    import configure
  File "C:\Users\Administrator\Desktop\node\configure.py", line 1716, in <module>
    run_gyp(gyp_args)
  File "tools\gyp_node.py", line 54, in run_gyp
    rc = gyp.main(args)
  File "tools\gyp\pylib\gyp\__init__.py", line 547, in main
    return gyp_main(args)
  File "tools\gyp\pylib\gyp\__init__.py", line 523, in gyp_main
    options.duplicate_basename_check)
  File "tools\gyp\pylib\gyp\__init__.py", line 100, in Load
    generator = __import__(generator_name, globals(), locals(), generator_name)
  File "tools\gyp\pylib\gyp\generator\msvs.py", line 17, in <module>
    import gyp.generator.ninja as ninja_generator
  File "tools\gyp\pylib\gyp\generator\ninja.py", line 23, in <module>
    from cStringIO import StringIO
ModuleNotFoundError: No module named 'cStringIO'
Failed to create vc project files.
Python 3 is not yet fully supported, to avoid issues Python 2 should be installed.

C:\Users\Administrator\Desktop\node>

cclauss · 2019-08-29T20:37:28Z

Feel free to cherrypick in #29371 if it helps you make progress.

It would be cool if we could set up a Jenkins, Travis CI, or GitHub Actions-based CI, or other CI that exercised Python 3 on Windows.

bnoordhuis · 2019-08-30T18:19:31Z

My review was requested but I don't think I'm the right person to sign off on this. My Windows shell scripting skills are close to non-existent.

Trott · 2019-08-31T00:51:52Z

FWIW, since this has been open for more than a week, one approval is enough to land it as long as there isn't any opposition to it (which I'm not seeing any) and CI passes (which it did).

Trott · 2019-08-31T00:52:54Z

vcbuild.bat

@@ -689,6 +689,9 @@ goto exit

 :create-msvs-files-failed
 echo Failed to create vc project files.
+if %VCBUILD_PYTHON_VERSION%==3 (
+  echo Python 3 is not yet fully supported, to avoid issues Python 2 should be installed.


Nit that can totally be ignored, especially since this message eventually will go away anyway.

Suggested change

echo Python 3 is not yet fully supported, to avoid issues Python 2 should be installed.

echo Python 3 is not yet fully supported. To avoid issues, Python 2 should be installed.

Trott

A bit rubber-stamp-y of me since I don't have a Windows machine available to test on, but LGTM as far as I can tell.

When looking for Python in the registry, as specified in PEP514, this was not able to handle installations in a path with spaces, like Program Files. This ensures the whole path is used, fixing the issue. PR-URL: #29236 Reviewed-By: Christian Clauss <[email protected]> Reviewed-By: Rich Trott <[email protected]>

If there is no Python 2 available, use Python 3. This allows to test running configure with Python 3. PR-URL: #29236 Reviewed-By: Christian Clauss <[email protected]> Reviewed-By: Rich Trott <[email protected]>

Trott · 2019-09-03T04:07:36Z

Landed in ab841d5...d18b6a7

joaocgreis · 2019-09-03T14:51:17Z

@Trott thanks for landing this!

When looking for Python in the registry, as specified in PEP514, this was not able to handle installations in a path with spaces, like Program Files. This ensures the whole path is used, fixing the issue. PR-URL: #29236 Reviewed-By: Christian Clauss <[email protected]> Reviewed-By: Rich Trott <[email protected]>

If there is no Python 2 available, use Python 3. This allows to test running configure with Python 3. PR-URL: #29236 Reviewed-By: Christian Clauss <[email protected]> Reviewed-By: Rich Trott <[email protected]>

When looking for Python in the registry, as specified in PEP514, this was not able to handle installations in a path with spaces, like Program Files. This ensures the whole path is used, fixing the issue. PR-URL: #29236 Reviewed-By: Christian Clauss <[email protected]> Reviewed-By: Rich Trott <[email protected]>

If there is no Python 2 available, use Python 3. This allows to test running configure with Python 3. PR-URL: #29236 Reviewed-By: Christian Clauss <[email protected]> Reviewed-By: Rich Trott <[email protected]>

When looking for Python in the registry, as specified in PEP514, this was not able to handle installations in a path with spaces, like Program Files. This ensures the whole path is used, fixing the issue. PR-URL: nodejs#29236 Reviewed-By: Christian Clauss <[email protected]> Reviewed-By: Rich Trott <[email protected]>

If there is no Python 2 available, use Python 3. This allows to test running configure with Python 3. PR-URL: nodejs#29236 Reviewed-By: Christian Clauss <[email protected]> Reviewed-By: Rich Trott <[email protected]>

joaocgreis added 2 commits August 20, 2019 22:02

build,win: find Python in paths with spaces

f0b1a6e

When looking for Python in the registry, as specified in PEP514, this was not able to handle installations in a path with spaces, like Program Files. This ensures the whole path is used, fixing the issue.

build,win: accept Python 3 if 2 is not available

acab2ec

If there is no Python 2 available, use Python 3. This allows to test running configure with Python 3.

joaocgreis added windows Issues and PRs related to the Windows platform. build Issues and PRs related to build files or the CI. python PRs and issues that require attention from people who are familiar with Python. labels Aug 20, 2019

nodejs-github-bot added install Issues and PRs related to the installers. tools Issues and PRs related to the tools directory. labels Aug 20, 2019

joaocgreis removed the install Issues and PRs related to the installers. label Aug 20, 2019

This comment has been minimized.

Sign in to view

joaocgreis mentioned this pull request Aug 20, 2019

build: find Python 3 or Python 2 in configure #25878

Closed

4 tasks

This comment has been minimized.

Sign in to view

cclauss mentioned this pull request Aug 26, 2019

build: ongoing list of actions for Python 3 compatibility #25789

Closed

63 tasks

cclauss approved these changes Aug 27, 2019

View reviewed changes

cclauss mentioned this pull request Aug 29, 2019

tools: from io import StringIO in ninja.py #29371

Closed

4 tasks

cclauss requested review from gengjiawen, bnoordhuis, richardlau and Trott August 30, 2019 04:02

Trott added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Aug 31, 2019

Trott reviewed Aug 31, 2019

View reviewed changes

Trott approved these changes Aug 31, 2019

View reviewed changes

Trott closed this Sep 3, 2019

BridgeAR mentioned this pull request Sep 3, 2019

v12.10.0 proposal #29429

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Accept Python 3 on Windows #29236

Accept Python 3 on Windows #29236

joaocgreis commented Aug 20, 2019

This comment has been minimized.

nodejs-github-bot commented Aug 20, 2019 •

edited by joaocgreis

Loading

cclauss commented Aug 21, 2019

joaocgreis commented Aug 21, 2019

Trott commented Aug 23, 2019

cclauss commented Aug 23, 2019 •

edited

Loading

This comment has been minimized.

cclauss commented Aug 25, 2019

cclauss left a comment

joaocgreis commented Aug 29, 2019

cclauss commented Aug 29, 2019

bnoordhuis commented Aug 30, 2019

Trott commented Aug 31, 2019

Trott Aug 31, 2019 •

edited

Loading

Trott left a comment

Trott commented Sep 3, 2019

joaocgreis commented Sep 3, 2019

	echo Python 3 is not yet fully supported, to avoid issues Python 2 should be installed.
	echo Python 3 is not yet fully supported. To avoid issues, Python 2 should be installed.

Accept Python 3 on Windows #29236

Accept Python 3 on Windows #29236

Conversation

joaocgreis commented Aug 20, 2019

Checklist

This comment has been minimized.

nodejs-github-bot commented Aug 20, 2019 • edited by joaocgreis Loading

cclauss commented Aug 21, 2019

joaocgreis commented Aug 21, 2019

Trott commented Aug 23, 2019

cclauss commented Aug 23, 2019 • edited Loading

This comment has been minimized.

cclauss commented Aug 25, 2019

cclauss left a comment

Choose a reason for hiding this comment

joaocgreis commented Aug 29, 2019

cclauss commented Aug 29, 2019

bnoordhuis commented Aug 30, 2019

Trott commented Aug 31, 2019

Trott Aug 31, 2019 • edited Loading

Choose a reason for hiding this comment

Trott left a comment

Choose a reason for hiding this comment

Trott commented Sep 3, 2019

joaocgreis commented Sep 3, 2019

nodejs-github-bot commented Aug 20, 2019 •

edited by joaocgreis

Loading

cclauss commented Aug 23, 2019 •

edited

Loading

Trott Aug 31, 2019 •

edited

Loading