Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to build vmprof on Windows with Python 3.5.2 #92

Closed
debnet opened this issue Jul 23, 2016 · 9 comments
Closed

Unable to build vmprof on Windows with Python 3.5.2 #92

debnet opened this issue Jul 23, 2016 · 9 comments

Comments

@debnet
Copy link

debnet commented Jul 23, 2016

Hello people.

Linkage failed when building vmprof on Windows with Python 3.5.2 x86-64 and Visual Studio Community 2015 with Python support.

Here the log:

C:\Users\Marc>pip install vmprof
Collecting vmprof
  Using cached vmprof-0.2.8.tar.gz
Requirement already satisfied (use --upgrade to upgrade): requests in c:\python35\lib\site-packages (from vmprof)
Installing collected packages: vmprof
  Running setup.py install for vmprof ... error
    Complete output from command c:\python35\python.exe -u -c "import setuptools, tokenize;__file__='D:\\Temp\\Users\\Marc\\pip-build-d2p21sz2\\vmprof\\setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record D:\Temp\Users\Marc\pip-dtex470n-record\install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    creating build
    creating build\lib.win-amd64-3.5
    creating build\lib.win-amd64-3.5\tests
    copying tests\cpuburn.py -> build\lib.win-amd64-3.5\tests
    copying tests\test_config.py -> build\lib.win-amd64-3.5\tests
    copying tests\test_reader.py -> build\lib.win-amd64-3.5\tests
    copying tests\test_run.py -> build\lib.win-amd64-3.5\tests
    copying tests\test_stats.py -> build\lib.win-amd64-3.5\tests
    copying tests\__init__.py -> build\lib.win-amd64-3.5\tests
    creating build\lib.win-amd64-3.5\vmprof
    copying vmprof\binary.py -> build\lib.win-amd64-3.5\vmprof
    copying vmprof\cli.py -> build\lib.win-amd64-3.5\vmprof
    copying vmprof\profiler.py -> build\lib.win-amd64-3.5\vmprof
    copying vmprof\reader.py -> build\lib.win-amd64-3.5\vmprof
    copying vmprof\show.py -> build\lib.win-amd64-3.5\vmprof
    copying vmprof\stats.py -> build\lib.win-amd64-3.5\vmprof
    copying vmprof\upload.py -> build\lib.win-amd64-3.5\vmprof
    copying vmprof\vmprofdemo.py -> build\lib.win-amd64-3.5\vmprof
    copying vmprof\__init__.py -> build\lib.win-amd64-3.5\vmprof
    copying vmprof\__main__.py -> build\lib.win-amd64-3.5\vmprof
    creating build\lib.win-amd64-3.5\vmprof\log
    copying vmprof\log\constants.py -> build\lib.win-amd64-3.5\vmprof\log
    copying vmprof\log\marks.py -> build\lib.win-amd64-3.5\vmprof\log
    copying vmprof\log\merge_point.py -> build\lib.win-amd64-3.5\vmprof\log
    copying vmprof\log\objects.py -> build\lib.win-amd64-3.5\vmprof\log
    copying vmprof\log\parser.py -> build\lib.win-amd64-3.5\vmprof\log
    copying vmprof\log\__init__.py -> build\lib.win-amd64-3.5\vmprof\log
    running egg_info
    writing requirements to vmprof.egg-info\requires.txt
    writing top-level names to vmprof.egg-info\top_level.txt
    writing vmprof.egg-info\PKG-INFO
    writing entry points to vmprof.egg-info\entry_points.txt
    writing dependency_links to vmprof.egg-info\dependency_links.txt
    warning: manifest_maker: standard file '-c' not found

    reading manifest file 'vmprof.egg-info\SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file 'vmprof.egg-info\SOURCES.txt'
    running build_ext
    building '_vmprof' extension
    creating build\temp.win-amd64-3.5
    creating build\temp.win-amd64-3.5\Release
    creating build\temp.win-amd64-3.5\Release\src
    C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Ic:\python35\include -Ic:\python35\include "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\winrt" /Tcsrc/_vmprof.c /Fobuild\temp.win-amd64-3.5\Release\src/_vmprof.obj
    _vmprof.c
    d:\temp\users\marc\pip-build-d2p21sz2\vmprof\src\vmprof_common.h(67): warning C4311: 'cast de type' : troncation de pointeur de 'PyCodeObject *' à 'unsigned long'
    d:\temp\users\marc\pip-build-d2p21sz2\vmprof\src\vmprof_common.h(67): warning C4312: 'cast de type' : la conversion de 'unsigned long' en 'void *' d'une taille supérieure
    d:\temp\users\marc\pip-build-d2p21sz2\vmprof\src\vmprof_common.h(96): warning C4267: '=' : conversion de 'size_t' en 'char', perte possible de données
    d:\temp\users\marc\pip-build-d2p21sz2\vmprof\src\vmprof_main_win32.h(31): warning C4267: 'fonction' : conversion de 'size_t' en 'unsigned int', perte possible de données
    d:\temp\users\marc\pip-build-d2p21sz2\vmprof\src\vmprof_main_win32.h(48): warning C4267: 'initialisation' : conversion de 'size_t' en 'int', perte possible de données
    d:\temp\users\marc\pip-build-d2p21sz2\vmprof\src\vmprof_main_win32.h(72): warning C4312: 'cast de type' : la conversion de 'DWORD' en 'void *' d'une taille supérieure
    src/_vmprof.c(42): warning C4311: 'cast de type' : troncation de pointeur de 'PyCodeObject *' à 'unsigned long'
    src/_vmprof.c(42): warning C4312: 'cast de type' : la conversion de 'unsigned long' en 'void *' d'une taille supérieure
    src/_vmprof.c(69): warning C4311: 'cast de type' : troncation de pointeur de 'PyCodeObject *' à 'unsigned long'
    C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\amd64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:c:\python35\libs /LIBPATH:c:\python35\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\LIB\amd64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\LIB\amd64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.10586.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.10586.0\um\x64" /EXPORT:PyInit__vmprof build\temp.win-amd64-3.5\Release\src/_vmprof.obj /OUT:build\lib.win-amd64-3.5\_vmprof.cp35-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.5\Release\src\_vmprof.cp35-win_amd64.lib
    _vmprof.obj : warning LNK4197: exportation 'PyInit__vmprof' spÚcifiÚe Ó plusieurs reprisesá; premiÞre spÚcification utilisÚe
       CrÚation de la bibliothÞque build\temp.win-amd64-3.5\Release\src\_vmprof.cp35-win_amd64.lib et de l'objet build\temp.win-amd64-3.5\Release\src\_vmprof.cp35-win_amd64.exp
    _vmprof.obj : error LNK2001: symbole externe non rÚsolu _PyThreadState_Current
    build\lib.win-amd64-3.5\_vmprof.cp35-win_amd64.pyd : fatal error LNK1120: 1 externes non rÚsolus
    error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\BIN\\amd64\\link.exe' failed with exit status 1120

    ----------------------------------------
Command "c:\python35\python.exe -u -c "import setuptools, tokenize;__file__='D:\\Temp\\Users\\Marc\\pip-build-d2p21sz2\\vmprof\\setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record D:\Temp\Users\Marc\pip-dtex470n-record\install-record.txt --single-version-externally-managed --compile" failed with error code 1 in D:\Temp\Users\Marc\pip-build-d2p21sz2\vmprof\

Thank you. ;)
@fijal
Copy link
Member

fijal commented Jul 25, 2016

vmprof is not working on win64 (as a 64bit application, it works fine under 32bit python). It might be a very shallow problem, but it was never ported nor tested, we should make it more clear on the website somewhere

@debnet
Copy link
Author

debnet commented Jul 25, 2016

When do you plan to port vmprof to x86-64 arch in your roadmap? :)

@trin5tensa trin5tensa mentioned this issue Aug 23, 2016
Closed
@trin5tensa
Copy link

trin5tensa commented Aug 23, 2016 •
edited
Loading

vmprof support has been added to the latest release of PyCharm 2016.2.1 and there are likely a lot of other PyCharm users on Windows who are having exactly this problem.
Is it possible to direct pip to compile a 32 bit executable which can be run on a Windows 64 bit machine?

@fijal
Copy link
Member

fijal commented Aug 23, 2016

I'm not a windows expert, but I would think by having a 32bit python you would do that

@trin5tensa
Copy link

trin5tensa commented Aug 23, 2016 •
edited
Loading

If I understand you correctly, you are saying:

  1. The compiler is deciding to create a 64 bit or a 32 bit executable based on the version of Python that is installed on the target machine and not on the fact the machine has a 64 bit architecture.
  2. If the python version happens to be the Intel 64 architecture version the attempt to install vmprof will fail.
  3. Your proposed solution is to install the Windows x86 version of Python in place of the Windows x86-64 version.

It seems likely that the Python devs. have gone to the trouble of offering a special version for Intel 64 architecture because they think there is some advantage in having this separate version. Your proposed solution asks the user to throw this advantage away. May I respectfully suggest that the people who want the faster 64 bit version of Python are interested in execution speed and so would be very interested in an installable code profiler.

Is this related to #85?

@fijal
Copy link
Member

fijal commented Aug 24, 2016

Yes, your understanding is correct.

As far as python-dev is concerned, it's legit to install both versions. The difference is very simple - the x86_64 architecture lets you address more memory (so your programs can have a heap bigger than 4G) at a cost of larger pointers (which means your memory usage grows by a factor of up to 2, depending on your use case).

Generally speaking, x86_64 should be faster with new registers and whatnot but in practice it never is because of the cache impact of larger pointers and clever tricks that let the processor optimize 32bit programs.

@trin5tensa
Copy link

trin5tensa commented Aug 28, 2016 •
edited
Loading

I was able to install 32 bit version of python on my Windows computer without affecting the 64 bit version.

Sadly, your solution did not work and the installation of vmprof in the 32 bit environment failed with exactly the same error as before. A complete log follows.

It is worth looking at the paths employed during the installation process. Sometimes it is working in the 32 bit program folder (e.g. D:\Program Files (x86)\Microsoft Visual Studio 14.0\...) and sometimes in the 64 bit program folder (e.g. Complete output from command "d:\program files\python\python 3.5.2\python.exe") Incidentally, I have double checked to make sure that pip install was run from D:\Program Files (x86)\Python\Python 3.5.2\Scripts\pip.exe

Collecting vmprof
    Using cached vmprof-0.3.8.tar.gz
Collecting requests (from vmprof)
      Using cached requests-2.11.1-py2.py3-none-any.whl
Collecting six (from vmprof)
  Using cached six-1.10.0-py2.py3-none-any.whl
Installing collected packages: requests, six, vmprof
  Running setup.py install for vmprof: started
    Running setup.py install for vmprof: finished with status 'error'
    Complete output from command "d:\program files\python\python 3.5.2\python.exe" -u -c "import setuptools, tokenize;__file__='D:\\Users\\Stephen\\AppData\\Local\\Temp\\pip-build-hdsmh04o\\vmprof\\setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record D:\Users\Stephen\AppData\Local\Temp\pip-uo11qqn_-record\install-record.txt --single-version-externally-managed --compile:

running install
running build
running build_py
creating build
creating build\lib.win-amd64-3.5
creating build\lib.win-amd64-3.5\jitlog
copying jitlog\constants.py -> build\lib.win-amd64-3.5\jitlog
copying jitlog\marks.py -> build\lib.win-amd64-3.5\jitlog
copying jitlog\merge_point.py -> build\lib.win-amd64-3.5\jitlog
copying jitlog\objects.py -> build\lib.win-amd64-3.5\jitlog
copying jitlog\parser.py -> build\lib.win-amd64-3.5\jitlog
copying jitlog\query.py -> build\lib.win-amd64-3.5\jitlog
copying jitlog\upload.py -> build\lib.win-amd64-3.5\jitlog
copying jitlog\__init__.py -> build\lib.win-amd64-3.5\jitlog
copying jitlog\__main__.py -> build\lib.win-amd64-3.5\jitlog
creating build\lib.win-amd64-3.5\vmprof
copying vmprof\cli.py -> build\lib.win-amd64-3.5\vmprof
copying vmprof\profiler.py -> build\lib.win-amd64-3.5\vmprof
copying vmprof\reader.py -> build\lib.win-amd64-3.5\vmprof
copying vmprof\show.py -> build\lib.win-amd64-3.5\vmprof
copying vmprof\stats.py -> build\lib.win-amd64-3.5\vmprof
copying vmprof\upload.py -> build\lib.win-amd64-3.5\vmprof
copying vmprof\__init__.py -> build\lib.win-amd64-3.5\vmprof
copying vmprof\__main__.py -> build\lib.win-amd64-3.5\vmprof
creating build\lib.win-amd64-3.5\vmshare
copying vmshare\binary.py -> build\lib.win-amd64-3.5\vmshare
copying vmshare\__init__.py -> build\lib.win-amd64-3.5\vmshare
running egg_info
writing dependency_links to vmprof.egg-info\dependency_links.txt
writing requirements to vmprof.egg-info\requires.txt
writing vmprof.egg-info\PKG-INFO
writing entry points to vmprof.egg-info\entry_points.txt
writing top-level names to vmprof.egg-info\top_level.txt
warning: manifest_maker: standard file '-c' not found

reading manifest file 'vmprof.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '*' found under directory 'vmprof\test'
warning: no previously-included files matching '*' found under directory 'jitlog\test'
writing manifest file 'vmprof.egg-info\SOURCES.txt'
running build_ext
building '_vmprof' extension
creating build\temp.win-amd64-3.5
creating build\temp.win-amd64-3.5\Release
creating build\temp.win-amd64-3.5\Release\src
D:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD "-Id:\program files\python\python 3.5.2\include" "-Id:\program files\python\python 3.5.2\include" "-ID:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" "-ID:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt" "-ID:\Program Files (x86)\Windows Kits\8.1\include\shared" "-ID:\Program Files (x86)\Windows Kits\8.1\include\um" "-ID:\Program Files (x86)\Windows Kits\8.1\include\winrt" /Tcsrc/_vmprof.c /Fobuild\temp.win-amd64-3.5\Release\src/_vmprof.obj
_vmprof.c
d:\users\stephen\appdata\local\temp\pip-build-hdsmh04o\vmprof\src\vmprof_common.h(97): warning C4312: 'type cast': conversion from 'long' to 'void *' of greater size
d:\users\stephen\appdata\local\temp\pip-build-hdsmh04o\vmprof\src\vmprof_common.h(103): warning C4311: 'type cast': pointer truncation from 'PyCodeObject *' to 'unsigned long'
d:\users\stephen\appdata\local\temp\pip-build-hdsmh04o\vmprof\src\vmprof_common.h(103): warning C4312: 'type cast': conversion from 'unsigned long' to 'void *' of greater size
d:\users\stephen\appdata\local\temp\pip-build-hdsmh04o\vmprof\src\vmprof_common.h(87): warning C4244: 'initializing': conversion from 'Py_ssize_t' to 'int', possible loss of data
d:\users\stephen\appdata\local\temp\pip-build-hdsmh04o\vmprof\src\vmprof_common.h(130): warning C4267: '=': conversion from 'size_t' to 'char', possible loss of data
d:\users\stephen\appdata\local\temp\pip-build-hdsmh04o\vmprof\src\vmprof_main_win32.h(31): warning C4267: 'function': conversion from 'size_t' to 'unsigned int', possible loss of data
d:\users\stephen\appdata\local\temp\pip-build-hdsmh04o\vmprof\src\vmprof_main_win32.h(48): warning C4267: 'initializing': conversion from 'size_t' to 'int', possible loss of data
d:\users\stephen\appdata\local\temp\pip-build-hdsmh04o\vmprof\src\vmprof_main_win32.h(72): warning C4312: 'type cast': conversion from 'DWORD' to 'void *' of greater size
src/_vmprof.c(42): warning C4311: 'type cast': pointer truncation from 'PyCodeObject *' to 'unsigned long'
src/_vmprof.c(42): warning C4312: 'type cast': conversion from 'unsigned long' to 'void *' of greater size
src/_vmprof.c(69): warning C4311: 'type cast': pointer truncation from 'PyCodeObject *' to 'unsigned long'
D:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO "/LIBPATH:d:\program files\python\python 3.5.2\libs" "/LIBPATH:d:\program files\python\python 3.5.2\PCbuild\amd64" "/LIBPATH:D:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\LIB\amd64" "/LIBPATH:D:\Program Files (x86)\Windows Kits\10\lib\10.0.10240.0\ucrt\x64" "/LIBPATH:D:\Program Files (x86)\Windows Kits\8.1\lib\winv6.3\um\x64" /EXPORT:PyInit__vmprof build\temp.win-amd64-3.5\Release\src/_vmprof.obj /OUT:build\lib.win-amd64-3.5\_vmprof.cp35-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.5\Release\src\_vmprof.cp35-win_amd64.lib
_vmprof.obj : warning LNK4197: export 'PyInit__vmprof' specified multiple times; using first specification
   Creating library build\temp.win-amd64-3.5\Release\src\_vmprof.cp35-win_amd64.lib and object build\temp.win-amd64-3.5\Release\src\_vmprof.cp35-win_amd64.exp
_vmprof.obj : error LNK2001: unresolved external symbol _PyThreadState_Current
build\lib.win-amd64-3.5\_vmprof.cp35-win_amd64.pyd : fatal error LNK1120: 1 unresolved externals
error: command 'D:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\BIN\\x86_amd64\\link.exe' failed with exit status 1120

----------------------------------------

@debnet
Copy link
Author

debnet commented Sep 15, 2016

Any improvements?

@trin5tensa
Copy link

@debnet. There is a pip issue #18 where Python Packaging Authority member pfmoore states,
"I suspect there won't be much change to what's currently available - the long-term goal is to make the build process independent of the install process, and in the meantime, the recommendation is really to build wheels. Either get the project to supply (Windows and/or OSX) wheels on PyPI, or if that's not practical, build wheels yourself locally (setup.py bdist_wheel should always be possible) and serve them from a local directory or a devpi instance."
Sadly. I haven't got time right now to find out how to build a local wheel for vmprof to see if it solves the problem.

@Kojoley Kojoley mentioned this issue Sep 24, 2016
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@fijal @debnet @trin5tensa