Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update setup.py packaging before dropping Python 3.4 support #281

Closed
nitzmahone opened this issue Mar 20, 2019 · 7 comments
Closed

Update setup.py packaging before dropping Python 3.4 support #281

nitzmahone opened this issue Mar 20, 2019 · 7 comments

Comments

@nitzmahone
Copy link
Member

setup.py needs to be updated to a newer technology than distutils before we drop support for another Python version. setuptools should be the "easiest" to leave most of the existing custom code in place to support the extension build and integrated testing, etc, but there are other options that I'm sure @webknjaz would love to talk about ;).

We properly updated setup.py's metadata when dropping Python 2.6 and 3.3 support, however, our setup.py is still based on distutils, which doesn't understand python_requires (and prevents pypi from extracting the necessary metadata to ensure that unsupported Pythons will fall back to the last release that did support them). So today, anyone that's doing pip install pyyaml with an unsupported Python will install the latest release, but it will bomb out at install-time with something like PyYAML requires Python '>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*' but the running Python is 2.6.9 instead of installing a prior release that did support it.

Since we've released one that way, the fallback to 3.13 will never work again unless all the intermediate releases were pulled (uh, no ;) ). So either we say "tough noogies" or we release a single 5.x version with Python 2.6 and 3.3 support listed (on a packaging framework that supports the right metadata) to fix the glitch.

@nitzmahone
Copy link
Member Author

also related: #210 (comment)

@webknjaz
Copy link

Setuptools is fine here, among alternatives I'd say Poetry looks nice, but when I checked it out it was lacking one feature which I personally wanted (grabbing a version from git tag). Yet, it gets better.

I think if pyyaml was to cover the most popular env cases with wheels it wouldn't matter a lot how setup.py is constructed because it would almost never be invoked given that users use modern enough Pip, which they should. Python 2.7 is shipped with up-to-date packaging tools so it should be fine to require them: https://github.com/python/cpython/tree/v2.7.16/Lib/ensurepip/_bundled.

The solution should start with pyproject.toml which would be a proper entry point for building things and setuptools is 100% compatible with it. It also provides us with a way to have easily replaceable build backends without ever affecting end-users.

Also, looking at the current setup.py, I'd say that we should move static stuff to setup.cfg and rip off test command which is widely discouraged in favor of the proper test runners.

@webknjaz
Copy link

Since we've released one that way, the fallback to 3.13 will never work

.post0 release can always fix this...

@MRigal MRigal mentioned this issue May 2, 2019
@4383
Copy link

4383 commented May 11, 2019

@webknjaz some PR are currently under development to allow poetry to retrieve project version from git tag, I also propose my own version of this feature, but I think this kind of feature is not for now.

Else another alternative can be to use pbr, pbr is really straightforward to use and already have a lot of cool features (version, changelog, authors, requirements, etc...)

@webknjaz
Copy link

pbr is horrible. Please don't suggest it. I consider it a bug itself. Poetry is still dumb and PyPA members whom I trust don't like how it's done, so I'd not rush into using it. I only recommend what I've battle-tested myself.

Totktonada added a commit to tarantool/tarantool-python that referenced this issue Aug 21, 2020
`pip install pyyaml` fails in AppVeyor on 64-bit Python 3.4 with the
following message (full log is in [1]):

 | Collecting pyyaml
 |   Downloading <...>/PyYAML-5.3.1.tar.gz (269kB)
 | <...>
 | building '_yaml' extension
 | Traceback (most recent call last):
 |   <...>
 |   File "C:\Python34-x64\lib\distutils\msvc9compiler.py", line 287, in query_vcvarsall
 |     raise ValueError(str(list(result.keys())))
 | ValueError: ['path']

It is cumulative effect of several problems.

The first one is that pip does not choose a PyYAML version, which
supports given Python version (it is PyYAML 5.2, see [2]), but tries to
install a last one (it is PyYAML 5.3.1 ATM). There are PyYAML issues
[3], [4] and [5], which are not exacly about this, but looks tightly
related. Despite Python 3.4 is not officially supported in PyYAML, it
works on 3.4 in fact, so it is not the problem per se.

Next, there are no wheels for Python 3.4 for PyYAML 5.3.1 on PyPI (which
is logical), so pip downloads the source tarball and tries to build it.
And the exception shown above is raised.

It looks similar to [6], so I would guess that it is a compiler setup
problem within an AppVeyor Windows image. I started the discussion [7]
regarding it, but we need to work the problem around.

Side note: 32-bit Python 3.4 works fine, the problem appears only for
the 64-bit Python 3.4 build.

In order to workaround the compiler setup problem we can just choose a
PyYAML version for which PyPI provides wheels for the corresponding
platform / Python version / Python build. It is 5.2b1, see [8].

Python 3.4 reaches EOL 2019-03-18 with releasing 3.4.10 (see [9]). I
would not consider it as the reason to drop support of it on our side,
especially when we're tell about a library (our users has its users,
etc) and it is hard to estimate the impact. The problem we meet here is
just about a testing dependency in a particular environment. So let's
keep Python 3.4 support until there will be a better reason to stop.

[1]: https://ci.appveyor.com/project/tarantool/tarantool-python/builds/34769695/job/ww3s69pkuh1n9d1y
[2]: yaml/pyyaml#345
[3]: yaml/pyyaml#281
[4]: yaml/pyyaml#375
[5]: yaml/pyyaml#398
[6]: https://stackoverflow.com/a/43711372/1598057
[7]: http://help.appveyor.com/discussions/problems/28023
[8]: https://pypi.org/project/PyYAML/5.2b1/#files
[9]: https://www.python.org/downloads/release/python-3410/
Totktonada added a commit to tarantool/tarantool-python that referenced this issue Aug 21, 2020
`pip install pyyaml` fails in AppVeyor on 64-bit Python 3.4 with the
following message (full log is in [1]):

 | Collecting pyyaml
 |   Downloading <...>/PyYAML-5.3.1.tar.gz (269kB)
 | <...>
 | building '_yaml' extension
 | Traceback (most recent call last):
 |   <...>
 |   File "C:\Python34-x64\lib\distutils\msvc9compiler.py", line 287, in query_vcvarsall
 |     raise ValueError(str(list(result.keys())))
 | ValueError: ['path']

It is cumulative effect of several problems.

The first one is that pip does not choose a PyYAML version, which
supports given Python version (it is PyYAML 5.2, see [2]), but tries to
install a last one (it is PyYAML 5.3.1 ATM). There are PyYAML issues
[3], [4] and [5], which are not exacly about this, but looks tightly
related. Despite Python 3.4 is not officially supported in PyYAML, it
works on 3.4 in fact, so it is not the problem per se.

Next, there are no wheels for Python 3.4 for PyYAML 5.3.1 on PyPI (which
is logical), so pip downloads the source tarball and tries to build it.
And the exception shown above is raised.

It looks similar to [6], so I would guess that it is a compiler setup
problem within an AppVeyor Windows image. I started the discussion [7]
regarding it, but we need to work the problem around.

Side note: 32-bit Python 3.4 works fine, the problem appears only for
the 64-bit Python 3.4 build.

In order to workaround the compiler setup problem we can just choose a
PyYAML version for which PyPI provides wheels for the corresponding
platform / Python version / Python build. It is 5.2b1, see [8].

Python 3.4 reaches EOL 2019-03-18 with releasing 3.4.10 (see [9]). I
would not consider it as the reason to drop support of it on our side,
especially when we're tell about a library (our users has its users,
etc) and it is hard to estimate the impact. The problem we meet here is
just about a testing dependency in a particular environment. So let's
keep Python 3.4 support until there will be a better reason to stop.

[1]: https://ci.appveyor.com/project/tarantool/tarantool-python/builds/34769695/job/ww3s69pkuh1n9d1y
[2]: yaml/pyyaml#345
[3]: yaml/pyyaml#281
[4]: yaml/pyyaml#375
[5]: yaml/pyyaml#398
[6]: https://stackoverflow.com/a/43711372/1598057
[7]: http://help.appveyor.com/discussions/problems/28023
[8]: https://pypi.org/project/PyYAML/5.2b1/#files
[9]: https://www.python.org/downloads/release/python-3410/
Totktonada added a commit to tarantool/tarantool-python that referenced this issue Aug 21, 2020
`pip install pyyaml` fails in AppVeyor on 64-bit Python 3.4 with the
following message (full log is in [1]):

 | Collecting pyyaml
 |   Downloading <...>/PyYAML-5.3.1.tar.gz (269kB)
 | <...>
 | building '_yaml' extension
 | Traceback (most recent call last):
 |   <...>
 |   File "C:\Python34-x64\lib\distutils\msvc9compiler.py", line 287, in query_vcvarsall
 |     raise ValueError(str(list(result.keys())))
 | ValueError: ['path']

It is cumulative effect of several problems.

The first one is that pip does not choose a PyYAML version, which
supports given Python version (it is PyYAML 5.2, see [2]), but tries to
install a last one (it is PyYAML 5.3.1 ATM). There are PyYAML issues
[3], [4] and [5], which are not exacly about this, but looks tightly
related. Despite Python 3.4 is not officially supported in PyYAML, it
works on 3.4 in fact, so it is not the problem per se.

Next, there are no wheels for Python 3.4 for PyYAML 5.3.1 on PyPI (which
is logical), so pip downloads the source tarball and tries to build it.
And the exception shown above is raised.

It looks similar to [6], so I would guess that it is a compiler setup
problem within an AppVeyor Windows image. I started the discussion [7]
regarding it, but we need to work the problem around.

Side note: 32-bit Python 3.4 works fine, the problem appears only for
the 64-bit Python 3.4 build.

In order to workaround the compiler setup problem we can just choose a
PyYAML version for which PyPI provides wheels for the corresponding
platform / Python version / Python build. It is 5.2b1, see [8].

Python 3.4 reaches EOL 2019-03-18 with releasing 3.4.10 (see [9]). I
would not consider it as the reason to drop support of it on our side,
especially when we're tell about a library (our users has its users,
etc) and it is hard to estimate the impact. The problem we meet here is
just about a testing dependency in a particular environment. So let's
keep Python 3.4 support until there will be a better reason to stop.

[1]: https://ci.appveyor.com/project/tarantool/tarantool-python/builds/34769695/job/ww3s69pkuh1n9d1y
[2]: yaml/pyyaml#345
[3]: yaml/pyyaml#281
[4]: yaml/pyyaml#375
[5]: yaml/pyyaml#398
[6]: https://stackoverflow.com/a/43711372/1598057
[7]: http://help.appveyor.com/discussions/problems/28023
[8]: https://pypi.org/project/PyYAML/5.2b1/#files
[9]: https://www.python.org/downloads/release/python-3410/
@mcepl
Copy link

mcepl commented Nov 23, 2021

Also, could you please eliminate setup.py test and run tests in some more standardized way (python3 -munittest discover or pytest), please?

@nitzmahone
Copy link
Member Author

Funny you mention it, I'd created a prototype pytest wrapper awhile back to surface pyyaml's "special" test suite as individual pytest tests- dusted it off and PR'd it at #588.

I think all the points of the original issue have been addressed in the past couple releases, so I'm closing...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants