Ensure "pip install datasette" still works with Python 3.6

[simonw]

Original title: Can I keep "pip install datasette" working on Python 3.6?

I dropped support for 3.6 in:

• Drop support for Python 3.6 #1577

I'm getting reports that pip3 install datasette throws an error on that Python, even though I haven't made that new release yet - presumably due to lack of pinning of Uvicorn: https://twitter.com/ldodds/status/1484289475195080706

Is it possible to get pip on that version of Python to install the highest possible version of the packages that are still known to support Python 3.6?

If so, how?

[simonw]

On Twitter: https://twitter.com/simonw/status/1484317711672877065

Here's the problem: Uvicorn only added python_requires to their setup.py a few days ago, which means the releases they have out on PyPI at the moment don't specify the Python version they need, which is why this mechanism doesn't work as expected:

• Add the python_requires version specifier encode/uvicorn#1328

[simonw]

I think there are two possible solutions then:

1. Convince Uvicorn to publish one last 0.16.1 version which includes that python_requires= line, such that there's a version of Uvicorn on PyPI that Python 3.6 can still install.
2. Release a 0.60.1 version of Datasette which pins that Uvicorn version, and hence can be installed.

I've made the request for 1) in Uvicorn Gitter here: https://gitter.im/encode/community?at=61ea044a6d9ba23328d0fa28

I'm going to investigate option 2) myself now.

[simonw]

I used the combo of pyenv and pipenv to run tests and figure out what the most recent versions of each dependency were that worked on Python 3.6. I also clicked around in the latest releases on pages such as https://pypi.org/project/aiofiles

cd /tmp
git clone [email protected]:simonw/datasette
cd /tmp/datasette
pipenv shell --python 3.6.10
pip install -e '.[test]'
pytest

I also used pip freeze | grep black to see which version was installed, since packages with python_requires= in them would automatically install the highest compatible version.

[simonw]

Problem: if I ship this, it will be the most recent release of Datasette - but unlike other previous releases it has exactly pinned versions of all of the dependencies. Which is bad for people who run pip install datasette but want to not be stuck to those exact library versions.

So maybe I ship this as 0.60.1, then ship a 0.60.2 release directly afterwards which unpins the dependencies again and requires Python 3.7?

[simonw]

Another option from https://twitter.com/samuel_hames/status/1484327636860293121 - environment markers, described in https://www.python.org/dev/peps/pep-0508/#environment-markers

Found some examples of those in use using GitHub code search: https://cs.github.com/?scopeName=All+repos&scope=&q=%22%3Bpython_version%22+path%3Asetup.py - in particular https://github.com/xmendez/wfuzz/blob/1b695ee9a87d66a7d7bf6cae70d60a33fae51541/setup.py#L31-L38

install_requires = [
    'pycurl',
    'pyparsing<2.4.2;python_version<="3.4"',
    'pyparsing>=2.4*;python_version>="3.5"',
    'six',
    'configparser;python_version<"3.5"',
    'chardet',
]

So maybe I can ship 0.60.1 with loose dependencies except for the uvicorn one on Python 3.6, using an environment marker.

Here's my setup.py at the moment:

datasette/setup.py

Lines 44 to 61 in ffca55d

	install_requires=[
	"asgiref>=3.2.10,<3.5.0",
	"click>=7.1.1,<8.1.0",
	"click-default-group~=1.2.2",
	"Jinja2>=2.10.3,<3.1.0",
	"hupper~=1.9",
	"httpx>=0.20",
	"pint~=0.9",
	"pluggy>=1.0,<1.1",
	"uvicorn~=0.11",
	"aiofiles>=0.4,<0.9",
	"janus>=0.6.2,<1.1",
	"asgi-csrf>=0.9",
	"PyYAML>=5.3,<7.0",
	"mergedeep>=1.1.1,<1.4.0",
	"itsdangerous>=1.1,<3.0",
	"python-baseconv==1.2.2",
	],

One other problem: there might be packages in that list right now which don't specify their 3.6 Python version but which will, at some point in the future, release a new version that doesn't work with 3.6 (like Uvicorn did) - in which case Python 3.6 installs would break in the future.

I think what I'll do then is ship the 0.60.1 Python 3.6 version with strict upper limits on each version which are the current, tested-with-Datasette-on-Python3.6 versions.

[simonw]

This whole thing reminds me of my ongoing internal debate about version pinning: should the Datasette package released to PyPI pin to the exact versions of the dependencies that are known to work, or should it allow a range of dependencies so users can pick other versions of the dependencies to use in their environment?

As I understand it, the general rule is to use exact pinning for applications but use ranges for libraries.

Datasette is almost entirely an application... but it can also be used as a library - and in fact I'm hoping to encourage that usage more in the future, see:

• Documentation on using Datasette as a library #1398

I'd also like to release a packaged version of Datasette that doesn't require Uvicorn, for running on AWS Lambda and other function-as-a-service platforms. Those platforms have their own HTTP layer and hence don't need the Uvicorn dependency.

Maybe the answer is to have a datasette-core package which provides the core of Datasette with unpinned dependencies and no Uvicorn, and then have the existing datasette package provide the Datasette CLI tool with Uvicorn and pinned dependencies?

[simonw]

https://github.com/simonw/datasette/runs/4890775227?check_suite_focus=true - the tests passed on Python 3.6 for this commit with the pinned dependencies: 41060e7

[simonw]

OK, the environment markers approach seems to work!

[simonw]

Out of curiosity, I installed this latest setup.py file using both Python 3.6 and Python 3.10, ran pip freeze on both of them and created a Gist to compare the difference. The result is here: https://gist.github.com/simonw/2e7d5b1beba675ef9a5bcd310cadc372/revisions

From that, it looks like the Python packages in my dependencies which have released new versions that don't work with Python 3.6 are:

• https://pypi.org/project/janus/#history
• https://pypi.org/project/Pint/#history
• https://pypi.org/project/platformdirs/#history
• https://pypi.org/project/uvicorn/#history (already discussed)

Sure enough, for the first three of those browsing through their recent versions on PyPI confirms that they switched from e.g. "Requires: Python >=3.6" on https://pypi.org/project/janus/0.7.0/ to "Requires: Python >=3.7" on https://pypi.org/project/janus/1.0.0/

[simonw]

So I don't need to release 0.60.1 AND 0.60.2 after all - I can just release 0.60.1 with a bug fix that it no longer breaks installation for Python 3.6.

[simonw]

Just shipped 0.60.1 with the fix - and tested that pip install datasette does indeed work correctly on Python 3.6.

[simonw]

Uvicorn have a release out now that would have fixed this issue if I hadn't shipped my own fix: https://github.com/encode/uvicorn/releases/tag/0.17.0.post1