Consider dropping Python 2 support

[JustAnotherArchivist]

It's been almost two years since Python 2 has reached end-of-life. Some things already broke, and this will only get worse over time. Also, the code is significantly more complex (everything involving six; see also my struggles on #405).

I suggest dropping Python 2 support and removing the accompanying code cruft.

@jjjake has previously said in #405 that he 'would like to support [Python 2] as long as we're able to'. As such, I don't expect this to be acted on immediately. But I think it is inevitable as dependencies have begun to drop Py2 support, slowly making it harder to keep the support here.

Relevant issues so far (in no particular order):

• ItemLocate error causes crash under Python 2: ItemLocateError broken on Python 2 #385
• contextlib2, an upstream dependency of schema, no longer supports Python 2; by extension, the self-contained binary is now broken: SyntaxError: Non-ASCII character '\xc3' in file ./ia on line 3 #420 (comment)
• ia tasks on Python 2 can't work reliably (without API changes) due to the lack of appropriate decoding error handlers: UnicodeDecodeError when trying to display task logs with invalid UTF-8 #359 and Fix crash in ia tasks when a task log contains invalid UTF-8 #360

As of right now, the contextlib2 development breaks new ia installations entirely on Python 2 unless you manually install an older version of that package. It could be pinned, but that's just delaying the inevitable.

Further things to look forward to:

• pip has removed Python 2 support as of version 21.0: Drop Support for Python 2 in pip 21.0 pypa/pip#6148 Drop support for Python 2.7 and Python 3.5 pypa/pip#9189
• Requests maintainers have previously stated that they will drop support once pip does: Clarification of timeline for removing Python 2.7 support psf/requests#5211

[maxz]

Since many of the libraries that we rely on will soon be dropping their Python 2 support, I'm generally in favour of also dropping the Python 2 support for internetarchive.

In the past I've had to deal with environments where I could only easily run Python 2, since running Python 3 would have required that I cross compile it myself and somehow get it to permanently reside in the memory of those rather special systems, so there are certain use cases I see for which it would be good to keep Python 2 support.

But the only realistic option to keep the Python 2 support would be to pin the latest Python 2 supporting versions of all our dependencies, which is pretty bad from a security point of view.
Backporting the newer versions of all our dependencies to Python 2 is just not a realistic option.

Therefore it is most likely the best option to just drop the Python 2 support.

I don't see any reason to do so right away, but as soon as the first major dependency drops the support, we pretty much also have to do so. Maybe we should prepare for that time by stating in all important places, that the Python 2 support will be dropped to give the users time to prepare (and try to find out the rough time when this will take place).

[JustAnotherArchivist]

urllib3, upstream dependency of requests, has dropped Python 2 support, though this isn't released yet: urllib3/urllib3#2044

[maxz]

They additionally dropped the Python 3.5 support in that pull request, which we should also watch.

[cclauss]

👍 Next week it is 700 days since Python 2 went EOL. Let's cut a final release that supports legacy Python and then move forward.

[jjjake]

Ok. Thanks to everyone for pushing on this, and thanks for your patience. It's been a long time coming! I think it's time, let's start talking about what a final PY2 release would look like (any outstanding bugs in PY2 we should try and deal with before shipping the last PY2 version, etc.), and go from there. Thanks again!

[JustAnotherArchivist]

Some thoughts in no particular order:

• The only direct and fixable Py2 bug in the actual code I'm aware of is ItemLocateError broken on Python 2 #385, which isn't a major issue in my opinion. Would be nice to fix, sure, but not convinced it deserves an extra release.
• UnicodeDecodeError when trying to display task logs with invalid UTF-8 #359 can't be fixed in Py2 without unreasonable effort. Fix crash in ia tasks when a task log contains invalid UTF-8 #360 uses errors='surrogateescape' even in the Py2 code, but that's not an really issue because the handler is only checked when something blows up anyway. No action needed here, I think.
• contextlib2 is actually alright, sort of. As long as users have a modern enough version of pip, they should automatically get the older version of the package when installing under Py2.
• The test suite dependencies are broken on Py2. I fixed this and will send a PR once Synchronise tested Python versions in tox.ini and .github/workflows/tox.yml #450 is merged (since it depends on that). But since that's in the test suite, no release needed for when it's fixed as the test code isn't shipped anyway.
• SyntaxError: Non-ASCII character '\xc3' in file ./ia on line 3 #420 is caused by how the pex is built rather than the Python code/package itself, so that doesn't require code changes I think, just possibly some updates to pex-requirements.txt (e.g. pinning contextlib2 despite not being a direct dependency).
• Otherwise, as far as I'm aware, it has mostly been a case of making code development harder due to having to constantly come up with Python 2 workarounds.

So overall, I can't really think of any particular significant things that should be fixed for a final Py2-compatible release. However, I also never used ia on Py2 myself, so I may be missing something.

[JustAnotherArchivist]

Considering the above and the recent release of version 2.3.0, can we drop Py2 yet? :-)

[jjjake]

@JustAnotherArchivist Yeah, I think we should move ahead with dropping PY2 support! Thanks for taking the time to consider all of the above, and following up on it.

[JustAnotherArchivist]

Yay! I'll work on a PR to remove the cruft.