Potentially merge Python 2 and 3 stubs for many modules

[JukkaL]

We'd need to add aliases for string-like types such as str/bytes/unicode, and these would be different between Python 2 and 3. We'd need to decide where to define the type aliases, either in the stub of the module that uses them or in a shared typeshed-related (?) module that is only available as a stub.

[vlasovskikh]

Here is what we found useful in PyCharm (written using our old syntax):

string             # Py2: str | unicode, Py3: str
bytestring         # Py2: str | unicode, Py3: bytes
bytes              # Py2: str, Py3: bytes
unicode            # Py2: unicode, Py3: str

[matthiaskramm]

Sharing code between pyi files sounds even more useful for https://github.com/JukkaL/typeshed/issues/3
Imagine a package with a lot of files (like e.g. numpy), and only two or three change between versions.
In terms of storing a duplicate modules in the github repo, we could just use symlinks.

However, concerning type aliases, I'm not sure how often we'll see the case that the Python 2 and the Python 3 version off the stub are exactly the same, except one uses unicode and the other uses str.

[gvanrossum]

Once mypy (and other systems) support version checks as described in PEP 484 (https://www.python.org/dev/peps/pep-0484/#version-and-platform-checking) we should be able to unify many more stubs.

[srittau]

Here is a list of modules that a simple diff between stdlib/2 and stdlib/3 uncovered (please edit as appropriate):

• _ast.pyi
• _threading_local.pyi
• _warnings.pyi
• abc.pyi
• ast.pyi
• atexit.pyi
• builtins.pyi
• compileall.pyi
• email
• encodings
• fcntl.pyi
• fnmatch.pyi
• functools.pyi
• gc.pyi
• getopt.pyi
• getpass.pyi
• gzip.pyi
• hashlib.pyi
• heapq.pyi
• imp.pyi
• inspect.pyi
• io.pyi
• itertools.pyi
• _json.pyi
• macpath.pyi
• multiprocessing
• ntpath.pyi
• nturl2path.pyi
• os (see Merge os stubs into 2and3 #1427)
• pipes.pyi
• platform.pyi
• posixpath.pyi
• posix.pyi (see Fill in stdlib/3/posix.pyi #616)
• random.pyi
• re.pyi
• resource.pyi
• runpy.pyi
• shelve.pyi
• shlex.pyi
• signal.pyi
• smtplib.pyi
• spwd.pyi
• sre_constants.pyi
• sre_parse.pyi
• stat.pyi
• string.pyi
• subprocess.pyi
• symbol.pyi
• sys.pyi
• tempfile.pyi
• textwrap.pyi
• tokenize.pyi
• types.pyi
• typing.pyi

The packages (marked in italics) need special consideration, since they might contain submodules that don't exist in both versions.

[srittau]

Also json, where the stub is currently a module in stdlib/2, but a package in stdlib/3. In the standard library it is a package in both.

[hauntsaninja]

Can we close this?

In 2020, it seems like it could be unnecessary burden (and potentially cause errors in py2 stubs by py3-minded people) to merge py2 and py3 stubs using version checks. It's a different matter if the files are identical, but a quick command makes it seem like that's not really the case, see details.

Files which have the same content hash, compare to what's enforced by check_consistent.py

~/dev/typeshed master λ fd . --type f | xargs md5 -s | pyp -b 'd = defaultdict(list)' 'z = x.split(); d[z[-1]].append(z[1])' -a '[v for v in d.values() if len(v) > 1]'
['(stdlib/2/_winreg.pyi)', '(stdlib/3/winreg.pyi)']
['(stdlib/2/copy_reg.pyi)', '(stdlib/3/copyreg.pyi)']
['(stdlib/2/os2emxpath.pyi)', '(stdlib/3/os/path.pyi)', '(stdlib/2/os/path.pyi)', '(stdlib/2and3/ntpath.pyi)', '(stdlib/2and3/posixpath.pyi)']
['(stdlib/2/__builtin__.pyi)', '(stdlib/2and3/builtins.pyi)']
['(stdlib/3/enum.pyi)', '(third_party/2/enum.pyi)']
['(stdlib/3/xmlrpc/__init__.pyi)', '(stdlib/3/email/mime/__init__.pyi)', '(stdlib/3/concurrent/__init__.pyi)', '(stdlib/2/email/mime/__init__.pyi)', '(stdlib/3/urllib/__init__.pyi)', '(stdlib/2/distutils/__init__.pyi)', '(third_party/2and3/dateutil/__init__.pyi)', '(third_party/2and3/geoip2/__init__.pyi)', '(third_party/2and3/tornado/__init__.pyi)', '(stdlib/2and3/wsgiref/__init__.pyi)', '(third_party/2and3/pyVmomi/__init__.pyi)', '(stdlib/2and3/xml/etree/__init__.pyi)', '(third_party/2and3/requests/packages/urllib3/packages/__init__.pyi)', '(third_party/2and3/requests/packages/urllib3/contrib/__init__.pyi)', '(stdlib/2and3/distutils/__init__.pyi)', '(third_party/2and3/werkzeug/contrib/__init__.pyi)', '(stdlib/2and3/distutils/command/bdist.pyi)', '(third_party/2and3/werkzeug/middleware/__init__.pyi)', '(stdlib/2and3/distutils/command/install_lib.pyi)', '(stdlib/2and3/distutils/command/clean.pyi)', '(stdlib/2and3/distutils/command/bdist_rpm.pyi)', '(stdlib/2and3/distutils/command/install_data.pyi)', '(stdlib/2and3/distutils/command/build_ext.pyi)', '(stdlib/2and3/distutils/command/check.pyi)', '(stdlib/2and3/distutils/command/build_scripts.pyi)', '(stdlib/2and3/distutils/command/install_scripts.pyi)', '(stdlib/2and3/distutils/command/install_headers.pyi)', '(stdlib/2and3/distutils/command/build.pyi)', '(stdlib/2and3/distutils/command/bdist_packager.pyi)', '(stdlib/2and3/distutils/command/__init__.pyi)', '(stdlib/2and3/distutils/command/bdist_dumb.pyi)', '(stdlib/2and3/distutils/command/register.pyi)', '(stdlib/2and3/distutils/command/bdist_wininst.pyi)', '(stdlib/2and3/distutils/command/build_clib.pyi)', '(stdlib/2and3/distutils/command/sdist.pyi)', '(third_party/2and3/cryptography/__init__.pyi)', '(third_party/2and3/google/protobuf/util/__init__.pyi)', '(third_party/2and3/cryptography/hazmat/bindings/openssl/__init__.pyi)', '(third_party/2and3/cryptography/hazmat/bindings/__init__.pyi)', '(third_party/2and3/google/protobuf/compiler/__init__.pyi)', '(third_party/2and3/google/protobuf/internal/__init__.pyi)', '(third_party/2and3/google/__init__.pyi)', '(third_party/2and3/pymysql/constants/__init__.pyi)', '(third_party/2/fb303/__init__.pyi)', '(third_party/2/kazoo/__init__.pyi)', '(third_party/2/kazoo/recipe/__init__.pyi)', '(third_party/2/OpenSSL/__init__.pyi)', '(third_party/2/scribe/__init__.pyi)', '(third_party/2/concurrent/__init__.pyi)', '(third_party/2and3/Crypto/Random/Fortuna/__init__.pyi)', '(tests/mypy_blacklist.txt)', '(third_party/3/jwt/contrib/__init__.pyi)']
['(stdlib/2/email/MIMEText.pyi)', '(stdlib/2/email/mime/text.pyi)']
['(stdlib/3/concurrent/futures/__init__.pyi)', '(third_party/2/concurrent/futures/__init__.pyi)']
['(stdlib/3/concurrent/futures/process.pyi)', '(third_party/2/concurrent/futures/process.pyi)']
['(stdlib/3/concurrent/futures/_base.pyi)', '(third_party/2/concurrent/futures/_base.pyi)']
['(stdlib/3/concurrent/futures/thread.pyi)', '(third_party/2/concurrent/futures/thread.pyi)']
['(stdlib/3/ipaddress.pyi)', '(third_party/2/ipaddress.pyi)']
['(stdlib/3/unittest/mock.pyi)', '(third_party/2and3/mock.pyi)']
['(stdlib/3/pathlib.pyi)', '(third_party/2/pathlib2.pyi)']
['(stdlib/3/socketserver.pyi)', '(stdlib/2/SocketServer.pyi)']
['(stdlib/3/encodings/utf_8.pyi)', '(stdlib/2/encodings/utf_8.pyi)']
['(stdlib/3/encodings/__init__.pyi)', '(stdlib/2/encodings/__init__.pyi)']
['(stdlib/3.7/dataclasses.pyi)', '(third_party/3/dataclasses.pyi)']
['(third_party/2and3/backports/ssl_match_hostname.pyi)', '(third_party/2and3/requests/packages/urllib3/packages/ssl_match_hostname/_implementation.pyi)']
['(stdlib/2and3/threading.pyi)', '(stdlib/2and3/_dummy_threading.pyi)']
['(stdlib/2and3/xml/dom/pulldom.pyi)', '(stdlib/2and3/xml/dom/minidom.pyi)']
['(third_party/2and3/cryptography/hazmat/primitives/asymmetric/__init__.pyi)', '(third_party/2and3/cryptography/hazmat/primitives/__init__.pyi)', '(third_party/2and3/cryptography/hazmat/__init__.pyi)']
['(third_party/2and3/markupsafe/_native.pyi)', '(third_party/2and3/markupsafe/_speedups.pyi)']
['(third_party/3/six/moves/email_mime_multipart.pyi)', '(third_party/2/six/moves/email_mime_multipart.pyi)']
['(third_party/3/six/moves/email_mime_base.pyi)', '(third_party/2/six/moves/email_mime_base.pyi)']
['(third_party/3/six/moves/BaseHTTPServer.pyi)', '(third_party/3/six/moves/SimpleHTTPServer.pyi)', '(third_party/3/six/moves/CGIHTTPServer.pyi)']
['(third_party/3/six/moves/urllib_response.pyi)', '(third_party/2/six/moves/urllib_response.pyi)']
['(third_party/3/six/moves/urllib_request.pyi)', '(third_party/2/six/moves/urllib_request.pyi)']
['(third_party/3/six/moves/tkinter_tkfiledialog.pyi)', '(third_party/3/six/moves/tkinter_filedialog.pyi)']
['(third_party/3/six/moves/email_mime_nonmultipart.pyi)', '(third_party/2/six/moves/email_mime_nonmultipart.pyi)']
['(third_party/3/six/moves/urllib/__init__.pyi)', '(third_party/2/six/moves/urllib/__init__.pyi)']
['(third_party/3/docutils/__init__.pyi)', '(third_party/3/docutils/parsers/__init__.pyi)', '(third_party/3/docutils/parsers/rst/nodes.pyi)', '(third_party/3/docutils/parsers/rst/__init__.pyi)']
['(third_party/3/contextvars.pyi)', '(stdlib/3.7/contextvars.pyi)']

[JelleZijlstra]

I agree, this hardly seems worth it any more. Hopefully in the not-too-distant future we can drop support for Python 2 completely.

[hauntsaninja]

Okay, sounds good! Closing this and the similar #1427