Automating patching/generating of stubs

[zdenop]

Can you please create new version for recent version (5.12.1) PyQt5?
Your package requires PyQt5==5.11.3

[stlehmann]

Done :)

[stlehmann]

Some sort of automation would be great here.

[zdenop]

Thanks. IMO also https://github.com/stlehmann/PyQt5-stubs/tree/5.12.1.0/requirements should be updated....

[kazum1kun]

Qt 5.13 is out, would you consider updating it again to latest Qt? Thanks!

[bjwest]

Seriously! Should we need to ask for this? I understand you may be busy so how about automating the process somehow?

[The-Compiler]

@bjwest Your tone seems unnecessarily demanding for something someone does for free, in their spare time. Everyone (including @stlehmann, see the comment above) seems to agree that more automation seems like a good idea, so why not start contributing to make the process easier (and thus faster for everyone)?

[stlehmann]

@bjwest You are very welcome to present some sort of automation concept here.
@The-Compiler thanks for pointing that out.

I would very much like to automate the process but it would be good to have someone with experience in this matter. I think the first step would be to create a hook to the PyQt5 repository that creates a new branch and merges the current PyQt5 stubs. Do you guys know someone with experience in git hooks?

[The-Compiler]

@stlehmann IMHO the best way forward would be to see what kind of stub changes makes sense to contribute upstream to sip (maybe hidden behind a commandline flag, if there's a reason to diverge between IDE-friendly autocompletion stubs and mypy-friendly type checking stubs).

Are you at Europython? If so, let's talk or maybe even work on PyQt stubs at the sprints?

(Also, note that there isn't a public PyQt5 repository, you'd need to get new releases from the website or PyPI)

[stlehmann]

@The-Compiler I would really love to got to Europython but sadly I couldn' t make it :(

What you're suggesting means we should initiate some communication with some PyQt5 guys. That might actually be a good way to go. This way we tackle the issues at their roots. Has somebody got some good contacts to PyQt5 development team?

[The-Compiler]

The PyQt mailing list is the best way to get in contact with the author - I've done so before and provided some observations (before you started PyQt5-stubs IIRC).

What would be needed is a list of concrete issues with the upstream stubs, a proposed solution (e.g. for handling signals correctly) and some testing to make sure it's still usable for autocompletion in IDEs (which I think is the primary goal of PyQt's upstream stubs).

[stlehmann]

Updated to 5.13.0. I also documented the most common issues in issue.md and wrote to the mailing list about a fixing strategy. Let's see what happens.

[kazum1kun]

Thanks a lot! Very helpful to have those.

[rwarren]

We now need 5.13.1 :)

Looking into doing this is a PR rather than a polite poke, I'm trying the build.py script to build in docker and yank the stub files... but is that all that needs doing? it looks like there may be more involved (e.g. the sip.pyi file).

Any tips on building properly so I can just PR this next time?

FWIW: this is prompted by me trying out PyCharm which nicely advertises stub files, then fails because they don't match.

[The-Compiler]

We now need 5.13.1 :)

The API between 5.13.0 and .1 should be the same, no?

[rwarren]

The API between 5.13.0 and .1 should be the same, no?

You would think so. All I can say is PyCharm thinks they are incompatible. I'm unsure why... I'm just exploring these stubs for the first time now.

[stlehmann]

@rwarren There already exists a buildenv that builds latest PyQt5 and yanks the stub files. But they need to get modified in order to work. At the moment this is handy-work that is why it might take some time to react on new releases of PyQt5.

[guizhenwei]

Now PyQt5.13.2 was released.

It would be great to keep PyQt5-Stubs updated with upstream.

PyCharm complains about incompatiable with the latest version of PyQt5.13.2 while installing PyQt5-stubs 5.13.1.

[kamichal]

Instead of hooking the PyQt5 repo, there is also another possibility for semi-automation.
Github allows for manual CI/CD pipeline trigger (just click a button).
That manually-fired job could fetch the newest version of Qt and perform the jobs that have to be done to build a new package. I think it's simplier because doesn't require other projects to be involved or modified. I'm aware that there is also manual work, but seems that would be the last obstacle on the way.

[The-Compiler]

@kamichal That's the smallest problem - if it was just about launching a script all 1-2 months, doing that by hand vs. on CI isn't a big difference. The problem is that most of those changes aren't easily scriptable without fixing them at the source where the bindings are generated.

[The-Compiler]

Taking the freedom to rename this issue, because this mostly morphed into a discussion about how those changes could be automated.

From what I've heard, the code generation part of sip (and thus I'm guessing stubs generation as well) should be rewritten in Python (instead of C). I'm not sure when that will be happening though, but if we're going to hack on sip, it'd probably be best to wait until that's done.

After that, I can see a couple of ways forward:

• Upstream changes to PyQt/sip. Upstream generally has a different goal with those stubs (mainly making autocompletion in IDEs as usable as possible, not type-checking via mypy), so I suspect this won't be possible for all cases. But at least for things which are clearly errors, this sounds like a reasonable idea. Maybe we could even upstream something like a --bindings-type mypy and all our changes?
• Automate the patching of .pyi files. This seems difficult: It needs a way to read/write .pyi files and a way to get more information (e.g. what's a signal?) from somewhere.
• Use sip to export to XML (if that's still available?) and write our own generator from there.
• Write our own small stubs generator based on the .sip files directly. That might be some effort, but also has a lot of benefits: We can generate whatever we want without having to work with upstream (which can be difficult, due to the "patches via ML and no public repository" style of working), and we can generate the stubs directly from PyQt sources without having to run sip or install Qt.

[The-Compiler]

Okay, so we have 2275 errors with the current master branch and mypy from what it seems... Fun!

Given that:

• The current PyQt stubs aren't really high quality (at least when using them with mypy)
• It's somewhat difficult to contribute fixes upstream with the only medium available being a mailinglist and a mercurial repository.
• The generator for those stubs is written in C, and there's no concrete plans on when that will change.

I wonder it would be easier to generate our own bindings from scratch rather than patching upstream's... Either starting from the XML exports (which still seem to be available, but will need our own build of those modules still), or parsing the .sip files ourselves (perhaps reusing parts of flex/bison based parser from sip via Python).

For reference, here's sip's type hint generation code: https://www.riverbankcomputing.com/hg/sip/file/tip/code_generator/type_hints.c

[stlehmann]

A great bunch of errors it is.

wonder it would be easier to generate our own bindings from scratch rather than patching upstream's

I can not really estimate the amount of time it would take but that sounds like be a huge amount of work. I actually would like to keep the effort within reasonable amounts because we don't know if they might one day provide high quality upstream stubs which would make this project redundant.

I just came up with another idea which I want to throw in the round. How about automatically parsing the mypy output and applying fixes accordingly. There are a lot of errors that repeat themselves and are easily fixed. E.g. we have a lot of errors with incompatible supertypes like this:

PyQt5-stubs/QtSensors.pyi:71: error: Argument 1 of "filter" is incompatible with supertype "QSensorFilter"; supertype defines the argument type as "QSensorReading"

This is an error that can actually not be fixed because of the class design of Qt being incompatible with the Liskov substitution principle. So the only solution I came up with is adding # typing: ignore at the end of the line.

This means we can parse mypy output looking for let's say "incompatible with supertype" by using Regex and add # type: ignore to all matching lines.

This is just the simplest of examples. But I think that might be aplicable in a way like this:

• Checking for a pattern by regex and extract all necessary information (line nr., error type)
• Call a function that handles this case (parameters: file, line number)
• The function than can modify the stub file to fix the error

I think that could eliminate a fair amount of the errors we get. Additionally we could add the functionality that @The-Compiler proposed and use a list with signals to replace signal definitions.

[The-Compiler]

As for finding out what is a signal, it just occurred to me that it's easiest to ask Qt directly at runtime. This prints all signals:

from PyQt5 import Qt, QtCore
from PyQt5 import QtWebEngine, QtWebEngineCore, QtWebEngineWidgets


def print_signals(module):
    for clsname in dir(module):
        cls = getattr(module, clsname)
        if not hasattr(cls, '__dict__'):
            continue
        for name in cls.__dict__:
            if name.startswith('_'):
                continue
            try:
                attr = getattr(cls, name)
            except AttributeError as e:
                continue

            if isinstance(attr, QtCore.pyqtSignal):
                print(f'{cls.__module__}.{cls.__qualname__}.{name}')

print_signals(Qt)
print_signals(QtWebEngine)
print_signals(QtWebEngineCore)
print_signals(QtWebEngineWidgets)

So the remaining question (at least IMHO) would be how to automatically edit the existing .pyi files (if we don't want to generate new ones). I wonder if LibCST or redbaron or perhaps jedi (or their lower-level equivalents, baron and parso); or perhaps lib2to3 (deprecated) can be used to do those kind of edits...

[bluebird75]

Since PyQt5 is in maintenance mode, very few new releases are expected, so automating the upstream conversion to our stubs is more work than just merging the latest version.

As for automating some fixes, we already have a reasonable amount of them and this will be expanded in the future.