Resistance to small improvements

[encukou]

An issue doing incremental improvement: if the next small step is too small relative to the disruption (deprecation cycles, proliferation of similar API, outdated books, ...), it doesn't make sense to do it (yet). Perfect is the enemy of good :(

How we can avoid being stuck with every mistake forever?

[encukou]

One solution is to switch to a new API, merging all the known papercuts into a single slash.

[iritkatriel]

Switching to a new api is only a solution if the new api can evolve, by design. Otherwise we will be stuck with its mistakes forever. (Or until we switch to a yet newer api. Since no api will ever go away, we can’t really keep doing this.)

[gvanrossum]

Depending on the timescale, at some point these problems solve themselves. E.g. maybe Python dies (i.e., becomes legacy software). Or maybe CPython dies and some other implementation "wins" (for some time). "Perfect is the enemy of good", and that applies to planning for future changes too. Almost all designs eventually develop serious problems due to challenges that couldn't possibly be foreseen (or were deprioritized in the name of progress), and that's a good thing.

IOW, we will always in some sense be "fighting the last war", and we shouldn't feel (too) bad about that. Anticipate what we can, and then make a move.

[vstinner]

How we can avoid being stuck with every mistake forever?

So far, what I did is to fix the C API and help people affected by my incompatible changes. In practice, I can handle (alone) the number of affected projects (at least, the majority of affected projects).

When I changed Py_TYPE() and Py_SIZE() to disallow using them as l-values and it affected too many projects, I reverted the change. I fixed most affected projects, and I did again the change later once most projects got released with the updated code.

My main issue remains the friction with other core developers who don't understand my whole plain and/or disagree with the benefits (compared to the cost). In my case, it leaded to a burnout. At some point, I thought that ok, maybe the majority of Python developers and users want a language / C API which no longer change and I should just move on to a different project. There a similar issue with CPython still limited to 1 CPU at the same time for CPU-bound workloads.

I also thought that the Steering Council who endorse the responsibility of C API changes, but we had a disagreement on the speed of such changes. Again, my feeling is that the SC wants stability at any cost.

Said differently, this challenge is not only technical. It's also political and so a human issue :-)

[iritkatriel]

Said differently, this challenge is not only technical. It's also political and so a human issue :-)

I think we all see this. The question is whether we can find technical solutions to the human issue, or organisational decisions that can reduce the impact of the human issue.

Technical solutions are well defined processes for API evolution, which are baked into the design of the API from the outset.

Organisational solutions will involve more clarity about how we do things. How sacred is stability? Right now anyone can shout down any change by yelling "backward compatibility", and the process is something like "whoever shouts loudest wins". This leads mostly to paralysis (when people choose to avoid proposing changes), and occasionally to conflict and burnout when people do try (particularly when it involves core devs who are unable or unwilling to communicate in a respectful manner). If we don't want that, then we need some kind of decision at the org level about how we will (or will not) evolve the API.

[vstinner]

The alternative is the D-day migration: one day everybody has to migrate to a new API, the missing ones will be left without any support. We tried that with Python 2 to Python 3 migration. It didn't go well. I'm now more confident about incremental changes: spread changes over many Python versions, limit changes per verison. Also, provide a practical solution to support old and new API in a single code base.

I explained that in my talk: https://github.com/vstinner/talks/blob/main/2023-PyconUS/python-incompatible-changes.pdf

In my experience, the bad part of C API changes is that most people don't get ther rationale for the long term goal. It's not well communicated.

[iritkatriel]

In my experience, the bad part of C API changes is that most people don't get ther rationale for the long term goal. It's not well communicated.

I don't disagree, but I don't see this as something that is leading us to an organisational solution to this problem. It would be great if people always explained their motivation clearly, and then other people took the time to listen to the explanations and politely ask for clarification where it's needed, before making their mind up about the proposal. But this doesn't always happen, and we can't really make an organisational decision that will make it happen. So I think that to say that it should happen is a truism that doesn't point to a way forward on this.

[vstinner]

If you are looking for a solution: we should write an informal PEP on points that most people agree on even if they cannot be solved easily or soon, to give a long term design goal of the C API. So people know where Python goes and why small changes are made.

Petr already wrote such "North Star" document.

[iritkatriel]

If you are looking for a solution: we should write an informal PEP on points that most people agree on even if they cannot be solved easily or soon, to give a long term design goal of the C API. So people know where Python goes and why small changes are made.

Petr already wrote such "North Star" document.

We are working here towards a document that will enumerate the problems we agree exist in the C API. These are problems that we agree need to be avoided in any new API we design, but that doesn't mean we agree on whether they should be fixed incrementally in the current C API.

Also, this issue is about incremental changes in general - the ones that exist today, as well as the ones we will create in the new API. The question, as I see it, is how do we not end up with the new C API in the same place we are now with respect to incremental changes.

[ronaldoussoren]

It helps if (small) changes have a clear benefit that helps code in the version those changes were introduced in. Changes with vague future benefits are harder to sell.

The downside such an approach is that this might lead to disruptive changes when an API must be changed due to improvements in the interpreter, such as recent changes to frame objects that require code to use an accessor function instead of directly peeking in a struct field because the information is no longer there.

@vstinner's pythoncapi-compat project also helps in making it easier to migrate to a new API without dropping support for older python versions.

[vstinner]

One solution is to switch to a new API, merging all the known papercuts into a single slash.

I discussed this solution in issue capi-workgroup/api-revolution#9.

[vstinner]

An issue doing incremental improvement: if the next small step is too small relative to the disruption (deprecation cycles, proliferation of similar API, outdated books, ...)

See also issue #60: "If an incompatible change has to be introduced, how can it be prepared to reduce its impact on users?".