PDEP-17: Backwards compatibility and deprecation policy

[Aloqeely]

cc @pandas-dev/pandas-core @pandas-dev/pandas-triage

An implementation of the PandasDeprecationWarning variable mentioned here can be seen in #58169

[WillAyd]

Thanks for starting this @Aloqeely . Interested to see how this conversation goes - this is an important topic!

[WillAyd]

Just some random thoughts:

1. We should state here what the public API is - I assume just anything in the pd. namespace. We've talked about that before but not sure we've ever stated officially
2. We should probably also clarify what "experimental" means in this PDEP. A recent example has been the nullable integer types. Those came out in January 2019 as part of the 0.24 release but are still labeled experimental

For the sake of proposing something on point 2 I'd say we should drop the experimental label when something survives a full major release cycle

[rhshadrach]

I assume just anything in the pd. namespace.

I understand what you're going for, but core is in the pd. namespace. I think we'll have to use "documented as public", or explicitly list it out in this PDEP.

[Aloqeely]

I found this page in the docs that defines the public API: https://pandas.pydata.org/docs/reference/index.html, going to hyperlink to it.

[Aloqeely]

I'd say we should drop the experimental label when something survives a full major release cycle

Should we really enforce something like this? I feel like the duration depends on the type of functionality and I can see some cases where we'd want to keep something as experimental for a bit longer.

[WillAyd]

Would starting with 2 full release cycles in the proposal mitigate that concern? I don't have a strong point of view on what that duration is; I just think its important to have something that we all agree upon so things don't stay "experimental" forever

[Aloqeely]

That sounds more reasonable but do we really mind if things stay experimental for a long period? I personally don't think we should have rules on anything experimental as that sort of defeats the point, but will leave it up for discussion.

[lithomas1]

Personally, I'd leave some room for ambiguity here.

Maybe we should just say that we should review experimental features every major release to decide whether they should be promoted to a regular feature.

[github-actions]

This pull request is stale because it has been open for thirty days with no activity. Please update and respond to this comment if you're still interested in working on this.

[Aloqeely]

@pandas-dev/pandas-core I'd like to start a vote soon but I'd really appreciate more input from the team before doing that.

[lithomas1]

Thanks for working on this - and once again sorry for the long silence.

This looks mostly good to me, just had some minor nits.

[jorisvandenbossche]

@Aloqeely thanks a lot of working on this!

For the abstract or motivation, I would reference our existing policy https://pandas.pydata.org/pandas-docs/version/2.2/development/policies.html and summarize the main addition (my current understanding is that we are mostly following what we had written before, but expanding it / clarifying some details about timelines and warning classes to use)

[jorisvandenbossche]

I know that MAJOR.MINOR.PATCH is the terminology typically used in semver, but how do people feel about using a terminology like major.feature.bugfix ?

That feels closer to how we also communicate about it (e.g. when announcing it, we didn't call pandas 2.2.0 a "minor" release)

[Aloqeely]

This is definitely easier to understand, but don't you think it might be a bit misleading? Because we do release features even in bugfix (patch) releases.

[Aloqeely]

After digging into past release notes, it seems like v2.2.1 is the only patch release with a new 'feature', so maybe that was just an odd case. I'm OK with this.

[jorisvandenbossche]

Yes, I suppose that was an odd case (and it was also only a packaging feature, i.e. a new extra, not an actual code feature)

There might always be exceptions (when there is a good reason), but I think at least the general rule is that bug-fix / patch releases only contain bug fixes (and then even mostly regression fixes)

[WillAyd]

Maybe worth clarifying that we consider "features" from the perspective of a user and not a developer? The latter can happen at any time

[rhshadrach]

I find "bugfix" misleading. We typically do not backport mere bugfixes, only those that fix regressions.

[jorisvandenbossche]

I think one of the main changes in this proposal compared to our current practice, is to more consistently use first a DeprecationWarning and then FutureWarning for higher visibility (this was one of the points that came out of the discussion with a downstream package author in #54970, and triggered doing this PDEP I think).
And to be explicit, I am +1 on that proposal.

[Aloqeely]

I think one of the main changes in this proposal compared to our current practice, is to more consistently use first a DeprecationWarning and then FutureWarning for higher visibility (this was one of the points that came out of the discussion with a downstream package author in #54970, and triggered doing this PDEP I think).

Exactly.

[Aloqeely]

Planning to start the vote in 15 days if there is no unresolved discussion by then.

[attack68]

I just voted for this but wanted to add a comment here: As a developer I think it would be helpful to guide users what to do in these situations in docs, hyper linked from the console-message. My users may have any version of pandas installed (often corporates use quite old versions packaged internally) which means a solution must work across multiple versions. My code often contains version "ifs" and then different branches with "TODO"s as remnants to tidy it up once my software has migrated to a minimum versions of pandas.

Sometimes deprecations can be avoided using current code, other times it uses new implementations which need different types of solutions.

[Aloqeely]

As a developer I think it would be helpful to guide users what to do in these situations in docs, hyper linked from the console-message.

It is usually the case that we guide users how to deal with a deprecation in the warning message if possible, as explicitly mentioned in the PDEP:

A deprecation's warning message should: - Mention how to achieve similar behavior if an alternative is available.

[simonjayhawkins]

Thanks @Aloqeely for working on this.

As a reader of this document, I would ask, "What is the main purpose of PDEP-17?" The abstract states, "This PDEP defines pandas' backwards compatibility and deprecation policy." This seems somewhat unclear as the motivation behind the PDEP since we already have a Version policy. It becomes apparent from following the in-depth discussion that the actual motivation was #54970. This should probably be conveyed in the document for future readers, so they don't need to dig deep into the discussion.

On a related note, to avoid ambiguity, does this document wholly supersede our current policy? The abstract mentions "The main additions ... ," but there is also a removal from the current policy. Specifically, the warning should include the pandas version in which the deprecation will be enforced is being removed. This is my understanding from the discussion, but I cannot see this explicitly mentioned.

Am I correct in thinking that the published policy on the web will be the text in the General policy section only and in its entirety, and not any modifications to the current policy?

Regarding the Public API, a common practice in the past has been to revert changes that break another established package and discuss the issue. Other packages may use experimental features or the internal API, and we do our best to accommodate that, even if sometimes reluctantly.

All that said... +1