Convention for (optional) annotation of non-inferable major version increments

[hvr]

Augmented Compatibility Annotation Scheme (version 2; WIP)

The Problem

A significant obstacle to being able to automate relaxation of PVP-style upper bounds is that major version increments conflate/overlay two different signals into a single one.

A related problem is that while version numbers are almost always properly assigned according to the PVP rules, mistakes can still happen.

Signal conflation in major version increment signalling

There are two categories of breaking changes that get signaled by the same mechanism:

1. Breaking changes that can be detected/inferred statically by the exposed API type signature
2. Breaking changes that can not be detected/inferred statically by the exposed API type signature

Major version increments that fall into the first category have the important property:

• Given a package which is known to be correct,
  when updating its dependee package to the subsequent new major version (of cat. 1), and it still compiles (in pedantic mode), its preexisting correctness is preserved.

IOW, in this case, we can infer statically and automatically that an upper bound relaxation to the next major version is indeed safe.

These harmless first category major version increments are likely the most frequent ones occurring on Hackage.

For the less frequent second category, however, we don't have any such properties. Even unittests won't give enough confidence (except for trivial enough cases where it's possible to encode all semantic properties a package relies on; but it's unrealistic to assume that authors will invest the required effort and discipline).

The ability to signal this kind of second category is however extremely important for our ecosystem, as otherwise we'd be unable to fix or improve our APIs at the semantic level, and would have to introduce new names each time we want to change or introduce new semantics for existing entities. This tradeoff is obviously undesirable.

In other words, the second category of major version increments is the one which cannot be handled automatically and therefore needs to be detected to avoid allowing incorrect package-build-plans which would be hard to detect and even harder to fix, especially if a massive automation had whitelisted them already.

Misattribution of version increment signalling

As aforementioned, the other problem relevant to automating management of version constraints of dependency are the (infrequently) occurring mistakes in version assignment. We should improve the tooling to help with reducing the risk of mistakes during version assignment; but there will always remain an element of risk.

Scenarios that can occur (listed in order of increasing danger):

• false positive: Signalling an incompatible change when there isn't one
  • e.g. text-1.0 signalled breaking compat even though there wasn't.
  • Other examples include packages which follow SemVer rules for >=1.0.0 (SemVer minor-ver-increments are seen as major-ver-increments in PVP)
• false negative: Failing to signal a backward incompatible change
  • cat. 1: e.g. Cabal-1.24.1 accidentally introduced a backward incompatible typesignature change (i.e. cat. 1) (reverted in Cabal-1.24.2; unfortunately, Stackage LTS had already picked up the broken 1.24.1 release)
  • cat. 2: e.g. directory-1.2.3 inadvertently introduced a semantic change (i.e. cat. 2) which went undetected for long time as it shipped with GHC 8.0.1 (and got reverted in directory-1.3; to be shipped with GHC 8.0.2)
  • Other examples include packages which follow SemVer (non-)rules for 0.y.z versions ("Anything may change at any time. The public API should not be considered stable.")

A different scenario (and actually not a case of misattribution) is when a package does a backward incompatible change, and signals that properly, but then reverts back to the original semantics (while again signalling this properly). This happened for aeson-0.9.* -> aeson-0.10.* -> aeson-0.11.*. So aeson-0.9 and aeson-0.11 are semantically compatible to each other.

The Solution

In order to address the problems stated, we need to

1. detect whether a major version increment is purely of the first category,
2. annotate the compatibility relation to non-directly-preceding major versions, and
3. be able to override PVP compatibility semantics retroactively to correct mistakes in the meta-data.

To this end, we can simply introduce a new field for .cabal files which can be interpreted by the build-bot infrastructure:

x-compatibility: <version-lhs> <compat-relation> <version-rhs>

By simply express this meta-data inside .cabal files, we can leverage the existing Hackage .cabal revision mechanism to add missing annotations retroactively, or to fix incorrectly annotated major versions.
Moreover, we make the standard case simple (i.e. no annotations are needed when the PVP has been applied correctly), and we make it possible to handle exceptional cases (annotations can be added in case of mistakes or deliberate deviations from the PVP).

The valid <compat-relation>s are

• compatible-with
• incompatible-with
• semantically-incompatible-with

version-rhs needs to refer to a single version (e.g. 1.2.3) or a version-constraint expression (e.g. >= 1.2 && < 1.3 or == 1.2.*); it must always refer to versions prior to the current package version.

version-lhs needs to be the same version as specified in the version: field. This is a safe-guard against leaving a stale compatibility annotation in your .cabal file, as each time you update the package version:, you'll get an error if you forgot to remove (or update) any preexisting x-compatibility: annotation.

By default, the following implicit x-compatibility annotations are inferred from the version: field (NB: obviously, for single-part version numbers, i.e. only A, a slightly different implicit default is constructed):

version: A.B.C.Ds
x-compatibility: A.B.C.Ds semantically-incompatible-with <A.B
x-compatibility: A.B.C.Ds incompatible-with >=A.B && <A.B.C

TODO define shadowing/combining/overriding rules in case of conflicting annotations

TODO reconsider whether we can reduce the power of <version-rhs> by allowing only single versions

TODO There doesn't seem to be any use-case for compatible-with; after all, the only reason we need to differentiate between major/minor version incs is for the purpose of efficiently inferring PVP-style upper bounds; but from the POV of empiric testing we only need to differentiate between semantic/non-semantic changes

Examples

directory

Annotations are only needed in directory-1.2.3,

name: directory
version: 1.2.3
-- retroactively revised via .cabal edit
x-compatibility: 1.2.3 semantically-incompatible-with <1.2.3

as well as directory-1.3.0 (releases inbetween don't need any annotation, as the property is transitive)

name: directory
version: 1.3.0
-- annotation included in 1.3.0 release (assuming `x-compatibility` was already known then)
x-compatibility: 1.3.0 incompatible-with >=1.2 && <1.2.3

aeson

Under the assumption that aeson-0.9 and aeson-0.11 have only a cat.1 incompatibility:

name: aeson
version: 0.11.0.0
x-compatibility: 0.11.0.0 incompatible-with ==0.9.*

time

We just need to reduce the severity of the major version increment that occured with 1.0.0 which was done mostly for marketing text as mature/stable, than for actual breaking changes.

name: text
version: 1.0.0
x-compatibility: 1.0.0 incompatible-with ==0.12.*

Cabal

The breaking change in 1.24.1 was a cat.1 one; we only need two annotations, one in

name: Cabal
version: 1.24.1.0
-- retroactively revised via .cabal edit
x-compatibility: 1.24.1.0 incompatible-with ==1.24.0.*

and another one in

name: Cabal
version: 1.24.2.0
x-compatibility: 1.24.2.0 incompatible-with ==1.24.1.*

---

Augmented Compatibility Annotation Scheme (version 1; superseeded)

The Problem

A significant obstacle to being able to automate relaxation of PVP-style upper bounds is that major version increments conflate/overlay two different signals into a single one.

There are two categories of breaking changes that get signaled by the same mechanism:

1. Breaking changes that can be detected/inferred statically by the exposed API type signature
2. Breaking changes that can not be detected/inferred statically by the exposed API type signature

Major version increments that fall into the first category have the important property:

• Given a package which is known to be correct,
  when updating its dependee package to the subsequent new major version (of cat. 1), and it still compiles (in pedantic mode), its preexisting correctness is preserved.

IOW, in this case, we can infer statically and automatically that an upper bound relaxation to the next major version is indeed safe.

These harmless first category major version increments are likely the most frequent ones occuring on Hackage.

For the less frequent second category, however, we don't have any such properties. Even unittests won't give enough confidence (except for trivial enough cases where it's possible to encode all semantic properties a package relies on; but it's unrealistic to assume that authors will invest the required effort and discipline).

The ability to signal this kind of second category is however extremely important for our ecosystem, as otherwise we'd be unable to fix or improve our APIs at the semantic level, and would have to introduce new names each time we want to change or introduce new semantics for existing entities. This tradeoff is obviously undesirable.

In other words, the second category of major version increments is the one which cannot be handled automatically and therefore needs to be detected to avoid allowing incorrect package-build-plans which would be hard to detect and even harder to fix, especially if a massive automation had whitelisted them already.

The Solution

Consequently, we need a way to detect whether the major version increment is purely of the first category.
All we need is literally a single bit of additional information, besides the major version increment.

To this end, we introduce a new convention, by defining a new optional annotation for .cabal files:

x-major-version-kind: <version> <kind>

where currently only the safe token is defined for <kind>.

Without any such annotation recorded for the current major version series we have no knowledge and therefore have to assume pessimistically that the current major version was a result of a category 2 major version increment. This default is also safe-guard, as otherwise we'd risk to err towards incorrectness which is hard to detect, which is what we want to avoid in the first place.

An example would be:

name: foobar
version: 1.2.0
x-major-version-kind: 1.2.0 safe

The version number must match the one declared in the version: field.
This is a safe-guard against leaving a stale x-major-version-kind: annotation in your .cabal file, as each time you update the package version:, you'll get an error if you forgot to remove (or update) any preexisting x-major-version-kind: annotation.

Moreover, since we simply express this meta-data inside .cabal files, we can leverage the existing Hackage .cabal revision mechanism to add missing annotations retroactively, or to fix incorrectly annotated major versions.

[ezyang]

I saw a very different good idea in this proposal: the idea that while PVP versioning should aspire to indicate when BC-breaking changes occur, people make mistakes, and version numbers can't change, so we should introduce a new way to talk about BC-breaking in the metadata. So here's a counterproposal.

Introduce a new field:

version-compatibility: <kind> <version>, ...

The intended semantics is to say, "this release is <kind> <version>". The valid kinds are breaking from, semantically breaking from and compatible with. For example, if we have:

name: p
version: 1.1.1
version-compatibility: breaking from 1.1.0

This states that 1.1.1 is a backwards breaking change from 1.1.0 (you wouldn't ever release a package like this, but you would add this field in post-facto if it was discovered that 1.1.1 broke compatibility.) If 1.1.2 fixed compatibility (and was incompatible with 1.1.1), you would then write:

name: p
version: 1.1.2
version-compatibility: breaking from 1.1.1, compatible with 1.1.0

You can use semantically breaking from as an extra warning that this is a breaking change that is not caught by the type checker.

By default, we use PVP to infer compatibility/breaking. So if you don't specify anything, there is implicitly a breaking from from your package to the last release of the previous major version; similar there is a compatible with from your package to the previous release (if there is one) in the same major version.

[hvr]

@ezyang I like the generalization, one nitpick though:

So if you don't specify anything, there is implicitly a breaking from from your package to the last release of the previous major version;

In order to satisfy the primary goal of the original proposal to make it possible to automate upper bound relaxation safely, by default we have to assume implicitly semantically breaking from

[hasufell]

This might be a bit bikeshedding, but it is about the practical use of PVP.

Afais, PVP is largely not adhered to by many haskellers and as a result, hackage maintainers are constantly updating/changing/fixing cabal files. This suggests that haskellers already have trouble with PVP.

Technically, this change is exactly what is theoretically needed for the PVP to be more useful, but my guess is it will complicate it further to a point that no one will bother using semantically-incompatible-with and other clues, leaving the work to hackage maintainers again.