[POC] Selectively override compat entries

[DilumAluthge]

Summary

This is a proof-of-concept (POC) of a feature that allows users to selectively override [compat] entries.

This is only a POC. If someone wants to develop this feature further, you are welcome to take this code as a starting point. In order to flesh this out into a full pull request, you will need to:

1. Come up with a better interface. (I don't think that specifying a ~/.julia/config/compat-overrides.toml file is a very good interface.)
2. Add tests
3. Add docs

The documentation of this feature should include a warning of the following form:

⚠️ Warning: Use of this feature may cause packages to break. It is the responsibility of the user (not the maintainers of any packages) to solve any problems that arise from the use of this feature. ⚠️

Motivation

Suppose that you want to use two packages PkgA and PkgB in the same environment. PkgB is a direct dependency of PkgA, and PkgA has a [compat] entry for PkgB that looks like this:

PkgB = "1"

Now, PkgB releases version 2.0.0. You would like to use PkgA and PkgB v2.0.0 in the same environment. Unfortunately, because PkgA is not compatible with PkgB v2.0.0, you cannot do so.

This pull request will allow you to use PkgA and PkgB v2.0.0 in the same environment.

Usage

Create a file at ~/.julia/config/compat-overrides.toml that has entries that look like this:

["uuid-for-PkgA"."version-number-of-PkgA"]
"uuid-for-PkgB" = "list, of, versions, of, PkgB, that, you, want, to, allow"

For example, suppose that the UUID of PkgA is 00000000-0000-0000-0000-000000000001, and the UUID of PkgB is 321657f4-b219-11e9-178b-2701a2544e81. And suppose that you want to use PkgA v5.6.7 and PkgB v2. Your ~/.julia/config/compat-overrides.toml file might look like this:

["00000000-0000-0000-0000-000000000001"."5.6.7"]
"00000000-0000-0000-0000-000000000002" = "1, 2"

Related pull requests

1. add an API option to ignore compat #1607 adds an API option to ignore [compat] entries. When the user selects that option, all [compat] entries (except the [compat] entries in your project) will be ignored. This is useful if you want to ignore all [compat] entries and just get the latest versions of everything. However, it does not allow you to only ignore some [compat] entries while preserving all other [compat] entries. In contrast, this pull request allows you to select which [compat] entries you want to override.

Example

In this example, we will use the following packages:

Name	UUID
MLJModelInterface	e80e1ace-859a-464e-9ed9-23947d8ae3ea
ScientificTypes	321657f4-b219-11e9-178b-2701a2544e81

First, observe that MLJModelInterface v0.2.7 is not compatible with ScientificTypes v1:

julia> import Pkg

julia> dir = mktempdir(; cleanup = true)
"/var/folders/jy/7hh5zyw95cg2lh66lfpthqq80000gn/T/jl_AY9nP9"

julia> Pkg.activate(dir)
  Activating new environment at `/var/folders/jy/7hh5zyw95cg2lh66lfpthqq80000gn/T/jl_AY9nP9/Project.toml`

julia> p1 = Pkg.PackageSpec(name = "MLJModelInterface", version = "0.2")
PackageSpec(
  name = MLJModelInterface
  version = VersionSpec("0.2")
)

julia> Pkg.add(p1)
    Updating registry at `~/.julia/registries/General`
    Updating git-repo `https://github.com/JuliaRegistries/General.git`
   Resolving package versions...
   Installed MLJModelInterface ─ v0.2.7
   Installed ScientificTypes ─── v0.8.1
Updating `/private/var/folders/jy/7hh5zyw95cg2lh66lfpthqq80000gn/T/jl_AY9nP9/Project.toml`
  [e80e1ace] + MLJModelInterface v0.2.7
Updating `/private/var/folders/jy/7hh5zyw95cg2lh66lfpthqq80000gn/T/jl_AY9nP9/Manifest.toml`
  [e80e1ace] + MLJModelInterface v0.2.7
  [321657f4] + ScientificTypes v0.8.1
  [9a3f8284] + Random
  Progress [========================================>]  2/2
2 dependencies successfully precompiled in 3 seconds

julia> Pkg.pin(p1)
   Resolving package versions...
Updating `/private/var/folders/jy/7hh5zyw95cg2lh66lfpthqq80000gn/T/jl_AY9nP9/Project.toml`
  [e80e1ace] ↓ MLJModelInterface v0.2.7 ⇒ v0.2.7 ⚲
Updating `/private/var/folders/jy/7hh5zyw95cg2lh66lfpthqq80000gn/T/jl_AY9nP9/Manifest.toml`
  [e80e1ace] ↓ MLJModelInterface v0.2.7 ⇒ v0.2.7 ⚲

julia> p2 = Pkg.PackageSpec(name = "ScientificTypes", version = "1")
PackageSpec(
  name = ScientificTypes
  version = VersionSpec("1")
)

julia> Pkg.add(p2)
   Resolving package versions...
ERROR: Unsatisfiable requirements detected for package MLJModelInterface [e80e1ace]:
 MLJModelInterface [e80e1ace] log:
 ├─possible versions are: 0.1.0-0.3.6 or uninstalled
 ├─restricted to versions 0.2.7 by an explicit requirement, leaving only versions 0.2.7
 └─restricted by compatibility requirements with ScientificTypes [321657f4] to versions: 0.3.5-0.3.6 or uninstalled — no versions left
   └─ScientificTypes [321657f4] log:
     ├─possible versions are: 0.1.0-1.1.1 or uninstalled
     └─restricted to versions 1 by an explicit requirement, leaving only versions 1.0.0-1.1.1
Stacktrace:
  [1] propagate_constraints!(graph::Pkg.Resolve.Graph, sources::Set{Int64}; log_events::Bool)
    @ Pkg.Resolve ~/Downloads/Pkg.jl/src/Resolve/graphtype.jl:1059
  [2] propagate_constraints!
    @ ~/Downloads/Pkg.jl/src/Resolve/graphtype.jl:1000 [inlined]
  [3] simplify_graph!(graph::Pkg.Resolve.Graph, sources::Set{Int64}; clean_graph::Bool)
    @ Pkg.Resolve ~/Downloads/Pkg.jl/src/Resolve/graphtype.jl:1514
  [4] simplify_graph!
    @ ~/Downloads/Pkg.jl/src/Resolve/graphtype.jl:1514 [inlined]
  [5] resolve_versions!(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, julia_version::VersionNumber)
    @ Pkg.Operations ~/Downloads/Pkg.jl/src/Operations.jl:329
  [6] targeted_resolve(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, preserve::Pkg.Types.PreserveLevel, julia_version::VersionNumber)
    @ Pkg.Operations ~/Downloads/Pkg.jl/src/Operations.jl:1125
  [7] tiered_resolve(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, julia_version::VersionNumber)
    @ Pkg.Operations ~/Downloads/Pkg.jl/src/Operations.jl:1110
  [8] _resolve(io::Base.TTY, env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, preserve::Pkg.Types.PreserveLevel, julia_version::VersionNumber)
    @ Pkg.Operations ~/Downloads/Pkg.jl/src/Operations.jl:1131
  [9] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, new_git::Vector{Base.UUID}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations ~/Downloads/Pkg.jl/src/Operations.jl:1147
 [10] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API ~/Downloads/Pkg.jl/src/API.jl:267
 [11] add
    @ ~/Downloads/Pkg.jl/src/API.jl:218 [inlined]
 [12] add(pkgs::Vector{Pkg.Types.PackageSpec}; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API ~/Downloads/Pkg.jl/src/API.jl:143
 [13] add
    @ ~/Downloads/Pkg.jl/src/API.jl:141 [inlined]
 [14] #add#22
    @ ~/Downloads/Pkg.jl/src/API.jl:138 [inlined]
 [15] add(pkg::Pkg.Types.PackageSpec)
    @ Pkg.API ~/Downloads/Pkg.jl/src/API.jl:138
 [16] top-level scope
    @ REPL[8]:1

Next, create a new file at ~/.julia/config/compat-overrides.toml with the following contents:

["e80e1ace-859a-464e-9ed9-23947d8ae3ea"."0.2.7"]
"321657f4-b219-11e9-178b-2701a2544e81" = "0.7, 0.8, 1"

Now, we try again to add MLJModelInterface v0.2.7 and ScientificTypes v1 into the same environment, and this time it works:

julia> import Pkg

julia> dir = mktempdir(; cleanup = true)
"/var/folders/jy/7hh5zyw95cg2lh66lfpthqq80000gn/T/jl_sDhzgN"

julia> Pkg.activate(dir)
  Activating new environment at `/var/folders/jy/7hh5zyw95cg2lh66lfpthqq80000gn/T/jl_sDhzgN/Project.toml`

julia> p1 = Pkg.PackageSpec(name = "MLJModelInterface", version = "0.2")
PackageSpec(
  name = MLJModelInterface
  version = VersionSpec("0.2")
)

julia> Pkg.add(p1)
    Updating registry at `~/.julia/registries/General`
    Updating git-repo `https://github.com/JuliaRegistries/General.git`
   Resolving package versions...
[ Info: Using compat override file at "/Users/dilum/.julia/config/compat-overrides.toml"
   Installed ScientificTypes ─── v1.1.1
   Installed MLJModelInterface ─ v0.2.7
Updating `/private/var/folders/jy/7hh5zyw95cg2lh66lfpthqq80000gn/T/jl_sDhzgN/Project.toml`
  [e80e1ace] + MLJModelInterface v0.2.7
Updating `/private/var/folders/jy/7hh5zyw95cg2lh66lfpthqq80000gn/T/jl_sDhzgN/Manifest.toml`
  [e80e1ace] + MLJModelInterface v0.2.7
  [321657f4] + ScientificTypes v1.1.1
  [9a3f8284] + Random
  Progress [========================================>]  2/2
2 dependencies successfully precompiled in 3 seconds

julia> Pkg.pin(p1)
   Resolving package versions...
Updating `/private/var/folders/jy/7hh5zyw95cg2lh66lfpthqq80000gn/T/jl_sDhzgN/Project.toml`
  [e80e1ace] ↓ MLJModelInterface v0.2.7 ⇒ v0.2.7 ⚲
Updating `/private/var/folders/jy/7hh5zyw95cg2lh66lfpthqq80000gn/T/jl_sDhzgN/Manifest.toml`
  [e80e1ace] ↓ MLJModelInterface v0.2.7 ⇒ v0.2.7 ⚲

julia> p2 = Pkg.PackageSpec(name = "ScientificTypes", version = "1")
PackageSpec(
  name = ScientificTypes
  version = VersionSpec("1")
)

julia> Pkg.add(p2)
   Resolving package versions...
Updating `/private/var/folders/jy/7hh5zyw95cg2lh66lfpthqq80000gn/T/jl_sDhzgN/Project.toml`
  [321657f4] + ScientificTypes v1.1.1
No Changes to `/private/var/folders/jy/7hh5zyw95cg2lh66lfpthqq80000gn/T/jl_sDhzgN/Manifest.toml`

julia> Pkg.pin(p2)
   Resolving package versions...
Updating `/private/var/folders/jy/7hh5zyw95cg2lh66lfpthqq80000gn/T/jl_sDhzgN/Project.toml`
  [321657f4] ↓ ScientificTypes v1.1.1 ⇒ v1.1.1 ⚲
Updating `/private/var/folders/jy/7hh5zyw95cg2lh66lfpthqq80000gn/T/jl_sDhzgN/Manifest.toml`
  [321657f4] ↓ ScientificTypes v1.1.1 ⇒ v1.1.1 ⚲

julia> import MLJModelInterface, ScientificTypes

julia> Pkg.status()
Status `/private/var/folders/jy/7hh5zyw95cg2lh66lfpthqq80000gn/T/jl_sDhzgN/Project.toml`
  [e80e1ace] MLJModelInterface v0.2.7 ⚲
  [321657f4] ScientificTypes v1.1.1 ⚲

[ExpandingMan]

I think the ability to do this selectively is a big deal as it probably has a lot more viable use cases than simply "ignore compat". 👍

[DilumAluthge]

I think the ability to do this selectively is a big deal as it probably has a lot more viable use cases than simply "ignore compat". 👍

It becomes inconvenient, however, if you want to override a large number of [compat] entries.

The nice part about #1607 is that you only need to specify a single keyword argument, and then you are able to override all [compat] entries and just get the latest versions of all packages.

In contrast, with this PR, if you want to override e.g. 50 different [compat] entries, you have to manually write out 50 entries in the ~/.julia/config/compat-overrides.toml file.

So I think this PR is probably more convenient if you just have a few [compat] entries that you want to override.

But if you want to override a large number of [compat] entries, I think #1607 is more suitable.

[DilumAluthge]

Anyway, I only created this PR as a proof-of-concept. I'm not planning on working on it further.

If there are one or two people that really want to see this feature end up in Pkg, they are welcome to take this code and turn it into a proper RFC.

[BioTurboNick]

This would be really useful for debugging compat issues.

Sometimes the latest versions really don't work, but a few back work fine. Figuring out where that line is would be simplified with this utility.

Coming from: https://discourse.julialang.org/t/should-upgrading-compat-bounds-be-considered-a-breaking-change-loopvectorization-and-arrayinterface/87817/12