opam list --installable doesn't take the currently installed packages into account

[kit-ty-kate]

If one of the package currently installed is not available anymore (I suspect mainly the compiler packages), then opam list --installable does not work anymore.

Here is a test-case using docker:

FROM ocurrent/opam:debian-10-opam
RUN opam init --disable-sandboxing -ay --bare
RUN opam repository set-url --set-default default git://github.com/ocaml/opam-repository.git
RUN opam switch create -y 4.09.1
RUN opam list -s --all-versions --installable --depopts --depends-on ocp-index.1.2
# That worked..

# Now let's set the repository to the commit just before ocaml 4.09.1 gets added
RUN opam repository set-url --set-default default git://github.com/ocaml/opam-repository.git#a322837f3918a7b1a220a3c47c1ca3353d1d6a01
RUN opam list -s --all-versions --installable --depopts --depends-on ocp-index.1.2
# That one doesn't work (empty output) :(

The first call to opam list --installable returns:

ocaml-top.1.1.5
ocp-browser.1.0.0
ocp-browser.1.2
ocp-index-top.0.4.1
ocp-index-top.0.4.3
ocp-index-top.0.5.0
user-setup.0.1
user-setup.0.2
user-setup.0.3
user-setup.0.4
user-setup.0.5
user-setup.0.6
user-setup.0.7

but the second does not return anything, and even opam list -s --all-versions --installable, if we would try that, does not return anything at all.

Would it be possible to add the currently installed packages to the global state to be consistent with opam install?

[rjbou]

Then opam list --installable would contains all installed packages... Seems logic, from switch installability point of view.

[AltGr]

There is special handling internally for what we call "orphan packages": packages which are installed but no longer available ; the policy is to try to keep them as long as they can be kept unchanged, but they can't be recompiled, for example.

The code is in OpamClient (and quite complex, involving computation of full dependency and reverse-dependency cones); it would make sense to try to do this processing too for installability computation, but that might be pretty tricky. It would not be reasonable to attempt to duplicate it, in any case.

[AltGr]

The above no longer applies; however, the issue remains. It makes sense from a logics point of view: if your current state is already invalid, whatever the package you add it is still invalid, ergo the package is considered not installable...

The simple solution would probably be to discard the information about installed packages when computing installability, since it shouldn't affect the result in any other way! I'll try that.

[AltGr]

(so the bug would actually be the opposite of what the issue subject says :D)

[AltGr]

alright, my above comments where a little off. What actually happens here is related to the switch invariant (or base packages in 2.0 terms): the switch gets to a situation where its invariant is no longer satisfiable. Theoretically at that point, it means the switch is in an invalid state and can't be fixed unless you manually change the invariant: so having no packages installable is actually the correct answer.

However, there is some magic about "orphan packages", that can allow you to go on while they are still installed and as long as you don't need to change them: this explains the discrepancy between opam install which works, and opam list --installable which is empty.

That has been changed in #4969 though: it removes that magic, resulting in a clearer and simpler theory, and removing a bunch of corner-case bugs along the way. This restores consistency with opam list --installable and opam install, but of course not in the way hoped in this report, since list --installable is not changed but it's opam install which will refuse to install.

I am open to discussions if that's not a suitable outcome, but otherwise think this can be closed.

[kit-ty-kate]

I think it would be worth making opam install --installable --depends-on <pkg> fail entirely if <pkg> is currently not installable (be it because of the current switch state or something else)

[kit-ty-kate]

We currently can’t make the difference between "this package has no revdeps" and "this package can’t even be installed"

[AltGr]

Agreed! Although that might be annoying to add since it is a condition on a conjuction of flags, while they are at the moment supposed to be independent...

We currently can’t make the difference between "this package has no revdeps" and "this package can’t even be installed"

Note that opam install --installable --depends-on A may not be empty even if A is not installable: you may have C that depends on A|B and thus remains installable even if A is not. (This may be more likely to happen with multiple versions of A)
Not that I think it's very useful for opam list to let the user query on that :).