Depext: Issues with variants in MacPorts

[MSoegtropIMC]

I added MacPorts solutions to a few opam conf packages. See: (ocaml/opam-repository#16912)

One issues is that depext doesn't seem to handle variants like "gtk3 +quartz" in a consistent manner. When printing what to install, gtk3 and +quartz are listed on separate lines, but when sorting packages alphabetically it seems to treat them as one. I guess this is just not handled at all explicitly - maybe it should.

It somehow works as is, but it looks like it might break any day by a change.

(See ocaml-opam/opam-depext#128)

[rjbou]

To check installed packages, we use port installed pkg1 pkg2 pkg3 and parse an output in the form :
zlib @1.2.11_0 (active)
For available package, it is port search --line --exact pkg1 pkg2 pkg3, and the parsed output:

diffutils  3.7  sysutils textproc devel  GNU diff utilities
--
No match for gcc found

You can find the code here (commands & regexp).

Can you share example outputs for packages with and without variants?

[MSoegtropIMC]

Here are a few examples for port installed

~$ port installed gtk3
The following ports are currently installed:
  gtk3 @3.24.14_1+quartz
  gtk3 @3.24.20_0+quartz
  gtk3 @3.24.21_0+quartz (active)
~$ port installed gtk3 +quartz
The following ports are currently installed:
  gtk3 @3.24.14_1+quartz
  gtk3 @3.24.20_0+quartz
  gtk3 @3.24.21_0+quartz (active)
~$ port installed gtk2 +quartz
The following ports are currently installed:
  gtk2 @2.24.32_0+quartz (active)
  gtk2 @2.24.32_0+x11
~$ port installed gtk2
The following ports are currently installed:
  gtk2 @2.24.32_0+quartz (active)
  gtk2 @2.24.32_0+x11
~$ port installed gtk2+quartz
None of the specified ports are installed.
~$ port installed gtk3+quartz
None of the specified ports are installed.

So in the query the variant is ignored, but it is given in the result.

For port search things are less helpful:

~$  port search --line --exact gtk3
gtk3	3.24.21	gnome x11	GTK+ widget toolkit
~$  port search --line --exact gtk3 +quartz
gtk3	3.24.21	gnome x11	GTK+ widget toolkit

That is regardless of the query one gets only one response. Let me find you something that works.

[rjbou]

We do only one syscall per query : one for installed with all retrieved system packages, and one for availability. The pkg1 pkg2 pkgn in my previous comments are the requested macports system packages.
You can display opam port command and its output with verbose level 3, ie -vvv.

[rjbou]

In fact more exactly, the command is port "installed" "pkg1" "pkg2" "pkg3", so if in depexts field there is a package name with a space, one of the argument is for example "gtk3 +quartz".

[MSoegtropIMC]

If I quote it, get this:

~$ port installed "gtk3 +quartz"
None of the specified ports are installed.

This explains why it tries to reinstall packages each time.

I would recommend to remove the variant (a word starting with + or - in case a default variant is switched off) when doing queries and inspect the result.

port search will only search for base ports and ignore variants. I fear you have to query available variants separately with:

port info --variants gtk3
variants: universal, quartz, x11

[rjbou]

I would recommend to remove the variant (a word starting with + or - in case a default variant is switched off) when doing queries and inspect the result.

That, it is possible.

I fear you have to query available variants separately with

hmm... It is possible to do the query for each package that specify a variant, but it will be time consuming, I suppose. Can you launch the port info with time?.
External dependencies retrieving is done at each switch load (not always, but at each time we need the list of available packages, a lot of usual commands are concerned), we need to ensure that depext loading doesn't take too much time.

[MSoegtropIMC]

I think variants are not that common, so it shouldn't matter that much. As far as I can see this is anyway only required for checking available packages which may take time, since it is usually followed by an install. The query of installed packages does include the variant information - I think it is more important that this is fast.

~$ time port info --variants gtk3
variants: universal, quartz, x11

real	0m0.161s
user	0m0.099s
sys	0m0.053s
~$ time port info --variants gtk3
variants: universal, quartz, x11

real	0m0.159s
user	0m0.098s
sys	0m0.053s
~$ time port info --variants gtk3
variants: universal, quartz, x11

real	0m0.155s
user	0m0.097s
sys	0m0.051s

[rjbou]

Can you check that branch ? You can find opam test instruction here.

The time it takes load full depext with the availability check. In fact, installed and available package are retrieved on some operations, and not only install, it can slow down opam list too for example. If too slow (I'm not in favor of keeping the available check commit), the availability part could not be kept with a port info query for each variant specified depext. Even if now there is only one or two in the opam repository, and 0.3s are acceptable, it can easily reach several seconds.

Is there another way than querying them one by one ? Or another way to specify in the depext format the needed variants? Maybe by checking the installation of the other package, like that only single queries are done.

[MSoegtropIMC]

Can you check that branch ? You can find opam test instruction here.

I can check it on Wednesday.

Is there another way than querying them one by one ?

I think it is faster to query several ports (all of them) at a time. You could e.g. do:

~$ time port info --name --variants gtk2 gtk3 gtksourceview3
name: gtk2
variants: universal, quartz, x11, bundle
--
name: gtk3
variants: universal, quartz, x11
--
name: gtksourceview3
variants: quartz, glade, universal

real	0m0.200s
user	0m0.133s
sys	0m0.059s

I think you can replace port search with port info. port info seems to be about 2x slower than port search but is more felxible.

[rjbou]

ping @MSoegtropIMC ?

[MSoegtropIMC]

@rjbou : sorry for not being responsive on this - I am currently working on a large multi platform opam based meta project (Coq platform) and have quite a few loose threads.

I wonder what I should test - as I said it did work before, but it looked like this was just luck. Can you post a short description on what you changed? I think the most important point is that the opam developers are aware of the concept of variants in MacPorts. Or should I just test if it still works?

[rjbou]

No worry :)

There is two changes :

• d1fbd48 that I'd keep if it works well, for installed packages variants handling, in case they are depext, in the form "gtk3 +quartz" (based on previousport installed the example output)
• d9462fc that I'm not keen to keep (time hole, one syscall per variant defined depext), for availability check with variant (also, based on previsou port info output)

[dra27]

@rjbou - what's the status of this for 2.1?

[MSoegtropIMC]

@dra27 : I guess @rjbou is still waiting on my feedback. I will put it on my prio to do list.

[dra27]

@MSoegtropIMC - I think she also has some more code to test and push (we had a dev meeting this morning), so there should be some additional news too, soon!

[MSoegtropIMC]

I just wanted to ask what the status is. Can I start testing or further changes expected?

[rjbou]

Referenced PR is good to test