Installing and recording new dependencies in a single step

[ELLIOTTCABLE]

At the moment, all dependencies must be encoded manually. In the OCaml ecosystem, that isn't too bad, just yet; but as opam 2.0 has started to make it a lot easier to build, develop, and publish multiple smaller/more focused packages, I imagine that will begin to shift.

As mentioned in #3606, one thing I'm really missing from my maintainer/contributor workflow in Node packages, is an equivalent of --save and --save-dev: I'm currently having to install a package, watch what version opam decides to resolve it to, manually copy that version into my opam file as a dependency, and then re-generate my opam.lock. A bit of a verbose process!

I'd love to see a behaviour to do all of this automatically — hopefully, one that includes sane SemVer defaults (see @whitequark's #2976?) for newly-installed packages, so that new users of my packages are likely to be protected from breaking changes I make as a default behaviour.

[rjbou]

There is something unclear... Why do you need to add the packages in the opam file and then regenerate the lock file? opam lock generates dependencies list according the current switch state, calling the solver with install criteria.

[ELLIOTTCABLE]

I'm not sure I understand the question — when I need to add a new dependency in my library, four things implicitly happen:

1. Some part of the source-code changes to make use of that dependency,
2. I install that dependency in the project, so that code will compile (in opam terms, I add it to my local switch),
3. I indicate to dependents of my library the known outer bounds on the new transitive dependency for their projects (in opam terms, I add it to my depends:)
4. and I correspondingly indicate to fellow collaborators and/or my CI the inner bounds on that dependency, i.e. the exact version(s) that are known to work and should be used when reproducing my development environment (in opam terms, I add it to my lockfile's depends:)

Does that answer your question? Happy to elaborate.

[rjbou]

I see better, thanks. afaik, it is not yet possible in opam to do it automatically (ping @AltGr?).

An option --save <opamfile>, but live thinking about it some rules needs to be defined, e.g:

• even if not given as install argument, should opam save the installed version number? As it is resolved given the current switch, it is not the most accurate regarding the package itself
• if version number is given, should the package be constrained to this one in the opam file? opam can't determine itself if the version number is as upper or inner bound. npm adds the ~version, but this is not necessarily needed to have this strong constraint
• if a package is already present, should opam overwrite it? In the case that there is not constraint already defined, it can overrides it, otherwise, it can be added, but not be sure to not have an irresolvable/null constraint

As it is a plugin, lock file generation can't be integrated.

[ELLIOTTCABLE]

I definately think opam has some different considerations here than npm, so we shouldn't blindly copy their details. For me, the takeaway from npm is "automating this particular workflow is nice" — from there on out, the decisions will have to be OCaml / compiled-software specific. (In fact, I was just talking about the diffences between the Node and OCaml problem-space with @zkat as it pertains to npm copying opam's interactive-resolution mechanism!)

Specifically to those points, here're my thoughts:

1. save/no-save being the default: I personally prefer saving being the default, but I'm also very, very used to npm's ways of doing things, and I didn't initially like that default-on behaviour. To boot, people in the OCaml community aren't nearly as used to this. Maybe default-to-not-saving, at least at first?

   This is especially interesting given switches: in the Node world, node_modules is intended to be a very close mirror of the package-lock.json. In keeping with this, by default, an unqualified npm install will remove packages, if they're not listed in the package.json. Meanwhile, over here, at least at the moment, it's pretty common for the switch-state to have just about no correlation to the indicated behaviour from the .opam file.

   Just a thought.

2. I absolutely think constraints should be persisted, especially if they're explicitly indicated by the user in the same command they requested --save with:

   opam install --save uucp>=11.0.0
   

   The user's wishes seem pretty clear there?

3. If there's already a constraint, it should probably be an error. It's really easy to forget “Oh, I found out my package very specifically doesn't work with newer versions of ”; and if you encoded that into your package's versioning constraints and immediately forgot it, you don't want an off-the-cuff installation of another package to lose that context, do you? ¯\_(ツ)_/¯

[ELLIOTTCABLE]

One thought I want to add: I touch on this above, but it seems to me that, in a way, the dependency-graph expressed in the .opam file is orthogonal to the actual state of the switch being used. This is resolved in one direction, on the ‘receiving end’ so to speak, by lockfiles (“ignore what I say, do what i actually did”) — but in the context of this discussion, it feels like we need a different (in fact, the opposite) solution in the other direction: “ignore what I've actually done, and make sure this applies to what I just said.”

In other words, could a separate dependency-resolution take place on --save, not just updating actually-installed versions in the current switch, but essentially finding a solution as if the actual dependencies expressed in the .opam were installed as-encoded? Do the dependency-resolution for --save against a ‘ghost-switch’, if you will.

[ELLIOTTCABLE]

Just bumping to show continued interest in the possibility of such a feature!

(Also, I'm totally okay with scoping this down to just --save, and the feature not concerning itself with lockfiles / the lock plugin, for now. Would still be a timesaver!)

[rjbou]

On the constraint persistence, I agree with you, but I have the feeling that not most of user uses those constraints: usually you install the package with no version, or you specify a specific one, while in the opam file, most of the time, constraints are not as (one specific version) strict. In this case, i'd say that it's the user that need to be more precise on version expression at install as it will be added in the opam file.

On the "forgotten breaking new version", generally in that case you reinstall an older version of the incriminated package, plus it will be with a constraint, that will be directly integrated in the opam file. Otherwise, for every package that is already present, the need to open opam file and manually update remain, what minimizes the use of this option.

i began a first draft of this option here, if you want to take a look.

---

If you want to check that dependencies in your opam are coherent, you can test this by creating an empty switch (that can be used afterwards for those checks also)

opam switch create empty --empty

and then when you want to check, you can run a fake install in this switch, whatever you current switch is

opam install <opamfile> --dry-run --switch empty -n

This will only check dependencies consistence, not installation status as it performs no real action.

---

An easier way to modify you opam file without editing them is to use the package opam-ed

[rjbou]

Also pin-depends should also be updated by the option? As, if pinned, the installed package will be installed from pin sources (not repo verseion), there will be an inconsistency if opam file don't integrate that information.

[rjbou]

@ELLIOTTCABLE The feature is implemented in a plugin: opam-otopop.