Sites, locations, relocation

[bobot]

This PR adds the notion of sites which are subdirectories of section directories (share, lib, ...) directory. These directories are at different places, in the local directory during development, in the install directory after installation. Moreover other packages could need to install files in another package directory such as plugins.

Sum-up of the features

• site can be defined in the package definition in dune-project. A site is a sub-directory in the share, lib, or other section of the package
• Other package can specify the site of another package in its install stanza in order to install files.
• site are not associated to a unique directory because a package A can be installed but we are building another package B which installs file in the site of A. So the directories associated to the site foo of A are $lib/A/foo and _build/install/lib/A/foo.
• a new plugin stanza allows to create a plugin in the site of another package. Currently it is restricted to an empty library which depends on other libraries
• a new generate_module stanza allows to create a module which for each site, lists the directories of this site. This module use the same infrastructure than the build-info. So the site is substituted during promote and installation. An environment variable allows also to append to the site directory.
• The generate_module gives also facilities for knowing the OCAMLPATH but more importantly to configure findlib so that fl_dynload could be used trivially.
• dune install as a new option --relocatable which will create a relocatable directory. All the libraries needed by the plugins or which have some external data, must be in the local workspace (not installed by opam).

The complete documentation is available in the diff

FAQ

• Which plugins are available? The installed plugins and all the plugins that have been built locally (the package dependency should be used) are available

[bobot]

Relocation mode is implemented. But since findlib loads the META files of all the dependencies, copying the local libraries META are not sufficient. For example the META for the thread library is not present. Creating a dumb META file is sufficient (done is the test in the run.t).

There are some solution:

• Let the developers create the missing META files (The whole MR is an experimental feature)
• Creating the one from ocaml, but we will need to keep that up-to-date
• replacing findlib.dynload based on our own META parsing:
  • using existing one but it depends on stdune
  • creating one that just look for the plugin(...) using just ocamllex, but its another META parser

@diml do you have an opinion?

EDIT: the existing parser can be made independent of stdune, which seems the best option

[bobot]

A first round of review can be done.

[bobot]

@nojb I would appreciate if you could take a look. The relocatable part is implemented and should work if you can put all your dependencies in the workspace (except for a misfeature for the libraries installed from OCaml, cf above comment). The tests in sites_locations shows how to use it. I'm going to test the relocation part with why3.

[bobot]

Artefact_substitution is here done on libraries, so cmx hash is at the end wrong in some sense. With sufficient Sys.opaque, it should not create bugs but it is fragile. If the generated module was always compiled with -opaque it would work.

[ghost]

Doesn't ocamlfind install these meta files?

[nojb]

@nojb I would appreciate if you could take a look. The relocatable part is implemented and should work if you can put all your dependencies in the workspace (except for a misfeature for the libraries installed from OCaml, cf above comment). The tests in sites_locations shows how to use it. I'm going to test the relocation part with why3.

I will try, but won't be able to do so until next week the earliest (too many things on my plate at the moment).

[bobot]

Doesn't ocamlfind install these meta files?

It does but we need to copy all of them during the creation of the installation directory during for relocation, in fact we still need to copy the cmxs and cma which is not yet done.

It is perhaps overkill for the little problem but now it is independent of findlib. The advantages are that it follows now the same rules than dune for the hardcoded ocamlpath; plugins can have the same name than librairies; we don't have to initialize multiple times findlib; no cache to cleanup. The drawbacks is 200 lines of code.

[bobot]

Thanks a lot @nojb. The implementation is finished and you will maybe appreciate that there is no more findlib dependency. I'm going to write the documentation.

[bobot]

I'm splitting this MR in understandable part:

• oneliner in cram [Cram sanitization] path separator used to separate directory #3329
• factorization of the eval function for artifact [Artifact] Move the artifact evaluation to its own private library #3330
• sites definition and use in install stanza
• sites locations in binaries and --relocatable
• plugins

[ghost]

Thanks for splitting!

[bobot]

All the requests have been fixed in ebf35fc and the branch has been rebased. During the rebase I had to move Install to dune_engine.

[rgrinberg]

@cwong-ocaml @aalekseyev this change is unfortunate, but it seems inevitable if artifact substitution is part of the engine.

[cwong-ocaml]

I think it might be a good idea to see if we can rewrite Install in terms of a lower-level filesystem-as-known-to-dune API (I can also work on putting something like that together); I'm sure this isn't the only module that would benefit from it.

[aalekseyev]

My hope would be that artifact substitution could be a part of the rules, instead. (I don't think we should think too much about that for this PR, though)

[rgrinberg]

@aalekseyev I'd appreciate a second glance at the documentation if you have some time. This doesn't have to delay the PR however.

I'll merge the PR today or tomorrow after one last pass.

[aalekseyev]

I think it's hard to figure out what its location is referring to here. Should it say the location of the executable? (is that even correct?)

[aalekseyev]

I had a quick look at the documentation and it sounds reasonable, but I really can't say I fully understand why the sites and plugins have to be related. It seems pretty sensible to me to collect non-plugin configurations in a site, or to store a plugin separately in a package that defines it, passing it to the runner exe by an alternative mechanism.

One technical question that I'm curious about: if you run a plugin-bearing executable from a dune rule, for example to test it, what happens? What site locations does it observe and what plugins will it find there? What is the correct way to declare dependencies for such a rule?

[rgrinberg]

@bobot merged!

don't forget to address any remaining comments here.

[bobot]

It seems pretty sensible to me to collect non-plugin configurations in a site, or to store a plugin separately in a package that defines it, passing it to the runner exe by an alternative mechanism.

Things can be duplicated, but the declaration, the installation machinery, the passing of the directory localisation in every case is the same for sites and for plugins. Plugins are in fact built on top of sites. I could have splitted the MR this way.

One technical question that I'm curious about: if you run a plugin-bearing executable from a dune rule, for example to test it, what happens?

If we have pkg1 which has an executable exe which accept plugins, and pkg2 defines such a plugin:

• If when testing exe you want to use the plugin pkg2 the right dependency is (package pkg2)
• The site for pkg1 are located in _build/install/default/{share,lib}/pkg1/ and directly available from a module generated by (generate_module (name mod) (sites pkg1) ...)

Even if pkg1 is also installed globally, the local exe will not see those directories.

If pkg1 is installed but pkg2 is a local package:

• same dependencies
• The installed site of pkg1 and the local _build/.../pkg1/ are the location of the site. That's why it is not a unique directory but a list of directories.

Do you think such precision should be added in the documentation? In concept? In (generate_module doc?

[aalekseyev]

Thanks. I'm just trying to understand for now.

If a package pkg3 is later added and it installs a new plugin into the site of pkg1, then the behavior of exe may change based on the existence of that plugin (it scans for all the plugins in a directory, right?). If that dependency is not tracked, then we have a problem (sandboxing may help, but if we're relying on sandboxing then we should make that explicit).

Maybe we could/should instead make it so that merely depending on (package pkg1) forces all its sites to be fully populated?

Alternatively, we could declare that the user must use sandboxing when depending on a package that has any sites.

I think a clarification on what happens when you depend on a (package ...) that has any sites can go into Dependency specification section in concepts.rst, the (package <pkg>) bullet point.

[rgrinberg]

@bobot Jeremie and I reviewed the docs and we think this feature is better documented in its own section (like js_of_ocaml for example). Additionally, it would also be nice to start the doc with a tutorial style introduction that does not assume much prior knowledge, in the style of tests.rst. We feel that such a feature needs a little more narrative for the user to better understand when it should be used.

[bobot]

There is already a section in advanced topic about sites/plugins, is it this kind of example you want me to move to its own file? like jsoo.rst?

[ghost]

The topic we are covering here is "how to load additional files at runtime". It feels important enough to get its own chapter in the manual, explaining each concept and feature one by one from the simpler ones to the most advanced ones. doc/tests.rst is a good example of that.

Initially, the manual was written in a "reference manual" style, because there was a lot to document and just describing what was there was the quickest way to mint the manual. However, while a reference manual is good it is not the easiest for users. So we are trying to move away from a reference manual and move towards a more classic book about to learn Dune.

Here, my reasoning is the following; as an application grows and becomes more complex, there will be a point were the developers will want to add support for loading additional files (including plugins) at runtime. To learn how to do that, they should look for a chapter in the manual that explains how to load additional files at runtime in applications developed with Dune. This chapter should explain everything from the ground up, so that at the end of it the developer knows what is a site, how Dune installs such additional files, etc...

[bobot]

Ok, done in #3794. Thank you for the explanation, I will add some more in this direction.