[new release] dune-build-info, dune, dune-configurator, dune-action-plugin, dune-private-libs and dune-glob (2.0.0+draft1)

[rgrinberg]

Embed build informations inside executable

• Project page: https://github.com/ocaml/dune
• Documentation: https://dune.readthedocs.io/

CHANGES:

• Introduce alias and package fields to the rule stanza. This is the
  preferred way of attaching rules to aliases. (Add alias field to rules dune#2744, @rgrinberg)

• Add field (optional) for executable stanzas (Add optional executables dune#2463, fixes executables: (optional) equivalent? dune#2433, @bobot)

• Infer targets for rule stanzas expressed in long form (Infer rule targets in long-form rule stanzas as well dune#2494, fixes Rule target inferrence doesn't work with write-file dune#2469,
  @NathanReb)

• Indicate the progress of the initial file tree loading (Show the progress of the file tree scanning dune#2459, fixes dune running in $HOME is super slow dune#2374,
  @bobot)

• Build .cm[ox] files for executables more eagerly. This speeds up builds at
  the cost of building unnecessary artifacts in some cases. Some of these extra
  artifacts can fail to built, so this is a breaking change. (Build .cmx/.o files more eagerly for executables dune#2268, @rgrinberg)

• Do not put the <package>.install files in the source tree unless -p or
  --promote-install-files is passed on the command line (opam pin dune fails as dune.install is not promoted dune#2329, @diml)

• Change implicit_transive_deps to be false. Implicit transitive deps now must
  be manually enabled (Change implicit_transive_deps to be false for 2.0 dune#2306, @rgrinberg)

• Compilation units of user defined executables are now mangled by default. This
  is done to prevent the accidental collision with library dependencies of the
  executable. (Mangle compilation units of dune generated modules dune#2364, fixes Add name mangling to dune binaries dune#2292, @rgrinberg)

• Enable (explicit_js_mode) by default. (Explicit js mode dune#1941, @nojb)

• Add an option to clear the console in-between builds with
  --terminal-persistence=clear-on-rebuild

• Stop symlinking object files to main directory for stanzas defined jbuild
  files (Remove compat symlinks for jbuild files dune#2440, @rgrinberg)

• Library names are now validated in a strict fashion. Previously, invalid names
  would be allowed for unwrapped libraries (Turn a bunch of compatibility warnings into errors dune#2442, @rgrinberg)

• mli only modules must now be explicitly declared. This was previously a
  warning and is now an error. (Turn a bunch of compatibility warnings into errors dune#2442, @rgrinberg)

• Modules filtered out from the module list via the Ordered Set Language must
  now be actual modules. (Turn a bunch of compatibility warnings into errors dune#2442, @rgrinberg)

• Actions which introduce targets where new targets are forbidden (e.g.
  preprocessing) are now an error instead of a warning. (Turn a bunch of compatibility warnings into errors dune#2442, @rgrinberg)

• No longer install a jbuilder binary. (Good bye jbuilder 👋 dune#2441, @diml)

• Stub names are no longer allowed relative paths. This was previously a warning
  and is now an error (Error when stub names are relative paths dune#2443, @rgrinberg).

• Define (paths ...) fields in (context ...) definitions in order to set or
  extend any PATH-like variable in the context environment. (Add (paths ...) field to (context ...) definition dune#2426, @nojb)

• The diff action will always normalize newlines before diffing. Perviousy, it
  would not do this normalization for rules defined in jbuild files. (Remove jbuild style diff comparison dune#2457,
  @rgrinberg)

• Modules may no longer belong to more than one stanza. This was previously
  allowed only in stanzas defined in jbuild files. (Remove more old jbuild support dune#2458, @rgrinberg)

• Remove support for jbuild-ignore files. They have been replaced by the the
  dirs stanza in dune files. (Remove support for jbuild-ignore files dune#2456, @rgrinberg)

• Add a new config option sandboxing_preference, the cli argument --sandbox,
  and the dep spec sandbox in dune language. These let the user control the level of
  sandboxing done by dune per rule and globally. The rule specification takes precedence.
  The global configuration merely specifies the default.
  (Global sandboxing setting dune#2213, @aalekseyev, @diml)

• Remove support for old style subsystems. Dune will now emit a warning to
  reinstall the library with the old style subsystem. (Remove support for old style subsystems dune#2480, @rgrinberg)

• Add action (with-stdin-from ) to redirect input from
  when performing . (Add (with-stdin-from ...) dune#2487, @nojb)

• Change the automatically generated odoc index to only list public modules.
  This only affects unwrapped libraries ([odoc] module visibility and entry list dune#2479, @rgrinberg)

• Set up formatting rules by default. They can be configured through a new
  (formatting) stanza in dune-project (Dune2 formatting dune#2347, fixes dune 2: set up formatting rules by default dune#2315, @emillon)

• Change default target from @install to @all. (Change the default target to @all dune#2449, fixes [PROPOSAL] Change the Default Target dune#1220,
  @rgrinberg)

• Include building stubs in @check rules. (@rgrinberg, Compile stubs for @check dune#2530)

• Get rid of ad-hoc rules for guessing the version. Dune now only
  relies on the version written in the dune-project file and no
  longer read VERSION or similar files (Simplify the computing of version numbers dune#2541, @diml)

• In (diff? x y) action, require x to exist and register a
  dependency on that file. (Improvements to diff? semantics dune#2486, @aalekseyev)

• On Windows, an .exe suffix is no longer added implicitly to binary names that
  already end in .exe. Second, when resolving binary names, .opt variants are no
  longer chosen automatically. (Cleanup handling of .opt and .exe dune#2543, @nojb)

• Make (diff? x y) move the correction file (y) away from the build
  directory to promotion staging area. This makes corrections work with
  sandboxing and in general reduces build directory pollution. (Improvements to diff? semantics dune#2486,
  @aalekseyev, fixes [RFC] Better semantic of diff? dune#2482)

• c_flags, c_names and cxx_names are now supported in executable
  and executables stanzas. (Allow c_flags, c_names, cxx_names in executable stanzas dune#2562, @nojb)
  Note: this feature has been subsequently extended into a separate
  foreign_stubs field. (Add support for foreign libraries dune#2659, RFC [RFC] Foreign libraries dune#2650, @snowleopard)

• Remove git integration from $ dune upgrade (Minor improvements for Build_info and install dune#2565, @rgrinberg)

• Add a --disable-promotion to disable all modification to the source
  directory. There's also a corresponding DUNE_DISABLE_PROMOTION environment
  variable. (Add --disable-promotion option dune#2588, fix Feature request: CLI flag or env var to disable .merlin generation dune#2568, @rgrinberg)

• Add a forbidden_libraries field to prevent some library from being
  linked in an executable. This help detecting who accidently pulls in
  unix for instance (Add a forbidden_libraries field to executable stanzas dune#2570, @diml)

• Fix incorrect error message when a variable is expanded in static context:
  %{lib:lib:..} when the library does not exist. (Fix incorrect message "unknown macro" dune#2597, fix Improve error message for %{lib:...} dune#1541,
  @rgrinberg)

• Add --sections option to $ dune install to install subsections of .install
  files. This is useful for installing only the binaries in a workspace for
  example. (Add --sections option to install/uninstall dune#2609, fixes dune install ignores the (vendor) markers dune#2554, @rgrinberg)

• Drop support for jbuild and jbuild-ignore files (Move jbuild support into a separate library dune#2607, @diml)

• Add a dune-action-plugin library for describing dependencies direcly in
  the executable source. Programs that use this feature can be run by a new
  action (dynamic-run ...). (dune-action-plugin - expressing dependencies directly in source code dune#2635, @staronj, @aalekseyev)

• Stop installing the ocaml-syntax-shims binary. In order to use
  future_syntax, one now need to depend on the ocaml-syntax-shims
  package (Stop installing the ocaml-syntax-shims binary dune#2654, @diml)

• Add support for dependencies that are re-exported. Such dependencies
  are marked withre_export and will automatically be provided to
  users of a library (Add re_exports for libraries dune#2605, @rgrinberg)

• Add a deprecated_library_name stanza to redirect old names after a
  library has been renamed (Add a deprecated_library_name stanza dune#2528, @diml)

• Error out when a preprocessor_deps field is present but not
  preprocess field is. It is a warning with Dune 1.x projects
  (Warn about unused preprocessor_deps dune#2660, @Julow)

• Dune will use -output-complete-exe instead of -custom when compiling
  self-contained bytecode executables whenever this options is available
  (OCaml version >= 4.10) (Use -output-complete-exe if available dune#2692, @nojb)

• Add action (with-accepted-exit-codes <pred> <action>) to specify the set of
  successful exit codes of <action>. <pred> is specified using the predicate
  language. (Add (with-exit-codes <pred> <action>) action dune#2699, @nojb)

• Do not setup rules for disabled libraries (Fix @all alias should only build enabled libraries dune#2491, fixes @all doesn't work with enabled_if on libraries dune#2272, @bobot)

• Configurator: filter out empty flags from pkg-config (Configurator: fix noisy empty compiler command-line arguments dune#2716, @AltGr)

• no_keep_locs is a no-op for projects that use lang dune older than 2.0. In
  projects where the language is at least 2.0, the field is now forbidden.
  (Make no_keep_locs a no-op dune#2752, fixes Delete (no_keep_locs) dune#2747, @rgrinberg)

• Extend support for foreign sources and archives via the (foreign_library ...)
  stanza as well as the (foreign_stubs ...) and (foreign_archives ...) fields.
  (Add support for foreign libraries dune#2659, RFC [RFC] Foreign libraries dune#2650, @snowleopard)

• Add (deprecated_package_names) field to (package) declaration in
  dune-project. The names declared here can be used in the (old_public_name)
  field of (deprecated_library_name) stanza. These names are interpreted as
  library names (not prefixed by a package name) and appropiate redirections are
  setup in their META files. This feaure is meant to migrate old libraries which
  do not follow Dune's convention of prefixing libraries with the package
  name. (Allow top-level libraries in old_public_name dune#2696, @nojb)

• The fields license, authors, maintainers, source, bug_reports,
  homepage, and documentation of dune-project can now be overriden on a
  per-package basis. (Allow to set per-package information dune#2774, @nojb)

[kit-ty-kate]

The tests part are having a hard time:

#=== ERROR while compiling dune-action-plugin.2.0.0+draft1 ====================#
# context              2.0.5 | linux/x86_64 | ocaml-base-compiler.4.06.1 | pinned(https://github.com/ocaml/dune/releases/download/2.0.0+draft1/dune-2.0.0+draft1.tbz)
# path                 ~/.opam/4.06/.opam-switch/build/dune-action-plugin.2.0.0+draft1
# command              ~/.opam/4.06/bin/dune build -p dune-action-plugin -j 2 @install @runtest
# exit-code            1
# env-file             ~/.opam/log/dune-action-plugin-29-ffb3fd.env
# output-file          ~/.opam/log/dune-action-plugin-29-ffb3fd.out
### output ###
# File "doc/dune", line 23, characters 0-136:
# 23 | (rule
# 24 |  (targets dune.inc.gen)
# 25 |  (deps
# 26 |   (package dune))
# 27 |  (action
# 28 |   (with-stdout-to
# 29 |    %{targets}
# 30 |    (run bash %{dep:update-jbuild.sh}))))
# Error: No rule found for alias .dune-files
# File "otherlibs/build-info/test/dune", line 1, characters 0-162:
# 1 | (alias
# 2 |  (name runtest)
# 3 |  (deps
# 4 |   (package dune)
# 5 |   (package dune-build-info))
# 6 |  (action
# 7 |   (progn
# 8 |    (run cram -test %{dep:run.t})
# 9 |    (diff? run.t run.t.corrected))))
# Error: No rule found for alias .dune-build-info-files
# File "test/blackbox-tests/dune.inc", line 1396, characters 0-263:
# 1396 | (alias
# 1397 |  (name pkg-config-quoting)
# 1398 |  (deps
# 1399 |   (package dune)
# 1400 |   (source_tree test-cases/pkg-config-quoting)
# 1401 |   (package dune-configurator))
# 1402 |  (action
# 1403 |   (chdir
# 1404 |    test-cases/pkg-config-quoting
# 1405 |    (progn (run %{exe:cram.exe} -test run.t) (diff? run.t run.t.corrected)))))
# Error: No rule found for alias .dune-configurator-files
# File "test/expect-tests/catapult/dune", line 13, characters 2-24:
# 13 |   ppx_inline_test.config)
#        ^^^^^^^^^^^^^^^^^^^^^^
# Error: Library "ppx_inline_test.config" not found.
# Hint: try: dune external-lib-deps --missing -p dune-action-plugin -j 2 @install
#   @runtest
# File "test/expect-tests/dag/dune", line 13, characters 2-24:
# 13 |   ppx_inline_test.config)
#        ^^^^^^^^^^^^^^^^^^^^^^
# Error: Library "ppx_inline_test.config" not found.
# Hint: try: dune external-lib-deps --missing -p dune-action-plugin -j 2 @install
#   @runtest
# File "test/expect-tests/dune_action_plugin/dune", line 8, characters 3-25:
# 8 |    ppx_inline_test.config dune-glob)
#        ^^^^^^^^^^^^^^^^^^^^^^
# Error: Library "ppx_inline_test.config" not found.
# Hint: try: dune external-lib-deps --missing -p dune-action-plugin -j 2 @install
#   @runtest
# File "test/expect-tests/dune_lang/dune", line 14, characters 2-24:
# 14 |   ppx_inline_test.config)
#        ^^^^^^^^^^^^^^^^^^^^^^
# Error: Library "ppx_inline_test.config" not found.
# Hint: try: dune external-lib-deps --missing -p dune-action-plugin -j 2 @install
#   @runtest
# File "test/expect-tests/fiber/dune", line 14, characters 2-24:
# 14 |   ppx_inline_test.config)
#        ^^^^^^^^^^^^^^^^^^^^^^
# Error: Library "ppx_inline_test.config" not found.
# Hint: try: dune external-lib-deps --missing -p dune-action-plugin -j 2 @install
#   @runtest
# File "test/expect-tests/memo/dune", line 15, characters 2-24:
# 15 |   ppx_inline_test.config)
#        ^^^^^^^^^^^^^^^^^^^^^^
# Error: Library "ppx_inline_test.config" not found.
# Hint: try: dune external-lib-deps --missing -p dune-action-plugin -j 2 @install
#   @runtest
# File "test/expect-tests/dune", line 20, characters 2-24:
# 20 |   ppx_inline_test.config)
#        ^^^^^^^^^^^^^^^^^^^^^^
# Error: Library "ppx_inline_test.config" not found.
# Hint: try: dune external-lib-deps --missing -p dune-action-plugin -j 2 @install
#   @runtest
# File "test/expect-tests/stdune/dune", line 15, characters 2-24:
# 15 |   ppx_inline_test.config)
#        ^^^^^^^^^^^^^^^^^^^^^^
# Error: Library "ppx_inline_test.config" not found.
# Hint: try: dune external-lib-deps --missing -p dune-action-plugin -j 2 @install
#   @runtest
# File "test/blackbox-tests/dune", line 6, characters 44-56:
# 6 |  (libraries stdune dune_re test_common dune configurator))
#                                                 ^^^^^^^^^^^^
# Error: Library "configurator" not found.
# Hint: try: dune external-lib-deps --missing -p dune-action-plugin -j 2 @install
#   @runtest
#     ocamldep otherlibs/action-plugin/test/depend-on-directory-without-targets/bin/.foo.eobjs/foo.ml.d (exit 2)
# (cd _build/default && /home/opam/.opam/4.06/bin/ocamldep.opt -modules -impl otherlibs/action-plugin/test/depend-on-directory-without-targets/bin/foo.ml) > _build/default/otherlibs/action-plugin/test/depend-on-directory-without-targets/bin/.foo.eobjs/foo.ml.d
# File "otherlibs/action-plugin/test/depend-on-directory-without-targets/bin/foo.ml", line 6, characters 7-14:
# Error: Syntax error
#     ocamldep otherlibs/action-plugin/test/dependency-rebuilt-but-not-changed/bin/.foo.eobjs/foo.ml.d (exit 2)
# (cd _build/default && /home/opam/.opam/4.06/bin/ocamldep.opt -modules -impl otherlibs/action-plugin/test/dependency-rebuilt-but-not-changed/bin/foo.ml) > _build/default/otherlibs/action-plugin/test/dependency-rebuilt-but-not-changed/bin/.foo.eobjs/foo.ml.d
# File "otherlibs/action-plugin/test/dependency-rebuilt-but-not-changed/bin/foo.ml", line 5, characters 7-11:
# Error: Syntax error
#     ocamldep otherlibs/action-plugin/test/depends-on-directory-with-glob/bin/.foo.eobjs/foo.ml.d (exit 2)
# (cd _build/default && /home/opam/.opam/4.06/bin/ocamldep.opt -modules -impl otherlibs/action-plugin/test/depends-on-directory-with-glob/bin/foo.ml) > _build/default/otherlibs/action-plugin/test/depends-on-directory-with-glob/bin/.foo.eobjs/foo.ml.d
# File "otherlibs/action-plugin/test/depends-on-directory-with-glob/bin/foo.ml", line 7, characters 7-14:
# Error: Syntax error
#     ocamldep otherlibs/action-plugin/test/depends-on-its-target-by-read-dir/bin/.foo.eobjs/foo.ml.d (exit 2)
# (cd _build/default && /home/opam/.opam/4.06/bin/ocamldep.opt -modules -impl otherlibs/action-plugin/test/depends-on-its-target-by-read-dir/bin/foo.ml) > _build/default/otherlibs/action-plugin/test/depends-on-its-target-by-read-dir/bin/.foo.eobjs/foo.ml.d
# File "otherlibs/action-plugin/test/depends-on-its-target-by-read-dir/bin/foo.ml", line 8, characters 7-8:
# Error: Syntax error
#     ocamldep otherlibs/action-plugin/test/depends-on-its-target/bin/.foo1.eobjs/foo2.ml.d (exit 2)
# (cd _build/default && /home/opam/.opam/4.06/bin/ocamldep.opt -modules -impl otherlibs/action-plugin/test/depends-on-its-target/bin/foo2.ml) > _build/default/otherlibs/action-plugin/test/depends-on-its-target/bin/.foo1.eobjs/foo2.ml.d
# File "otherlibs/action-plugin/test/depends-on-its-target/bin/foo2.ml", line 9, characters 14-18:
# Error: Syntax error: operator expected.
#     ocamldep otherlibs/action-plugin/test/do-not-rebuild-unneeded-dependency/bin/.client.eobjs/client.ml.d (exit 2)
# (cd _build/default && /home/opam/.opam/4.06/bin/ocamldep.opt -modules -impl otherlibs/action-plugin/test/do-not-rebuild-unneeded-dependency/bin/client.ml) > _build/default/otherlibs/action-plugin/test/do-not-rebuild-unneeded-dependency/bin/.client.eobjs/client.ml.d
# File "otherlibs/action-plugin/test/do-not-rebuild-unneeded-dependency/bin/client.ml", line 7, characters 11-15:
# Error: Syntax error: operator expected.
#     ocamldep otherlibs/action-plugin/test/one-absent-dependency/bin/.foo.eobjs/foo.ml.d (exit 2)
# (cd _build/default && /home/opam/.opam/4.06/bin/ocamldep.opt -modules -impl otherlibs/action-plugin/test/one-absent-dependency/bin/foo.ml) > _build/default/otherlibs/action-plugin/test/one-absent-dependency/bin/.foo.eobjs/foo.ml.d
# File "otherlibs/action-plugin/test/one-absent-dependency/bin/foo.ml", line 5, characters 7-11:
# Error: Syntax error
#     ocamldep otherlibs/action-plugin/test/one-dependency-with-chdir/bin/.foo.eobjs/foo.ml.d (exit 2)
# (cd _build/default && /home/opam/.opam/4.06/bin/ocamldep.opt -modules -impl otherlibs/action-plugin/test/one-dependency-with-chdir/bin/foo.ml) > _build/default/otherlibs/action-plugin/test/one-dependency-with-chdir/bin/.foo.eobjs/foo.ml.d
# File "otherlibs/action-plugin/test/one-dependency-with-chdir/bin/foo.ml", line 5, characters 7-11:
# Error: Syntax error
#     ocamldep otherlibs/action-plugin/test/one-dependency/bin/.foo.eobjs/foo.ml.d (exit 2)
# (cd _build/default && /home/opam/.opam/4.06/bin/ocamldep.opt -modules -impl otherlibs/action-plugin/test/one-dependency/bin/foo.ml) > _build/default/otherlibs/action-plugin/test/one-dependency/bin/.foo.eobjs/foo.ml.d
# File "otherlibs/action-plugin/test/one-dependency/bin/foo.ml", line 5, characters 7-11:
# Error: Syntax error
#     ocamldep otherlibs/action-plugin/test/one-directory-dependency/bin/.foo.eobjs/foo.ml.d (exit 2)
# (cd _build/default && /home/opam/.opam/4.06/bin/ocamldep.opt -modules -impl otherlibs/action-plugin/test/one-directory-dependency/bin/foo.ml) > _build/default/otherlibs/action-plugin/test/one-directory-dependency/bin/.foo.eobjs/foo.ml.d
# File "otherlibs/action-plugin/test/one-directory-dependency/bin/foo.ml", line 6, characters 7-14:
# Error: Syntax error
#     ocamldep otherlibs/action-plugin/test/two-stages-dependency-choose/bin/.client.eobjs/client.ml.d (exit 2)
# (cd _build/default && /home/opam/.opam/4.06/bin/ocamldep.opt -modules -impl otherlibs/action-plugin/test/two-stages-dependency-choose/bin/client.ml) > _build/default/otherlibs/action-plugin/test/two-stages-dependency-choose/bin/.client.eobjs/client.ml.d
# File "otherlibs/action-plugin/test/two-stages-dependency-choose/bin/client.ml", line 7, characters 11-15:
# Error: Syntax error: operator expected.
# File "test/blackbox-tests/dune", line 12, characters 19-28:
# 12 |  (libraries stdune dune_lang)
#                         ^^^^^^^^^
# Error: Library "dune_lang" not found.
# Hint: try: dune external-lib-deps --missing -p dune-action-plugin -j 2 @install
#   @runtest
# File "test/unit-tests/ocaml-config/dune", line 3, characters 19-31:
# 3 |  (libraries stdune ocaml_config)
#                        ^^^^^^^^^^^^
# Error: Library "ocaml_config" not found.
# Hint: try: dune external-lib-deps --missing -p dune-action-plugin -j 2 @install
#   @runtest
# File "test/unit-tests/dune", line 4, characters 19-28:
# 4 |  (libraries stdune dune_lang jbuild_support))
#                        ^^^^^^^^^
# Error: Library "dune_lang" not found.
# Hint: try: dune external-lib-deps --missing -p dune-action-plugin -j 2 @install
#   @runtest
# File "test/unit-tests/artifact_substitution/dune", line 3, characters 30-37:
# 3 |  (libraries stdune dune fiber dune_re)
#                                   ^^^^^^^
# Error: Library "dune_re" not found.
# Hint: try: dune external-lib-deps --missing -p dune-action-plugin -j 2 @install
#   @runtest

[camelus]

Commit: f8bd086

A pull request by opam-seasoned @rgrinberg.

☀️ All lint checks passed f8bd086

• These packages passed lint tests: dune-action-plugin.2.0.0+draft1, dune-build-info.2.0.0+draft1, dune-configurator.2.0.0+draft1, dune-glob.2.0.0+draft1, dune-private-libs.2.0.0+draft1, dune.2.0.0+draft1

---

☀️ Installability check (+6)

• new installable packages (6): dune.2.0.0+draft1 dune-action-plugin.2.0.0+draft1 dune-build-info.2.0.0+draft1 dune-configurator.2.0.0+draft1 dune-glob.2.0.0+draft1 dune-private-libs.2.0.0+draft1

[kit-ty-kate]

Should I do a PR restricting all the packages that are not compatible with dune 2.0 or do you want to have a look at all the failures first and do the PR yourself?

[rgrinberg]

Your help would be appreciated :)

…

On Nov 1, 2019, 8:50 PM +0900, Kate ***@***.***>, wrote: Should I do a PR restricting all the packages that are not compatible with dune 2.0 or do you want to have a look at all the failures first and do the PR yourself? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[kit-ty-kate]

The custom run on check.ocamllabs.io has finished. I'm going through it right now. Here is the diff you want to go through it as well (tmp url): http://check.ocamllabs.io:8080/diff/1572627330-871c2f16efe7de104a12d93452e99cb8fab92d13..1572726647-ace99caf8ab213b131e231ca974980b041f0ea90

While looking at the errors, this one does seem a little strange. Does this error seem ok to you?

#=== ERROR while compiling 0install.2.14 ======================================#
# context              2.0.5 | linux/x86_64 | ocaml-base-compiler.4.06.1 | file:///home/opam/opam-repository
# path                 ~/.opam/4.06.1/.opam-switch/build/0install.2.14
# command              /usr/bin/make all
# exit-code            2
# env-file             ~/.opam/log/0install-6-9b7fd4.env
# output-file          ~/.opam/log/0install-6-9b7fd4.out
### output ###
# dune build --root=. --profile=release @install
#        ocaml (internal)
# ocamlfind: Package `lablgtk2' not found
# (will skip building the GTK plugin)
# File "ocaml/zeroinstall/dune", line 3, characters 23-32:
# 3 |  (modules (:standard \ With_dbus Without_dbus))
#                            ^^^^^^^^^
# Error: Module With_dbus is excluded but it doesn't exist.
# make: *** [Makefile:44: all] Error 1

[kit-ty-kate]

Also I'm noticing something: I thought dune-project file was going to be mandatory in dune 2.0. Has this been postponed?

[ghost]

It's slightly different: it's required to mark the root of a project in a multi-project workspace. Before, having a file foo/foo.opam was enough for Dune to decide that foo was the root of a project. It's no longer a case with Dune 2.0.0 and unless there is also a file foo/dune-project, Dune will consider that foo is part of the same project as the root directory. It's a bit cleaner as the old convention was a bit magical.

In any case, this only affects users who use a mono-repo style workflow.

[ghost]

Regarding the 0install error, it seems to be a capitalisation issue as the file is named with_dbus.ml.

[kit-ty-kate]

It's slightly different: it's required to mark the root of a project in a multi-project workspace. Before, having a file foo/foo.opam was enough for Dune to decide that foo was the root of a project. It's no longer a case with Dune 2.0.0 and unless there is also a file foo/dune-project, Dune will consider that foo is part of the same project as the root directory. It's a bit cleaner as the old convention was a bit magical.

In any case, this only affects users who use a mono-repo style workflow.

ok, thanks for the explanation.

Regarding the 0install error, it seems to be a capitalisation issue as the file is named with_dbus.ml.

Ok so I'm guessing it's a new hard error for dune 2.0. I finished my first pass on the diff and opened #15206, however I've left out a few of the packages with less straightforward errors like this and will open a second PR shortly.

[ghost]

This looks like a bug in Dune actually, we are looking at it

[kit-ty-kate]

Ok, then here are the remaining failures. Tell me if some of them are normal and require the packages to be constrain:

#=== ERROR while compiling herdtools7.7.54 ====================================#
# context              2.0.5 | linux/x86_64 | ocaml-base-compiler.4.06.1 | file:///home/opam/opam-repository
# path                 ~/.opam/4.06.1/.opam-switch/build/herdtools7.7.54
# command              /bin/sh ./dune-build.sh /home/opam/.opam/4.06.1
# exit-code            1
# env-file             ~/.opam/log/herdtools7-6-a19e5c.env
# output-file          ~/.opam/log/herdtools7-6-a19e5c.out
### output ###
# Info: Creating file dune-project with this contents:
# | (lang dune 2.0)
# | (name herdtools7)
# File "litmus/tst.ml", line 24, characters 54-65:
# Error: Unbound value CLexer.main
# File "gen/critical.ml", line 40, characters 10-20:
# Error: Unbound module Top

#=== ERROR while compiling irmin-watcher.0.4.0 ================================#
# context              2.0.5 | linux/x86_64 | ocaml-base-compiler.4.06.1 | file:///home/opam/opam-repository
# path                 ~/.opam/4.06.1/.opam-switch/build/irmin-watcher.0.4.0
# command              ~/.opam/4.06.1/bin/dune build -p irmin-watcher -j 71
# exit-code            1
# env-file             ~/.opam/log/irmin-watcher-6-22c189.env
# output-file          ~/.opam/log/irmin-watcher-6-22c189.out
### output ###
# File "src/dune", line 3, characters 33-40:
# 3 |   (modules     core hook backend polling irmin_watcher)
#                                      ^^^^^^^
# Error: Module Polling doesn't exist.

#=== ERROR while compiling links-postgresql.0.8 ===============================#
# context              2.0.5 | linux/x86_64 | ocaml-base-compiler.4.06.1 | file:///home/opam/opam-repository
# path                 ~/.opam/4.06.1/.opam-switch/build/links-postgresql.0.8
# command              ~/.opam/4.06.1/bin/dune build -p links-postgresql -j 71
# exit-code            1
# env-file             ~/.opam/log/links-postgresql-6-53634f.env
# output-file          ~/.opam/log/links-postgresql-6-53634f.out
### output ###
# File "unit/dune", line 3, characters 11-21:
# 3 |   (package links-unit)
#                ^^^^^^^^^^
# Error: The current scope doesn't define package "links-unit".
# The only packages for which you can declare elements to be installed in this
# directory are:
# - links            (because of links.opam)
# - links-postgresql (because of links-postgresql.opam)

#=== ERROR while compiling minisat.0.2 ========================================#
# context              2.0.5 | linux/x86_64 | ocaml-base-compiler.4.06.1 | file:///home/opam/opam-repository
# path                 ~/.opam/4.06.1/.opam-switch/build/minisat.0.2
# command              ~/.opam/4.06.1/bin/dune build -p minisat
# exit-code            1
# env-file             ~/.opam/log/minisat-6-4eee66.env
# output-file          ~/.opam/log/minisat-6-4eee66.out
### output ###
# Error: Path src/minisat has already been scanned. Cannot scan it again
# through symlink src/minisat-C-1.14.1

#=== ERROR while compiling npy.0.0.8 ==========================================#
# context              2.0.5 | linux/x86_64 | ocaml-base-compiler.4.06.1 | file:///home/opam/opam-repository
# path                 ~/.opam/4.06.1/.opam-switch/build/npy.0.0.8
# command              ~/.opam/4.06.1/bin/dune build -p npy -j 71
# exit-code            1
# env-file             ~/.opam/log/npy-6-42ef32.env
# output-file          ~/.opam/log/npy-6-42ef32.out
### output ###
# Info: Creating file dune-project with this contents:
# | (lang dune 2.0)
# | (name npy)
#       ocamlc src/.npy.objs/byte/npy.{cmo,cmt} (exit 2)
# (cd _build/default && /home/opam/.opam/4.06.1/bin/ocamlc.opt -w -40 -g -bin-annot -I src/.npy.objs/byte -I /home/opam/.opam/4.06.1/lib/camlzip -intf-suffix .ml -no-alias-deps -o src/.npy.objs/byte/npy.cmo -c -impl src/npy.ml)
# File "src/npy.ml", line 407, characters 17-28:
# Error: Unbound module Zip
#     ocamlopt src/.npy.objs/native/npy.{cmx,o} (exit 2)
# (cd _build/default && /home/opam/.opam/4.06.1/bin/ocamlopt.opt -w -40 -g -I src/.npy.objs/byte -I src/.npy.objs/native -I /home/opam/.opam/4.06.1/lib/camlzip -intf-suffix .ml -no-alias-deps -o src/.npy.objs/native/npy.cmx -c -impl src/npy.ml)
# File "src/npy.ml", line 407, characters 17-28:
# Error: Unbound module Zip

And the weirder for the end:

#=== ERROR while compiling lwt-pipeline.0.1 ===================================#
# context              2.0.5 | linux/x86_64 | ocaml-base-compiler.4.06.1 | file:///home/opam/opam-repository
# path                 ~/.opam/4.06.1/.opam-switch/build/lwt-pipeline.0.1
# command              ~/.opam/4.06.1/bin/dune build -p lwt-pipeline -j 71
# exit-code            1
# env-file             ~/.opam/log/lwt-pipeline-6-1804c7.env
# output-file          ~/.opam/log/lwt-pipeline-6-1804c7.out
### output ###
# [...]
# File "src/pipeline.ml", line 75, characters 21-43:
# Error: This expression has type ('a, 'b) Result.result Lwt.t
#        but an expression was expected of type ('c, 'd) result Lwt.t
#        Type ('a, 'b) Result.result is not compatible with type
#          ('c, 'd) result 
#     ocamlopt src/.pipeline.objs/native/pipeline.{cmx,o} (exit 2)
# (cd _build/default && /home/opam/.opam/4.06.1/bin/ocamlopt.opt -w -40 -g -I src/.pipeline.objs/byte -I src/.pipeline.objs/native -I /home/opam/.opam/4.06.1/lib/lwt -intf-suffix .ml -no-alias-deps -o src/.pipeline.objs/native/pipeline.cmx -c -impl src/pipeline.ml)
# File "src/pipeline.ml", line 75, characters 21-43:
# Error: This expression has type ('a, 'b) Result.result Lwt.t
#        but an expression was expected of type ('c, 'd) result Lwt.t
#        Type ('a, 'b) Result.result is not compatible with type
#          ('c, 'd) result 

All of them were compiling fine before and there has been no changes in the opam-repository (same base commit, same dependencies taken in every cases).

[kit-ty-kate]

I had a bit of digging at the error in lwt-pipeline and it seems to be linked to the fact that dune used to insert automatically -I ~/.opam/<switch>/lib/result -I ~/.opam/<switch>/lib/seq to the ocamlc/ocamlopt calls and it's no longer the case.

[emillon]

for npy, it's caused by the fact that dune 2 does not bring transitive deps implicitly (camlzip brings zip).

[rgrinberg]

That’s the correct behaviour. Note that this can only cause breakage for users that did not commit a dune-project file. In which case dune will interpret the project as using 2.0. Our backwards compatibility policy applies to people that fix the due language version with a dune-project file.

…

On Nov 5, 2019, 1:58 AM +0900, Kate ***@***.***>, wrote: I had a bit of digging at the error in lwt-pipeline and it seems to be linked to the fact that dune used to insert automatically -I ~/.opam/<switch>/lib/result -I ~/.opam/<switch>/lib/seq to the ocamlc/ocamlopt calls and it's no longer the case. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[rgrinberg]

While looking at the errors, this one does seem a little strange. Does this error seem ok to you?

Here's the offending dune file:

(library
 (name        zeroinstall)
 (modules (:standard \ With_dbus Without_dbus))
 (modules_without_implementation feed_provider progress sigs ui)
 (libraries curl curl.lwt dynlink lwt_react support yojson
            (select dbus.ml from
             (obus.network-manager
              obus.notification    -> with_dbus.ml)
             (                     -> without_dbus.ml))))

I had a closer look and I think this is an issue with 0install. The way to fix it is to simply get rid of this modules field. There's no need to exclude sources that are used in select, dune already takes care of that automatically.

[ghost]

Hmm, are you sure about this?

[rgrinberg]

I tried to reproduce this issue but I was only able to reproduce this issue in conjunction with select. Note that casing doesn’t matter here and I can reproduce it with lowercased names. That’s the extent of my certainty. But really, I’m not sure what’s the correct behaviour here. On one hand, these selected modules are all valid sources. On the other hand they’re sources for a completely different module after selection. So I think the question is what is the most intuitive behavior.

…

On Nov 5, 2019, 9:20 PM +0900, Jérémie Dimino ***@***.***>, wrote: Hmm, are you sure about this? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[ghost]

Now that you mention it, this dune file certainly looks dodgy. If that's the only failure of this type, I suggest that we mark 0install incompatible with Dune 2.0.0 and send a patch to the author.

[kit-ty-kate]

Ok so 0install, npy and lwt-pipeline should be marked as incompatible for sure.
Am I guessing correctly if I say herdtools7, irmin-watcher and links* should also be marked as incompatible?
What about the error in minisat? The error seems to be of a different kind

[avsm]

I'd like to release a new version of irmin-watcher so that we don't end up with an upper bound constraint on dune 2.0 when its merged. What's the right fix here? It looks like a straightforward use of (select) that is now incompatible with dune 2.0 for some reason...

[rgrinberg]

For irmin-watcher, I think the fix should be simple and I’ve put up a PR that will make it compatible with dune 2.0 and dune 1.x. For 0install, the way that should be compatible with dune 1.x and 2.x would be to just list out all the modules. @diml It seems like we need an explicit modules when we use select with dune 1.x. Unfortunately this module list doesn’t work for 2.0 because it excludes non existent modules. We could allow people to use these module names for exclusions in 2.x. Currently, the way to write a select that is compatible with 2.0 and 1.x is to always write the modules field exhaustively without using OSL.

…

On Nov 6, 2019, 2:42 AM +0900, Anil Madhavapeddy , wrote: I'd like to release a new version of irmin-watcher so that we don't end up with an upper bound constraint on dune 2.0 when its merged. What's the right fix here? It looks like a straightforward use of (select) that is now incompatible with dune 2.0 for some reason... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[ghost]

There is another solution; the various choices don't have to be proper <module-name>.ml files:

(library
 (name        zeroinstall)
 (modules_without_implementation feed_provider progress sigs ui)
 (libraries curl curl.lwt dynlink lwt_react support yojson
            (select dbus.ml from
             (obus.network-manager
              obus.notification    -> dbus.with.ml)
             (                     -> dbus.without.ml))))

This is in fact how I imagined select would be used initially.

[rgrinberg]

There is another solution; the various choices don't have to be proper .ml files:

Should we just make users follow this convention then? I think it's unlikely they'll think of this themselves otherwise.

[rgrinberg]

I’ve submitted upstream fixes for 0install and irmin-watcher: 0install/0install#117 mirage/irmin-watcher#25

…

On Nov 6, 2019, 5:02 PM +0900, Jérémie Dimino ***@***.***>, wrote: There is another solution; the various choices don't have to be proper <module-name>.ml files: (library (name zeroinstall) (modules_without_implementation feed_provider progress sigs ui) (libraries curl curl.lwt dynlink lwt_react support yojson (select dbus.ml from (obus.network-manager obus.notification -> dbus.with.ml) ( -> dbus.without.ml)))) This is in fact how I imagined select would be used initially. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[ghost]

Should we just make users follow this convention then? I think it's unlikely they'll think of this themselves otherwise.

That would be best yeah. The only thing is that select is currently too generic for its own good, i.e. the destination file could be anything and doesn't have to be a .ml file. Though we could restrict select in 2.0.

[bobot]

FI the PR which removed the list of file names from select ocaml/dune#2506.

[rgrinberg]

There seems to be a difference in the way we handle symlinks. This is likely due to my change to port dune to use memoization in file tree. The symlink in minisat for example, is:

$ ls src/ minisat -> minisat-C-1.14.1 minisat-C-1.14.1 …

That indeed seems like something that should be forbidden. I’m actually not sure how the old code allowed this.

[rgrinberg]

All the include and missing module errors are accounted for. It's just the lack of dune-project file that is forcing these projects to use lang dune 2.0 before they're ready for it. The links issue seems to be another metadata issue - links-unit is not specified anywhere and the issue is fixed in the latest version.

[kit-ty-kate]

The rest of the packages should be constrained by #15229. I'll run another global check once this is done and if all is green we can call it successful.

[kit-ty-kate]

Ok everything is green now (except some unrelated failures) ... except just one package:

#=== ERROR while compiling dokeysto_lz4.3.0.1 =================================#
# context              2.0.5 | linux/x86_64 | ocaml-base-compiler.4.06.1 | file:///home/opam/opam-repository
# path                 ~/.opam/4.06.1/.opam-switch/build/dokeysto_lz4.3.0.1
# command              ~/.opam/4.06.1/bin/dune build -p dokeysto_lz4 -j 71
# exit-code            1
# env-file             ~/.opam/log/dokeysto_lz4-6-17e921.env
# output-file          ~/.opam/log/dokeysto_lz4-6-17e921.out
### output ###
# Info: Creating file dune-project with this contents:
# | (lang dune 2.0)
# | (name dokeysto)
#       ocamlc src/.dokeysto_cli.eobjs/byte/dune__exe__Dokeysto_cli.{cmi,cmo,cmt} (exit 2)
# (cd _build/default && /home/opam/.opam/4.06.1/bin/ocamlc.opt -w -40 -g -bin-annot -I src/.dokeysto_cli.eobjs/byte -I /home/opam/.opam/4.06.1/lib/dolog -I /home/opam/.opam/4.06.1/lib/minicli -I src/.dokeysto_lz4.objs/byte -I src/.dokeysto_lz4.objs/native -no-alias-deps -o src/.dokeysto_cli.eobjs/byte/dune__exe__Dokeysto_cli.cmo -c -impl src/dokeysto_cli.ml)
# File "src/dokeysto_cli.ml", line 5, characters 14-28:
# Error: Unbound module Dokeysto

I tested locally and with (lang dune 1.11) it works but I can't figure out why it doesn't work with dune 2.0. Did something changed when one library is defined locally but not built (not given to -p) but is available globally via opam?

Could you look into it?

[rgrinberg]

Could you look into it?

I think it's the same implicit_transitive_deps issue. This value is true for 1.11 but false for 2.0. This package does not have a dune-project file hence it's now automatically upgraded to 2.0 even though that is unintended.

[kit-ty-kate]

Oh, right, I was looking at the wrong thing in the dune file, my bad. Thanks!
All the failures are now dealt with. You can go forward with the release whenever you want.

[avsm]

The Irmin-watcher PR (mirage/irmin-watcher#25) seems to fail on dune 1.11.4 and I won't have time to look at it for a few days, so I just wanted to flag that up -- I'm still not entirely clear what a dune 1.x and 2.x compatible solution for (select) is.

[rgrinberg]

I pushed a commit that should fix this issue. Thanks for bringing this to my attention.

…

On Nov 7, 2019, 11:48 PM +0900, Anil Madhavapeddy ***@***.***>, wrote: The Irmin-watcher PR (mirage/irmin-watcher#25) seems to fail on dune 1.11.4 and I won't have time to look at it for a few days, so I just wanted to flag that up -- I'm still not entirely clear what a dune 1.x and 2.x compatible solution for (select) is. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[rgrinberg]

Second draft here: #15261