Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"Rule failed to generate the following targets" with virtual package #4195

Closed
dannywillems opened this issue Feb 4, 2021 · 4 comments · Fixed by #4233
Closed

"Rule failed to generate the following targets" with virtual package #4195

dannywillems opened this issue Feb 4, 2021 · 4 comments · Fixed by #4233

Comments

@dannywillems
Copy link
Contributor

I am working on the JavaScript support for a package called bls12-381 implementing the curve BLS12-381 using a Rust optimized implementation.
The initial version of the package bls12-381 was only for UNIX, and was using an intermediate Rust dependency called rustc-bls12-381.
The JavaScript part uses jsoo to call a wasm version of the rust code given above.

I decided to go with the following structure:

  • bls12-381 -> virtual package. Code located under src/virtual
  • bls12-381-unix -> UNIX version, implementing bls12-381, using the rust code compiled into a static library. Code located under src/unix
  • bls12-381-js -> JavaScript version, implementing bls12-381, using jsoo to wrap a NPM package containing the wasm code and providing a JavaScript layer above this wasm code. Code located under src/js, mostly calling bls12-381-js-gen under src/js-gen.

Intermediate packages are used for modularity/to ease the implementation of UNIX/JS support (bls12-381-gen, bls12-381-js-gen). So the structure is more:

- bls12-381-gen
  -> bls12-381
  |  --> bls12-381-js-gen
  |  | --> bls12-381-js
  | --> bls12-381-unix

When building and running the tests with dune build @runtest, everything is fine. However, when I try to install with opam (see below), I get a bunch of the following error for different files:

# Error: Rule failed to generate the following targets:
# - src/unix/.bls12_381_unix.objs/native/bls12_381.cmx

Expected Behavior

Compile without errors.

Actual Behavior

$ opam install bls12-381-unix

<><> Synchronising pinned packages ><><><><><><><><><><><><><><><><><><><><><><>
[bls12-381-unix.dev] no changes from file:///home/danny/codes/dannywillems/ocaml-bls12-381

The following actions will be performed:
  ∗ install bls12-381-unix dev*
[bls12-381-unix.dev] synchronised from file:///home/danny/codes/dannywillems/ocaml-bls12-381

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
[ERROR] The compilation of bls12-381-unix failed at "/home/danny/.opam/opam-init/hooks/sandbox.sh build dune build -j 3 -p bls12-381-unix @install".

#=== ERROR while compiling bls12-381-unix.dev =================================#
# context     2.0.6 | linux/x86_64 | ocaml-base-compiler.4.09.1 | pinned(file:///home/danny/codes/dannywillems/ocaml-bls12-381)
# path        ~/codes/dannywillems/ocaml-bls12-381/_opam/.opam-switch/build/bls12-381-unix.dev
# command     ~/.opam/opam-init/hooks/sandbox.sh build dune build -j 3 -p bls12-381-unix @install
# exit-code   1
# env-file    ~/.opam/log/bls12-381-unix-1063079-d2c37b.env
# output-file ~/.opam/log/bls12-381-unix-1063079-d2c37b.out
### output ###
# [...]
# File "src/dune_rules/virtual_rules.ml", line 40, characters 30-37:
# Error: Rule failed to generate the following targets:
# - src/unix/.bls12_381_unix.objs/byte/bls12_381__G2.cmi
# File "src/dune_rules/virtual_rules.ml", line 40, characters 30-37:
# Error: Rule failed to generate the following targets:
# - src/unix/.bls12_381_unix.objs/byte/bls12_381__Pairing.cmi
# File "src/dune_rules/virtual_rules.ml", line 40, characters 30-37:
# Error: Rule failed to generate the following targets:
# - src/unix/.bls12_381_unix.objs/native/bls12_381.cmx
# File "src/dune_rules/virtual_rules.ml", line 40, characters 30-37:
# Error: Rule failed to generate the following targets:
# - src/unix/.bls12_381_unix.objs/native/bls12_381.o



<><> Error report <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
┌─ The following actions failed
│ λ build bls12-381-unix dev
└─ 
╶─ No changes have been performed

$ opam install bls12-381-js

<><> Synchronising pinned packages ><><><><><><><><><><><><><><><><><><><><><><>
[bls12-381-js.dev] synchronised from file:///home/danny/codes/dannywillems/ocaml-bls12-381
[bls12-381-js] Installing new package description from upstream file:///home/danny/codes/dannywillems/ocaml-bls12-381

The following actions will be performed:
  ∗ install bls12-381-js-gen dev* [required by bls12-381-js]
  ∗ install bls12-381-js     dev*
===== ∗ 2 =====
Do you want to continue? [Y/n] Y
[bls12-381-js.dev] synchronised from file:///home/danny/codes/dannywillems/ocaml-bls12-381
[bls12-381-js-gen.dev] synchronised from file:///home/danny/codes/dannywillems/ocaml-bls12-381

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
∗ installed bls12-381-js-gen.dev
[ERROR] The compilation of bls12-381-js failed at "/home/danny/.opam/opam-init/hooks/sandbox.sh build dune build -j 3 -p bls12-381-js @install".

#=== ERROR while compiling bls12-381-js.dev ===================================#
# context     2.0.6 | linux/x86_64 | ocaml-base-compiler.4.09.1 | pinned(file:///home/danny/codes/dannywillems/ocaml-bls12-381)
# path        ~/codes/dannywillems/ocaml-bls12-381/_opam/.opam-switch/build/bls12-381-js.dev
# command     ~/.opam/opam-init/hooks/sandbox.sh build dune build -j 3 -p bls12-381-js @install
# exit-code   1
# env-file    ~/.opam/log/bls12-381-js-1063579-038aac.env
# output-file ~/.opam/log/bls12-381-js-1063579-038aac.out
### output ###
# [...]
# File "src/dune_rules/virtual_rules.ml", line 40, characters 30-37:
# Error: Rule failed to generate the following targets:
# - src/js/.bls12_381_js.objs/byte/bls12_381__Fr.cmi
# File "src/dune_rules/virtual_rules.ml", line 40, characters 30-37:
# Error: Rule failed to generate the following targets:
# - src/js/.bls12_381_js.objs/byte/bls12_381__G1.cmi
# File "src/dune_rules/virtual_rules.ml", line 40, characters 30-37:
# Error: Rule failed to generate the following targets:
# - src/js/.bls12_381_js.objs/byte/bls12_381__G2.cmi
# File "src/dune_rules/virtual_rules.ml", line 40, characters 30-37:
# Error: Rule failed to generate the following targets:
# - src/js/.bls12_381_js.objs/byte/bls12_381__Pairing.cmi

Reproduction

  1. git clone https://gitlab.com/dannywillems/ocaml-bls12-381 && cd ocaml-bls12-381
  2. opam switch create . 4.09.1 --no-install
    2b. git checkout remove-pin-depends # temporary, require a MR to be merged in ocaml/opam-repository
    2c. opam pin add tezos-rust-libs.1.1 "git+https://gitlab.com/tezos/tezos-rust-libs.git#f0a269087fe86dbe1cf471bb76e42fb4a481e226" # temporary, like 2b.
  3. opam pin add bls12-381-gen.dev ./ --no-action
  4. opam pin add bls12-381.dev . --no-action
  5. opam pin add bls12-381-unix.dev . --no-action
  6. opam pin add bls12-381-js-gen . --no-action
  7. opam pin add bls12-381-js . --no-action
  8. opam install bls12-381-gen --with-test
  9. opam install bls12-381 --with-test
  10. opam instal bls12-381-unix --with-test # will fail with the error above.
  11. opam instal bls12-381-js-gen --with-test # fine.
  12. opam instal bls12-381-js --with-test # will fail, with the error above.

However, when using dune build @runtest, it does work fine.

Specifications

  • Version of dune (output of dune --version): 2.8.2
  • Version of ocaml (output of ocamlc --version): 4.09.1
  • Operating system (distribution and version): ubuntu 20.04
  • Version of opam: 2.06.
@dannywillems
Copy link
Contributor Author

dannywillems commented Feb 4, 2021 •
edited
Loading

Running without sandbox in ~/codes/dannywillems/ocaml-bls12-381/_opam/.opam-switch/build/bls12-381-js.dev:

dune build -j 3 -p bls12-381-js @install

works fine
Running with the given path in the error (~/codes/dannywillems/ocaml-bls12-381/_opam/.opam-switch/build/bls12-381-js.dev) with the sandbox fails:

~/.opam/opam-init/hooks/sandbox.sh build dune build -j 3 -p bls12-381-js @install
File "src/dune_rules/virtual_rules.ml", line 40, characters 30-37:
Error: Rule failed to generate the following targets:
- src/js/.bls12_381_js.objs/byte/bls12_381.cmo
File "src/dune_rules/virtual_rules.ml", line 40, characters 30-37:
Error: Rule failed to generate the following targets:
- src/js/.bls12_381_js.objs/byte/bls12_381__Fq12.cmi
File "src/dune_rules/virtual_rules.ml", line 40, characters 30-37:
Error: Rule failed to generate the following targets:
- src/js/.bls12_381_js.objs/byte/bls12_381__Fr.cmi
File "src/dune_rules/virtual_rules.ml", line 40, characters 30-37:
Error: Rule failed to generate the following targets:
- src/js/.bls12_381_js.objs/byte/bls12_381__G1.cmi
File "src/dune_rules/virtual_rules.ml", line 40, characters 30-37:
Error: Rule failed to generate the following targets:
- src/js/.bls12_381_js.objs/byte/bls12_381__G2.cmi
File "src/dune_rules/virtual_rules.ml", line 40, characters 30-37:
Error: Rule failed to generate the following targets:
- src/js/.bls12_381_js.objs/byte/bls12_381__Pairing.cmi

Then, it looks like the issue comes from the sandbox. I continue to investigate.

@dannywillems
Copy link
Contributor Author

bls12-381-unix.0.4.1 has been merged in the public opam-repository, see ocaml/opam-repository#18112. The CI of opam-repository didn't detect it. However, the issue appears when running

opam install bls12-381-unix

@dannywillems dannywillems mentioned this issue Feb 15, 2021
Merged
@dannywillems
Copy link
Contributor Author

To avoid confusion, I am sometimes talking about bls12-381-js and sometimes about bls12-381-unix: the error appears for both, so i mix them. Looks like it is related to virtual packages.

@dannywillems
Copy link
Contributor Author

It looks like it is because there is a missing permission.
In the script sandbox.sh, there is:

[...]
# This case-switch should remain identical between the different sandbox implems
COMMAND="$1"; shift
case "$COMMAND" in
    build)
        add_mounts ro "$OPAM_SWITCH_PREFIX"
        add_mounts rw "$PWD"
        add_ccache_mount
        add_dune_cache_mount
        ;;
    install)
        add_mounts rw "$OPAM_SWITCH_PREFIX"
[...]
esac

https://github.com/ocaml/dune/blob/main/src/dune_rules/virtual_rules.ml#L40, where the exception is raised, the src and dst values are:

- Path src = /Users/dannywillems/.opam2/default/lib/bls12-381/bls12_381.cmo
- Path dest = _build/default/src/unix/.bls12_381_unix.objs/byte/bls12_381.cmo
- Path src = /Users/dannywillems/.opam2/default/lib/bls12-381/bls12_381.cmx
- Path dest = _build/default/src/unix/.bls12_381_unix.objs/native/bls12_381.cmx
- Path src = /Users/dannywillems/.opam2/default/lib/bls12-381/bls12_381.o
- Path dest = _build/default/src/unix/.bls12_381_unix.objs/native/bls12_381.o
- Path src =
- /Users/dannywillems/.opam2/default/lib/bls12-381/bls12_381__Fq12.cmi
- Path dest =
- _build/default/src/unix/.bls12_381_unix.objs/byte/bls12_381__Fq12.cmi
- Path src =
- /Users/dannywillems/.opam2/default/lib/bls12-381/bls12_381__Fq12.cmi
- Path dest =
- _build/default/src/unix/.bls12_381_unix.objs/public_cmi/bls12_381__Fq12.cmi

I then tried to simply add the write permissions on $OPAM_SWITCH_PREFIX (brutal method...), like this:

[...]
    build)
        add_mounts rw "$OPAM_SWITCH_PREFIX"  # HERE, ro -> rw.
        add_mounts rw "$PWD"
        add_ccache_mount
        add_dune_cache_mount
        ;;
[...]

and it does install without any issue.

dannywillems added a commit to dannywillems/dune that referenced this issue Feb 15, 2021
@dannywillems
Virtual_rules: use copy instead of symlink when copying signatures
cb43ba1 
Fix ocaml#4195
In sandbox, symlink requires rw permissions on the directory of the sources to create a symlink. Copy does not require
@dannywillems dannywillems mentioned this issue Feb 15, 2021
Merged
dannywillems added a commit to dannywillems/dune that referenced this issue Feb 15, 2021
@dannywillems
Virtual_rules: use copy instead of symlink when copying signatures
b3c3fa5 
Fix ocaml#4195
In sandbox, symlink requires rw permissions on the directory of the sources to create a symlink. Copy does not require
dannywillems added a commit to dannywillems/dune that referenced this issue Feb 15, 2021
@dannywillems
Virtual_rules: use copy instead of symlink when copying signatures
e449f2c 
Fix ocaml#4195
In sandbox, symlink requires rw permissions on the directory of the sources to create a symlink. Copy does not require
dannywillems added a commit to dannywillems/dune that referenced this issue Feb 15, 2021
@dannywillems
Virtual_rules: use copy instead of symlink when copying signatures
e9252e9 
Fix ocaml#4195
In sandbox, symlink requires rw permissions on the directory of the sources to create a symlink. Copy does not require
dannywillems added a commit to dannywillems/dune that referenced this issue Feb 15, 2021
@dannywillems
Virtual_rules: use copy instead of symlink when copying signatures
8e9ef85 
Fix ocaml#4195
In sandbox, symlink requires rw permissions on the directory of the sources to create a symlink. Copy does not require

Signed-off-by: Danny Willems <[email protected]>
dannywillems added a commit to dannywillems/dune that referenced this issue Feb 15, 2021
@dannywillems
Virtual_rules: use copy instead of symlink when copying signatures
466cefe 
Fix ocaml#4195
In sandbox, symlink requires rw permissions on the directory of the sources to create a symlink. Copy does not require

Signed-off-by: Danny Willems <[email protected]>
dannywillems added a commit to dannywillems/dune that referenced this issue Feb 15, 2021
@dannywillems
Virtual_rules: use copy instead of symlink when copying signatures
abf8a74 
Fix ocaml#4195
In sandbox, symlink requires rw permissions on the directory of the sources to create a symlink. Copy does not require

Signed-off-by: Danny Willems <[email protected]>
dannywillems added a commit to dannywillems/dune that referenced this issue Feb 16, 2021
@dannywillems
Use stat instead of lstat in refresh_and_chmod
80ea2f7 
Fix ocaml#4195

Signed-off-by: Danny Willems <[email protected]>
ghost pushed a commit to dannywillems/dune that referenced this issue Feb 16, 2021
@dannywillems @jeremiedimino
No longer call chmod on symbolic links
cbc5dbe 
Use stat instead of lstat in refresh_and_chmod

Fix ocaml#4195

Signed-off-by: Danny Willems <[email protected]>
@ghost ghost closed this as completed in #4233 Feb 16, 2021
ghost pushed a commit that referenced this issue Feb 16, 2021
@dannywillems
No longer call chmod on symbolic links (#4233)
3d07283 
By using stat instead of lstat in refresh_and_chmod

Fix #4195

Signed-off-by: Danny Willems <[email protected]>
rgrinberg pushed a commit that referenced this issue Mar 7, 2021
@dannywillems @rgrinberg
No longer call chmod on symbolic links (#4233)
7269719 
By using stat instead of lstat in refresh_and_chmod

Fix #4195

Signed-off-by: Danny Willems <[email protected]>
@rgrinberg rgrinberg mentioned this issue Mar 7, 2021
Closed
rgrinberg added a commit to rgrinberg/opam-repository that referenced this issue Mar 7, 2021
@rgrinberg
[new release] dune-build-info, dune, dune-configurator, dune-site, du…
26052da 
…ne-action-plugin, dune-private-libs and dune-glob (2.8.3)

CHANGES:

- Make `patdiff` show refined diffs (ocaml/dune#4257, fixes ocaml/dune#4254, @hakuch)

- Fixed a bug that could result in needless recompilation under Windows due to
  case differences in the result of `Sys.getcwd` (observed under `emacs`).
  (ocaml/dune#3966, @nojb).

- Restore compatibility with Coq < 8.10 for coq-lang < 0.3 , document
  that `(using coq 0.3)` does require Coq 8.10 at least (ocaml/dune#4224, fixes
  ocaml/dune#4142, @ejgallego)

- Add a META rule for 'compiler-libs.native-toplevel' (ocaml/dune#4175, @AltGr)

- No longer call `chmod` on symbolic links (fixes ocaml/dune#4195, @dannywillems)

- Dune no longer automatically create or edit `dune-project` files
  (ocaml/dune#4239, fixes ocaml/dune#4108, @jeremiedimino)

- Have `dune` communicate the location of the standard library directory to
  `merlin` (ocaml/dune#4211, fixes ocaml/dune#4188, @nojb)

- Workaround incorrect exception raised by Unix.utimes (OCaml PR#8857) in
  Path.touch on Windows (ocaml/dune#4223, @dra27)

- `dune ocaml-merlin` is now able to provide configuration for source files in
  the `_build` directory. (ocaml/dune#4274, @voodoos)

- Automatically delete left-over Merlin files when rebuilding for the first time
  a project previously built with Dune `<= 2.7`. (ocaml/dune#4261, @voodoos, @aalekseyev)

- Fix `ppx.exe` being compiled for the wrong target when cross-compiling
  (ocaml/dune#3751, fixes ocaml/dune#3698, @toots)

- `dune top` correctly escapes the generated toplevel directives, and make it
  easier for `dune top` to locate C stubs associated to concerned libraries.
  (ocaml/dune#4242, fixes ocaml/dune#4231, @nojb)

- Do not pass include directories containing native objects when compiling
  bytecode (ocaml/dune#4200, @nojb)
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
1 participant
@dannywillems