Pinning local package warns of rsync permission error

[ElectreAAS]

I didn't expect a simple opam pin foo -yn . to print 14K lines of error, but it happened.

Steps to reproduce

Create a simple project:

# In dune-project
(lang dune 3.7)

(generate_opam_files true)

(package
 (name foo))

# In lib/dune
(library
 (public_name foo))

# In lib/foo.ml
let f x y = x + y

# In root_file.txt
Hello

Run dune build to generate opam file.
Run sudo chown root:root root_file.txt and sudo chmod 600 root_file.txt to make it unavailable to the current user.

Problem

opam pin -yn . produces the following output:

[WARNING] Rsync partially failed:
          sending incremental file list
          ./
          dune-project
          foo.opam
          rsync: send_files failed to open "/home/ambre/opam-bug/root_file.txt": Permission denied (13)
          lib/
          lib/dune
          lib/foo.ml

          sent 1,174 bytes  received 160 bytes  2,668.00 bytes/sec
          total size is 511  speedup is 0.38
          rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1207) [sender=3.1.3]

Package foo does not exist, create as a NEW package? [Y/n] y
foo is now pinned to file:///home/ambre/opam-bug (version ~dev)

Environment check

$ opam config report
# opam config report
# opam-version         2.1.0 
# self-upgrade         no
# system               arch=x86_64 os=linux os-distribution=ubuntu os-version=20.04
# solver               builtin-mccs+glpk
# install-criteria     -removed,-count[avoid-version,changed],-count[version-lag,request],-count[version-lag,changed],-count[missing-depexts,changed],-changed
# upgrade-criteria     -removed,-count[avoid-version,changed],-count[version-lag,solution],-count[missing-depexts,changed],-new
# jobs                 7
# repositories         1 (http) (default repo at 094b4999)
# pinned               0
# current-switch       self
# ocaml:native         true
# ocaml:native-tools   true
# ocaml:native-dynlink true
# ocaml:stubsdir       /home/ambre/.opam/self/lib/ocaml/stublibs:/home/ambre/.opam/self/lib/ocaml
# ocaml:preinstalled   false
# ocaml:compiler       5.0.0

[kit-ty-kate]

There are two different issues in this:

• When using the rsync backend it is more or less expected to have rsync yell because it doesn't have access to the potentially critical files you're trying to use. However we could try to ask rsync to reduce the verbosity of the error (I'm not sure if there is an option for that though)
• When using the git backend however the issue still appears and that's quite surprising because the directory isn't part of the git repository and should be ignored. @rjbou is there a reason why we call rsync even when we're fetching the directory using git?

[ElectreAAS]

Note that in the real scenario, the root-owned files are inside foo/bar/root_here, and there exists a file foo/.gitignore which mentions bar/, but that information isn't in the toplevel .gitignore.
Hope that is useful info

[rjbou]

On failure, we display stderr. It's hard to truncate stderr and be sure to give enough information to user on what's failing.

On git & rsync, it is mainly used with working-dir: we don't only take versioned files.

You directory isn't versioned, so opam automatically made a path pin

foo is now pinned to file:///home/ambre/opam-bug (version ~dev)

[kit-ty-kate]

On git & rsync, it is mainly used with working-dir: we don't only take versioned files.

I don’t get that. What do you mean? When the git repository is local we don’t actually use git clone to copy the repository?

[rjbou]

When the git repository is local we don’t actually use git clone to copy the repository?

It is. In the example that you gave (step to reproduce), there is no git repository, i was talking about that. You have the same rsync issue with a git pinned repo ?

[kit-ty-kate]

as said in #5469 (comment), yes.

[rjbou]

Opened another issue for rsync+git+permission warning #5472.

I don’t get that. What do you mean? When the git repository is local we don’t actually use git clone to copy the repository?

We do, but afterwards we rsync non-versioned files.

[rjbou]

From dev meeting: Check rsync option to minimise its error output, and if possible link it to opam verbosity level.