Is cached_digest doing extra work on cmt files?

[ejgallego]

Dear dune devs,

I have been observing some re-hashing of files on what should be a no-op rebuild, example from the Coq sources after doing dune build coq.install after a previous full build:

[dirty / mtime] _build/install/default/lib/coq/vernac/vernacextend.cmt {old: 1613989865.805196/ new: 1613990621.966763}
[dirty / perm ] _build/install/default/lib/coq/vernac/vernacextend.cmt {old: 292/ new: 511}
[dirty / size ] _build/install/default/lib/coq/vernac/vernacextend.cmt {old: 112027/ new: 64}
[dirty / mtime] _build/install/default/lib/coq/vernac/vernacextend.cmti {old: 1613989865.479193/ new: 1613990621.967763}
[dirty / perm ] _build/install/default/lib/coq/vernac/vernacextend.cmti {old: 292/ new: 511}
[dirty / size ] _build/install/default/lib/coq/vernac/vernacextend.cmti {old: 40355/ new: 65}

so indeed, all .cmt{,i} files in the source tree seem to be re-digested, however it seems to me they should not. After the digest, nothing is rebuilt, I guess the digest matches the stored one, tho cmt tend to be leaves themselves.

• dune version: master
• OCaml version: 4.12.0-rc1

[ghost]

There is a known reproducibility issue with cmt/cmti files, so that might just be it. /cc @snowleopard who has been observing such issues inside Jane Street.

[snowleopard]

I think @jeremiedimino refers to this issue with non-reproducible cmt files: ocaml/ocaml#9991.

However, it seems to me that @ejgallego's report may be about a different problem: in this case we expect no files to be rebuild at all, but for some reason some cmt files get to be rebuild (and fortunately the results are deterministic: if the above bug with non-reproducible cmts actually occurred we would expect to see further rebuilds).

[ghost]

I see. This reminds of a conversation we had some time ago, where we said it'd be nice to have a way to debug what files Dune's consider as changed. To understand better why it re-run some rules.

[ejgallego]

I don't think the issue is specific to cmt files, I could reproduce on others; it seems to me that it is due to symlinks not being tracked properly.

[ejgallego]

I see. This reminds of a conversation we had some time ago, where we said it'd be nice to have a way to debug what files Dune's consider as changed. To understand better why it re-run some rules.

I have written some support like that a few times; I'd be happy to finish the support, but a couple of issues remain:

• main issue is how to integrate that without affecting the hot path, if guess a simple boolean shouldn't be too hard, but I still worry about that
• should we limit explanations to hashes, or to full rules?

[ghost]

main issue is how to integrate that without affecting the hot path, if guess a simple boolean shouldn't be too hard, but I still worry about that

I think so as well, but it'd be easier to reason about this with the code. So I'd just open a PR and we can discuss this during the review.

should we limit explanations to hashes, or to full rules?

I'd say whatever is the most useful for debugging. If both are useful, it seems fine to me to support both.

[ejgallego]

This seems fixed now, likely some of the PRs related to #4412