[feature] Improve compatibility with compiler caches like ccache or sccache

[monsdar]

We recently set up sccache with a local S3 storage for our CI infrastructure. With this runners should be able to reuse cached compilations instead of recompiling from source all the time. The problem now is that due to how Conan stores packages internally the cache is not used in most cases:

sccache (and ccache or other such solutions) use the absolute path of the source-files to determine whether a source-file has changed. With Conan the absolute path to the sources is relying on attributes of the package: $CONAN_HOME/data/<name>/<version>/<user>/<channel>. Dependencies also have their path changed frequently: $CONAN_HOME/data/bar/.5.6/otheruser/otherchannel/package/<someSHA>/. See this comment for reference.

Especially in our case this is a hard problem, as we extend SemVer with metadata regarding the Git SHA and Build ID, so versions are changing quite a bit: Foo/1.0.0+abcdefg.GL00137@user/channel. Pathes are never the same between builds.

From mozilla/sccache#35 it does not seem that sccache is interested in solving this, it's quite a complex problem for them as well.

I'd like to ask if anyone on the Conan-side has experience with this type of scenario. Perhaps there's already a solution that'd help, perhaps we can come up with an idea to enable a better usage of compile caches with Conan.

• I've read the CONTRIBUTING guide.

[monsdar]

One idea I could come up would be to add a new option for Conan:

• conan_use_central_build_directory would always build the package in a simplified path like $CONAN_HOME/build/<name>
• After a successful build it would copy the resulting package into the more complex final path $CONAN_HOME/data/<name>/<version>/<user>/<channel>
• When conan_use_central_build_directory is activated it will run builds sequentially to make sure no two builds are using the same build directory. Could be achieved with a simple lock to the directory.

A solution like this would be non-breaking and easy to adopt. Absolute pathes would not differ between builds, so support for ccache, scchache is given. Having the builds to be run sequentially is not a problem within a CI scenario (and most user-scenarios as well I guess).

Is this too shortsighted and I've missed an important bit?

[jgsogo]

Hi, @monsdar . It is not the first time we get some issues related to compiler cache systems and Conan, and so far we haven't come up with any solution.... but if any, I think it should follow your approach: an opt-in where the user takes all the responsibility if something fails.

There are some issues that will make it very hard for Conan to adopt it as default:

• concurrency between different Conan processes running at the same time, it is really hard, and we have some issues still opened related to the locking mechanism we have in the cache, it is not easy to find/develop a system-level locking mechanism 😞
• if the source folder is shared between different versions, then every build needs to remove it and copy/download the sources again, this can be time-consuming for big packages... (we can think about a workaround for this one).

I would love to see implemented the CCACHE_BASEDIR, it would be much easier for us. Let's tag this one as a look into, but mostly with Conan 2.0 in mind, there were some proposals to add multiple revisions to the Conan cache and also to obfuscate the Conan cache to get shorter paths (and store in a database the matching between a configuration and the path), this can be considered as an opt-in for some scenarios.

[monsdar]

Both issues would not apply in our CI usecase:

• Only one build process runs per job per runner
• Builds always work on a freshly cloned repository

Due to the opt-in people cannot really expect these to be solved, for someone using this for CI these shouldn't be blockers. Of course users will see that feature and try to use for different purposes and then ask for proper locking and incremental builds :)

[drussel]

This issue probably can be handled as part of the generating the data for the configuration (like conanbuildinfo.cmake)

• create a directory in your build directory with symlinks to the include/lib dirs for the referenced conan packages
• have the CONAN_PKG targets in conanbuildinfo.cmake use these instead of the absolute paths

This sort of approach would have the advantages that all user-wide data (in .conan) keeps the safe paths and all decisions that potentially impact compile time safety are on a per-project basis.

I think windows now has working symlinks now too.