Improve fetching performance (mainly for CI) #10
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This may turn out to be too much (but it does at least build). Motivation: CI's git checkouts consume an unmanageably large amount of cache (over 3 GiB per snapshot out of 10 GiB quota), and must be cached for acceptable build speeds. Dropping submodules should help somewhat directly, but I want to try switching most of CI over to fetchTree's github fetcher, which won't include submodules. This change should help maintain parity. I'm also seeing a submodule fetch failure in CI (for stan-mode) that I may not need to debug if the package functions without that submodule.
This seems to be much more space-efficient: ~/.cache/nix/tarball-cache is about 700MiB uncompressed, under 300MiB as tzst (using tar's defaults, CI uses zstdmt but I assume will be in the same ballpark). The gitv3 cache is multiple GiB. CI will still build doom-example using fetchGit. I intend to shrink the number of modules enabled in the example to keep gitv3 cache size under control.
Using the lockfile hash as cache key did not make much sense, since changes to Unstraightened itself change what we fetch. And a static restore key means we never flush our cache: that might be ok if cache size was reasonable, but it is not (over 3 GiB). Switch to using the date as cache key, and only restore caches from the current month. Add tarballs as a second cache. I considered caching all of ~/.cache/nix, but do not want to find out the hard way any (new) caches are not portable if I add CI for other platforms. The tarball-cache seems to be another git tree, so it should be ok.
This now matters for CI, as the example is still built using fetchGit instead of the github fetcher. The example was doing double duty as an example and a stress-test, enabling some problematic or interesting modules. That is no longer necessary (once the branch with dynamically generated init.el with all modules/flags enabled lands).
This should not be doing anything, and I saw it stall (to retry requests) once or twice.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This disables submodule fetches, which may have broken something.