[Question] What to cache for CI using the node_modules linker?

[jasonlimantoro]

It is common in CI to cache folders across multiple stages.
What is the suggested way to cache for yarn v2?

So far, I see two options:

1. Caching .yarn and node_modules (and packages/{package}/node_modules for monorepo)

Advantages:

• yarn install only needs to be run on the first job. The subsequent jobs can reuse the cache (no need to do yarn install)

Disadvantages:

• restoring and uploading cache might be the bottleneck (node_modules takes a long time for download/upload).

2. Caching .yarn only

Advantages:

• No more downloading and uploading node_modules

Disadvantages:

• yarn install will be required to be run on every jobs. (The cached yarn install will be faster though). Without yarn install, running any yarn script will throw the following error:

Usage Error: Couldn't find the node_modules state file - running an install might help (findPackageLocation)

Any other thoughts?

[arcanis]

At work we check in the CI cache on the repo and use PnP, so no cache is really needed (apart the repo itself). We still run an install to hydrate the build artifacts, but I plan to find a way to get rid of that.

[larixer]

The approach to caching heavily depends on the project and the CI. Sometimes CI cache compression and decompression takes more time, than Yarn downloading artifacts from the registry, in this case caching .yarn/cache not worth it. Caching all node_modules locations and .yarn/install-state.gz should more often then not be beneficial, because any native tool typically creates a lot of files much faster than Node.js, plus you will avoid rebuilding your install-time dependencies. Also, make sure to use compressionLevel: 0 in .yarnrc.yml if you are on Yarn 2 or 3, (it is 0 by default on Yarn 4+) if you use node-modules linker all the time, this affects install times very much.

[theolavaux]

@larixer I'm a bit confused now, I want to use Yarn 2 (with nodeLinker: node-modules and without enableGlobalCache) in Bitbucket Pipelines with Cypress and I came across this documentation.
https://docs.cypress.io/guides/continuous-integration/introduction#Caching

If I understand right, these are CI artifacts to keep after the dependencies installation

• node_modules
• .yarn/install-state.gz

And these are to be cached

• .yarn/cache
• ~/.cache/Cypress

I thought .yarn/install-state.gz shouldn't be commited as https://yarnpkg.com/getting-started/qa says but your comment above left me wondering.

[Lonli-Lokli]

@arcanis can you please explain how currently caching is done within node? We are using node16, pnp (without commiting artefacts, eg nonzeroconfig) and would like to reuse cache on gitlab. unfortunately there is no documentation for CI on berry website, only on classic one

[Zainakbar786]

Hello!
I was mainly thinking to commit my .yarn/cache folder on my repo so the CI won't have to install every package from remote repositories and we'd save pipeline time 😄.
Regards picsapkstudio

[Lonli-Lokli]

@Zainakbar786 buy you will spend a lot of time for cloning a fresh repo, which will be always longer during the time, even if you decide to disable cache in future

[xg-wang]

@larixer thanks for the answer. I have some additional questions about caching node_modules and install-state.gz: in a monorepo setup what would happen if I don't restore all the node_modules (instead I only restore the root node_modules)? If I have to restore all the node_modules, what if a package is deleted/added between difference source code versions?

[belgattitude]

When using node_modules, I agree with @larixer. Just ./.yarn/cache ($(yarn config get cacheFolder)).

Besides that, I really like to take benefit of enbaleGlobalCache: false with a customized version of @action/cache rather than the builtin cache in setup/node. See actions/setup-node#325 and the action gist: https://gist.github.com/belgattitude/042f9caf10d029badbde6cf9d43e400a

Some reasons

• Most of my repos have renovate bot enabled. So the yarn.lock changes often. (a cleanup example: https://gist.github.com/belgattitude/042f9caf10d029badbde6cf9d43e400a#cleanup-caches)
• Locally when switching branches with different yarn.lock, the install feels almost instant.
• In some rare circumstances, I was able to work or release in case of an npm outage.

More real-world action example I use: https://github.com/belgattitude/nextjs-monorepo-example/blob/main/.github/actions/yarn-nm-install/action.yml. There' probably more small enhancements to be made. But for now I'm fine.

When possible "pnp" will probably give better results, but I've been often blocked.

[zain909]

can you please explain how currently caching is done within the node? We are using node16, PNP (without committing artifacts, eg nonzeroconfig) and would like to reuse the cache on GitLab. unfortunately, there is no documentation for CI on the berry website, only on the classic one.
Regards: The News Addicts

[williamworse]

unfortunately, there is no documentation for CI on the berry website, only on the classic one.
Regards: Funding pips

[dannyrb]

This is most painful for us in a monorepo where some projects use different versions of playwright, and the linker time can really start to grow. Today we're at > 4-minutes.

We're eyeballing a strategy where time-consuming-to-build dependencies are maintained in a separate tree that isn't restored when the root/primary tree is; in this way, "building binaries" only occurs in the actions that require these dependencies.

[belgattitude]

@dannyrb there's 2 strategies I'd suggest to try related to playwright:

1. Be sure to install @playwright/test rather than playwright. (or set the env: PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1 in the CI)
2. Install playwright via npx playwright install on the ci or use an action similar to this one

That should help.

PS: Or use the docker but it slightly more complex.

[belgattitude]

I've got an updated gist here:

https://gist.github.com/belgattitude/042f9caf10d029badbde6cf9d43e400a

TLDR;

• Always cache the .yarn/cache

• Optional: when having multiple jobs that run the install... Try the gist above with cache-node-modules: true and cache-install-state: true to see the if the cost of action/cache (de-)compression time gives a better overal result. (in general yes). But be warned sometimes you might run into issues as some postinstall scripts might not be run (ie: prisma). Depending on cost/benefit it's something that can hacked.

I'll update the gist to allow blacklisting some bots (dependabot, renovate), generally they don't need to create a cache for node-modules/install-state.

[larixer]

@belgattitude The approach to caching heavily depends on the project and the CI. Sometimes CI cache compression and decompression takes more time, than Yarn downloading artifacts from the registry, in this case even caching .yarn/cache not worth it. Sometimes your project has dependencies with slow postinstall scripts, in this case you might find beneficial caching node_modules. I'd say it depends and unfortunately you should experiment with your project and CI. Make sure though to use compressionLevel: 0 in .yarnrc.yml if you use node-modules linker all the time, this affects install times very much.

[belgattitude]

Thanks @larixer you're totally right, it all depends. For information I keep some simple benchmarks in this repo. It's interesting cause there's a lot of common dependencies, like storybook, vite, nextjs remix, vitest..

Here some benchs with and without cache (just saving yarn cache folder):

With cache

CI Scenario	Install	CI fetch cache	Total	CI cache size	CI persist cache
yarn4 nm mixed-compression	±51s	±4s	55s	201Mb	(±5s)
yarn4 nm no compression	±39s	±6s	45s	155Mb	(±8s)
pnpm8.5.1	±23s	±16s	39s	253Mb	(±30s)

Without cache

CI Scenario	Install	Diff with cached run
yarn4 mixed-compression / no cache	±110s	slower than 55s
yarn4 no compression / no cache	±63s	slower than 45s
pnpm85.1 / no cache	±80s	slower than 39s

I really like the fact that with enableGlobalCache: false, the cache is automatically purged / does not have to be rotated like with other package managers. All P/R starts warm or almost warm. Very nice.

BTW the setup/node will be soon improved for yarn... actions/setup-node#325.