Add an option to skip caching node_modules

[jrjohnson]

Why?
Some packages do not expect the node_modules directory to be cached and
have trouble installing themselves a second time.

Most recently we have been unable to build a project that contains sharp which has a compilation step that stores data outside of the node_modules directory. We have to manually force a build wit a cleared cache each time. In the past we have had similar issues with node-sass.

What changed?
If the $CACHE_NODE_MODULES environmental variable is false the build will skip
caching or attempting to use a cached node_modules directory. If
$CACHE_NODE_MODULES is unset or true then node_modules will continue to
be cached.

[brycekahle]

@jrjohnson Do you have more information on what data those modules are storing outside of node_modules? That seems like a bad practice.

[jrjohnson]

@brycekahle I'm honestly not that certain. It seems to be standard when there is a native compilation step. My builds usually fail with a can't find ../build type of message.

Bad practice or not this regularly forces us to Retry Deploy with the clear cache option. Which means we also loose the excellent yarn cache and have to fetch all packages from the network.

[edmorley]

I believe many packages save the built assets in directories in node_modules/ instead - which yarn then tracks as artifacts in .yarn-integrity (so knows not to purge them). Putting them outside of node_modules seems like something the packages in question should fix. (That said, perhaps adding more options around caching behaviour would be a useful feature for the build script anyway, so still useful even if in this case it's being used to work around upstream bugs?)

[brycekahle]

I'm not rejecting your proposed solution. I'm trying to understand the problem better before taking any action. Do you have an example of the error you see when running a build with a cache?

[jrjohnson]

I understand. I'm happy to answer as many questions as you have and try something else. I just figured a PR would be easier to discuss than a more abstract issue. I have always felt that node_moduels should not be committed or cached since it is the result of building package.json. I know that isn't necessarily everyone's opinion though and in netlify's case it probably saves significant computation costs to keep it around so I'm more than happy to explore other solutions.

The specific message we get when a build fails is:
Cannot find module '../build/Release/sharp.node'

https://app.netlify.com/sites/ilios-frontend/deploys/5aaff42192226a1217716b63

Detailed build log:

10:32:17 AM: Build ready to start
10:32:20 AM: Fetching cached dependencies
10:32:20 AM: Starting to download cache of 166.7MB
10:32:21 AM: Finished downloading cache in 1.56782041s
10:32:21 AM: Starting to extract cache
10:32:29 AM: Finished extracting cache in 7.884087812s
10:32:29 AM: Finished fetching cache in 9.604046968s
10:32:29 AM: Starting to prepare the repo for build
10:32:30 AM: Preparing Git Reference refs/heads/master
10:32:31 AM: Starting build script
10:32:31 AM: Installing dependencies
10:32:32 AM: Started restoring cached node version
10:32:34 AM: Finished restoring cached node version
10:32:34 AM: Attempting node version '6' from .nvmrc
10:32:35 AM: v6.13.1 is already installed.
10:32:36 AM: Now using node v6.13.1 (npm v3.10.10)
10:32:36 AM: Attempting ruby version 2.2.9, read from environment
10:32:37 AM: Using ruby version 2.2.9
10:32:37 AM: Using PHP version 5.6
10:32:37 AM: Started restoring cached node modules
10:32:37 AM: Finished restoring cached node modules
10:32:37 AM: Started restoring cached yarn cache
10:32:37 AM: Finished restoring cached yarn cache
10:32:38 AM: Installing NPM modules using Yarn version 0.18.1
10:32:39 AM: yarn install v0.18.1
10:32:39 AM: [1/4] Resolving packages...
10:32:41 AM: [2/4] Fetching packages...
10:33:57 AM: warning [email protected]: The platform "linux" is incompatible with this module.
10:33:57 AM: info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
10:33:57 AM: [3/4] Linking dependencies...
10:33:57 AM: warning Unmet peer dependency "stylelint@^8.3.0 || ^9.0.0".
10:33:57 AM: warning Unmet peer dependency "stylelint@^8.0.0 || ^9.0.0".
10:33:57 AM: warning Unmet peer dependency "stylelint@^8.3.0 || ^9.0.0".
10:34:24 AM: [4/4] Building fresh packages...
10:34:24 AM: Done in 105.25s.
10:34:24 AM: NPM modules installed using Yarn
10:34:24 AM: Started restoring cached go cache
10:34:25 AM: Finished restoring cached go cache
10:34:25 AM: unset GOOS;
10:34:25 AM: unset GOARCH;
10:34:25 AM: export GOROOT='/opt/buildhome/.gimme/versions/go1.10.linux.amd64';
10:34:25 AM: export PATH="/opt/buildhome/.gimme/versions/go1.10.linux.amd64/bin:${PATH}";
10:34:25 AM: go version >&2;
10:34:25 AM: export GIMME_ENV='/opt/buildhome/.gimme/env/go1.10.linux.amd64.env';
10:34:25 AM: go version go1.10 linux/amd64
10:34:25 AM: Installing missing commands
10:34:25 AM: Verify run directory
10:34:25 AM: Executing user command: ember build -e preview
10:34:27 AM: Cannot find module '../build/Release/sharp.node'
10:34:27 AM: Stack Trace and Error Report: /tmp/error.dump.adb60804a027f99a811061ca2f4faf5c.log
10:34:27 AM: Caching artifacts
10:34:27 AM: Started saving node modules
10:34:27 AM: Finished saving node modules
10:34:27 AM: Started saving yarn cache
10:34:27 AM: Finished saving yarn cache
10:34:27 AM: Started saving pip cache
10:34:27 AM: Finished saving pip cache
10:34:27 AM: Started saving emacs cask dependencies
10:34:27 AM: Finished saving emacs cask dependencies
10:34:27 AM: Started saving maven dependencies
10:34:27 AM: Finished saving maven dependencies
10:34:27 AM: Started saving boot dependencies
10:34:27 AM: Finished saving boot dependencies
10:34:27 AM: Started saving go dependencies
10:34:27 AM: Finished saving go dependencies
10:34:27 AM: Error running command: Build script returned non-zero exit code: 1
10:34:27 AM: Failing build: Failed to build site
10:34:27 AM: failed during stage 'building site': Build script returned non-zero exit code: 1
10:34:27 AM: Finished processing build request in 2m8.7273574s

[edmorley]

(Sorry for another drive by!)
Actually looking at the sharp package repo, it seems that the build directory is within node_modules after all.

From the log above, it seems that that Netlify site/project is using a very old version of yarn (v0.18.1 vs the netlify build script default of v1.3.2), so this might be a case of just needing to update (or unset) your YARN_VERSION environment variable, to pick up this fix?
yarnpkg/yarn#3224

[jrjohnson]

@edmorley we're not declaring a specific yarn version anywhere that I can find. I will investigate that some more to see if it is a solution to this issue. Thanks.

Edit: Forcing yarn with YARN_VERSION=1.5.1 doesn't help with this issue. The only way I can get a build to run is to clear the cache.

[jrjohnson]

@brycekahle just checking back in to see if you need more information.

[brycekahle]

@jrjohnson Would you mind reaching out to support (and referencing this issue)? I'm pretty sure we have others who use sharp and aren't running into this problem.

[jrjohnson]

Support was unable to diagnose my issue. But suggested I add rm -rf node_modules to my build command to work around whatever is going on here and that works. I'm going to close this since I don't think you want to merge it.