Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[rush] rush add in not-yet-rush-install-ed repo when using yarn causes integrity check failure #2929

Open
MichaelBelousov opened this issue Sep 24, 2021 · 10 comments

Comments

@MichaelBelousov
Copy link

MichaelBelousov commented Sep 24, 2021

Summary

I wanted to add a package to one of my subproject's dependencies, but had not yet ran rush install. I would expect the dependency to be added to the package.json and rush update to be successfully called, installing the package.

This does not happen if you run rush install first, but I've seen it occur when I had rush install-ed, I just haven't reproduced those yet.

maybe related: #1706 (comment)

Repro steps

  1. clone a monorepo using rush with yarn, e.g. this mvr repo I made (and use for a separate issue but works here too).
  2. run cd projects/proj1 && rush add -p lodash.pick, (this is specific to the provided repo, you can cd into whichever project you have on your own repo, and add whatever package you like.
  3. boom, you should get something like the following:
Invoking package manager: /home/mike/.rush/node-v12.18.4/yarn-1.22.11/node_modules/yarn/bin/yarn.js install --link-folder yarn-link --cache-folder /home/mike/work/rush-yarn-test/common/temp/yarn-cache --non-interactive

yarn install v1.22.11
[1/4] Resolving packages...
[2/4] Fetching packages...
error Integrity check failed for "@rush-temp/proj1" (computed integrity doesn't match our records, got "sha512-G/im+psEW6mOwoUgSYAuGN156YsEDiM5ZiIJduau35ZOtznera1fDiHde+jIPNd5NnjxrCIhw3x6du55lGSwNQ== sha1-7racjnd6SVJhArCSqAA9DKnXePc=")
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

The command failed:
 /home/mike/work/rush-yarn-test/common/temp/yarn-local/node_modules/.bin/yarn install --link-folder yarn-link --cache-folder /home/mike/work/rush-yarn-test/common/temp/yarn-cache --non-interactive
ERROR: Error: The command failed with exit code 1

Trying again (attempt #2)...

yarn install v1.22.11
[1/4] Resolving packages...
[2/4] Fetching packages...
error Integrity check failed for "@rush-temp/proj1" (computed integrity doesn't match our records, got "sha512-G/im+psEW6mOwoUgSYAuGN156YsEDiM5ZiIJduau35ZOtznera1fDiHde+jIPNd5NnjxrCIhw3x6du55lGSwNQ== sha1-7racjnd6SVJhArCSqAA9DKnXePc=")
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

The command failed:
 /home/mike/work/rush-yarn-test/common/temp/yarn-local/node_modules/.bin/yarn install --link-folder yarn-link --cache-folder /home/mike/work/rush-yarn-test/common/temp/yarn-cache --non-interactive
ERROR: Error: The command failed with exit code 1

Trying again (attempt #3)...

yarn install v1.22.11
[1/4] Resolving packages...
[2/4] Fetching packages...
error Integrity check failed for "@rush-temp/proj1" (computed integrity doesn't match our records, got "sha512-G/im+psEW6mOwoUgSYAuGN156YsEDiM5ZiIJduau35ZOtznera1fDiHde+jIPNd5NnjxrCIhw3x6du55lGSwNQ== sha1-7racjnd6SVJhArCSqAA9DKnXePc=")
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

The command failed:
 /home/mike/work/rush-yarn-test/common/temp/yarn-local/node_modules/.bin/yarn install --link-folder yarn-link --cache-folder /home/mike/work/rush-yarn-test/common/temp/yarn-cache --non-interactive
ERROR: Error: The command failed with exit code 1

Giving up after 3 attempts


ERROR: The command failed with exit code 1


Error: The command failed with exit code 1

    at Function._processResult (/home/mike/.rush/node-v12.18.4/rush-5.54.0/node_modules/@microsoft/rush-lib/lib/utilities/Utilities.js:565:19)
    at Function._executeCommandInternal (/home/mike/.rush/node-v12.18.4/rush-5.54.0/node_modules/@microsoft/rush-lib/lib/utilities/Utilities.js:556:19)
    at Function.executeCommand (/home/mike/.rush/node-v12.18.4/rush-5.54.0/node_modules/@microsoft/rush-lib/lib/utilities/Utilities.js:192:19)
    at Function.executeCommandWithRetry (/home/mike/.rush/node-v12.18.4/rush-5.54.0/node_modules/@microsoft/rush-lib/lib/utilities/Utilities.js:212:27)
    at RushInstallManager.installAsync (/home/mike/.rush/node-v12.18.4/rush-5.54.0/node_modules/@microsoft/rush-lib/lib/logic/installManager/RushInstallManager.js:419:35)
    at RushInstallManager.doInstallAsync (/home/mike/.rush/node-v12.18.4/rush-5.54.0/node_modules/@microsoft/rush-lib/lib/logic/base/BaseInstallManager.js:127:24)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async PackageJsonUpdater.doRushAdd (/home/mike/.rush/node-v12.18.4/rush-5.54.0/node_modules/@microsoft/rush-lib/lib/logic/PackageJsonUpdater.js:126:17)
    at async AddAction.runAsync (/home/mike/.rush/node-v12.18.4/rush-5.54.0/node_modules/@microsoft/rush-lib/lib/cli/actions/AddAction.js:143:9)
    at async RushCommandLineParser.onExecute (/home/mike/.rush/node-v12.18.4/rush-5.54.0/node_modules/@rushstack/ts-command-line/lib/providers/CommandLineParser.js:199:13)

Standard questions

Please answer these questions to help us investigate your issue more quickly:

Question Answer
@microsoft/rush globally installed version? 5.54.0
rushVersion from rush.json? 5.54.0
useWorkspaces from rush.json? No
Operating system? Linux
Would you consider contributing a PR? Yes
Node.js version (node -v)? 12.18.4
@MichaelBelousov
Copy link
Author

related: #2928

@MichaelBelousov
Copy link
Author

MichaelBelousov commented Sep 24, 2021

I managed to find an order of commands that miraculously makes it work without deleting caches, the lockfile, etc:

In a fresh mvr repo (./reset.sh if necessary):

# run the failed add
pushd projects/proj1 && rush add -p lodash.pick && popd


# I need to do this command twice (bizarrely, it fails the first time but succeeds the next!)
for i in $(seq 2)
do
  pushd common/temp
  # I used the same paths as rush does, but I make them non-absolute and use global `yarn` here for brevity
  yarn install --link-folder yarn-link --cache-folder common/temp/yarn-cache --non-interactive --update-checksums
  popd
done
# now the previously `rush add` will work
cd projects/proj1 && rush add -p lodash.pick

The error the first time yarn --update-checksums is ran is:

error An unexpected error occurred: "ENOENT: no such file or directory, open '/home/mike/work/rush-yarn-test/common/temp/yarn-cache/v6/npm-@rush-temp-proj1-0.0.0-d12365c6ba1a4ab080edcc54c46602e10fff4f75-integrity/node_modules/@rush-temp/proj1/.yarn-metadata.json'".

@MichaelBelousov
Copy link
Author

MichaelBelousov commented Sep 24, 2021

I can't find it anywhere and it failed when I tried, so I might as well ask, is yarn v2 supported by rush? Maybe there's a fix for needing to run --update-checksums twice in yarn-v2

@MichaelBelousov
Copy link
Author

it looks like yarn install fails the first time because there was no @rush-temp/proj1 package in common/temp/node_modules until it is ran the first time

@MichaelBelousov
Copy link
Author

MichaelBelousov commented Sep 24, 2021

I used tar + diff <(tar -tf state1.tar) <(tar -tf state2.tar) to break down what changes between each command:

pushd projects/proj1 && rush add -p lodash.pick && popd
# the following are added:
# common/temp/yarn-preinstall.lock
# common/temp/yarn.lock
# common/temp/projects/proj1/**
# common/temp/yarn-local
# common/temp/.npmrc
# common/temp/yarn-cache/**
# common/temp/package.json
# I'll add later what yarn commands were run which should explain a lot of this
pushd common/temp
yarn install --link-folder yarn-link --cache-folder common/temp/yarn-cache --non-interactive --update-checksums
popd
# first time this runs, the following are added:
# common/temp/yarn-error.log (because of the error)
# common/temp/yarn-cache/v6/npm-@rush-temp-proj1-0.0.0-d12365c6ba1a4ab080edcc54c46602e10fff4f75/node_modules/@rush-temp/proj1/** (the temp project is cached)
# the above directory includes the file that causes an error, maybe a race occurs and the file is created during/after the erroring code which needs it?
pushd common/temp
yarn install --link-folder yarn-link --cache-folder common/temp/yarn-cache --non-interactive --update-checksums
popd
# second time this is ran, it doesn't cause any errors and adds the following:
# common/temp/node_modules/@rush-temp/
# common/temp/node_modules/lazy-from/** (this is the initial dep of proj1)
# common/temp/node_modules/.yarn-integrity
pushd projects/proj1 && rush add -p lodash.pick && popd
# this time lodash.pick is added correctly, to common/temp/node_modules and the yarn-cache

@MichaelBelousov
Copy link
Author

MichaelBelousov commented Sep 24, 2021

due to having a(n albeit ugly) workaround, lack of upgraded yarn support in rush (I tried forcing v2 and v3 but there are multiple issues) I'm not going to look into this further, I'd rather look at electron-tools supporting pnpm

@iclanton
Copy link
Member

Our support for Yarn in general isn't as good as PNPM. @octogonz, do you know if we've looked into supporting Yarn V2?

@octogonz
Copy link
Collaborator

By "Yarn V2" i assume you mean Yarn Plug'n'Play. (V2 is also able to do the Yarn Classic installation model.) I did investigate supporting that. The work would be similar to what we did for PNPM workspaces. The main problem is that we need someone to volunteer to do the work. Plug'n'Play is hard to justify investing in since it has more compatibility issues than PNPM, and not really any significant practical advantages besides a theoretically simpler design. (Feedback welcome on this point.)

@tim-stasse
Copy link

@octogonz Support for Yarn (2/3) — like you said, really Plug'n'Play (PnP) — is about the only thing stopping me from using rushstack!

Regarding its practical advantages, combining the offline cache with PnP enables a zero install philosophy, this has significant benefits for the software supply chain & security, and I wouldn't dismiss them quickly. It's also something which rushstack could always use & benefit from itself 😉.

I'd also be more than happy to contribute & help with adding support for this. If someone else was to do it but needed a hand, I'm relatively familiar with the yarn codebase now, too (though that said, I'm not familiar with rushstack at this point 😅).

@ntduan
Copy link

ntduan commented Oct 8, 2021

I have the same problem, but only when installing @rushstack/eslint-config.

@iclanton iclanton moved this to Needs Investigation in Bug Triage Aug 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Needs Investigation
Development

No branches or pull requests

5 participants