Installing package from a branch with yarn.lock present installs old version of package

[askielboe]

This is bug report and a follow-up to issue #2280.

Problem:

When installing from a git branch with a correct yarn.lock file present, yarn installs an old version of the package instead of the most recent commit (HEAD), as defined in yarn.lock.

This can lead to inconsistent install states when when using a yarn.lock file.

Steps to reproduce:

The following steps simulate a case where a package has been installed from the master branch of a GitHub repository, and what happens when a user tries to install a new commit to master using yarn.

1. Create a new repository with a single commit of a file version_1.0 and push it to GitHub (in this example askielboe/yarn-bug).

2. Create a package.json file with the following content:

{
  "name": "yarn-bug",
  "version": "1.0.0",
  "dependencies": {
    "yarn-bug": "askielboe/yarn-bug#master"
  }
}

3. Run yarn install at the location of package.json: yarn correctly installs the package with a single file node_modules/yarn-bug/version_1.0.

4. Create a new commit in yarn-bug adding a file version_2.0, and push the commit to GitHub.

5. Run yarn install again: yarn installs from cache without checking for new commits at the origin. This means that the yarn installation of yarn-bug will not include the HEAD commit with the new file version_2.0.

6. Delete node_modules and yarn.lock and run yarn install: With yarn.lock gone, yarn now correctly pulls the latest commit and generates a new yarn.lock file with the HEAD hash.

7. Delete node_modules (but keep yarn.lock) and run yarn install: yarn now installs the initial commit again, instead of the HEAD commit referenced in yarn.lock. Somehow yarn looks at its cache and picks a commit (by random?) to install instead of checking whether it is the HEAD, or the most recent pulled commit.

Notes & workarounds:

• Running yarn upgrade after deleting node_modules will install the correct version of the package.

• Clearing the yarn cache by running yarn cache clean yarn-bug and then yarn install will install the correct HEAD commit without the need to delete yarn.lock.

• Even when using --frozen-lockfile yarn still installs an old commit (ignoring the correct hash in yarn.lock). This can lead to inconsistent state and critical issues in continuous integration, when requiring packages from a git branch.

The bug(s) can be reproduced using the following script: (replace askielboe/yarn-bug with a reference to your own test-repository). Please read through the script before running it, as it creates new directories and clears the yarn cache: https://gist.github.com/askielboe/0c0bc3fdf88193bb17fc4fef5d332ad1

This bug report has been created using yarn 1.2.1 and node 8.7.0 on macOS 10.12.6.

[bestander]

Thanks, @askielboe, this is a great bug report.
The core team will look a this eventually but they are very busy at the moment.
Do you think you could spend some time debugging git-resolver.js and see why is it resolving to the wrong revision?

[askielboe]

I'll take a look if I have time.

Meanwhile I've updated the bug report to reflect the fact that you have to do an install between git commits for the bug to arise. I believe it's due to the fact that yarn then keeps references to several commits in its cache, resulting in inconsistent installations.

I've also included a new script to reproduce the bug in this way.

[askielboe]

I've done a bit of debugging. The yarn codebase is a bit opaque to me unfortunately, and I won't be able to spend more time this week, but here is what I've managed to find so far.

It seems that the issue appears when the package is fetched with a manifest where uid is equal to the package version instead of the commit hash. This happens when fresh: false, and makes BaseFetcher create a folder in yarn cache with path $(yarn cache dir)/npm-yarn-bug-0.0.0, excluding a commit hash.

When calling yarn install, yarn will use the cached version in npm-yarn-bug-0.0.0 regardless of when it was created, ignoring the commit hash in yarn.lock. This happens even if there are caches of the correct commit in the yarn cache (i.e. $(yarn cache dir)/npm-yarn-bug-0.0.0-13dba66e55f4b8b8259577f417d73aef1809c9d8).

If the cache folder npm-yarn-bug-0.0.0 is deleted, yarn installs the correct version until a new npm-yarn-bug-0.0.0 folder is generated.

This is the manifest I get when yarn creates the non-hashed cache folder:

{ name: 'yarn-bug',
    version: '0.0.0',
    _reference:
     PackageReference {
       resolver: [Object],
       lockfile: [Object],
       requests: [Array],
       config: [Object],
       registry: 'npm',
       version: '0.0.0',
       name: 'yarn-bug',
       uid: '0.0.0',
       remote: [Object],
       dependencies: [],
       permissions: {},
       patterns: [Array],
       optional: false,
       level: 0,
       ignore: false,
       incompatible: false,
       fresh: false,
       location: null },
    _remote:
     { resolved: 'https://codeload.github.com/askielboe/yarn-bug/tar.gz/13dba66e55f4b8b8259577f417d73aef1809c9d8',
       type: 'tarball',
       reference: 'https://codeload.github.com/askielboe/yarn-bug/tar.gz/13dba66e55f4b8b8259577f417d73aef1809c9d8',
       hash: '',
       registry: 'npm' },
    fresh: false }

And this is the manifest I get when yarn created a cache folder including a git hash (which results in the correct version getting installed):

{ name: 'yarn-bug',
    version: '0.0.0',
    _reference:
     PackageReference {
       resolver: [Object],
       lockfile: [Object],
       requests: [Array],
       config: [Object],
       registry: 'npm',
       version: '0.0.0',
       name: 'yarn-bug',
       uid: '13dba66e55f4b8b8259577f417d73aef1809c9d8',
       remote: [Object],
       dependencies: [],
       permissions: {},
       patterns: [Array],
       optional: false,
       level: 0,
       ignore: false,
       incompatible: false,
       fresh: true,
       location: null },
    _remote:
     { resolved: 'https://codeload.github.com/askielboe/yarn-bug/tar.gz/13dba66e55f4b8b8259577f417d73aef1809c9d8',
       type: 'tarball',
       reference: 'https://codeload.github.com/askielboe/yarn-bug/tar.gz/13dba66e55f4b8b8259577f417d73aef1809c9d8',
       registry: 'npm',
       hash: undefined },
    fresh: true }

The main differences are:

• not working -> working
• uid: '0.0.0' -> uid: '13dba66e55f4b8b8259577f417d73aef1809c9d8'
• fresh: false -> fresh: true
• hash: '' -> hash: undefined

[edmorley]

#4722 seems related (CC @SEAPUNK)

[SEAPUNK]

it might be the exact same issue, but not 100% sure
i'll keep my issue up just in case

[bestander]

@askielboe, that is a great analysis and a huge help, thanks a lot for digging that deep.
It looks like the problem is in cache folder for git dependencies, maybe the name is using package.json version instead of git hash.

cc @arcanis as he knows more about Yarn/git integration.

More help getting to the bottom of this is always welcome.

[shtse8]

I have the same issue, any updates for fixing this bug?

[raduflp]

the issue is reproducible to this day :(

[xmnlab]

I have the same issue here also.

[cesarizu]

This is still happening with yarn version 1.15.2.

[sergeyzenchenko]

Same issue for me in the latest version (1.21.1). Any updates on this one?

[jtomaszewski]

Happens to me at 1.22.0 as well.

I go around it with yarn cache clean xxx && rm -rf node_modules/xxx && yarn --check-files but I had it appearing over and over again, which is quite annoying I have to admit.

[dword-design]

Are there any plans on fixing this? Otherwise I'll have to get back to NPM since Yarn literally it installs wrong dependencies on production.

[cah-brian-gantzler]

From what I see, when the package.json refers to a branch, the commit for the current HEAD is stored in the yarn.lock, then when more commits are added, a yarn install always installed the saved commit instead of the new HEAD.

Any updates on fixing this? been living with constantly doing yarn upgrade my-package while in development.

[raphaelschwinger]

Im also affected by this issue and experience the same behavior as @cah-briangantzler.
Since yarn 2 yarn upgrade is replaced with yarn up so I use the following workaround:

• yarn up PACKAGE_NAME -> this installs the latest version available in the npm repo
• revert change version in package.json to the gitlab branch
• yarn install

[rachelslurs]

It seems the only way to get things to work reliably is to use commit= instead of branch= 😞 OR to update the commit in the resolution field of yarn.lock

Using yarn 3.2.0 and still having this issue. @raphaelschwinger 's workaround didn't work for me.

[pacaplan]

I believe this is a feature, not a bug. This is in fact the entire purpose of having a yarn.lock file: to ensure that every time you run yarn install, it installs the same version of each dependency.

What would happen if yarn did not work this way?

1. First time yarn install is run, yarn.lock points to commit aaaaaa of package yarn-bug
2. App is tested and release to production
3. Developer modifies package.json, e.g adding / upgrading unrelated dependency
4. Developer runs yarn install
5. Yarn.lock now points to commit bbbbbb of package yarn-bug

So the package yarn-bug has now been inadvertently upgraded to a new commit that could potentially introduce bugs. This is the reason why versions numbers and commit hashes are locked down in yarn.lock, so that packages are only upgraded to a new version / commit when explicitly requested.

[Kapcash]

@pacaplan The thing is, if you define your dependency to target a branch and not a specific commit, I would expect it to actually updates to the HEAD of the branch at any install.
If I want to fix the version to a specific commit, I can use the commit hash on my dependency ("yarn-bug": "https://github.com/askielboe/yarn-bug.git#f3fb60b37da376cbe480b22587e73604d99f4f9e").

I feel like this is developer's responsibility to tell yarn to install from a commit or a branch. We know it's risky that it updates from the HEAD, but that's what expect anyway!

[dword-design]

My solution to this problem was just to not use Git repos in Yarn/npm because it has some disadvantages I found out during my work on Open-Source projects:

• Some CI systems don't have Git installed
• When you transpile code, you will have to add the dist folder to the repo so that it will be in the installed node_modules folder

I know it's not a fix to the actual problem but maybe you don't have to solve the problem after all 😃.

[SebaSOFT]

This is in fact a bug, not a feature. When dealing with branches and handling internal dependencies you want the immediate back and forth between the library and the project, having the resolution be stick to the commit should check if the commit is actual HEAD for the branch it intends to represent.