yarn install overwrites linked modules with cached version

[lukebennett]

Do you want to request a feature or report a bug?

Bug

What is the current behavior?

Running yarn install for an app that has linked modules results in yarn copying its cached version of each module to the symlinked directory, overwriting the local copy in the process.

If the current behavior is a bug, please provide the steps to reproduce.

I struggled to reproduce this from scratch, it's possible that it's down to a specific combination of factors with my setup. This involves local copies of private/scoped npm modules yarn linked from an application, and referenced in package.json via file:../xxx. The version of the modules yarn has in its cache appears to be from when I first installed and ran it a few days ago (there have been numerous changes to the modules since), so it's possible the problem lies with its caching strategy as much as its installation strategy.

One observation I've made is that the problem occurs even when using yarn install --force to supposedly bypass the cache.

Happy to answer any questions and/or try anything to help pinpoint the problem.

What is the expected behavior?

yarn install should skip linked modules like npm intstall does. Performing the same steps above using npm install does not result in this problem.

Please mention your node.js, yarn and operating system version.

node v6.2.2
yarn v0.16.0
Mac OS X v10.10.5

[bmayen]

This isn't specific to cached packages. I have a published package [email protected]. Locally I link an unpublished package [email protected]. My project has a package.json dependency for [email protected] and I link to that. When I run yarn after linking, it tries to install [email protected] from the repo instead of using the linked version and it fails because [email protected] hasn't been published yet.

[jimdebeer]

really need this

[mstdokumaci]

this is breaking

[haggholm]

This applies even when you install different packages. E.g.

cd @my/local-dep
yarn link
cd ../../@my/main-project
yarn link @my/local-dep
yarn add lodash

Even though I didn’t even explicitly do anything with @my/local-dep, yarn still goes through the modules and installs anything not locally installed, even if it’s linked.

This happens even if I specify --no-lockfile.

It’s a particularly nasty problem because it overwrites my local working copy of @my/local-dep, so it will overwrite anything you haven’t committed yet, effectively erasing local work.

$ yarn --version
0.16.1

[joefiorini]

I've noticed this issue as well. In my testing it seems to be limited to scoped modules, but that may not be true for everyone. My use case:

Our team works in a monorepo. To get shared packages locally, the only way to have an efficient development process is to symlink them. Since we can't save symlinked installs to package.json, we use file: specifiers with relative paths, followed by a library called linklocal that goes finds and symlinks local dependencies.

It's kind of a hack, but it works very well. However, npm shrinkwrap is so painful for us that switching to yarn is extremely advantageous, even if we have to work around a few warts due to early adoption.

This issue is very bad for that above workflow though. Anytime yarn tries to install we run the risk of losing work.

What exactly is the advantage of using the cache for local dependencies? Since they would never hit the network anyway, couldn't yarn just treat anything that's a file: reference as a cache miss and either skip copying it entirely (as npm does) or copy it in from the local dir?

[haggholm]

I’d be much happier if Yarn just left linked dependencies alone—if it’s a symlink, either do nothing on the assumption that I want to work with it as a symlink, or if for some reason that can’t be done, issue an error and refuse to do anything. Even deleting the symlinks and replacing them with normal dependencies would be better—still a nuisance, but at least not as bad as the current behaviour.

1. If I run yarn link, I should never have to worry about Yarn changing my files.
2. There’s no reason why Yarn should be modifying files outside of node_modules (and its own caches).

IMHO, the preferred behaviour would be to leave symlinked directories alone, unless there’s a version mismatch, in which case error out with a descriptive message.

[bmayen]

unless there’s a version mismatch, in which case error out with a descriptive message.

What would be the advantage of this? Why not allow linked packages to be any version? I believe that's how NPM currently works.

[damonmaria]

Having the same issue. It seems like it's replacing the linked module with the published version. As tho it's installing the package into the directory without realising it's a symlinked package. I don't think yarn's cache is the reason.

[joefiorini]

I was able to get around this issue for time being by using yarn link and then not adding the dependencies to my package.json at all. The script that starts our dev server & installs deps for CI/production builds also handles running yarn link for our shared packages, which ensures it happens everywhere we need to build our code.

[wclr]

This is very confusing behaviour of yarn, a big source of hassle, have you managed fix the logic behind it? I believe yarn should not check for cache when dealing with file: modules (it also refers to other non-versioned sources like remote git repo).

[wclr]

I've described inconsistent behaviour or yarn that leads to the issue: #1794

[wclr]

Btw in new 0.17.6 it seems that yarn doesn't replace content of simlinked folder (in node_modules) with version from cache, but installs new dependencies (that changed in linked package.json - which is nice)

Can others confirm that this not an issue anymore?
@lukebennett @joefiorini @haggholm

[travisbloom]

@whitecolor can confirm that with 0.17.6 the problem still persists. simlinked folder is having code overwritten by yarn.

[travisbloom]

This is happening with a privately scoped npm repo (@rocketrip/name-of-package)

[wclr]

@travisbloom right it seem to be true when you link scoped package, it works ok with not scoped symlinks (doesn't touch it).

@wyze can you look in to it, to close issues with scoped packages?

[tgriesser]

Just added a fix for this in #1970

[DenysVuika]

Still an issue with 0.17.9 and a huge blocker for most of the scenarios involving npm/yarn link.

[rbilgil]

+1, This is a big issue that caused us to lose uncommitted work from a linked repo. Would be good to get it fixed.

[AlastairTaft]

Can confirm I don't get this issue in the latest version 0.19.1.

[tgriesser]

Yep, this can probably be closed now

[AlastairTaft]

I am still seeing this issue on [email protected]. I'm using 'lerna' so if I have packageA depending on packageB via a symlink and then run yarn upgrade packageB inside the packageA directory It ends up overwriting my non-commited local changes of packageB.

[wclr]

@AlastairTaft can you make a list of simple steps to reproduce?

[dbpieter]

@whitecolor

Simple reproduction with file: protocol.

• Install a package through the registry (for example "lodash")
• Change your package.json so that lodash uses file protocol (eg: "lodash": file:./custom_packages/lodash")
• yarn install
• observe node_modules => lodash version from cache is used instead of the one from custom_packages

yarn v0.22.0

[wclr]

@dalmaer and on the latest stable 23.2?

[bestander]

We just landed a fix to file: dependencies on master. Check out in 0.24 next week

…

On 27 April 2017 at 15:09, Alex ***@***.***> wrote: @dalmaer <https://github.com/dalmaer> and on the latest stable 23.2? — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#1214 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACBdWJx3UiG6uRUHgZeNqqGlsUyGyd2Pks5r0KGIgaJpZM4KaX2W> .

[oliversalzburg]

This is still happening. I just opened #3288 to report the same behavior. Happens with 0.23.4. We're not using file protocol. Could possibly be related to scoped/private as that is where we're seeing it.

[oliversalzburg]

Is there any workaround for this which I am not aware of? Needless to say, having your working copies constantly overwritten by cached content without you realizing it has a major impact on development.

[wclr]

@oliversalzburg I advice to try this for development with local packages: https://github.com/whitecolor/yalc (it can make your life really easier)

[oliversalzburg]

@whitecolor Awesome. Thanks a lot. I'll check it out.

[jonschlinkert]

I'm still trying to confirm if this is what happened, but it appears that yarn overwrote local files across multiple files with cached symlinked versions. I'm using yarn 0.23.4

[oliversalzburg]

@bestander Can you reopen this issue? It is not solved. I would really appreciate some feedback on this.

[bestander]

@oliversalzburg please open a new one with a scenario how to reproduce it.
Ideally if you could provide a bash script to make it automated.

[oliversalzburg]

@bestander I had already opened #3288, but I'm having a really hard time reproducing the issue in isolation. It happens multiple times every day in our main project tree though. It would be really nice to investigate this with someone who is more familiar with the code base and maybe give me some pointers on what to look for.

[danilo-drs]

I'm getting this issue on version 0.24.5.
I have some packages that i developed and they are not published on any server, they are just linked.
My projects uses this linked packages and some published packages like express or bluebird.
when I try to run "yarn" to install all or a specific package it returns the error:

error Couldn't find package "package-name" on the "npm" registry

steps to reproduce:

cd path/to/a-dev-package
yarn link
cd path/to/a/project
yarn link a-dev-package
yarn add bluebird
it returns :
error Couldn't find package "a-dev-package" on the "npm" registry

[bestander]

Thanks, @danilo-drs, I've extracted this into a separate issue #3645

[stelf]

I believe this issue is not solved in v0.27.5. node here is 6.11 and npm 5.04.

Steps (note: A is a dependency to B in package.json):

1. yarn link / npm link module A (a symlink goes into the local $PREFIX/lib/node_modules/)
2. chdir into module B's dir
3. yarn link A (a symlink to A goes into B/node_modules/).
4. yarn install => complains about versions from npmjs.com, rather than taking into consideration the local

[wclr]

@stelf yarn takes into account what is in your package.json to keep consistency. So it actually doesn't take into account that you linked the module (there is no any notice about it, except symlink directory in node_modules ). There is link: dependencies in yarn, but it is more for linking local deps in monorepo. For working with external modules I successfully use yalc

[fcoury]

This is still happening with yarn v1.2.1:

> $ yarn link mongodb-promise                                                                                                                                         yarn link v1.2.1
success Using linked module for "mongodb-promise".
✨  Done in 0.07s.

> $ ls -lad node_modules/mongodb-promise                                                                                                                              
lrwxr-xr-x  1 fcoury  staff  21 Oct 18 18:52 node_modules/mongodb-promise -> ../../mongodb-promise

fcoury@iMac ~/code/faxing                                                                                                                                              > $ yarn                                                                                                                                                              [±master ●]
yarn install v1.2.1
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved lockfile.
✨  Done in 1.99s.

> $ ls -lad node_modules/mongodb-promise                                                                                                                              drwxr-xr-x  9 fcoury  staff  306 Oct 18 18:52 node_modules/mongodb-promise

So this is the expected behavior?

[chaddjohnson]

Still outstanding and needs to be fixed.

When I run yarn add lodash (for example) it removes any links I had set up for my project, and I have to manually run yarn link my-package-name again.

[bestander]

Looks like the original issue was fixed but there may be other cases when it is still relevant.
For everyone seeing this please open a new issue with the repro script.
It would be hard to reopen this one as it tracks too many things.

[chaddjohnson]

Created new issue #4770 as requested.