yarn dedupes inconsistently

[TheLudd]

After a discusion on discord with @rally25rs earlier this week I was advised to create an issue here for a discussion on how yarn handles deduping.

First, running yarn dedupe produces this output which is also available here:

error The dedupe command isn't necessary. yarn install will already dedupe.

However, yarn install will infact not dedupe if you have a duplicated structure. Duplicated structures are created today when upgrading specific packages. The flow is usually something like this:

• Your project depends on react
• It also depend on a react module that itself depends on react
• Your project upgrades react, and even though the new react version is in the range of the other modules specified react dependency, react is duplicated in the node_modules tree.

This is a problem because you can get duplicate instances of react which is not valid. This is just an example however, there are other packages than react, such as our own internal modules at my company, where duplicated modules cause problems.

The only way now to deduplicate is to run a general yarn upgrade which has the side effect of upgrading all other dependencies as well.

I am not sure if the current behavior is intended or not but IMO yarn should always deduplicate as much as possible. If I remember correctly, this was a goal of yarn when it was initially launched.

Could yarn be implemented to dedupe on upgrades?

[rally25rs]

@yarnpkg/core looking for some discussion on the correct behavior here...

Some of my notes from our discord discussion:

if multiple patterns resolve to the same version you'll get more than 1 key to the same value
so in your example, if the root project depends on a@^1.0.0 and b depends on a@^1.0.1 (in other words the dep tree is:

package.json
  |- a@^1.0.0
  +- b
      +- a@^1.0.1

In the lockfile you get something like:

[a@^1.0.0, a@^1.0.1] -> [email protected]

assuming that [email protected] is latest at the time it was added.

if you yarn upgrade a because [email protected] exists then it unlocks the root's dependency on a (removes a@^1.0.0) and leaves you with

[a@^1.0.1] -> [email protected]

then re-adds the latest a@^1.0.0 which now resolves to [email protected] so the lockfile becomes

[a@^1.0.1] -> [email protected]
[a@^1.0.0] -> [email protected]

(not saying that's "correct", just saying I'm pretty sure that's how the code currently works)

---

@TheLudd suggests that after yarn upgrade a yarn should dedupe a@^1.0.0 and [email protected] to both point to [email protected] which is the max version satisfying both ranges.

My feedback was:

It's possible that if you had

package.json
  |- [email protected]
  |     +- [email protected]
  +- [email protected]

if running yarn upgrade b changed it to

package.json
  |- [email protected]
  |     +- [email protected]
  +- [email protected]

some people might find that to be a bug (yarn changed a's dependencies but I didn't upgrade a) ... but I could certainly see an argument either way.

My concern is that I think the behavior might actually be a bit ... undefined at the moment.
If they used the same pattern then I think the above change would happen, but if they used different patterns then it would not change a -> b and become

package.json
  |- [email protected]
  |     +- [email protected]
  +- [email protected]

so whether or not a -> b changes or not is unclear... It's be nice to get some more opinions on what's "correct" in these cases and make sure it at least behaves in a consistent way.

---

What I'm talking about with the "inconsistent" behavior is that if the dep tree used the same ranges from the original example:

package.json
  |- a@^1.0.0
  +- b
      +- a@^1.0.0

In the lockfile you get:

[a@^1.0.0] -> [email protected]

then yarn upgrade a will also change the version of a that b is using, because it is using a hoisted and deduped version of a

[bestander]

I totally agree.
Let's do it!
I mean the upgrade command to scan the tree and upgrade in other places.

[epmatsw]

Just want to add another vote that the deduplication approach is the better one. The current state can make it difficult/impossible to upgrade a dependency without unlocking the full dependency tree or manually editing the lockfile, which is a big ask when you only want to upgrade a single package.

[epmatsw]

I also think this conversation is discussing a solution for #3967

[Bnaya]

I'm using
https://www.npmjs.com/package/yarn-tools
fix-duplicates
And it really works great.