Node must de-duplicate environment variables before calling CreateProcess (and upon env.extend())

[asklar]

• Version: 12.9.1
• Platform: x64
• Subsystem: Windows 10

What steps will reproduce the bug?

yarn sets up some environment variables (like npm_config_cache) and calls execSync. This calls the Win32 API CreateProcess and passes an environment block which is a merge of the existing environment block (i.e. from the calling process) with some additional stuff that includes npm_config_cache among others.

Environment variables in Windows are case-insensitive, but CreateProcess does not seem to sanitize the environment block. This is a problem because if the calling process had set up NPM_CONFIG_CACHE (all caps), then yarn will just set the lowercase variant of the variable, and node will call CreateProcess which will contain the variable twice). This breaks some tools that create dictionaries out of the environment block because they expect to run into each variable exactly once (regardless of casing).

In my case we have a nodejs script that we invoke via yarn, and the nodejs script calls a Windows build utility (msbuild) which crashes when trying to set up options for the compiler when it finds this option twice. We run into this when building in our CI in Azure DevOps pipeline, which sets up NPM_CONFIG_CACHE in the environment.

Related:

• https://github.com/actions/virtual-environments/pull/1566/files
• MSBuild should sanitize environment block on startup dotnet/msbuild#5726

How often does it reproduce? Is there a required condition?

100%

What is the expected behavior?

node should de-duplicate variables before passing them to CreateProcess

What do you see instead?

CreateProcess gets an environment block with both variables:

Additional information

[bzoz]

This is basically a duplicate of #34667

When spawn gets the object with the env pairs there is no way for it to know which one is the "true one", there is no record of which was added later. This is a known, documented limitation and unfortunately, there is no way of fixing this in Node.

In this case, yarn needs to be fixed to correctly handle environment variables on Windows.

[bnoordhuis]

Closing as a duplicate.

[asklar]

@bnoordhuis @bzoz The documentation only mentions PATH as being the special case but in fact all environment variables are susceptible to this problem. Could you please update the documentation to reflect this?

Moreover, the doc says it will sort the variables lexicographically and use the first occurrence of PATH. This is the same behavior that the OS GetEnvironmentVariable will do, so once you get into the duplicate variable case, it doesn't matter which one was added last - the only one that is reachable via GetEnvironmentVariable is going to be the first lexicographically, so spawn could sanitize the env block by doing exactly that, only keep the first instance of each variable

[bzoz]

Those points make sense, both the doc update and the sanitization.

I think keeping only one entry for each env variable would be a semver-major though.

[asklar]

@bzoz any clue when we might see this fixed?

[bzoz]

I've opened a PR with a fix: #35210. I guess this is a semver-major though, so I would expect this to be shipped with the next major release of Node.

[asklar]

thanks a lot @bzoz. From the release doc it sounds like the next release is October so that would be ok assuming we can get this in soon 🤞