npm install task (not msbuild) fails with EPERM on hosted build

[rosieks]

It looks like there are some problems with permissions while installing various packages over npm install.

When I run my build for the 1st time I received the following errors:

npm ERR! Error: EPERM, open 'C:\NPM\Cache\bb33797c-ache-minimatch-2-0-8-package-tgz.lock'
npm ERR! { [Error: EPERM, open 'C:\NPM\Cache\bb33797c-ache-minimatch-2-0-8-package-tgz.lock']
npm ERR! errno: 50,
npm ERR! code: 'EPERM',
npm ERR! path: 'C:\\NPM\\Cache\\bb33797c-ache-minimatch-2-0-8-package-tgz.lock' }
npm ERR! 

After that I just run my build for the 2nd time without any modifications and I got the following errors:

npm ERR! Error: EPERM, open 'C:\NPM\Cache\e360d466-Cache-inherits-2-0-1-package-tgz.lock'
npm ERR! { [Error: EPERM, open 'C:\NPM\Cache\e360d466-Cache-inherits-2-0-1-package-tgz.lock']
npm ERR! errno: 50,
npm ERR! code: 'EPERM',
npm ERR! path: 'C:\\NPM\\Cache\\e360d466-Cache-inherits-2-0-1-package-tgz.lock' }
npm ERR! 

I receiving that errors on VSO

[bryanmacfarlane]

the front page of the agent repo talks about the npm issue
https://github.com/Microsoft/vso-agent

[rosieks]

Front page talks about installing vsoagent, but I have that issue on hosted pool/agent while running npm install task during build?

[bryanmacfarlane]

Ahhh. interesting. Are you installing from npm globally? (-g)

[rosieks]

I don't think so. I just add Npm Install task to my build and setup working directory to folder where is my package.json

[lsoltys]

I have the same problem with 'Npm install' build step on Visual Studio Online using hosted pool agent to build.

[bryanmacfarlane]

OK - we will investigate today

[densto88]

Hey guys, sorry you're hitting this issue. I think I see what the problem is, it should be resolved shortly.

Thanks,
Dennis

[bryanmacfarlane]

Closing since the fix is about to rollout to hosted

[bryanmacfarlane]

Re-opening for Dennis to confirm

[densto88]

It looks like my original theory wasn't correct. Can either of you provide me your visualstudio.com account name so I can look up which data center you're account is in? You can email me at [email protected] if you're uncomfortable posting it here.

Other questions that may help us diagnose the issue:
Have you been able to do this exact step successfully in the past? If so, when did it last succeed and when did you first start seeing it fail?

Does the step fail for all npm packages or just certain ones? If so, which have you seen it succeed for and which failed?

Thanks,
Dennis

[lsoltys]

Hi, My VSO account name is rcs-pol. For the first time I used 'npm install' build step yesterday and I had these problems from this first time. The only successful step was about 3 hours ago. I thought you had solved the problem. However now this step fails again. Error from build log:
2015-07-16T13:06:32.9426115Z ##[error]npm ERR! EEXIST, open 'C:\NPM\Cache\a82d4a1c--Cache-lodash-3-10-0-package-tgz.lock'
2015-07-16T13:06:32.9436102Z ##[error]File exists: C:\NPM\Cache\a82d4a1c--Cache-lodash-3-10-0-package-tgz.lock
2015-07-16T13:06:32.9436102Z ##[error]Move it away, and try again.
2015-07-16T13:06:32.9446106Z ##[error]npm ERR! System Windows_NT 6.2.9200
2015-07-16T13:06:32.9456106Z ##[error]npm ERR! command "C:\Program Files\nodejs\node.exe" "C:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js" "install"
2015-07-16T13:06:32.9496111Z ##[error]npm ERR! cwd C:\a\4f767487\EuradSB\src\EuradWebClient
2015-07-16T13:06:32.9516111Z ##[error]npm ERR! node -v v0.10.32
2015-07-16T13:06:32.9526111Z ##[error]npm ERR! npm -v 1.4.28
2015-07-16T13:06:32.9546110Z ##[error]npm ERR! path C:\NPM\Cache\a82d4a1c--Cache-lodash-3-10-0-package-tgz.lock
2015-07-16T13:06:32.9576107Z ##[error]npm ERR! code EEXIST
2015-07-16T13:06:32.9716105Z ##[error]npm ERR! errno 47
2015-07-16T13:06:33.6864108Z ##[error]npm
2015-07-16T13:06:34.1110534Z ##[error]Unexpected exit code 47 returned from tool npm.cmd

[bryanmacfarlane]

Our theory is currently this: http://stackoverflow.com/questions/26491311/lock-file-gets-in-the-way-when-trying-to-generate-yeoman-chrome-extension

If that is the issue, we need to update npm on our build image

[cwoolum]

I've gotten this issue randomly as well. I've run into an issue previously where if multiple builds that call "npm install" run at the same time, the lock file from the first npm install will cause the second one to fail. I didn't have much time to try and fix the issue with NPM and ended up only being able to limit builds to one concurrent NPM install at once. I did read that you can override the global cache folder per run using --cache=/folder but didn't get much farther on it.

npm/npm#2500

[bryanmacfarlane]

We're you running multiple agents on the same machine (that might be another workaround). Interesting that you can set cache folder - we could set local to each agent. Although it seems like it's npm's job to protect's it's cache. We'll look at the workaround. thx

[cwoolum]

I think local to each agent would be the best bet. My specific use case was with Jenkins and multiple agents but the idea was the same.

[AbraaoAlves]

This happened to me too. A quick solution was to make a agent for a Linux machine already set up. The problem is that we have no control over the npm. Eg: install npm try to compile many packages by OS and depending on the package it may require admin access, installed tools on machine, or even a more advanced npm version (like node-sass package that requires npm ^ 2.x ). Things we can not do in HostedBuild

One of the things that can put an end to these problems will be docker support , which will give us control of the environment even in the hosted environment.

[brutaldev]

Also having the same issues here, I have been unable to use npm as a pre-build step as it always fails for the same error number 4048 (but a different file path each time).

EXEC : npm ERR! error : EPERM, rename 'C:\NPM\Cache\gulp\3.9.0\package\package.json' [d:\a\src\AppProject.csproj]
     npm ERR!     at Error (native)
     npm ERR!  { [Error: EPERM, rename 'C:\NPM\Cache\gulp\3.9.0\package\package.json']
     npm ERR!   errno: -4048,
     npm ERR!   code: 'EPERM',
     npm ERR!   path: 'C:\\NPM\\Cache\\gulp\\3.9.0\\package\\package.json',
     npm ERR!   parent: 'AppProject' }
     npm ERR! 
     npm ERR! Please try running this command again as root/Administrator.

[thechrisjohnson]

The image with this fix went out on Friday/Monday. Are you still seeing the permission issue?

[brutaldev]

@thechrisjohnson Yes, I just queued up another build and still got the same failure message.

Build started 7/30/2015 4:39:31 PM.
...
npm install
npm ERR! Windows_NT 6.3.9600
npm ERR! argv "C:\\Program Files\\nodejs\\\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "install"
npm ERR! node v0.12.7
npm ERR! npm  v2.11.3
npm ERR! path C:\NPM\Cache\gulp\3.9.0\package\package.json
npm ERR! code EPERM
npm ERR! errno -4048

Anything else you need from me to investigate further?

[thechrisjohnson]

I'm unable to repro this issue just running npm install on a test project I have. What packages are you trying to install when this occurs?

[brutaldev]

@thechrisjohnson It works on my machine, but only fails when running the build in VSO using the Hosted Build Controller.

[thechrisjohnson]

Oh, this is a xaml based build?

[chrispat]

Can you provide a package.json file that repros that we can try?

From: Werner van Deventer [mailto:[email protected]]
Sent: Thursday, July 30, 2015 2:08 PM
To: Microsoft/vso-agent-tasks [email protected]
Subject: Re: [vso-agent-tasks] npm install task fail on installing various packages (#323)

@thechrisjohnsonhttps://github.com/thechrisjohnson It works on my machine, but only fails when running the build in VSO using the Hosted Build Controller.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/323#issuecomment-126421083.

[brutaldev]

@thechrisjohnson
Created a codeless project just to build with Gulp, it has no assembly output.

Just some simple MSBUILD Exec tasks in the .csproj file.

<Project ToolsVersion="12.0" DefaultTargets="GulpBuild" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
...
<PropertyGroup>
  <CompileDependsOn>
    $(CompileDependsOn);
    GulpBuild;
  </CompileDependsOn>
</PropertyGroup>
...
<Target Name="GulpBuild">
    <Exec Command="if not exist C:\Users\buildguest\AppData\Roaming\npm mkdir C:\Users\buildguest\AppData\Roaming\npm" />
    <Exec Command="npm install" />
    <Exec Command="node_modules\.bin\bower install" />
    <Exec Command="gulp release" />
  </Target>

Deploy copy steps and so forth omitted since it can't get past the npm install. This was based on a number of different sites:
http://www.codecadwallader.com/2015/03/15/integrating-gulp-into-your-tfs-builds-and-web-deploy/
http://blogs.msdn.com/b/nicktrog/archive/2015/03/05/adding-gulp-to-visual-studio-online-builds.aspx
http://www.davepaquette.com/archive/2015/04/08/integrating-gulp-and-bower-with-visual-studio-online-hosted-builds.aspx

You can use any packages I guess, it fails with different names on each build so it appears random as to when exactly it fails. Here is a sample of my package.json.

{
  "name": "MyProject",
  "private": "true",
  "version": "0.0.1",
  "description": "",
  "dependencies": {},
  "devDependencies": {
    "bower": "^1.4.1",
    "browser-sync": "^2.7.12",
    "del": "~1.2.0",
    "gulp": "^3.9.0",
    "gulp-autoprefixer": "~2.3.1",
    "gulp-concat": "~2.5.2",
    "gulp-consolidate": "~0.1.2",
    "gulp-csso": "~1.0.0",
    "gulp-declare": "~0.3.0",
    "gulp-ext-replace": "^0.2.0",
    "gulp-foreach": "^0.1.0",
    "gulp-handlebars": "^4.0.0",
    "gulp-hb": "^2.6.0",
    "gulp-if": "~1.2.5",
    "gulp-imagemin": "~2.2.1",
    "gulp-jsbeautifier": "0.0.8",
    "gulp-jshint": "~1.11.0",
    "gulp-load-plugins": "~0.10.0",
    "gulp-minify-html": "~1.0.3",
    "gulp-minify-inline": "^0.1.0",
    "gulp-newer": "~0.5.0",
    "gulp-ng-annotate": "^1.0.0",
    "gulp-plumber": "~1.0.1",
    "gulp-print": "~1.1.0",
    "gulp-sass": "^2.0.3",
    "gulp-uglify": "~1.2.0",
    "gulp-usemin": "~0.3.11",
    "gulp-wrap": "~0.11.0",
    "imagemin-jpegtran": "^4.2.0",
    "imagemin-pngquant": "~4.1.0",
    "jshint-stylish": "~2.0.0",
    "merge-stream": "^0.1.7",
    "run-sequence": "~1.1.1"
  }
}

[thechrisjohnson]

Thanks! I was able to repro the issue and will push out a fix within the next day or so.

[brutaldev]

Awesome! Let me know when it's done and I'll give it a test.

[bryanmacfarlane]

It would also be good to try the same thing from the new build system (+ definition from the web view)
http://aka.ms/vsopreview

[rosieks]

I've tried to run my orginal build (based on new build system) and now NPM part works.

[bryanmacfarlane]

@Bartmax - the npm bug is a race condition that repros on windows with low I/O. This isn't a CI issue - I sent npm a repro (way up in the thread) with a package.json and npm cmd line (no CI involved). We just run cmd line just like Travis. Travis also doesn't build on windows.

And yes, as I pointed out in the thread way above, there's an inexpensive workaround which is to pass --force as additional args to npm. That causes it to bypass the cache which avoids the race and deep graphs with multiple writes without locking to the cache dir ( npm/npm#9696 ). We are considering just doing that on hosted (with retries like other CIs do) since it's always a fresh machine and the cache doesn't help much.

[tghamm]

We're getting this error using the NPM task (with --force) on our self-hosted instance consistently (we just set it up in the last couple of days):

npm ERR! syscall rename
npm ERR! Error: EPERM: operation not permitted, rename     'C:\Users\tfsbuild\AppData\Roaming\npm-    cache\minimist\1.2.0\package\package.json.414d5643225a5c0e9714b0cdc0eaf8b4' ->     'C:\Users\tfsbuild\AppData\Roaming\npm-cache\minimist\1.2.0\package\package.json'
npm ERR! at Error (native)
npm ERR! { [Error: EPERM: operation not permitted, rename 'C:\Users\tfsbuild\AppData\Roaming\npm-cache\minimist\1.2.0\package\package.json.414d5643225a5c0e9714b0cdc0eaf8b4' -> 'C:\Users\tfsbuild\AppData\Roaming\npm-cache\minimist\1.2.0\package\package.json']
npm ERR! errno: -4048,
npm ERR! code: 'EPERM',
npm ERR! syscall: 'rename',
npm ERR! path: 'C:\\Users\\tfsbuild\\AppData\\Roaming\\npm-cache\\minimist\\1.2.0\\package\\package.json.414d5643225a5c0e9714b0cdc0eaf8b4',
npm ERR! dest: 'C:\\Users\\tfsbuild\\AppData\\Roaming\\npm-cache\\minimist\\1.2.0\\package\\package.json',
npm ERR! parent: 'gulp-util' }
npm ERR!
npm ERR! Please try running this command again as root/Administrator.

Is there any other way to get this working?

[bryanmacfarlane]

@tghamm - npm is writing to the cache folder:

EPERM: operation not permitted, rename 'C:\Users\tfsbuild\AppData\Roaming\npm-cache ...

Which implies --force is not being passed (force fetches and bypassed the cache)

Can you:

• gather your CI steps and inputs
• set system.debug to true on the variables tab of the definition
• gather task output
• since you have your own vm, gather the npm log the output refers to
• one more - your package.json (or at a minimum the dependencies section)
  Zip that up and send it bigbldt at microsoft dot com

[tghamm]

@bryanmacfarlane will do. It's actually working intermittently now so I'm waiting for it to fail again.

[konrad-ha]

Using the "--force" works for now but is far from an optimal solution. We're trying to roll out a rather large pipeline using npm right now and I'm afraid this might get a serious issue.

[bryanmacfarlane]

@konrad-ha - agreed. The only right fix is npm fixing the race condition. --force, retries, and faster I/O are all workarounds/mitigations.

[joaomoreno]

How about a focused hackathon by Microsoft dedicated to fix this for npm?

[ivanz]

I have been running my own VSO agent on a VM with premium storage data disk for 10 days and even though it dramatically improves the situation (and build times) - I still see the occasional EPERM error (very rare though so far)

[Bartmax]

what about a central cache/disk for npm in the microsoft infrastructure. Instead of downloading everything, the vm can attach the cache drive and download there or use it as cache improving speed by a lot and prevent this kind of errors. (same can be done for nuget).

if the vm is run without cache (some option?), no disk will be attached and you can opt in for the current behavior (for those needing that stuff actually gets downloaded)

Personally, It will make me very happy to use hosted agent and don't waste download speed / bandwidth.

[AMoghrabi]

Our team city agents also run into this issue. Using the --force argument does not work unfortunately. Here is a sample build log:

    [11:25:59][Step 1/1] Execute npm_install (13m:23s)
    [11:25:59][Execute npm_install] Running npm with parameters: ["install", "--force"]
    [11:26:03][Execute npm_install] npm WARN using --force I sure hope you know what you are doing.
    ...
    ...
    11:38:47][Execute npm_install] npm ERR! Windows_NT 6.1.7601
    [11:38:48][Execute npm_install] npm ERR! argv "C:\\Program Files\\nodejs\\\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "install" "--force"
    [11:38:48][Execute npm_install] npm ERR! node v0.12.7
    [11:38:48][Execute npm_install] npm ERR! npm  v2.11.3
    [11:38:48][Execute npm_install] npm ERR! path C:\Users\teamcityagent\AppData\Roaming\npm-cache\inherits\2.0.1\package\package.json
    [11:38:48][Execute npm_install] npm ERR! code EPERM
    [11:38:48][Execute npm_install] npm ERR! errno -4048
    [11:38:48][Execute npm_install] 
    [11:38:48][Execute npm_install] npm ERR! Error: EPERM, rename 'C:\Users\teamcityagent\AppData\Roaming\npm-cache\inherits\2.0.1\package\package.json'
    [11:38:48][Execute npm_install] npm ERR!     at Error (native)
    [11:38:48][Execute npm_install] npm ERR!  { [Error: EPERM, rename 'C:\Users\teamcityagent\AppData\Roaming\npm-cache\inherits\2.0.1\package\package.json']
    [11:38:48][Execute npm_install] npm ERR!   errno: -4048,
    [11:38:48][Execute npm_install] npm ERR!   code: 'EPERM',
    [11:38:48][Execute npm_install] npm ERR!   path: 'C:\\Users\\teamcityagent\\AppData\\Roaming\\npm-cache\\inherits\\2.0.1\\package\\package.json',
    [11:38:48][Execute npm_install] npm ERR!   parent: 'npmconf' }
    [11:38:48][Execute npm_install] npm ERR! 
    [11:38:48][Execute npm_install] npm ERR! Please try running this command again as root/Administrator.
    [11:39:22][Execute npm_install] 

Our agents are running on Windows 7 and currently it is not feasible to upgrade the agents to use an SSD...

[bryanmacfarlane]

Lots of progress in finding the root issue from Microsoft open tech. The issue is in npm2. More here: npm/npm#9696

The good news is it's fixed in npm3 ([sudo] npm install npm@3). We're progressing to see if a fix can be pushed to npm2.

We are looking to move hosted build pool to npm3 and we're almost done with premium storage in the hosted pool (even if npm3 fixes, still much faster I/O).

So to sum up the workarounds:

• own VM with premium storage, and/or
• own VM with npm3
• on hosted pool use --force

[joaomoreno]

Very cool!

[Maarten88]

Based on this thread, I may have found a workaround: first install npm@3, then run that using a Command Line task to do the actual npm install:

(npm task) -> npm install npm@3
(Command Line task) -> $(Build.SourcesDirectory)\[your project]\node_modules\.bin\npm.cmd

My npm install tasks finished successfully without having to restart it multiple times for the first time in months!

[Jonlondon]

@Maarten88 +1

[bryanmacfarlane]

Hosted build image is getting npm3 this sprint.

Note that if you have your own agent and you install npm3 (or node5.x) and it resolves in your path (confirm with npm -v), then you can use the npm task (no need use command line task but that's fine as well).

[chrispat]

We had to roll back npm3 because it broke Cordova so it will be a bit longer before we can move to it.

From: Bryan MacFarlane [mailto:[email protected]]
Sent: Friday, January 15, 2016 8:09 AM
To: Microsoft/vso-agent-tasks [email protected]
Cc: Chris Patterson [email protected]
Subject: Re: [vso-agent-tasks] npm install task (not msbuild) fails with EPERM on hosted build (#323)

Hosted build image is getting npm3 this sprint.

Note that if you have your own agent and you install npm3 (or node5.x) and it resolves in your path (confirm with npm -v), then you can use the npm task (no need use command line task but that's fine as well).

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/323#issuecomment-171955379.

[san7hos]

What is the progress on this?

Would it help to simply include node_modules into the repository and basically skip the npm install? Is it a good idea?

[ChBrain]

I would not do that. You kind of break the concept behind npm install. npm install npm@3 takes usually less than 2 minutes for us. npm install of course has to download quite some stuff (depending on project), but thats the point, to keep the dependencies up to date and to latest release.

[bryanmacfarlane]

Progress:

Microsoft open tech fixed the root problem in npm2.
npm/npm#9696
npm/npm@e982858

When that is released, we will update our hosted image to it. On your machines you can do the same or install npm@3

We are also in the process of rolling out premium storage disks to our hosted pool scale units at no cost to our customers. Early numbers show significant I/O gains and have cut the time in half for some of our sample builds. This will minimize the likelyhood of hitting the bug.

This upcoming sprint, we will offer a drop down in the npm task to specify which npm version. It will be leveraging a tools concept we will be adding: https://github.com/Microsoft/vso-agent-tasks/blob/master/docs/tools.md

[ghost]

Good news Bryan and thanks to MS open tech. Will there be an announcement on a blog somewhere when the upcoming sprint is deployed? Better yet would be a note here, it would be fantastic for those subscribed to this thread. -Paul

[san7hos]

Thanks for the information. If I understand you correctly: when using the Hosted Agent, we need to wait until the fix is released. However, the upcoming sprint will bring some improvements sooner and the fix will be released a bit later.

[KaiTwiggy]

There has been some time from the last comment so i was wondering has there been any progress on this issue ? The workaround that @Maarten88 posted works fine, but it would be much cleaner if VSO would provide different versions of npm out of the box.

I was wondering if the dropdown version selection will as well apply to node and other dependencies or is it only npm for now ?

[bryanmacfarlane]

Closing.

It was confirmed as an npm2 bug and the fix was made to npm2
It's not a bug in npm3.
We finished moving most pools to premium storage in the hosted pool (at our cost) which minimizes the chance if you hit it.