[BUG] `prepare` and `prepack` scripts are not handled as documented when installing a git dependency #1865

kimamula · 2020-09-27T06:24:53Z

Current Behavior:

I tested installing a git dependency which has either prepare or prepack script with npm CLI v7.0.0-beta.12 and confirmed the following behaviors.

a package with a prepare script: devDependencies are NOT installed but the prepare script is executed
a package with a prepack script: devDependencies are NOT installed and the prepack script is NOT executed

Expected Behavior:

a package with a prepare script: devDependencies are installed and the prepare script is executed
a package with a prepack script: devDependencies are installed and the prepack script is executed
- Though I've not found any document saying that devDependencies are installed when installing a git dependency with a prepack script and actually they are not installed even when npm CLI v6 is used, I believe npm should install them since running a prepack script without installing devDependencies does not make sense.

Steps To Reproduce:

prepare:

// https://github.com/kimamula/prepare-test/blob/master/package.json
{
  "name": "prepare-test",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "prepare": "tsc"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "typescript": "^4.0.3"
  }
}

$ npm install github:kimamula/prepare-test
npm ERR! code 127
npm ERR! git dep preparation failed
npm ERR! command /Users/kenji/.nodenv/versions/12.18.3/bin/node /Users/kenji/.nodenv/versions/12.18.3/lib/node_modules/npm/bin/npm-cli.js install --only=dev --prod --ignore-prepublish --no-progress --no-save --cache=/Users/kenji/.npm/_cacache --prefer-offline=false --prefer-online=false --offline=false --before=
npm ERR! > [email protected] prepare
npm ERR! > tsc
npm ERR! npm WARN invalid config before="" set in command line options
npm ERR! npm WARN invalid config Must be one of: null, valid Date string
npm ERR! sh: tsc: command not found
npm ERR! npm ERR! code 127
npm ERR! npm ERR! path /Users/kenji/.npm/_cacache/tmp/git-clone-a73f300b
npm ERR! npm ERR! command failed
npm ERR! npm ERR! command sh -c tsc
npm ERR! 
npm ERR! npm ERR! A complete log of this run can be found in:
npm ERR! npm ERR!     /Users/kenji/.npm/_cacache/_logs/2020-09-27T06_13_19_941Z-debug.log

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/kenji/.npm/_logs/2020-09-27T06_13_19_972Z-debug.log

prepack:

// https://github.com/kimamula/prepack-test/blob/master/package.json
{
  "name": "prepack-test",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "prepack": "tsc"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "typescript": "^4.0.3"
  }
}

$ npm install github:kimamula/prepack-test

changed 1 package, and audited 1 package in 6s

found 0 vulnerabilities

-> tsc should generate index.js file, but it's not generated.

Environment:

OS: macOS 10.15.6
Node: 12.18.3
npm: 7.0.0-beta.12

The text was updated successfully, but these errors were encountered:

darcyclarke · 2020-10-01T22:07:14Z

@kimamula just to confirm, does this behavior reflect how [email protected] works today? Or, have we changed how that works?

kimamula · 2020-10-02T00:40:21Z

@darcyclarke The behavior of [email protected] is different (see #1229).

a package with a prepare script: working as expected (devDependencies are installed and the prepare script is executed)
a package with a prepack script: devDependencies are NOT installed but the prepack script is executed

aomarks · 2020-10-21T22:17:21Z

I seem to be hitting a similar case but where the prepare script is running without dependencies being available under NPM 7. This particular dependency is not published to NPM, so I am installing it as a Git dependency specification, not sure if that is related.

With NPM 6, the following worked:

npm install --save-dev git+https://github.com/codemirror/google-modes.git#57b26bb0e76ca5d3b83b12faf13ce1054d34bdd

But with NPM 7 it fails, seemingly because the grammar-mode binary that this package's prepare script depends on is not available:

npm install --save-dev git+https://github.com/codemirror/google-modes.git#57b26bb0e76ca5d3b83b12faf13ce1054d34bdd
npm ERR! code 127
npm ERR! git dep preparation failed
npm ERR! command /home/aomarks/node/bin/node /home/aomarks/node/lib/node_modules/npm/bin/npm-cli.js install --only=dev --prod --ignore-prepublish --no-progress --no-save --cache=/home/aomarks/.npm/_cacache --prefer-offline=false --prefer-online=false --offline=false --before=
npm ERR! > [email protected] prepare
npm ERR! > npm run build && npm test
npm ERR! 
npm ERR! 
npm ERR! > [email protected] build
npm ERR! > npm run build-js && npm run build-ts && npm run build-c && npm run build-cpp && npm run build-clif && npm run build-fbs && npm run build-py && npm run build-go && npm run build-java && npm run build-html && npm run build-ng && npm run build-kotlin
npm ERR! 
npm ERR! 
npm ERR! > [email protected] build-js
npm ERR! > grammar-mode --es-module src/javascript.grammar --output src/javascript.mode.js && rollup -c rollup.config.js src/javascript.js -o dist/javascript.js
npm ERR! npm WARN invalid config before="" set in command line options
npm ERR! npm WARN invalid config Must be one of: null, valid Date string
npm ERR! sh: 1: grammar-mode: not found
npm ERR! npm ERR! code 127
npm ERR! npm ERR! path /home/aomarks/.npm/_cacache/tmp/git-clone-ae2eb4f5
npm ERR! npm ERR! command failed
npm ERR! npm ERR! command sh -c grammar-mode --es-module src/javascript.grammar --output src/javascript.mode.js && rollup -c rollup.config.js src/javascript.js -o dist/javascript.js
npm ERR! 
npm ERR! npm ERR! A complete log of this run can be found in:
npm ERR! npm ERR!     /home/aomarks/.npm/_cacache/_logs/2020-10-21T22_12_07_990Z-debug.log
npm ERR! npm ERR! code 127
npm ERR! npm ERR! path /home/aomarks/.npm/_cacache/tmp/git-clone-ae2eb4f5
npm ERR! npm ERR! command failed
npm ERR! npm ERR! command sh -c npm run build-js && npm run build-ts && npm run build-c && npm run build-cpp && npm run build-clif && npm run build-fbs && npm run build-py && npm run build-go && npm run build-java && npm run build-html && npm run build-ng && npm run build-kotlin
npm ERR! 
npm ERR! npm ERR! A complete log of this run can be found in:
npm ERR! npm ERR!     /home/aomarks/.npm/_cacache/_logs/2020-10-21T22_12_08_013Z-debug.log
npm ERR! npm ERR! code 127
npm ERR! npm ERR! path /home/aomarks/.npm/_cacache/tmp/git-clone-ae2eb4f5
npm ERR! npm ERR! command failed
npm ERR! npm ERR! command sh -c npm run build && npm test
npm ERR! 
npm ERR! npm ERR! A complete log of this run can be found in:
npm ERR! npm ERR!     /home/aomarks/.npm/_cacache/_logs/2020-10-21T22_12_08_023Z-debug.log

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/aomarks/.npm/_logs/2020-10-21T22_12_08_070Z-debug.log

npm/cli#1865

zloirock · 2020-10-26T06:56:38Z

This bug causes problems for core-js.

https://travis-ci.org/github/zloirock/core-js/builds/738834230

isaacs · 2020-11-05T18:51:03Z

This is a bug in pacote. npm/pacote#53

isaacs · 2020-11-05T18:55:37Z

Will be fixed in the next cli release (ie, tomorrow, likely included with the Node.js 15 release going out next week).

jaydenseric · 2020-11-06T02:51:42Z

Are these related issues?

[BUG] Errors when installing a GitHub repo dependency with a prepare script using a command from a dev dependency #2084
[BUG] Installing git dependencies that share dependencies with the parent package fails #2106

isaacs · 2020-11-06T18:11:25Z

Yeah, those look like duplicates.

fregante · 2020-12-30T08:27:30Z

Can anyone double-check that this problem has actually been fixed? prepack is not being run in npm 7.0.9 nor the latest npm (7.3.0). prepare works.

npm i fregante/select-dom#edd18f4 --verbose # with prepack
npm i fregante/select-dom#32f1864 --verbose # with prepare

Testing on Node v15.2.1, macOS 10.15.7. I tried with npm 6 and prepack is being run (but it fails, likely due to missing dev dependencies)

aomarks · 2021-03-03T00:13:18Z

I can confirm it works for me now, at least for my particular case outlined here: #1865 (comment)

fregante · 2021-03-03T00:34:42Z

Still no luck for me on npm 7.6.0. This still results in an semi-empty package folder

npm i fregante/select-dom#edd18f4 --verbose # with prepack

zimtsui · 2021-07-12T08:30:38Z

why is this issue closed? the problem has not been solved in 7.19.1

@master

During npm install, the Lighthouse package which is installed from GitHub (@master branch) triggers the newly added `prepack` script [1], but due to a bug in NPM [2] the devDependencies are not available when the script is run, thus making the installation fail. [1]: GoogleChrome/lighthouse#13261 [2]: npm/cli#1865

@master

During npm install, the Lighthouse package which is installed from GitHub (@master branch) triggers the newly added `prepack` script [1], but due to a bug in NPM [2] the devDependencies are not available when the script is run, thus making the installation fail. [1]: GoogleChrome/lighthouse#13261 [2]: npm/cli#1865

jameskerr · 2022-01-13T20:01:12Z

There were two problems documented in this issue.

dev dependencies were not being installed
prepack was not running

The first has been fixed, the second has not.

Repro Example

I have a test package called npm-lib with the following package.json

{
  "scripts": {
    "prepack": "ls | cowsay > prepacked.txt",
    "prepare": "ls | cowsay > prepared.txt"
  },
  "devDependencies": {
    "cowsay": "^1.5.0"
  }
}

I add this to another package like so with the commit hash:

"dependencies": {
    "npm-lib": "github:jameskerr/npm-lib#7cb685db9a232c44057b2be3a97cff183972b4cd"
  },

I then run npm install with npm@6 and npm@7 and look at the contents of node_modules/npm-lib.

npm version 6.14.15

$ ls node_modules/npm-lib
README.md     
index.js      
package.json  
prepacked.txt 
prepared.txt

npm version 7.24.2

ls node_modules/npm-lib
README.md    
index.js     
package.json 
prepared.txt

The contents of the prepared.txt show that the dev dependency (cowsay) was installed:

cat node_modules/npm-lib/prepared.txt
 ___________________
/ README.md         \
| index.js          |
| main.go           |
| node_modules      |
| package-lock.json |
| package.json      |
\ prepared.txt      /
 -------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Should I open a new issue?

fwcd · 2023-01-12T01:08:45Z

I can confirm that the issue that devDependencies are not installed before prepare is executed is not solved, I have a TypeScript package with a package.json like this:

{
  "scripts": {
    "prepare": "tsc ..."
  },
  "devDependencies": {
    "typescript": "..."
  }
}

Running npm install with npm 9.2.0 errors with

npm ERR! sh: tsc: command not found

whereas yarn handles this installation perfectly well. As a temporary workaround the required devDependencies can be installed globally, in this case the TypeScript compiler, but as a solution this isn't great.

This still seems to cause trouble though, see npm/cli#1865

aarondill · 2023-02-04T05:54:55Z

I can confirm that on npm version 9.4.1, prepack is still not being called when installing git dependencies.
the test in #1865 (comment) demonstrate this issue perfectly. prepare is run, but prepack is never called.

aarondill · 2023-02-04T06:22:42Z

workspaces/libnpmpack/lib/index.js is the only place in this codebase which calls prepack, it seems like the dependency pacote anticipates the cli to handle prepack(pacote/blob/main/lib/dir.js#L39-L41), yet it's only handled on npm pack and npm publish
A new issue in npm/pacote has been created to correspond with this one until the problem is solved: npm/pacote#257

ljharb · 2023-02-04T16:31:28Z

Why would prepack run with git deps, when they’re never packed or published?

fregante · 2023-02-04T16:36:25Z

That makes sense, but the lifecycle of npm scripts still seems alien to me. I recall prepack shortly being the preferred way to execute scripts before publishing a few years back.

Installing from git bypasses the entire packaging and publishing sequence that repos go through when running npm publish, in a way I'd expect the same sequence to be repeated when running npm install user/repo. Otherwise we get these weird lifecycle events that vary depending on how the package is installed.

ljharb · 2023-02-04T16:38:38Z

Prepack works great for a “before publish” step; i use it - but git deps are never published. prepare is used when you want to support installing from git (which i have no interest in), and it should run pre publish and pre install.

fregante · 2023-02-04T16:42:37Z

Prepack works great for a “before publish” step; i use it -

Exactly, but why do you use it? Is it to run a build? Then it'd make sense to have a script that runs when installing from git

to support installing from git (which i have no interest in)

Do you say that as a module author? Then yeah, that's not great, is it. As a module author myself, I shouldn't need to bother about that either.

Does npm install support installing from git or not? These half-implementations really cause a lot of confusion. Why should anyone use prepack if prepare is the right one to support both? I wish there was one way to do things in npm land.

ljharb · 2023-02-04T16:57:43Z

I use it to autogenerate an npmignore file, which isn’t a relevant task unless it’s being published.

As a module author, i always need to bother with the way my users install things - anything i support is something i have to support, and since i believe nothing should ever be consumed except from an npm registry, that’s what i wish to support.

npm install supports that, but the thing you’re installing may not.

aarondill · 2023-02-05T01:16:37Z

Why should anyone use prepack if prepare is the right one to support both?

prepare runs on local npm install with no arguments, thus when cloning a repo and installing dependencies, the prepare script is run, which would likely build the project, as well as filling the terminal with lots of information that's only relevant to building(as many build steps output debug information)

From the docs, prepack should run everywhere prepare does, except for local npm install, thus still allowing usage of npm pack and installation from git repos, without auto-building when installing dependencies.

jakebailey · 2023-03-30T21:46:39Z

Not to pile on here, but, we were actually considering adding a prepack to TypeScript to make bisecting checker changes simpler, only to find out that prepack doesn't actually work in v7+ anymore.

prepare is the wrong script to be using for our needs, as we'd incur a full build at a plain npm install or even npm ci at the repo root, wheras prepack is much better and used by yarn. But, given npm doesn't seem to support it anymore, we just gave up on it.

Does this need to be filed as a brand new issue? Reopened? Is npm/pacote#257 the right place for this now?

aovchinn · 2023-05-09T18:13:07Z

is this happening again ?
I tested it on npm 7 and 9
created #6435

jakebailey · 2023-05-09T19:57:02Z

@aovchinn Not sure your linked issue is relavent to this issue; this issue is about npm not executing the prepack script on installs via git in v7+, not about missing things on PATH.

Er, no, sorry. This issue contains multiple issues all in one.

kimamula added Bug thing that needs fixing Needs Triage needs review for next steps Release 7.x work is associated with a specific npm 7 release labels Sep 27, 2020

darcyclarke added beta and removed Needs Triage needs review for next steps labels Oct 1, 2020

darcyclarke added this to the OSS - Sprint 16 milestone Oct 1, 2020

darcyclarke modified the milestones: OSS - Sprint 16, OSS - Sprint 17, OSS - Sprint 18 Oct 5, 2020

darcyclarke removed the beta label Oct 19, 2020

darcyclarke modified the milestones: OSS - Sprint 18, OSS - Sprint 19 Oct 19, 2020

zloirock added a commit to zloirock/core-js that referenced this issue Oct 26, 2020

temporarily disable node@15 in travis because of npm@7 bug

42e22ad

npm/cli#1865

zloirock mentioned this issue Oct 27, 2020

CI: Travis CI -> Github actions zloirock/core-js#875

Merged

isaacs mentioned this issue Nov 5, 2020

[BUG] npmInstallCommand is incorrect npm/pacote#53

Closed

isaacs closed this as completed in npm/pacote@f536ec0 Nov 5, 2020

isaacs reopened this Nov 5, 2020

isaacs closed this as completed in 1dad328 Nov 6, 2020

darcyclarke mentioned this issue Nov 13, 2020

deps: upgrade npm to 7.0.11 nodejs/node#36112

Closed

RomanH2 mentioned this issue Mar 23, 2021

devDependencies of git dependency are missing during prepare when installing with --prod in main package #2920

Closed

sgratzl mentioned this issue Jan 27, 2022

possible fix for npm install lineupjs/lineupjs#541

Merged

3 tasks

fwcd added a commit to ProjectLighthouseCAU/nighthouse that referenced this issue Jan 12, 2023

Run build on prepare again

f5194ee

This still seems to cause trouble though, see npm/cli#1865

This was referenced Feb 4, 2023

[BUG] prepack is not called on installation of git packages npm/pacote#257

Open

change prepublishOnly to prepack uetchy/create-create-app#64

Open

hut8 mentioned this issue Jun 3, 2023

Fix tests and large number of security vulnerabilities supervillain-software-group/axios-rest-resource#18

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] `prepare` and `prepack` scripts are not handled as documented when installing a git dependency #1865

[BUG] `prepare` and `prepack` scripts are not handled as documented when installing a git dependency #1865

kimamula commented Sep 27, 2020 •

edited

Loading

darcyclarke commented Oct 1, 2020

kimamula commented Oct 2, 2020

aomarks commented Oct 21, 2020 •

edited

Loading

zloirock commented Oct 26, 2020

isaacs commented Nov 5, 2020

isaacs commented Nov 5, 2020

jaydenseric commented Nov 6, 2020

isaacs commented Nov 6, 2020

fregante commented Dec 30, 2020 •

edited

Loading

aomarks commented Mar 3, 2021

fregante commented Mar 3, 2021

zimtsui commented Jul 12, 2021

jameskerr commented Jan 13, 2022

fwcd commented Jan 12, 2023

aarondill commented Feb 4, 2023

aarondill commented Feb 4, 2023 •

edited

Loading

ljharb commented Feb 4, 2023

fregante commented Feb 4, 2023

ljharb commented Feb 4, 2023

fregante commented Feb 4, 2023 •

edited

Loading

ljharb commented Feb 4, 2023 •

edited

Loading

aarondill commented Feb 5, 2023 •

edited

Loading

jakebailey commented Mar 30, 2023

aovchinn commented May 9, 2023

jakebailey commented May 9, 2023 •

edited

Loading

[BUG] prepare and prepack scripts are not handled as documented when installing a git dependency #1865

[BUG] prepare and prepack scripts are not handled as documented when installing a git dependency #1865

Comments

kimamula commented Sep 27, 2020 • edited Loading

Current Behavior:

Expected Behavior:

Steps To Reproduce:

Environment:

darcyclarke commented Oct 1, 2020

kimamula commented Oct 2, 2020

aomarks commented Oct 21, 2020 • edited Loading

zloirock commented Oct 26, 2020

isaacs commented Nov 5, 2020

isaacs commented Nov 5, 2020

jaydenseric commented Nov 6, 2020

isaacs commented Nov 6, 2020

fregante commented Dec 30, 2020 • edited Loading

aomarks commented Mar 3, 2021

fregante commented Mar 3, 2021

zimtsui commented Jul 12, 2021

jameskerr commented Jan 13, 2022

fwcd commented Jan 12, 2023

aarondill commented Feb 4, 2023

aarondill commented Feb 4, 2023 • edited Loading

ljharb commented Feb 4, 2023

fregante commented Feb 4, 2023

ljharb commented Feb 4, 2023

fregante commented Feb 4, 2023 • edited Loading

ljharb commented Feb 4, 2023 • edited Loading

aarondill commented Feb 5, 2023 • edited Loading

jakebailey commented Mar 30, 2023

aovchinn commented May 9, 2023

jakebailey commented May 9, 2023 • edited Loading

[BUG] `prepare` and `prepack` scripts are not handled as documented when installing a git dependency #1865

[BUG] `prepare` and `prepack` scripts are not handled as documented when installing a git dependency #1865

kimamula commented Sep 27, 2020 •

edited

Loading

aomarks commented Oct 21, 2020 •

edited

Loading

fregante commented Dec 30, 2020 •

edited

Loading

aarondill commented Feb 4, 2023 •

edited

Loading

fregante commented Feb 4, 2023 •

edited

Loading

ljharb commented Feb 4, 2023 •

edited

Loading

aarondill commented Feb 5, 2023 •

edited

Loading

jakebailey commented May 9, 2023 •

edited

Loading