Skip to content

Commit

Permalink
Yarn Plug'n'Play: Implementation (yarnpkg#6382)
Browse files Browse the repository at this point in the history
* Initial pnp implementation

* Adds the pnp-env directory for quick checks

* Improves the "yarn node" command so that it works with pnp

Branch: yarn-node-pnp

* Adds the pnp tests to the pkg-tests testsuite

Branch: pkg-tests

* Fixes various issues with the pnp map generation

Branch: pnp-map-improvements

* Remove the scriptsPrependNodePath option

Branch: lifecycle-wrappers-remove-option

* Adds tests to ensure that the lifecycle scripts are called with the right binaries

Branch: lifecycle-wrappers-tests

* Fixes linting

Branch: lifecycle-wrappers-flowlint

* Improve the error message when an optional required dependency hasn't been installed

Branch: lifecycle-wrappers-improve-pnp-error

* Implements lifecycle wrappers

Branch: lifecycle-wrappers-main

* Adds .pnp.js files to the gitignore

* Fallbacks to the toplevel dependencies when a transitive dependency cannot be resolved

* Fixes an issue inside the pkg-tests helper

Branch: pkg-tests-helper-definition

* Improves the fixtures so that they also return info about their dev and peer dependencies

Branch: dev-peer-fixtures

* Finishes to hide plugnplay behind a yarnrc option (`plugnplay-experimental`)

Branch: option-plugnplay-experimental

* Adds failing tests for pnp peer dependencies

Branch: pnp-peer-dependencies-failing

* Prettifies the generated pnp files

* Adds the plugnplay flag to the integrity check

* Removes the pnp file at link-time if installing with pnp disabled

* Improves pnp maps compatibility

* Prevents "yarn check" from checking the node_modules existence when under pnp

* Adds a missing package to the request cache

* Fixes tests

Branch: pnp-fix-tests

* Reimplements the resolution to correctly account for peer dependencies

Branch: better-faster-stronger

* Externalizes the creation of proxy scripts

Branch: portable-proxy-scripts

* Implements support for `yarn run` within pnp-enabled installations

Branch: pnp-yarn-run

* Makes it possible for dependency binaries to require their own dependencies

Branch: pnp-yarn-run-dependencies

* Moves the pnp embed api into its own file

* Adds the test folder to Jest error messages

Branch: jest-better-message

* Improves peer dependency tests

Branch: peer-deps-tests

* Adds tests for require.resolve

Branch: require-resolve-tests

* Refactors the pnp file

Branch: pnp-file-refactoring

* Updates the build-webpack script to include a custom resolver

Branch: webpack-resolver

* Fixes tests

Branch: fix-tests

* Generates virtual when using peer dependencies

Branch: virtual-packages

* Refactors the generated pnp files to use the newly generated maps

Branch: pnp-refactor-find

* Implements Module._findPath

Branch: module-findpath

* Adds tests for workspaces

Branch: workspace-tests

* Fixes top level detection

Branch: fix-top-level-detection

* Implements workspaces support in pnp

Branch: pnp-workspaces

* Automatically adds workspaces as dependencies of the top-level

Branch: auto-workspace-dependencies

* Updates the cache path to include the 'node_modules' string

Branch: node-modules-cache

* Prevents pnp from bootstrapping when running non-pnp-installed scripts

Branch: fix-npm-run

* Installs peer dependencies symlinks inside a project folder

Branch: per-project-virtual-deps

* Adds a sample application that showcases webpack, babel, react, jest

Branch: sample-app

* Ignores the .pnp directory

Branch: ignore-pnp

* Implements a blacklist that throws nicer errors when a package is required through a realpath'd path.

Branch: location-blacklist

* Changes the order the locations are matched to package locators

Branch: location-to-locator-match-order

* Automatically adds packages as dependencies of themselves if possible

Branch: implicit-self

* Updates the node resolution

Branch: node-resolution-improvements

* Ensures that binaries are set as executable in the cache

Branch: chmod-bins

* Disables integrity checks when running under pnp

Branch: pnp-no-integrity

* Makes the .pnp.js file an actual executable that can be used as a resolution server

Branch: pnp-executable

* Removes pnp-env

* Fixes a few tests

* Fixes snapshots

* Fixes the lockfile not being written when using pnp

Branch: fix-lockfile

* Fixes the `yarn node` command being incorrectly forwarded arguments

Branch: fix-yarn-node

* Uses symlinks instead of a script for bin indirection to allow calling them directly through Node

Branch: bin-symlinks

* Adds a `yarn bin <name>` command that returns the path of the specified bin on the disk

Branch: yarn-bin-name

* Implements an --into option to yarn node/yarn run

Branch: opt-into

* Implements the `--pnp` option

Branch: pnp-option

* Adds new tests for checking that packages are correctly locked

Branch: lock-tests

* Bugfixes

This diff ships with two fixes:

- Fixes calling a pnp script from a non-pnp scripts
- Fixes relative requires from binaries

It also adds tests for all those cases

Branch: bugfixes

* Fixes workspace registration

Branch: workspace-registration-fix

* Uses --enable-pnp (alias --pnp) and --disable-pnp

Branch: instalconfig-pnp

* Adds the issuer into the error messages when requesting a package one shouldn't have access to.

Branch: via-issuer

* Implements extendedQualifiedPathResolution

Branch: extended-qualified-path-resolution

* Changes the return of the pnp daemon to return json data

* Implements custom shebangs for the pnp file

* Changes the return of the pnp-exposed functions to return null with builtins

* Various fixes & improvements

* Adds a test, prettier, fixes a test

* Don't iterate on the registries

* Renames YARN_PLUGNPLAY_EXPERIMENTAL into YARN_PLUGNPLAY_OVERRIDE

* Avoids touching the .pnp.js file when it doesn't need to change

* Reworks the cache path to contain the "node_modules/<pkg-name>" string

* Shims resolve#isCore

* Improves error messages

* Fixes the environment cast to allow passing false/0

* Implements a pnp blacklist settings

* Fixes the fallback resolution to use _resolveFilename instead of _finePath

* Fixes fallback relative path resolution

* Implements require.cache

* Prevents pnp from being enabled on Windows

* Removes absolute paths from the pnp files

* Implements a super basic offline cache integration

* feat(pnp): eject package command (yarnpkg#92)

* test(pnp): support ejecting packages

* feat(pnp): eject package command

* test(pnp): use fs.readdir instead of fs.readdirSync

* test(pnp): do not expect specific error message

* refactor(pnp): move eject logic to package linker

* fix(pnp): change ejected folder to .pnp/ejected/pkgName-pkgVersion/node_modules/pkgName

* fix(pnp): do not re-eject package

* test(pnp): do not rely on installConfig

* Fixes yarn bin

* Preserves the node_modules components in zip paths

* Fixes the offline cache

* Adds a VERSIONS field into the generated resolver

* Exposes the "extensions" option to "resolveRequest"

* Renames yarn eject into yarn unplug

* Tweaks yarn unplug

* Removes packageMainEntry from the package information

* Makes "unplug" print the list of unplugged packages

* Unplugs postinstall packages automatically

* Renames things

* Adds a warning on Windows to notify that PnP settings are ignored at the moment

* Fixes the default shebang

* Exports pnpapi

* Reworks the resolve shim to only affect liftoff

* Guards the pnp file against fs extensions

* Fixes the resolve shim

* Fixes a broken test

* Re-enables the focus tests

* Stops relying on bash for test scripts

* Fixes nohoist

* Revert "Fixes a broken test"

This reverts commit 84358aa.

* Ensures that the getPackageInformation function returns an absolute path
  • Loading branch information
arcanis authored Sep 24, 2018
1 parent 5682d55 commit e905f74
Show file tree
Hide file tree
Showing 109 changed files with 5,030 additions and 501 deletions.
1 change: 1 addition & 0 deletions .babelrc
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@
"stage-0"
],
"plugins": [
["babel-plugin-inline-import", { "extensions": [ ".tpl.js" ] }],
["transform-inline-imports-commonjs"],
["transform-runtime", { "polyfill": false, "regenerator": true }]
]
Expand Down
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -21,5 +21,7 @@ test/fixtures/**/.fbkpm
/__tests__/fixtures/request-cache/GET/localhost/.bin
.idea
.yarn-meta
.pnp.js
.pnp
/packages/lockfile/index.js
.vscode/
2 changes: 1 addition & 1 deletion __tests__/commands/install/integration.js
Original file line number Diff line number Diff line change
Expand Up @@ -236,7 +236,7 @@ test('changes the cache path when bumping the cache version', () =>

await mockConstants(config, {CACHE_VERSION: 42}, async config => {
await cache(config, reporter, {}, ['dir']);
expect((JSON.parse(String(inOut.read())): any).data).toMatch(/[\\\/]v42[\\\/]?$/);
expect((JSON.parse(String(inOut.read())): any).data).toMatch(/[\\\/]v42([\\\/].*)?$/);
});
}));

Expand Down
Binary file not shown.
Binary file not shown.
1 change: 1 addition & 0 deletions __tests__/integration.js
Original file line number Diff line number Diff line change
Expand Up @@ -506,6 +506,7 @@ test('relative cache folder', async () => {

const [stdoutOutput, _] = await runYarn(['cache', 'dir'], {cwd: `${base}/sub`});

// The dirname is to remove the "v2" part
expect(await fs.realpath(path.dirname(stdoutOutput.toString()))).toEqual(await fs.realpath(`${base}/foo`));
});

Expand Down
10 changes: 5 additions & 5 deletions __tests__/lifecycle-scripts.js
Original file line number Diff line number Diff line change
Expand Up @@ -48,31 +48,31 @@ async function execCommand(cmd: string, packageName: string, env = process.env):

test.concurrent('should add the global yarnrc arguments to the command line', async () => {
const stdout = await execCommand('cache dir', 'yarnrc-cli');
expect(stdout.replace(/\\/g, '/')).toMatch(/^(C:)?\/tmp\/foobar\/v[0-9]+\n$/);
expect(stdout.replace(/\\/g, '/')).toMatch(/^(C:)?\/tmp\/foobar\/v[0-9]+(\/.*)?\n$/);
});

test.concurrent(
'should add the command-specific yarnrc arguments to the command line if the command name matches',
async () => {
const stdout = await execCommand('cache dir', 'yarnrc-cli-command-specific-ok');
expect(stdout.replace(/\\/g, '/')).toMatch(/^(C:)?\/tmp\/foobar\/v[0-9]+\n$/);
expect(stdout.replace(/\\/g, '/')).toMatch(/^(C:)?\/tmp\/foobar\/v[0-9]+(\/.*)?\n$/);
},
);

test.concurrent("should not add the command-specific yarnrc arguments if the command name doesn't match", async () => {
const stdout = await execCommand('cache dir', 'yarnrc-cli-command-specific-ko');
expect(stdout.replace(/\\/g, '/')).not.toMatch(/^(C:)?\/tmp\/foobar\/v[0-9]+\n$/);
expect(stdout.replace(/\\/g, '/')).not.toMatch(/^(C:)?\/tmp\/foobar\/v[0-9]+(\/.*)?\n$/);
});

test.concurrent('should allow overriding the yarnrc values from the command line', async () => {
const stdout = await execCommand('cache dir --cache-folder /tmp/toto', 'yarnrc-cli');
expect(stdout.replace(/\\/g, '/')).toMatch(/^(C:)?\/tmp\/toto\/v[0-9]+\n$/);
expect(stdout.replace(/\\/g, '/')).toMatch(/^(C:)?\/tmp\/toto\/v[0-9]+(\/.*)?\n$/);
});

// Test disabled for now, cf rc.js
test.concurrent('should resolve the yarnrc values relative to where the file lives', async () => {
const stdout = await execCommand('cache dir', 'yarnrc-cli-relative');
expect(stdout.replace(/\\/g, '/')).toMatch(/^(C:)?(\/[^\/]+)+\/foobar\/hello\/world\/v[0-9]+\n$/);
expect(stdout.replace(/\\/g, '/')).toMatch(/^(C:)?(\/[^\/]+)+\/foobar\/hello\/world\/v[0-9]+(\/.*)?\n$/);
});

test.concurrent(
Expand Down
6 changes: 3 additions & 3 deletions __tests__/package-resolver.js
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ jasmine.DEFAULT_TIMEOUT_INTERVAL = 90000;

const path = require('path');

// regexp which verifies that cache path contains semver + hash
const cachePathRe = /-\d+\.\d+\.\d+-[\dabcdef]{40}$/;
// regexp which verifies that the cache path contains a path component ending with semver + hash
const cachePathRe = /-\d+\.\d+\.\d+-[\dabcdef]{40}[\\\/]/;

async function createEnv(configOptions): Object {
const lockfile = new Lockfile();
Expand Down Expand Up @@ -82,7 +82,7 @@ addTest(
'@foo/[email protected]',
'npm',
async cacheFolder => {
const folder = path.join(cacheFolder, 'npm-@foo', 'bar');
const folder = path.join(cacheFolder, 'npm-@foo-bar', 'node_modules', '@foo', 'bar');
await fs.mkdirp(folder);
await fs.writeFile(
path.join(folder, constants.METADATA_FILENAME),
Expand Down
1 change: 1 addition & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@
"babel-eslint": "^7.2.3",
"babel-loader": "^6.2.5",
"babel-plugin-array-includes": "^2.0.3",
"babel-plugin-inline-import": "^2.0.6",
"babel-plugin-transform-builtin-extend": "^1.1.2",
"babel-plugin-transform-inline-imports-commonjs": "^1.0.0",
"babel-plugin-transform-runtime": "^6.4.3",
Expand Down
2 changes: 1 addition & 1 deletion packages/pkg-tests/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
"babel-preset-env": "^1.6.1",
"babel-preset-flow": "^6.23.0",
"flow-bin": "^0.66.0",
"jest": "^22.3.0",
"jest": "^23.0.0",
"prettier": "^1.10.2"
},
"scripts": {
Expand Down
8 changes: 6 additions & 2 deletions packages/pkg-tests/pkg-tests-core/sources/utils/fs.js
Original file line number Diff line number Diff line change
Expand Up @@ -174,8 +174,12 @@ exports.readJson = async function readJson(source: string): Promise<any> {
}
};

exports.chmod = function chmod(target: string, mod: number): Promise<void> {
return fs.chmod(target, mod);
exports.chmod = async function chmod(target: string, mod: number): Promise<void> {
await fs.chmod(target, mod);
};

exports.realpath = function realpath(source: string): Promise<string> {
return fs.realpath(source);
};

exports.makeFakeBinary = async function(
Expand Down
45 changes: 37 additions & 8 deletions packages/pkg-tests/pkg-tests-core/sources/utils/tests.js
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,17 @@ export type PackageRunDriver = (

export type PackageDriver = any;

let whitelist = new Map();

exports.setPackageWhitelist = async function whitelistPackages(
packages: Map<string, Set<string>>,
fn: () => Promise<void>,
) {
whitelist = packages;
await fn();
whitelist = new Map();
};

exports.getPackageRegistry = function getPackageRegistry(): Promise<PackageRegistry> {
if (getPackageRegistry.promise) {
return getPackageRegistry.promise;
Expand Down Expand Up @@ -182,7 +193,12 @@ exports.startPackageServer = function startPackageServer(): Promise<string> {
return processError(res, 404, `Package not found: ${name}`);
}
const versions = Array.from(packageEntry.keys());
let versions = Array.from(packageEntry.keys());
const whitelistedVersions = whitelist.get(name);
if (whitelistedVersions) {
versions = versions.filter(version => whitelistedVersions.has(version));
}
const data = JSON.stringify({
name,
Expand Down Expand Up @@ -300,29 +316,42 @@ exports.generatePkgDriver = function generatePkgDriver({runDriver}: {|runDriver:
}

return async function(): Promise<void> {
const path = await fsUtils.createTemporaryFolder();
const path = await fsUtils.realpath(await fsUtils.createTemporaryFolder());

const registryUrl = await exports.startPackageServer();

// Writes a new package.json file into our temporary directory
await fsUtils.writeJson(`${path}/package.json`, await deepResolve(packageJson));

const run = (...args) => {
let callDefinition = {};

if (args.length > 0 && typeof args[args.length - 1] === 'object') {
callDefinition = args.pop();
}

return runDriver(path, args, {
registryUrl,
...definition,
...subDefinition,
...callDefinition,
});
};

const source = async script => {
return JSON.parse((await run('node', '-p', `JSON.stringify(${script})`)).stdout.toString());
return JSON.parse((await run('node', '-p', `JSON.stringify((() => ${script})())`)).stdout.toString());
};

await fn({
path,
run,
source,
});
try {
await fn({
path,
run,
source,
});
} catch (error) {
error.message = `Temporary fixture folder: ${path}\n\n` + error.message;
throw error;
}
};
};

Expand Down
8 changes: 5 additions & 3 deletions packages/pkg-tests/pkg-tests-fixtures/default-index.js
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,9 @@

module.exports = require(`./package.json`);

for (const key of Object.keys(module.exports.dependencies || {})) {
// $FlowFixMe The whole point of this file is to be dynamic
module.exports.dependencies[key] = require(key);
for (const key of [`dependencies`, `devDependencies`, `peerDependencies`]) {
for (const dep of Object.keys(module.exports[key] || {})) {
// $FlowFixMe The whole point of this file is to be dynamic
module.exports[key][dep] = require(dep);
}
}
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
/* @flow */

module.exports = require(`./package.json`);

for (const key of Object.keys(module.exports.dependencies || {})) {
module.exports.dependencies[key] = require(key);
for (const key of [`dependencies`, `devDependencies`, `peerDependencies`]) {
for (const dep of Object.keys(module.exports[key] || {})) {
// $FlowFixMe The whole point of this file is to be dynamic
module.exports[key][dep] = require(dep);
}
}
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
/* @flow */

module.exports = require(`./package.json`);

for (const key of Object.keys(module.exports.dependencies || {})) {
module.exports.dependencies[key] = require(key);
for (const key of [`dependencies`, `devDependencies`, `peerDependencies`]) {
for (const dep of Object.keys(module.exports[key] || {})) {
// $FlowFixMe The whole point of this file is to be dynamic
module.exports[key][dep] = require(dep);
}
}
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
/* @flow */

module.exports = require(`./package.json`);

for (const key of Object.keys(module.exports.dependencies || {})) {
module.exports.dependencies[key] = require(key);
for (const key of [`dependencies`, `devDependencies`, `peerDependencies`]) {
for (const dep of Object.keys(module.exports[key] || {})) {
// $FlowFixMe The whole point of this file is to be dynamic
module.exports[key][dep] = require(dep);
}
}
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
/* @flow */

module.exports = require(`./package.json`);

for (const key of Object.keys(module.exports.dependencies || {})) {
module.exports.dependencies[key] = require(key);
for (const key of [`dependencies`, `devDependencies`, `peerDependencies`]) {
for (const dep of Object.keys(module.exports[key] || {})) {
// $FlowFixMe The whole point of this file is to be dynamic
module.exports[key][dep] = require(dep);
}
}
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
/* @flow */

module.exports = require(`./package.json`);

for (const key of Object.keys(module.exports.dependencies || {})) {
module.exports.dependencies[key] = require(key);
for (const key of [`dependencies`, `devDependencies`, `peerDependencies`]) {
for (const dep of Object.keys(module.exports[key] || {})) {
// $FlowFixMe The whole point of this file is to be dynamic
module.exports[key][dep] = require(dep);
}
}
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
/* @flow */

module.exports = require(`./package.json`);

for (const key of Object.keys(module.exports.dependencies || {})) {
module.exports.dependencies[key] = require(key);
for (const key of [`dependencies`, `devDependencies`, `peerDependencies`]) {
for (const dep of Object.keys(module.exports[key] || {})) {
// $FlowFixMe The whole point of this file is to be dynamic
module.exports[key][dep] = require(dep);
}
}
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
/* @flow */

module.exports = require(`./package.json`);

for (const key of Object.keys(module.exports.dependencies || {})) {
module.exports.dependencies[key] = require(key);
for (const key of [`dependencies`, `devDependencies`, `peerDependencies`]) {
for (const dep of Object.keys(module.exports[key] || {})) {
// $FlowFixMe The whole point of this file is to be dynamic
module.exports[key][dep] = require(dep);
}
}
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
/* @flow */

module.exports = require(`./package.json`);

for (const key of Object.keys(module.exports.dependencies || {})) {
module.exports.dependencies[key] = require(key);
for (const key of [`dependencies`, `devDependencies`, `peerDependencies`]) {
for (const dep of Object.keys(module.exports[key] || {})) {
// $FlowFixMe The whole point of this file is to be dynamic
module.exports[key][dep] = require(dep);
}
}
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
/* @flow */

module.exports = require(`./package.json`);

for (const key of Object.keys(module.exports.dependencies || {})) {
module.exports.dependencies[key] = require(key);
for (const key of [`dependencies`, `devDependencies`, `peerDependencies`]) {
for (const dep of Object.keys(module.exports[key] || {})) {
// $FlowFixMe The whole point of this file is to be dynamic
module.exports[key][dep] = require(dep);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
#!/usr/bin/env node

console.log(process.cwd());
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
#!/usr/bin/env node

const secret = require('./secret');

console.log(secret);
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
#!/usr/bin/env node

const noDeps = require('no-deps');

console.log(noDeps.name);
console.log(noDeps.version);
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
#!/usr/bin/env node

for (let t = 2; t < process.argv.length; ++t) {
console.log(process.argv[t]);
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
/* @flow */

module.exports = require(`./package.json`);

for (const key of [`dependencies`, `devDependencies`, `peerDependencies`]) {
for (const dep of Object.keys(module.exports[key] || {})) {
// $FlowFixMe The whole point of this file is to be dynamic
module.exports[key][dep] = require(dep);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
{
"name": "has-bin-entries",
"version": "1.0.0",
"bin": {
"has-bin-entries": "./bin.js",
"has-bin-entries-with-require": "./bin-with-require.js",
"has-bin-entries-with-relative-require": "./bin-with-relative-require.js",
"has-bin-entries-get-pwd": "./bin-get-pwd.js"
},
"dependencies": {
"no-deps": "1.0.0"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
module.exports = 42;
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
/* @flow */

module.exports = require(`./package.json`);

for (const key of Object.keys(module.exports.dependencies || {})) {
module.exports.dependencies[key] = require(key);
for (const key of [`dependencies`, `devDependencies`, `peerDependencies`]) {
for (const dep of Object.keys(module.exports[key] || {})) {
// $FlowFixMe The whole point of this file is to be dynamic
module.exports[key][dep] = require(dep);
}
}
Loading

0 comments on commit e905f74

Please sign in to comment.