Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FIX] improve tarball insertion #6808

Merged
merged 5 commits into from
Nov 26, 2019
Merged

[FIX] improve tarball insertion #6808

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
9133c58
improve packaging
runspired Nov 26, 2019
bea016c
add log to figure out CI and update ilios
runspired Nov 26, 2019
67dc107
make extra sure we aren't trolling ourselves
runspired Nov 26, 2019
77e97f5
cleanup
runspired Nov 26, 2019
0952f7c
improve read-file
runspired Nov 26, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 9 additions & 3 deletions bin/-tarball-info.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,13 @@ function generatePackageReference(version, tarballName) {
}

function insertTarballsToPackageJson(fileLocation, options = {}) {
const pkgInfo = require(fileLocation);
// in some flows we have the potential to have previously written
// to the package.json already prior to calling this method.
// reading it in this way this ensures we get the latest and not
// a stale module from require
const location = require.resolve(fileLocation);
const pkgInfo = JSON.parse(fs.readFileSync(location, 'utf8'));

if (options.isRelativeTarball) {
pkgInfo.version = `${pkgInfo.version}-sha.${CurrentSha}`;
}
Expand All @@ -182,11 +188,11 @@ function insertTarballsToPackageJson(fileLocation, options = {}) {

if (!options.isRelativeTarball) {
const resolutions = (pkgInfo.resolutions = pkgInfo.resolutions || {});
resolutions[packageName] = pkg.tarballLocation;
resolutions[packageName] = pkg.reference;
}
});

fs.writeFileSync(fileLocation, JSON.stringify(pkgInfo, null, 2));
fs.writeFileSync(location, JSON.stringify(pkgInfo, null, 2));
}

module.exports = {
Expand Down
84 changes: 52 additions & 32 deletions bin/test-external-partner-project.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,26 @@ const execa = require('execa');
const debug = require('debug')('test-external');
const rimraf = require('rimraf');
const chalk = require('chalk');

const cliArgs = require('command-line-args');
const projectRoot = path.resolve(__dirname, '../');
const externalProjectName = process.argv[2];
const gitUrl = process.argv[3];
const skipSmokeTest =
(process.argv[4] && process.argv[4] === '--skip-smoke-test') ||
(process.argv[5] && process.argv[5] === '--skip-smoke-test');
const skipClone =
(process.argv[4] && process.argv[4] === '--skip-clone') || (process.argv[5] && process.argv[5] === '--skip-clone');

let cliOptionsDef = [{ name: 'projectName', defaultOption: true }];
let cliOptions = cliArgs(cliOptionsDef, { stopAtFirstUnknown: true });
const externalProjectName = cliOptions.projectName;
let argv = cliOptions._unknown || [];
cliOptionsDef = [{ name: 'gitUrl', defaultOption: true }];
cliOptions = cliArgs(cliOptionsDef, { stopAtFirstUnknown: true, argv });
const gitUrl = cliOptions.gitUrl;
argv = cliOptions._unknown || [];
cliOptionsDef = [
{ name: 'skipSmokeTest', type: Boolean, defaultValue: false },
{ name: 'skipClone', type: Boolean, defaultValue: false },
{ name: 'skipTest', type: Boolean, defaultValue: false },
];
cliOptions = cliArgs(cliOptionsDef, { argv });

const { skipSmokeTest, skipClone, skipTest } = cliOptions;

// we share this for the build
const cachePath = '../__external-test-cache';
const tempDir = path.join(projectRoot, cachePath);
Expand Down Expand Up @@ -109,34 +120,43 @@ try {
try {
debug('Preparing Package To Run Tests Against Commit');
insertTarballsToPackageJson(packageJsonLocation);
// we must use npm because yarn fails to install
// the nested tarballs correctly (just as it fails to pack them correctly)
// we rimraf node_modules first because it was previously
// installed using yarn's resolution algorithm
if (!skipSmokeTest) {
execExternal(`rm -rf node_modules`);
}
execExternal(`npm install`);

// clear node_modules installed for the smoke-test
execExternal(`rm -rf node_modules`);
// we are forced to use yarn so that our resolutions will be respected
// in addition to the version file link we insert otherwise nested deps
// may bring their own ember-data
//
// For this reason we don't trust the lock file
// we also can't trust the cache
execExternal(`yarn install --no-lockfile --cache-folder=tmp/yarn-cache`);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW, npm and yarn aren't drop in replacements for each other. There are quite a few bizarre inconsistencies (e.g. around using file: or link: references for deps/devDeps).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before we were using npm regardless of whether the partner used yarn or npm because yarn was failing to extract the tarballs. This appears to have been fixed! However, I discovered a bug with npm wherein both during link and when installing from tarball nested dependencies specifying ember-data wont resolve to our tarball even if it is "in range" (I think this may be due to it being an alpha).

So we can't use npm safely, so using yarn+resolutions (to force the nested deps) always seems like the way to go.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW, tarballs can never be "in range". They are always treated as "exotic" and the system cannot make assumptions about greater than / less than.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rwjblue all the more reason to not install with npm unless they’ve added support for resolutions that I don’t know about (based on testing they don’t seem to respect them)

} catch (e) {
debug(e);
throw new Error(`Unable to npm install tarballs for ember-data\` for ${externalProjectName}`);
console.log(`Unable to npm install tarballs for ember-data\` for ${externalProjectName}. Original error below:`);

throw e;
}

try {
debug('Running tests against EmberData commit');
execExternal(`ember test`, true);
} catch (e) {
commitTestPassed = false;
if (!skipTest) {
try {
debug('Running tests against EmberData commit');
execExternal(`ember test`, true);
} catch (e) {
commitTestPassed = false;
}
}

if (skipSmokeTest && !commitTestPassed) {
throw new Error('Commit may result in a regression, but the smoke test was skipped.');
} else if (!smokeTestPassed && !commitTestPassed) {
throw new Error(`Commit may result in a regression, but the smoke test for ${externalProjectName} also failed.`);
} else if (smokeTestPassed && !commitTestPassed) {
throw new Error(`Commit results in a regression in ${externalProjectName}`);
} else if (!smokeTestPassed) {
console.log(`Commit may resolve issues present in the smoke test for ${externalProjectName}`);
if (skipTest) {
console.log(`Skipped Tests: Commit viability unknown`);
} else {
console.log(`Commit does not regress ${externalProjectName}`);
if (skipSmokeTest && !commitTestPassed) {
throw new Error('Commit may result in a regression, but the smoke test was skipped.');
} else if (!smokeTestPassed && !commitTestPassed) {
throw new Error(`Commit may result in a regression, but the smoke test for ${externalProjectName} also failed.`);
} else if (smokeTestPassed && !commitTestPassed) {
throw new Error(`Commit results in a regression in ${externalProjectName}`);
} else if (!smokeTestPassed) {
console.log(`Commit may resolve issues present in the smoke test for ${externalProjectName}`);
} else {
console.log(`Commit does not regress ${externalProjectName}`);
}
}
2 changes: 1 addition & 1 deletion package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@
"test-external:travis-web": "./bin/test-external-partner-project.js travis-web https://github.com/travis-ci/travis-web.git",
"test-external:storefront": "./bin/test-external-partner-project.js storefront https://github.com/embermap/ember-data-storefront.git",
"test-external:factory-guy": "./bin/test-external-partner-project.js factory-guy https://github.com/danielspaniel/ember-data-factory-guy.git",
"test-external:ilios-frontend": "./bin/test-external-partner-project.js ilios-frontend https://github.com/ilios/frontend.git --skip-smoke-test",
"test-external:ilios-frontend": "./bin/test-external-partner-project.js ilios-frontend https://github.com/ilios/frontend.git --skipSmokeTest",
"test-external:ember-resource-metadata": "./bin/test-external-partner-project.js ember-resource-metadata https://github.com/ef4/ember-resource-metadata.git",
"test-external:ember-data-relationship-tracker": "./bin/test-external-partner-project.js ember-data-relationship-tracker https://github.com/ef4/ember-data-relationship-tracker.git"
},
Expand Down