Skip to content

Commit

Permalink
Merge branch 'master' into serial-prettier
Browse files Browse the repository at this point in the history
  • Loading branch information
72636c authored Oct 12, 2021
2 parents 6217cb8 + fa7ed19 commit 757c4d3
Show file tree
Hide file tree
Showing 10 changed files with 76 additions and 32 deletions.
6 changes: 3 additions & 3 deletions .changeset/changelog.js
Original file line number Diff line number Diff line change
Expand Up @@ -27,11 +27,11 @@ const defaultChangelogFunctions = {
(changeset) => `- Updated dependencies [${changeset.commit}]`,
);

const updatedDepenenciesList = dependenciesUpdated.map(
const updatedDependenciesList = dependenciesUpdated.map(
(dependency) => ` - ${dependency.name}@${dependency.newVersion}`,
);

return [...changesetLinks, ...updatedDepenenciesList].join("\n");
return [...changesetLinks, ...updatedDependenciesList].join("\n");
},
getReleaseLine: async (changeset) => {
const [firstLine, ...futureLines] = changeset.summary
Expand Down Expand Up @@ -155,7 +155,7 @@ const gitHubChangelogFunctions = {

const suffix = links.pull ?? links.commit;

return `\n\n- ${formattedFirstLine}${
return `\n- ${formattedFirstLine}${
suffix ? ` (${suffix})` : ""
}\n${futureLines.map((l) => ` ${l}`).join("\n")}`;
},
Expand Down
7 changes: 7 additions & 0 deletions .changeset/metal-rings-listen.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
"skuba": patch
---

lint: Use worker threads when running `--serial`ly

This aims to reduce the memory footprint of `skuba lint --serial`. ESLint and Prettier are now run in worker threads so their memory can be more readily freed on thread exit.
26 changes: 13 additions & 13 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,15 +8,15 @@

This avoids rewriting sequences like "distroless" as "libroless".

* **template:** Remove `unknown` specifier in catch clauses ([#580](https://github.com/seek-oss/skuba/pull/580))
- **template:** Remove `unknown` specifier in catch clauses ([#580](https://github.com/seek-oss/skuba/pull/580))

Strict TypeScript 4.4 now defaults to typing catch clause variables as `unknown`.

- **build-package, lint:** Handle worker thread errors more gracefully ([#583](https://github.com/seek-oss/skuba/pull/583))

The worker threads now correctly propagate an exit code and log errors instead of triggering an `UnhandledPromiseRejectionWarning`.

* **format, lint:** Limit Prettier to 25 parallel file I/O operations ([#584](https://github.com/seek-oss/skuba/pull/584))
- **format, lint:** Limit Prettier to 25 parallel file I/O operations ([#584](https://github.com/seek-oss/skuba/pull/584))

This should alleviate file descriptor issues that are not handled by `graceful-fs` such as `EBADF: bad file description, close`.

Expand All @@ -28,7 +28,7 @@

This reduces the number of Node.js processes spawned by `skuba lint`. We've also been able to significantly enhance our logging output as a result, particularly when the `--debug` flag is supplied.

* **build-package, lint:** Add `--serial` flag ([#556](https://github.com/seek-oss/skuba/pull/556))
- **build-package, lint:** Add `--serial` flag ([#556](https://github.com/seek-oss/skuba/pull/556))

This explicitly disables concurrent command execution.

Expand All @@ -48,13 +48,13 @@

This avoids creating a separate Node.js process just to run the REPL.

* **Buildkite.annotate:** Add development API for writing annotations ([#558](https://github.com/seek-oss/skuba/pull/558))
- **Buildkite.annotate:** Add development API for writing annotations ([#558](https://github.com/seek-oss/skuba/pull/558))

- **format:** Execute ESLint with `--report-unused-disable-directives` ([#512](https://github.com/seek-oss/skuba/pull/512))

`skuba format` will now flag unused disable directives, and will [automatically remove](https://eslint.org/blog/2021/06/whats-coming-in-eslint-8.0.0#unused-disable-directives-are-now-fixable) them once ESLint v8 is released.

* **deps:** Prettier 2.4 ([#507](https://github.com/seek-oss/skuba/pull/507))
- **deps:** Prettier 2.4 ([#507](https://github.com/seek-oss/skuba/pull/507))

This includes TypeScript 4.4 support. See the [release notes](https://prettier.io/blog/2021/09/09/2.4.0.html) for more information.

Expand All @@ -64,15 +64,15 @@

Note that new syntax in TypeScript 4.4 will only be supported by `skuba format` and `skuba lint` once ESLint v8 is released.

* **format:** Run ESLint and Prettier in process ([#539](https://github.com/seek-oss/skuba/pull/539))
- **format:** Run ESLint and Prettier in process ([#539](https://github.com/seek-oss/skuba/pull/539))

This eliminates the overhead of spinning up separate Node.js processes. We've also been able to significantly enhance our logging output as a result, particularly when the `--debug` flag is supplied.

- **build:** Remove experimental Babel support ([#513](https://github.com/seek-oss/skuba/pull/513))

There's limited upside to switching to [Babel-based builds](https://github.com/seek-oss/skuba/tree/master/docs/deep-dives/babel.md) for backend use cases, and it would be difficult to guarantee backwards compatibility with existing `tsconfig.json`-based configuration. Dropping Babel dependencies reduces our package size and resolves [SNYK-JS-SETVALUE-1540541](https://app.snyk.io/vuln/SNYK-JS-SETVALUE-1540541).

* **lint:** Support Buildkite annotations ([#558](https://github.com/seek-oss/skuba/pull/558))
- **lint:** Support Buildkite annotations ([#558](https://github.com/seek-oss/skuba/pull/558))

`skuba lint` can now output issues as Buildkite annotations.

Expand All @@ -82,33 +82,33 @@

- **template:** pino-pretty ^7.0.0 ([#506](https://github.com/seek-oss/skuba/pull/506))

* **template:** Configure environment variables and volume mounts for Buildkite annotations ([#558](https://github.com/seek-oss/skuba/pull/558))
- **template:** Configure environment variables and volume mounts for Buildkite annotations ([#558](https://github.com/seek-oss/skuba/pull/558))

- **template:** serverless-plugin-canary-deployments ^0.7.0 ([#508](https://github.com/seek-oss/skuba/pull/508))

* **template/lambda-sqs-worker\*:** Prime dev ECR cache in Buildkite pipeline ([#503](https://github.com/seek-oss/skuba/pull/503))
- **template/lambda-sqs-worker\*:** Prime dev ECR cache in Buildkite pipeline ([#503](https://github.com/seek-oss/skuba/pull/503))

This should result in faster "Deploy Dev" times as the ECR cache will already be warm.

- **template:** seek-jobs/gantry v1.4.1 ([#504](https://github.com/seek-oss/skuba/pull/504))

* **template:** Remove `@types/node` resolution override ([#498](https://github.com/seek-oss/skuba/pull/498))
- **template:** Remove `@types/node` resolution override ([#498](https://github.com/seek-oss/skuba/pull/498))

Jest 27.1 is compatible with newer versions of `@types/node`.

- **template/\*-rest-api:** Suggest using a secure header middleware ([#579](https://github.com/seek-oss/skuba/pull/579))

* **template/lambda-sqs-worker-cdk:** Run "Test, Lint & Build" step in prod ([#503](https://github.com/seek-oss/skuba/pull/503))
- **template/lambda-sqs-worker-cdk:** Run "Test, Lint & Build" step in prod ([#503](https://github.com/seek-oss/skuba/pull/503))

This reduces our dependence on a dev environment to successfully deploy to prod.

- **build-package, lint:** Simplify logging prefix ([#535](https://github.com/seek-oss/skuba/pull/535))

* **Jest.mergePreset:** Allow `watchPathIgnorePatterns` ([#555](https://github.com/seek-oss/skuba/pull/555))
- **Jest.mergePreset:** Allow `watchPathIgnorePatterns` ([#555](https://github.com/seek-oss/skuba/pull/555))

- **build-package, lint:** Limit max concurrency to CPU core count ([#540](https://github.com/seek-oss/skuba/pull/540))

* **template:** Remove Yarn cache from worker Docker images ([#499](https://github.com/seek-oss/skuba/pull/499))
- **template:** Remove Yarn cache from worker Docker images ([#499](https://github.com/seek-oss/skuba/pull/499))

This shrinks the cached Docker images that our worker templates generate.

Expand Down
4 changes: 2 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -41,15 +41,15 @@
"devDependencies": {
"@changesets/cli": "2.17.0",
"@changesets/get-github-info": "0.5.0",
"@types/concurrently": "6.2.1",
"@types/concurrently": "6.3.0",
"@types/ejs": "3.1.0",
"@types/express": "4.17.13",
"@types/fs-extra": "9.0.13",
"@types/koa": "2.13.4",
"@types/lodash.mergewith": "4.6.6",
"@types/module-alias": "2.0.1",
"@types/npm-which": "3.0.1",
"@types/picomatch": "2.2.6",
"@types/picomatch": "2.3.0",
"@types/supertest": "2.0.11",
"express": "4.17.1",
"koa": "2.13.3",
Expand Down
4 changes: 2 additions & 2 deletions src/cli/lint.int.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -106,8 +106,8 @@ test.each`
await prepareTempDirectory(baseDir, tempDir);

await expect(
// Force `--serial` as worker threads can't run in a TypeScript context.
lint([...args, '--serial'], tscOutputStream),
// Disable worker threads as they can't run in a TypeScript context.
lint(args, tscOutputStream, false),
).resolves.toBeUndefined();

const tempDirRegex = new RegExp(tempDir, 'g');
Expand Down
29 changes: 26 additions & 3 deletions src/cli/lint/external.ts
Original file line number Diff line number Diff line change
Expand Up @@ -39,19 +39,42 @@ const lintConcurrently = async ({ tscOutputStream, ...input }: Input) => {
return { eslint, prettier, tscOk };
};

const lintSerially = async (input: Input) => {
/**
* Run linting tools `--serial`ly for resource-constrained environments.
*
* Note that we still run ESLint and Prettier in worker threads as a
* counterintuitive optimisation. Memory can be more readily freed on worker
* thread exit, which isn't as easy with a monolithic main thread.
*/
const lintSerially = async ({ tscOutputStream, ...input }: Input) => {
const eslint = await runESLintInWorkerThread(input);
const prettier = await runPrettierInWorkerThread(input);
const tscOk = await runTscInNewProcess({ ...input, tscOutputStream });

return { eslint, prettier, tscOk };
};

const lintSeriallyWithoutWorkerThreads = async (input: Input) => {
const eslint = await runESLintInCurrentThread(input);
const prettier = await runPrettierInCurrentThread(input);
const tscOk = await runTscInNewProcess(input);

return { eslint, prettier, tscOk };
};

export const externalLint = async (input: Input) => {
const selectLintFunction = (input: Input) => {
if (!input.workerThreads) {
return lintSeriallyWithoutWorkerThreads;
}

// `--debug` implies `--serial`.
const isSerial = input.debug || input.serial;

const lint = isSerial ? lintSerially : lintConcurrently;
return isSerial ? lintSerially : lintConcurrently;
};

export const externalLint = async (input: Input) => {
const lint = selectLintFunction(input);

const tscOutputStream = new StreamInterceptor();
tscOutputStream.pipe(input.tscOutputStream ?? process.stdout);
Expand Down
2 changes: 2 additions & 0 deletions src/cli/lint/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,13 @@ import type { Input } from './types';
export const lint = async (
args = process.argv,
tscOutputStream: NodeJS.WritableStream | undefined = undefined,
workerThreads = true,
) => {
const opts: Input = {
debug: hasDebugFlag(args),
serial: hasSerialFlag(args),
tscOutputStream,
workerThreads,
};

await externalLint(opts);
Expand Down
10 changes: 10 additions & 0 deletions src/cli/lint/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,4 +22,14 @@ export interface Input {
* Defaults to `process.stdout`.
*/
tscOutputStream?: NodeJS.WritableStream;

/**
* Whether to allow usage of Node.js worker threads.
*
* This may be set to `false` when there is a worker thread incompatibility,
* such as calling in from a TypeScript context in our Jest tests.
*
* Defaults to `true`.
*/
workerThreads?: boolean;
}
3 changes: 2 additions & 1 deletion src/utils/exec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ import { cpus } from 'os';
import stream from 'stream';
import util from 'util';

import type { Color } from 'chalk';
import concurrently from 'concurrently';
import execa, { ExecaChildProcess } from 'execa';
import npmRunPath from 'npm-run-path';
Expand Down Expand Up @@ -62,7 +63,7 @@ export type Exec = (
interface ExecConcurrentlyCommand {
command: string;
name: string;
prefixColor?: string;
prefixColor?: typeof Color;
}

interface ExecConcurrentlyOptions {
Expand Down
17 changes: 9 additions & 8 deletions yarn.lock
Original file line number Diff line number Diff line change
Expand Up @@ -1196,12 +1196,13 @@
"@types/connect" "*"
"@types/node" "*"

"@types/concurrently@6.2.1":
version "6.2.1"
resolved "https://registry.yarnpkg.com/@types/concurrently/-/concurrently-6.2.1.tgz#48540055add9d76c55d444aaaf6f4c964c405452"
integrity sha512-l+AH7GuHYSgIYoXAufh7P9/oYzRobM+m863XobAARzFiafcGdCjMQQ1grLbVc69+AJTBbO3S8mtt2UPsipk6WA==
"@types/concurrently@6.3.0":
version "6.3.0"
resolved "https://registry.yarnpkg.com/@types/concurrently/-/concurrently-6.3.0.tgz#ef6956d0e120f78d1da93d4435da6fe872256d04"
integrity sha512-R0M9Dt330vtNgNlBYQYniDfI3I3Sk6cu27M1EKS5VFoio2rZqTMuOBGe9DX+ISM0B6y+RGrgtpWCv2k2VBLtbQ==
dependencies:
"@types/node" "*"
chalk "^4.1.0"

"@types/connect@*":
version "3.4.35"
Expand Down Expand Up @@ -1408,10 +1409,10 @@
resolved "https://registry.yarnpkg.com/@types/parse-json/-/parse-json-4.0.0.tgz#2f8bb441434d163b35fb8ffdccd7138927ffb8c0"
integrity sha512-//oorEZjL6sbPcKUaCdIGlIUeH26mgzimjBB77G6XRgnDl/L5wOnpyBGRe/Mmf5CVW3PwEBE1NjiMZ/ssFh4wA==

"@types/picomatch@2.2.6":
version "2.2.6"
resolved "https://registry.yarnpkg.com/@types/picomatch/-/picomatch-2.2.6.tgz#9f9f460d50e7f2f63f08fa8751aafe6bfd554eba"
integrity sha512-24Y8b5ENcUDAtmcvVOIzdCwQVuXDjGTSQJl6Y0RTkct4ivJZursOSRL1SVv14Tvtql1arS68LUpspbw3HD74DA==
"@types/picomatch@2.3.0":
version "2.3.0"
resolved "https://registry.yarnpkg.com/@types/picomatch/-/picomatch-2.3.0.tgz#75db5e75a713c5a83d5b76780c3da84a82806003"
integrity sha512-O397rnSS9iQI4OirieAtsDqvCj4+3eY1J+EPdNTKuHuRWIfUoGyzX294o8C4KJYaLqgSrd2o60c5EqCU8Zv02g==

"@types/prettier@^2.1.5":
version "2.4.1"
Expand Down

0 comments on commit 757c4d3

Please sign in to comment.