Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add pooling and request limiting to prevent memory leaks #110

Closed
Closed
Show file tree
Hide file tree
Changes from 47 commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
6c12e78
Add pooling and request limiting to prevent memory leaks
seanmorris Sep 14, 2023
e968403
Adding Pool class.
seanmorris Sep 14, 2023
b515f13
Reverting change.
seanmorris Sep 14, 2023
5c7b52d
Removing console.log
seanmorris Sep 14, 2023
a33b0c5
Max request options.
seanmorris Sep 14, 2023
f8326cd
Tweaks.
seanmorris Sep 14, 2023
cb383fe
Tweaks.
seanmorris Sep 14, 2023
ac4ea1b
Initialize all instances
seanmorris Sep 14, 2023
0b9044f
Tweak.
seanmorris Sep 14, 2023
fc0f995
Tweak.
seanmorris Sep 14, 2023
e18b1fd
Consolidating FINALLY block.
seanmorris Sep 15, 2023
0ffaad7
Cleaner pooling algorithm.
seanmorris Sep 15, 2023
3d26067
Comments
seanmorris Sep 15, 2023
909d08a
Comments.
seanmorris Sep 15, 2023
1c91f34
Comments.
seanmorris Sep 15, 2023
a86c4c9
Catch spawn failures
seanmorris Sep 15, 2023
4b634bb
Making private methods privater.
seanmorris Sep 15, 2023
3f7b796
Tweaks.
seanmorris Sep 15, 2023
db3c5d6
Tweaks.
seanmorris Sep 15, 2023
68f6290
Correcting CLI switches, killing instances when done.
seanmorris Sep 16, 2023
9167e77
Detect error in test.
seanmorris Sep 16, 2023
70c3a22
Correcting multi-await
seanmorris Sep 19, 2023
aebc3d5
Correcting multi-await
seanmorris Sep 19, 2023
96a2e82
Scaling back default maxRequests
seanmorris Sep 27, 2023
0cb98b7
Pull request tweaks.
seanmorris Oct 3, 2023
a73874f
Doc Comments.
seanmorris Oct 5, 2023
326b410
Doc Comments.
seanmorris Oct 5, 2023
966a9df
Testing tweaks.
seanmorris Oct 5, 2023
488ad3c
Testing tweaks.
seanmorris Oct 5, 2023
a7708af
Testing tweaks.
seanmorris Oct 5, 2023
e01d6d4
All tests pass.
seanmorris Oct 5, 2023
eca5995
Passing linter
seanmorris Oct 5, 2023
3d8a1c2
Passing linter
seanmorris Oct 5, 2023
271c935
Passing linter
seanmorris Oct 5, 2023
a7581dc
Temporarily skip typecheck
seanmorris Oct 5, 2023
8d968d7
Revert temp change.
seanmorris Oct 5, 2023
04a3e5e
Send 500 error to browser instead of re-queueing request.
seanmorris Oct 6, 2023
65c28e2
Correcting default fatal handler
seanmorris Oct 6, 2023
a1b809e
Abstracting node pools
seanmorris Oct 6, 2023
102adbc
Tweaks.
seanmorris Oct 6, 2023
954d33f
Tweaks
seanmorris Oct 6, 2023
6bc105d
Tweaks
seanmorris Oct 6, 2023
b3c6a08
Tweaks
seanmorris Oct 6, 2023
be0c0cb
Merge branch 'trunk' into sm-preventing-memory-leaks
seanmorris Nov 22, 2023
fabcafa
Bumping versions.
seanmorris Dec 4, 2023
56484b5
Non-working copy operation
seanmorris Dec 4, 2023
c59bea8
Revering extra change.
seanmorris Dec 6, 2023
c133a1e
PR comments.
seanmorris Dec 7, 2023
a657040
PR comments.
seanmorris Dec 8, 2023
75bd3eb
Separating debug flags into its own PR
seanmorris Dec 8, 2023
c3b2269
Incrementing verion numbers
seanmorris Dec 21, 2023
1f9f696
Formatting.
seanmorris Dec 21, 2023
f352bb8
Lint & Tests.
seanmorris Dec 21, 2023
0e6c8ba
Merge conflict.
seanmorris Dec 27, 2023
0ad0b9d
PR comment tweaks.
seanmorris Dec 27, 2023
5d18b81
Restoring codemirror dependency
seanmorris Dec 27, 2023
b09b13b
Revering extra change
seanmorris Dec 27, 2023
0bd1222
PR comment tweaks.
seanmorris Dec 27, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 49 additions & 25 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 5 additions & 5 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -30,11 +30,11 @@
"@codemirror/state": "6.2.0",
"@codemirror/theme-one-dark": "6.1.1",
"@codemirror/view": "6.9.3",
"@php-wasm/node": "0.1.56",
"@php-wasm/progress": "0.1.56",
"@php-wasm/universal": "0.1.56",
"@php-wasm/web": "0.1.57",
"@wp-playground/blueprints": "0.1.56",
"@php-wasm/node": "0.3.1",
"@php-wasm/progress": "0.3.1",
"@php-wasm/universal": "0.3.1",
"@php-wasm/web": "0.3.1",
"@wp-playground/blueprints": "0.3.1",
"@wp-playground/client": "0.2.0",
"classnames": "^2.3.2",
"comlink": "^4.4.1",
Expand Down
2 changes: 1 addition & 1 deletion packages/interactive-code-block/project.json
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
"cwd": "dist/packages",
"commands": [
"cp ../../packages/interactive-code-block/README.md ./interactive-code-block/",
"rm interactive-code-block/assets/php_5*",
"rm -f interactive-code-block/assets/php_5*",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why only add -f in this line of code but not the other ones? And why update the interactive-code-block?

Copy link
Contributor Author

@seanmorris seanmorris Dec 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was getting an error during the build process due to the attempted removal of a non-existent file. -f prevents rm from throwing errors when you try to remove a file that isn't there. I've reverted it locally and the issue doesn't seem to be happening anymore. I only applied this to php5 because I've seen that we're deprecating that.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aha! PHP is not in the Playground repo anymore so that entire rm line could be removed in a separate PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@adamziel See, that worries me. If someone still has that directory, and then moves to a version that doesn't have an rm call for it, then it won't be removed. I think we should definitely open the PR now, BUT I think we should hold off on actually merging it for a predetermined amount of time, so that this can run off. Let me know your thoughts.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How would someone still have that directory after git pull and npm install?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@adamziel If someone pulled the package freshly then that directory wouldn't exist. But if someone were already working with it, and had it in a state where that directory exists, and then checked out a version where the rm call is gone, then the directory might persist unexpectedly.

Copy link
Collaborator

@adamziel adamziel Dec 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The way I understand git is that checking out a newer revision where a file is removed does remove that file from the filesystem. Am I getting that wrong?

"rm interactive-code-block/assets/php_7_0*",
"rm interactive-code-block/assets/php_7_1*",
"rm interactive-code-block/assets/php_7_2*",
Expand Down
5 changes: 5 additions & 0 deletions packages/nx-extensions/src/executors/built-script/executor.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,11 @@ const dirname = __dirname;

export default async function runExecutor(options: BuiltScriptExecutorSchema) {
const args = [
...(options['inspect'] ? ['--inspect-brk'] : []),
adamziel marked this conversation as resolved.
Show resolved Hide resolved
...(options['inspect-brk'] ? ['--inspect-brk'] : []),
...(options['trace-exit'] ? ['--trace-exit'] : []),
...(options['trace-uncaught'] ? ['--trace-uncaught'] : []),
...(options['trace-warnings'] ? ['--trace-warnings'] : []),
'--loader',
join(dirname, 'loader.mjs'),
options.scriptPath,
Expand Down
5 changes: 5 additions & 0 deletions packages/nx-extensions/src/executors/built-script/schema.d.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
export interface BuiltScriptExecutorSchema {
scriptPath: string;
inspect: boolean;
'inspect-brk': boolean;
'trace-exit': boolean;
'trace-uncaught': boolean;
'trace-warnings': boolean;
__unparsed__: string;
} // eslint-disable-line
25 changes: 25 additions & 0 deletions packages/nx-extensions/src/executors/built-script/schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,31 @@
"description": "Path of the script to run.",
"x-prompt": "What script would you like to run?"
},
"inspect": {
"type": "boolean",
"description": "Use Node debugging client.",
"x-prompt": "Would you like to connect a Node debugging client?"
},
"inspect-brk": {
"type": "boolean",
"description": "Use Node debugging client. Break immediately on script execution start.",
"x-prompt": "Would you like to connect a Node debugging client and break immediately on script execution start?"
},
"trace-exit": {
"type": "boolean",
"description": "Prints a stack trace whenever an environment is exited proactively, i.e. invoking process.exit().",
"x-prompt": "Would you like print a stacktrace on exit?"
},
"trace-uncaught": {
"type": "boolean",
"description": "Print stack traces for uncaught exceptions; usually, the stack trace associated with the creation of an Error is printed, whereas this makes Node.js also print the stack trace associated with throwing the value (which does not need to be an Error instance).",
"x-prompt": "Would you like print a stacktrace on uncaught promise rejection?"
},
"trace-warnings": {
"type": "boolean",
"description": "Print stack traces for process warnings (including deprecations).",
"x-prompt": "Would you like print a stacktrace on warning?"
},
"__unparsed__": {
"hidden": true,
"type": "array",
Expand Down
2 changes: 1 addition & 1 deletion packages/vscode-extension/project.json
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@
"cp ./LICENSE dist/packages/vscode-extension",
"cp packages/vscode-extension/package.json dist/packages/vscode-extension",
"cp packages/vscode-extension/README.md dist/packages/vscode-extension",
"cp node_modules/@php-wasm/node/*.wasm dist/packages/vscode-extension"
"find node_modules/@php-wasm/node/ -type f -name php_*.wasm | while read FILE; do cp $FILE dist/packages/vscode-extension; done"
adamziel marked this conversation as resolved.
Show resolved Hide resolved
],
"parallel": false
}
Expand Down
31 changes: 15 additions & 16 deletions packages/wp-now/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,24 +6,23 @@ It uses automatic mode detection to provide a fast setup process, regardless of

![Demo GIF of wp-now](https://github.com/WordPress/wordpress-playground/assets/36432/474ecb85-d789-4db9-b820-a19c25da79ad)


## Table of Contents

- [Requirements](#requirements)
- [Usage](#usage)
- [Automatic Modes](#automatic-modes)
- [Arguments](#arguments)
- [Technical Details](#technical-details)
- [Using Blueprints](#using-blueprints)
- [Defining Custom URLs](#defining-custom-urls)
- [Defining Debugging Constants](#defining-debugging-constants)
- [Known Issues](#known-issues)
- [Comparisons](#comparisons)
- [Laravel Valet](#laravel-valet)
- [wp-env](#wp-env)
- [Contributing](#contributing)
- [Testing](#testing)
- [Publishing](#publishing)
- [Requirements](#requirements)
- [Usage](#usage)
- [Automatic Modes](#automatic-modes)
- [Arguments](#arguments)
- [Technical Details](#technical-details)
- [Using Blueprints](#using-blueprints)
- [Defining Custom URLs](#defining-custom-urls)
- [Defining Debugging Constants](#defining-debugging-constants)
- [Known Issues](#known-issues)
- [Comparisons](#comparisons)
- [Laravel Valet](#laravel-valet)
- [wp-env](#wp-env)
- [Contributing](#contributing)
- [Testing](#testing)
- [Publishing](#publishing)

## Requirements

Expand Down
8 changes: 8 additions & 0 deletions packages/wp-now/src/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ export interface CliOptions {
port?: number;
blueprint?: string;
reset?: boolean;
maxRequests?: number;
maxJobs?: number;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know it's late in the process but now as this is basically ready I'm questioning how clear these names are. we're not limiting the total number of requests, and what is a job?

what if we either renamed them or created them as sub-options like poolOptions?

something like maxRequestsPerWorkerLifespan is verbose but does far more to communicate the role of the config, and maxPhpWorkers (or even the already existing numberOfPhpInstances) skips the term for one that's already in scope when creating one of these config values.

}

export const enum WPNowMode {
Expand All @@ -43,6 +45,8 @@ export interface WPNowOptions {
wpContentPath?: string;
wordPressVersion?: string;
numberOfPhpInstances?: number;
maxRequests?: number;
maxJobs?: number;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's the interplay here between maxJobs and numberOfPhpInstances?
we didn't recreate the same config value did we?

could we use numberOfPhpInstances to set maxJobs or are these somehow indicating different things: number of workers in a pool vs. number of pools? would it make sense to only use one and then to leave the existing config for WPNowOptions?

blueprintObject?: Blueprint;
reset?: boolean;
}
Expand All @@ -54,6 +58,8 @@ export const DEFAULT_OPTIONS: WPNowOptions = {
projectPath: process.cwd(),
mode: WPNowMode.AUTO,
numberOfPhpInstances: 1,
maxRequests: 128,
maxJobs: 1,
reset: false,
};

Expand Down Expand Up @@ -111,6 +117,8 @@ export default async function getWpNowConfig(
phpVersion: args.php as SupportedPHPVersion,
projectPath: args.path as string,
wordPressVersion: args.wp as string,
maxRequests: args.maxRequests as number,
adamziel marked this conversation as resolved.
Show resolved Hide resolved
maxJobs: args.maxJobs as number,
port,
reset: args.reset as boolean,
};
Expand Down
44 changes: 44 additions & 0 deletions packages/wp-now/src/node-pool.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
import { PHPResponse } from '@php-wasm/universal';
import { NodePHP } from '@php-wasm/node';
import { Pool } from './pool';

export type nodePoolOptions = {
spawn?: () => Promise<NodePHP>;
reap?: (instance: NodePHP) => void;
fatal?: (instance: NodePHP, error: any) => any;
maxRequests?: number;
maxJobs?: number;
};

const defaultSpawn = async () => await NodePHP.load('8.2');
adamziel marked this conversation as resolved.
Show resolved Hide resolved

const defaultFatal = (instance: NodePHP, error: any) =>
new PHPResponse(
500,
{},
new TextEncoder().encode(
`500 Internal Server Error:\n\n${String(
error && error.stack ? error.stack : error
)}`
)
);

const defaultReap = (instance: NodePHP) => {
try {
instance.exit();

Check failure on line 28 in packages/wp-now/src/node-pool.ts

View workflow job for this annotation

GitHub Actions / Lint and typecheck

Property 'exit' does not exist on type 'NodePHP'.
} catch {
void 0;
adamziel marked this conversation as resolved.
Show resolved Hide resolved
}
};

export class NodePool extends Pool {
constructor({
maxRequests = 128,
maxJobs = 1,
spawn = defaultSpawn,
fatal = defaultFatal,
reap = defaultReap,
} = {}) {
super({ maxRequests, maxJobs, spawn, fatal, reap });
}
}
Loading
Loading