Add / #639 Standalone Built Plugin Tests #671

10upsimon · 2023-03-14T00:07:23Z

Summary

Fixes #639

Relevant technical choices

Introduction of a new `test-plugins` cli command

This CLI command is an extension of the existing ./bin/plugin/cli.js entry point and can be execute in one of two manners;

Via npm run test-plugins for tests in a single site environment
Via npm run test-plugins-multisite for tests in a multisite environment
Via the more vanilla form ./bin/plugin/cli.js test-plugins for single site site testing
1. Via the more vanilla form ./bin/plugin/cli.js test-plugins --sitetype=multi for multisite testing

The former 2 is the preferred means, and the means used by the Github workflow that executes standalone plugin integration testing.

Why a `test-plugins` cli command as opposed to just Github Workflow set of steps?

Authors may want to validate standalone plugins locally following changes to one or more of the modules. A cli command off of the existing ./bin/plugin/cli.js seemed like a logical and accessible choice and flows naturally from the recently introduced build-plugins command.

`test-plugins` logic overview

The logical flow undertaken by the script is as follows:

A check for the presence of a .wp-env.json file is performed as this is critical to the logic that follows. If one is not present, a suitable error is produced and an exit with code 1 is performed to ensure that the test fails.
The presence of the build directory is checked. If no build directory is found, a suitable error is produced and an exit with code 1 is performed to ensure that the test pipeline fails. Users should run npm run build-plugins as a prerequisite to the standalone plugin integration tests. This step is however performed as part of the Github workflow introduced in this PR.
Each plugin in the build dir is then looped over and the steps that follow are performed.
Test assets from the plugin-tests folder are copied to each standalone plugin directory. These include a suitable composer.json file to install necessary testing dependencies, a tests folder containing a single test suite used for all standalone plugins and phpunit.xml.dist used for all standalone plugins, that defines the test(s) to execute per plugin. All plugins perform the same tests, so updating the tests within the plugin-tests/tests folder will have bearing on all built plugins.
composer install --no-interaction is performed against each plugin directory within the build directory. This is handled by setting the --working-dir argument of the command to reference the suitable plugin within the loop.
The .wp-env.json file within plugin-test is update and its plugins property update to an array reflective of the plugins inside of the build directory. It is then copied to the root directory. This ensures that when we start the wp-env environment, all built plugins are already active. This array of plugins within the .wp-env.json file does not include the top level performance plugin.
The wp-env environment is started. This command is executed synchronously.
npm run pretest-php is executed in order for the applicable packages at a top level to be installed.
Plugins are again iterated and test are executed against each via the command wp-env run phpunit 'phpunit -c /var/www/html/wp-content/plugins/${ plugin }/phpunit.xml.dist --verbose --testdox' where ${plugin} is the applicable plugin in the loop. This command is executed synchronously.
If a test fails, i.e the above command returned exit status 1, the process is halted and an overall exit with code 1 is returned, forcing a fail on the test pipeline.
This process continues for all plugins, until an error/failure (if any) is encountered.
If no errors/failures are encountered, the wp-env environment is stopped and an exit with code 0 is performed. This signals a successful execution of the test cycle. At this stage, the integration tests for all plugins would have passed OK.

Why does the `test-plugins` command handle `wp-env start`?

Because we need to first ensure that the.wp-env.json file contains valid plugin pointers to the plugins within the build directory or else there would be no way to activate them as the environment would be unaware of their existence. Further to this, at time of authoring this code it is not possible to run the standalone plugins alongside the performance plugin.

What if there are I/O operation errors?

All I/O operations are executed synchronously and with suitable exception catching. Exits with code 1 are performed, therefore producing a suitable error message, and failing the test pipeline.

Can I test plugin manually outside of the Github Workflow?

Yes, you're encouraged to. As long as you've run npm run build-plugins first, running npm run test-plugins should result in a successful execution of the integration test suite for all standalone plugins in build. HOWEVER, be aware that this command does change the plugins property of the .wp-env.json file to reference the built plugins, and removes the reference to the top level performance plugin.

For example, your .wp-env.json files plugins property may change from something like this:

"plugins": [ "." ],

To this:

"plugins": [ "./build/webp-uploads" ],

Simply reinstate it to [ "." ] and re-run wp-env start in order to resume a WordPress environment with just the performance plugin active.

Introduction of a new `Standalone Plugin Integration Testing / PHP` Github workflow

NOTE: This workflow currently runs against push and pull actions agains feature, release and trunk branches. Once the process of the checks are confirmed via this PR, the workflow will be suitably updated to be more concise and specific.

This workflow executes the following operations:

Ensures the presence of a suitable Ubuntu based PHP container.
Sets up Node.js
Caches npm
Runs npm ci agains the root package.json file
Executes the npm run build-plugins command to generate standalone plugins, as these are not part of the repository source
Executes npm run test-plugins following a successful step above, and continues to run integration tests for all built plugins as described further above, in a single site environment.
1. Executes npm run test-plugins-multisite following a successful step above, and continues to run integration tests for all built plugins as described further above, in a multisite environment.

Checklist

PR has either [Focus] or Infrastructure label.
PR has a [Type] label.
PR has a milestone or the no milestone label.

…uild prior to unit testing of each.

…entrypoint file

…dPress plugin assets

…s fail accordingly

…gnal success to test pipeline, although should not be too necessary it is a safety net

…peline failure

…idate test pipeline failure

mukeshpanchal27

Thanks @10upsimon, Great start. Left some nit-pick feedback.

bin/plugin/commands/test-plugins.js

plugin-tests/phpunit.xml.dist

plugin-tests/tests/test-standalone.php

.gitattributes

mukeshpanchal27 · 2023-03-14T11:31:02Z

.github/workflows/php-test-standalone-plugins.yml

@@ -0,0 +1,58 @@
+name: Standalone Plugin Integration Testing
+
+on:


Does this workflow only work when the plugins.json file is updated?

@mukeshpanchal27 for now, it runs at the same time php unit tests run, but this will change once I have approval on the process itself. Thank you!

joemcgill · 2023-03-15T01:03:15Z

Thanks @10upsimon for the thorough writeup of the approach you've chosen to take here. There's a lot to dig into, so before getting too deep into details, I'd like to share some initial observations and questions.

Overall, it seems like we're adding a lot of complexity in the setup to account for what is a limited problem—testing the build of individual plugins while we're in the in-between state of having these modules available as both part of the Performance Lab plugin AND as individual plugins. Currently, the unit tests don't appear to be running full test suites, just ensuring that there are no fatals, but we're taking a long path to get to that result. I'm curious if we could simplify this quite a bit if we chose to focus on simply ensuring that no fatals are produced when we load a site with all plugins (including the performance lab plugin) activated at the same time?
It looks like this approach is spinning up separate wp-env environments for each built plugin. Is that correct? If so, that seems fairly heavy handed to me. Assuming we eventually need to run separate unit test suites for all built plugins, could we modify the approach so that all of the PHPUnit tests are run in a single container that loads the plugin code in a single bootstrap process, rather than as individual independent tests?
If we could run tests on all of these plugins from the same container, would there still be a need to copy a composer.json to each plugin folder?

10upsimon · 2023-03-15T11:31:02Z

@joemcgill many thanks for your early review feedback, to address your points:

Overall, it seems like we're adding a lot of complexity in the setup to account for what is a limited problem—testing the build of individual plugins while we're in the in-between state of having these modules available as both part of the Performance Lab plugin AND as individual plugins. Currently, the unit tests don't appear to be running full test suites, just ensuring that there are no fatals, but we're taking a long path to get to that result. I'm curious if we could simplify this quite a bit if we chose to focus on simply ensuring that no fatals are produced when we load a site with all plugins (including the performance lab plugin) activated at the same time?

I believe we should be running tests in 2x scenarios:

Testing standalone plugins without the PL plugin active, this is what we are accounting for in this iteration of work
Testing standalone plugins with the PL plugin active, which was not possible prior, but is now that Ensure module load.php files really only load other code to prevent conflicts in standalone plugins #674

To be clear, simply ensuring that no fatals are produced when we load a site with all plugins is exactly what we are doing here. I believe the perceived complexity stems from the fact that because we are not enabling the PL plugin, there are some prerequisites needing to be performed to successfully run tests, just like we do for the PL plugin. In summary, the actual steps taken to arrive at the tests is quite minimal and only involves copying a small handful of assets and executing the tests.

It looks like this approach is spinning up separate wp-env environments for each built plugin. Is that correct? If so, that seems fairly heavy handed to me. Assuming we eventually need to run separate unit test suites for all built plugins, could we modify the approach so that all of the PHPUnit tests are run in a single container that loads the plugin code in a single bootstrap process, rather than as individual independent tests?

This is not the case. This approach builds a plugin string of all plugins that reside in the build directory, applies that to the existing .wp-env.json file, and starts the (single) WordPress environment that references n plugins from the build directory as active plugins.

Regarding running separate unit tests for all plugins, this is currently handled by the PL plugin which runs tests against each module. Will it not always be the case that we'll maintain the PL plugin and handle standalone plugins as builds therefrom, i.e via npm run build-plugins? If so, we should leave the module based tests as they are, as there is no need to cover those again in standalone plugin tests. If we need to adjust for that in future, we can. If we are not going to be building standalone plugins from this "monorepo" using npm run build-plugins indefinitely, then I'd argue that the tests can be adapted for the standalone plugin repo's when that time comes.

If we could run tests on all of these plugins from the same container, would there still be a need to copy a composer.json to each plugin folder?

I could not get it to work correctly without having the PL plugin active as well (which already sets up the prerequisites for tests) and I think forcing the presence of the PL plugin is the incorrect path to take. That being said, I could attempt to simply run composer install --no-interaction on the top level composer file and that might actually solve for what you're suggesting, I'll give that a whirl today.

Thanks for your thoughtful feedback @joemcgill , much appreciated!

joemcgill · 2023-03-15T22:56:09Z

Thanks for the update, @10upsimon, I had definitely misunderstood part of how this was working, in terms of spinning up separate environments for each plugin. I've done some local testing and things are much more clear. I'll consolidate any additional feedback in a proper code review soon.

felixarntz

@10upsimon This looks great to me! Three nit-picks below, I only have one broader point of feedback for consideration: I think the approach of reading .wp-env.json and replacing the plugins entry in it is a creating solution, however it may be simpler to create a static .wp-env.json for this purpose in another location (e.g. in plugin-tests/.wp-env.json) with the plugins already in it, since the list of plugins we already know from the beginning (see the plugins.json file), so I don't think we technically have to "detect" them from the build directory. Your approach here would allow to e.g. only manually build one plugin and then the test code would only test that one - which may be useful, but at the same time I wonder whether that'll ever be used, given that we would always want to have standalone plugins available for all plugins in the plugins.json configuration.

I'm curious what you think about it and what your rationale for the approach was. Definitely not saying that we have to change it, I'm just pointing out an alternative solution that we can consider.

felixarntz · 2023-03-16T00:17:27Z

.gitattributes

@@ -30,3 +30,5 @@

 /.gitattributes export-ignore
 /.gitignore export-ignore
+
+/plugin-tests export-ignore


Nit-pick, maybe put this exclude together with the other directories that contain tooling code (lines 8-10 above).

felixarntz · 2023-03-16T00:25:17Z

bin/plugin/commands/test-plugins.js

+	// Buffer .wp-env.json content var.
+	let wpEnvFileContent = '';
+
+	try {
+		wpEnvFileContent = fs.readFileSync( wpEnvFile, 'utf-8' );
+	} catch ( e ) {
+		log( formats.error( `Error reading file "${ wpEnvFile }": "${ e }"` ) );
+
+		// Return with exit code 1 to trigger a failure in the test pipeline.
+		process.exit( 1 );
+	}
+
+	// If the contents of the file were incorrectly read or exception was not captured and value is blank, abort.
+	if ( '' === wpEnvFileContent ) {
+		log(
+			formats.error(
+				`File content for "${ wpEnvFile }" is empty, aborting.`
+			)
+		);
+
+		// Return with exit code 1 to trigger a failure in the test pipeline.
+		process.exit( 1 );
+	}
+
+	// If we do not have a match on the wp-env enabled plugins regex, abort.
+	if ( ! wpEnvPluginsRegex.test( wpEnvFileContent ) ) {
+		log(
+			formats.error(
+				`Unable to find plugins property/key in WP Env config file: "${ wpEnvFile }". Please ensure that it is present and try agagin.`
+			)
+		);
+
+		// Return with exit code 1 to trigger a failure in the test pipeline.
+		process.exit( 1 );
+	}
+
+	// Let the user know we're re-writing the .wp-env.json file.
+	log( formats.success( `Rewriting plugins property in ${ wpEnvFile }` ) );
+
+	// Attempt replacement of the plugins property in .wp-env.json file to match built plugins.
+	try {
+		// Create plugins property from built plugins.
+		wpEnvPluginsRegexReplacement = `"plugins": [ "${ builtPlugins
+			.map( ( item ) => `${ builtPluginsDir }${ item }` )
+			.join( '", "' ) }" ],`;
+
+		fs.writeFileSync(
+			wpEnvFile,
+			wpEnvFileContent.replace(
+				wpEnvPluginsRegex,
+				wpEnvPluginsRegexReplacement
+			)
+		);
+	} catch ( e ) {
+		log(
+			formats.error(
+				`Error replacing content in ${ wpEnvFile } using regex "${ wpEnvPluginsRegex }": "${ e }"`
+			)
+		);
+
+		// Return with exit code 1 to trigger a failure in the test pipeline.
+		process.exit( 1 );
+	}


Maybe move this code into its own function? The handler function is extremely long, and I think especially this here is pretty much one logical step that could be broken apart into its own function unit.

felixarntz · 2023-03-16T00:25:52Z

bin/plugin/commands/test-plugins.js

+	} );
+
+	// Run tests per plugin.
+	log( formats.success( `About to execute standalone plugins tests.` ) );


This output is probably unnecessary :)

…vel one

mukeshpanchal27

Thanks @10upsimon, I left some findings and feedback.

bin/plugin/cli.js

mukeshpanchal27 · 2023-03-21T06:12:50Z

bin/plugin/commands/test-plugins.js

+ * External dependencies
+ */
+const fs = require( 'fs-extra' );
+const { execSync, spawnSync } = require( 'node:child_process' );


Suggested change

const { execSync, spawnSync } = require( 'node:child_process' );

const { execSync, spawnSync } = require( 'child_process' );

When i run command in local environment it show me Error: Cannot find module 'node:child_process' error. After applying the changes it working for me.

mukeshpanchal27 · 2023-03-21T06:13:42Z

bin/plugin/commands/test-plugins.js

+		process.exit( 1 );
+	}
+
+	// Only try to read plugin dirs if build ir exists.


Suggested change

// Only try to read plugin dirs if build ir exists.

// Only try to read plugin dirs if build is exists.

mukeshpanchal27 · 2023-03-21T06:14:45Z

bin/plugin/commands/test-plugins.js

+
+		// Copy over test files.
+		try {
+			fs.copySync( pluginTestAssets, `./build/${ plugin }/`, {


Suggested change

fs.copySync( pluginTestAssets, `./build/${ plugin }/`, {

fs.copySync( pluginTestAssets, `${ builtPluginsDir }${ plugin }/`, {

Use already define variable.

joemcgill

A couple small suggestions, but after reading all this again, I think the approach makes sense and is looking good. One other thing I'd like to confirm is whether running these tests during a deployment workflow (see: #638) has the potential of polluting the build artifacts that would be uploaded to the SVN repo, given that we're copying certain files into the plugin folder in order to run these tests.

joemcgill · 2023-03-22T19:32:46Z

bin/plugin/commands/test-plugins.js

+	// Attempt replacement of the plugins property in .wp-env.json file to match built plugins.
+	try {
+		// Create plugins property from built plugins.
+		wpEnvPluginsRegexReplacement = `"plugins": [ "${ builtPlugins
+			.map( ( item ) => `${ builtPluginsDir }${ item }` )
+			.join( '", "' ) }" ],`;
+
+		fs.writeFileSync(
+			wpEnvFile,
+			wpEnvFileContent.replace(
+				wpEnvPluginsRegex,
+				wpEnvPluginsRegexReplacement
+			)
+		);
+	} catch ( e ) {
+		log(
+			formats.error(
+				`Error replacing content in ${ wpEnvFile } using regex "${ wpEnvPluginsRegex }": "${ e }"`
+			)
+		);


Rather than overwriting the wp-env.json file, we can create a new wp-env.override.json file, which we could also optionally add to the .gitignore so anyone running this locally won't accidentally have changes in their working branch.

See: https://developer.wordpress.org/block-editor/reference-guides/packages/packages-env/#wp-env-override-json

Thanks @joemcgill , taking a look at this as it seems a necessary change to me 👍

@joemcgill this has been implemented in a recent commit.

joemcgill · 2023-03-22T19:56:02Z

plugin-tests/tests/bootstrap.php

+$GLOBALS['wp_tests_options'] = array(
+	'active_plugins' => array( basename( TESTS_PLUGIN_DIR ) . '/load.php' ),
+);


Do we also need to optionally load the main Plugin Lab plugin as well, in order to ensure there are no fatals due to both being loaded at the same time, or is that out of scope for this specific issue?

@joemcgill I figured this was out of the scope of this task/issue, but given that we now have a working ability to run them side by side, maybe we should add this test case?

Yes I think that it's a good idea that we have the ability to do both, but let's handle that in a separate issue so it doesn't block progress here.

@joemcgill this has been handled in a recent commit.

joemcgill · 2023-03-22T20:01:24Z

plugin-tests/composer.json

+  "require-dev": {
+    "dealerdirect/phpcodesniffer-composer-installer": "^0.7",
+    "phpcompatibility/php-compatibility": "^9.3",
+    "phpunit/phpunit": "^4|^5|^6|^7|^8|^9",


Fun fact: PHPUnit doesn't need to be required if we're using yoast/phpunit-polyfills, because it requires PHPUnit as a dependency itself and handles proper versioning for you.

@joemcgill have removed, thanks. We should probably consider this rule for the main plugin, as that is where I borrowed the file from.

Agreed! It's some thing that we could check for in all of our plug-ins that use the polyfills library

…Press/performance into feature/creating-standalone-plugins

…alone-built-plugin-tests

…d excluded from version control

10upsimon · 2023-03-23T22:55:16Z

@joemcgill @felixarntz @mukeshpanchal27 this is ready for a final review I believe. I've requested a re-review from all 3 of you. Notable changes:

I've split the logic into smaller functions and lightened the handler substantially. I decided specifically on splitting the logic of replacing/building the wp-env.override.json file and running the unit tests to ease the process of redoing the full test cycle with the WPP plugin active as well.
Carrying on from the above, I've now added a second round of testing with the WPP plugin active as well. This is done within the same cli cycle, there is no separate workflow for it.
Multisite tests have been added via a new --sitetype flag, it defaults to 'single' but can accept 'multi'. I've made this more accessible via a new npm command npm run test-plugins-multisite
Logic has been altered to obtain the built plugins list from the plugins.json file as opposed to directory traversing. I've still added a check against each plugin to ensure that the directory exists though. If not, it is removed (via .filter()) from the array of plugins and a suitable warning is issued in the cli output. NOTE: I have NOT failed the test pipeline in this case.
Various minor changes relating to prior review comments

The only outstanding item is a full and proper implementation of the GitHub workflow rules for the changed files. If we're happy with the rest of the code, I can work on that as a final deliverable.

Thank you.

.github/workflows/php-test-standalone-plugins.yml

joemcgill

Thanks Simon! Looks like it's working now. Left a couple additional comments, but nothing blocking.

joemcgill · 2023-03-27T19:03:12Z

bin/plugin/commands/test-plugins.js

+	// Create an array of plugins from entries in plugins JSON file.
+	builtPlugins = Object.keys( pluginsJsonFileContentAsJson )
+		.filter( ( item ) => {
+			if ( ! fs.pathExistsSync( `${ settings.builtPluginsDir }${ pluginsJsonFileContentAsJson[ item ].slug }` ) ) {


Do we need to add any handling here in case the builtPluginsDir value does not end in a slash?

@joemcgill I do not think this is absolutely necessary, as this is not a dynamically defined value, we hard define it as part of the settings object we pass to doRunStandalonePluginTests(). We can potentially handle this in a future PR if the need arises?

joemcgill · 2023-03-27T19:15:02Z

bin/plugin/commands/test-plugins.js

+		process.exit( 1 );
+	}
+
+	// Copy the newly modified ./plugins-tests/.wp-env.json file to the root level.


Why not write directly to this file rather than writing to the original and then copying?

@joemcgill we have to copy it regardless the first time, but I agree that a check for file existing and then a copy over and final write is probably best, as if it's existing already, we may as well write to it. To be clear, it won't exist already as we need to create this overrides file, so I am not actually sure how much of a difference this makes, but I am going to effect this in case this os ever enhanced to handle multiple writes (which it is currently but you hav further feedback in this review to change that).

joemcgill · 2023-03-27T19:35:08Z

bin/plugin/commands/test-plugins.js

+			`Re-running standalone plugin integration tests with WPP plugin active.`
+		)
+	);
+	// Add the root level WPP plugin to the built plugins array.


This works, but we could probably speed this up a bit by doing the first run with the WPP plugin active and then using a WP CLI command to disable it for the second run. That way we don't have to worry about restarting the environment.

Great suggestion, I am in the process of changing this.

@joemcgill this has been refactored in the latest commit at the time of writing, please see my general update comment further down.

… and activate plugins ability as part of running tests to optimize the overal process

10upsimon · 2023-03-31T12:01:25Z

@joemcgill @mukeshpanchal27 I've pushed a round of changes, all of which can be seen in the commit at f485ccf

I've refactored a few areas, but high level takeaways are:

I now only copy the wp-env overrides file once, and I copy it and then write to it as per your suggestion @joemcgill
Added a variable to act as a state tracking agains whether we've started wp-env already, so that it is not unnecessary re-started
Moved the wp-env stop command out of the testing function, as that is called multiple times. wp-env is now simply stopped as the final action of the handler logic
Added two new settings objects for passing into doRunUnitTests() function: disablePlugins and enablePlugins. Instead of stopping and starting the env with different plugins in the plugins JSON object, we now start them with all plugins including the main WPP one, and we disable and enable it as needed via these new settings objects
Use of wp-cli commands to deactivate and activate plugins (in our case only the main WPP one) to speed up the testing process. This has greatly sped up the total time to test

joemcgill

Nice updates @10upsimon , this is looking good, and I can confirm that everything is working as expected locally. The only thing that I noticed while testing this command locally that does not exceed affect our GitHub workflow is that the performancePluginSlug value may be incorrect if somebody has cloned this repo to a folder named something other than "performance". Unless that's an easy fix here, I think we could dress that has an enhancement later.

felixarntz

@10upsimon Code-wise this looks pretty much good to go, however I found two things we should address, both related to running locally. We need to also make sure this works smoothly locally, not only in the CI workflow.

Nit-pick, one thing in your PR description is probably out of date now? See this:

HOWEVER, be aware that this command does change the plugins property of the .wp-env.json file to reference the built plugins, and removes the reference to the top level performance plugin.

I believe this is no longer accurate, maybe from a previous iteration of the PR? We can probably remove it now that the code here uses the .wp-env.override.json` file.

bin/plugin/commands/test-plugins.js

…nv overrides file as part of cleanup operations.

10upsimon · 2023-03-31T19:59:52Z

@joemcgill @felixarntz thanks for the review, I've addressed final pieces of feedback in e36d516

performancePluginSlug is now dynamic and based off of the base directory in which the npm run test-plugins command is run from, which is the root level folder in which the repo was cloned to.
As part of cleanup operations, we are now removing the .wp-env.override.json file from the base/root directory.

bin/plugin/commands/test-plugins.js

…be executed immediately after spinning up wp-env

joemcgill

Approving again. This is working great locally now too. 🎉

felixarntz

Thank you @10upsimon for the last updates, great work!

10upsimon added 5 commits March 14, 2023 01:58

Track plugin test assets to be copied to each built plugin within ./b…

a453083

…uild prior to unit testing of each.

Added test-plugins command logic

135fdb5

Added test-plugins command reference and necessary imports to cli.js …

2a16c5a

…entrypoint file

Added test-plugins command to package.json

f4f9733

Set plugin-tests to be export ignored so as not to be part of any Wor…

828265b

…dPress plugin assets

10upsimon added [Type] Enhancement A suggestion for improvement of an existing feature Infrastructure Issues for the overall performance plugin infrastructure no milestone PRs that do not have a defined milestone for release Creating standalone plugins labels Mar 14, 2023

10upsimon added 8 commits March 14, 2023 02:09

Removed unnneded bin folder from within plugin-tests folder

387a8af

Fixed incorrect variable reference within catch condition

2b73a24

Better logging and error handling

e065b71

Added exit status 1 for all failures so that test pipelines/worklflow…

c525470

…s fail accordingly

Returned with exit code 0 at the end of the plugin test process to si…

0257c5e

…gnal success to test pipeline, although should not be too necessary it is a safety net

Added standalone plugin integration testing GitHub workflow

cb2e4f5

Forcing failed standalone plugin integration test to validate test pi…

3f5f992

…peline failure

Removing forced failure for standalone plugin integration test to val…

9ca5131

…idate test pipeline failure

mukeshpanchal27 reviewed Mar 14, 2023

View reviewed changes

Code and comment formatting following initial draft PR review

1ab29d3

10upsimon marked this pull request as ready for review March 14, 2023 13:20

10upsimon requested review from JustinyAhin and felixarntz as code owners March 14, 2023 13:20

10upsimon requested a review from mukeshpanchal27 March 14, 2023 13:20

felixarntz reviewed Mar 16, 2023

View reviewed changes

Add base .wp-env.json file to plugin-tests as opposed to using top le…

b6a3f5b

…vel one

mukeshpanchal27 reviewed Mar 21, 2023

View reviewed changes

joemcgill reviewed Mar 22, 2023

View reviewed changes

10upsimon added 3 commits March 24, 2023 00:28

Merge branch 'feature/creating-standalone-plugins' of github.com:Word…

3dd9a4c

…Press/performance into feature/creating-standalone-plugins

Merge branch 'feature/creating-standalone-plugins' into add/639-stand…

fc9403d

…alone-built-plugin-tests

Remove incorrectly added wp-env json file in server-timing folder

95fed30

10upsimon requested review from joemcgill, felixarntz and mukeshpanchal27 March 23, 2023 22:34

Altered logic so that copied wp-env json file is an overrides one, an…

0efd1e6

…d excluded from version control

10upsimon self-assigned this Mar 23, 2023

10upsimon added the Needs Review label Mar 23, 2023

10upsimon linked an issue Mar 23, 2023 that may be closed by this pull request

Implement GitHub workflow to have a basic test for standalone plugins #639

Closed

mukeshpanchal27 reviewed Mar 24, 2023

View reviewed changes

.github/workflows/php-test-standalone-plugins.yml Outdated Show resolved Hide resolved

Changes to GH workflow file for standalone plugin integration tests

fb93494

10upsimon requested a review from mukeshpanchal27 March 24, 2023 10:01

joemcgill reviewed Mar 27, 2023

View reviewed changes

Tidy up test run logic to not unnecessarily restart wp-env | Deactive…

f485ccf

… and activate plugins ability as part of running tests to optimize the overal process

joemcgill approved these changes Mar 31, 2023

View reviewed changes

felixarntz requested changes Mar 31, 2023

View reviewed changes

bin/plugin/commands/test-plugins.js Outdated Show resolved Hide resolved

bin/plugin/commands/test-plugins.js Show resolved Hide resolved

Made main wpp plugin slug dynamic based on process cwd | Deleted wp-e…

e36d516

…nv overrides file as part of cleanup operations.

joemcgill reviewed Mar 31, 2023

View reviewed changes

bin/plugin/commands/test-plugins.js Outdated Show resolved Hide resolved

Fixed typo and moved operation that deletes wp-env overrides file to …

555c071

…be executed immediately after spinning up wp-env

10upsimon requested a review from felixarntz March 31, 2023 20:18

joemcgill approved these changes Mar 31, 2023

View reviewed changes

felixarntz approved these changes Mar 31, 2023

View reviewed changes

felixarntz merged commit 8dce595 into feature/creating-standalone-plugins Mar 31, 2023

felixarntz deleted the add/639-standalone-built-plugin-tests branch March 31, 2023 20:36

mukeshpanchal27 mentioned this pull request Apr 3, 2023

Implement GitHub workflow to have a basic test for standalone plugins #639

Closed

mukeshpanchal27 mentioned this pull request Mar 25, 2024

Remove plugin-tests folder #1087

Merged

		@@ -0,0 +1,58 @@
		name: Standalone Plugin Integration Testing

		on:

	const { execSync, spawnSync } = require( 'node:child_process' );
	const { execSync, spawnSync } = require( 'child_process' );

	// Only try to read plugin dirs if build ir exists.
	// Only try to read plugin dirs if build is exists.

	fs.copySync( pluginTestAssets, `./build/${ plugin }/`, {
	fs.copySync( pluginTestAssets, `${ builtPluginsDir }${ plugin }/`, {

Add / #639 Standalone Built Plugin Tests #671

Add / #639 Standalone Built Plugin Tests #671

Conversation

10upsimon commented Mar 14, 2023 • edited Loading

Summary

Relevant technical choices

Introduction of a new test-plugins cli command

Why a test-plugins cli command as opposed to just Github Workflow set of steps?

test-plugins logic overview

Why does the test-plugins command handle wp-env start?

What if there are I/O operation errors?

Can I test plugin manually outside of the Github Workflow?

Introduction of a new Standalone Plugin Integration Testing / PHP Github workflow

Checklist

mukeshpanchal27 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joemcgill commented Mar 15, 2023

10upsimon commented Mar 15, 2023

joemcgill commented Mar 15, 2023

felixarntz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mukeshpanchal27 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joemcgill left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

10upsimon commented Mar 23, 2023

joemcgill left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

10upsimon Mar 31, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

10upsimon commented Mar 31, 2023

joemcgill left a comment

Choose a reason for hiding this comment

felixarntz left a comment

Choose a reason for hiding this comment

10upsimon commented Mar 31, 2023

joemcgill left a comment

Choose a reason for hiding this comment

felixarntz left a comment

Choose a reason for hiding this comment

10upsimon commented Mar 14, 2023 •

edited

Loading

Introduction of a new `test-plugins` cli command

Why a `test-plugins` cli command as opposed to just Github Workflow set of steps?

`test-plugins` logic overview

Why does the `test-plugins` command handle `wp-env start`?

Introduction of a new `Standalone Plugin Integration Testing / PHP` Github workflow

10upsimon Mar 31, 2023 •

edited

Loading