Env: Download WordPress PHPUnit Into Container #41780

ObliviousHarmony · 2022-06-17T09:38:50Z

What?

This pull request downloads the WordPress PHPUnit test files into the wordpress and tests-wordpress container. This closes #23171.

Why?

Currently, the best option we have for installing the WordPress PHPUnit test library is using wp-phpunit/wp-phpunit. This is not a bad solution, however, it comes with a few problems:

If you select a version of WordPress that is incompatible with the installed version of wp-phpunit/wp-phpunit your tests will fail.
There is no option to download the current trunk test files. This can lead to problems when trying to test against the next version of WordPress.

Since we know what version of WordPress the environment contains, a better solution would be to just download it and mount it in the container.

How?

This pull request works by scanning ~/.wp-env/{hash}/{coreSource}/wp-includes/version.php for the current WordPress version. An alternative that was considered was using wp core version, but, I wanted to find a solution that could give us the version to include the library in the docker-compose file.

We can then use this information to figure out what version of the WordPress PHPUnit library to install. We store it in the ~/.wp-env/{hash}/ directory and mount it at /phpunit in the container. To get around the need for anything other than git, we download from WordPress/wordpress-develop. It mirrors from SVN so we can rely on master being the bleeding edge and the tags containing WordPress versions. We're also using a sparse checkout to prevent having the entire WordPress/wordpress-develop repository checked out.

Lastly, in order to use it in unit tests, we copy the wp-config.php file and set the required constants. Once we create the containers with the WP_PHPUNIT__DIR we can now run unit tests without wp-phpunit/wp-phpunit

Note: I looked at removing wp-phpunit/wp-phpunit but it felt out of scope for this PR. It requires changes to the test suite to properly use Yoast/PHPUnit-Polyfills.

Testing Instructions

This is tricky to test, unfortunately. We're going to have to remove wp-phpunit/wp-phpunit. The tests won't pass unless you do the legwork, but, just removing it can let you see that the test suite boots up. That is really all you need to know this in particular works.

Run composer remove wp-phpunit/wp-phpunit --dev to get rid of wp-phpunit/wp-phpunit. In theory, the PHPUnit tests should now not even be able to boot.
Run npm run wp-env run tests-wordpress "/var/www/html/wp-content/plugins/gutenberg/vendor/bin/phpunit -c /var/www/html/wp-content/plugins/gutenberg/phpunit.xml.dist --verbose"
Note that the PHPUnit suite boots up and loads WordPress but fails because of wp-phpunit/wp-phpunit failures.
Poke around ~/.wp-env/{hash} and check out the WordPress-PHPUnit and tests-WordPress-PHPUnit directories. They should contain all of the test files with a sparse checkout.
Use the WP_ENV_CORE environment variable to try out a few different options for WordPress installs. A local one, a zip, a git repository, etc. Verify that the correct version of the PHPUnit library is installed in each case.

I have prepared #41852 with all of the commits from this pull request. It makes the appropriate changes for the Gutenberg PHPUnit tests to be ran in wp-env. The PHPUnit test workflow in that pull request uses the provided test files. That pull request passing is a demonstration of this working in CI.

ObliviousHarmony · 2022-06-18T01:02:06Z

One potential optimization for the git checkout could be to add a remote to an empty repository instead of cloning. This would mean nothing is downloaded that we don't want. I think we could fetch the specific tag or branch and then do the sparse checkout. I'm actually not completely sure if the sparse checkout saves us anything as-is, but it felt nice to empty the directory haha.

With that said, as far as I can tell, performance isn't really an issue. It's just a thought though. I think the depth of 1 already aggressively optimizes what we're cloning in the first place, but, I can take another pass at the download next week if we want to try and bleed whatever performance out that we can.

ObliviousHarmony · 2022-06-18T04:54:59Z

I went ahead with the performance optimization anyway. It should now only be downloading the latest files for the test library. It's a bit faster and the fact that the tests and development files are downloaded in parallel helps too!

ObliviousHarmony · 2022-06-18T07:34:37Z

As I was working on this I noticed that the test config doesn't have a different table prefix. I wasn't sure if it mattered at the time, but now, it feels like it should. Is there a good reason we shouldn't change the prefix so that unit tests don't truncate all of the tables?

This commit adds support for downloading the WordPress PHPUnit source and mounting it at /phpunit in the container. It uses the version of WordPress installed in the container to decide what version to download.

This commit minimizes the amount of data that needs to be downloaded when we pull in the test framework. It does this by not cloning the entire repository and only checking out once we've set up the sparse checkout.

This commit moves the test library from `/phpunit' to `/wordpress-phpunit`. This change is to avoid potential confusion. `/phpunit` sounds like a tool while `/wordpress-phpunit` is not so ambiguous.

ObliviousHarmony · 2022-06-22T00:35:28Z

I've opened a PR making the appropriate changes for this to be used in Gutenberg. That pull request includes the commits from this one, so, it should also be a demonstration of the functionality here.

ndiego · 2022-07-12T16:29:25Z

@ObliviousHarmony and @noahtallen the linked issue and this PR came up in today's Editor Bug Scrub. Just checking in to see what the status of this PR is and what needs to be done to get it across the line.

noahtallen

Nice, overall I think this is good to go! I think we can also remove the line which creates the old wp-phpunit-config.php file, as the new file does the same thing.

On this branch, after removing wp-phpunit from composer, npm run test-unit-php passes. As a result, it's using these changes successfully. This also works after removing the file I mentioned above.

So this change should be backwards compatible, which is a nice benefit :)

noahtallen · 2022-07-12T21:32:59Z

packages/env/test/build-docker-compose-config.js

 		expect( devVolumes ).toEqual( expect.arrayContaining( localSources ) );
+
+		localSources = [


Nit: I'd just define this as a separate variable.

noahtallen · 2022-07-12T21:36:12Z

packages/env/lib/commands/start.js

+			),
+			readWordPressVersion( config.env.tests.coreSource, spinner, debug ),
+		] );
+		await downloadWPPHPUnit(


Do we need to run this for every wp-env install? I wonder if we should instead have a top-level config option for including phpunit. That said, if this isn't causing any meaningful performance impact, we can probably leave it as is.

I considered this initially but instead decided to heavily optimize the develop checkout. It only downloads the phpunit files. I've also taken care to parallelize the download, so as far as performance goes, I've found this to be incredibly fast. It only downloads something like 21MB. For what it's worth, I'm pretty sure this is lighter than the phpunit container we're currently spinning up.

For me, this was a tradeoff between standardization and performance. Everyone who is using wp-env should be writing unit tests. All of these tests will likely require the PHPUnit files, so, I didn't see any good reasons not to include them by default.

That's a good reason. The easier we can make unit tests to use, the better.

The entire point of including the test files is that they are better to use than a package like this. We should have an opinionated default and allow them to override it if they really want to.

Since we've made it so that the default is no longer to support `wp-phpunit/wp-phpunit`, we need to explicitly override the default `wp-env` behavior. This will be cleaned up in a future pull request that switches over entirely to the test files in the environment.

Mamaduka · 2022-07-15T13:11:39Z

@noahtallen, is this good to merge?

ObliviousHarmony · 2022-07-15T15:04:15Z

No @Mamaduka, sorry, I need to check on something first. The PHP unit tests in this PR have a different output than trunk. There are risky tests now where there weren't, so, I think something about the test suite has changed and needs debugged.

noahtallen

Left some notes about the changelog and readme. I also want to test the upgrade path for other plugins, so I'll run this branch on a different plugin locally now!

packages/env/CHANGELOG.md

packages/env/README.md

ObliviousHarmony · 2022-07-15T19:07:43Z

Thanks for the feedback @noahtallen, I've gone ahead and made some changes to the changelog and documentation as requested.

noahtallen

Alright, I think this is good to go!

ObliviousHarmony requested a review from noahtallen as a code owner June 17, 2022 09:38

ObliviousHarmony mentioned this pull request Jun 20, 2022

Execute PHP Unit Tests using wp-env woocommerce/woocommerce#32436

Closed

ObliviousHarmony added 6 commits June 20, 2022 15:10

Fixed WPConfig References

d26af6c

Added WordPress PHPUnit Library To wp-env

f14dcef

This commit adds support for downloading the WordPress PHPUnit source and mounting it at /phpunit in the container. It uses the version of WordPress installed in the container to decide what version to download.

Fixed Test Suite

f8f5dcf

Improved WordPress/wp-develop Git Performance

0bf2927

This commit minimizes the amount of data that needs to be downloaded when we pull in the test framework. It does this by not cloning the entire repository and only checking out once we've set up the sparse checkout.

Fixed Linting Errors

68a5e1e

Moved test library to /wordpress-phpunit

99a70d2

This commit moves the test library from `/phpunit' to `/wordpress-phpunit`. This change is to avoid potential confusion. `/phpunit` sounds like a tool while `/wordpress-phpunit` is not so ambiguous.

ObliviousHarmony mentioned this pull request Jun 22, 2022

Utilize WordPress PHPUnit Test Library Included By wp-env #41852

Merged

Fixed Test Config

caa47fb

noahtallen approved these changes Jul 12, 2022

View reviewed changes

ObliviousHarmony added 5 commits July 14, 2022 15:35

Removed wp-phpunit/wp-phpunit Native Support

49fdcca

The entire point of including the test files is that they are better to use than a package like this. We should have an opinionated default and allow them to override it if they really want to.

Fixed Unit Tests

f0c2721

Since we've made it so that the default is no longer to support `wp-phpunit/wp-phpunit`, we need to explicitly override the default `wp-env` behavior. This will be cleaned up in a future pull request that switches over entirely to the test files in the environment.

Merge branch 'trunk' into add/phpcs-installation

27cffcb

Documented PHPUnit Test File Inclusion

3650a2d

Added Changelog

d9dc775

ObliviousHarmony requested a review from noahtallen July 14, 2022 23:17

ObliviousHarmony added 3 commits July 15, 2022 11:00

Fixed Coding Standards

eb07442

Merge branch 'trunk' into add/phpcs-installation

cf5608a

Fixed WP_TESTS_DIR Override

47d307a

noahtallen reviewed Jul 15, 2022

View reviewed changes

packages/env/CHANGELOG.md Outdated Show resolved Hide resolved

packages/env/CHANGELOG.md Outdated Show resolved Hide resolved

packages/env/README.md Outdated Show resolved Hide resolved

packages/env/README.md Outdated Show resolved Hide resolved

Improved documentation and changelog

efe7b23

Added wp-phpunit/wp-phpunit Notice to Changelog

c9a1a23

noahtallen approved these changes Jul 15, 2022

View reviewed changes

noahtallen merged commit 9b13f84 into WordPress:trunk Jul 15, 2022

github-actions bot added this to the Gutenberg 13.8 milestone Jul 15, 2022

noahtallen mentioned this pull request Jul 15, 2022

Update ETK phpunit to work with new wp-env improvements Automattic/wp-calypso#65653

Closed

Mamaduka added [Type] Bug An existing feature does not function as intended [Tool] Env /packages/env labels Jul 27, 2022

This was referenced Aug 1, 2022

Fix is_decoupled() warning when frontend is running on different port Automattic/vip-decoupled-bundle#57

Merged

Fix wp-env WordPress and phpunit version mismatch when no core version is set #43020

Closed

swissspidy mentioned this pull request Aug 15, 2022

Env: Error in readWordPressVersion #43222

Closed

noisysocks mentioned this pull request Oct 12, 2022

wp-env: Use case insensitive regex when checking WP version string #44887

Merged

ockham mentioned this pull request Jan 16, 2023

Workflows: Run PHP unit tests also against current WP - 1 #46983

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Env: Download WordPress PHPUnit Into Container #41780

Env: Download WordPress PHPUnit Into Container #41780

ObliviousHarmony commented Jun 17, 2022 •

edited

Loading

ObliviousHarmony commented Jun 18, 2022

ObliviousHarmony commented Jun 18, 2022

ObliviousHarmony commented Jun 18, 2022

ObliviousHarmony commented Jun 22, 2022

ndiego commented Jul 12, 2022

noahtallen left a comment

noahtallen Jul 12, 2022

noahtallen Jul 12, 2022

ObliviousHarmony Jul 13, 2022

noahtallen Jul 14, 2022

Mamaduka commented Jul 15, 2022

ObliviousHarmony commented Jul 15, 2022 •

edited

Loading

noahtallen left a comment

ObliviousHarmony commented Jul 15, 2022

noahtallen left a comment

		expect( devVolumes ).toEqual( expect.arrayContaining( localSources ) );

		localSources = [

Env: Download WordPress PHPUnit Into Container #41780

Env: Download WordPress PHPUnit Into Container #41780

Conversation

ObliviousHarmony commented Jun 17, 2022 • edited Loading

What?

Why?

How?

Testing Instructions

ObliviousHarmony commented Jun 18, 2022

ObliviousHarmony commented Jun 18, 2022

ObliviousHarmony commented Jun 18, 2022

ObliviousHarmony commented Jun 22, 2022

ndiego commented Jul 12, 2022

noahtallen left a comment

Choose a reason for hiding this comment

noahtallen Jul 12, 2022

Choose a reason for hiding this comment

noahtallen Jul 12, 2022

Choose a reason for hiding this comment

ObliviousHarmony Jul 13, 2022

Choose a reason for hiding this comment

noahtallen Jul 14, 2022

Choose a reason for hiding this comment

Mamaduka commented Jul 15, 2022

ObliviousHarmony commented Jul 15, 2022 • edited Loading

noahtallen left a comment

Choose a reason for hiding this comment

ObliviousHarmony commented Jul 15, 2022

noahtallen left a comment

Choose a reason for hiding this comment

ObliviousHarmony commented Jun 17, 2022 •

edited

Loading

ObliviousHarmony commented Jul 15, 2022 •

edited

Loading