Fix wp-env WordPress and phpunit version mismatch when no core version is set

[alecgeatches]

What?

This adds a check in wp-env to query for the latest release version of WordPress before pulling tests/phpunit test files. This replaces the previous default of pulling the trunk version of phpunit files when no core version is specified in .wp-env.json.

Why?

See investigation of the problem in Automattic/vip-decoupled-bundle#57 (comment), and related issue #41780 where phpunit changes were made.

Currently, when no core config is specified in .wp-env.json, wp-env uses WordPress source from the wordpress docker image which is the latest WordPress release code. phpunit code is pulled from trunk. This can cause test failures when breaking changes are made in trunk that are not yet present in WordPress' latest release.

In this case in the current trunk version of WordPress, wp-db.php has been renamed to class-wpdb.php. During installation of WordPress for testing, phpunit includes class-wpdb.php. However, class-wpdb.php does not exist on the latest release version of WordPress used by wp-env. This causes an ugly failure during bootstrapping which is tricky to track down.

This can be solved by specifying a core WordPress version, but for config files without a version there is always a possibility for a desync. The root problem is that WordPress' code is running from the wordpress docker container and the WordPress version being used isn't yet known during phpunit file downloading.

How?

These changes use the WordPress version check API before tests/phpunit cloning to determine the latest WordPress release version when no pinned version is specified. This makes a few assumptions:

1. The latest WordPress version returned by the API (e.g. 6.0.1) corresponds with a tagged branch in https://github.com/WordPress/wordpress-develop.
2. The docker image for WordPress used internally by wp-env will match the latest WordPress version when no core version is specified.

It would be ideal if the code was able to determine the version of WordPress used by docker directly, but I'm not sure how this would be accomplished. The tests/phpunit directory download is setup during configuration before docker containers start running. Since we don't have direct access to the docker image, the code uses the assumptions above to pull the "correct" version of WordPress test files to match the release container. The code will also default to trunk when the API fails to resolve, so the failure case should be the same behavior as before.

Testing Instructions

For testing, I'm using the repository linked above where I discovered the issue. Here is a reproduction of the original problem using the current version of wp-env:

1. Install the current latest version of wp-env:

   npm install -g @wordpress/env

2. Clone the vip-decoupled-bundle repo for testing

   git clone [email protected]:Automattic/vip-decoupled-bundle.git
   cd vip-decoupled-bundle/

3. Edit .wp-env.json and remove the "core" value to use the default WordPress version.

   {
   "plugins": [
       "."
   ],
   +    "config": {}
   -    "config": {},
   -    "core": "WordPress/WordPress#6.0.1"
   }

4. Run wp-env provisioning:

   wp-env start

5. Install and run tests:

   composer install
   composer run test

6. You should see the error documented in this comment resulting from a test/phpunit and WordPress version mismatch.

Using the files from this branch, here's how to test the fix:

1. In a separate directory, clone the fix/wordpress-phpunit-version-mismatch branch:

   git clone [email protected]:alecgeatches/gutenberg.git -b fix/wordpress-phpunit-version-mismatch --depth 1 gutenberg-alecg
   cd gutenberg-alecg/

2. Enter the wp-env package and install dependencies:

   cd packages/env/
   nvm use
   npm install

3. From the vip-decoupled-bundle directory, recreate the enviroment with branch version of wp-env:

   # In vip-decoupled-bundle/
   
   wp-env destroy
   node <path-to>/gutenberg-alecg/packages/env/bin/wp-env start

4. Run tests again:

   composer run test

The tests should now succeed, as the downloaded phpunit files now target WordPress 6.0.1.

[github-actions]

👋 Thanks for your first Pull Request and for helping build the future of Gutenberg and WordPress, @alecgeatches! In case you missed it, we'd love to have you join us in our Slack community, where we hold regularly weekly meetings open to anyone to coordinate with each other.

If you want to learn more about WordPress development in general, check out the Core Handbook full of helpful information.

[alecgeatches]

Tagging @ObliviousHarmony to check out these phpunit test file code changes for recent related work in #41780. Thank you!

[ObliviousHarmony]

Thanks @alecgeatches,

@noahtallen actually already noticed this mistake and has opened #42826 to resolve it. We talked about how best to approach the problem and decided that if we're going to default to the latest version, we should assume that version throughout wp-env. I appreciate that we all came to the exact same solution for getting the latest version though!

[noahtallen]

Yeah, thanks for working on this! One of the differences with my approach in #42826 is that we now download a WordPress copy so that the phpunit code can find a version to match to (rather than assuming that what Docker provides will always match the latest stable version). This way, we can control the WordPress version rather than Docker providing whatever they want.

[alecgeatches]

@ObliviousHarmony @noahtallen Whoops, I missed #42826 when looking for an already existing issue. I like the changes you've made over there better, and happy this is fixed. Thank you!