Skip to content

Commit

Permalink
Merge pull request #372 from Automattic/add/e2e-latest-release
Browse files Browse the repository at this point in the history
Add initial E2E test for latest-release.php
  • Loading branch information
gudmdharalds authored Sep 21, 2023
2 parents 47d9f98 + 02c6b90 commit 33c7009
Show file tree
Hide file tree
Showing 5 changed files with 167 additions and 6 deletions.
12 changes: 7 additions & 5 deletions .github/PULL_REQUEST_TEMPLATE
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@ Draft for a patch:

TODO:
- [ ] Added patch for [describe issue]
- [ ] Add/update tests -- unit and/or integrated (if needed)
- [ ] Ensure only one function is tested per test file.
- [ ] Add/update tests -- unit/integrated/E2E (if needed)
- [ ] Ensure only one function/functionality is tested per test file.
- [ ] Add to, or update, `Scan run detail` report as applicable
- [ ] Check status of automated tests
- [ ] Ensure `PHPDoc` comments are up to date for functions added or altered
Expand All @@ -29,8 +29,8 @@ TODO:
- [ ] Update `--help` message
- [ ] Implement [new feature / logic]
- [ ] Add to, or update, `Scan run detail` report as applicable
- [ ] Add/update tests -- unit and/or integrated
- [ ] Ensure only one function is tested per test file.
- [ ] Add/update tests -- unit/integrated/E2E
- [ ] Ensure only one function/functionality is tested per test file.
- [ ] Ensure `PHPDoc` comments are up to date for functions added or altered
- [ ] Update repository documentation (README.md, RELEASING.md, TESTING.md, TOOLS-UPDATE.md)
- [ ] Assign appropriate [priority](https://github.com/Automattic/vip-go-ci/blob/trunk/CONTRIBUTING.md#priorities) and [type of change labels](https://github.com/Automattic/vip-go-ci/blob/trunk/CONTRIBUTING.md#type-of-change-labels).
Expand All @@ -47,8 +47,10 @@ TODO:
- [ ] Add same version number in defines.php
- [ ] Assign a milestone corresponding to the version number to this PR and PRs that will form the release
- [ ] Assign label ([Changelog and version](https://github.com/Automattic/vip-go-ci/blob/trunk/CONTRIBUTING.md#type-of-change-labels))
- [ ] Run unit-test suite
- [ ] Unit-test suite run successful
- [ ] Integration-test suite successful (without secrets)
- [ ] Run integration-test suite with secrets
- [ ] E2E-test suite run successful
- [ ] Manual testing
- [ ] Pull request with PHP linting issues
- [ ] Pull request with PHPCS issues
Expand Down
35 changes: 35 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,41 @@ jobs:
run: |
rm -f tests/config-secrets.ini
e2e-testing:
name: Run E2E tests (PHP ${{ matrix.php }})
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
php:
- '8.0'
- '8.1'
- '8.2'
steps:
- name: Check out the source code
uses: actions/checkout@v3

- name: Ask git to fetch latest branch and other branches
run: git fetch origin latest && git pull

- name: Set up PHP
uses: shivammathur/[email protected]
with:
coverage: none
php-version: "${{ matrix.php }}"
tools: phpunit:9
env:
fail-fast: 'true'

- name: Configure PHPUnit
run: sed "s:PROJECT_DIR:$(pwd):g" phpunit.xml.dist > phpunit.xml

- name: Run E2E tests
run: phpunit --testsuite=e2e-tests
env:
VIPGOCI_TESTING_DEBUG_MODE: 'true'


php-code-compatibility:
name: PHP code compatibility (8.2)
runs-on: ubuntu-latest
Expand Down
8 changes: 7 additions & 1 deletion TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Introduction

`vip-go-ci` relies on both manual and automated testing. Much of the functionality `vip-go-ci` provides is automatically tested using it's extensive unit and integration test suites. _Most_ of the tests in the test suites are run automatically when code is committed and pushed to the repository, though _some_ integration tests need to be run manually (due to secrets, see below). The manual testing that should be performed is functional, testing the final behaviour of the software.
`vip-go-ci` relies on both manual and automated testing. Much of the functionality `vip-go-ci` provides is automatically tested using it's extensive unit, integration and E2E (End-to-End) test suites. _Most_ of the tests in the test suites are run automatically when code is committed and pushed to the repository, though _some_ integration tests need to be run manually (due to secrets, see below). The manual testing that should be performed is functional, testing the final behaviour of the software.

## Automated testing

Expand Down Expand Up @@ -80,6 +80,12 @@ Integration tests will execute the scanning utilities — PHPCS, SVG scanner and

By using this command, you will run the tests of the test-suite which can be run (depending on tokens and other detail), and get feedback on any errors or warnings. Note that when run, requests will be made to the GitHub API using anonymous calls (unless configured to use an access-token as shown above). It can happen that the GitHub API returns with an error indicating that the maximum limit of API requests has been reached; the solution is to wait and re-run or switch to authenticted calls.

### E2E test suite

The E2E (End-to-End) tests can be run using the following command:

> VIPGOCI_TESTING_DEBUG_MODE=true phpunit --testsuite=e2e-tests
### Test isolation

Note that the test suite uses the `@runTestsInSeparateProcesses` and `@preserveGlobalState` PHPUnit flags to avoid any influence of one test on another. Further, tests should include all required files in `setUp()` function to avoid the same function being defined multiple times across multiple tests during the same run. Combining the usage of `@runTestsInSeparateProcesses` and the inclusion of required files in `setUp()` means each test is independent of other tests, which enables functions to be defined for each test easily and avoids leakage between tests.
Expand Down
3 changes: 3 additions & 0 deletions phpunit.xml.dist
Original file line number Diff line number Diff line change
Expand Up @@ -15,5 +15,8 @@
<testsuite name="integration-tests">
<directory>PROJECT_DIR/tests/integration</directory>
</testsuite>
<testsuite name="e2e-tests">
<directory>PROJECT_DIR/tests/e2e</directory>
</testsuite>
</testsuites>
</phpunit>
115 changes: 115 additions & 0 deletions tests/e2e/LatestReleaseTest.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
<?php
/**
* Verify that latest-release.php behaves as it should.
*
* @package Automattic/vip-go-ci
*/

declare(strict_types=1);

namespace Vipgoci\Tests\E2E;

use PHPUnit\Framework\TestCase;

/**
* Class that implements the testing.
*
* @runTestsInSeparateProcesses
* @preserveGlobalState disabled
*/
final class LatestReleaseTest extends TestCase {
/**
* Temporary file for contents of defines.php.
*
* @var $temp_file_name
*/
private mixed $temp_file_name = '';

/**
* Setup function. Require files, etc.
*
* @return void
*/
protected function setUp() :void {
$this->temp_file_name = tempnam(
sys_get_temp_dir(),
'vipgoci-defines-php-file'
);
}

/**
* Clean up.
*
* @return void
*/
protected function tearDown() :void {
if ( false !== $this->temp_file_name ) {
unlink( $this->temp_file_name );
}
}

/**
* Verify that return value from the script matches
* real version number. Also verify that the format
* is correct.
*
* @return void
*/
public function testResults(): void {
// phpcs:disable WordPress.PHP.DiscouragedPHPFunctions.system_calls_exec
if ( false === $this->temp_file_name ) {
$this->markTestSkipped(
'Unable to create temporary file'
);

return;
}

/*
* Get 'defines.php' from latest branch,
* put contents of the file into temporary file
* and then retrieve the version number
* by including the file.
*/
exec( 'git -C . show latest:defines.php > ' . $this->temp_file_name );

require_once $this->temp_file_name;

$correct_version_number = VIPGOCI_VERSION;

/*
* Run latest-release.php to get latest version number.
*/
$returned_version_number = exec( 'php latest-release.php' );

/*
* Verify format of version number is correct.
*/
$version_number_preg = '/^(\d+\.)?(\d+\.)?(\*|\d+)$/';

$this->assertSame(
1,
preg_match(
$version_number_preg,
$correct_version_number
)
);

$this->assertSame(
1,
preg_match(
$version_number_preg,
$returned_version_number
)
);

/*
* Verify both version numbers match.
*/
$this->assertSame(
$correct_version_number,
$returned_version_number
);
// phpcs:enable WordPress.PHP.DiscouragedPHPFunctions.system_calls_exec
}
}

0 comments on commit 33c7009

Please sign in to comment.