Measure Code Coverage

[swissspidy]

While working on #2408 I realized there was currently no easy way to measure PHP code coverage for the plugin.

With this PR, code coverage analysis is automatically run on Travis CI and then uploaded to Coveralls. This is the only service that wp-dev-lib supports out of the box. I've previously only used Codecov, but I guess they are pretty much the same.

Also, this PR disables Xdebug when no coverage analysis is needed, for a nice little extra boost.

Wanna get some code coverage analysis today? Fear not! It's as easy as following these steps:

1. Make sure Xdebug is installed and active on your computer. Otherwise use pecl install xdebug
2. Run phpunit --coverage-clover build/logs/clover.xml. This might take a while.
3. In IntellJ, open Analyze -> Show Code Coverage Data and select the newly created XML file.
4. Get visualization like this:

[swissspidy]

After disabling Xdebug when not needed, the Travis jobs are now super fast: https://travis-ci.org/ampproject/amp-wp/builds/537875524 ⚡️

[westonruter]

@swissspidy The code coverage job is failing: https://travis-ci.org/ampproject/amp-wp/jobs/537889379

Read environment variables
Dump submitting json file: /tmp/wordpress/src/wp-content/plugins/amp/build/logs/coveralls-upload.json
File size: 2,012.67 kB
Submitting to https://coveralls.io/api/v1/jobs
Client error occurred. status: 422 Unprocessable Entity
Couldn't find a repository matching this job.
elapsed time: 1.488 sec memory: 14.00 MB

[swissspidy]

@westonruter That is to be expected, as the the project hasn’t been set up on Coveralls yet. See todo above:

Verify whether Coveralls can actually be added (GitHub permissions)

I don‘t have the permissions to add the Coveralls GitHub app to this repository. Without that, Coveralls cannot set things up.

[swissspidy]

Looks like the main repo uses Codecov, so I am inclined to use the same: https://codecov.io/gh/ampproject/amphtml/

Relevant wp-dev-lib issue: xwp/wp-dev-lib#153

How GlotPress implemented it:

https://github.com/GlotPress/GlotPress-WP/blob/a08b7865ba0a28d4a8e84f8e81df66f1c16a9c4a/.travis.yml#L67-L73
https://github.com/GlotPress/GlotPress-WP/blob/a08b7865ba0a28d4a8e84f8e81df66f1c16a9c4a/phpunit.xml.dist#L30-L32

[swissspidy]

Just pushed some config changes that should make it work with Codecov. Nice bonus is that Codecov apparently makes it super easy to upload reports for both PHP and JS coverage.

I also requested permissions in order to get access to this repo with Codecov:

Probably necessary because I am only a collaborator on this repo. Maybe you can directly add it without problems, @westonruter.

[swissspidy]

The last build failed because of PHPUnit errors:

PHP Fatal error:  Uncaught Error: Class 'AMP_Base_Sanitizer' not found in /tmp/wordpress/src/wp-content/plugins/amp/tests/test-class-amp-theme-support.php:1907
Stack trace:
#0 phar:///home/travis/phpunit-bin/5.7/phpunit/phpunit/Util/Fileloader.php(52): include_once()
#1 phar:///home/travis/phpunit-bin/5.7/phpunit/phpunit/Util/Fileloader.php(36): PHPUnit_Util_Fileloader::load('/tmp/wordpress/...')
#2 phar:///home/travis/phpunit-bin/5.7/phpunit/phpunit/Framework/TestSuite.php(316): PHPUnit_Util_Fileloader::checkAndLoad('/tmp/wordpress/...')
#3 phar:///home/travis/phpunit-bin/5.7/phpunit/phpunit/Framework/TestSuite.php(393): PHPUnit_Framework_TestSuite->addTestFile('/tmp/wordpress/...')
#4 phar:///home/travis/phpunit-bin/5.7/phpunit/phpunit/Util/Configuration.php(946): PHPUnit_Framework_TestSuite->addTestFiles(Array)
#5 phar:///home/travis/phpunit-bin/5.7/phpunit/phpunit/Util/Configuration.php(843): PHPUnit_Util_Configuration->getTestSuite(Object(DOMElement), NULL)
#6 phar:///home/travis/phpunit-bin/5.7/phpunit/phpunit/TextUI/Com in /tmp/wordpress/src/wp-content/plugins/amp/tests/test-class-amp-theme-support.php on line 1907
Fatal error: Uncaught Error: Class 'AMP_Base_Sanitizer' not found in /tmp/wordpress/src/wp-content/plugins/amp/tests/test-class-amp-theme-support.php on line 1907
Error: Class 'AMP_Base_Sanitizer' not found in /tmp/wordpress/src/wp-content/plugins/amp/tests/test-class-amp-theme-support.php on line 1907

No idea why this happens. Seems like some autoloader issue, but I couldn't reproduce it locally.

See https://travis-ci.org/ampproject/amp-wp/jobs/538685863#L685-L710

[westonruter]

PHP Fatal error: Uncaught Error: Class 'AMP_Base_Sanitizer' not found in /tmp/wordpress/src/wp-content/plugins/amp/tests/test-class-amp-theme-support.php:1907

@swissspidy This happens when amp.php short-curcuits due to the plugin not having been built, as I see you have discovered!

[westonruter]

@swissspidy I've put in the request to get the GitHub app installed.

I suppose we need to update wp-dev-lib to enable generation of the badge in readme.md.

[swissspidy]

Yeah wp-dev-lib doesn‘t support codecov yet. Not urgent, as it‘s mostly important for ourselves.

Let‘s hope the request gets through soon. Maybe we can ping some people directly?

[swissspidy]

The issue with PHP coverage is a bit odd: https://travis-ci.org/ampproject/amp-wp/jobs/539185192#L979-L1007

PHPUnit says a coverage report was built, but Codecov can't find anything at build/logs/clover.xml.

[westonruter]

PHPUnit says a coverage report was built, but Codecov can't find anything at build/logs/clover.xml.

I believe the problem is that the travis.after_script.sh hasn't been run. This is where code coverage is performed. So that needs to be sourced as well, before running bash <(curl -s https://codecov.io/bash) -cF php -f ...

[swissspidy]

Hmm that just uploads the report to Coveralls, no? We already (try to) upload it to Codecov in the script step, so that doesn‘t really apply to this setup.

The actual code coverage analysis is done by PHPUnit while running the tests.

[swissspidy]

Finally! It works! 🥳

Check out the newest code coverage report here: https://codecov.io/gh/ampproject/amp-wp/tree/140e6b3abdd1c86cf10cdbee8c2fa3e19b85824d

It combines both PHP and JS coverage, which is nice.

@westonruter If you wanna review this again, that would be awesome 🙂

[westonruter]

Nice persistent work.

[amedina]

Excellent! Em sex, 31 de mai de 2019 às 12:07, Pascal Birchler < [email protected]> escreveu:

…

Merged #2434 <#2434> into develop. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2434?email_source=notifications&email_token=AAD3R2M6KA7HYSONBHJCJG3PYFZPDA5CNFSM4HP5F4B2YY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGORX4NQXA#event-2381895772>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAD3R2ITOI7FOKXWAV4LLZ3PYFZPDANCNFSM4HP5F4BQ> .