drone PHPunit out-of-memory errors

[phil-davis]

drone is quite often getting:

PHPUnit 6.5.14 by Sebastian Bergmann and contributors.

Runtime:       PHPDBG 7.1.29-1+ubuntu16.04.1+deb.sury.org+1
Configuration: /drone/src/tests/phpunit-autotest.xml

.............................................................   61 / 9675 (  0%)
........................S....................................  122 / 9675 (  1%)
.............................................................  183 / 9675 (  1%)
.............................................................  244 / 9675 (  2%)
.............................................................  305 / 9675 (  3%)
.............................................................  366 / 9675 (  3%)
.............................................................  427 / 9675 (  4%)
.............................................................  488 / 9675 (  5%)
.............................................................  549 / 9675 (  5%)
.............................................................  610 / 9675 (  6%)
.............................................................  671 / 9675 (  6%)
.............................................................  732 / 9675 (  7%)
..............
[PHP Fatal error:  Allowed memory size of 4294967296 bytes exhausted (tried to allocate 65536 bytes) in /drone/src/lib/composer/pear/archive_tar/Archive/Tar.php on line 1695]
[PHP Fatal error:  Allowed memory size of 4294967296 bytes exhausted (tried to allocate 65536 bytes) in /drone/src/lib/private/Shutdown/ShutDownManager.php on line 58] 

e.g. https://drone.owncloud.com/owncloud/core/18922/155

It seems to be happening a lot more often this week. It is getting very annoying for CI.

• work out what the test case/environment is where this happens
• fix it

[phil-davis]

@micbar @DeepDiver1975 maybe you know straight away what the problem is ;)
Or can allocate someone to sort it out?

[DeepDiver1975]

This seems to be related to the coverage generation:

+ phpdbg -d memory_limit=4096M -rr ./lib/composer/bin/phpunit --configuration tests/phpunit-autotest.xml --coverage-clover tests/output/coverage/autotest-clover-postgres.xml

[DeepDiver1975]

something to try out:

• move the coverage builds on 7.2 or 7.3

[micbar]

I had something similar on my local machine with coverage.

had been related to blacklisting /whitelisting for coverage beeing configured too "greedy". Maybe it scans far more directories than it should. I found out that whitelisting specific directories took less resources than whitelisting all and restrict afterwards.

[patrickjahns]

Also another idea to investigate:

Move over to ubuntu 18.04 and install the packages there. It might be that current packages have some issues

[phil-davis]

Note:
This PHPunit out of memory problem seemed to become a bit more common in the last week or so. But actually we have also seen this in the past, including on PHPunit5, and now still seeing it on PHPunit6. So something is "on the edge" and a small change in the run-time environment can trigger it.
Needs a "proper" fix anyway, not just allocating a bit more memory in some drone agent.

[phil-davis]

This seems to have got worse today. I am looking into it.

[patrickjahns]

@phil-davis
I've retriggered building owncloudci/php packages - it will change from 7.1.29 -> 7.1.30 at least and fetch latest php packages for other versions as well.
Lets see if this also helps

[phil-davis]

@patrickjahns now on an ordinary PR, e.g. #35695 I am getting phan fail on PHP 7.2
https://drone.owncloud.com/owncloud/core/18989/98

ReflectionException: Method Closure::__invoke() does not exist in /drone/src/vendor-bin/phan/vendor/phan/phan/src/Phan/Language/Element/FunctionFactory.php:96

I get the same locally if I run phan with PHP 7.2 or 7.3
phan passes with PHP 7.1

phan is currently pinned to 0.12.11 because we got a bit stuck on sorting out stuff that it reports.

What to do for today? Just run phan on PHP 7.1 in drone?

Note: that error message goes away with phan 1.1.10. But then I get a long list of phan analysis messages. The list is lots of PhanUndeclaredMethod and is similar when I use the latest "version 1" of phan = 1.3.5

The very latest phan 2.2.2 gives:

ERROR: Phan 2.x requires php-ast 1.0.1+ because it depends on AST version 70. php-ast 0.1.6 is installed.

so that adds another small challenge.

[phil-davis]

This could be looked into again after we drop PHP 7.0 and move up to phpunit7.
The disabled tests (PR #35693 ) will need to be enabled again and see what happens...

[phil-davis]

Fixed by #37078