Lazy dataprovider loading #3191

SamMousa · 2018-07-04T08:19:20Z

Q	A
PHPUnit version	7.1.5
PHP version	7.2
Installation Method	Composer

This is not a bug, but more of a feature request / proposal.

Currently the result of a dataprovider is always traversed, meaning that a generator will be fully traversed before the first test starts.

This approach has several downsides:

The whole data set must fit in memory.
Actual testing is delayed.

Changes required would be relatively minor.

PHPUnit\Util\Test::getProvidedData() return type should be changed to ?Traversable.
Instead of testing the dataset beforehand we should test each element separately, we can easily do this by wrapping the Traversable.
TestSuite::createTest() needs some minor changes (can't check if a Traversable is empty).

I'm open for implementing if there's agreement that this would be an improvement.

The text was updated successfully, but these errors were encountered:

sebastianbergmann · 2018-07-04T12:32:26Z

Depends on #10.

keradus · 2018-07-04T21:20:04Z

note: currently there is a possibility to filter tests by data provider label, this shall not be broken

epdenouden · 2019-03-29T10:39:26Z

note: currently there is a possibility to filter tests by data provider label, this shall not be broken

Just to clarify, you are referring to --filter someName also acting on @dataprovider row names or is there some other filtering mechanism I am not yet aware of?

This works with the caveat that the current mechanism discovers available tests by instantiation and then simply using the TestSuite-iterator. To get filters on tests working I have to force load the data rows when a filter gets injected in a DataProviderTestSuite.

Frankly, I am not too familiar with the ins and outs of RecursiveIteratorAggregate. I've always wondered if the execution reordering could have been implemented as a fancy Iterator.

./phpunit \
  --testdox \
  --colors=always \
  --no-configuration \
  --filter true \
  DataProviderFilterTest tests/_files/DataProviderFilterTest.php

PHPUnit 8.1-g78c9461c9 by Sebastian Bergmann and contributors.

DataProviderFilter
 ✔ True with data set 0
 ✔ True with data set 1
 ✔ True with data set 2
 ✔ True with data set 3

Time: 71 ms, Memory: 6.00 MB

OK (4 tests, 4 assertions)

keradus · 2019-06-08T17:05:56Z

Just to clarify, you are referring to --filter someName also acting on @dataProvider row names or is there some other filtering mechanism I am not yet aware of?

example of data provider:

public function provideFoo() {
    return [
        'foo_bar' => [1, 2],
        'baz_123' => [3, 4],
    ];
}

then, you can --filter foo_bar

epdenouden · 2019-06-20T10:06:12Z

@keradus Yes, this scenario works in the prototype. It did require some new logic to check if a dataprovider requires being loaded for filtering to work, but yes, this works fine!

epdenouden · 2019-06-20T10:37:25Z

Hooray! I've got some first results as of yesterday:

Prefetched is the @dataprovider implementation from v8.x
JIT is the experimental just-in-time loading mechanism
JIT+unload is the JIT combined with data unloading during the run

Benefits of improved data provider handling

compared to current implementation speed and resource consumption is equal or better
already saves between 5%-10% memory on PHPUnit's self-test suite, which is light on data sets
smoother resource usage and lower spikes during initialization
works with selectors like --testsuite, --group and --filter
no changes to tests or configuration required

Opportunities for further improvement

Generator row-by-row loading could further smooth load spikes
cache data sets when used by multiple tests
data providers loaded on-demand could be executed non-statically and have access to resources created by fixtures
steadily climbing memory usage hints at low-hanging fruit in TestResult

Testing

The effect of on-demand creation of tests based on @dataprovider rows is visible in the test counts:

.......   61 / 1010 (  6%)      .......   61 / 1827 (  3%)
.......  122 / 1010 ( 12%)      .......  122 / 1827 (  6%)
.......  183 / 1044 ( 17%)      .......  183 / 1827 ( 10%)
.......  244 / 1138 ( 21%)      .......  244 / 1827 ( 13%)
.......  305 / 1172 ( 26%)      .......  305 / 1827 ( 16%)
.......  366 / 1253 ( 29%)      .......  366 / 1827 ( 20%)
.......  427 / 1323 ( 32%)      .......  427 / 1827 ( 23%)
.......  488 / 1334 ( 36%)      .......  488 / 1827 ( 26%)
.......  549 / 1334 ( 41%)      .......  549 / 1827 ( 30%)
.......  610 / 1334 ( 45%)      .......  610 / 1827 ( 33%)
.......  671 / 1336 ( 50%)      .......  671 / 1827 ( 36%)
.......  732 / 1339 ( 54%)      .......  732 / 1827 ( 40%)
.......  793 / 1365 ( 58%)      .......  793 / 1827 ( 43%)
.......  854 / 1393 ( 61%)      .......  854 / 1827 ( 46%)
.......  915 / 1394 ( 65%)      .......  915 / 1827 ( 50%)
.......  976 / 1394 ( 70%)      .......  976 / 1827 ( 53%)
....... 1037 / 1395 ( 74%)      ....... 1037 / 1827 ( 56%)
....... 1098 / 1400 ( 78%)      ....... 1098 / 1827 ( 60%)
....... 1159 / 1400 ( 82%)      ....... 1159 / 1827 ( 63%)
....... 1220 / 1402 ( 87%)      ....... 1220 / 1827 ( 66%)
....... 1281 / 1431 ( 89%)      ....... 1281 / 1827 ( 70%)
....... 1342 / 1449 ( 92%)      ....... 1342 / 1827 ( 73%)
....... 1403 / 1466 ( 95%)      ....... 1403 / 1827 ( 76%)
....... 1464 / 1510 ( 96%)      ....... 1464 / 1827 ( 80%)
....... 1525 / 1548 ( 98%)      ....... 1525 / 1827 ( 83%)
....... 1586 / 1827 ( 86%)      ....... 1586 / 1827 ( 86%)
....... 1647 / 1827 ( 90%)      ....... 1647 / 1827 ( 90%)
....... 1708 / 1827 ( 93%)      ....... 1708 / 1827 ( 93%)
....... 1769 / 1827 ( 96%)      ....... 1769 / 1827 ( 96%)
....    1827 / 1827 (100%)      ....    1827 / 1827 (100%)

Notes

PHPUnit's unit self-tests only use simple and small dataproviders: no large sets, no complex datatypes, no database interaction
memory usage during a run of ./phpunit --testsuite unit with xdebug.trace_format=1
exact measurements are slightly different between environments

keradus · 2019-06-24T05:32:20Z

It did require some new logic to check if a dataprovider requires being loaded for filtering to work

out of curiosity, can you elaborate that logic? like, how to detect if dataprovider shall be loaded or not to filter by keys provided by dataprovider ?

epdenouden · 2019-06-24T08:22:42Z

out of curiosity, can you elaborate that logic? like, how to detect if dataprovider shall be loaded or not to filter by keys provided by dataprovider ?

Unfortunately there is no magic here. Filtering by name requires the names to be available which in turn forces us to load the data provider. This works by loading the data upon DataProviderTestSuite::injectFilter, making the data is available for the iterator filter.

Many optimizations came to mind, introducing their own edge cases. Any ideas welcome as always! I intend to build a proper implementation and your critical eye is very much appreciated.

epdenouden · 2019-06-24T08:26:08Z

if @keradus orders a meal in a restaurant, does the kitchen iterate through the whole menu only to return his order?

epdenouden · 2019-06-24T11:01:12Z

ping @SamMousa on-demand data providers are a possible :)

keradus · 2019-06-24T12:19:09Z

if @epdenouden ordered anything with salami, does the kitchen knows where to look for it, or iterate over all dishes ingredients?

problematic part is that we can filter not only by items (test classes, methods) but also ingredients (case description)

epdenouden · 2019-06-24T22:15:55Z

if @epdenouden ordered anything with salami, does the kitchen knows where to look for it, or iterate over all dishes ingredients?

problematic part is that we can filter not only by items (test classes, methods) but also ingredients (case description)

It will iterate through al dishes looking for salami as an ingredient. This works by simply reverting to the old data provider behaviour when a --filter is used. There are still some efficiency gains possible, but the main advantage of not loading the providers is gone

It might be possible, though! I am not (yet) familiar enough with the whole test filtering logic.

epdenouden · 2019-06-25T16:03:36Z

Superseded by #3736

keradus · 2019-09-08T08:55:28Z

This works by simply reverting to the old data provider behaviour when a --filter is used. There are still some efficiency gains possible, but the main advantage of not loading the providers is gone

It may happen that a lot of great effort made to improve data providers will be simply ignored due to fact that someone used a filter to run only some or the tests ? What a pity, I almost always use filtering, @epdenouden

sebastianbergmann closed this as completed Jul 4, 2018

epdenouden added the type/enhancement A new idea that should be implemented label Mar 29, 2019

epdenouden mentioned this issue Apr 2, 2019

Fatal error triggered in PHPUnit data provider of a test without stacktrace #3582

Closed

epdenouden reopened this Jun 20, 2019

epdenouden self-assigned this Jun 20, 2019

epdenouden added the experimental label Jun 20, 2019

epdenouden mentioned this issue Jun 25, 2019

Reduce resource footprint of data-driven tests #3736

Closed

20 tasks

epdenouden closed this as completed Jun 25, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Lazy dataprovider loading #3191

Lazy dataprovider loading #3191

SamMousa commented Jul 4, 2018

sebastianbergmann commented Jul 4, 2018

keradus commented Jul 4, 2018

epdenouden commented Mar 29, 2019 •

edited

Loading

keradus commented Jun 8, 2019

epdenouden commented Jun 20, 2019

epdenouden commented Jun 20, 2019 •

edited

Loading

keradus commented Jun 24, 2019

epdenouden commented Jun 24, 2019 •

edited

Loading

epdenouden commented Jun 24, 2019

epdenouden commented Jun 24, 2019

keradus commented Jun 24, 2019

epdenouden commented Jun 24, 2019

epdenouden commented Jun 25, 2019

keradus commented Sep 8, 2019

Lazy dataprovider loading #3191

Lazy dataprovider loading #3191

Comments

SamMousa commented Jul 4, 2018

sebastianbergmann commented Jul 4, 2018

keradus commented Jul 4, 2018

epdenouden commented Mar 29, 2019 • edited Loading

keradus commented Jun 8, 2019

epdenouden commented Jun 20, 2019

epdenouden commented Jun 20, 2019 • edited Loading

Benefits of improved data provider handling

Opportunities for further improvement

Testing

Notes

keradus commented Jun 24, 2019

epdenouden commented Jun 24, 2019 • edited Loading

epdenouden commented Jun 24, 2019

epdenouden commented Jun 24, 2019

keradus commented Jun 24, 2019

epdenouden commented Jun 24, 2019

epdenouden commented Jun 25, 2019

keradus commented Sep 8, 2019

epdenouden commented Mar 29, 2019 •

edited

Loading

epdenouden commented Jun 20, 2019 •

edited

Loading

epdenouden commented Jun 24, 2019 •

edited

Loading