[META] Fix random test failures

[jenlampton]

There are several tests that fail randomly, usually only on the Travis CI infrastructure. Let's fix them, or replace them with others so that they don't do more harm than good.

Still giving us trouble:

• AJAX: AJAX forms on cached pages
• Form API: Forms using form storage on cached page
• SimpleTest: Installation profile module tests
• Update: Update core functionality Random test failure: Update: Update core functionality (UpdateCoreTestCase) #1965
• Image: Image style flood protection (ImageStyleFloodProtection):
  fail Correct number of images (1) return 403 access denied. image.test:1792
• Random failure: ModuleUninstallTestCase->testUninstallLocale (module.test) #5226 | Random failure: ModuleUninstallTestCase->testUninstallLocale (module.test)
• Fixed: Functional tests: LayoutUpgradePathTest fails too often in GHA. #6293 | Functional tests: LayoutUpgradePathTest fails too often in GHA

Fixed:

• Path pattern bulk updating (PathPatternBulkUpdateTestCase) - Fixed Random test failures with PathPatternBulkUpdateTestCase #4862
• Random failures or timeouts in BatchProcessingTestCase (race condition in all batch processing) - Random failures or timeouts in BatchProcessingTestCase (race condition in all batch processing) #5205
• Comment: Link with label 1 new comment not found. (CommentInterfaceTest) Fixed: Random failures in CommentInterfaceTest - Link with label 1 new comment not found. #5189
• Locale: Translation import (LocaleImportFunctionalTest) - Fixed: Random test failure: Locale: Translation import (LocaleImportFunctionalTest) #2611
• Bootstrap: Page cache test (BootstrapPageCacheTestCase) - Issue #4418: Avoid race-condition in page header test. Fixing regression from #4306. backdrop#3173
• Comment: Comment interface (CommentInterfaceTest) - Issue #1478: Avoid race-condition in comment-new test checking. backdrop#2601
• Image: Image style flood protection - Issue #1478: Fix random failures in image style flood test. backdrop#1201
• Language settings file not valid JSON - Fixed in Random failure in "Field translations tests": The configuration file "language.settings" could not be read.  #511
• Path: Path pattern basic tests - Issue #1478: Fix random failures in user tests and Module API. backdrop#1200
• User admin test case - Issue #1478: Fix random failures in user tests and Module API. backdrop#1200
• Module: Module API - Issue #1478: Fix random failures in user tests and Module API. backdrop#1200
• Bootstrap: Boot and exit hook invocation - Issue #1478: Fix random failures in user tests and Module API. backdrop#1200
• Bootstrap: Page cache test - Issue #1478: Fix random failures in user tests and Module API. backdrop#1200
• Bootstrap: Page cache test (still) - Issue #1478: Fixing failing page compression test. backdrop#1383
• Batch API: Batch processing (BatchProcessingTestCase) - Issue #1478: Remove random values from Batch processing. backdrop#1983
• Layout: Layout Interface Test - Issue #1478: Fixing PHP 5.3 encoding of JSON backslashes. backdrop#1450
• Layout: Layout BlockText Test (LayoutBlockTextTest) - Issue #1478: Avoid random content in layout block content. backdrop#1942
• Layout: Fix Random LayoutInterfaceTest Failures in PHP 5.3 - Fix Random LayoutInterfaceTest Failures in PHP 5.3 #3080
• Image: Image styles path and URL functions (ImageStylesPathAndUrlUnitTest) - Issue #1478: Remove random string from image style test. backdrop#1984
• Date: Date Theme (DateThemeTestCase) (was meant to be fixed in 1.13.2 with DateThemeTestCase randomly(?) fails on testDateDisplayCombination #3794; same failure noticed with 1.14.1) follow-up issue: DateThemeTestCase fails at certain times of day #4180:
  Start and end date on same day with timezone, with remaining days renders correctly. date_themes.test:137

[quicksketch]

I got a couple more randoms today. I think some of these new ones might be related to Background Fetch if the server isn't running fast enough. This was on PHP 5.4 and the tests ran for 30 minutes, compared to the normally speedy 10-15 minutes.

---- Bootstrap: Boot and exit hook invocation (HookBootExitTestCase) ----


Status    Group      Filename          Line Function                            
--------------------------------------------------------------------------------
Fail      Other      bootstrap.test     377 HookBootExitTestCase->testHookBootE
    Page cache cleared.
Fail      Other      bootstrap.test     380 HookBootExitTestCase->testHookBootE
    hook_boot called with aggressive cache and no cached page.
Fail      Other      bootstrap.test     381 HookBootExitTestCase->testHookBootE
    hook_exit called with aggressive cache and no cached page.


---- Layout: Layout Interface Test (LayoutInterfaceTest) ----


Status    Group      Filename          Line Function                            
--------------------------------------------------------------------------------
Fail      Other      layout.test        608 LayoutInterfaceTest->testMainConten
    "The "Main page content" block may only be added once to this
    layout." found
Exception Notice     layout.test        614 LayoutInterfaceTest->testMainConten
    Undefined offset: 0
Exception Notice     layout.test        618 LayoutInterfaceTest->testMainConten
    Undefined offset: 0
Fail      Other      layout.test        621 LayoutInterfaceTest->testMainConten
    Found the Save layout button
Fail      Other      layout.test        621 LayoutInterfaceTest->testMainConten
    Found the requested form fields at
Fail      Other      layout.test        622 LayoutInterfaceTest->testMainConten
    Layout saved successfully after removing old content block.
Fail      Other      layout.test        632 LayoutInterfaceTest->testMainConten
    Main content shown in sidebar after moving main content block.


---- Module: Module API (ModuleUnitTest) ----


Status    Group      Filename          Line Function                            
--------------------------------------------------------------------------------
Fail      Other      module.test         84 ModuleUnitTest->testModuleImplement
    The module implements cache is populated after requesting a page.

[quicksketch]

Details about the "User admin test case" fail:

---- User: User administration (UserAdminTestCase) ----
Status    Group      Filename          Line Function                            
--------------------------------------------------------------------------------
Fail      Other      user.test         1441 UserAdminTestCase->testUserAdmin() 
    User C blocked
Fail      E-mail     user.test         1450 UserAdminTestCase->testUserAdmin() 
    Activation mail sent to user C

[quicksketch]

Filed a PR at backdrop/backdrop#1200 that addresses 4 of these failures. Updated the summary.

[quicksketch]

I think I've got all these worked out. For image styles and module API, the added sleep() timeouts should help the slower servers successfully pass when dealing with time-sensitive issues. The user and path alias tests I rewrote to use more robust XPath checking of elements instead of guessing based on the User ID.

The Layout: Layout Interface Test (LayoutInterfaceTest) is the only one on which I'm unclear. If that test fail keeps coming up, hopefully we will have at least narrowed the scope of the random test failures.

All the fixes so far have gone into 1.2.x and 1.x, so I'm tagging this for 1.2.4.

[quicksketch]

Here's dump of the current fail on the Layout Interface test fail report:

---- Layout: Layout Interface Test (LayoutInterfaceTest) ----
Status    Group      Filename          Line Function                            
--------------------------------------------------------------------------------
Fail      Other      layout.test        608 LayoutInterfaceTest->testMainConten
    "The "Main page content" block may only be added once to this
    layout." found
Exception Notice     layout.test        614 LayoutInterfaceTest->testMainConten
    Undefined offset: 0
Exception Notice     layout.test        618 LayoutInterfaceTest->testMainConten
    Undefined offset: 0
Fail      Other      layout.test        621 LayoutInterfaceTest->testMainConten
    Found the Save layout button
Fail      Other      layout.test        621 LayoutInterfaceTest->testMainConten
    Found the requested form fields at
Fail      Other      layout.test        622 LayoutInterfaceTest->testMainConten
    Layout saved successfully after removing old content block.
Fail      Other      layout.test        632 LayoutInterfaceTest->testMainConten
    Main content shown in sidebar after moving main content block.

Looks like our old friend "Invalid JSON" is still happening as well:

---- User: User email administration (UserAdminSettingsEmailFormTest) ----
Status    Group      Filename          Line Function                            
--------------------------------------------------------------------------------
Exception Uncaught e config.inc         898 ConfigFileStorage->read()          
    ConfigStorageReadException: The configuration file "user.mail" is
    not properly formatted JSON. in ConfigFileStorage->read() (line 898 of
    /home/travis/build/jenlampton/backdrop/core/includes/config.inc).

[quicksketch]

We're still getting a lot of failing page cache tests:

---- Bootstrap: Page cache test (BootstrapPageCacheTestCase) ----
Status    Group      Filename          Line Function                            
--------------------------------------------------------------------------------

Fail      Other      bootstrap.test     204 BootstrapPageCacheTestCase->testPag
    Cached page request.

I think this might have to do with the background fetch functionality and our use of sleep() to try and wait long enough, but that is not guaranteed, especially with the slower Travis CI servers. I filed a new PR at backdrop/backdrop#1367.

[quicksketch]

I merged in backdrop/backdrop#1367, which passed the fist attempt. We'll see how it does with new PRs as they come in.

[Gormartsen]

I am able to reproduce next issue with backdrop/backdrop#1367 applied.

Bootstrap: Page cache test (BootstrapPageCacheTestCase) 155 passes, 1 fail, 0 exceptions, and 41 debug messages

Test run duration: 58 sec

Detailed test results
---------------------


---- Bootstrap: Page cache test (BootstrapPageCacheTestCase) ----


Status    Group      Filename          Line Function
--------------------------------------------------------------------------------
Fail      Other      bootstrap.test     202 BootstrapPageCacheTestCase->testPag
    Initial page requests returned before shutdown functions are executed.

I am doing investigation here.

[Gormartsen]

PS: random fail happen again here: https://travis-ci.org/backdrop/backdrop/jobs/130238636

---- Bootstrap: Page cache test (BootstrapPageCacheTestCase) ----
Status    Group      Filename          Line Function                            
--------------------------------------------------------------------------------
Fail      Other      bootstrap.test     328 BootstrapPageCacheTestCase->testPag
    Page was cached.

[Gormartsen]

This test is failing here:

  $start = microtime(TRUE);
    $this->backdropGet('system-test/sleep/shutdown/5', array(), $headers);
    $total = microtime(TRUE) - $start;
    $this->assertEqual($this->backdropGetHeader('X-Backdrop-Cache'), 'MISS', 'Initial page request was miss.');
    echo "Total: " . $total ."\n";
    $this->assertTrue($total < 5, 'Initial page requests returned before shutdown functions are executed.');

test with display $total has next result:

Bootstrap: Page cache test (BootstrapPageCacheTestCase) 155 passes, 1 fail, 0 exceptions, and 41 debug messages
Total: 5.0353920459747

Test run duration: 58 sec

Detailed test results
---------------------


---- Bootstrap: Page cache test (BootstrapPageCacheTestCase) ----


Status    Group      Filename          Line Function
--------------------------------------------------------------------------------
Fail      Other      bootstrap.test     203 BootstrapPageCacheTestCase->testPag
    Initial page requests returned before shutdown functions are executed.

It mean that

 $this->backdropGet('system-test/sleep/shutdown/5', array(), $headers);

takes 5.03 sec to process

data from function system_test_sleep:

<div id="start">1463241968.3128</div>
  5 seconds will be slept at shutdown. 
<div id="end">1463241968.3128</div>

[quicksketch]

It mean that [...] takes 5.03 sec to process

Well in this case it's a "good" failure! Because the functionality is actually broken or not working. The page isn't being returned immediately, it's getting held up by a shutdown function when it shouldn't be. We've had some issues where backdrop_page_footer() doesn't flush and close the connection properly based on differing PHP configurations (e.g. mod_php vs. FastCGI vs. PHP-FPM). So this likely needs to be fixed in backdrop_page_footer() rather than adjusting the test.

[Gormartsen]

Thank you for details @quicksketch . It is different setup from travis. I am going to figure out this one.

[Gormartsen]

@quicksketch I fixed this (system-test/sleep/shutdown/5) issue. PR is here: backdrop/backdrop#1378

[quicksketch]

New PR for fixing the comment-new race-condition that causes what might be our most common random failure: backdrop/backdrop#2601

[herbdool]

RTBC for this PR backdrop/backdrop#2601 (comment) @quicksketch

[klonos]

From time to time, I see this:

"Generated 20 URL aliases." found path_pattern.test:929

[klonos]

...never mind; was already in the issue summary as Path: Path pattern bulk updating (PathPatternBulkUpdateTestCase)

[quicksketch]

A new PR for a failure that's been happening frequently since 1.16.0: backdrop/backdrop#3173

[indigoxela]

Seems like CommentInterfaceTest is back...

[quicksketch]

We've made a lot of progress recently since the switch to GitHub Actions (a lot of these had been showing up more frequently). The following now seem resolved:

• Random test failures with PathPatternBulkUpdateTestCase Fixed Random test failures with PathPatternBulkUpdateTestCase #4862
• Random failures or timeouts in BatchProcessingTestCase (race condition in all batch processing) Random failures or timeouts in BatchProcessingTestCase (race condition in all batch processing) #5205
• Random failures in CommentInterfaceTest - Link with label 1 new comment not found. Fixed: Random failures in CommentInterfaceTest - Link with label 1 new comment not found. #5189

The fixing of #5205 should fix intermittent failures in several other tests (anything that used batch processing). I think this may include #1965, but I'll leave that issue open to confirm.