[TEST] Multiple Users Reported Failing Tests

[ghost]

Problem

Since around 2 months, multiple users were reporting test-failures.

List of Issues

• last 6 tests failing #3842 last 6 tests failing
• 6 tests failed #4342 - 6 tests failed
• 6 out of 36 tests failed: is that critical? #4079 - 6 out of 36 tests failed: is that critical?
• Plugin and Nodeos test failed. #4758 - Plugin and Nodeos test failed. -> closed incorrectly, blaming WSL
• rebuild v1.1.0, make test has failed #4764 - rebuild v1.1.0, make test has failed
• a #4765 - Nodeos_run_test and nodeos_sanity_test failing on Ubuntu 18.04
• nodeos,cleos,keosd command not found #4883 (comment)
• make test What is the reason for failure? #5314
• ... (list is incomplete)

Effective Root Cause

• The CI Test-System does not use the same local tests. This way, the test-failures were not detected in an automated manner since months
  • Solution: [TEST] Unify CI Tests with Local Tests #4984 Unify CI Tests with Local Tests

Possible Causes

Own tests on an quite slow i7-16GB laptop show that the failures seem sporadic (not sure though)

• possibly some test-timeouts end up in problems on slower machines
• possibly a full clean-up (processes, data-dirs) is needed for some of the tests

Priority

This issue should get a quite high priority (similar to e.g. build-system-stabilization, like #4923)

Subtasks

(I am currently at this. Anyone can jump in and provide the logs, ideally as file attachments to this issue)

• Find out if the failures occur always at the same tests: => no
• Find out if the failures occur when test is run alone (ctest -R test-name): => sporadically
• Produce test-logs: [TEST] Sporadic Failure, nodeos_voting_lr_test #5036 nodeos_voting_lr_test (last failing tests)
• Get feedback from core-devs

Update

With version 1.1.3, test-errors have disappeared, except for #5036 nodeos_voting_lr_test (fix is in queue)

Full Test-Sequence

ubuntu 18.04, fresh install (Native, Virtual or WSL), then

sudo apt update
sudo apt upgrade
# fetch repo & submodules
./eosio_build.sh
# launch mongodb, as instructed
cd build
ctest --output-on-failure

(it seems apt update/upgrade is not deterministic, but ignoring for now)

[ghost]

The good news: test pass now (version 1.1.3), VirtualBox on Win10pro

The bad news: Total Test time (real) = 3557.49 sec - obviously, I need a new machine.

100% tests passed, 0 tests failed out of 41

Label Time Summary:
long_running_tests    = 1301.26 sec*proc (2 tests)

Total Test time (real) = 3557.49 sec

laz@laz-vb:~/eos/build$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.1 LTS
Release:	18.04
Codename:	bionic
laz@laz-vb:~/eos/build$ 

[jgiszczak]

The long running tests will always take a considerable amount of time as they are dependent on wall clock time. Several of the tests not in the long running section are also dependent on real time.

[ghost]

Ah, I see. Thanks for clarifying.

(But I guess I'll need anyway a classy workstation to minimize build-times)

[ghost]

@brianjohnson5972 , can you please reopen #5036, as I'm still getting errors (1 of 5 runs) ?

[ghost]

Closing in favor of #5263