backupccl, importccl: testrace, stress and stressrace on Bazel are always red

[adityamaru]

https://teamcity.cockroachdb.com/project.html?projectId=Cockroach_Ci_Tests&buildTypeId=&tab=testDetails&testNameId=161655107273305799&order=START_DATE_DESC&branch_Cockroach_Ci_Tests=__all_branches__&itemsCount=-1

https://teamcity.cockroachdb.com/project.html?projectId=Cockroach_Ci_Tests&buildTypeId=&tab=testDetails&testNameId=-7298309073118688234&order=START_DATE_DESC&branch_Cockroach_Ci_Tests=__all_branches__&itemsCount=-1

The test history for the importccl and backupccl packages show that testrace, stress and stressrace builds in bazel are rarely (if ever) green. A quick look at some of the builds point to shards timing out in the case of testrace.

stress and stressrace seem to fail with:

 INFO: Found 1 test target...
[17:11:07][Run stress tests] [0 / 3] [Prepa] BazelWorkspaceStatusAction stable-status.txt
[17:11:10][Run stress tests] [524 / 2,491] GoStdlib external/io_bazel_rules_go/stdlib_/pkg; 2s processwrapper-sandbox ... (16 actions, 15 running)
[17:11:12][Run stress tests] [540 / 2,491] GoStdlib external/io_bazel_rules_go/stdlib_/pkg; 4s processwrapper-sandbox ... (16 actions, 15 running)
[17:11:16][Run stress tests] [558 / 2,491] GoStdlib external/io_bazel_rules_go/stdlib_/pkg; 7s processwrapper-sandbox ... (16 actions, 15 running)
[17:11:20][Run stress tests] [567 / 2,491] GoStdlib external/io_bazel_rules_go/stdlib_/pkg; 12s processwrapper-sandbox ... (16 actions running)
[17:11:24][Run stress tests] [578 / 2,491] GoStdlib external/io_bazel_rules_go/stdlib_/pkg; 16s processwrapper-sandbox ... (16 actions running)
[17:11:28][Run stress tests] [579 / 2,491] GoStdlib external/io_bazel_rules_go/stdlib_/pkg; 20s processwrapper-sandbox ... (16 actions running)
[17:11:34][Run stress tests] [591 / 2,491] GoStdlib external/io_bazel_rules_go/stdlib_/pkg; 26s processwrapper-sandbox ... (16 actions running)
[17:11:40][Run stress tests] [612 / 2,491] GoStdlib external/io_bazel_rules_go/stdlib_/pkg; 32s processwrapper-sandbox ... (16 actions, 15 running)
[17:11:47][Run stress tests] [760 / 2,491] CcCmakeMakeRule c-deps/libproj/include; 38s processwrapper-sandbox ... (16 actions, 15 running)
[17:11:55][Run stress tests] [1,123 / 2,491] CcCmakeMakeRule c-deps/libproj/include; 46s processwrapper-sandbox ... (16 actions, 15 running)
[17:12:04][Run stress tests] [1,425 / 2,491] CcCmakeMakeRule c-deps/libproj/include; 55s processwrapper-sandbox ... (16 actions, 15 running)
[17:12:15][Run stress tests] [1,768 / 2,491] Executing genrule //pkg/sql/parser:sql-goyacc; 20s processwrapper-sandbox ... (16 actions, 15 running)
[17:12:28][Run stress tests] [1,879 / 2,491] Executing genrule //pkg/sql/parser:sql-goyacc; 33s processwrapper-sandbox ... (16 actions, 15 running)
[17:12:42][Run stress tests] [2,114 / 2,801] Executing genrule //pkg/sql/parser:sql-goyacc; 47s processwrapper-sandbox ... (16 actions, 15 running)
[17:12:58][Run stress tests] [2,453 / 3,082] Executing genrule //pkg/sql/parser:sql-goyacc; 63s processwrapper-sandbox ... (16 actions, 15 running)
[17:13:21][Run stress tests] [2,737 / 3,212] Executing genrule //pkg/sql/parser:sql-goyacc; 86s processwrapper-sandbox ... (8 actions, 7 running)
[17:13:45][Run stress tests] [2,798 / 3,212] Executing genrule //pkg/sql/parser:sql-goyacc; 110s processwrapper-sandbox ... (4 actions running)
[17:14:10][Run stress tests] [2,853 / 3,212] Executing genrule //pkg/sql/parser:sql-goyacc [for host]; 87s processwrapper-sandbox ... (5 actions running)
[17:14:40][Run stress tests] [3,023 / 3,212] Executing genrule //pkg/sql/parser:sql-goyacc [for host]; 117s processwrapper-sandbox ... (6 actions running)
[17:15:16][Run stress tests] [3,178 / 3,212] Executing genrule //pkg/sql/colexec/colexecproj:gen-proj-non-const-ops; 20s processwrapper-sandbox ... (3 actions, 2 running)
[17:16:00][Run stress tests] [3,212 / 3,213] [Prepa] Testing //pkg/ccl/backupccl:backupccl_test
[17:16:00][Run stress tests] FAIL: //pkg/ccl/backupccl:backupccl_test (see /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/execroot/cockroach/bazel-out/k8-dbg/testlogs/pkg/ccl/backupccl/backupccl_test/test.log)
[17:16:00][Run stress tests] ==================== Test output for //pkg/ccl/backupccl:backupccl_test:
[17:16:00][Run stress tests] /bin/bash: stress: command not found
[17:16:00][Run stress tests] ================================================================================
[17:16:00][Run stress tests] INFO: From Testing //pkg/ccl/backupccl:backupccl_test:
[17:16:00][Run stress tests] Target //pkg/ccl/backupccl:backupccl_test up-to-date:
[17:16:00][Run stress tests]   /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/execroot/cockroach/bazel-out/k8-dbg/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test
[17:16:00][Run stress tests] INFO: Elapsed time: 307.360s, Critical Path: 251.42s
[17:16:00][Run stress tests] INFO: 3025 processes: 299 internal, 2726 processwrapper-sandbox.
[17:16:00][Run stress tests] INFO: Build completed, 1 test FAILED, 3025 total actions

This is a tracking issue for work that needs to be done to get these builds green!

[blathers-crl]

cc @cockroachdb/bulk-io

[adityamaru]

I'm unsure which team this issue falls on. I assume it is bulkio who is responsible for maintaining these packages but I think we (at least I) lack the expertise to know what is misconfigured. If someone from the bazel team could recommend a way forward I'm happy to see this through.

[stevendanna]

@adityamaru At least for the tests that require the stress command, I have a feeling the problem is in the teamcity scripts or related bazci tool. It doesn't seem to be just our package. For example: https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Ci_Tests_MaybeStressrace/3674362?expandedTest=build%3A%28id%3A3674362%29%2Cid%3A5959&showLog=3674362_5959_5819.5824.5958.5959&logFilter=debug