Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ccl/streamingccl/streamingest: TestTenantStreamingMultipleNodes failed #99900

Closed
cockroach-teamcity opened this issue Mar 29, 2023 · 49 comments · Fixed by #106310
Closed

ccl/streamingccl/streamingest: TestTenantStreamingMultipleNodes failed #99900

cockroach-teamcity opened this issue Mar 29, 2023 · 49 comments · Fixed by #106310
Assignees
Labels
branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-disaster-recovery
Milestone

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Mar 29, 2023

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ a34af06d8e7230e4052db80d55da21a06013f81b:

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1924075900
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:181: condition failed to evaluate within 2m0s: expected job status succeeded, but got running
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1924075900
--- FAIL: TestTenantStreamingMultipleNodes (312.13s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-26123

@cockroach-teamcity cockroach-teamcity added branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Mar 29, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.1 milestone Mar 29, 2023
@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ aec78f33d45a8376a0ecec885688bae60dbfb85c:

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes3770019881
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:181: condition failed to evaluate within 2m0s: expected job status succeeded, but got running
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes3770019881
--- FAIL: TestTenantStreamingMultipleNodes (287.89s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ 7e72aae900c3ff4b44f1643c2d7ba55fbb2cbe23:

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes655755448
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:181: condition failed to evaluate within 2m0s: expected job status succeeded, but got running
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes655755448
--- FAIL: TestTenantStreamingMultipleNodes (304.95s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ 5d5abe704b0551556c35d51aa673d3162f493aa8:

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes2582884448
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:181: condition failed to evaluate within 2m0s: expected job status succeeded, but got running
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes2582884448
--- FAIL: TestTenantStreamingMultipleNodes (309.66s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ 63b683eab3e0ebfff812bb54fdac936c101d9383:

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1976777852
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:181: condition failed to evaluate within 2m0s: expected job status succeeded, but got running
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1976777852
--- FAIL: TestTenantStreamingMultipleNodes (318.51s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ 63b683eab3e0ebfff812bb54fdac936c101d9383:

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes2217391373
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:181: condition failed to evaluate within 2m0s: expected job status succeeded, but got running
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes2217391373
--- FAIL: TestTenantStreamingMultipleNodes (313.07s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ 35b724947ac464e6af45ce5193e34035fcacb1ef:

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1350450497
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:181: condition failed to evaluate within 2m0s: expected job status succeeded, but got running
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1350450497
--- FAIL: TestTenantStreamingMultipleNodes (343.32s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ e847e4358c004d4a8977ae26749768ad1d417fba:

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes3645054781
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:181: condition failed to evaluate within 2m0s: expected job status succeeded, but got running
    panic.go:522: -- test log scope end --
--- FAIL: TestTenantStreamingMultipleNodes (350.94s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ b432e8c20339de5cfa7c811a9ee6f5dc98d15a1e:

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1142741274
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:181: condition failed to evaluate within 2m0s: expected job status succeeded, but got running
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1142741274
--- FAIL: TestTenantStreamingMultipleNodes (368.82s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ 571a1744c72415894dc0e742bd89e74938ea71f4:

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes2174277395
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:181: condition failed to evaluate within 2m0s: expected job status succeeded, but got running
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes2174277395
--- FAIL: TestTenantStreamingMultipleNodes (336.03s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ de239a7438f44d382c9aefceb65d9c39911dabd2:

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes3270845987
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:181: condition failed to evaluate within 2m0s: expected job status succeeded, but got running
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes3270845987
--- FAIL: TestTenantStreamingMultipleNodes (346.80s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ 2f96695f75b07c872ec5f146acc1fa198135768f:

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1125718273
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:181: condition failed to evaluate within 2m0s: expected job status succeeded, but got running
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1125718273
--- FAIL: TestTenantStreamingMultipleNodes (345.65s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ a9eb6d220fc0957d4f6f8261d447991e9d03b4ad:

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes3000248953
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:181: condition failed to evaluate within 2m0s: expected job status succeeded, but got running
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes3000248953
--- FAIL: TestTenantStreamingMultipleNodes (351.06s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ c8a4703f9853a442ac676b44d074b43eb387f60c:

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1704441274
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:181: condition failed to evaluate within 2m0s: expected job status succeeded, but got running
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1704441274
--- FAIL: TestTenantStreamingMultipleNodes (359.40s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ cf84c1c11f2f9c1a4a979033fa0af3befb2ccbcf:

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes2230028440
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:181: condition failed to evaluate within 2m0s: expected job status succeeded, but got running
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes2230028440
--- FAIL: TestTenantStreamingMultipleNodes (378.02s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ ad16885ca3b4567ed5eb34646fe8281fd2d740e3:

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1182253152
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:181: condition failed to evaluate within 2m0s: expected job status succeeded, but got running
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1182253152
--- FAIL: TestTenantStreamingMultipleNodes (357.72s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ 84905b8396e7cd7ef999b264faaafe13110d631a:

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes591484688
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:181: condition failed to evaluate within 2m0s: expected job status succeeded, but got running
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes591484688
--- FAIL: TestTenantStreamingMultipleNodes (328.37s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ 6cce6c746150307a9ecf3b529dfd633a6985c110:

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes3301648202
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:181: condition failed to evaluate within 2m0s: expected job status succeeded, but got running
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes3301648202
--- FAIL: TestTenantStreamingMultipleNodes (353.28s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ 6cce6c746150307a9ecf3b529dfd633a6985c110:

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes595926273
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:181: condition failed to evaluate within 2m0s: expected job status succeeded, but got running
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes595926273
--- FAIL: TestTenantStreamingMultipleNodes (352.75s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ 12607684868d04f957b3385f142e4e8b9b55c799:

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1379670400
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:181: condition failed to evaluate within 2m0s: expected job status succeeded, but got running
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1379670400
--- FAIL: TestTenantStreamingMultipleNodes (342.65s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ 2f86475f2e3d713b4c8c976c34dba99acce7c4c7:

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1861851717
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:181: condition failed to evaluate within 2m0s: expected job status succeeded, but got running
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1861851717
--- FAIL: TestTenantStreamingMultipleNodes (327.26s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ ad6ce866ea3b5c5bb47ba9a0ac19b721a0c98add:

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes3964597765
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:181: condition failed to evaluate within 2m0s: expected job status succeeded, but got running
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes3964597765
--- FAIL: TestTenantStreamingMultipleNodes (365.27s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ 3919dbf5423e3c9c3313d849580ceb44f4b895fd:

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1979244431
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:181: condition failed to evaluate within 2m0s: expected job status succeeded, but got running
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1979244431
--- FAIL: TestTenantStreamingMultipleNodes (357.58s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ 46b5a2cdc03a1bcbf59aab077cd062a8db62cc1b:

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1318314385
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:394: error executing 'ALTER TABLE d.scattered SCATTER': dial tcp 127.0.0.1:33621: connect: connection refused
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1318314385
--- FAIL: TestTenantStreamingMultipleNodes (442.64s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@lidorcarmel
Copy link
Contributor

infra flake, these are pre-emptible VMs and the non-pre-emptible did not fail. test-eng thread: https://cockroachlabs.slack.com/archives/C023S0V4YEB/p1688147468497699

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ e12e85479312972b551677203849d29aeb38ad5f:

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes126114287
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:394: error executing 'ALTER TABLE d.scattered SCATTER': read tcp 127.0.0.1:46074->127.0.0.1:41109: read: connection reset by peer
    panic.go:522: -- test log scope end --
--- FAIL: TestTenantStreamingMultipleNodes (439.07s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ e12e85479312972b551677203849d29aeb38ad5f:

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes963233390
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:394: error executing 'ALTER TABLE d.scattered SCATTER': dial tcp 127.0.0.1:46121: connect: connection refused
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes963233390
--- FAIL: TestTenantStreamingMultipleNodes (517.19s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ e12e85479312972b551677203849d29aeb38ad5f:

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes4034190256
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:394: error executing 'ALTER TABLE d.scattered SCATTER': dial tcp 127.0.0.1:45985: connect: connection refused
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes4034190256
--- FAIL: TestTenantStreamingMultipleNodes (417.61s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ a26ee7efd223c92cda531152837302f03a14f461:

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1267786531
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:394: error executing 'ALTER TABLE d.scattered SCATTER': dial tcp 127.0.0.1:34205: connect: connection refused
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1267786531
--- FAIL: TestTenantStreamingMultipleNodes (506.15s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ f1108313b2b51ffab176b6b230bcc15b0259f170:

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes3476952680
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:394: error executing 'ALTER TABLE d.scattered SCATTER': dial tcp 127.0.0.1:40319: connect: connection refused
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes3476952680
--- FAIL: TestTenantStreamingMultipleNodes (501.11s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ 3eeb4361863fd8ad291a635676b2adf44b6acb4d:

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1277052916
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:394: error executing 'ALTER TABLE d.scattered SCATTER': dial tcp 127.0.0.1:33899: connect: connection refused
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1277052916
--- FAIL: TestTenantStreamingMultipleNodes (394.29s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ 3eeb4361863fd8ad291a635676b2adf44b6acb4d:

Fatal error:

panic: use of Span after Finish. Span: stream-ingestion-processor. Finish previously called at: <stack not captured. Set debugUseAfterFinish>

Stack:

goroutine 515827 [running]:
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish(0x73fddb8?)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:182 +0x12e
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).Tracer(...)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:225
github.com/cockroachdb/cockroach/pkg/util/tracing.ChildSpan({0x73fddb8, 0xc00e045bc0}, {0x5e54718, 0x16})
	github.com/cockroachdb/cockroach/pkg/util/tracing/tracer.go:1584 +0x46
github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamclient.(*partitionedStreamSubscription).Subscribe(0xc019105220, {0x73fddb8?, 0xc00e045bc0?})
	github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamclient/partitioned_stream_client.go:308 +0x65
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x25
golang.org/x/sync/errgroup.(*Group).Go.func1()
	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0xa5
Log preceding fatal error

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1538837135
    test_log_scope.go:79: use -show-logs to present logs inline

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ 5a53114bf5ae9e5ec672d43acf599f145bf22341:

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes943034635
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:394: error executing 'ALTER TABLE d.scattered SCATTER': dial tcp 127.0.0.1:43957: connect: connection refused
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes943034635
--- FAIL: TestTenantStreamingMultipleNodes (498.44s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ 15480b8496e7928845330055d191e331b228eb66:

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1845039945
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:346: condition failed to evaluate within 45s: stream ingestion has not recorded any progress yet, waiting to advance pos 1688886776.231145429,0
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1845039945
--- FAIL: TestTenantStreamingMultipleNodes (471.11s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ f62527e83e458f5f4521497e7bfd5c60b70f1e31:

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes2466137716
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:394: error executing 'ALTER TABLE d.scattered SCATTER': dial tcp 127.0.0.1:35607: connect: connection refused
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes2466137716
--- FAIL: TestTenantStreamingMultipleNodes (399.25s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ 7bfb0ce4cf4e5d3c6c51fd97ced445f4c05fbdc8:

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1175092655
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:394: error executing 'ALTER TABLE d.scattered SCATTER': dial tcp 127.0.0.1:45055: connect: connection refused
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1175092655
--- FAIL: TestTenantStreamingMultipleNodes (406.60s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ 0efd1568d09a5c6a54ae41b0bbf42c682331d24b:

=== RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1671833762
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:346: condition failed to evaluate within 45s: stream ingestion has not recorded any progress yet, waiting to advance pos 1689231126.745784816,0
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1671833762
--- FAIL: TestTenantStreamingMultipleNodes (578.80s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@msbutler msbutler assigned msbutler and unassigned lidorcarmel Jul 13, 2023
craig bot pushed a commit that referenced this issue Jul 13, 2023
106310: streamingest: unskip a multi-node c2c test, and skip under deadlock r=lidorcarmel a=lidorcarmel

These tests are slow and flaky under deadlock.

Epic: none
Fixes: #105956
Fixes: #99900

Release note: None

106765: rfcs: rename the filename to adhere to the pattern r=yuzefovich a=yuzefovich

Recently merged RFC on query SST metrics was missing the prepended date in the filename.

Epic: None.

Release note: None

Co-authored-by: Lidor Carmel <[email protected]>
Co-authored-by: Yahor Yuzefovich <[email protected]>
@craig craig bot closed this as completed in 8f009ec Jul 13, 2023
lidorcarmel added a commit to lidorcarmel/cockroach that referenced this issue Jul 13, 2023
These tests are slow and flaky under deadlock.

Epic: none
Fixes: cockroachdb#105956
Fixes: cockroachdb#99900

Release note: None
lidorcarmel added a commit to lidorcarmel/cockroach that referenced this issue Jul 14, 2023
These tests are slow and flaky under deadlock.

Epic: none
Fixes: cockroachdb#105956
Fixes: cockroachdb#99900

Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-disaster-recovery
Projects
No open projects
Archived in project
Development

Successfully merging a pull request may close this issue.

3 participants