Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ccl/streamingccl/streamingest: TestRandomClientGeneration failed #99343

Closed
cockroach-teamcity opened this issue Mar 23, 2023 · 11 comments · Fixed by #100813
Closed

ccl/streamingccl/streamingest: TestRandomClientGeneration failed #99343

cockroach-teamcity opened this issue Mar 23, 2023 · 11 comments · Fixed by #100813
Assignees
Labels
branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-disaster-recovery
Milestone

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Mar 23, 2023

ccl/streamingccl/streamingest.TestRandomClientGeneration failed with artifacts on release-23.1 @ 751b17200b6026b73351c57cab734415d2c4b02a:

=== RUN   TestRandomClientGeneration
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestRandomClientGeneration1740823187
    test_log_scope.go:79: use -show-logs to present logs inline
*
* INFO: Running test with the default test tenant. If you are only seeing a test case failure when this message appears, there may be a problem with your test case running within tenants.
*
*
* INFO: Running test with the default test tenant. If you are only seeing a test case failure when this message appears, there may be a problem with your test case running within tenants.
*
*
* INFO: Running test with the default test tenant. If you are only seeing a test case failure when this message appears, there may be a problem with your test case running within tenants.
*
    stream_ingestion_processor_test.go:404: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest/stream_ingestion_processor_test.go:404
        	            				github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest/stream_ingestion_processor_test.go:552
        	Error:      	"0" is not greater than "0"
        	Test:       	TestRandomClientGeneration
    panic.go:540: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestRandomClientGeneration1740823187
--- FAIL: TestRandomClientGeneration (67.69s)

Parameters: TAGS=bazel,gss,race

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-25832

@cockroach-teamcity cockroach-teamcity added branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Mar 23, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.1 milestone Mar 23, 2023
@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestRandomClientGeneration failed with artifacts on release-23.1 @ f351747ed97862fc037717cadec23f18073fb6be:

=== RUN   TestRandomClientGeneration
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestRandomClientGeneration745508372
    test_log_scope.go:79: use -show-logs to present logs inline
*
* INFO: Running test with the default test tenant. If you are only seeing a test case failure when this message appears, there may be a problem with your test case running within tenants.
*
*
* INFO: Running test with the default test tenant. If you are only seeing a test case failure when this message appears, there may be a problem with your test case running within tenants.
*
*
* INFO: Running test with the default test tenant. If you are only seeing a test case failure when this message appears, there may be a problem with your test case running within tenants.
*
    stream_ingestion_processor_test.go:404: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest/stream_ingestion_processor_test.go:404
        	            				github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest/stream_ingestion_processor_test.go:552
        	Error:      	"0" is not greater than "0"
        	Test:       	TestRandomClientGeneration
    panic.go:540: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestRandomClientGeneration745508372
--- FAIL: TestRandomClientGeneration (74.80s)

Parameters: TAGS=bazel,gss,race

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestRandomClientGeneration failed with artifacts on release-23.1 @ b995ddff53ee95a30537a366c6cd8e0e13fcee79:

=== RUN   TestRandomClientGeneration
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestRandomClientGeneration986735949
    test_log_scope.go:79: use -show-logs to present logs inline
*
* INFO: Running test with the default test tenant. If you are only seeing a test case failure when this message appears, there may be a problem with your test case running within tenants.
*
*
* INFO: Running test with the default test tenant. If you are only seeing a test case failure when this message appears, there may be a problem with your test case running within tenants.
*
*
* INFO: Running test with the default test tenant. If you are only seeing a test case failure when this message appears, there may be a problem with your test case running within tenants.
*
    stream_ingestion_processor_test.go:404: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest/stream_ingestion_processor_test.go:404
        	            				github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest/stream_ingestion_processor_test.go:552
        	Error:      	"0" is not greater than "0"
        	Test:       	TestRandomClientGeneration
    panic.go:540: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestRandomClientGeneration986735949
--- FAIL: TestRandomClientGeneration (66.27s)

Parameters: TAGS=bazel,gss,race

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestRandomClientGeneration failed with artifacts on release-23.1 @ 186218edb86b5974501b2976cfd0704d4d6d1beb:

=== RUN   TestRandomClientGeneration
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestRandomClientGeneration460408357
    test_log_scope.go:79: use -show-logs to present logs inline
    stream_ingestion_processor_test.go:404: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest/stream_ingestion_processor_test.go:404
        	            				github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest/stream_ingestion_processor_test.go:552
        	Error:      	"0" is not greater than "0"
        	Test:       	TestRandomClientGeneration
    panic.go:540: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestRandomClientGeneration460408357
--- FAIL: TestRandomClientGeneration (61.91s)

Parameters: TAGS=bazel,gss,race

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestRandomClientGeneration failed with artifacts on release-23.1 @ 3551dc089ac3efcb3772df2109164dcb01e79d8f:

=== RUN   TestRandomClientGeneration
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestRandomClientGeneration550885663
    test_log_scope.go:79: use -show-logs to present logs inline
*
* INFO: Running test with the default test tenant. If you are only seeing a test case failure when this message appears, there may be a problem with your test case running within tenants.
*
*
* INFO: Running test with the default test tenant. If you are only seeing a test case failure when this message appears, there may be a problem with your test case running within tenants.
*
*
* INFO: Running test with the default test tenant. If you are only seeing a test case failure when this message appears, there may be a problem with your test case running within tenants.
*
    stream_ingestion_processor_test.go:404: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest/stream_ingestion_processor_test.go:404
        	            				github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest/stream_ingestion_processor_test.go:552
        	Error:      	"0" is not greater than "0"
        	Test:       	TestRandomClientGeneration
    panic.go:540: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestRandomClientGeneration550885663
--- FAIL: TestRandomClientGeneration (67.96s)

Parameters: TAGS=bazel,gss,race

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@stevendanna stevendanna self-assigned this Mar 28, 2023
@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestRandomClientGeneration failed with artifacts on release-23.1 @ a34af06d8e7230e4052db80d55da21a06013f81b:

=== RUN   TestRandomClientGeneration
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestRandomClientGeneration260716344
    test_log_scope.go:79: use -show-logs to present logs inline
    stream_ingestion_processor_test.go:404: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest/stream_ingestion_processor_test.go:404
        	            				github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest/stream_ingestion_processor_test.go:552
        	Error:      	"0" is not greater than "0"
        	Test:       	TestRandomClientGeneration
    panic.go:540: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestRandomClientGeneration260716344
--- FAIL: TestRandomClientGeneration (63.42s)

Parameters: TAGS=bazel,gss,race

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestRandomClientGeneration failed with artifacts on release-23.1 @ 7e72aae900c3ff4b44f1643c2d7ba55fbb2cbe23:

=== RUN   TestRandomClientGeneration
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestRandomClientGeneration1843982076
    test_log_scope.go:79: use -show-logs to present logs inline
*
* INFO: Running test with the default test tenant. If you are only seeing a test case failure when this message appears, there may be a problem with your test case running within tenants.
*
*
* INFO: Running test with the default test tenant. If you are only seeing a test case failure when this message appears, there may be a problem with your test case running within tenants.
*
*
* INFO: Running test with the default test tenant. If you are only seeing a test case failure when this message appears, there may be a problem with your test case running within tenants.
*
    stream_ingestion_processor_test.go:404: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest/stream_ingestion_processor_test.go:404
        	            				github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest/stream_ingestion_processor_test.go:552
        	Error:      	"0" is not greater than "0"
        	Test:       	TestRandomClientGeneration
    panic.go:540: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestRandomClientGeneration1843982076
--- FAIL: TestRandomClientGeneration (73.02s)

Parameters: TAGS=bazel,gss,race

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@msbutler msbutler self-assigned this Mar 31, 2023
@msbutler
Copy link
Collaborator

@stevendanna I think #97845 is responsible for this failure.

./dev test pkg/ccl/streamingccl/streamingest -f TestRandomClientGeneration --race --stress --cpus 8 --timeout=5m passes on the first commit in the PR (related to modifying the kv request), and fails on the last commit in the PR.

@msbutler
Copy link
Collaborator

msbutler commented Mar 31, 2023

oh fun, it has something to do with the locking implementation I recommended 67ba17f . The test only fails when this commit is included....

@stevendanna stevendanna added release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Mar 31, 2023
@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestRandomClientGeneration failed with artifacts on release-23.1 @ 63b683eab3e0ebfff812bb54fdac936c101d9383:

=== RUN   TestRandomClientGeneration
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestRandomClientGeneration1509922753
    test_log_scope.go:79: use -show-logs to present logs inline
*
* INFO: Running test with the default test tenant. If you are only seeing a test case failure when this message appears, there may be a problem with your test case running within tenants.
*
*
* INFO: Running test with the default test tenant. If you are only seeing a test case failure when this message appears, there may be a problem with your test case running within tenants.
*
*
* INFO: Running test with the default test tenant. If you are only seeing a test case failure when this message appears, there may be a problem with your test case running within tenants.
*
    stream_ingestion_processor_test.go:404: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest/stream_ingestion_processor_test.go:404
        	            				github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest/stream_ingestion_processor_test.go:552
        	Error:      	"0" is not greater than "0"
        	Test:       	TestRandomClientGeneration
    panic.go:540: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestRandomClientGeneration1509922753
--- FAIL: TestRandomClientGeneration (69.07s)

Parameters: TAGS=bazel,gss,race

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestRandomClientGeneration failed with artifacts on release-23.1 @ 63b683eab3e0ebfff812bb54fdac936c101d9383:

=== RUN   TestRandomClientGeneration
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestRandomClientGeneration388574145
    test_log_scope.go:79: use -show-logs to present logs inline
    stream_ingestion_processor_test.go:404: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest/stream_ingestion_processor_test.go:404
        	            				github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest/stream_ingestion_processor_test.go:552
        	Error:      	"0" is not greater than "0"
        	Test:       	TestRandomClientGeneration
    panic.go:540: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestRandomClientGeneration388574145
--- FAIL: TestRandomClientGeneration (63.03s)

Parameters: TAGS=bazel,gss,race

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestRandomClientGeneration failed with artifacts on release-23.1 @ b432e8c20339de5cfa7c811a9ee6f5dc98d15a1e:

=== RUN   TestRandomClientGeneration
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestRandomClientGeneration3724495846
    test_log_scope.go:79: use -show-logs to present logs inline
    stream_ingestion_processor_test.go:404: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest/stream_ingestion_processor_test.go:404
        	            				github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest/stream_ingestion_processor_test.go:552
        	Error:      	"0" is not greater than "0"
        	Test:       	TestRandomClientGeneration
    panic.go:540: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestRandomClientGeneration3724495846
--- FAIL: TestRandomClientGeneration (62.47s)

Parameters: TAGS=bazel,gss,race

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

msbutler added a commit to msbutler/cockroach that referenced this issue Apr 6, 2023
This patch fixes 4 bugs in TestRandomClientGeneration that were
responsible for the persistent flakiness and lack of coverage in this test:
- the randomeStreamClient no longer instantiates keys with a table prefix that
	collides with the job info table prefix. This collision was the original
  cause of the flakes reported in cockroachdb#99343.
- getPartitionSpanToTableId() now generates a correct map from source partition
	key space to table Id. Previously, the key spans in the map didn't contain
keys that mapped to anything logical in the cockroach key space.
- assertKVs() now checks for keys in the destination tenant keyspace.
- assertKVs() now actually asserts that kvs were found. Before, the assertion
  could pass if no keys were actually checked, which has been happening for
  months and allowed the bugs above to infest this test.

Fixes cockroachdb#99343

Release note: None
craig bot pushed a commit that referenced this issue Apr 10, 2023
100813: streamingccl: deflake TestRandomClientGeneration r=adityamaru a=msbutler

This patch fixes 4 bugs in TestRandomClientGeneration that were responsible for the persistent flakiness and lack of coverage in this test:
- the randomeStreamClient no longer instantiates keys with a table prefix that collides with the job info table prefix. This collision was the original cause of the flakes reported in #99343.
- getPartitionSpanToTableId() now generates a correct map from source partition key space to table Id. Previously, the key spans in the map didn't contain keys that mapped to anything logical in the cockroach key space.
- assertKVs() now checks for keys in the destination tenant keyspace.
- assertKVs() now actually asserts that kvs were found. Before, the assertion could pass if no keys were actually checked, which has been happening for months and allowed the bugs above to infest this test.

Fixes #99343

Release note: None

100952: cli: trash `TestNoLinkForbidden` r=rail a=rickystewart

This test does not work:
1. The test [has been broken](#74119) for years.
2. The test is not sensible in the Bazel world anyway, and under remote execution the test fails with an error like the following:

```
 build.go:59: go/build: go list github.com/cockroachdb/cockroach/pkg/cmd/cockroach: fork/exec GOROOT/bin/go: no such file or directory
```

The bug to replace this test with working functionality based on Bazel is #81526.

Epic: CRDB-17165
Release note: None
Closes #74119.

100965: sql: link issue to unimplemented mutations in udfs r=mgartner,rytaft a=rharding6373

Links an issue to the unimplemented errors for mutations in UDFs.

Epic: None
Informs: #87289
Fixes: #99715

Release note: None

Co-authored-by: Michael Butler <[email protected]>
Co-authored-by: Ricky Stewart <[email protected]>
Co-authored-by: rharding6373 <[email protected]>
@craig craig bot closed this as completed in 3539505 Apr 10, 2023
msbutler added a commit to msbutler/cockroach that referenced this issue Apr 11, 2023
This patch fixes 4 bugs in TestRandomClientGeneration that were
responsible for the persistent flakiness and lack of coverage in this test:
- the randomeStreamClient no longer instantiates keys with a table prefix that
	collides with the job info table prefix. This collision was the original
  cause of the flakes reported in cockroachdb#99343.
- getPartitionSpanToTableId() now generates a correct map from source partition
	key space to table Id. Previously, the key spans in the map didn't contain
keys that mapped to anything logical in the cockroach key space.
- assertKVs() now checks for keys in the destination tenant keyspace.
- assertKVs() now actually asserts that kvs were found. Before, the assertion
  could pass if no keys were actually checked, which has been happening for
  months and allowed the bugs above to infest this test.

Fixes cockroachdb#99343

Release note: None
blathers-crl bot pushed a commit that referenced this issue Apr 11, 2023
This patch fixes 4 bugs in TestRandomClientGeneration that were
responsible for the persistent flakiness and lack of coverage in this test:
- the randomeStreamClient no longer instantiates keys with a table prefix that
	collides with the job info table prefix. This collision was the original
  cause of the flakes reported in #99343.
- getPartitionSpanToTableId() now generates a correct map from source partition
	key space to table Id. Previously, the key spans in the map didn't contain
keys that mapped to anything logical in the cockroach key space.
- assertKVs() now checks for keys in the destination tenant keyspace.
- assertKVs() now actually asserts that kvs were found. Before, the assertion
  could pass if no keys were actually checked, which has been happening for
  months and allowed the bugs above to infest this test.

Fixes #99343

Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-disaster-recovery
Projects
No open projects
Archived in project
Development

Successfully merging a pull request may close this issue.

3 participants