Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kv/kvserver: TestProtectedTimestamps failed #93497

Closed
cockroach-teamcity opened this issue Dec 13, 2022 · 5 comments · Fixed by #105641
Closed

kv/kvserver: TestProtectedTimestamps failed #93497

cockroach-teamcity opened this issue Dec 13, 2022 · 5 comments · Fixed by #105641
Assignees
@arulajmani
Labels
branch-master Failures and bugs on the master branch. branch-release-22.2 Used to mark GA and release blockers, technical advisories, and bugs for 22.2 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. skipped-test T-kv KV Team
Milestone
22.2

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Dec 13, 2022 •
edited by cockroach-jira-scripts
Loading

kv/kvserver.TestProtectedTimestamps failed with artifacts on release-22.2 @ 9a14474ed902e168ec6e6d1ea586d13eb92a2288:

=== RUN   TestProtectedTimestamps
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/33e1d369c27b9c01b2b6009c561815a3/logTestProtectedTimestamps534488675
    test_log_scope.go:79: use -show-logs to present logs inline
    client_protectedts_test.go:267: 
        	Error Trace:	/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1884/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_test_/kvserver_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvserver/client_protectedts_test.go:267
        	            				/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1884/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_test_/kvserver_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvserver/soon.go:69
        	            				/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1884/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_test_/kvserver_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvserver/retry.go:208
        	            				/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1884/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_test_/kvserver_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvserver/soon.go:75
        	            				/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1884/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_test_/kvserver_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvserver/soon.go:57
        	            				/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1884/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_test_/kvserver_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvserver/soon.go:40
        	            				/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1884/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_test_/kvserver_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvserver/client_protectedts_test.go:260
        	Error:      	Should be true
        	Test:       	TestProtectedTimestamps
        	Messages:   	1670927709.442153182,0 >= 1670927709.442153181,2147483647
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/33e1d369c27b9c01b2b6009c561815a3/logTestProtectedTimestamps534488675
--- FAIL: TestProtectedTimestamps (7.66s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-22365
@cockroach-teamcity cockroach-teamcity added branch-release-22.2 Used to mark GA and release blockers, technical advisories, and bugs for 22.2 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Dec 13, 2022
@cockroach-teamcity cockroach-teamcity added this to the 22.2 milestone Dec 13, 2022
@blathers-crl blathers-crl bot added the T-kv KV Team label Dec 13, 2022
@DrewKimball
Copy link
Collaborator

Saw an instance of this on master as well in #93673.

@renatolabs
Copy link
Contributor

22.1 issue: #93183.

@cockroach-teamcity cockroach-teamcity mentioned this issue Jan 7, 2023
Closed
@github-actions
Copy link

We have marked this test failure issue as stale because it has been
inactive for 1 month. If this failure is still relevant, removing the
stale label or adding a comment will keep it active. Otherwise,
we'll close it in 5 days to keep the test failure queue tidy.

@nvanbenschoten nvanbenschoten added skipped-test GA-blocker branch-master Failures and bugs on the master branch. labels Mar 27, 2023
@tbg
Copy link
Member

tbg commented May 31, 2023

Just picked this up, fails now with

--- FAIL: TestProtectedTimestamps (16.80s)
    test_log_scope.go:161: test logs captured to: /tmp/cockroach/_tmp/33e1d369c27b9c01b2b6009c561815a3/logTestProtectedTimestamps1101236470
    test_log_scope.go:79: use -show-logs to present logs inline
    client_protectedts_test.go:78:
                Error Trace:    github.com/cockroachdb/cockroach/pkg/kv/kvserver_test/pkg/kv/kvserver/client_protectedts_test.go:78
                Error:          Received unexpected error:
                                pq: could not validate zone config: RangeMaxBytes 262144 less than minimum allowed 67108864
                Test:           TestProtectedTimestamps
    panic.go:522: -- test log scope end --

@cockroach-teamcity
Copy link
Member Author

kv/kvserver.TestProtectedTimestamps failed with artifacts on release-22.2 @ 8a2050b00bac7e247df26c63aa0c8c1ca4fd7996:

=== RUN   TestProtectedTimestamps
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/33e1d369c27b9c01b2b6009c561815a3/logTestProtectedTimestamps26030002
    test_log_scope.go:79: use -show-logs to present logs inline
*
* INFO: Running test with the default test tenant. If you are only seeing a test case failure when this message appears, there may be a problem with your test case running within tenants.
*
    client_protectedts_test.go:267: 
        	Error Trace:	/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2800/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_test_/kvserver_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvserver/client_protectedts_test.go:267
        	            				/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2800/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_test_/kvserver_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvserver/soon.go:69
        	            				/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2800/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_test_/kvserver_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvserver/retry.go:213
        	            				/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2800/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_test_/kvserver_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvserver/soon.go:75
        	            				/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2800/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_test_/kvserver_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvserver/soon.go:57
        	            				/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2800/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_test_/kvserver_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvserver/soon.go:40
        	            				/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2800/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_test_/kvserver_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvserver/client_protectedts_test.go:260
        	Error:      	Should be true
        	Test:       	TestProtectedTimestamps
        	Messages:   	1686568838.400605959,0 >= 1686568838.400605958,2147483647
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/33e1d369c27b9c01b2b6009c561815a3/logTestProtectedTimestamps26030002
--- FAIL: TestProtectedTimestamps (6.53s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@arulajmani arulajmani mentioned this issue Jun 27, 2023
Merged
craig bot pushed a commit that referenced this issue Jun 27, 2023
@arulajmani
Merge #105383 #105641
3c2d5a4 
105383: concurrency: split out replicated/unreplicated lock holder information r=nvanbenschoten a=arulajmani

The MVCC keyspace is the source of truth for replicated locks. We've
previously concluded that trying to keep the in-memory state of
replicated locks in sync is fraught with subtle issues. We instead do
the dumb thing and forget replicated locks in a few places (see comment
about mvccResolveWriteIntent in tryUpdateLockLocked). It follows that
we don't need to track as much information about replicated locks
(like we do for unreplicated locks). For example, we do not need to
track the sequence number history for replicated locks.

This patch splits out lock holder information about replicated locks and
unreplicated locks into 2 different structs. As mentioned above, we no
longer track seqeunce numbers for the former.

Informs #102270

Release note: None

105641: kvserver: deflake TestProtectedTimestamps r=irfansharif a=arulajmani

This patch fixes a few (hopefully all) issues with TestProtectedTimestamps. In particular,

- The range max bytes used by the test was broken after the lower bound was bumped in a37e053. We up the value.
- There was flakiness at various points in the test as a result of lease transfers. We change the test to run on a single node test cluster to get around this.

Fixes: #93497

Release note: None

Co-authored-by: Arul Ajmani <[email protected]>
@craig craig bot closed this as completed in 0147fcc Jun 27, 2023
@blathers-crl blathers-crl bot mentioned this issue Jun 27, 2023
Merged
blathers-crl bot pushed a commit that referenced this issue Jun 27, 2023
@arulajmani
kvserver: deflake TestProtectedTimestamps
32fd7e1 
This patch fixes a few (hopefully all) issues with
TestProtectedTimestamps. In particular,

- The range max bytes used by the test was broken after
the lower bound was bumped in a37e053.
We up the value.
- There was flakiness at various points in the test as a result of lease
transfers. We change the test to run on a single node test cluster to
get around this.

Fixes: #93497

Release note: None
@blathers-crl blathers-crl bot mentioned this issue Jun 27, 2023
Merged
blathers-crl bot pushed a commit that referenced this issue Jun 27, 2023
@arulajmani
kvserver: deflake TestProtectedTimestamps
2ec3584 
This patch fixes a few (hopefully all) issues with
TestProtectedTimestamps. In particular,

- The range max bytes used by the test was broken after
the lower bound was bumped in a37e053.
We up the value.
- There was flakiness at various points in the test as a result of lease
transfers. We change the test to run on a single node test cluster to
get around this.

Fixes: #93497

Release note: None
@blathers-crl blathers-crl bot mentioned this issue Jun 27, 2023
Closed
blathers-crl bot pushed a commit that referenced this issue Jun 27, 2023
@arulajmani
kvserver: deflake TestProtectedTimestamps
cf6eb79 
This patch fixes a few (hopefully all) issues with
TestProtectedTimestamps. In particular,

- The range max bytes used by the test was broken after
the lower bound was bumped in a37e053.
We up the value.
- There was flakiness at various points in the test as a result of lease
transfers. We change the test to run on a single node test cluster to
get around this.

Fixes: #93497

Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@arulajmani arulajmani

Labels
branch-master Failures and bugs on the master branch. branch-release-22.2 Used to mark GA and release blockers, technical advisories, and bugs for 22.2 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. skipped-test T-kv KV Team
Projects
None yet
Milestone
22.2
Development

Successfully merging a pull request may close this issue.

kvserver: deflake TestProtectedTimestamps arulajmani/cockroach
7 participants
@tbg @nvanbenschoten @arulajmani @irfansharif @cockroach-teamcity @DrewKimball @renatolabs