Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kv/kvserver: TestNewRangefeedForceLeaseRetry failed #131808

Closed
cockroach-teamcity opened this issue Oct 3, 2024 · 2 comments · Fixed by #131845
Closed

kv/kvserver: TestNewRangefeedForceLeaseRetry failed #131808

cockroach-teamcity opened this issue Oct 3, 2024 · 2 comments · Fixed by #131845
Assignees
@iskettaneh
Labels
A-testing Testing tools and infrastructure branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 branch-release-24.2 Used to mark GA and release blockers, technical advisories, and bugs for 24.2 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-3 Issues/test failures with no fix SLA
Milestone
24.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Oct 3, 2024 •
edited by cockroach-jira-scripts
Loading

kv/kvserver.TestNewRangefeedForceLeaseRetry failed on release-24.1 @ 25fab96bed932edb16c5e32e830fad62ec7383b5:

=== RUN   TestNewRangefeedForceLeaseRetry
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestNewRangefeedForceLeaseRetry4045324648
    test_log_scope.go:81: use -show-logs to present logs inline
    replica_rangefeed_test.go:1688: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/kv/kvserver_test/pkg/kv/kvserver/replica_rangefeed_test.go:1688
        	Error:      	Received unexpected error:
        	            	liveness record changed while incrementing epoch for liveness(nid:2 epo:1 exp:1727929763.238261911,0); actual is liveness(nid:2 epo:1 exp:1727929763.134542165,0); is the node still live?
        	            	(1) liveness record changed while incrementing epoch for liveness(nid:2 epo:1 exp:1727929763.238261911,0); actual is liveness(nid:2 epo:1 exp:1727929763.134542165,0); is the node still live?
        	            	Error types: (1) *liveness.ErrEpochCondFailed
        	Test:       	TestNewRangefeedForceLeaseRetry
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestNewRangefeedForceLeaseRetry4045324648
--- FAIL: TestNewRangefeedForceLeaseRetry (0.52s)

Parameters:

  • attempt=1
  • run=25
  • shard=2
Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/replication

This test on roachdash | Improve this report!

Jira issue: CRDB-42896
@cockroach-teamcity cockroach-teamcity added branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv-replication labels Oct 3, 2024
@cockroach-teamcity cockroach-teamcity added this to the 24.1 milestone Oct 3, 2024
@kvoli
Copy link
Collaborator

kvoli commented Oct 3, 2024

@iskettaneh should we backport your fix in #123551 (comment) to 24.1 as well? This issue appears the same.

@kvoli kvoli added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. A-testing Testing tools and infrastructure P-3 Issues/test failures with no fix SLA and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Oct 3, 2024
@iskettaneh iskettaneh mentioned this issue Oct 3, 2024
Closed
@iskettaneh iskettaneh mentioned this issue Oct 3, 2024
Merged
craig bot pushed a commit that referenced this issue Oct 4, 2024
@fqazi @iskettaneh @tbg
Merge #131797 #131845 #131905
42ed8a3 
131797: workload/schemachanger: address flakes and stabilize this test r=fqazi a=fqazi

This patch will address the following flakes:

1. Allow constraint violations on add column / alter pk, since concurrent inserts can cause these errors on commit
2. Fix aborted txn error inside ADD FOREIGN KEY, which was because one of the intropection queries was never run inside a child txn.
3. Address command is too large error in insert, which could cause inserts to fail when a large number of columns existed on a table.
4. Address a bug inside INSERT foreign key validation when multiple rows were inserted in the same batch
5. Handle crdb_internal computed cols in colIsRefByComputed, which removes a potential error from DROP COLUMN

Fixes: #131345
Fixes: #130923
Fixes: #126967

131845: kvserver: deflake TestNewRangefeedForceLeaseRetry r=iskettaneh a=iskettaneh

This commit lets the test waits for N1's view of N2's lease expiration to match N2's view. This is important in the rare case where N1 tries to increase N2's epoch, but it has a stale view of the lease expiration time.

Fixes: #131808

Release note: None

131905: cli: add --user flag to client cmds r=tbg a=tbg

I originally sent #130827 in reaction to finding that `./cockroach gen
haproxy` didn't work with SQL urls that used client certs for a non-root
user (it would erroneously expect to be pointed at the root client certs).

That PR caused problems too; now one needed to specify `--certs` even
though the certs were in the URL. There's a fix for that too (#131894)
but it all seems pretty tangled up.

This PR takes a more straightforward route: we revert #130827 and add
the `--user` flag to all client commands (who all already get the
`--sql` flag.

This should have fewer unintended consequences, and solves the problem.

Fixes #131802.
Fixes #131812.
Fixes #131814.
Fixes #131815.
Fixes #131816.
Fixes #131817.

Epic: none
Release note: None



Co-authored-by: Faizan Qazi <[email protected]>
Co-authored-by: Ibrahim Kettaneh <[email protected]>
Co-authored-by: Tobias Grieger <[email protected]>
@craig craig bot closed this as completed in 637c9c8 Oct 4, 2024
@blathers-crl blathers[crl]
Copy link

blathers-crl bot commented Oct 4, 2024

Based on the specified backports for linked PR #131845, I applied the following new label(s) to this issue: branch-release-23.1, branch-release-23.2, branch-release-24.2. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@blathers-crl blathers-crl bot added branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 branch-release-24.2 Used to mark GA and release blockers, technical advisories, and bugs for 24.2 labels Oct 4, 2024
blathers-crl bot pushed a commit that referenced this issue Oct 4, 2024
@iskettaneh
kvserver: deflake TestNewRangefeedForceLeaseRetry
4ed2e28 
This commit lets the test waits for N1's view of N2's lease expiration
to match N2's view. This is important in the rare case where N1 tries to
increase N2's epoch, but it has a stale view of the lease expiration
time.

Fixes: #131808

Release note: None
@blathers-crl blathers-crl bot mentioned this issue Oct 4, 2024
Merged
blathers-crl bot pushed a commit that referenced this issue Oct 4, 2024
@iskettaneh
kvserver: deflake TestNewRangefeedForceLeaseRetry
83501ca 
This commit lets the test waits for N1's view of N2's lease expiration
to match N2's view. This is important in the rare case where N1 tries to
increase N2's epoch, but it has a stale view of the lease expiration
time.

Fixes: #131808

Release note: None
@blathers-crl blathers-crl bot mentioned this issue Oct 4, 2024
Merged
blathers-crl bot pushed a commit that referenced this issue Oct 4, 2024
@iskettaneh
kvserver: deflake TestNewRangefeedForceLeaseRetry
3439745 
This commit lets the test waits for N1's view of N2's lease expiration
to match N2's view. This is important in the rare case where N1 tries to
increase N2's epoch, but it has a stale view of the lease expiration
time.

Fixes: #131808

Release note: None
@blathers-crl blathers-crl bot mentioned this issue Oct 4, 2024
Merged
blathers-crl bot pushed a commit that referenced this issue Oct 4, 2024
@iskettaneh
kvserver: deflake TestNewRangefeedForceLeaseRetry
810fbc3 
This commit lets the test waits for N1's view of N2's lease expiration
to match N2's view. This is important in the rare case where N1 tries to
increase N2's epoch, but it has a stale view of the lease expiration
time.

Fixes: #131808

Release note: None
@blathers-crl blathers-crl bot mentioned this issue Oct 4, 2024
Merged
@jlinder jlinder reopened this Oct 9, 2024
@jlinder jlinder closed this as completed Oct 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@iskettaneh iskettaneh

Labels
A-testing Testing tools and infrastructure branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 branch-release-24.2 Used to mark GA and release blockers, technical advisories, and bugs for 24.2 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-3 Issues/test failures with no fix SLA
Projects
None yet
Milestone
24.1
Development

Successfully merging a pull request may close this issue.

kvserver: deflake TestNewRangefeedForceLeaseRetry iskettaneh/cockroach
4 participants
@jlinder @cockroach-teamcity @kvoli @iskettaneh