Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rpc: TestRemoteOffsetUnhealthy failed #116898

Closed
cockroach-teamcity opened this issue Dec 20, 2023 · 3 comments · Fixed by #121279
Closed

rpc: TestRemoteOffsetUnhealthy failed #116898

cockroach-teamcity opened this issue Dec 20, 2023 · 3 comments · Fixed by #121279
Assignees
@kvoli
Labels
A-testing Testing tools and infrastructure branch-master Failures and bugs on the master branch. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-3 Issues/test failures with no fix SLA T-kv KV Team
Milestone
24.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Dec 20, 2023 •
edited by jlinder
Loading

rpc.TestRemoteOffsetUnhealthy failed with artifacts on master @ 7a7bbffa8959c9004991602b9d124de365aa8a17:

=== RUN   TestRemoteOffsetUnhealthy
I231220 21:19:37.955065 402 2@rpc/peer.go:527  [rnode=2,raddr=127.0.0.1:44837,class=default,rpc] 19  connection is now healthy
I231220 21:19:37.969569 408 2@rpc/peer.go:527  [rnode=3,raddr=127.0.0.1:41535,class=default,rpc] 20  connection is now healthy
I231220 21:19:37.975695 97 2@rpc/peer.go:527  [rnode=4,raddr=127.0.0.1:34655,class=default,rpc] 21  connection is now healthy
I231220 21:19:37.982762 411 2@rpc/peer.go:527  [rnode=1,raddr=127.0.0.1:38141,class=default,rpc] 22  connection is now healthy
I231220 21:19:37.996518 440 2@rpc/peer.go:527  [rnode=3,raddr=127.0.0.1:41535,class=default,rpc] 23  connection is now healthy
I231220 21:19:38.003363 468 2@rpc/peer.go:527  [rnode=4,raddr=127.0.0.1:34655,class=default,rpc] 24  connection is now healthy
I231220 21:19:38.009764 499 2@rpc/peer.go:527  [rnode=1,raddr=127.0.0.1:38141,class=default,rpc] 25  connection is now healthy
I231220 21:19:38.015776 514 2@rpc/peer.go:527  [rnode=2,raddr=127.0.0.1:44837,class=default,rpc] 26  connection is now healthy
E231220 21:19:38.136548 521 2@rpc/peer.go:601  [rnode=4,raddr=127.0.0.1:34655,class=default,rpc] 27  failed connection attempt (last connected 0s ago): operation "conn heartbeat" timed out after 120ms (given timeout 100ms): grpc: context deadline exceeded [code 4/DeadlineExceeded]
    context_test.go:1172: initial connection heartbeat failed: operation "conn heartbeat" timed out after 120ms (given timeout 100ms): grpc: context deadline exceeded [code 4/DeadlineExceeded]
--- FAIL: TestRemoteOffsetUnhealthy (0.21s)
Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-34836
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv KV Team labels Dec 20, 2023
@cockroach-teamcity cockroach-teamcity added this to the 24.1 milestone Dec 20, 2023
@jayshrivastava
Copy link
Contributor

Another one https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_BazelEssentialCi/13384826?hideProblemsFromDependencies=false&hideTestsFromDependencies=false&expandBuildChangesSection=true&expandBuildTestsSection=true

@shralex shralex added the P-3 Issues/test failures with no fix SLA label Jan 12, 2024
@nvanbenschoten nvanbenschoten added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. A-testing Testing tools and infrastructure labels Jan 16, 2024
@nvanbenschoten nvanbenschoten mentioned this issue Jan 16, 2024
Closed
@arulajmani arulajmani removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Feb 5, 2024
@msbutler
Copy link
Collaborator

msbutler commented Feb 26, 2024 •
edited
Loading

another one, on 23.2 https://teamcity.cockroachdb.com/viewLog.html?buildId=14133134&buildTypeId=Cockroach_BazelEssentialCi

@msbutler
Copy link
Collaborator

Another one: https://teamcity.cockroachdb.com/viewLog.html?buildId=14339755&buildTypeId=Cockroach_BazelEssentialCi

could we skip this test until it is fixed?

@cockroach-teamcity cockroach-teamcity mentioned this issue Mar 27, 2024
Closed
@kvoli kvoli mentioned this issue Mar 28, 2024
Merged
@kvoli kvoli self-assigned this Mar 28, 2024
craig bot pushed a commit that referenced this issue Mar 28, 2024
@kvoli
Merge #121279
fbae232 
121279: rpc: expect eventual connection in remote offset unhealthy test r=arulajmani a=kvoli

`TestRemoteOffsetUnhealthy` manually connects a grid of peers to later assert on the remote offset between rpc connections. The initial connection setup can timeout under resource constraints.

Retry the initial connection when setting up the test.

Fixes: #121238
Fixes: #116898
Release note: None

Co-authored-by: Austen McClernon <[email protected]>
@craig craig bot closed this as completed in 2fe2f25 Mar 28, 2024
blathers-crl bot pushed a commit that referenced this issue Mar 28, 2024
@kvoli
rpc: expect eventual connection in remote offset unhealthy test
7e46ae8 
`TestRemoteOffsetUnhealthy` manually connects a grid of peers to later
assert on the remote offset between rpc connections. The initial
connection setup can timeout under resource constraints.

Retry the initial connection when setting up the test.

Fixes: #121238
Fixes: #116898
Release note: None
@blathers-crl blathers-crl bot mentioned this issue Mar 28, 2024
Merged
blathers-crl bot pushed a commit that referenced this issue Mar 28, 2024
@kvoli
rpc: expect eventual connection in remote offset unhealthy test
997a118 
`TestRemoteOffsetUnhealthy` manually connects a grid of peers to later
assert on the remote offset between rpc connections. The initial
connection setup can timeout under resource constraints.

Retry the initial connection when setting up the test.

Fixes: #121238
Fixes: #116898
Release note: None
@blathers-crl blathers-crl bot mentioned this issue Mar 28, 2024
Merged
@github-project-automation github-project-automation bot moved this to Closed in KV Aug 28, 2024
@miraradeva miraradeva mentioned this issue Sep 6, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kvoli kvoli

Labels
A-testing Testing tools and infrastructure branch-master Failures and bugs on the master branch. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-3 Issues/test failures with no fix SLA T-kv KV Team
Projects
No open projects
KV
Status: Closed
Milestone
24.1
Development

Successfully merging a pull request may close this issue.

rpc: expect eventual connection in remote offset unhealthy test kvoli/cockroach
7 participants
@nvanbenschoten @shralex @arulajmani @jayshrivastava @cockroach-teamcity @msbutler @kvoli