server: TestStatusEngineStatsJson failed #99261

cockroach-teamcity · 2023-03-22T17:17:49Z

server.TestStatusEngineStatsJson failed with artifacts on release-23.1 @ dc054b59d684e421238d7044539463df051186b7:

=== RUN   TestStatusEngineStatsJson
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/74885ac1e7f972ae9e97608ed3fdb458/logTestStatusEngineStatsJson1821925975
    test_log_scope.go:79: use -show-logs to present logs inline
    status_test.go:282: status: 503 Service Unavailable, content-type: application/json, body: {
          "error": "connection error: desc = \"transport: error while dialing: connection interrupted (did the remote node shut down or are there networking issues?)\"",
          "code": 14,
          "message": "connection error: desc = \"transport: error while dialing: connection interrupted (did the remote node shut down or are there networking issues?)\"",
          "details": [
          ]
        }, error: <nil>
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/74885ac1e7f972ae9e97608ed3fdb458/logTestStatusEngineStatsJson1821925975
--- FAIL: TestStatusEngineStatsJson (15.37s)

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

server: TestStatusEngineStatsJson failed #95491 server: TestStatusEngineStatsJson failed [A-observability-inf C-test-failure O-robot T-observability-inf branch-master]

/cc @cockroachdb/obs-inf-prs @cockroachdb/server _{This test on roachdash | Improve this report!

Jira issue: CRDB-25791

Epic: CRDB-28893}

The text was updated successfully, but these errors were encountered:

knz · 2023-04-03T12:56:05Z

This continues to flake on master and 23.1 despite my attempt to make it better in #100115 and #100284.

knz · 2023-04-03T12:56:35Z

Example failure https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_UnitTests_BazelUnitTests/9401853?buildTab=overview&showRootCauses=false&expandBuildProblemsSection=true&expandBuildTestsSection=true&expandBuildChangesSection=true#%2Ftmp%2F_tmp%2Fb0e29eaabc4b10fcc0559880c50b11bd

99934: changefeedccl: Remove skipped tests that decayed over time r=miretskiy a=miretskiy Remove Fixes #32232 Remove TestChangefeedNodeShutdown. This test has been disabled since 2018; Other tests exist (e.g. `TestChangefeedHandlesDrainingNodes`) that verify restart behavior. Fixes #51842 Remove BenchmarkChangefeedTicks benchmark. This benchmark has been skipped since 2019. Attempts could be made to revive it; however, this benchmark had a lot of code, which accomplished questionable goals. The benchmark itself was unrepresentative (by using dependency injection), too small to be meaningful (1000 rows), and most likely would be too noise and inconclusive. We have added other micro benchmarks over time; and we conduct large scale testing, including with roachtests. Release note: None 100342: upgrades: remove migration that waits for schema changes r=rafiss a=rafiss We can also remove some skipped tests, since they no longer apply. informs: #96751 Release note: None 100345: upgrades: unskip TestIsAtLeastVersionBuiltin r=rafiss a=rafiss informs: #96751 Release note: None 100484: server,testutils: add some extra logging for TestStatusEngineStatsJson r=abarganier a=knz Informs #99261 Release note: None Epic: None Co-authored-by: Yevgeniy Miretskiy <[email protected]> Co-authored-by: Rafi Shamim <[email protected]> Co-authored-by: Raphael 'kena' Poss <[email protected]>

ajwerner · 2023-04-05T20:03:37Z

https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_BazelEssentialCi/9455763?showRootCauses=false&expandBuildChangesSection=true&expandBuildProblemsSection=true&expandBuildTestsSection=true

cockroach-teamcity · 2023-04-20T03:08:01Z

server.TestStatusEngineStatsJson failed with artifacts on master @ 568c68a5b2ba48ba08cd69c96876dd8ac25ade3c:

=== RUN   TestStatusEngineStatsJson
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/38c80cbed3f79a398570824e9fcc636b/logTestStatusEngineStatsJson2636581669
    test_log_scope.go:79: use -show-logs to present logs inline
    status_test.go:279: using admin URL https://127.0.0.1:44241
    status_test.go:286: condition failed to evaluate within 45s: status: 503 Service Unavailable, content-type: application/json, body: {
          "error": "connection error: desc = \"transport: error while dialing: gRPC connection unexpectedly re-dialed: connection interrupted (did the remote node shut down or are there networking issues?)\"",
          "code": 14,
          "message": "connection error: desc = \"transport: error while dialing: gRPC connection unexpectedly re-dialed: connection interrupted (did the remote node shut down or are there networking issues?)\"",
          "details": [
          ]
        }, error: <nil>
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/38c80cbed3f79a398570824e9fcc636b/logTestStatusEngineStatsJson2636581669
--- FAIL: TestStatusEngineStatsJson (84.09s)

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-04-24T10:47:33Z

server.TestStatusEngineStatsJson failed with artifacts on master @ 3c23346a6d1c91dfe8ed1c5285966f9b9e487601:

=== RUN   TestStatusEngineStatsJson
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/04ff595e8b8a4cdf61532aa9937d7edf/logTestStatusEngineStatsJson3168219894
    test_log_scope.go:79: use -show-logs to present logs inline
    status_test.go:279: using admin URL https://127.0.0.1:36583
    status_test.go:286: condition failed to evaluate within 45s: status: 503 Service Unavailable, content-type: application/json, body: {
          "error": "connection error: desc = \"transport: error while dialing: gRPC connection unexpectedly re-dialed: connection interrupted (did the remote node shut down or are there networking issues?)\"",
          "code": 14,
          "message": "connection error: desc = \"transport: error while dialing: gRPC connection unexpectedly re-dialed: connection interrupted (did the remote node shut down or are there networking issues?)\"",
          "details": [
          ]
        }, error: <nil>
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/04ff595e8b8a4cdf61532aa9937d7edf/logTestStatusEngineStatsJson3168219894
--- FAIL: TestStatusEngineStatsJson (87.12s)

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-04-28T00:35:44Z

server.TestStatusEngineStatsJson failed with artifacts on master @ ddc247fd1fa4c7c1d2f212fcdd101ef200b772ff:

=== RUN   TestStatusEngineStatsJson
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/04ff595e8b8a4cdf61532aa9937d7edf/logTestStatusEngineStatsJson2340365415
    test_log_scope.go:79: use -show-logs to present logs inline
    status_test.go:279: using admin URL https://127.0.0.1:43193
    status_test.go:286: condition failed to evaluate within 45s: status: 503 Service Unavailable, content-type: application/json, body: {
          "error": "connection error: desc = \"transport: error while dialing: gRPC connection unexpectedly re-dialed: connection interrupted (did the remote node shut down or are there networking issues?)\"",
          "code": 14,
          "message": "connection error: desc = \"transport: error while dialing: gRPC connection unexpectedly re-dialed: connection interrupted (did the remote node shut down or are there networking issues?)\"",
          "details": [
          ]
        }, error: <nil>
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/04ff595e8b8a4cdf61532aa9937d7edf/logTestStatusEngineStatsJson2340365415
--- FAIL: TestStatusEngineStatsJson (109.60s)

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-04-28T16:17:33Z

server.TestStatusEngineStatsJson failed with artifacts on release-23.1 @ 1f41dc622e53e7f2dcf27d80ed808a47605510b8:

=== RUN   TestStatusEngineStatsJson
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/38c80cbed3f79a398570824e9fcc636b/logTestStatusEngineStatsJson2954868437
    test_log_scope.go:79: use -show-logs to present logs inline
    status_test.go:284: condition failed to evaluate within 45s: status: 503 Service Unavailable, content-type: application/json, body: {
          "error": "connection error: desc = \"transport: error while dialing: gRPC connection unexpectedly re-dialed: connection interrupted (did the remote node shut down or are there networking issues?)\"",
          "code": 14,
          "message": "connection error: desc = \"transport: error while dialing: gRPC connection unexpectedly re-dialed: connection interrupted (did the remote node shut down or are there networking issues?)\"",
          "details": [
          ]
        }, error: <nil>
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/38c80cbed3f79a398570824e9fcc636b/logTestStatusEngineStatsJson2954868437
--- FAIL: TestStatusEngineStatsJson (100.05s)

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-05-06T06:27:04Z

server.TestStatusEngineStatsJson failed with artifacts on master @ 5d775e992ade315becd6b4fc41ed1fb85d8e35d6:

=== RUN   TestStatusEngineStatsJson
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/115b56b7b7bd2dfb6f4ea1f5f6513edb/logTestStatusEngineStatsJson1398007110
    test_log_scope.go:79: use -show-logs to present logs inline
    status_test.go:279: using admin URL https://127.0.0.1:35639
    status_test.go:286: condition failed to evaluate within 45s: status: 503 Service Unavailable, content-type: application/json, body: {
          "error": "connection error: desc = \"transport: error while dialing: gRPC connection unexpectedly re-dialed: connection interrupted (did the remote node shut down or are there networking issues?)\"",
          "code": 14,
          "message": "connection error: desc = \"transport: error while dialing: gRPC connection unexpectedly re-dialed: connection interrupted (did the remote node shut down or are there networking issues?)\"",
          "details": [
          ]
        }, error: <nil>
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/115b56b7b7bd2dfb6f4ea1f5f6513edb/logTestStatusEngineStatsJson1398007110
--- FAIL: TestStatusEngineStatsJson (56.25s)

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-05-08T23:34:39Z

server.TestStatusEngineStatsJson failed with artifacts on master @ 439c515b2a0058648731da73993c409544404da1:

=== RUN   TestStatusEngineStatsJson
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/dbd7076bfe20e3183065a44fa4f7e406/logTestStatusEngineStatsJson483250565
    test_log_scope.go:79: use -show-logs to present logs inline
    status_test.go:280: using admin URL https://127.0.0.1:41589
    status_test.go:287: condition failed to evaluate within 45s: status: 503 Service Unavailable, content-type: application/json, body: {
          "error": "connection error: desc = \"transport: error while dialing: gRPC connection unexpectedly re-dialed: connection interrupted (did the remote node shut down or are there networking issues?)\"",
          "code": 14,
          "message": "connection error: desc = \"transport: error while dialing: gRPC connection unexpectedly re-dialed: connection interrupted (did the remote node shut down or are there networking issues?)\"",
          "details": [
          ]
        }, error: <nil>
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/dbd7076bfe20e3183065a44fa4f7e406/logTestStatusEngineStatsJson483250565
--- FAIL: TestStatusEngineStatsJson (73.18s)

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-05-10T20:27:21Z

server.TestStatusEngineStatsJson failed with artifacts on master @ 09ec0ee7c658bc139adab023fca8a61f1cd6789f:

=== RUN   TestStatusEngineStatsJson
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/dbd7076bfe20e3183065a44fa4f7e406/logTestStatusEngineStatsJson835366702
    test_log_scope.go:79: use -show-logs to present logs inline
    status_test.go:280: using admin URL https://127.0.0.1:36079
    status_test.go:287: condition failed to evaluate within 45s: status: 503 Service Unavailable, content-type: application/json, body: {
          "error": "connection error: desc = \"transport: error while dialing: gRPC connection unexpectedly re-dialed: connection interrupted (did the remote node shut down or are there networking issues?)\"",
          "code": 14,
          "message": "connection error: desc = \"transport: error while dialing: gRPC connection unexpectedly re-dialed: connection interrupted (did the remote node shut down or are there networking issues?)\"",
          "details": [
          ]
        }, error: <nil>
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/dbd7076bfe20e3183065a44fa4f7e406/logTestStatusEngineStatsJson835366702
--- FAIL: TestStatusEngineStatsJson (73.25s)

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-05-17T06:29:42Z

server.TestStatusEngineStatsJson failed with artifacts on master @ 76078816b69ee6d1f8dacd581ef464660fb1b697:

=== RUN   TestStatusEngineStatsJson
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/115b56b7b7bd2dfb6f4ea1f5f6513edb/logTestStatusEngineStatsJson3831849725
    test_log_scope.go:79: use -show-logs to present logs inline
    status_test.go:280: using admin URL https://127.0.0.1:35511
    status_test.go:287: condition failed to evaluate within 45s: status: 503 Service Unavailable, content-type: application/json, body: {
          "error": "connection error: desc = \"transport: error while dialing: gRPC connection unexpectedly re-dialed: connection interrupted (did the remote node shut down or are there networking issues?)\"",
          "code": 14,
          "message": "connection error: desc = \"transport: error while dialing: gRPC connection unexpectedly re-dialed: connection interrupted (did the remote node shut down or are there networking issues?)\"",
          "details": [
          ]
        }, error: <nil>
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/115b56b7b7bd2dfb6f4ea1f5f6513edb/logTestStatusEngineStatsJson3831849725
--- FAIL: TestStatusEngineStatsJson (64.35s)

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

andrewbaptist · 2023-05-17T13:52:51Z

Hit here: https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_BazelEssentialCi/10113825

Refs: cockroachdb#99261 Reason: flaky test Generated by bin/skip-test. Release justification: non-production code changes Release note: None Epic: None

103624: server: skip TestStatusEngineStatsJson r=smg260 a=smg260 Refs: #99261 Reason: flaky test Generated by bin/skip-test. Release justification: non-production code changes Release note: None Epic: None Co-authored-by: Miral Gadani <[email protected]>

tbg · 2023-05-19T18:03:37Z

I've been poking at this test a bit. So far I got to the point where I'm certain that the EngineStats endpoint isn't even reached. So I think the problem is somewhere in the http-rpc bridge. I'll try to remember how this all works and printf-debug my way through...

tbg · 2023-05-19T18:12:28Z

Ah, I see that we are setting up the dialer for grpc-gateway here:

cockroach/pkg/server/grpc_gateway.go

Lines 80 to 85 in b8518eb

    
           // Eschew `(*rpc.Context).GRPCDial` to avoid unnecessary moving parts on the 
        
           // uniquely in-process connection. 
        
           dialOpts, err := rpcContext.GRPCDialOptions(ctx, GRPCAddr, rpc.DefaultClass) 
        
           if err != nil { 
        
           	return nil, nil, nil, err 
        
           }

It uses GRPCDialOptions, which also puts in a onlyOnceDialer - but I don't think we want that here... under stress, we can probably fail to dial due to some timeout. And then the onlyOnceDialer doesn't allow reconnections, so this connection will be permanently broken.

tbg · 2023-05-19T18:59:14Z

Yeah, I think that's it. I changed the above call to not use the onlyOnceDialer by classifying it as loopbackTransport¹

@knz could you chime in on whether that's the semantically correct fix? It does seem like that to me but there might be nuance that I'm missing.

https://github.com/cockroachdb/cockroach/blob/3d3910b15061844ff809bcb6fa8c0ca52f8fcf95/pkg/rpc/context.go#L1631-L1637 ↩

cockroach-teamcity · 2023-05-22T17:04:16Z

server.TestStatusEngineStatsJson failed with artifacts on release-23.1 @ 128261565cccf92b771975db161ceda9340a659a:

=== RUN   TestStatusEngineStatsJson
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/dbd7076bfe20e3183065a44fa4f7e406/logTestStatusEngineStatsJson1794869553
    test_log_scope.go:79: use -show-logs to present logs inline
    status_test.go:284: condition failed to evaluate within 45s: status: 503 Service Unavailable, content-type: application/json, body: {
          "error": "connection error: desc = \"transport: error while dialing: gRPC connection unexpectedly re-dialed: connection interrupted (did the remote node shut down or are there networking issues?)\"",
          "code": 14,
          "message": "connection error: desc = \"transport: error while dialing: gRPC connection unexpectedly re-dialed: connection interrupted (did the remote node shut down or are there networking issues?)\"",
          "details": [
          ]
        }, error: <nil>
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/dbd7076bfe20e3183065a44fa4f7e406/logTestStatusEngineStatsJson1794869553
--- FAIL: TestStatusEngineStatsJson (62.62s)

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

rafiss · 2023-06-27T15:46:24Z

reopening as the test is still skipped

cockroach-teamcity added branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Mar 22, 2023

cockroach-teamcity added this to the 23.1 milestone Mar 22, 2023

blathers-crl bot added T-observability-inf A-observability-inf labels Mar 22, 2023

cockroach-teamcity mentioned this issue Mar 30, 2023

server: TestStatusEngineStatsJson failed #95491

Closed

knz added the branch-master Failures and bugs on the master branch. label Apr 3, 2023

knz mentioned this issue Apr 3, 2023

server,testutils: add some extra logging for TestStatusEngineStatsJson #100484

Merged

dhartunian added the GA-blocker label Apr 12, 2023

dhartunian self-assigned this Apr 21, 2023

dhartunian removed the GA-blocker label Apr 25, 2023

sumeerbhola mentioned this issue May 10, 2023

allocator2: add generic clearableMemoMap #102420

Merged

adityamaru mentioned this issue May 15, 2023

backupccl,kvserver: log failed ExportRequest trace on client and server #102793

Merged

kvoli mentioned this issue May 17, 2023

asim: simulate recovery actions #101745

Merged

smg260 mentioned this issue May 18, 2023

server: skip TestStatusEngineStatsJson #103624

Merged

smg260 pushed a commit to smg260/cockroach that referenced this issue May 18, 2023

server: skip TestStatusEngineStatsJson

07bcdb3

Refs: cockroachdb#99261 Reason: flaky test Generated by bin/skip-test. Release justification: non-production code changes Release note: None Epic: None

smg260 added the skipped-test label May 19, 2023

tbg mentioned this issue May 19, 2023

grpcutil: improve gRPC logging #103387

Merged

tbg mentioned this issue May 23, 2023

roachtest: admission-control/tpcc-olap/nodes=3/cpu=8/w=50/c=96 failed #103692

Closed

This was referenced May 23, 2023

rpc: grpc-gateway loopback conn mistakenly uses onlyOnceDialer and causes sticky permanent RPC errors #103762

Closed

rpc: use the loopback conn also for GRPCDialOptions #103764

Merged

exalate-issue-sync bot assigned knz and unassigned dhartunian May 23, 2023

craig bot closed this as completed in 741c91b May 23, 2023

blathers-crl bot mentioned this issue May 23, 2023

release-23.1: rpc: use the loopback conn also for GRPCDialOptions #103775

Merged

rafiss reopened this Jun 27, 2023

knz assigned knz and unassigned knz Jun 27, 2023

knz mentioned this issue Jun 27, 2023

server: unskip TestStatusEngineStatsJson #105629

Merged

craig bot closed this as completed in 08343f5 Jun 27, 2023

blathers-crl bot mentioned this issue Jun 27, 2023

release-23.1: server: unskip TestStatusEngineStatsJson #105637

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

server: TestStatusEngineStatsJson failed #99261

server: TestStatusEngineStatsJson failed #99261

cockroach-teamcity commented Mar 22, 2023 •

edited by knz

Loading

knz commented Apr 3, 2023

knz commented Apr 3, 2023

ajwerner commented Apr 5, 2023

cockroach-teamcity commented Apr 20, 2023

cockroach-teamcity commented Apr 24, 2023

cockroach-teamcity commented Apr 28, 2023

cockroach-teamcity commented Apr 28, 2023

cockroach-teamcity commented May 6, 2023

cockroach-teamcity commented May 8, 2023

cockroach-teamcity commented May 10, 2023

cockroach-teamcity commented May 17, 2023

andrewbaptist commented May 17, 2023

tbg commented May 19, 2023

tbg commented May 19, 2023

tbg commented May 19, 2023

cockroach-teamcity commented May 22, 2023

rafiss commented Jun 27, 2023

server: TestStatusEngineStatsJson failed #99261

server: TestStatusEngineStatsJson failed #99261

Comments

cockroach-teamcity commented Mar 22, 2023 • edited by knz Loading

knz commented Apr 3, 2023

knz commented Apr 3, 2023

ajwerner commented Apr 5, 2023

cockroach-teamcity commented Apr 20, 2023

cockroach-teamcity commented Apr 24, 2023

cockroach-teamcity commented Apr 28, 2023

cockroach-teamcity commented Apr 28, 2023

cockroach-teamcity commented May 6, 2023

cockroach-teamcity commented May 8, 2023

cockroach-teamcity commented May 10, 2023

cockroach-teamcity commented May 17, 2023

andrewbaptist commented May 17, 2023

tbg commented May 19, 2023

tbg commented May 19, 2023

tbg commented May 19, 2023

Footnotes

cockroach-teamcity commented May 22, 2023

rafiss commented Jun 27, 2023

cockroach-teamcity commented Mar 22, 2023 •

edited by knz

Loading