rpc: panic in newPeer under stress

[yuzefovich]

On d9d421e when running ./dev test pkg/sql -f=TestRelocateNonVoters --stress --race --cpus=4 -- --test_env=COCKROACH_TEST_TENANT=true on the gceworker I got the following panic:

* ERROR: a panic has occurred!
* child [5 127.0.0.1:44713 default] already exists 
* (1) attached stack trace
*   -- stack trace: 
*   | runtime.gopanic
*   | › GOROOT/src/runtime/panic.go:890
*   | [...repeated from below...]
* Wraps: (2) assertion failure
* Wraps: (3) attached stack trace
*   -- stack trace: 
*   | github.com/cockroachdb/cockroach/pkg/util/metric/aggmetric.(*childSet).add
*   | › github.com/cockroachdb/cockroach/pkg/util/metric/aggmetric/agg_metric.go:107
*   | github.com/cockroachdb/cockroach/pkg/util/metric/aggmetric.(*AggGauge).AddChild
*   | › github.com/cockroachdb/cockroach/pkg/util/metric/aggmetric/gauge.go:87
*   | github.com/cockroachdb/cockroach/pkg/rpc.(*Metrics).acquire
*   | › github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/metrics.go:189
*   | github.com/cockroachdb/cockroach/pkg/rpc.(*Context).newPeer
*   | › github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/peer.go:178
*   | github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal
*   | › github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:2157
*   | github.com/cockroachdb/cockroach/pkg/rpc.(*Context).GRPCDialNode
*   | › github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:2114
*   | github.com/cockroachdb/cockroach/pkg/rpc/nodedialer.(*Dialer).dial
*   | › github.com/cockroachdb/cockroach/pkg/rpc/nodedialer/nodedialer.go:170
*   | github.com/cockroachdb/cockroach/pkg/rpc/nodedialer.(*Dialer).Dial
*   | › github.com/cockroachdb/cockroach/pkg/rpc/nodedialer/nodedialer.go:103
*   | github.com/cockroachdb/cockroach/pkg/sql.runnerRequest.run
*   | › github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:112
*   | github.com/cockroachdb/cockroach/pkg/sql.(*runnerCoordinator).init.func1
*   | › github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:152
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
*   | › github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484
*   | runtime.goexit
*   | › src/runtime/asm_amd64.s:1594
* Wraps: (4) child [5 127.0.0.1:44713 default] already exists 
* Error types: (1) *withstack.withStack (2) *assert.withAssertionFailure (3) *withstack.withStack (4) *errutil.leafError

Seems like this code was added recently in #99191.

Stack trace is here

Jira issue: CRDB-30494

[blathers-crl]

cc @cockroachdb/replication

[erikgrinaker]

This was supposed to have been fixed by #107697. Might be a different failure mode.

[blathers-crl]

Hi @erikgrinaker, please add branch-* labels to identify which branch(es) this release-blocker affects.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}