roachtest: acceptance/gossip/restart failed

[cockroach-teamcity]

(roachtest).acceptance/gossip/restart failed on master@425eaa8fb05fc32b2c42827b85338daa52f4177c:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: artifacts/acceptance/gossip/restart/run_1
	gossip.go:225,gossip.go:290,acceptance.go:91,test_runner.go:753: pq: result is ambiguous (error=unable to dial n3: breaker open [exhausted])

More

Artifacts: /acceptance/gossip/restart
Related:

• roachtest: acceptance/gossip/restart-node-one failed #47960 roachtest: acceptance/gossip/restart-node-one failed C-test-failure O-roachtest O-robot branch-release-19.2 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

Jira issue: CRDB-4327

[tbg]

I think the statement that failed is

cockroach/pkg/cmd/roachtest/gossip.go

Lines 224 to 226 in f80ec9c

	if _, err := db.Exec(`UPSERT INTO test.kv (k, v) VALUES (1, 0)`); err != nil {
	t.Fatal(err)
	}

I assume we're getting an ambiguous result all the way to the client because the upsert decomposes into a read-write cycle... somehow? But this should never go back to the client in this case, no?

cc @asubiotto

[asubiotto]

I think the optimizer plans a check separately from the actual upsert operation. I'm not very familiar with this code, but why do you think this shouldn't go back to the client if the transport is exhausted? Based on the ClientVisibleAmbiguousError interface, looks like we do expect to return these to the client sometimes.

[andreimatei]

@tbg what were you thinking here?

[tbg]

Oh - looks like I wasn't thinking at all. Sorry about the noise. I guess my thinking was that this particular UPSERT is idempotent, and so it should never have to return an ambiguous result. But at the same time I know that we're not smart enough to do that (and it may not even be desirable, not sure).

Unclear what exactly to do with that failure. Do we really need to return an ambiguous result when a breaker is open? We didn't even attempt to send.

[cockroach-teamcity]

(roachtest).acceptance/gossip/restart failed on master@436a82a518b142c3a3212f49bb595956b57ac68c:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: artifacts/acceptance/gossip/restart/run_1
	cluster.go:2138,cluster.go:2157,cluster.go:2261,gossip.go:215,gossip.go:290,acceptance.go:94,test_runner.go:753: pgurl for nodes [] empty: :1 from
		stdout:
		
		stderr:

More

Artifacts: /acceptance/gossip/restart
Related:

• roachtest: acceptance/gossip/restart-node-one failed #47960 roachtest: acceptance/gossip/restart-node-one failed C-test-failure O-roachtest O-robot branch-release-19.2 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

[knz]

looked at this in this round of triaging - the latest failure is still with the same symptoms.

[tbg]

Looking at this again as part of triage. @asubiotto I think there is something for SQL to do here yet - I believe the error comes from

cockroach/pkg/sql/colflow/colrpc/outbox.go

Lines 151 to 159 in 7345931

	conn, err := dialer.Dial(ctx, nodeID, rpc.DefaultClass)
	if err != nil {
	log.Warningf(
	ctx,
	"Outbox Dial connection error, distributed query will fail: %+v",
	err,
	)
	return err
	}

I may be wrong- but it seems to come from SQL, it's this one here:

cockroach/pkg/rpc/nodedialer/nodedialer.go

Lines 142 to 155 in f80ec9c

	func (n *Dialer) dial(
	ctx context.Context,
	nodeID roachpb.NodeID,
	addr net.Addr,
	breaker *wrappedBreaker,
	class rpc.ConnectionClass,
	) (_ *grpc.ClientConn, err error) {
	// Don't trip the breaker if we're already canceled.
	if ctxErr := ctx.Err(); ctxErr != nil {
	return nil, ctxErr
	}
	if breaker != nil && !breaker.Ready() {
	err = errors.Wrapf(circuit.ErrBreakerOpen, "unable to dial n%d", nodeID)
	return nil, err

This error should not become ambiguous since no connection attempt has been made (also, this is in the read portion of the query? Maybe I am wrong about the callsite here).

If we see this again, we should grab the logs before they're gone to make sure.

[cockroach-teamcity]

(roachtest).acceptance/gossip/restart failed on master@e6c1a2abe0bb9008d904aad0f23eeff9ef217430:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: artifacts/acceptance/gossip/restart/run_1
	gossip.go:225,gossip.go:290,acceptance.go:96,test_runner.go:754: pq: result is ambiguous (error=unable to dial n4: breaker open [exhausted])

More

Artifacts: /acceptance/gossip/restart

See this test on roachdash
_{powered by pkg/cmd/internal/issues}