distsql: better handling of failures and dead nodes

[RaduBerinde]

In 1.0, we avoid hosts that known to be dead, but we may fail the first attempt of a query if we don't know the host is dead.

This issue tracks implementing better handling of these situations. The idea is to support multiple flows for the same query on a single node, and to reschedule flows (which we failed to schedule) on the gateway node.

One important aspect of "error" in this discussion is version mismatch: if we can reschedule flows that can't run on a host because of its version, we can provide query availability (with decreased performance) during an upgrade even when there are incompatible changes in distsql.

[cuongdo]

how is this looking for 1.1?

[RaduBerinde]

We have a good idea of what we want to do here but nothing is implemented at this point.

[rjnn]

This is being addressed by #17497, which should merge in the 1.1 timeframe.

[andreimatei]

The version mismatched problem has been address some. The node failed recently case remains open.

[vivekmenezes]

Probably obvious but this is a high priority issue for us to fix.

[vivekmenezes]

radu and andrei are thinking about this problem and trying to figure out what to do in the next release.

[nstewart]

This issue is preventing me from doing standard reporting. Is there an update? I'm getting psycopg2.InternalError: [n3] communication error: rpc error: code = Canceled desc = context canceled and I see in my 5 node cluster a couple nodes restarted. Happy to create a new ticket but I see #19882 was closed in favor of this one.

[jordanlewis]

@nstewart, does the query complete if you try it again? Or are you never able to complete your query?

[nstewart]

My script runs ~10 queries serially. Each query runs for about several minutes (but not hours). I've run the script several times over the weekend and it fails at different queries each time with context canceled. So it looks like retrying does work in some cases.

I'm betting after I add retry loops this thing will eventually complete (I'm going to do this in the short term so I'm not blocked), but since these aren't serializable retries and there is only one, single-threaded client for this database, I wonder if that would be masking a bigger problem.

[jordanlewis]

Yes, surely doing that would be masking a problem. During your script's run, are nodes restarting a lot?

[nstewart]

Yes, I am running a 5 node cluster and I saw 1 node restart once and another restart 4 times. I saved the debug zip here: #28278

[github-actions]

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
10 days to keep the issue queue tidy. Thank you for your contribution
to CockroachDB!