distsql: Nil pointer panic in PopulateEndpoints

[bdarnell]

https://sentry.io/cockroach-labs/cockroachdb/issues/565592900/

*log.safeError: conn_executor.go:521: panic while executing 1 statements: SELECT _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _ FROM ((SELECT _, _, _, _, _, _, _, _, _, _ FROM _._._ WHERE (_ = (-_)) AND (_ = _) ORDER BY _, _, _ LIMIT _) JOIN (SELECT _, _, _, _, _, _, _, _, _, _, _, _, _, _ FROM _._._ WHERE (_ = (-_)) AND (_ = _) ORDER BY _, _ LIMIT _) USING (_, _)) ORDER BY _, _, _: caused by <redacted>
  File "github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go", line 489, in func2
  File "github.com/cockroachdb/cockroach/pkg/sql/distsqlplan/physical_plan.go", line 676, in PopulateEndpoints
  File "github.com/cockroachdb/cockroach/pkg/sql/distsql_physical_planner.go", line 2463, in FinalizePlan
  File "github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go", line 531, in PlanAndRun
  File "github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go", line 672, in execWithDistSQLEngine
...
(8 additional frame(s) were not displayed)

conn_executor.go:521: panic while executing 1 statements: SELECT _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _ FROM ((SELECT _, _, _, _, _, _, _, _, _, _ FROM _._._ WHERE (_ = (-_)) AND (_ = _) ORDER BY _, _, _ LIMIT _) JOIN (SELECT _, _, _, _, _, _, _, _, _, _, _, _, _, _ FROM _._._ WHERE (_ = (-_)) AND (_ = _) ORDER BY _, _ LIMIT _) USING (_, _)) ORDER BY _, _, _: caused by <redacted>

[jordanlewis]

                p2.Spec.Input[s.DestInput].Streams = append(p2.Spec.Input[s.DestInput].Streams, endpoint)
                if endpoint.Type == distsqlrun.StreamEndpointSpec_REMOTE {
                        var ok bool
                        endpoint.TargetAddr, ok = nodeAddresses[p2.Node]
                        if !ok {
                                panic(fmt.Sprintf("node %d node in nodeAddresses map", p2.Node))
                        }
                }

The panic comes from the explicit panic listed there.

[solongordon]

I think I see where this got broken. getNodeIDForScan used to return the gateway node ID if a node was unhealthy or incompatible. Now it returns the unhealthy node ID.

0cd1da0#diff-8840c1839af40078c429f6fed21d7916L899

I'll dig in a bit more but if that's all it is, it'll be an easy fix.

[solongordon]

This bug is present as of 2.0.2.

[solongordon]

Just realized this code path is thankfully only used by LIMIT queries, which somewhat reduces the scope of this bug.