StatementStatistics.Nodes appears to be incomplete

[matthewtodd]

In my investigations for #89949, I believe I've discovered we are under-reporting the nodes involved in a distsql query in crdb_internal.statement_statistics and friends.

To reproduce, create a multi-region database with a populated regional by row table, and select * from it. The resulting statistics->>'statistics'->>'nodes' value in crdb_internal.statement_statistics should show at least one node from each region, but it will instead only show the gateway node. (Note that EXPLAIN (DISTSQL) however does show all the involved nodes.)

I believe the suspect code is in getNodesFromPlanner in executor_statement_metrics.go.

Observed on a 22.2 release as well as on (current) pre-23.1 master.

Jira issue: CRDB-24263

[xinhaoz]

It looks like that getNodesFromPlanner only works when tracing is on while we always provide at least the gateway node as part of the list. Maybe we need to figure out how to better communicate that there was no observability here (and it could have run on more nodes) when we don't have the trace? @kevin-v-ngo