sql: insert-stmt-stats is not version gated and fails in mixed-version state

[stevendanna]

Describe the problem

In a mixed-version state (with nodes running the 22.2 binary but still on a 22.1 cluster version), I see a number of log lines such as:

W220912 15:21:54.889174 398 sql/sqlstats/persistedsqlstats/flush.go:117 ⋮ [n1] 2118  ‹failed to flush statement statistics›: flush statement ‹738159740542152745›'s statistics: ‹insert-stmt-stats›: ‹INSERT› has more ‹expressions› than ‹target columns›, 11 expressions for 10 targets

This is the result of an insert statement that requires a newly added column but which doesn't use a version gate to ensure that the column was added.

cockroach/pkg/sql/sqlstats/persistedsqlstats/flush.go

Lines 475 to 481 in fc308c1

	insertStmt := `
	INSERT INTO system.statement_statistics
	VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11)
	ON CONFLICT (crdb_internal_aggregated_ts_app_name_fingerprint_id_node_id_plan_hash_transaction_fingerprint_id_shard_8,
	aggregated_ts, fingerprint_id, transaction_fingerprint_id, app_name, plan_hash, node_id)
	DO NOTHING
	`

To avoid this, we likely need to conditionally insert the index recommendations based on a version gate on

cockroach/pkg/clusterversion/cockroach_versions.go

Line 219 in fc308c1

AlterSystemStatementStatisticsAddIndexRecommendations

We simply log this failure, so the only impact is that the stats we had hoped to persist are not persisted and some log spam.

To Reproduce

This can be seen in the import/mixed-versions roachtest.

Jira issue: CRDB-19707

[blathers-crl]

Hi @ajwerner, please add branch-* labels to identify which branch(es) this release-blocker affects.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}

[ajwerner]

@maryliag I don't know if this rises to the level of release-blocker, but it very well might