sql: Add telemetry counter when descriptor corruption error is encountered

[vy-ton]

In 20.2, we added descriptor validation on write. In 21.1, we will start logging/creating Sentry reports for namespace validation errors.

Desired behavior
Whenever we encounter a corruption error, we should increment a telemetry counter(s). The number of counters should be determined by the granularity level we have for corruption errors (Ex. descriptor corruption vs namespace corruption).

[postamar]

I've been giving this some thought. Are there any arguments against adding telemetry for all descriptor and namespace validation failures?

[vy-ton]

I would love that! Would it be a single counter for all descriptor validation failures?

[postamar]

I'm in the process of finding out how to best do this but I should have an answer for you today.

[postamar]

I eventually came up with a solution involving a small number of counters with the following dimensions:

• failure class: was this a descriptor self-check? or a cross-reference check?
• failure subject type: was this descriptor a table descriptor? a database descriptor? etc.
• failure time: did the failure occur at read time or at write time? Note that validations also take place when attempting to rename things like columns and whatnot. I opted to not count these because they're ultimately the result of user error. These errors usually have proper PG codes anyway (but not always!)

Please correct me if I'm wrong but I believe this addresses the original ask and it's not too big a change that it goes against the spirit of the stability period.

[vy-ton]

I'll let @ajwerner or @jordanlewis weigh in on the right dimensions.

Initially, I would likely aggregate all description corruption-related counters to view the trendline. So this proposal SGTM.