jobs: add job metrics per-type to track success, failure, and cancel

[fqazi]

Fixes: #59711

Previously, there were only over all counters tracking how many
jobs were completed, cancelled, or failed. This was inadequate
because it didn't make it easy to tell in aggregate what job
types they were. To address this, this patch will add counters
for different job types for tracking success, failure, and
cancellation.

Release justification: Low risk change only adding a metric inside
the crdb_internal.feature_usage table
Release note: None

[cockroach-teamcity]

This change is 

[ajwerner]

I'd prefer that this instead parallel the metrics as the life-cycle actually moves as opposed to the methods that get called. This will deal with retries and I also think that it's a good idea for us to couple these counter with the other extremely similar counters we already have. See

cockroach/pkg/jobs/registry.go

Line 1160 in c0d8340

jm := r.metrics.JobMetrics[jobType]

Reviewed 1 of 6 files at r1.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @dt)

[ajwerner]

I don't think this is what you want either. This is now adding metrics which are not telemetry counters (at least, I'm pretty sure they aren't).

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @dt)

[ajwerner]

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @dt and @fqazi)

---

pkg/sql/sqltelemetry/schema.go, line 155 at r2 (raw file):

Quoted 4 lines of code…

func SchemaJobSuccessCounter(jobName string) telemetry.Counter {
	jobName = strings.ToLower(strings.Replace(jobName, " ", "_", -1))
	return telemetry.GetCounter(fmt.Sprintf("sql.schema.job.%s.successful", jobName))
}

It's a slightly pre-mature optimization but I'd prefer if you just statically allocate all of these counters rather than generating these strings and doing it lazily.

I'm also not sure that they need to be in this package. I think you can happily import telemetry in the jobs package and just stick the vars there.

[fqazi]

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @dt and @fqazi)

pkg/sql/sqltelemetry/schema.go, line 155 at r2 (raw file):

Quoted 4 lines of code…
It's a slightly pre-mature optimization but I'd prefer if you just statically allocate all of these counters rather than generating these strings and doing it lazily.

I'm also not sure that they need to be in this package. I think you can happily import telemetry in the jobs package and just stick the vars there.

Good idea, I'll move them into the job package and allocate them once. I originally tried the telemetry package but ran into a circular dependency issue with the protobuf.

[ajwerner]

Reviewed 1 of 8 files at r2, 1 of 3 files at r3.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @ajwerner, @dt, and @fqazi)

---

pkg/jobs/metrics.go, line 168 at r3 (raw file):

		Successful: telemetry.GetCounterOnce(fmt.Sprintf("sql.schema.job.%s.successful", jobName)),
		Failed:     telemetry.GetCounterOnce(fmt.Sprintf("sql.schema.job.%s.failed", jobName)),
		Canceled:   telemetry.GetCounterOnce(fmt.Sprintf("sql.schema.job.%s.canceled", jobName)),

I don't think that the schema string makes a lot of sense here. Jobs are used in a lot of contexts outside of schema changes. I don't even know if sql makes sense. What if the prefix was just job.%s.<event>

[fqazi]

Reviewed 1 of 8 files at r2, 1 of 3 files at r3.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @ajwerner, @dt, and @fqazi)

pkg/jobs/metrics.go, line 168 at r3 (raw file):

		Successful: telemetry.GetCounterOnce(fmt.Sprintf("sql.schema.job.%s.successful", jobName)),
		Failed:     telemetry.GetCounterOnce(fmt.Sprintf("sql.schema.job.%s.failed", jobName)),
		Canceled:   telemetry.GetCounterOnce(fmt.Sprintf("sql.schema.job.%s.canceled", jobName)),

I don't think that the schema string makes a lot of sense here. Jobs are used in a lot of contexts outside of schema changes. I don't even know if sql makes sense. What if the prefix was just job.%s.<event>

I'll simplify it down to that format

[ajwerner]

LGTM

[fqazi]

bors r+

[craig]

Build failed (retrying...):

• GitHub CI (Cockroach)

[craig]

Canceled.

[fqazi]

bors r+

[craig]

Build succeeded:

• GitHub CI (Cockroach)

[vy-ton]

@fqazi Can you confirm these are the names of the new telemetry counters?

job.<job type>.[successful | failed | canceled]
<job type> can be: 
backup
restore
schema_change
import
changefeed
create_stats
schema_change_gc
type_schema_change
stream_ingestion
migration

[fqazi]

They are the following @vy-ton

<job type > is the internal mapping:
backup               
restore              
schema_change        
import               
changefeed           
create_stats         
auto_create_stats    
schema_change_gc     
typedesc_schema_change
stream_ingestion     
new_schema_change    
migration            

[vy-ton]

cc @mwang1026, you might be interested in these success, failure, and cancel telemetry counters for all job types