roachtest: schemachange/mixed-versions failed [debug doctor validation: existing descriptors that still need to be dropped]

[cockroach-teamcity]

roachtest.schemachange/mixed-versions failed with artifacts on release-23.1.22-rc @ 6ed4ef16634f683adfb7d77b4ebf414e0c1e42a7:

(mixedversion.go:594).Run: mixed-version test failure while running step 15 (run "run schemachange workload and validation after upgrade has finalized"): full command output in run_060543.459340130_n4_cockroach-debug-doct.log: COMMAND_PROBLEM: exit status 125
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• roachtest: schemachange/mixed-versions failed [functions cannot have more than 100 arguments] #124390 roachtest: schemachange/mixed-versions failed [functions cannot have more than 100 arguments] [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-master]

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-39037

[annrpom]

run_060543.459340130_n4_cockroach-debug-doct: 06:05:43 cluster.go:2164: > ./cockroach debug doctor examine cluster --url {pgurl:4}
Examining 317 descriptors and 195 namespace entries...
Examining 1186 jobs...
ERROR: validation failed
Failed running "debug doctor examine cluster"
job 972289588003012611: running schema change GC refers to missing table descriptor(s) [336]; existing descriptors that still need to be dropped [337 350]; job safe to delete: false.
run_060543.459340130_n4_cockroach-debug-doct: 06:05:44 cluster.go:2171: > result: COMMAND_PROBLEM: exit status 125

[rafiss]

Do you think #124424 would resolve this?

[annrpom]

@rafiss no, that PR will just touch examining descriptors (to skip over dropped descriptors, skipping over ValidateJobReferencesInDescriptor -- which produces this error).

the fail state we see here is likely coming from the ValidateDescriptorReferencesInJob check while examining jobs:

cockroach/pkg/jobs/validate.go

Lines 85 to 89 in 27cffe8

	case jobspb.TypeSchemaChangeGC:
	isSafeToDelete := existing.Len() == 0 && len(j.Progress.GetSchemaChangeGC().Indexes) == 0
	errorAccFn(errors.AssertionFailedf("%s schema change GC refers to missing table descriptor(s) %+v; "+
	"existing descriptors that still need to be dropped %+v; job safe to delete: %v",
	j.Status, missing.Ordered(), existing.Ordered(), isSafeToDelete))

i am a bit unsure as to why we would do ValidateDescriptorReferencesInJob in both descriptor and job examining -- let me ask during standup

[annrpom]

Upon checking this out further, I see that there is a difference between the two validations and both seem necessary:

• ValidateDescriptorReferencesInJob (during ExamineJobs)
• ValidateJobReferencesInDescriptor (during ExamineDescriptors)

I also don't think this was due to:

1. the wrong jobs table being written to
2. a dropped descriptor being wrongfully scanned over

Which are the two things that backport addresses

---

Elaboration on point 2:

trying to mock this state in doctor_test.go leads to getting an expected job examination of no errors

for posterity, here is the test I was trying to get to reproduce:

		{
			descTable: doctor.DescriptorTable{
				{
					ID: 2,
					DescBytes: toBytes(t, &descpb.Descriptor{Union: &descpb.Descriptor_Table{
						Table: &descpb.TableDescriptor{ID: 2},
					}}),
				},
				{
					ID: 3,
					DescBytes: toBytes(t, &descpb.Descriptor{Union: &descpb.Descriptor_Table{
						Table: &descpb.TableDescriptor{ID: 3, State: descpb.DescriptorState_DROP},
					}}),
				},
			},
			jobsTable: doctor.JobsTable{
				{
					ID:      100,
					Payload: &jobspb.Payload{Details: jobspb.WrapPayloadDetails(jobspb.SchemaChangeGCDetails{})},
					Progress: &jobspb.Progress{Details: jobspb.WrapProgressDetails(
						jobspb.SchemaChangeGCProgress{
							Tables: []jobspb.SchemaChangeGCProgress_TableProgress{
								{ID: 1, Status: jobspb.SchemaChangeGCProgress_CLEARED},
								{ID: 2, Status: jobspb.SchemaChangeGCProgress_CLEARING},
								{ID: 3, Status: jobspb.SchemaChangeGCProgress_WAITING_FOR_CLEAR},
							},
						})},
					Status: jobs.StatusRunning,
				},
			},
			expected: `Examining 1 jobs...
job 100: running schema change GC refers to missing table descriptor(s) [3]; existing descriptors that still need to be dropped [2]; job safe to delete: false.
`,
		},

But, since descID 3 is marked as dropped, we actually get:

Examining 1 jobs...

This doesn't seem to be a frequent failure. I'll close this one out as unreproducable for now and gently remind myself to take care of my ongoing backports. Maybe what had happened was that a dropped descriptor got gced before its job entry got to update its job status?