release-22.1: sql: improve EXPLAIN (DDL) output

[blathers-crl]

Backport 2/2 commits from #79673 on behalf of @postamar.

/cc @cockroachdb/release

---

Previously, the EXPLAIN (DDL, STAGES) and EXPLAIN (DDL, DEPS) statements
were printing graphviz URLs to help engineers during the development of
the declarative schema changer.

Now that the declarative schema changer is going live, the audience for
EXPLAIN (DDL) changes to become the users. The graphviz URLs were, at
any rate, not very useful for schema changes involving many objects, due
to limitations in the rendering engine.

This commit gets rid of the STAGES and DEPS flags to provide a unified
plan which is similar to the data-driven test output in the scplan
package. The VERBOSE flag can now be used to render a superset of the
information in the default rendering.

Touches #77158.

Release note (sql change): EXPLAIN (DDL), when invoked on statements
supported by the declarative schema changer, prints a plan of what the
schema changer will do. This can be useful for the user to anticipate
the complexity of a schema change (anything involving Backfill or
Validation operations might be slow to run). This can be useful for
troubleshooting. EXPLAIN (DDL, VERBOSE) produces a more detailed plan.

---

EXPLAIN (DDL) Example:

Schema change plan for DROP VIEW ‹test›.public.‹v1dep› CASCADE;
 ├── StatementPhase
 │    └── Stage 1 of 1 in StatementPhase
 │         ├── 4 elements transitioning toward ABSENT
 │         │    ├── PUBLIC → TXN_DROPPED View:{DescID: 147}
 │         │    ├── PUBLIC → TXN_DROPPED View:{DescID: 148}
 │         │    ├── PUBLIC → TXN_DROPPED View:{DescID: 149}
 │         │    └── PUBLIC → TXN_DROPPED View:{DescID: 150}
 │         └── 4 Mutation operations
 │              ├── MarkDescriptorAsDroppedSynthetically {"DescID":147}
 │              ├── MarkDescriptorAsDroppedSynthetically {"DescID":148}
 │              ├── MarkDescriptorAsDroppedSynthetically {"DescID":149}
 │              └── MarkDescriptorAsDroppedSynthetically {"DescID":150}
 ├── PreCommitPhase
 │    └── Stage 1 of 1 in PreCommitPhase
 │         ├── 56 elements transitioning toward ABSENT
 │         │    ├──      PUBLIC → ABSENT     Namespace:{DescID: 147, Name: v1dep, ReferencedDescID: 104}
 │         │    ├──      PUBLIC → ABSENT     Owner:{DescID: 147}
 │         │    ├──      PUBLIC → ABSENT     UserPrivileges:{DescID: 147, Name: admin}
 │         │    ├──      PUBLIC → ABSENT     UserPrivileges:{DescID: 147, Name: root}
 │         │    ├── TXN_DROPPED → DROPPED    View:{DescID: 147}
 │         │    ├──      PUBLIC → ABSENT     ObjectParent:{DescID: 147, ReferencedDescID: 105}
 │         │    ├──      PUBLIC → ABSENT     TableComment:{DescID: 147}
 │         │    ├──      PUBLIC → WRITE_ONLY Column:{DescID: 147, ColumnID: 1}
 │         │    ├──      PUBLIC → ABSENT     ColumnName:{DescID: 147, Name: name, ColumnID: 1}
 │         │    ├──      PUBLIC → ABSENT     ColumnType:{DescID: 147, ColumnFamilyID: 0, ColumnID: 1}
 │         │    ├──      PUBLIC → ABSENT     ColumnComment:{DescID: 147, ColumnID: 1}
 │         │    ├──      PUBLIC → ABSENT     Namespace:{DescID: 148, Name: v2dep, ReferencedDescID: 104}
 │         │    ├──      PUBLIC → ABSENT     Owner:{DescID: 148}
 │         │    ├──      PUBLIC → ABSENT     UserPrivileges:{DescID: 148, Name: admin}
 │         │    ├──      PUBLIC → ABSENT     UserPrivileges:{DescID: 148, Name: root}
 │         │    ├── TXN_DROPPED → DROPPED    View:{DescID: 148}
 │         │    ├──      PUBLIC → ABSENT     ObjectParent:{DescID: 148, ReferencedDescID: 105}
 │         │    ├──      PUBLIC → ABSENT     TableComment:{DescID: 148}
 │         │    ├──      PUBLIC → WRITE_ONLY Column:{DescID: 148, ColumnID: 1}
 │         │    ├──      PUBLIC → ABSENT     ColumnName:{DescID: 148, Name: n1, ColumnID: 1}
 │         │    ├──      PUBLIC → ABSENT     ColumnType:{DescID: 148, ColumnFamilyID: 0, ColumnID: 1}
 │         │    ├──      PUBLIC → ABSENT     ColumnComment:{DescID: 148, ColumnID: 1}
 │         │    ├──      PUBLIC → WRITE_ONLY Column:{DescID: 148, ColumnID: 2}
 │         │    ├──      PUBLIC → ABSENT     ColumnName:{DescID: 148, Name: n2, ColumnID: 2}
 │         │    ├──      PUBLIC → ABSENT     ColumnType:{DescID: 148, ColumnFamilyID: 0, ColumnID: 2}
 │         │    ├──      PUBLIC → ABSENT     ColumnComment:{DescID: 148, ColumnID: 2}
 │         │    ├──      PUBLIC → ABSENT     Namespace:{DescID: 149, Name: v3dep, ReferencedDescID: 104}
 │         │    ├──      PUBLIC → ABSENT     Owner:{DescID: 149}
 │         │    ├──      PUBLIC → ABSENT     UserPrivileges:{DescID: 149, Name: admin}
 │         │    ├──      PUBLIC → ABSENT     UserPrivileges:{DescID: 149, Name: root}
 │         │    ├── TXN_DROPPED → DROPPED    View:{DescID: 149}
 │         │    ├──      PUBLIC → ABSENT     ObjectParent:{DescID: 149, ReferencedDescID: 105}
 │         │    ├──      PUBLIC → ABSENT     TableComment:{DescID: 149}
 │         │    ├──      PUBLIC → WRITE_ONLY Column:{DescID: 149, ColumnID: 1}
 │         │    ├──      PUBLIC → ABSENT     ColumnName:{DescID: 149, Name: name, ColumnID: 1}
 │         │    ├──      PUBLIC → ABSENT     ColumnType:{DescID: 149, ColumnFamilyID: 0, ColumnID: 1}
 │         │    ├──      PUBLIC → ABSENT     ColumnComment:{DescID: 149, ColumnID: 1}
 │         │    ├──      PUBLIC → WRITE_ONLY Column:{DescID: 149, ColumnID: 2}
 │         │    ├──      PUBLIC → ABSENT     ColumnName:{DescID: 149, Name: n1, ColumnID: 2}
 │         │    ├──      PUBLIC → ABSENT     ColumnType:{DescID: 149, ColumnFamilyID: 0, ColumnID: 2}
 │         │    ├──      PUBLIC → ABSENT     ColumnComment:{DescID: 149, ColumnID: 2}
 │         │    ├──      PUBLIC → ABSENT     Namespace:{DescID: 150, Name: v4dep, ReferencedDescID: 104}
 │         │    ├──      PUBLIC → ABSENT     Owner:{DescID: 150}
 │         │    ├──      PUBLIC → ABSENT     UserPrivileges:{DescID: 150, Name: admin}
 │         │    ├──      PUBLIC → ABSENT     UserPrivileges:{DescID: 150, Name: root}
 │         │    ├── TXN_DROPPED → DROPPED    View:{DescID: 150}
 │         │    ├──      PUBLIC → ABSENT     ObjectParent:{DescID: 150, ReferencedDescID: 105}
 │         │    ├──      PUBLIC → ABSENT     TableComment:{DescID: 150}
 │         │    ├──      PUBLIC → WRITE_ONLY Column:{DescID: 150, ColumnID: 1}
 │         │    ├──      PUBLIC → ABSENT     ColumnName:{DescID: 150, Name: n2, ColumnID: 1}
 │         │    ├──      PUBLIC → ABSENT     ColumnType:{DescID: 150, ColumnFamilyID: 0, ColumnID: 1}
 │         │    ├──      PUBLIC → ABSENT     ColumnComment:{DescID: 150, ColumnID: 1}
 │         │    ├──      PUBLIC → WRITE_ONLY Column:{DescID: 150, ColumnID: 2}
 │         │    ├──      PUBLIC → ABSENT     ColumnName:{DescID: 150, Name: n1, ColumnID: 2}
 │         │    ├──      PUBLIC → ABSENT     ColumnType:{DescID: 150, ColumnFamilyID: 0, ColumnID: 2}
 │         │    └──      PUBLIC → ABSENT     ColumnComment:{DescID: 150, ColumnID: 2}
 │         └── 29 Mutation operations
 │              ├── DrainDescriptorName {"Namespace":{"DatabaseID":104,"DescriptorID":147,"Name":"v1dep","SchemaID":105}}
 │              ├── RemoveDroppedColumnType {"ColumnID":1,"TableID":147}
 │              ├── DrainDescriptorName {"Namespace":{"DatabaseID":104,"DescriptorID":148,"Name":"v2dep","SchemaID":105}}
 │              ├── RemoveDroppedColumnType {"ColumnID":1,"TableID":148}
 │              ├── RemoveDroppedColumnType {"ColumnID":2,"TableID":148}
 │              ├── DrainDescriptorName {"Namespace":{"DatabaseID":104,"DescriptorID":149,"Name":"v3dep","SchemaID":105}}
 │              ├── RemoveDroppedColumnType {"ColumnID":1,"TableID":149}
 │              ├── RemoveDroppedColumnType {"ColumnID":2,"TableID":149}
 │              ├── DrainDescriptorName {"Namespace":{"DatabaseID":104,"DescriptorID":150,"Name":"v4dep","SchemaID":105}}
 │              ├── RemoveDroppedColumnType {"ColumnID":1,"TableID":150}
 │              ├── RemoveDroppedColumnType {"ColumnID":2,"TableID":150}
 │              ├── MarkDescriptorAsDropped {"DescID":149}
 │              ├── RemoveViewBackReferencesInRelations {"BackReferencedViewID":149}
 │              ├── RemoveAllTableComments {"TableID":149}
 │              ├── MarkDescriptorAsDropped {"DescID":150}
 │              ├── RemoveViewBackReferencesInRelations {"BackReferencedViewID":150}
 │              ├── RemoveAllTableComments {"TableID":150}
 │              ├── MarkDescriptorAsDropped {"DescID":148}
 │              ├── RemoveViewBackReferencesInRelations {"BackReferencedViewID":148}
 │              ├── RemoveAllTableComments {"TableID":148}
 │              ├── MarkDescriptorAsDropped {"DescID":147}
 │              ├── RemoveViewBackReferencesInRelations {"BackReferencedViewID":147}
 │              ├── RemoveAllTableComments {"TableID":147}
 │              ├── SetJobStateOnDescriptor {"DescriptorID":146,"Initialize":true}
 │              ├── SetJobStateOnDescriptor {"DescriptorID":147,"Initialize":true}
 │              ├── SetJobStateOnDescriptor {"DescriptorID":148,"Initialize":true}
 │              ├── SetJobStateOnDescriptor {"DescriptorID":149,"Initialize":true}
 │              ├── SetJobStateOnDescriptor {"DescriptorID":150,"Initialize":true}
 │              └── CreateSchemaChangerJob {"NonCancelable":true}
 └── PostCommitNonRevertiblePhase
      └── Stage 1 of 1 in PostCommitNonRevertiblePhase
           ├── 11 elements transitioning toward ABSENT
           │    ├──    DROPPED → ABSENT View:{DescID: 147}
           │    ├── WRITE_ONLY → ABSENT Column:{DescID: 147, ColumnID: 1}
           │    ├──    DROPPED → ABSENT View:{DescID: 148}
           │    ├── WRITE_ONLY → ABSENT Column:{DescID: 148, ColumnID: 1}
           │    ├── WRITE_ONLY → ABSENT Column:{DescID: 148, ColumnID: 2}
           │    ├──    DROPPED → ABSENT View:{DescID: 149}
           │    ├── WRITE_ONLY → ABSENT Column:{DescID: 149, ColumnID: 1}
           │    ├── WRITE_ONLY → ABSENT Column:{DescID: 149, ColumnID: 2}
           │    ├──    DROPPED → ABSENT View:{DescID: 150}
           │    ├── WRITE_ONLY → ABSENT Column:{DescID: 150, ColumnID: 1}
           │    └── WRITE_ONLY → ABSENT Column:{DescID: 150, ColumnID: 2}
           └── 21 Mutation operations
                ├── LogEvent {"Statement":"DROP VIEW ‹test›...","StatementTag":"DROP VIEW","TargetStatus":1}
                ├── DeleteDescriptor {"DescriptorID":147}
                ├── MakeColumnAbsent {"ColumnID":1,"TableID":147}
                ├── LogEvent {"Statement":"DROP VIEW ‹test›...","StatementTag":"DROP VIEW","TargetStatus":1}
                ├── DeleteDescriptor {"DescriptorID":148}
                ├── MakeColumnAbsent {"ColumnID":1,"TableID":148}
                ├── MakeColumnAbsent {"ColumnID":2,"TableID":148}
                ├── LogEvent {"Statement":"DROP VIEW ‹test›...","StatementTag":"DROP VIEW","TargetStatus":1}
                ├── DeleteDescriptor {"DescriptorID":149}
                ├── MakeColumnAbsent {"ColumnID":1,"TableID":149}
                ├── MakeColumnAbsent {"ColumnID":2,"TableID":149}
                ├── LogEvent {"Statement":"DROP VIEW ‹test›...","StatementTag":"DROP VIEW","TargetStatus":1}
                ├── DeleteDescriptor {"DescriptorID":150}
                ├── MakeColumnAbsent {"ColumnID":1,"TableID":150}
                ├── MakeColumnAbsent {"ColumnID":2,"TableID":150}
                ├── RemoveJobStateFromDescriptor {"DescriptorID":146}
                ├── RemoveJobStateFromDescriptor {"DescriptorID":147}
                ├── RemoveJobStateFromDescriptor {"DescriptorID":148}
                ├── RemoveJobStateFromDescriptor {"DescriptorID":149}
                ├── RemoveJobStateFromDescriptor {"DescriptorID":150}
                └── UpdateSchemaChangerJob {"IsNonCancelable":true}

EXPLAIN (DDL, VERBOSE) Example:

• Schema change plan for DROP SEQUENCE ‹test›.public.‹sq1› CASCADE;
│
├── • StatementPhase
│   │
│   └── • Stage 1 of 1 in StatementPhase
│       │
│       ├── • 1 element transitioning toward ABSENT
│       │   │
│       │   └── • Sequence:{DescID: 113}
│       │         PUBLIC → TXN_DROPPED
│       │
│       └── • 1 Mutation operation
│           │
│           └── • MarkDescriptorAsDroppedSynthetically
│                 DescID: 113
│
├── • PreCommitPhase
│   │
│   └── • Stage 1 of 1 in PreCommitPhase
│       │
│       ├── • 9 elements transitioning toward ABSENT
│       │   │
│       │   ├── • Namespace:{DescID: 113, Name: sq1, ReferencedDescID: 104}
│       │   │     PUBLIC → ABSENT
│       │   │
│       │   ├── • Owner:{DescID: 113}
│       │   │   │ PUBLIC → ABSENT
│       │   │   │
│       │   │   └── • skip PUBLIC → ABSENT operations
│       │   │         rule: "skip element removal ops on descriptor drop"
│       │   │
│       │   ├── • UserPrivileges:{DescID: 113, Name: admin}
│       │   │   │ PUBLIC → ABSENT
│       │   │   │
│       │   │   └── • skip PUBLIC → ABSENT operations
│       │   │         rule: "skip element removal ops on descriptor drop"
│       │   │
│       │   ├── • UserPrivileges:{DescID: 113, Name: root}
│       │   │   │ PUBLIC → ABSENT
│       │   │   │
│       │   │   └── • skip PUBLIC → ABSENT operations
│       │   │         rule: "skip element removal ops on descriptor drop"
│       │   │
│       │   ├── • Sequence:{DescID: 113}
│       │   │   │ TXN_DROPPED → DROPPED
│       │   │   │
│       │   │   ├── • Precedence dependency from ABSENT Namespace:{DescID: 113, Name: sq1, ReferencedDescID: 104}
│       │   │   │     rule: "dependent element removal before descriptor drop"
│       │   │   │
│       │   │   ├── • Precedence dependency from ABSENT Owner:{DescID: 113}
│       │   │   │     rule: "dependent element removal before descriptor drop"
│       │   │   │
│       │   │   ├── • Precedence dependency from ABSENT UserPrivileges:{DescID: 113, Name: admin}
│       │   │   │     rule: "dependent element removal before descriptor drop"
│       │   │   │
│       │   │   ├── • Precedence dependency from ABSENT UserPrivileges:{DescID: 113, Name: root}
│       │   │   │     rule: "dependent element removal before descriptor drop"
│       │   │   │
│       │   │   ├── • Precedence dependency from ABSENT ObjectParent:{DescID: 113, ReferencedDescID: 105}
│       │   │   │     rule: "dependent element removal before descriptor drop"
│       │   │   │
│       │   │   ├── • Precedence dependency from ABSENT TableComment:{DescID: 113}
│       │   │   │     rule: "dependent element removal before descriptor drop"
│       │   │   │
│       │   │   ├── • Precedence dependency from ABSENT ColumnDefaultExpression:{DescID: 114, ColumnID: 2}
│       │   │   │     rule: "column DEFAULT removed before dropping dependent types and sequences"
│       │   │   │
│       │   │   └── • Precedence dependency from ABSENT ColumnDefaultExpression:{DescID: 115, ColumnID: 2}
│       │   │         rule: "column DEFAULT removed before dropping dependent types and sequences"
│       │   │
│       │   ├── • ObjectParent:{DescID: 113, ReferencedDescID: 105}
│       │   │     PUBLIC → ABSENT
│       │   │
│       │   ├── • TableComment:{DescID: 113}
│       │   │   │ PUBLIC → ABSENT
│       │   │   │
│       │   │   └── • skip PUBLIC → ABSENT operations
│       │   │         rule: "skip table comment removal ops on descriptor drop"
│       │   │
│       │   ├── • ColumnDefaultExpression:{DescID: 114, ColumnID: 2}
│       │   │     PUBLIC → ABSENT
│       │   │
│       │   └── • ColumnDefaultExpression:{DescID: 115, ColumnID: 2}
│       │         PUBLIC → ABSENT
│       │
│       └── • 11 Mutation operations
│           │
│           ├── • DrainDescriptorName
│           │     Namespace:
│           │       DatabaseID: 104
│           │       DescriptorID: 113
│           │       Name: sq1
│           │       SchemaID: 105
│           │
│           ├── • RemoveColumnDefaultExpression
│           │     ColumnID: 2
│           │     TableID: 114
│           │
│           ├── • UpdateBackReferencesInSequences
│           │     BackReferencedColumnID: 2
│           │     BackReferencedTableID: 114
│           │     SequenceIDs:
│           │     - 113
│           │
│           ├── • RemoveColumnDefaultExpression
│           │     ColumnID: 2
│           │     TableID: 115
│           │
│           ├── • UpdateBackReferencesInSequences
│           │     BackReferencedColumnID: 2
│           │     BackReferencedTableID: 115
│           │     SequenceIDs:
│           │     - 113
│           │
│           ├── • MarkDescriptorAsDropped
│           │     DescID: 113
│           │
│           ├── • RemoveAllTableComments
│           │     TableID: 113
│           │
│           ├── • SetJobStateOnDescriptor
│           │     DescriptorID: 113
│           │     Initialize: true
│           │
│           ├── • SetJobStateOnDescriptor
│           │     DescriptorID: 114
│           │     Initialize: true
│           │
│           ├── • SetJobStateOnDescriptor
│           │     DescriptorID: 115
│           │     Initialize: true
│           │
│           └── • CreateSchemaChangerJob
│                 Authorization:
│                   UserName: root
│                 DescriptorIDs:
│                 - 113
│                 - 114
│                 - 115
│                 JobID: 1
│                 NonCancelable: true
│                 Statements:
│                 - statement: DROP SEQUENCE sq1 CASCADE
│                   redactedstatement: DROP SEQUENCE ‹test›.public.‹sq1› CASCADE
│                   statementtag: DROP SEQUENCE
│
└── • PostCommitNonRevertiblePhase
    │
    └── • Stage 1 of 1 in PostCommitNonRevertiblePhase
        │
        ├── • 1 element transitioning toward ABSENT
        │   │
        │   └── • Sequence:{DescID: 113}
        │         DROPPED → ABSENT
        │
        └── • 6 Mutation operations
            │
            ├── • LogEvent
            │     Authorization:
            │       UserName: root
            │     Element:
            │       Sequence:
            │         sequenceId: 113
            │     Statement: DROP SEQUENCE ‹test›.public.‹sq1› CASCADE
            │     StatementTag: DROP SEQUENCE
            │     TargetMetadata:
            │       SourceElementID: 1
            │       SubWorkID: 1
            │     TargetStatus: 1
            │
            ├── • CreateGcJobForTable
            │     StatementForDropJob:
            │       Statement: DROP SEQUENCE test.public.sq1 CASCADE
            │     TableID: 113
            │
            ├── • RemoveJobStateFromDescriptor
            │     DescriptorID: 113
            │     JobID: 1
            │
            ├── • RemoveJobStateFromDescriptor
            │     DescriptorID: 114
            │     JobID: 1
            │
            ├── • RemoveJobStateFromDescriptor
            │     DescriptorID: 115
            │     JobID: 1
            │
            └── • UpdateSchemaChangerJob
                  IsNonCancelable: true
                  JobID: 1

---

Release justification: low-risk high-value change pertaining to declarative schema changer

[blathers-crl]

Thanks for opening a backport.

Please check the backport criteria before merging:

• Patches should only be created for serious issues or test-only changes.
• Patches should not break backwards-compatibility.
• Patches should change as little code as possible.
• Patches should not change on-disk formats or node communication protocols.
• Patches should not add new functionality.
• Patches must not add, edit, or otherwise modify cluster versions; or add version gates.

If some of the basic criteria cannot be satisfied, ensure that the exceptional criteria are satisfied within.

• There is a high priority need for the functionality that cannot wait until the next release and is difficult to address in another way.
• The new functionality is additive-only and only runs for clusters which have specifically “opted in” to it (e.g. by a cluster setting).
• New code is protected by a conditional check that is trivial to verify and ensures that it only runs for opt-in clusters.
• The PM and TL on the team that owns the changed code have signed off that the change obeys the above rules.

Add a brief release justification to the body of your PR to justify this backport.

Some other things to consider:

• What did we do to ensure that a user that doesn’t know & care about this backport, has no idea that it happened?
• Will this work in a cluster of mixed patch versions? Did we test that?
• If a user upgrades a patch version, uses this feature, and then downgrades, what happens?

[cockroach-teamcity]

This change is 

[ajwerner]

Needs a rewrite for some reason.

[postamar]

We may have forgotten to backport something. Investigating.

[postamar]

I think it's because #79294 hasn't been merged yet.

[ajwerner]

I think it's because #79294 hasn't been merged yet.

Ack, rebasing it now

[ajwerner]

Err, so are you 😓

[postamar]

After #79294 gets merged, I'll rebase this one again before merging.