[compute] map LIR to dataflow

[mgree]

This PR introduces two new introspection sources and two new introspection views. These novel forms of introspection allow us to map LIR operators down to dataflow operators; we can now attribute existing introspection data about dataflows to LIR operators.

Introspection

The two new sources are ComputeLog sources that run per worker.

mz_introspection.mz_compute_dataflow_global_ids_per_worker

Maps dataflow identifiers to the global IDs used internally for things that get built as dataflows.

   name    | nullable | type  | comment
-----------+----------+-------+---------
 id        | f        | uint8 | dataflow ID
 worker_id | f        | uint8 |
 global_id | f        | text  |

mz_introspection.mz_compute_lir_mapping_per_worker

Tracks attribution information for LIR terms (in terms of FlatPlan).

       name        | nullable | type  | comment
-------------------+----------+-------+---------
 global_id         | f        | text  |
 lir_id            | f        | uint8 | AST node number
 worker_id         | f        | uint8 |
 operator          | f        | text  | rendered string
 parent_lir_id     | t        | uint8 | parent AST node number
 nesting           | f        | uint2 | nesting (used for indentation)
 operator_id_start | t        | uint8 | first dataflow operator (inclusive)
 operator_id_end   | t        | uint8 | last dataflow oeprator (exclusive)

Views

We use two introspection views to work with these per-worker sources. It ought to be the case that all workers agree about this metadata (though they may not agree on, say, the amount of memory a dataflow operator is using!).

So: these are just views that set worker_id = 0.

mz_introspection.mz_dataflow_global_ids

   name    | nullable | type  | comment
-----------+----------+-------+---------
 id        | f        | uint8 |
 global_id | f        | text  |

mz_introspection.mz_lir_mapping

       name        | nullable | type  | comment
-------------------+----------+-------+---------
 global_id         | f        | text  |
 lir_id            | f        | uint8 |
 operator          | f        | text  |
 parent_lir_id     | t        | uint8 |
 nesting           | f        | uint2 |
 operator_id_start | t        | uint8 |
 operator_id_end   | t        | uint8 |

Attributing to LIR

We can see a sample interaction as follows:

CREATE TABLE t(x INT NOT NULL, y INT, z TEXT);
CREATE VIEW v AS
  SELECT t1.x AS x, t1.z AS z1, t2.z AS z2
  FROM t AS t1, t AS t2
  WHERE t1.x = t2.y;
CREATE INDEX v_idx_x ON v(x);

\! sleep 1

SELECT global_id, lir_id, REPEAT(' ', MAX(nesting) * 2) || operator AS operator, SUM(duration_ns) AS duration, SUM(count) AS count
    FROM           mz_introspection.mz_lir_mapping mlm
         LEFT JOIN mz_introspection.mz_compute_operator_durations_histogram mcodh
         ON (mlm.operator_id_start <= mcodh.id AND mcodh.id < mlm.operator_id_end)
GROUP BY global_id, lir_id, operator
ORDER BY global_id, lir_id DESC;

which yields an output like:

 global_id | lir_id |          operator          | duration | count
-----------+--------+----------------------------+----------+-------
 u2        | 4      | Join::Differential 1 » 3   |  1261568 |    17
 u2        | 3      |   Arrange 2                |   466944 |    16
 u2        | 2      |     Get::Collection u1     |    69632 |     7
 u2        | 1      |   Arrange 0                |   417792 |    16
 u2        | 0      |     Get::Collection u1     |    73728 |     7
 u3        | 6      | Arrange 5                  |   454656 |    17
 u3        | 5      |   Get::PassArrangements u2 |          |

A place to store our bicycles

• Should these be mz_internal or mz_introspection? It needs to be mz_introspection to pass tests.

• Should I just do the indentation up front?

• Should I track other metadata (e.g., parents/children of LIR nodes?) I added parent_lir_id to allow for more complex reconstructions.

Motivation

• This PR adds a known-desirable feature.

The first step in attribution/plan profiling. https://github.com/MaterializeInc/database-issues/issues/6551

Tips for reviewer

I've tried to break things down so that each commit is sensible, but there's a little bit of back-and-forth in exactly what I store in the views.

Checklist

• This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
• This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
• If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
• If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
• If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.

[sthm]

I'd need to play around with it to get a better intuition how this works in practice. But from the description above, it seems to provide a an easy way to map operator ids back to object ids in a way that preserves the hierarchy and dependencies of operators.

The example shows how this mapping can be used to add operator level statistics mz_compute_operator_durations_histogram back to something that looks like a query plan. From a quick skim of the documentation, it seems like it would also work for mz_arrangement_sizes and even more complicated queries like operator skew from dataflow troubleshooting, which is great.

[antiguru]

Looks good, thank you! Left some inline comments.

[antiguru]

Do you want to include your 'Attribution to Lir' example here, without duration/count?

[mgree]

Good idea!

[mgree]

Turned back on in #30692.