Enable collapsing asset groups in the asset graph UI

[mkleinbort-ic]

What's the use case?

We are encountering issues scaling Dagster to more assets while preserving visibility of asset dependencies.

Beyond a few hundred assets the DAG looks uninteligible - which is to be expected.

A major mitigation to this would be the ability to have "sub-groups" and possibly collapsable groups in the Dagit UI.

Kedro has a good implementation (this is a GIF). Here you can see the "group" Data Processing as a single node when zoomed-out, but it can also be expanded.

Ideas of implementation

I think this would require a modification to the group_name parameter to accept a list, then one could collapse based on the entries in the list.

In this idea, an asset would be declared as

@asset(group_name=['data_processing'])
def foo():
    ...

@asset(group_name=['data_processing'])
def bar():
    ...

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

[mkleinbort-ic]

Here is a link to a Kedro interactive demo that shows the functionality.

https://demo.kedro.org/

And here is some documentation on how they've approached dag layout (they seem to have put a lot of thought into it)

https://github.com/kedro-org/kedro-viz/blob/main/LAYOUT_ENGINE.md

[smackesey]

cc @bengotow

[mkleinbort-ic]

On reflection, this issue could be approached in two phases:

Phase 1: Collapsible Groups
Phase 2: Nested Groups

Phase 1: Collapsible Groups

The first phase would be a UI toggle in the corner of the group bounding box. Here is a proposal:

Clicking on the [ - ] would collapse the group into a single node with a single input & output point for all edges incoming/outgoing from the assets in the group:

Phase 2: Nested Groups

This would allow one do define groups within groups. Hopefully the collapsible groups feature can be extended without too much effort.

[bengotow]

This looks great! We've played around a version of the DAG visualization that does more of a "graph with a box, which reveals another graph when you zoom in", but I think a simple collapsing behavior might be best.

One challenge we've had with this is that groups in asset graphs can get super large, and collapsing them reflows the rest of the graph so much that it's hard to tell what happened. "eg: I hide this huge box, and everything animates all over the place." Maybe we can find a way to mitigate that -- if we restricted the types of reflow allowed after collapsing a box to just sliding other nodes into the empty space, it might be easy enough to follow.

[mkleinbort-ic]

I'm also very ok with a full re-flow of the dag.

In many ways I want to collapse a group to help me think through my DAG in a new way - say "Forgetting that my data extraction group is 100 separate assets - what stuff depends on it, and what stuff does it depend on"?

A re-flow would be helpful in this context.

[mkleinbort-ic]

Hi @sryza
Just want to emphasize how amazing this feature would be! It's definitely the #1 feature I'm looking forward to in Dagster.

Is it currently in the Dagit backlog? I could not find it here: https://github.com/dagster-io/dagster/projects/8

[sryza]

Hey @mkleinbort-ic - I just added it to the Dagit backlog. @braunjj - I think you've seen this already, but just in case you haven't, pinging you.

[mkleinbort-ic]

Any updates on this?

[mkleinbort-ic]

Hey @hellendag - is this in the roadmap? Will hack something on my end if not.

[salazarm]

@mkleinbort-ic checkout #16657

[mkleinbort-ic]

It's a cool feature to navegate the graph. I'm using it at the moment.

But just to note that it's a different feature to what I'm looking for. In my deployments this pattern is quite common:

Usually when computing multiple metrics for an asset, or when extracting data from multiple API endpoints.

I find these "super wide" sections make the DAG quite hard to visualize and reason about, so it'd be nice to collapse them to show the data lineage from source asset to final asset without these distractions.

[salazarm]

@mkleinbort-ic @bengotow is currently prototyping this feature.

[mkleinbort-ic]

This can be closed as per the recent feature update (👏👏👏👏)
I am moving the conversation about nested groups to #14530