An alternative to using code locations for separating/categorising assets in a Kubernetes OSS deployment

[stevenmurphy12]

This is partly a question, partly a suggestion.

I'm building a new data platform centred around Dagster, replacing an existing system. We have x number of customers, and each of those customers has a variable amount of pipelines.

I'd like to be able to get a view of the pipelines per customer. The way I've been doing that at the moment is by having a separate code location deployment for each one (I'm running a K8s deployment of Dagster). Then each of the assets within a particular pipeline will be grouped (by asset group), for easier visual identification.

In the screenshot below you can see this: customers are on the left, asset groups on the right.

This solution feels less than optimal. Right now I've got a monorepo with all our customer pipelines in it, building a single Docker image. I believe will serve our purposes fine (we can break it out later if need be). In my K8s helm chart values, I've got multiple cod e location deployments referring to the same image.

My concern is that as we ramp up our onboarding of more customers, the amount of code location pods running in K8s will ramp up significantly. It seems like quite a significant waste of cluster resources, for simply displaying an asset graph for the front end.

It'd be useful to have some other means of tagging assets in such a way that they simulate the view I get in the screenshot above (or similar). For me that might look like:

@asset(group='pipeline1', customer='mycustomer')
def my_asset():
   ...

Though I'd be quite happy with a generalised option, doesn't have to be customer)

Thank you