Support dynamic graph execution

[groszewn]

Currently, the inference graph must be statically defined within the SeldonDeployment. We have multiple models that would be reused across multiple different inference graphs, leading to an increase in resource usage (since each graph spins up its own underlying model pods). This also means we need to deploy a new inference graph to our cluster for any slightly modified graph that our consumers may need.

It would be great to be able to dynamically define the inference graph at request-time as opposed to deploy-time to decrease both the amount of resources used and production deployments needed. Some sort of model registry within the cluster could potentially be a way to discover what model services are available for use.

[ukclivecox]

Sounds interesting. Can you expand a bit on how you see this working?

Would this allow models to be running in multiple shared graphs?

[groszewn]

I feel like this would be more of an additional service that leverages SeldonDeployments as opposed to an extension of the current executor. The mlgraph repo pretty accurately captures how I'm thinking about this.

I'm envisioning a registry of all deployed SeldonDeployments and their corresponding schema (depending on the outcome of #1420) that the orchestration service would be aware of and leverage. Users would pass in the required inputs and defined inference graph to be executed (likely reusing the graph structure in the SeldonDeployment specification).

For statically defined graphs, a dependency map that tracks enough information between graphs to ensure the exact same model service is truly intended to be used (runtime arguments, environment variables, resource requests/limits, mounted volumes, etc.) would become unwieldy pretty quickly. I would likely see the option for models to be running in multiple shared graphs as a feature of runtime-defined graphs only.

[ukclivecox]

This can be solved via Tempo