Gateways supporting LLMServerPool as a an HTTPRoute BackendRef

[kfswain]

Our current Envoy integration relies on EnvoyExtensionPolicy and EnvoyPatchPolicy this is very manual, and not sustainable.
(See: #18)

We're trying to settle on a single implementation that this project will work on to extend to support LLMServerPool as a Gateway API backend. This will enable us to run e2e tests against these concepts and iterate more quickly. That implementation should be:

1. An existing conformant implementation of Gateway API
2. Part of CNCF
3. Envoy-based for simplicity of extension mechanisms
4. Open to contributions from us to support this new type of backend

We propose extending existing this gateway implementation to act as the controller for the LLMServerPool object. (See: https://github.com/kubernetes-sigs/llm-instance-gateway/blob/main/docs/proposals/002-api-proposal/proposal.md#llmserverpool). As well as updating HTTPRoute to support a LLMServerPool as a backendRef.

At a high level we expect this to look like:

• Upon creation of an LLMServerPool the controller creates: An ext-proc deployment/service. An original_dst cluster.
• Upon creation of an HTTPRoute with an LLMServerPool as a backendRef: A Listener that routes requests to the appropriate original_dst cluster (there may be multiple LLMServerPools), and configure ext_proc to operate on requests sent to this cluster

[arkodg]

hey @kfswain ive created envoyproxy/gateway#4423 to make a decision on supporting this in Envoy Gateway

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale