[RFC] Provider Runtime Interface (PRI)

[hasheddan]

What problem are you facing?

Crossplane supports two types of packages: Configurations and Providers. In practice, both of these packages need only be a single package.yaml file in a tarball in the first layer of an OCI image that contains a stream of YAML documents. However, Providers are unique in that they also point to another image that is contains the controller for the CRDs in the YAML stream. When a Provider is installed in a Crossplane cluster, Crossplane unpacks the package image, installs all of the CRDs in the YAML stream, then creates a Deployment using the the referenced controller image (which can be further configured by referencing a ControllerConfig).

However, the fact the the controllers are run as a Deployment is merely an implementation detail. They could be provisioned and run as another Kubernetes primitive, a higher-level construct (e.g. a function), or a runtime completely external to the cluster (e.g. a server, a managed service, another Kubernetes cluster etc.). Enabling each of these use cases opens up a set of unique deployment models for Crossplane itself:

• Utilizing external runtimes means that Crossplane could run with a minimized Kubernetes API server, perhaps even one that is bundled into Crossplane itself. For example, running a Crossplane instance could be as simple as docker run crossplane/crossplane:latest, and providers could be run by using a Docker implementation of the "Provider Runtime Interface (PRI)" that deployed the provider controllers as containers that shared a network namespace with Crossplane.
• Decoupling the control plane from the provider runtime means that the two components can be operated independently, potentially by different parties. This enables scenarios where credentials may need to be managed externally (at the provider runtime), but package management and composition can be managed by a third party.
• A PRI implementation could opt to handle some aspects of the data plane. For example, something like crisscross could be implemented by the PRI implementation provisioning an intermediate controller that brokers communication to smaller functions that can be written an any language (essentially simplifying the interface to be notified of Kubernetes events). (Note: I am not entirely sold on this being the way to do crisscross, but seems a least related to this effort).

How could Crossplane help solve your problem?

There are no shortage of runtimes available to run a containerized process, but each requires specific steps and configuration to run and be able to communicate to the cluster where Crossplane is running. Because of this, it does not make sense for Crossplane to bundle in all of the logic for provisioning against multiple runtimes. In fact, there is already doubt (#2468) around whether the ControllerConfig should be promoted to a stable Crossplane API (disclaimer: I think I am the primary one who is doubtful).

A cleaner and more extensible story for deploying Providers against heterogeneous runtimes would be to define an interface (similar to CRI, CSI, etc.). This would allow for users to set up the the appropriate PRI implementation(s) when deploying Crossplane, each of which could support dedicated configuration mechanism(s).

Development of such an interface could impact:

• Promote ControllerConfig to v1beta1 #2468
• API Server (and clients) becomes unresponsive with too many CRDs #2649
• Is it possible for package manager to use node image cache #2647
• Overwrite package parameters in configuration  #2367
• Default Provider Service Account always created when overridden in ControllerConfig  #2295
• Configuration of installed dependencies #1985
• Configuration package global parameters #1922

[negz]

I could imagine https://virtual-kubelet.io/ as an alternative avenue to run provider controllers outside of Kubernetes, though in its current implementation that approach would not support all of the deployment models suggested in the original post (e.g. it assumes a full API server with Pods etc).

I think relating these deployment models to some problems we know (or suspect) our users have would help a lot. The most immediate customer use case I can think of is ours at Upbound, where we don't want to run provider controllers in the same API server where Crossplane is running.

[fire-ant]

I like the idea of running a Provider per tenant workload where Crossplane is either:

• running in the CAPI mgmt plane
• Running as a dedicated cluster deployed by CAPI using KCP

this allows me to utilise a provider runtime with the specific remit of setting up predicates such as IAM/KMS/VPC as part of the tenant bootstrap process and also manage the separation of concerns with greater granularity regarding access to the cloud resources.

It also ensures a leaner/cleaner implementation where I can reuse components and (in the event it’s needed) means that I can run a single crossplane instance of each release that a number of providers need. That is the one issue I foresee in decoupling but it can be handled/tested accordingly.

[fire-ant]

I should also say that the KCP is just a kubernetes-centric way of deploying the crossplane controllers and of course running in an alternative managed service which was transparent to the provider runtime would be ideal too (though it might be a little chicken/egg as to how that's managed lifecycle wise). As a candidate/target, KCP would have the advantage of being consistent across clouds and would have a relatively known bootstrap/deploy process.

[tobru]

An idea could also be to use vcluster instances to run a Provider per tenant, but not having to spin up a full Kubernetes cluster.

[s4rd1nh4]

An idea could also be to use vcluster instances to run a Provider per tenant, but not having to spin up a full Kubernetes cluster.

virtualcluster is viable alternative alligned with cluster-api

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[turkenh]

Crossplane supports two types of packages: Configurations and Providers.

@hasheddan, as part of #3670, we are considering introducing another package type for Secret Store Plugins, which is originally proposed here. As we made progress on the implementation, we realized that the deployment model proposed in the design (e.g. run plugins as sidecars for all providers and core Crossplane) would be painful and we are considering to propose changes there as such we will run secret store plugins (basically a gRPC server talking to ESS) as a separate deployment with a K8s service in front. This makes it look more like Provider packages in the following aspect:

However, Providers are unique in that they also point to another image that is contains the controller for the CRDs in the YAML stream. When a Provider is installed in a Crossplane cluster, Crossplane unpacks the package image, installs all of the CRDs in the YAML stream, then creates a Deployment using the the referenced controller image (which can be further configured by referencing a ControllerConfig).

So, we believe we would have similar configuration needs for the plugin Deployments, and we would need something similar to ControllerConfigs. I am wondering if we could have something more generic than Provider Runtime Interface that could help for future package types like this.

/cc @ezgidemirel

[dvgitit]

What is the recommended way to specify settings for a company internal proxy, if ControllerConfig is deprecated?

[jbw976]

@dvgitit see the note at the top of #3601:

⚠️ ControllerConfig was marked as deprecated in v1.11, but it will not be removed until a comparable alternative is available.

So ControllerConfig is still available now while a replacement is being proposed/worked on in #2671 😊