Proposal: Deprecate and remove native secret store support

[negz]

What problem are you facing?

Crossplane has supported generating and writing connection details to Kubernetes secrets since its first release.

Connection details are the details needed to connect to a resource. For example you need to know an RDS instance's address, username, and password to connect to its default database. Connection details are often sensitive.

Claims, XRs, and MRs all have a spec.writeConnectionSecretToRef field. When you specify this field, Crossplane writes connection details to the named Kubernetes secret.

Crossplane v1.7.0 (release notes) added alpha support for "external secret stores". External secret stores allow Crossplane to write connection details to an external secret store, like HashiCorp Vault. You can read the design document here.

You configure Crossplane to write connection details to an external secret store using the spec.publishConnectionDetailsTo field of a claim, XR, or MR. Crossplane's external secret stores are write-only. You can't read secrets, for example provider credentials, from an external secret store. Adding support for reading from external secret stores is tracked in #2985.

As of Crossplane v1.15.0 (two years later), external secret stores are still an alpha feature. They're off by default, and must be enabled using the --enable-external-secret-stores flag.

At the time we designed Crossplane's external secret stores:

• We couldn't find any existing projects that could push Kubernetes secrets to an external store like Vault. Projects like https://external-secrets.io/ were early, and only supported pulling secrets.
• Some Crossplane users had expressed that they could never write connection details to a Kubernetes secret, which they deemed too insecure.

Since then, External Secrets Operator has added support for pushing Kubernetes secrets to external secret stores using a PushSecret resource. You can read about that here.

How could Crossplane help solve your problem?

I propose we deprecate and remove Crossplane's native external secret store support. Removing external secret store support means Crossplane would only be able to write connection details to a Kubernetes secret.

You could still write connection details to external secret stores like Vault by pairing Crossplane with an https://external-secrets.io PushSecret.

I believe External Secrets Operator can already do everything Crossplane's native external secret store support can do, and more. It supports more secret stores than Crossplane does. It also supports pulling external secrets, e.g. for provider credentials.

Dropping support for native external secret stores allows us to focus on other areas of the project, instead of spending our time and energy supporting things that External Secrets Operator can already do.

I believe the only remaining argument for Crossplane maintaining native external secret store support is to avoid ever writing connection details to Kubernetes secrets. While I believe this is a real constraint for some people, I don't think it's enough people to warrant Crossplane building and maintaining its own external secret store support.

Here are a few issues and articles that make the case that there's diminishing security returns in trying to completely avoid Kubernetes secrets:

• https://dev.to/canelasevero/yet-another-rant-in-favor-of-kubernetes-secrets-432k
• https://www.macchaffee.com/blog/2022/k8s-secrets/
• Clarity: secrets store CSI driver vs external secrets... what to use? external-secrets/external-secrets#478

I think the assertion that relatively few people need to strictly avoid Kubernetes secrets is backed up by the fact that there's been little community demand for and engagement with Crossplane's native external secret stores:

• Promote External Secret Stores to Beta #4164 has seen only engagement from ~5 community members.
• https://github.com/crossplane-contrib/ess-plugin-vault has seen only 3 issues and 5 PRs in the year since it was created.
• In the two years since we added external secret store support we've seen relatively few related GitHub issues on c/c.

[balorette]

I understand the desire to streamline development efforts, I believe that maintaining this feature is crucial for secure use cases, especially when integrating solutions like HashiCorp Vault.

Couple quick points on it:

Using an external secret store like Vault, we can add an additional layer of protection to sensitive connection details. This includes features such as secret injection and secret writing, which significantly enhance the security of our infrastructure. With native support, we can seamlessly integrate these security measures into our workflows, ensuring that our secrets are well-protected.

Running the control plane as a solution for end users to consume, we can conveniently write their secrets to Vault for consumption. This eliminates the need for users to manage multiple solutions and simplifies their experience. By maintaining native support, it allows customers to utilize a trusted and comprehensive solution for their security needs.

Managing multiple solutions for a single domain (IaC provisioning) to handle external secrets can be cumbersome and inefficient. While using another solution (External Secret Store Operation) may provide additional integration, it is more of a pattern to consider. By maintaining native external secret store support, we eliminate the need for an additional product solely to secure Crossplane. This reduction in the number of products needed not only simplifies our infrastructure but also reduces the potential for misconfigurations and security vulnerabilities.

Furthermore, as more customers, particularly those in areas with a high focus on security, consider using our product, the integration of external secrets into solutions like HashiCorp Vault will become an imperative feature. Customers in highly regulated businesses seek a first-class integration that allows them to leverage the advanced security capabilities of Vault without the need to manage multiple solutions.

[negz]

@balorette I totally get the desire to integrate with external systems like Vault.

I think this is clear to you already given what you wrote, but I want to reiterate that that would still be possible. This would change how that's done, not that it can be done.

In fact, ESO supports integrating with many more stores than Crossplane. Today our alpha ESS supports only Vault, while ESO supports Vault while ESO lists 26 external secret stores in their documentation, including AWS Secrets Manager and Azure Key Vault. ESO also supports pulling secrets from external stores like Vault, not just pushing them.

Managing multiple solutions for a single domain (IaC provisioning) to handle external secrets can be cumbersome and inefficient. While using another solution (External Secret Store Operation) may provide additional integration, it is more of a pattern to consider.

I wonder if this would incur meaningfully more operational burden relative to our current ESS integration? I don't think what we offer at the moment is very streamlined. e.g. Compare:

• https://docs.crossplane.io/knowledge-base/integrations/vault-as-secret-store/
• https://external-secrets.io/latest/provider/hashicorp-vault/

Today with ESS I believe the flow is:

1. Use Helm to install the Vault agent sidecar injector
2. Use Helm to install Crossplane's Vault ESS Plugin
3. Update Crossplane to set the --enable-external-secret-stores flag
4. Add a DeploymentRuntimeConfig for each Provider to set the --enable-external-secret-stores flag
5. Create a VaultConfig to configure Crossplane's Vault ESS Plugin to talk to Vault
6. Create a StoreConfig to configure Crossplane to talk to its ESS Plugin
7. Create a StoreConfig for each provider, to configure it to talk to Crossplane's ESS Plugin
8. Create claims/XRs/MRs that write connection details to Vault by setting spec.publishConnectionDetailsTo

With ESO there's more (and simpler) authentication options, but assuming you use the Vault agent sidecar, the flow is:

1. Use Helm to install the Vault agent sidecar injector
2. Use Helm to install ESO
3. Create a ClusterSecretStore to configure ESO to talk to Vault ¹
4. Create claims/XRs/MRs that write connection details to a Secret by setting spec.writeConnectionSecretToRef
5. Push secrets to Vault by creating a PushSecret.

Footnotes

1. ESO doesn't use the Vault agent sidecar injector. Instead it supports various ways of authenticating to Vault, including Kubernetes service accounts, AWS IRSA, and reading a Vault token from a secret. ↩

[negz]

Push secrets to Vault by creating a PushSecret.

One shortcoming of ESO is that you currently need to create a PushSecret for each secret you want to push to an external secret store like Vault.

There's a desire to support pushing all secrets from a namespace, per external-secrets/external-secrets#3183.

I think pushing based on label and type would be convenient too. All Crossplane connection secrets are of type: connection.crossplane.io/v1alpha1. So ideally you could easily configure "push all Crossplane connection secrets (in this namespace)" to Vault.

https://github.com/crossplane/crossplane-runtime/blob/v1.15.1/pkg/resource/resource.go#L41

I'd be happy to contribute this to ESO if it was the difference between having to maintain our own ESS implementation or not.

[haarchri]

@negz @turkenh think we need to work on this Deprecation Notice when we want to drop ESS support https://github.com/crossplane/crossplane/pull/2940/files#diff-40b5c705a7ba08a514b14b156d26120fcd0c6a8b4326096ceecbb7d3bd971f29R47

// This field is planned to be replaced in a future release in favor of
// PublishConnectionDetailsWithStoreConfigRef. Currently, both could be
// set independently and connection details would be published to both
// without affecting each other as long as related fields at MR level
// specified.
// +optional
WriteConnectionSecretsToNamespace *string json:"writeConnectionSecretsToNamespace,omitempty"

// PublishConnectionDetailsWithStoreConfig specifies the secret store config
// with which the connection secrets of composite resource dynamically
// provisioned using this composition will be published.
turkenh marked this conversation as resolved.
// +optional
// +kubebuilder:default={"name": "default"}
PublishConnectionDetailsWithStoreConfigRef *xpv1.Reference json:"publishConnectionDetailsWithStoreConfigRef,omitempty"

cc: @ytsarev

[shuknk8s]

We have strict security requirements to push the generated secrets directly to external secret manager like vault NOT to create insecure kubernetes secrets for connection details and passwords(ex rds , aurora db) in crossplane cluster and then push. Hence not supporting ESS is going to negatively impact us and we may have to discard crossplane/upbound as solution. ESO can only push existing insecure k8s secret to external secret manager store so it is not at all a secure solution, I would urge you to support ESS, ESO pushsecret is not an equivalent solution to ESS. ESO requires an existing kubernetes secrets in the cluster to push to externakl secret store , ESS does not need that by design which we really appreciate from cyber security stand point

[wlwwu]

I want to know whether we can add support of writing connection details to Kubernetes secrets with annotation and label. just as publishconnectionto functions.

[github-actions]

Crossplane does not currently have enough maintainers to address every issue and pull request. This issue has been automatically marked as stale because it has had no activity in the last 90 days. It will be closed in 14 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.