Generate Secret References for Sensitive Parameters under the spec.initProvider API tree

[ulucinar]

Description of your changes

While working on adding the User.mq resource in crossplane-contrib/provider-upjet-aws, we hit the limitation that currently we do not generate secret references for the sensitive parameters under the spec.initProvider API. Currently, one can specify an ActiveMQ broker user with crossplane-contrib/provider-upjet-aws as follows:

apiVersion: mq.aws.upbound.io/v1beta1
kind: Broker
metadata:
  name: test-broker
spec:
  forProvider:
    ...
    user:
    - passwordSecretRef:
        key: password
        name: mq-secret
        namespace: upbound-system
      username: admin

However, we cannot fully specify the broker user using the spec.initProvider API tree because there's no spec.initProvider.user[*].passwordSecretRef and a partial specification like the following:

apiVersion: mq.aws.upbound.io/v1beta1
kind: Broker
metadata:
  name: test-broker
spec:
  forProvider:
    # broker users are defined under spec.initProvider
    ...
  initProvider:
    user:
      username: admin

will not function as the required user password is missing in the user specification, which renders the spec.initProvider.user API useless.

This PR adds support for the generation of secret references for sensitive parameters under the spec.initProvider API tree. As a result, it now becomes possible to fully specify a broker user as follows:

apiVersion: mq.aws.upbound.io/v1beta1
kind: Broker
metadata:
  name: test-broker
spec:
  initProvider:
    ...
    user:
    - passwordSecretRef:
        key: password
        name: mq-secret
        namespace: upbound-system
      username: admin
  forProvider:
    ...

The PR proposes both the code generation changes & the runtime secret resolution changes:

Code generation changes:

1. Upjet now generates the secret references under the spec.initProvider API tree. As an example, Broker.mq now has spec.initProvider.user[*].passwordSecretRef as a sensitive parameter.
2. This will result in the marking of certain previously required sensitive fields as optional under the spec.forProvider API tree because they have now counterparts under the spec.initProvider subtree, i.e., they can now be specified from under spec.initProvider.
3. Upjet now generates, in zz_*_terraformed.go files, connection details mappings without the spec.forProvider prefixes in CRD paths. For example, what upjet used to generate as follows:

// GetConnectionDetailsMapping for this SharedDirectory
func (tr *SharedDirectory) GetConnectionDetailsMapping() map[string]string {
  return map[string]string{"notes": "spec.forProvider.notesSecretRef"}
}

now becomes:

// GetConnectionDetailsMapping for this SharedDirectory
func (tr *SharedDirectory) GetConnectionDetailsMapping() map[string]string {
  return map[string]string{"notes": "notesSecretRef"}
}

This is because, the mapping is now not 1:1. The sensitive value for the notes Terraform configuration argument can now be resolved from either a spec.forProvider reference or a soec.initProvider reference.

Runtime changes:

Upjet runtime now has to be able to resolve secret references from under the spec.initProvider tree and the mapping between the Terraform configuration arguments & secret references is no longer one-to-one. If both a spec.initProvider and a spec.forProvider secret reference target the same Terraform configuration argument, then the spec.forProvider's reference prevails over.

I have:

• Read and followed Upjet's contribution process.
• Run make reviewable to ensure this PR is ready for review.
• Added backport release-x.y labels to auto-backport this PR if necessary.

How has this code been tested

Manually tested via the User.users crossplane-contrib/provider-upjet-azuread resource.

[sergenyalcin]

Thanks @ulucinar LGTM!

[ulucinar]

The CLI-based external client relies on Terraform's ignore_changes lifecycle meta-argument to ignore changes to MR fields that are exclusively set in spec.initProvider and not set in spec.forProvider for drift detection. This PR only considers ignoring changes in spec.initProvider sensitive fields (i.e., secret references) for the no-fork architecture and such changes will be considered as drift in the old architecture. We can work on this problem if there's need for it. The spec.initProvider use cases are rather limited and further, not many will involve the newly added sensitive fields.

A possible implementation could use the mapping from the spec.initProvider sensitive field to the corresponding Terraform target sensitive argument and place that sensitive argument to the ignore_changes block. But let's tackle with it if there's any real use case for it.