Dynamic resource references

[a-hilaly]

The ACK ecosystem currently supports referencing resources from the same or different GVKs within a static implementation. For example, you can reference an s3::bucket in a lambda::function object. However, this implementation does not allow for the referencing of multiple resource types.

Ideally, we would like to expand this feature to support the referencing of any object managed by ACK controllers. This would allow for resources like AWS EventBridge Rule targets to reference a wide range of resources, such as sns::topic and sqs::queue.

The ACK ecosystem's current ability to reference resources is a new feature that was developed in response to a recent problem. @embano1 and I will continue exploring and working on a design document for this feature, which will involve refactoring the original resource reference implementation to allow for the referencing of multiple resource types. This will require moving the generation logic to the code-generator/pkg/generate package and switching to the k8s dynamic client.

[a-hilaly]

/cc @jaypipes @azpaulp @jljaco @RedbackThomson

[embano1]

+1 to simplify the code base (eventually) using dynamic client and make it easier to compose resources using TypedLocalObjectReferences.

For example, as an EventBridge controller user I want to reference an SQS queue as a target which is also managed by ACK:

apiVersion: eventbridge.services.k8s.aws/v1alpha1
kind: Rule
metadata:
  name: rule-01
  namespace: namespace-01
spec:
  name: rule-01
  eventPattern: "{\"detail-type\":\"testevent\"}"
  # keep as is
  eventBusRef:
    name: eventbus-01
  # allows by arn and TypedLocalObjectReference
  targets:
    - id: someResourceFromArn
      arn: arn:aws:1234567890:someresource
    - id: queueFromRef
      objectRef:
        apiGroup: sqs.services.k8s.aws/v1alpha1
        kind: Queue
        name: queue-01

To be discussed: do we allow cross-namespace resource references (ObjectReference) or resources must be within the same namespace? I can see use cases for both.

Alternative approaches: in Knative we use duck typing on Kubernetes resources in references. E.g. we could create an Arnable reference object which is any Kubernetes object with a set arn in status:

status:
  ackResourceMetadata:
    arn: <arn>

[jaypipes]

@a-hilaly @embano1 @RedbackThomson @jljaco I think it's probably a good idea to just use the dynamic client anyway for everything... that way we can remove the need to include in the go.mod file the referenced controller libs.

[embano1]

Yup, will help with a lot of imports and deps! Kinda feels like generics :-p

[jljaco]

After some discussion, @RedbackThomson and I would like to see a more full-fledged design proposal for this feature before we can potentially slate it for future work.

@a-hilaly @jaypipes @embano1 anyone have bandwidth to come up with a more detailed proposal?

[embano1]

Happy to assist as EventBridge is going to benefit a lot from this and I brought this up with @a-hilaly based on my experience with Knative, but can't sign up for owning this AI.

[jaypipes]

After some discussion, @RedbackThomson and I would like to see a more full-fledged design proposal for this feature before we can potentially slate it for future work.

@a-hilaly @jaypipes @embano1 anyone have bandwidth to come up with a more detailed proposal?

@jljaco apologies, I don't have bandwidth to work on this one right now

[ack-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/aws-controllers-k8s/community.
/lifecycle stale

[ack-bot]

Issues go stale after 180d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 60d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/aws-controllers-k8s/community.
/lifecycle stale

[ack-bot]

Issues go stale after 180d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 60d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/aws-controllers-k8s/community.
/lifecycle stale

[ack-bot]

Stale issues rot after 60d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 60d of inactivity.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/aws-controllers-k8s/community.
/lifecycle rotten

[a-hilaly]

@AlyIbrahim @kejne @joestadler @migueleliasweb We just open sourced kro, a new composition engine for Kubernetes, which can be used with ACK as well as any custom resource definitions. kro solves the problem of dynamic references, so you won't need to use ACK references at all. :) Just instruct kro to query status fields and inject them into the spec of other resources.

[migueleliasweb]

Yeah, I just saw it actually. Thanks @a-hilaly . It looks sweet! I was just discussing this with my team!

Looks similar to https://docs.crossplane.io/latest/concepts/compositions/ (I'm sure there was some inspiration there 😅)

Do you think moving forward ACK will be ported more directly to inside kro or will it be kept its own thing and just used within the compositions?

[a-hilaly]

Do you think moving forward ACK will be ported more directly to inside kro or will it be kept its own thing and just used within the compositions?

Thank you for the feedback @migueleliasweb ! kro focuses exclusively on Kubernetes resource orchestration and composition by interacting only with the Kubernetes API. ACK and kro will continue to operate as independent projects and can be used together if needed. kro will maintain its fundamentally generic design, ensuring full compatibility with ANY Kubernetes resource (native and custom) across all the environments.

On the crossplane composition topic, indeed there are some similarities. both try to solve the same problem but using different approaches - kro mainly leverages CEL to express resource relationship + inferring the correct deployment order, and brings the concept of SimpleSchema to overcome the verbosity and complexity of authoring CRDs :). Also see awslabs/kro#135 (comment)