be able to pass annotations to the generated pods

[davidkarlsen]

See #251.
I want to be able to set annotations on the pods generated from the trigger mesh custom resources, in order to do istio side-car injection on openshift. This is only possible through annotating the pods when using openshift, not via namespace labelling.

[antoineco]

@davidkarlsen I've been thinking about this a little bit more and had an idea I wanted to share with you.

Just passing annotations wouldn't fit all scenarios, because we cannot know for sure whether the user wants to have those annotations set on:

• both the Deployment metadata and Pod template
• just the Pod template
• just the Deployment
• which Pod(s), if the event source runs more than one Pod (we have that case in another repo)
• a mixture of that: what annotation is set where?

Besides, in the future someone may require the injection of extra environment variables (http_proxy is just one example), or want to customize container-level resources and limits, etc. and this doesn't fit in the abstraction offered by CRDs anymore.

Kubernetes used to have something called PodPresets for this kind of use case, but it was removed recently 😞. The current "recommended" way to inject extra configuration into arbitrary objects upon creation/update is to create an admission controller (https://docs.giantswarm.io/guides/creating-your-own-admission-controller/), but this is quite involved just for adding an annotation...

---

My proposal is that we implement a configuration object that can be used to bind annotations to any Kubernetes object: Deployments, Pods, custom objects... The spec could look like this (it is very much inspired by https://github.com/mattmoor/bindings):

apiVersion: settings.triggermesh.io/v1
kind: AnnotationsBinding
metadata:
  name: istio-injection

spec:

  # 'subject' selects what object(s) the binding applies to, either by name or label selector
  subject:
    apiVersion: v1
    kind: Pod
    selector:
      matchLabels:
        # applies to all AWS event sources
        app.kubernetes.io/part-of: aws-event-sources
        # applies to all SQS event sources
        app.kubernetes.io/name: awssqssource
        # applies to a specific SQS event source
        app.kubernetes.io/instance: my-source

  # 'values' is a map of annotations to inject into the object(s) selected by 'subject'
  values:
    sidecar.istio.io/inject: 'true'
    other.istio.annotation: some value

By keeping the labels selector broad, you could have all your Pods covered (not only TriggerMesh's) with a single binding inside the namespace.

Example: if the Pod that runs the SQS source is created like this:

apiVersion: v1
kind: Pod
metadata:
  name: awssqssource-my-source
  labels:
    app.kubernetes.io/part-of: aws-event-source
    app.kubernetes.io/name: awssqssource
    app.kubernetes.io/instance: my-source
spec:
  containers:
  - names: adapter
    env:
    - name: ARN
      value: arn:aws:sqs:us-east-2:123456789012:my-queue
    - name: K_SINK
      value: http://...
    # ...

then the binding will ensure the following annotations are always present:

apiVersion: v1
kind: Pod
metadata:
  name: awssqssource-my-source
  labels:
    app.kubernetes.io/part-of: aws-event-source
    app.kubernetes.io/name: awssqssource
    app.kubernetes.io/instance: my-source
    
    # injected by the binding
    annotations:
      sidecar.istio.io/inject: 'true'
      other.istio.annotation: some value

spec:
  containers:
  - names: adapter
    env:
    - name: ARN
      value: arn:aws:sqs:us-east-2:123456789012:my-queue
    - name: K_SINK
      value: http://...
    # ...

Does that sound like something that could help with your use case, or is the overhead not worth the effort compared to deploying Pods manually?

[davidkarlsen]

That design sounds good, however I have researched the istio capabilities in this space, and it appears it can't do it (yet, at least, envoyproxy/envoy#11308 was just recently merged to support it) - so for now I think I just need to go with bare pods.

BTW: I found the ContainerSource api https://knative.dev/docs/eventing/samples/container-source/ - it's seems a bit more native than bare pods, do you support running like this (i.e. will the image respect the sink configuration)? Still quite new to k-native.

[antoineco]

BTW: I found the ContainerSource api [...] do you support running like this?

We do!

https://github.com/triggermesh/aws-event-sources/blob/master/config/samples/awssqs-containersource.yaml

Just replace latest with the current release version, which is v1.1.0, set your ARN, and ensure the reference to the secret containing your credentials is correct (we support the format value: my-api-key instead of valueFrom if you prefer passing credentials directly in the spec).

[davidkarlsen]

Yay - that worked! :)
But I notice the body/payload is base64 encoded when I receive it in my service, while it's sent as plain text/json?

[antoineco]

Hmm, that shouldn't be the case since the body of SQS messages are plain strings (as opposed to raw bytes).

Here is what the source generates for the JSON payload {"msg": "Hello, world!}

☁️  cloudevents.Event
Validation: valid
Context Attributes,
  specversion: 1.0
  type: com.amazon.sqs.message
  source: arn:aws:sqs:us-west-2:123456789012:antoineco
  subject: AIDAQUHRFMYWSR2PNVTVZ
  id: b2e63f6a-0542-44c2-858a-6946b9183c72
  time: 2021-01-18T17:25:39.284582652Z
  datacontenttype: application/json
Data,
  {
    "Attributes": {
      "ApproximateFirstReceiveTimestamp": "1610990739235",
      "ApproximateReceiveCount": "1",
      "SenderId": "AIDAQUHRFMYWSR2PNVTVZ",
      "SentTimestamp": "1610990739219"
    },
    "Body": "{\"msg\": \"Hello, world!\"}",
    "MD5OfBody": "4e1903c62c593f658fd238bc20522d4d",
    "MD5OfMessageAttributes": null,
    "MessageAttributes": null,
    "MessageId": "b2e63f6a-0542-44c2-858a-6946b9183c72",
    "ReceiptHandle": "AQEBIuKaF3ivdEH/XW0zvWOD/IOZ2xz4UCtxH20d1PkwI0TWbwJFf6MVleSzLrTwdMuFovmeg321MLXVQDVp7OumBfrb1iqCuITEeBei9K8An8eTMblZlRxkrlWPdXnbmpLGsnEniiwnPobnkwZNLU35ab7V1JJRLShZ8E8/G6XwkMoy6X4eK0eWobRN5hoFB877MyL9obp7Ke9kEWQNDyWTAP7FUF14EuGqOfRdcghWUJXF+/J1pfcMSV1z8mVan5Wh7Q17kqG8HzBacr8cisI+wsMTmGeYuqK4ogmaxQxwzoDev/uTHXeiGrTWSHVLBUu53MCVtYKt9BmgJrD84O3MaScUmuw9Np4OlNafySlGCliucq2k9BQAXWsTwlOfdCfQQvroYiQfZtpRrnlWep+Bew=="
  }

If you think that is a bug, would you mind opening a new issue with a description of what is happening, and the expected behaviour?

[davidkarlsen]

Doh - I was fooled by the logging which outputs it b64 encoded when presented as a byte[]. Everything works like it should.

[antoineco]

Correct, []byte is often base64-encoded when serialized to JSON instead of being converted to a string, because JSON can't handle raw bytes, and the contents could contain binary characters.

Glad it works!