fix(#1980): Support data types in Kamelets

[christophd]

• Adds input/output/error data type spec to Kamelet CRD. The data type specifications provides additional information to the user such as what kind of input is required to use a Kamelet and what output is produced by the Kamelet.
• The data type specifications can be used by tooling and data type converters to improve the overall usability of Kamelets
• Deprecate former types field and EventTypeSpec

Fixes #1980

Sample source Kamelet using the new dataTypes structure:

apiVersion: camel.apache.org/v1alpha1
kind: Kamelet
metadata:
  name: event-source
  labels:
    camel.apache.org/kamelet.type: "source"
spec:
  definition:
    title: "Event"
    description: "Source uses different output data types"
    required:
      - token
    properties:
      token:
        title: Token
        description: The authentication token
        type: string
  dataTypes:
    out:
      headers:
        eventId:
          title: "Event Id"
          description: "The id of the event"
          type: string
        eventSubject:
          title: "Event Subject"
          description: "The event subject"
          type: string
      types:
        json:
          mediaType: application/json
          dependencies:
            - camel:jackson
          schema:
            # the JSON schema
        binary:
          mediaType: application/octet-stream
          headers:
            compressed:
              title: Data Compression
              type: boolean
              default: false
      default: json
  template:
    from:
      uri: "kamelet:source"
      steps:
        [...]

The dataTypes section may define message headers that the Kamelet produces. Also the section defines one to many supported data types for in/out/error. In the sample two data types are defined for the source Kamelet as an output (json and binary). Each data type specification is able to define a mediaType, additional dependencies, headers and a schema.

In a KameletBinding users may choose the output data type when referencing the Kamelet:

apiVersion: camel.apache.org/v1alpha1
kind: KameletBinding
metadata:
  name: event-binding
spec:
  source:
    ref:
      kind: Kamelet
      apiVersion: camel.apache.org/v1alpha1
      name: event-source
      dataTypes:
        out: 
          format: binary
  sink:
    ref:
      kind: Kamelet
      apiVersion: camel.apache.org/v1alpha1
      name: event-sink

The binding explicitly defines the output data type on the event-source Kamelet reference. The Kamelet source produces the output as binary accordingly.

Same applies for input data types where the Kamelet reference in a binding may choose from different supported input types of a Kamelet.

Release Note

feat: Support data types in Kamelets

[manstis]

Completely unrelated to your PR @christophd but how did you re-generate this?

I couldn't get it to generate with make generate on my PR.

[christophd]

@manstis I called

make clean
make generate
make build
make images

One of them did the trick for me 😅

[manstis]

One of them did the trick for me

LOL.. None of them re-generated it for me.. oh well... the mystery deepens.

[squakez]

make generate is the one in charge to refresh autogenerated code (which include deepcopies)

[squakez]

Nice work, thanks! I have a couple of questions though:

1. What is be the default behavior when the data type is not selected?
2. Would it be possible to include some E2E test covering the new scenario? Ie, creating a Kamelet that has some customized data type.

[christophd]

@squakez thanks for the questions

What is be the default behavior when the data type is not selected?

Each Kamelet has a default behavior. There is a default field in the dataTypes section for in/out/error and this should reflect the default behavior. Additional logic will be applied only when users define some data type in the binding.

Would it be possible to include some E2E test covering the new scenario? Ie, creating a Kamelet that has some customized data type.

Yes, of course. Many Kamelets being used in the e2e tests already use the dataTypes section (they have been using the old types section and I refactored the Kamelets to use the new structure). For the KameletBinding adding some e2e tests regarding data types will be a future task when the actual implementation follows. This PR only adds the data type specifications to the CRDs. I am going to work on the implementation in the binding as soon as this has been accepted and merged

[squakez]

Okey, then, if I understand correctly, this PR only set the specification, but it would not yet be usable, waiting for the implementation in Kamelets/Camel code. Is that right?

[christophd]

@squakez yes, right adding data types information to a Kamelet is just informational for now. Using a data type in a KameletBinding does not have any effect at the moment.

[squakez]

Okey, then, IMO we can leave this on hold while the development is not ready. I don't like have things that have no real usage as they can confuse customers trying to use it. However the spec configuration will be good to go for me once the business logic will be implemented. I will mark this as WIP if it's okey for you.

[christophd]

Added the implementation of the data types support in KameletBindings.

The idea is to allow the user to specify the data types for output/input on Kamelet references in a binding (see source and sink Kamelets in example below).

apiVersion: camel.apache.org/v1alpha1
kind: KameletBinding
metadata:
  name: event-binding
spec:
  source:
    ref:
      kind: Kamelet
      apiVersion: camel.apache.org/v1alpha1
      name: event-source
    dataTypes:
      out:
        format: cloudevents
  sink:
    ref:
      kind: Kamelet
      apiVersion: camel.apache.org/v1alpha1
      name: event-sink
    dataTypes:
      in:
        format: binary

Once the user explicitly defines the data types in the binding Camel K automatically adds respective steps (using the data-type-action Kamelet) in order to apply the data types for input/output (see generated template-flow.yaml below).

from:
  uri: kamelet:event-source/source
  steps:
    - kamelet:
        name: data-type-action/source-out
    - kamelet:
        name: data-type-action/sink-in
    - to: kamelet:event-sink/sink

This is for data types being set on source and sink Kamelet references in a binding. Also, the user is able to add data types to Kamelet reference used in the steps section of a binding:

apiVersion: camel.apache.org/v1alpha1
kind: KameletBinding
metadata:
  name: event-binding
spec:
  source:
    ref:
      kind: Kamelet
      apiVersion: camel.apache.org/v1alpha1
      name: event-source
  steps:
    - ref:
        kind: Kamelet
        apiVersion: camel.apache.org/v1alpha1
        name: log-action
      dataTypes:
        out:
          format: binary
  sink:
    ref:
      kind: Kamelet
      apiVersion: camel.apache.org/v1alpha1
      name: event-sink

This time the generated template-flow.yaml looks like this:

from:
  uri: kamelet:event-source/source
  steps:
    - pipeline:
        id: action-0-pipeline
        steps:
          - kamelet:
              name: log-action/action-0
          - kamelet:
              name: data-type-action/action-0-out
    - to: kamelet:event-sink/sink

Camel K uses a pipeline step in order to apply the data type action as an output of the log-action step.

The automatically added data type-action Kamelet steps will apply the given data format using the format name and an optional component scheme. At the moment the data type conversion logic is loaded from the camel-kamelet-utils library that holds some default data type implementations for AWS-S3, AWS-DDB and common String and binary data types (see https://github.com/apache/camel-kamelets/tree/main/library/camel-kamelets-utils/src/main/java/org/apache/camel/kamelets/utils/format/converter).

[christophd]

👀 @oscerd @lburgazzoli please have a look, too 🙏

[christophd]

FYI the YAKS tests failing because I need to release a new YAKS version supporting Kamelet data types first. This is being addressed in citrusframework/yaks#469

[squakez]

Nice work. I think that once all those changes are ready we should consider if it makes sense to promote Kamelet API from v1alpha1 to v1.