Bug: field.set processor can not set a new field at the .Payload level

[tsinghgill]

Bug description

We are trying to set the field field: ".Payload.Data" however it throws an error. According to the docs this processor will create the field if it does not exist. The below configuration does not work.

processors: - id: add-data-field plugin: "field.set" settings: field: ".Payload.Data" value: "{{.Payload.After}}"

We tested this configuration locally, where we set field: ".Payload.After.Data" and this works:

processors: - id: add-data-field plugin: "field.set" settings: field: ".Payload.After.Data" value: "{{.Payload.After}}"

Steps to reproduce

Set this configuration in your pipeline, essentially cloning the .Payload.After to .Payload.Data:

processors: - id: add-data-field plugin: "field.set" settings: field: ".Payload.Data" value: "{{.Payload.After}}"

Version

0.10.1 darwin/arm64

[lovromazgon]

We can't do that, the record will always have the structure of an opencdc record. We can't add an arbitrary field anywhere in the record, that only applies to .Payload.After, .Payload.Before and .Key, as only those fields can contain structured data.

In other words, the fields in an OpenCDC record are always the same, can't be removed, renamed or new fields attached. You should look at it as the envelope that carries the actual data.

What you might be able to do is use the template record formatter, which lets you use a Go template and be very flexible with what is actually written into the destination: https://conduit.io/docs/connectors/output-formats/