[Transform] Add _meta field to TransformConfig

[przemekwitek]

This PR adds a new field (called _meta) to the TransformConfig.
This _meta field stores an arbitrary key-value map.
Keys are strings.
Values are arbitrary objects (possibly also maps).
The _meta map is updatable via _update endpoint in Transform API.

Relates #77506

[elasticmachine]

Pinging @elastic/ml-core (Team:ML)

[hendrikmuhs]

some comments:

• we should add a mapping(FLATTENED) to TransformInternalIndex, this is not strictly required as dynamic mappings are turned off, but we do the same for other fields -> consistency
• it would be nice to not allow arbitrary sized meta entries, the other parts of the config are somewhat limited by nature (assuming no one is writing a monster aggregation), but meta is unlimited
  • however that's hard to achieve with the current parser
  • we could as alternative limit the Rest Request (RestPutTransformAction)
• +1 regarding the idea of using LinkedHashMap to preserve order

[przemekwitek]

some comments:

we should add a mapping(FLATTENED) to TransformInternalIndex, this is not strictly required as dynamic mappings are
turned off, but we do the same for other fields -> consistency

Done.

it would be nice to not allow arbitrary sized meta entries, the other parts of the config are somewhat limited by nature
(assuming no one is writing a monster aggregation), but meta is unlimited
however that's hard to achieve with the current parser
we could as alternative limit the Rest Request (RestPutTransformAction)

And also RestUpdateTransformAction, right?
What limit do you think makes sense here? Any prior art?

+1 regarding the idea of using LinkedHashMap to preserve order

Done.

[droberts195]

I would expect this to cause an assertion failure when used in conjunction with mapOrdered if we have good test coverage:

elasticsearch/server/src/main/java/org/elasticsearch/common/io/stream/StreamOutput.java

Line 536 in 39234ab

assert false == (map instanceof LinkedHashMap);

For a linked hash map just writeMap should be fine.

[benwtrent]

Correct, I think this is probably fine, it iterates the keys and since LinkedHashMap has consistent order, that will work.

On reading though, this means that you cannot use readMap, you will need to readOrderedMap or whatever.

[droberts195]

LGTM, if you could just fix a couple of field names in the docs

[hendrikmuhs]

And also RestUpdateTransformAction, right?
What limit do you think makes sense here? Any prior art?

no prior art as far as I know.

I think if we limit to 5MB we still allow a lot of room for super large in-config painless scripts.

Note, I am not concerned about a single put or update request. The real issue is getting the configs in some internal places where we sometimes retrieve 100, sometimes up to 10k at a time.

[przemekwitek]

run elasticsearch-ci/part-2

[przemekwitek]

I think if we limit to 5MB we still allow a lot of room for super large in-config painless scripts.

Ok. I've added a limit of 5MB to both Put and Update. I've manually verified (by temporarily lowering the limit) that an exception when the request is too large.

[hendrikmuhs]

LGTM