[data views] Field editor endpoint versioning and schema validation

[mattkime]

Summary

• Move field preview to internal route
• Add versioning to route
• Endpoint is called with version
• Response schema validation

Closes #159158

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[davismcphee]

Looks good overall! But I did run into one output validation issue when testing composite runtime fields.

This is the field I was trying to create (I disabled output validation to get this screenshot):

With the validation enabled, I got this toast:

And this error response:

{
    "statusCode": 500,
    "error": "Internal Server Error",
    "message": "Failed output validation: [request body.values]: types that failed validation:\n- [request body.values.0.composite_field.test]: types that failed validation:\n - [request body.0]: expected value of type [boolean] but got [Array]\n - [request body.1]: expected value of type [number] but got [Array]\n - [request body.2]: expected value of type [string] but got [Array]\n - [request body.3]: expected a plain object value, but found [Array] instead.\n- [request body.values.1]: expected value of type [array] but got [Object]"
}

And for reference, the script I was using:

Map map = new HashMap();
map.put("first","Foo");
map.put("last", "Bar");

emit(map);
emit('map', map); 
emit('test', 'test');
emit('test2', 2555);
emit('test3', [99, 99]);

And the success response with validation disabled:

{
  "composite_field.test": ["test"],
  "composite_field.first": ["Foo"],
  "composite_field.last": ["Bar"],
  "composite_field.test2": [2555],
  "composite_field.test3": [[99, 99]],
  "composite_field.map": [
    {
      "last": "Bar",
      "first": "Foo"
    }
  ]
}

[mattkime]

@davismcphee I need to talk to the painless engineers - either you've found an undocumented feature or a bug. Either way, passing maps and arrays to emit isn't documented - https://www.elastic.co/guide/en/elasticsearch/painless/8.8/painless-runtime-fields-context.html

[davismcphee]

@mattkime I think calling emit with a map is supported for composite runtime fields, it's just very poorly documented. It looks like the feature was originally implemented here: elastic/elasticsearch#75108. And the documentation here shows a map being passed to emit as a result of grok().extract(), but that was the only (vague) mention of it that I could find in the documentation.

I'm not sure if emit('key', map) is considered valid since I couldn't find any references in the documentation, I just tried it in testing and it seemed to work.

The array in this case is actual a geo_point field, and it also seems to work as intended in the Discover grid with validation disabled:

To be fair though, I couldn't find explicit mention of using geo_point values in composite fields in any of our documentation either. I just landed on the emit('key', [lat, lon]) syntax through trial and error. Presumably this is an alternate syntax for emit(lat, lon) reserved for composite fields since the usual syntax would conflict with the composite field emit('key', value) syntax, but maybe this isn't intentional and geo_point values shouldn't be supported in composite fields.

Either way though, composite fields seem to fail validation regardless of which values are used. I used complex types in my example to be comprehensive in testing, but even a composite field containing just primitive types currently fails validation. I believe it's because we're expecting Record<string, ValueType> here, but the painless API seems to return Record<string, ValueType[]> instead:

kibana/src/plugins/data_view_field_editor/server/routes/field_preview.ts

Lines 45 to 58 in a1b6664

	response: {
	200: {
	body: schema.object({
	values: schema.oneOf([
	// composite field
	schema.recordOf(schema.string(), valueSchema),
	// primitive field
	schema.arrayOf(valueSchema),
	]),
	error: schema.maybe(schema.object({}, { unknowns: 'allow' })),
	status: schema.maybe(schema.number()),
	}),
	},
	},

[mattkime]

@davismcphee I spoke to some ES engineers and the consensus was that passing an array or map to emit('fieldName', value) isn't valid. I've created a issue against the ES repo - elastic/elasticsearch#97126

Past that, everything should work now.

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 47fc40f

Metrics [docs]

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
dataViewFieldEditor	23.3KB	23.3KB	+13.0B

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
enterpriseSearch	14	16	+2
securitySolution	413	417	+4
total			+6

Total ESLint disabled count

id	before	after	diff
enterpriseSearch	15	17	+2
securitySolution	492	496	+4
total			+6

History

• 💚 Build #138319 succeeded 6067848
• 💚 Build #136365 succeeded a1b6664
• 💔 Build #136363 failed e1a9dca
• 💚 Build #135925 succeeded bc6de5d
• 💔 Build #135427 failed 4edb27b
• 💔 Build #135253 failed 4027116

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @mattkime

[davismcphee]

Thanks for clarifying which subfield types are supported! Since maps and arrays aren't valid, everything seems to be working as expected, LGTM 👍