Discuss: validate default value for constant_keyword

[mtojek]

Pipeline tests ignore changes around default field values: elastic/package-spec#194 .

I assume that this is another entrypoint for data, which previously have been set only by Metricbeat or Filebeat. Now this data is set once the ingest pipeline finishes its job and just before the event is stored in Elasticsearch. In the future we have to figure out a workaround for this.

System tests don't use any expected data files. It looks like a field for improvement.

cc @andrewkroh @jsoriano .

[andrewkroh]

Validation for constant_keyword seems similar to the validation that we would need for runtime fields. In order to access values for these types of fields we would want to index the documents and then query for fields as opposed to only looking at _source.

Perhaps this would be a new type of test? I like that pipeline tests are very fast today and simply check that the pipeline transforms the input into the _source output. It might be nice to have something that's similar to pipeline testing in that it reads from a file, but instead sends data into the data stream (like system tests) and then validates an expected data file against a query that uses fields.

BTW there is one small piece of validation that checks if the field does exist in the _source that it matches the declared value. But that sort of usage should be atypical because if you declare value in the mapping then you normally don't want to also set it in the _source.

elastic-package/internal/fields/validate.go

Line 303 in 2278b9e

if err := ensureConstantKeywordValueMatches(key, valStr, definition.Value); err != nil {

[mtojek]

I was thinking about improving system tests by marking event.dataset and event.module as mandatory and simply checking them (and their values). In this case the only drawback would be necessity of having system tests for which the coverage is not superb :) On the other hand it might be a quick win.

EDIT:

At the moment it would be great to decide (confirm) if ALL integrations must set these fields forever or is it a temporary workaround.