[Discuss] Enable synthetic source on new datasets by default

[ruflin]

Synthetic source is now part of Elasticsearch and hopefully is soon GA: elastic/elasticsearch#86603 Especially for metrics data, synthetic sources brings a wide variety of advantages. In Fleet it will soon be possible to enable synthetic source per data stream to test it out.

To migrate to synthetic source quickly I'm proposing that all new metrics datasets we create synthetic source will be enabled by default.

In case we find alignment on the above proposal it would mean:

• Update package dev docs with this recommendation
• Inform teams about the approach

@jsoriano @gizas @tommyers-elastic @lalit-satapathy Please chime in.

[jsoriano]

Depends on elastic/kibana#141211 (but integrations can already start using the source_mode flag).

[lalit-satapathy]

To migrate to synthetic source quickly I'm proposing that all new metrics datasets we create synthetic source will be enabled by default.

Sounds good for new datasets developed. I plan to close on the validation in couple of integrations to ensure no surprises in testing the integrations with the new mode.

@jsoriano, Will use the source_mode flag instead, for the testing if it is ready.

[ruflin]

Making synthetic source a default is still blocked by elastic/kibana#141211 and making the synthetic source feature GA which is not the case as of the upcoming 8.5 AFAIK. As we have now an option to enable it, lets make sure we continue testing and as soon as we are unblocked on the Fleet side and synthetic source goes GA, we can start to adopt the feature more heavily.

@jsoriano What do you think about having a hint for newly created metric datasets in elastic-package tooling or similar to push devs to use this feature when it is ready? My concern is that otherwise it will not be fully adapted or by mistake not be used.

[jsoriano]

@jsoriano What do you think about having a hint for newly created metric datasets in elastic-package tooling or similar to push devs to use this feature when it is ready?

For new data streams created with elastic-package create, it would be an option to automatically set source_mode: synthetic in the manifest. This is something we could already do, and Kibana will leverage it once elastic/kibana#141211 is ready, with the option to toggle it off if from the UI it causes problems. I have created an issue for this: elastic/elastic-package#1026.

[P1llus]

After discussion around this subject on Slack, the current situation in which synthetics is an option in the UI for ALL packages is unfortunately not going to work.

Any non-metrics package will by default not work with synthetics in its current state because we have event.original:

This field is included in all non-metrics packages.

Also, since synthetics is not supported on packages with flattened or wildcard mappings, which is pretty much all of them at some point, having this available by default is not really feasible.

I would suggest we put this behind a feature-flag for now, or make sure its only added to packages of type metrics.

[ruflin]

The issue description state all new metrics datasets so I think the part forward here is still ok? @P1llus I can update the title to make it more clear, or do I miss something?

[P1llus]

@ruflin the change was rolled out to all datasets at the moment, and after a discussion I was asked to post an update here :)

[ruflin]

Lets make sure we don't confuse the UI toggle with the discussion here which is the default behaviour for metrics dataset which are unrelated feature.

Here is a follow up discussion around the UI toggle: elastic/kibana#148317

[botelastic]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!