-
Notifications
You must be signed in to change notification settings - Fork 8.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Fleet] Support local routing rules during integration installation #155910
Comments
We need to discuss the names and the order of the different ingest pipelines. Data streams managed by fleet have a
It seems like the suggestion is to add routing rules directly to the default pipeline of the integration. While this is possible, there are a few things to watch out for when it comes to ordering and we should at least discuss whether to create dedicated routing pipelines. Option 1: put package-defined routing rules in the default pipeline
Option 2: dedicated pipeline for package-provided routing rules We could put all package-provided routing rules in a dedicated pipeline. This may make it easier to show all package-provided routing rules in a routing rules UI.
Option 2a: dedicated pipeline for custom routing rules Similar to the above, a custom pipeline that's dedicated to just routing rules might make it easier for a routing rules UI to display and add user-provided routing rules. The downside is that this UI would then have a lot of assumptions about the pipeline names and there may be routing rules in other pipelines that it would either not show or are read-only.
|
@felixbarny - Thanks for providing that summary. I think leaving room for user customizations is definitely an important consideration here. However, this issue is only concerning itself with the "package-sourced" rules implementation. I think it's good to future proof our plans here to make sure we're not going to collide with any follow-up implementations around user-provided routing rules. To that end, I think Option 1 that you've listed is still the most compelling to me. Introducing new pipelines and naming schemes means we need to carefully document and tutorialize these customization practices, whereas by simply appending routing rules after the existing One thing I want to avoid is a user opening their integration's pipeline definition and being overwhelmed by how many specially named processors, pipelines, etc exist and not being able to quickly add customizations. The simplicity of a single The organization and neatness around grouping various processors into separate pipelines seems compelling from a technical standpoint, but from an end-user perspective I worry it'd be noisy and needlessly complex. I'm in favor of keeping things simple here, especially for our MVP implementation. Unless there are any objections, I'll update the description here to capture the ordering of the |
hey @kpollich if I understand correctly this comment
should be changed to
since the dynamic dataset. Can you please confirm that since I am adding the same comment in my connected PR. thanks |
@nchaulet once you start working on this it would be great to kick an initial call with the observability folks in order to get a broad understanding of this. |
Ref elastic/package-spec#514
When integrations are installed, Fleet should honor any local routing rules defined by the integration. This means that when an integration defines a "catch-all" dataset that routes data to that integration's own data streams, the resulting ingest pipeline for the "catch-all" dataset contains
reroute
processors as appropriate.For example, when an
nginx
data stream is defined as followsA resulting
logs-nginx-1.2.3
ingest pipeline should be generated that includes the followingprocessors
:Note: This issue does not include the support of "injected" routing rules that are defined on non-local datasets. That will be implemented in a follow-up captured by #157422.
Acceptance Criteria
routing_rule
defined in a data stream manifest, a correspondingreroute
processor is generated on the resulting ingest pipeline that routes to the provideddataset
value on the defined rule@custom
pipeline processor is called BEFORE thereroute
processors we generate - this ensures user customizations and routing overrides can be provided if neededtag
value set to the datastream name from which the routing rule was sourcednamespace
andif
value provided by the ruleThe text was updated successfully, but these errors were encountered: