pipeline object?

[willemdh]

Atm I'm using an on_failure processor on pipeline filters, which logs errors to pipeline_error.

Wouldn't it be useful to have an ecs object pipeline which could hold:

pipeline.name
pipeline.error
pipeline.type (logstash|ingest)

or is there already a place in ecs for these kind of data?

[webmat]

I totally agree!

I see a bigger discussion that needs to happen for this. Here's some things I want to look into:

• Pipeline errors (as opposed to "userland" errors), make sure Ingest Node pipelines don't overwrite legit error.message fields :-) And of course additional metadata as you suggest here.
• Better support for complex pipeline metadata (multiple agents: syslog => beats => logstash => es). Right now I think agent/observer is too rigid
• Pipeline step timings. The "seen" timestamp of everyone in the chain, in an example like above.
• Looping back on the first point: do we need the ability to track pipeline errors for the various steps in the pipeline?

There's likely other things I'm forgetting as well :-)

[MikePaquette]

@willemdh @webmat Is it the general rule that the pipeline info is piggybacked onto an existing event, rather than included in a separate out-of-band event?

[webmat]

Yeah it should be added to the same event. However in doing so, there's a risk of overwriting information from the initial log.

Say for example you're ingesting exceptions:

• You put your exception message at error.message (aka the "userland" error)
• An error in the ingest node pipeline overwrites error.message with the ingest node error (pipeline error)
• insert sad face in your event ;-)

[webmat]

So what I'm outlining above is that I want to make it clearer where pipeline metadata should go, vs the original event's data.

[MikePaquette]

Yep, I agree with the approach. Sounds like a pipeline field set could be a good solution.

[webmat]

We created meta-issue #940 to discuss capturing pipeline details in ECS. Closing in favor of the meta issue.