[Logs+] Add pipeline that parses JSON log events into top-level fields

[eyalkoren]

Closes #95522

Note for reviewer

I added a validation for pipeline-dependencies, similar to the validation we have for composable index templates, so that a pipeline can be installed only if the pipelines it refers to are already installed.
Due to the complexities related to fully-automated resolution of pipeline dependencies, I simplified the solution by letting each registry manually define the required dependencies. I think this makes sense, as the pipelines required by another pipeline are part of the specific information known by the declaring registry, similar to the pipeline's ID and its related file.

NOTE that the validation doesn't care for versions of pipeline dependencies, so it is satisfied if the required pipelines are installed, based on their IDs, regardless of their version. I think this is a sufficient requirement, since it prevents race condition and it doesn't affect ingested data consistency in rolling upgrades (meaning- if the referred pipeline changes, ingested documents may be inconsistent either way).

[felixbarny]

There shouldn't be parsing-related rejections.

Right, it's "just" mapping issues that could happen as a result of adding more fields to the mapping that are parsed out of the JSON message.

As long as we don't map anything explicitly to object types, does this PR increase risk related to subobjects: false?

What I meant is that the JSON may contain conflicting keys, such as "foo" and "foo.bar" or that some of the keys in the JSON conflict with metadata fields added by Filebeat, such as "host" vs "host.name".

Maybe this is what you mean but we could for now already add the JSON pipeline with the registry but not call it from the logs-*-* pipeline. So if users want to use it, they can "opt-in" by calling it from the logs@custom pipeline.

It would be a trivial change to call the JSON pipeline by default once we have more safety measures in place.

[eyalkoren]

Maybe this is what you mean but we could for now already add the JSON pipeline with the registry but not call it from the logs-- pipeline. So if users want to use it, they can "opt-in" by calling it from the logs@custom pipeline.

Yes, that is what I meant. For logs users and for anyone else that is interested in this capability.
EDIT: sorry, this is specific to logs atm as it only looks for the message field. Maybe there's a way to generalize it somehow, but definitely out of scope at this stage.

[ruflin]

Maybe we can split this up into 2 phases. First have this "json" pipeline in place. Think of it like the ecs templates that can easily embeded and we keep optimising. In a second phase, we can enable it by default. By then, the pipeline is also tested. Or instead, we might make it a config option on the data stream.

[ruflin]

We need to discuss the naming of this pipeline. Because at the moment it would conflict with a pipeline for the dataset: json and namespace: pipeline. We need to come up with a non conflicting naming convention for these global asset. Ideally the component templates and pipelines follow the same logic. @kpollich @joshdover You might have ideas here too.

[joshdover]

What's the point of this being a separate pipeline at all? Are we going to be reusing it elsewhere?

[eyalkoren]

@joshdover I think this separation just proved to be useful with the concerns raised by @felixbarny - it will now allow us to remove it from the default pipeline, but still allow users to easily opt in by calling it from the logs@custom pipeline

[eyalkoren]

Demonstrated now in this PR's test

[eyalkoren]

I think I don't have any more input for this.
@ruflin @felixbarny let me know once you decide on the followings, so I can bring this to completion-

• do we merge it, or block until other safety valves are in place?
• if merging- do we disable it by default for now?
• how to name the pipeline?

[felixbarny]

do we merge it, or block until other safety valves are in place?
if merging- do we disable it by default for now?

Let's merge it but make it opt-in for now.

how to name the pipeline?

Taking inspiration from pipeline names used in the elastic/integrations repo. I'd suggest pipeline-json-message.

[eyalkoren]

The pipeline is now disabled by default, with easy opt-in option, as the test shows.
So we are only left with a decision about pipeline naming - @ruflin

Taking inspiration from pipeline names used in the elastic/integrations repo. I'd suggest pipeline-json-message.

I think it doesn't match very well to all other config files around it

[felixbarny]

Because at the moment it would conflict with a pipeline for the dataset: json and namespace: pipeline.

I think that's more of a theoretical concern, isn't it? Package-provided pipelines have the structure <type>-<dataset>-<version>, so I don't see the conflict with logs-json-pipeline. But if we still want to avoid that association, we should probably prefix pipelines that we set up in ES with pipeline-, such as pipeline-logs-json or pipeline-json-message.

[ruflin]

I think that's more of a theoretical concern, isn't it?

Not sure that is why I'm not comfortable with it. All the current integrations have a version prefixed, but what about the integrations that are built in Kibana? I rather be safe on this one.

I expect over the coming months we keep adding some more reusable assets to Elasticsearch like ecs templates, other ingest pipelines. For a user, it should be very easy to understand that these are assets loaded by the system and globally available, ideally easy to remember names / convention. As this is the first asset of this kind that makes it in, we should come up with the convention.

Here some thoughts:

• We don't need the pipeline part in the name, it is already a pipeline
• Should we prefix with the type these things are for? logs@json? If for all signals, signals@json? {type}@{processor}
  • How do the ECS templates play into this? logs@ecs-core, ecs-core@signals. But this is broader? ecs@system, ecs@managed ?
• The above should work for assets with a version identifier and without one. For ECS mappings, I expect these to have a version, but some base templates / pipelines not.

[eyalkoren]

As agreed offline- we'll change to logs@json-message and follow up with a proper definition of the builtin pipeline naming convention