[Logs+] Default pipeline for logs data streams #95971
Conversation
eyalkoren
added
>enhancement
:Data Management/Data streams
|
Pinging @elastic/es-data-management (Team:Data Management) |
elasticsearchmachine
added
Team:Data Management
|
Hi @eyalkoren, I've created a changelog YAML for you. |
…pipelines are added
…nto default-timestamp-for-logs
x-pack/plugin/core/src/main/resources/logs-default-pipeline.json
Outdated
Show resolved
Hide resolved
...eams/src/yamlRestTest/resources/rest-api-spec/test/data_stream/220_logs_default_pipeline.yml
Outdated
Show resolved
Hide resolved
|
@jbaiera Please verify that my fix for |
| "set": { | ||
| "description": "If '@timestamp' is missing, set it with the ingest timestamp", | ||
| "field": "@timestamp", | ||
| "override": false, | ||
| "copy_from": "_ingest.timestamp" | ||
| } |
There was a problem hiding this comment.
@joegallo FYI this is how we're setting a default timestamp. We'll want to have millisecond precision so we're not truncating here. As the @timestamp field is mapped as a date, not date_nanos, we don't need to truncate microseconds and nanoseconds, I think.
There was a problem hiding this comment.
A set w/ copy_from is very fast. 👍 and ❤️
| "pipeline" : { | ||
| "name": "logs@custom", | ||
| "ignore_missing_pipeline": true, | ||
| "description": "A custom pipeline for logs data streams, which does not exist by default" | ||
| } |
There was a problem hiding this comment.
@joshdover @kpollich FYI: the logs-*-* index template will contain this default logs pipeline that calls out a missing logs@custom extension point.
x-pack/plugin/core/src/main/resources/logs-default-pipeline.json
Outdated
Show resolved
Hide resolved
Co-authored-by: James Baiera <[email protected]>
Closes #95551
Closes #95537