Skip to content

Commit

Permalink
[Fleet] Remove subseconds from event.ingested
Browse files Browse the repository at this point in the history
The `event.ingested` field is added to all documents ingested via
Fleet plus Agent. By removing the subseconds we can be better compression
of the values in Elasticsearch.

The primary user of `event.ingested` today is the the Security Detection Engine
as a tie-breaker in search_after, but once it moves to the using the
point-in-time API the need for precision will be lessened because PIT has
an implicit tie-breaker.

Relates elastic#103944
Relates elastic/beats#22388
  • Loading branch information
andrewkroh committed Jul 1, 2021
1 parent c78c350 commit d72151e
Show file tree
Hide file tree
Showing 2 changed files with 27 additions and 1 deletion.
8 changes: 7 additions & 1 deletion x-pack/plugins/fleet/server/constants/fleet_es_assets.ts
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ export const FLEET_GLOBAL_COMPONENT_TEMPLATE_CONTENT = {
properties: {
ingested: {
type: 'date',
format: 'strict_date_time_no_millis||strict_date_optional_time||epoch_millis',
},
agent_id_status: {
ignore_above: 1024,
Expand All @@ -42,7 +43,12 @@ processors:
- set:
description: Add time when event was ingested.
field: event.ingested
value: '{{{_ingest.timestamp}}}'
copy_from: _ingest.timestamp
- script:
description: Remove sub-seconds from event.ingested to improve storage efficiency.
tag: truncate-subseconds-event-ingested
source: ctx.event.ingested = ctx.event.ingested.withNano(0).format(DateTimeFormatter.ISO_OFFSET_DATE_TIME);
ignore_failure: true
- remove:
description: Remove any pre-existing untrusted values.
field:
Expand Down
20 changes: 20 additions & 0 deletions x-pack/test/fleet_api_integration/apis/epm/final_pipeline.ts
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,26 @@ export default function (providerContext: FtrProviderContext) {
);
});

it('all docs should contain event.ingested without sub-seconds', async () => {
const res = await es.index({
index: 'logs-log.log-test',
body: {
'@timestamp': '2020-01-01T09:09:00',
message: 'hello',
},
});

const { body: doc } = await es.get({
id: res.body._id,
index: res.body._index,
});
// @ts-expect-error
const ingestTimestamp = doc._source.event.ingested;

// 2021-06-30T12:06:28Z
expect(ingestTimestamp).to.match(/^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z$/);
});

it('For a doc written without api key should write the correct api key status', async () => {
const res = await es.index({
index: 'logs-log.log-test',
Expand Down

0 comments on commit d72151e

Please sign in to comment.