From e19cb4f2d9ab156c0575e18098421070cc5aad92 Mon Sep 17 00:00:00 2001 From: Bharat Pasupula <123897612+bhapas@users.noreply.github.com> Date: Thu, 6 Apr 2023 13:51:44 +0200 Subject: [PATCH 1/3] x-pack/filebeat: Add Instance Id to trace filename (#35024) * Add Instance Id to trace filename * Add for CEL input * Add changelog * fix lint issues * restore lint changes * Make * placeholder * add docs * update note (cherry picked from commit 545f0f80ae1ae7b954e6a26f2cd1fe40436bb696) --- CHANGELOG.next.asciidoc | 38 +++++++++++++++++++ .../filebeat/docs/inputs/input-cel.asciidoc | 3 ++ .../docs/inputs/input-httpjson.asciidoc | 3 ++ x-pack/filebeat/input/cel/input.go | 4 ++ x-pack/filebeat/input/httpjson/input.go | 4 ++ 5 files changed, 52 insertions(+) diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc index 98700ce9414..098cb1cf681 100644 --- a/CHANGELOG.next.asciidoc +++ b/CHANGELOG.next.asciidoc @@ -43,6 +43,44 @@ https://github.com/elastic/beats/compare/v8.2.0\...main[Check the HEAD diff] *Filebeat* [Gcs Input] - Added missing locks for safe concurrency {pull}34914[34914] +- [Auditbeat System Package] Added support for Apple Silicon chips. {pull}34433[34433] +- [Azure blob storage] Changed logger field name from `container` to `container_name` so that it does not clash + with the ecs field name `container`. {pull}34403[34403] +- [GCS] Added support for more mime types & introduced offset tracking via cursor state. Also added support for + automatic splitting at root level, if root level element is an array. {pull}34155[34155] +- [httpsjon] Improved error handling during pagination with chaining & split processor {pull}34127[34127] +- [Azure blob storage] Added support for more mime types & introduced offset tracking via cursor state. {pull}33981[33981] +- Fix EOF on single line not producing any event. {issue}30436[30436] {pull}33568[33568] +- Fix handling of error in states in direct aws-s3 listing input {issue}33513[33513] {pull}33722[33722] +- Fix `httpjson` input page number initialization and documentation. {pull}33400[33400] +- Add handling of AAA operations for Cisco ASA module. {issue}32257[32257] {pull}32789[32789] +- Fix gc.log always shipped even if gc fileset is disabled {issue}30995[30995] +- Fix handling of empty array in httpjson input. {pull}32001[32001] +- Fix reporting of `filebeat.events.active` in log events such that the current value is always reported instead of the difference from the last value. {pull}33597[33597] +- Fix splitting array of strings/arrays in httpjson input {issue}30345[30345] {pull}33609[33609] +- Fix Google workspace pagination and document ID generation. {pull}33666[33666] +- Fix PANW handling of messages with event.original already set. {issue}33829[33829] {pull}33830[33830] +- Rename identity as identity_name when the value is a string in Azure Platform Logs. {pull}33654[33654] +- Fix 'requires pointer' error while getting cursor metadata. {pull}33956[33956] +- Fix input cancellation handling when HTTP client does not support contexts. {issue}33962[33962] {pull}33968[33968] +- Update mito CEL extension library to v0.0.0-20221207004749-2f0f2875e464 {pull}33974[33974] +- Fix CEL result deserialisation when evaluation fails. {issue}33992[33992] {pull}33996[33996] +- Fix handling of non-200/non-429 status codes. {issue}33999[33999] {pull}34002[34002] +- [azure-eventhub input] Switch the run EPH run mode to non-blocking {pull}34075[34075] +- [google_workspace] Fix pagination and cursor value update. {pull}34274[34274] +- Fix handling of quoted values in auditd module. {issue}22587[22587] {pull}34069[34069] +- Fixing system tests not returning expected content encoding for azure blob storage input. {pull}34412[34412] +- [Azure Logs] Fix authentication_processing_details parsing in sign-in logs. {issue}34330[34330] {pull}34478[34478] +- Prevent Elasticsearch from spewing log warnings about redundant wildcard when setting up ingest pipelines. {issue}34249[34249] {pull}34550[34550] +- Gracefully handle Windows event channel not found errors in winlog input. {issue}30201[30201] {pull}34605[34605] +- Fix the issue of `cometd` input worker getting closed in case of a network connection issue and an EOF error. {issue}34326[34326] {pull}34327[34327] +- Fix for httpjson first_response object throwing false positive errors by making it a flag based object {issue}34747[34747] {pull}34748[34748] +- Fix errors and panics due to re-used processors {pull}34761[34761] +- Add missing Basic Authentication support to CEL input {issue}34609[34609] {pull}34689[34689] +- [Gcs Input] - Added missing locks for safe concurrency {pull}34914[34914] +- Fix the ignore_inactive option being ignored in Filebeat's filestream input {pull}34770[34770] +- Fix TestMultiEventForEOFRetryHandlerInput unit test of CometD input {pull}34903[34903] +- Add input instance id to request trace filename for httpjson and cel inputs {pull}35024[35024] *Heartbeat* diff --git a/x-pack/filebeat/docs/inputs/input-cel.asciidoc b/x-pack/filebeat/docs/inputs/input-cel.asciidoc index 03859f5281d..37605964a5b 100644 --- a/x-pack/filebeat/docs/inputs/input-cel.asciidoc +++ b/x-pack/filebeat/docs/inputs/input-cel.asciidoc @@ -561,6 +561,9 @@ It is possible to log HTTP requests and responses in a CEL program to a local fi This option is enabled by setting the `resource.tracer.filename` value. Additional options are available to tune log rotation behavior. +To differentiate the trace files generated from different input instances, a placeholder `*` can be added to the filename and will be replaced with the input instance id. +For Example, `http-request-trace-*.ndjson`. + Enabling this option compromises security and should only be used for debugging. [float] diff --git a/x-pack/filebeat/docs/inputs/input-httpjson.asciidoc b/x-pack/filebeat/docs/inputs/input-httpjson.asciidoc index 92fd6d15936..87efc658dc0 100644 --- a/x-pack/filebeat/docs/inputs/input-httpjson.asciidoc +++ b/x-pack/filebeat/docs/inputs/input-httpjson.asciidoc @@ -619,6 +619,9 @@ It is possible to log httpjson requests and responses to a local file-system for This option is enabled by setting the `request.tracer.filename` value. Additional options are available to tune log rotation behavior. +To differentiate the trace files generated from different input instances, a placeholder `*` can be added to the filename and will be replaced with the input instance id. +For Example, `http-request-trace-*.ndjson`. + Enabling this option compromises security and should only be used for debugging. [float] diff --git a/x-pack/filebeat/input/cel/input.go b/x-pack/filebeat/input/cel/input.go index c0648bcfa2a..b45ff36d063 100644 --- a/x-pack/filebeat/input/cel/input.go +++ b/x-pack/filebeat/input/cel/input.go @@ -110,6 +110,10 @@ func (input) run(env v2.Context, src *source, cursor map[string]interface{}, pub ctx := ctxtool.FromCanceller(env.Cancelation) + if cfg.Resource.Tracer != nil { + cfg.Resource.Tracer.Filename = strings.ReplaceAll(cfg.Resource.Tracer.Filename, "*", env.ID) + } + client, err := newClient(ctx, cfg, log) if err != nil { return err diff --git a/x-pack/filebeat/input/httpjson/input.go b/x-pack/filebeat/input/httpjson/input.go index 2abf2d7e730..c10c2e80098 100644 --- a/x-pack/filebeat/input/httpjson/input.go +++ b/x-pack/filebeat/input/httpjson/input.go @@ -112,6 +112,10 @@ func run( stdCtx := ctxtool.FromCanceller(ctx.Cancelation) + if config.Request.Tracer != nil { + config.Request.Tracer.Filename = strings.ReplaceAll(config.Request.Tracer.Filename, "*", ctx.ID) + } + httpClient, err := newHTTPClient(stdCtx, config, log) if err != nil { return err From 167cda5fcbe415145184f0607d69306ffd10e238 Mon Sep 17 00:00:00 2001 From: Bharat Pasupula <123897612+bhapas@users.noreply.github.com> Date: Thu, 6 Apr 2023 14:00:57 +0200 Subject: [PATCH 2/3] Update CHANGELOG.next.asciidoc --- CHANGELOG.next.asciidoc | 37 ------------------------------------- 1 file changed, 37 deletions(-) diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc index 098cb1cf681..20a40aecf41 100644 --- a/CHANGELOG.next.asciidoc +++ b/CHANGELOG.next.asciidoc @@ -43,43 +43,6 @@ https://github.com/elastic/beats/compare/v8.2.0\...main[Check the HEAD diff] *Filebeat* [Gcs Input] - Added missing locks for safe concurrency {pull}34914[34914] -- [Auditbeat System Package] Added support for Apple Silicon chips. {pull}34433[34433] -- [Azure blob storage] Changed logger field name from `container` to `container_name` so that it does not clash - with the ecs field name `container`. {pull}34403[34403] -- [GCS] Added support for more mime types & introduced offset tracking via cursor state. Also added support for - automatic splitting at root level, if root level element is an array. {pull}34155[34155] -- [httpsjon] Improved error handling during pagination with chaining & split processor {pull}34127[34127] -- [Azure blob storage] Added support for more mime types & introduced offset tracking via cursor state. {pull}33981[33981] -- Fix EOF on single line not producing any event. {issue}30436[30436] {pull}33568[33568] -- Fix handling of error in states in direct aws-s3 listing input {issue}33513[33513] {pull}33722[33722] -- Fix `httpjson` input page number initialization and documentation. {pull}33400[33400] -- Add handling of AAA operations for Cisco ASA module. {issue}32257[32257] {pull}32789[32789] -- Fix gc.log always shipped even if gc fileset is disabled {issue}30995[30995] -- Fix handling of empty array in httpjson input. {pull}32001[32001] -- Fix reporting of `filebeat.events.active` in log events such that the current value is always reported instead of the difference from the last value. {pull}33597[33597] -- Fix splitting array of strings/arrays in httpjson input {issue}30345[30345] {pull}33609[33609] -- Fix Google workspace pagination and document ID generation. {pull}33666[33666] -- Fix PANW handling of messages with event.original already set. {issue}33829[33829] {pull}33830[33830] -- Rename identity as identity_name when the value is a string in Azure Platform Logs. {pull}33654[33654] -- Fix 'requires pointer' error while getting cursor metadata. {pull}33956[33956] -- Fix input cancellation handling when HTTP client does not support contexts. {issue}33962[33962] {pull}33968[33968] -- Update mito CEL extension library to v0.0.0-20221207004749-2f0f2875e464 {pull}33974[33974] -- Fix CEL result deserialisation when evaluation fails. {issue}33992[33992] {pull}33996[33996] -- Fix handling of non-200/non-429 status codes. {issue}33999[33999] {pull}34002[34002] -- [azure-eventhub input] Switch the run EPH run mode to non-blocking {pull}34075[34075] -- [google_workspace] Fix pagination and cursor value update. {pull}34274[34274] -- Fix handling of quoted values in auditd module. {issue}22587[22587] {pull}34069[34069] -- Fixing system tests not returning expected content encoding for azure blob storage input. {pull}34412[34412] -- [Azure Logs] Fix authentication_processing_details parsing in sign-in logs. {issue}34330[34330] {pull}34478[34478] -- Prevent Elasticsearch from spewing log warnings about redundant wildcard when setting up ingest pipelines. {issue}34249[34249] {pull}34550[34550] -- Gracefully handle Windows event channel not found errors in winlog input. {issue}30201[30201] {pull}34605[34605] -- Fix the issue of `cometd` input worker getting closed in case of a network connection issue and an EOF error. {issue}34326[34326] {pull}34327[34327] -- Fix for httpjson first_response object throwing false positive errors by making it a flag based object {issue}34747[34747] {pull}34748[34748] -- Fix errors and panics due to re-used processors {pull}34761[34761] -- Add missing Basic Authentication support to CEL input {issue}34609[34609] {pull}34689[34689] -- [Gcs Input] - Added missing locks for safe concurrency {pull}34914[34914] -- Fix the ignore_inactive option being ignored in Filebeat's filestream input {pull}34770[34770] -- Fix TestMultiEventForEOFRetryHandlerInput unit test of CometD input {pull}34903[34903] - Add input instance id to request trace filename for httpjson and cel inputs {pull}35024[35024] *Heartbeat* From f832672ab588047ad3a2c1fc44353aff102d53ef Mon Sep 17 00:00:00 2001 From: Bharat Pasupula <123897612+bhapas@users.noreply.github.com> Date: Thu, 6 Apr 2023 14:01:55 +0200 Subject: [PATCH 3/3] Update CHANGELOG.next.asciidoc --- CHANGELOG.next.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc index 20a40aecf41..38be0f82f63 100644 --- a/CHANGELOG.next.asciidoc +++ b/CHANGELOG.next.asciidoc @@ -43,7 +43,7 @@ https://github.com/elastic/beats/compare/v8.2.0\...main[Check the HEAD diff] *Filebeat* [Gcs Input] - Added missing locks for safe concurrency {pull}34914[34914] -- Add input instance id to request trace filename for httpjson and cel inputs {pull}35024[35024] +- Add input instance id to request trace filename for httpjson and cel inputs {pull}35037[35037] *Heartbeat*