From 3584dfb5e500843f41ba153294aaf8f636ad7374 Mon Sep 17 00:00:00 2001 From: Felix Barnsteiner Date: Wed, 26 Aug 2020 11:59:32 +0200 Subject: [PATCH 1/6] Add transaction_ignore_urls spec --- specs/agents/tracing-instrumentation-http.md | 26 ++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/specs/agents/tracing-instrumentation-http.md b/specs/agents/tracing-instrumentation-http.md index c89c5129..98f0d2a7 100644 --- a/specs/agents/tracing-instrumentation-http.md +++ b/specs/agents/tracing-instrumentation-http.md @@ -35,6 +35,32 @@ Request and response headers, cookies, and form bodies should be sanitised (i.e. Agents may may include additional patterns if there are common conventions specific to language frameworks. +##### `transaction_ignore_urls` configuration + +Used to restrict requests to certain URLs from being instrumented. + +This property should be set to a list containing one or more strings. +When an incoming HTTP request is detected, +its request path will be tested against each element in this list. +For example, adding `/home/index` to this list would match and remove instrumentation from the following URLs: + +``` +https://www.mycoolsite.com/home/index +http://localhost/home/index +http://whatever.com/home/index?value1=123 +``` + +NOTE: +All errors that are captured during a request to an ignored URL are still sent to the APM Server regardless of this setting. + +| | | +|----------------|---| +| Type | `List` | +| Default | agent specific | +| Dynamic | `true` | +| Central config | `true` | + +Agents should test against this common set of test cases to ensure interoperability: [`wildcard_matcher_tests.json`](../../tests/agents/json-specs/wildcard_matcher_tests.json) #### HTTP client spans From 56d18114fc033189fd9ea05309d088ef8eadca00 Mon Sep 17 00:00:00 2001 From: Felix Barnsteiner Date: Wed, 26 Aug 2020 12:07:40 +0200 Subject: [PATCH 2/6] Link to wildcard matcher tests on type definition --- specs/agents/tracing-instrumentation-http.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/specs/agents/tracing-instrumentation-http.md b/specs/agents/tracing-instrumentation-http.md index 98f0d2a7..ac7fbb00 100644 --- a/specs/agents/tracing-instrumentation-http.md +++ b/specs/agents/tracing-instrumentation-http.md @@ -55,13 +55,11 @@ All errors that are captured during a request to an ignored URL are still sent t | | | |----------------|---| -| Type | `List` | +| Type | `List<`[`WildcardMatcher`](../../tests/agents/json-specs/wildcard_matcher_tests.json)`>` | | Default | agent specific | | Dynamic | `true` | | Central config | `true` | -Agents should test against this common set of test cases to ensure interoperability: [`wildcard_matcher_tests.json`](../../tests/agents/json-specs/wildcard_matcher_tests.json) - #### HTTP client spans We capture spans for outbound HTTP requests. These should have a type of `external`, and subtype of `http`. The span name should have the format ` `. From 73bc50083f36841ce7a1ab746688284728552337 Mon Sep 17 00:00:00 2001 From: Benjamin Wohlwend Date: Thu, 3 Sep 2020 10:53:47 +0200 Subject: [PATCH 3/6] add link to `path` definition in RFC3986 --- specs/agents/tracing-instrumentation-http.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/specs/agents/tracing-instrumentation-http.md b/specs/agents/tracing-instrumentation-http.md index ac7fbb00..a6d9296b 100644 --- a/specs/agents/tracing-instrumentation-http.md +++ b/specs/agents/tracing-instrumentation-http.md @@ -41,7 +41,8 @@ Used to restrict requests to certain URLs from being instrumented. This property should be set to a list containing one or more strings. When an incoming HTTP request is detected, -its request path will be tested against each element in this list. +its request [`path`](https://tools.ietf.org/html/rfc3986#section-3.3) +will be tested against each element in this list. For example, adding `/home/index` to this list would match and remove instrumentation from the following URLs: ``` From c9c1ca08d3a2c2186cb4d45b244edf8000481fe7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Juan=20=C3=81lvarez?= Date: Mon, 14 Sep 2020 09:33:48 +0200 Subject: [PATCH 4/6] Update outcome spec (#338) --- specs/agents/tracing-spans.md | 2 ++ specs/agents/tracing-transactions.md | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/specs/agents/tracing-spans.md b/specs/agents/tracing-spans.md index 66a0dc76..2028fb9d 100644 --- a/specs/agents/tracing-spans.md +++ b/specs/agents/tracing-spans.md @@ -10,6 +10,8 @@ The only semantic difference is that client errors set the `outcome` to `"failur Agents should try to determine the outcome for spans created by auto instrumentation, which is especially important for exit spans (spans representing requests to other services). +If an agent doesn't report the `outcome` (or reports `null`), the APM Server will set it based on `context.response.status_code`. If the status code is not available, then it will be set to `"unknown"`. + While the transaction outcome lets you reason about the error rate from the service's point of view, other services might have a different perspective on that. For example, if there's a network error so that service A can't call service B, diff --git a/specs/agents/tracing-transactions.md b/specs/agents/tracing-transactions.md index 53e16222..e9cf7f30 100644 --- a/specs/agents/tracing-transactions.md +++ b/specs/agents/tracing-transactions.md @@ -11,7 +11,7 @@ Transactions are created either by the built-in auto-instrumentation or an agent The `outcome` property denotes whether the transaction represents a success or a failure from the perspective of the entity that produced the event. The APM Server converts this to the [`event.outcome`](https://www.elastic.co/guide/en/ecs/current/ecs-allowed-values-event-outcome.html) field. This property is optional to preserve backwards compatibility. -If an agent doesn't report the `outcome` (or reports `null`), the APM Server sets the outcome to `"unknown"`. +If an agent doesn't report the `outcome` (or reports `null`), the APM Server will set it based on `context.http.response.status_code`. If the status code is not available, then it will be set to `"unknown"`. - `"failure"`: Indicates that this transaction describes a failed result. \ Note that client errors (such as HTTP 4xx) don't fall into this category as they are not an error from the perspective of the server. From a5c6b0afa55f3ef2c8c488db75b25c36f5a20a6e Mon Sep 17 00:00:00 2001 From: Felix Barnsteiner Date: Tue, 15 Sep 2020 15:18:22 +0200 Subject: [PATCH 5/6] Add sanitize_field_names spec (#334) --- specs/agents/README.md | 1 + specs/agents/sanitization.md | 16 ++++++++++++++++ 2 files changed, 17 insertions(+) create mode 100644 specs/agents/sanitization.md diff --git a/specs/agents/README.md b/specs/agents/README.md index 315b96d6..efd8a226 100644 --- a/specs/agents/README.md +++ b/specs/agents/README.md @@ -51,6 +51,7 @@ You can find details about each of these in the [APM Data Model](https://www.ela - [Metrics](metrics.md) - [Logging Correlation](log-correlation.md) - [Agent Configuration](configuration.md) +- [Data sanitization](sanitization.md) # Processes diff --git a/specs/agents/sanitization.md b/specs/agents/sanitization.md new file mode 100644 index 00000000..7ca0ce85 --- /dev/null +++ b/specs/agents/sanitization.md @@ -0,0 +1,16 @@ +## Data sanitization + +### `sanitize_field_names` configuration + +Sometimes it is necessary to sanitize, i.e., remove, +sensitive data sent to Elastic APM. +This config accepts a list of wildcard patterns of field names which should be sanitized. +These apply to HTTP headers (including cookies) and `application/x-www-form-urlencoded` data (POST form fields). +The query string and the captured request body (such as `application/json` data) will not get sanitized. + +| | | +|----------------|---| +| Type | `List<`[`WildcardMatcher`](../../tests/agents/json-specs/wildcard_matcher_tests.json)`>` | +| Default | `password, passwd, pwd, secret, *key, *token*, *session*, *credit*, *card*, authorization, set-cookie` | +| Dynamic | `true` | +| Central config | `true` | From 3c0076f4f880a8e8d4526881ccaedd223f8649a3 Mon Sep 17 00:00:00 2001 From: eyalkoren <41850454+eyalkoren@users.noreply.github.com> Date: Wed, 16 Sep 2020 15:26:02 +0300 Subject: [PATCH 6/6] Update metrics.md (#339) --- specs/agents/metrics.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/specs/agents/metrics.md b/specs/agents/metrics.md index 83b8544b..6bd2dd0e 100644 --- a/specs/agents/metrics.md +++ b/specs/agents/metrics.md @@ -19,19 +19,16 @@ Where applicable, all agents (excluding JavaScript RUM) should record the follow - `system.process.cgroup.memory.mem.limit.bytes` - `system.process.cgroup.memory.mem.usage.bytes` - - `system.process.cgroup.memory.stats.inactive_file.bytes` #### Metrics source ##### [cgroup-v1](https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt) - `system.process.cgroup.memory.mem.limit.bytes` - based on the `memory.limit_in_bytes` file - `system.process.cgroup.memory.mem.usage.bytes` - based on the `memory.usage_in_bytes` file - - `system.process.cgroup.memory.stats.inactive_file.bytes` - based on the `total_inactive_file` line in the `memory.stat` file ##### [cgroup-v2](https://www.kernel.org/doc/Documentation/cgroup-v2.txt) - `system.process.cgroup.memory.mem.limit.bytes` - based on the `memory.max` file - `system.process.cgroup.memory.mem.usage.bytes` - based on the `memory.current` file - - `system.process.cgroup.memory.stats.inactive_file.bytes` - based on the `inactive_file` line in the `memory.stat` file #### Discovery of the memory files