From e531896ad436948cfbe8a1ca16594dd8815bd998 Mon Sep 17 00:00:00 2001 From: Phillip Carter Date: Mon, 5 Aug 2024 04:55:48 +0000 Subject: [PATCH 1/6] Restructure the logs page --- content/en/docs/concepts/signals/logs.md | 160 ++++++++++++++++++----- 1 file changed, 128 insertions(+), 32 deletions(-) diff --git a/content/en/docs/concepts/signals/logs.md b/content/en/docs/concepts/signals/logs.md index e186218d0c62..b901ead3cb51 100644 --- a/content/en/docs/concepts/signals/logs.md +++ b/content/en/docs/concepts/signals/logs.md @@ -5,31 +5,135 @@ description: A recording of an event. --- A **log** is a timestamped text record, either structured (recommended) or -unstructured, with metadata. Of all telemetry signals, logs have the biggest +unstructured, with optional metadata. Of all telemetry signals, logs have the biggest legacy. Most programming languages have built-in logging capabilities or -well-known, widely used logging libraries. Although logs are an independent data -source, they may also be attached to spans. In OpenTelemetry, any data that is -not part of a distributed trace or a metric is a log. For example, _events_ are -a specific type of log. Logs often contain detailed debugging/diagnostic info, -such as inputs to an operation, the result of the operation, and any supporting -metadata for that operation. +well-known, widely used logging libraries. -For traces and metrics, OpenTelemetry takes the approach of a clean-sheet -design, specifies a new API, and provides full implementations of this API in -multiple language SDKs. +## OpenTelemetry Logs -OpenTelemetry's approach with logs is different. Because existing logging -solutions are widespread in language and operational ecosystems, OpenTelemetry -acts as a "bridge" between those logs, the tracing and metrics signals, and -other OpenTelemetry components. In fact, the API for logs is called the "Logs -Bridge API" for this reason. +OpenTelemetry does not define a bespoke API or SDK to create logs. Instead, OpenTelemetry logs +are simply the existing logs you already have from a logging framework or infrastructure component. Under the covers, OpenTelemetry SDKs and autoinstrumentation utilize several components to automatically correlate logs with [traces](/docs/signals/traces). -The [logs specification][] contains more details on this philosophy. +OpenTelemetry's support for logs is designed to be maximally compatible with what you already have, providing capabilities to wrap those logs with additional context and a common toolkit to parse and manipulate logs into a common format across many different sources. -To understand how logging in OpenTelemetry works, let's look at a list of -components that will play a part in instrumenting our code. +### OpenTelemetry Logs in the OpenTelemetry Collector -## Log Appender / Bridge +The [OpenTelemetry Collector](/docs/collector) provides several tools to work with logs: + +- Several receivers which parse logs from specific, known sources of log data +- The filelogreceiver, which reads logs from any file and provides features to parse them from different formats or use a regex +- Processors like the transformprocessor which lets you parse nested data, flatten nested structures, add/remove/update values, and more +- Exporters that let you emit log data in a non-OpenTelemetry format + +Often times, the first step in adopting OpenTelemetry involves deploying a Collector as a general-purposes logging agent. + +### OpenTelemetry Logs for Applications + +In applications, OpenTelemetry logs are created with any logging library or built-in logging capabilities. When you add autoinstrumentation or activate an SDK, OpenTelemetry will automatically correlate your existing logs with any active trace and span, wrapping the log body with their IDs. In other words, OpenTelemetry will automatically correlate your logs! + +### Language Support + +Logs are a [stable](/docs/specs/otel/versioning-and-stability/#stable) signal in +the OpenTelemetry specification. For the individual language specific +implementations of the Logs API & SDK, the status is as follows: + +{{% signal-support-table "logs" %}} + +## Structured, Unstructured, and Semistructured Logs + +OpenTelemetry does not distinguish between structured and unstructured logs. However, it is vitally important to understand the different kinds of formats that log data can manifest in. + +### Structured logs + +A structured log is a log whose textual format follows a consistent, machine-readable +format. Two of are JSON: + +```json +{ + "timestamp": "2024-08-04T12:34:56.789Z", + "level": "INFO", + "service": "user-authentication", + "environment": "production", + "message": "User login successful", + "context": { + "userId": "12345", + "username": "johndoe", + "ipAddress": "192.168.1.1", + "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36" + }, + "transactionId": "abcd-efgh-ijkl-mnop", + "duration": 200, + "request": { + "method": "POST", + "url": "/api/v1/login", + "headers": { + "Content-Type": "application/json", + "Accept": "application/json" + }, + "body": { + "username": "johndoe", + "password": "******" + } + }, + "response": { + "statusCode": 200, + "body": { + "success": true, + "token": "jwt-token-here" + } + } +} +``` + +and for infrastructure components, Common Log Format (CLF): + +``` +127.0.0.1 - johndoe [04/Aug/2024:12:34:56 -0400] "POST /api/v1/login HTTP/1.1" 200 1234 +``` + +It is also common to have different structured log formats mixed together. For example, and Extended Log Format (ELF) log mixes JSON with the whitespace-separated data in a CLF log. + +``` +192.168.1.1 - johndoe [04/Aug/2024:12:34:56 -0400] "POST /api/v1/login HTTP/1.1" 200 1234 "http://example.com" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36" {"transactionId": "abcd-efgh-ijkl-mnop", "responseTime": 150, "requestBody": {"username": "johndoe"}, "responseHeaders": {"Content-Type": "application/json"}} +``` + +To make the most use of this log, it is recommended to parse bothe the JSON and the ELF-related pieces into a shared format to make analysis on an Observability backend easier. + +Structured logs are the preferred way to use logs. Because they are emitted in a consistent format, they are easier to parse, which makes them easier to preprocess in an OpenTelemetry Collector, correlate with other data, and ultimate analyze in an Observability backend. Use structured logs if you can. + +### Unstructured Logs + +Unstructured logs are logs that don't follow a consistent structure. They may be more human-readable, and are often used in development. However, it is not preferred to use unstructured logs to contain useful debugging data if you can help it. + +Examples of unstructured logs: + +``` +[ERROR] 2024-08-04 12:45:23 - Failed to connect to database. Exception: java.sql.SQLException: Timeout expired. Attempted reconnect 3 times. Server: db.example.com, Port: 5432 + +System reboot initiated at 2024-08-04 03:00:00 by user: admin. Reason: Scheduled maintenance. Services stopped: web-server, database, cache. Estimated downtime: 15 minutes. + +DEBUG - 2024-08-04 09:30:15 - User johndoe performed action: file_upload. Filename: report_Q3_2024.pdf, Size: 2.3 MB, Duration: 5.2 seconds. Result: Success +``` + +To reiterate, structured logs are fine for debugging applications locally, but are very difficult to use at scale when observing large systems. It is recommended that you use structured logging if possible. + +### Semistructured logs + +A semistructured log is a log that does use some self-consistent patterns to distinguish data so that it's machine-readable, but may not use the same formatting and delimiters between data across different systems. + +Example of a semistructured log: + +``` +2024-08-04T12:45:23Z level=ERROR service=user-authentication userId=12345 action=login message="Failed login attempt" error="Invalid password" ipAddress=192.168.1.1 userAgent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36" +``` + +Although machine-readable, semistructured logs may require several different parsers to allow for analysis at scale. + +## OpenTelemetry Logging Componentry + +The following lists of concepts/components are what powers OpenTelemetry's logging support under the hood. + +### Log Appender / Bridge As an application developer, the **Logs Bridge API** should not be called by you directly, as it is provided for logging library authors to build log appenders / @@ -39,7 +143,7 @@ OpenTelemetry LogRecordExporter. OpenTelemetry language SDKs offer this functionality. -## Logger Provider +### Logger Provider > Part of the **Logs Bridge API** and should only be used if you are the author > of a logging library. @@ -49,20 +153,20 @@ A Logger Provider (sometimes called `LoggerProvider`) is a factory for lifecycle matches the application's lifecycle. Logger Provider initialization also includes Resource and Exporter initialization. -## Logger +### Logger > Part of the **Logs Bridge API** and should only be used if you are the author > of a logging library. A Logger creates log records. Loggers are created from Log Providers. -## Log Record Exporter +### Log Record Exporter Log Record Exporters send log records to a consumer. This consumer can be standard output for debugging and development-time, the OpenTelemetry Collector, or any open source or vendor backend of your choice. -## Log Record +### Log Record A log record represents the recording of an event. In OpenTelemetry a log record contains two kinds of fields: @@ -89,15 +193,7 @@ The top-level fields are: For more details on log records and log fields, see [Logs Data Model](/docs/specs/otel/logs/data-model/). -## Language Support - -Logs are a [stable](/docs/specs/otel/versioning-and-stability/#stable) signal in -the OpenTelemetry specification. For the individual language specific -implementations of the Logs API & SDK, the status is as follows: - -{{% signal-support-table "logs" %}} - -## Specification +### Specification To learn more about logs in OpenTelemetry, see the [logs specification][]. From 3aa69e838932b38bf34e0de5738c89fbdf2de907 Mon Sep 17 00:00:00 2001 From: Phillip Carter Date: Mon, 5 Aug 2024 04:58:19 +0000 Subject: [PATCH 2/6] appease the robots --- content/en/docs/concepts/signals/logs.md | 74 +++++++++++++++++------- 1 file changed, 53 insertions(+), 21 deletions(-) diff --git a/content/en/docs/concepts/signals/logs.md b/content/en/docs/concepts/signals/logs.md index b901ead3cb51..07a874912d8c 100644 --- a/content/en/docs/concepts/signals/logs.md +++ b/content/en/docs/concepts/signals/logs.md @@ -5,31 +5,45 @@ description: A recording of an event. --- A **log** is a timestamped text record, either structured (recommended) or -unstructured, with optional metadata. Of all telemetry signals, logs have the biggest -legacy. Most programming languages have built-in logging capabilities or +unstructured, with optional metadata. Of all telemetry signals, logs have the +biggest legacy. Most programming languages have built-in logging capabilities or well-known, widely used logging libraries. ## OpenTelemetry Logs -OpenTelemetry does not define a bespoke API or SDK to create logs. Instead, OpenTelemetry logs -are simply the existing logs you already have from a logging framework or infrastructure component. Under the covers, OpenTelemetry SDKs and autoinstrumentation utilize several components to automatically correlate logs with [traces](/docs/signals/traces). +OpenTelemetry does not define a bespoke API or SDK to create logs. Instead, +OpenTelemetry logs are simply the existing logs you already have from a logging +framework or infrastructure component. Under the covers, OpenTelemetry SDKs and +autoinstrumentation utilize several components to automatically correlate logs +with [traces](/docs/signals/traces). -OpenTelemetry's support for logs is designed to be maximally compatible with what you already have, providing capabilities to wrap those logs with additional context and a common toolkit to parse and manipulate logs into a common format across many different sources. +OpenTelemetry's support for logs is designed to be maximally compatible with +what you already have, providing capabilities to wrap those logs with additional +context and a common toolkit to parse and manipulate logs into a common format +across many different sources. ### OpenTelemetry Logs in the OpenTelemetry Collector -The [OpenTelemetry Collector](/docs/collector) provides several tools to work with logs: +The [OpenTelemetry Collector](/docs/collector) provides several tools to work +with logs: - Several receivers which parse logs from specific, known sources of log data -- The filelogreceiver, which reads logs from any file and provides features to parse them from different formats or use a regex -- Processors like the transformprocessor which lets you parse nested data, flatten nested structures, add/remove/update values, and more +- The filelogreceiver, which reads logs from any file and provides features to + parse them from different formats or use a regex +- Processors like the transformprocessor which lets you parse nested data, + flatten nested structures, add/remove/update values, and more - Exporters that let you emit log data in a non-OpenTelemetry format -Often times, the first step in adopting OpenTelemetry involves deploying a Collector as a general-purposes logging agent. +Often times, the first step in adopting OpenTelemetry involves deploying a +Collector as a general-purposes logging agent. ### OpenTelemetry Logs for Applications -In applications, OpenTelemetry logs are created with any logging library or built-in logging capabilities. When you add autoinstrumentation or activate an SDK, OpenTelemetry will automatically correlate your existing logs with any active trace and span, wrapping the log body with their IDs. In other words, OpenTelemetry will automatically correlate your logs! +In applications, OpenTelemetry logs are created with any logging library or +built-in logging capabilities. When you add autoinstrumentation or activate an +SDK, OpenTelemetry will automatically correlate your existing logs with any +active trace and span, wrapping the log body with their IDs. In other words, +OpenTelemetry will automatically correlate your logs! ### Language Support @@ -41,12 +55,14 @@ implementations of the Logs API & SDK, the status is as follows: ## Structured, Unstructured, and Semistructured Logs -OpenTelemetry does not distinguish between structured and unstructured logs. However, it is vitally important to understand the different kinds of formats that log data can manifest in. +OpenTelemetry does not distinguish between structured and unstructured logs. +However, it is vitally important to understand the different kinds of formats +that log data can manifest in. ### Structured logs -A structured log is a log whose textual format follows a consistent, machine-readable -format. Two of are JSON: +A structured log is a log whose textual format follows a consistent, +machine-readable format. Two of are JSON: ```json { @@ -91,19 +107,29 @@ and for infrastructure components, Common Log Format (CLF): 127.0.0.1 - johndoe [04/Aug/2024:12:34:56 -0400] "POST /api/v1/login HTTP/1.1" 200 1234 ``` -It is also common to have different structured log formats mixed together. For example, and Extended Log Format (ELF) log mixes JSON with the whitespace-separated data in a CLF log. +It is also common to have different structured log formats mixed together. For +example, and Extended Log Format (ELF) log mixes JSON with the +whitespace-separated data in a CLF log. ``` 192.168.1.1 - johndoe [04/Aug/2024:12:34:56 -0400] "POST /api/v1/login HTTP/1.1" 200 1234 "http://example.com" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36" {"transactionId": "abcd-efgh-ijkl-mnop", "responseTime": 150, "requestBody": {"username": "johndoe"}, "responseHeaders": {"Content-Type": "application/json"}} ``` -To make the most use of this log, it is recommended to parse bothe the JSON and the ELF-related pieces into a shared format to make analysis on an Observability backend easier. +To make the most use of this log, it is recommended to parse bothe the JSON and +the ELF-related pieces into a shared format to make analysis on an Observability +backend easier. -Structured logs are the preferred way to use logs. Because they are emitted in a consistent format, they are easier to parse, which makes them easier to preprocess in an OpenTelemetry Collector, correlate with other data, and ultimate analyze in an Observability backend. Use structured logs if you can. +Structured logs are the preferred way to use logs. Because they are emitted in a +consistent format, they are easier to parse, which makes them easier to +preprocess in an OpenTelemetry Collector, correlate with other data, and +ultimate analyze in an Observability backend. Use structured logs if you can. ### Unstructured Logs -Unstructured logs are logs that don't follow a consistent structure. They may be more human-readable, and are often used in development. However, it is not preferred to use unstructured logs to contain useful debugging data if you can help it. +Unstructured logs are logs that don't follow a consistent structure. They may be +more human-readable, and are often used in development. However, it is not +preferred to use unstructured logs to contain useful debugging data if you can +help it. Examples of unstructured logs: @@ -115,11 +141,15 @@ System reboot initiated at 2024-08-04 03:00:00 by user: admin. Reason: Scheduled DEBUG - 2024-08-04 09:30:15 - User johndoe performed action: file_upload. Filename: report_Q3_2024.pdf, Size: 2.3 MB, Duration: 5.2 seconds. Result: Success ``` -To reiterate, structured logs are fine for debugging applications locally, but are very difficult to use at scale when observing large systems. It is recommended that you use structured logging if possible. +To reiterate, structured logs are fine for debugging applications locally, but +are very difficult to use at scale when observing large systems. It is +recommended that you use structured logging if possible. ### Semistructured logs -A semistructured log is a log that does use some self-consistent patterns to distinguish data so that it's machine-readable, but may not use the same formatting and delimiters between data across different systems. +A semistructured log is a log that does use some self-consistent patterns to +distinguish data so that it's machine-readable, but may not use the same +formatting and delimiters between data across different systems. Example of a semistructured log: @@ -127,11 +157,13 @@ Example of a semistructured log: 2024-08-04T12:45:23Z level=ERROR service=user-authentication userId=12345 action=login message="Failed login attempt" error="Invalid password" ipAddress=192.168.1.1 userAgent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36" ``` -Although machine-readable, semistructured logs may require several different parsers to allow for analysis at scale. +Although machine-readable, semistructured logs may require several different +parsers to allow for analysis at scale. ## OpenTelemetry Logging Componentry -The following lists of concepts/components are what powers OpenTelemetry's logging support under the hood. +The following lists of concepts/components are what powers OpenTelemetry's +logging support under the hood. ### Log Appender / Bridge From eb1bc9d634d40693bf8dc39d1a7f5478bd5efb6e Mon Sep 17 00:00:00 2001 From: Phillip Carter Date: Mon, 5 Aug 2024 05:08:40 +0000 Subject: [PATCH 3/6] appease te robots more --- content/en/docs/concepts/signals/logs.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/content/en/docs/concepts/signals/logs.md b/content/en/docs/concepts/signals/logs.md index 07a874912d8c..e01a0378d0f8 100644 --- a/content/en/docs/concepts/signals/logs.md +++ b/content/en/docs/concepts/signals/logs.md @@ -29,7 +29,7 @@ with logs: - Several receivers which parse logs from specific, known sources of log data - The filelogreceiver, which reads logs from any file and provides features to - parse them from different formats or use a regex + parse them from different formats or use a regular expression - Processors like the transformprocessor which lets you parse nested data, flatten nested structures, add/remove/update values, and more - Exporters that let you emit log data in a non-OpenTelemetry format @@ -103,7 +103,7 @@ machine-readable format. Two of are JSON: and for infrastructure components, Common Log Format (CLF): -``` +```text 127.0.0.1 - johndoe [04/Aug/2024:12:34:56 -0400] "POST /api/v1/login HTTP/1.1" 200 1234 ``` @@ -111,7 +111,7 @@ It is also common to have different structured log formats mixed together. For example, and Extended Log Format (ELF) log mixes JSON with the whitespace-separated data in a CLF log. -``` +```text 192.168.1.1 - johndoe [04/Aug/2024:12:34:56 -0400] "POST /api/v1/login HTTP/1.1" 200 1234 "http://example.com" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36" {"transactionId": "abcd-efgh-ijkl-mnop", "responseTime": 150, "requestBody": {"username": "johndoe"}, "responseHeaders": {"Content-Type": "application/json"}} ``` @@ -133,7 +133,7 @@ help it. Examples of unstructured logs: -``` +```text [ERROR] 2024-08-04 12:45:23 - Failed to connect to database. Exception: java.sql.SQLException: Timeout expired. Attempted reconnect 3 times. Server: db.example.com, Port: 5432 System reboot initiated at 2024-08-04 03:00:00 by user: admin. Reason: Scheduled maintenance. Services stopped: web-server, database, cache. Estimated downtime: 15 minutes. @@ -153,7 +153,7 @@ formatting and delimiters between data across different systems. Example of a semistructured log: -``` +```text 2024-08-04T12:45:23Z level=ERROR service=user-authentication userId=12345 action=login message="Failed login attempt" error="Invalid password" ipAddress=192.168.1.1 userAgent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36" ``` From fe66ca2ec02f12fa026987a8e90dcf8ddb025418 Mon Sep 17 00:00:00 2001 From: Phillip Carter Date: Mon, 5 Aug 2024 06:51:43 -0700 Subject: [PATCH 4/6] Apply suggestions from code review Co-authored-by: Fabrizio Ferri-Benedetti --- content/en/docs/concepts/signals/logs.md | 36 +++++++++++------------- 1 file changed, 17 insertions(+), 19 deletions(-) diff --git a/content/en/docs/concepts/signals/logs.md b/content/en/docs/concepts/signals/logs.md index e01a0378d0f8..9159f3045511 100644 --- a/content/en/docs/concepts/signals/logs.md +++ b/content/en/docs/concepts/signals/logs.md @@ -12,12 +12,12 @@ well-known, widely used logging libraries. ## OpenTelemetry Logs OpenTelemetry does not define a bespoke API or SDK to create logs. Instead, -OpenTelemetry logs are simply the existing logs you already have from a logging -framework or infrastructure component. Under the covers, OpenTelemetry SDKs and +OpenTelemetry logs are the existing logs you already have from a logging +framework or infrastructure component. OpenTelemetry SDKs and autoinstrumentation utilize several components to automatically correlate logs with [traces](/docs/signals/traces). -OpenTelemetry's support for logs is designed to be maximally compatible with +OpenTelemetry's support for logs is designed to be fully compatible with what you already have, providing capabilities to wrap those logs with additional context and a common toolkit to parse and manipulate logs into a common format across many different sources. @@ -27,14 +27,14 @@ across many different sources. The [OpenTelemetry Collector](/docs/collector) provides several tools to work with logs: -- Several receivers which parse logs from specific, known sources of log data +- Several receivers which parse logs from specific, known sources of log data. - The filelogreceiver, which reads logs from any file and provides features to - parse them from different formats or use a regular expression + parse them from different formats or use a regular expression. - Processors like the transformprocessor which lets you parse nested data, - flatten nested structures, add/remove/update values, and more -- Exporters that let you emit log data in a non-OpenTelemetry format + flatten nested structures, add/remove/update values, and more. +- Exporters that let you emit log data in a non-OpenTelemetry format. -Often times, the first step in adopting OpenTelemetry involves deploying a +The first step in adopting OpenTelemetry frequently involves deploying a Collector as a general-purposes logging agent. ### OpenTelemetry Logs for Applications @@ -43,7 +43,7 @@ In applications, OpenTelemetry logs are created with any logging library or built-in logging capabilities. When you add autoinstrumentation or activate an SDK, OpenTelemetry will automatically correlate your existing logs with any active trace and span, wrapping the log body with their IDs. In other words, -OpenTelemetry will automatically correlate your logs! +OpenTelemetry automatically correlates your logs and traces. ### Language Support @@ -115,8 +115,8 @@ whitespace-separated data in a CLF log. 192.168.1.1 - johndoe [04/Aug/2024:12:34:56 -0400] "POST /api/v1/login HTTP/1.1" 200 1234 "http://example.com" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36" {"transactionId": "abcd-efgh-ijkl-mnop", "responseTime": 150, "requestBody": {"username": "johndoe"}, "responseHeaders": {"Content-Type": "application/json"}} ``` -To make the most use of this log, it is recommended to parse bothe the JSON and -the ELF-related pieces into a shared format to make analysis on an Observability +To make the most use of this log, parse both the JSON and +the ELF-related pieces into a shared format to make analysis on an observability backend easier. Structured logs are the preferred way to use logs. Because they are emitted in a @@ -127,9 +127,7 @@ ultimate analyze in an Observability backend. Use structured logs if you can. ### Unstructured Logs Unstructured logs are logs that don't follow a consistent structure. They may be -more human-readable, and are often used in development. However, it is not -preferred to use unstructured logs to contain useful debugging data if you can -help it. +more human-readable, and are often used in development. Examples of unstructured logs: @@ -141,9 +139,9 @@ System reboot initiated at 2024-08-04 03:00:00 by user: admin. Reason: Scheduled DEBUG - 2024-08-04 09:30:15 - User johndoe performed action: file_upload. Filename: report_Q3_2024.pdf, Size: 2.3 MB, Duration: 5.2 seconds. Result: Success ``` -To reiterate, structured logs are fine for debugging applications locally, but -are very difficult to use at scale when observing large systems. It is -recommended that you use structured logging if possible. +Structured logs are fine for debugging applications locally, but +are very difficult to use at scale when observing large systems. +Use structured logging if possible. ### Semistructured logs @@ -162,8 +160,8 @@ parsers to allow for analysis at scale. ## OpenTelemetry Logging Componentry -The following lists of concepts/components are what powers OpenTelemetry's -logging support under the hood. +The following lists of concepts and components power OpenTelemetry's +logging support. ### Log Appender / Bridge From 84f9489b6296024d4556a93d6719cd41ad8f23c2 Mon Sep 17 00:00:00 2001 From: Phillip Carter Date: Mon, 5 Aug 2024 14:31:32 +0000 Subject: [PATCH 5/6] updates --- content/en/docs/concepts/signals/logs.md | 60 +++++++++++++++--------- 1 file changed, 37 insertions(+), 23 deletions(-) diff --git a/content/en/docs/concepts/signals/logs.md b/content/en/docs/concepts/signals/logs.md index 9159f3045511..0e8165cd6dfc 100644 --- a/content/en/docs/concepts/signals/logs.md +++ b/content/en/docs/concepts/signals/logs.md @@ -9,20 +9,20 @@ unstructured, with optional metadata. Of all telemetry signals, logs have the biggest legacy. Most programming languages have built-in logging capabilities or well-known, widely used logging libraries. -## OpenTelemetry Logs +## OpenTelemetry logs OpenTelemetry does not define a bespoke API or SDK to create logs. Instead, OpenTelemetry logs are the existing logs you already have from a logging framework or infrastructure component. OpenTelemetry SDKs and autoinstrumentation utilize several components to automatically correlate logs -with [traces](/docs/signals/traces). +with [traces](/docs/concepts/signals/traces). OpenTelemetry's support for logs is designed to be fully compatible with what you already have, providing capabilities to wrap those logs with additional context and a common toolkit to parse and manipulate logs into a common format across many different sources. -### OpenTelemetry Logs in the OpenTelemetry Collector +### OpenTelemetry logs in the OpenTelemetry Collector The [OpenTelemetry Collector](/docs/collector) provides several tools to work with logs: @@ -37,7 +37,7 @@ with logs: The first step in adopting OpenTelemetry frequently involves deploying a Collector as a general-purposes logging agent. -### OpenTelemetry Logs for Applications +### OpenTelemetry logs for applications In applications, OpenTelemetry logs are created with any logging library or built-in logging capabilities. When you add autoinstrumentation or activate an @@ -45,7 +45,7 @@ SDK, OpenTelemetry will automatically correlate your existing logs with any active trace and span, wrapping the log body with their IDs. In other words, OpenTelemetry automatically correlates your logs and traces. -### Language Support +### Language support Logs are a [stable](/docs/specs/otel/versioning-and-stability/#stable) signal in the OpenTelemetry specification. For the individual language specific @@ -53,16 +53,20 @@ implementations of the Logs API & SDK, the status is as follows: {{% signal-support-table "logs" %}} -## Structured, Unstructured, and Semistructured Logs +## Structured, unstructured, and semistructured logs -OpenTelemetry does not distinguish between structured and unstructured logs. -However, it is vitally important to understand the different kinds of formats -that log data can manifest in. +OpenTelemetry does not technically distinguish between structured and +unstructured logs. You can use any log you have with OpenTelemetry. However, not +all log formats are equally useful! Structured logs, in particular, are +recommended for production observability because they are easy to parse and +analyze at scale. The following section explains the differences between +structured, unstructured, and semistructured logs. ### Structured logs A structured log is a log whose textual format follows a consistent, -machine-readable format. Two of are JSON: +machine-readable format. For applications, one of the most common formats is +JSON: ```json { @@ -101,14 +105,14 @@ machine-readable format. Two of are JSON: } ``` -and for infrastructure components, Common Log Format (CLF): +and for infrastructure components, Common Log Format (CLF) is commonly used: ```text 127.0.0.1 - johndoe [04/Aug/2024:12:34:56 -0400] "POST /api/v1/login HTTP/1.1" 200 1234 ``` It is also common to have different structured log formats mixed together. For -example, and Extended Log Format (ELF) log mixes JSON with the +example, an Extended Log Format (ELF) log can mix JSON with the whitespace-separated data in a CLF log. ```text @@ -117,17 +121,21 @@ whitespace-separated data in a CLF log. To make the most use of this log, parse both the JSON and the ELF-related pieces into a shared format to make analysis on an observability -backend easier. +backend easier. The `filelogreceiver` in the +[OpenTelemetry Collector](/docs/collector) contains standardized ways to parse +logs like this. -Structured logs are the preferred way to use logs. Because they are emitted in a -consistent format, they are easier to parse, which makes them easier to -preprocess in an OpenTelemetry Collector, correlate with other data, and -ultimate analyze in an Observability backend. Use structured logs if you can. +Structured logs are the preferred way to use logs. Because structured logs are +emitted in a consistent format, they are straightforward to parse, which makes +them easier to preprocess in an OpenTelemetry Collector, correlate with other +data, and ultimate analyze in an Observability backend. -### Unstructured Logs +### Unstructured logs Unstructured logs are logs that don't follow a consistent structure. They may be -more human-readable, and are often used in development. +more human-readable, and are often used in development. However, it is not +preferred to use unstructured logs for production observability purposes, since +they are much more difficult to parse and analyze at scale. Examples of unstructured logs: @@ -139,9 +147,15 @@ System reboot initiated at 2024-08-04 03:00:00 by user: admin. Reason: Scheduled DEBUG - 2024-08-04 09:30:15 - User johndoe performed action: file_upload. Filename: report_Q3_2024.pdf, Size: 2.3 MB, Duration: 5.2 seconds. Result: Success ``` -Structured logs are fine for debugging applications locally, but -are very difficult to use at scale when observing large systems. -Use structured logging if possible. +It is possible to store and analyze Unstructured logs in production, although you may need to do +substantial work to parse or otherwise pre-process them to be machine-readable. +For example, the above three logs will require a regular expression to parse +their timestamps and custom parsers to consitently extract the bodies of the log +message. This will typically be necessary for a logging backend to know how to +sort and organize the logs by timestamp. Although it's possible to parse +unstructured logs for analysis purposes, doing this may be more work than +switching to structured logging, such as via a standard logging framework in +your applications. ### Semistructured logs @@ -158,7 +172,7 @@ Example of a semistructured log: Although machine-readable, semistructured logs may require several different parsers to allow for analysis at scale. -## OpenTelemetry Logging Componentry +## OpenTelemetry logging components The following lists of concepts and components power OpenTelemetry's logging support. From a80a888f2e7cead4e4934fadd31690bab4954ccc Mon Sep 17 00:00:00 2001 From: Phillip Carter Date: Mon, 5 Aug 2024 14:33:43 +0000 Subject: [PATCH 6/6] fuck these robots man --- content/en/docs/concepts/signals/logs.md | 35 ++++++++++++------------ 1 file changed, 17 insertions(+), 18 deletions(-) diff --git a/content/en/docs/concepts/signals/logs.md b/content/en/docs/concepts/signals/logs.md index 0e8165cd6dfc..26c5dace5fce 100644 --- a/content/en/docs/concepts/signals/logs.md +++ b/content/en/docs/concepts/signals/logs.md @@ -17,8 +17,8 @@ framework or infrastructure component. OpenTelemetry SDKs and autoinstrumentation utilize several components to automatically correlate logs with [traces](/docs/concepts/signals/traces). -OpenTelemetry's support for logs is designed to be fully compatible with -what you already have, providing capabilities to wrap those logs with additional +OpenTelemetry's support for logs is designed to be fully compatible with what +you already have, providing capabilities to wrap those logs with additional context and a common toolkit to parse and manipulate logs into a common format across many different sources. @@ -119,11 +119,10 @@ whitespace-separated data in a CLF log. 192.168.1.1 - johndoe [04/Aug/2024:12:34:56 -0400] "POST /api/v1/login HTTP/1.1" 200 1234 "http://example.com" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36" {"transactionId": "abcd-efgh-ijkl-mnop", "responseTime": 150, "requestBody": {"username": "johndoe"}, "responseHeaders": {"Content-Type": "application/json"}} ``` -To make the most use of this log, parse both the JSON and -the ELF-related pieces into a shared format to make analysis on an observability -backend easier. The `filelogreceiver` in the -[OpenTelemetry Collector](/docs/collector) contains standardized ways to parse -logs like this. +To make the most use of this log, parse both the JSON and the ELF-related pieces +into a shared format to make analysis on an observability backend easier. The +`filelogreceiver` in the [OpenTelemetry Collector](/docs/collector) contains +standardized ways to parse logs like this. Structured logs are the preferred way to use logs. Because structured logs are emitted in a consistent format, they are straightforward to parse, which makes @@ -147,15 +146,15 @@ System reboot initiated at 2024-08-04 03:00:00 by user: admin. Reason: Scheduled DEBUG - 2024-08-04 09:30:15 - User johndoe performed action: file_upload. Filename: report_Q3_2024.pdf, Size: 2.3 MB, Duration: 5.2 seconds. Result: Success ``` -It is possible to store and analyze Unstructured logs in production, although you may need to do -substantial work to parse or otherwise pre-process them to be machine-readable. -For example, the above three logs will require a regular expression to parse -their timestamps and custom parsers to consitently extract the bodies of the log -message. This will typically be necessary for a logging backend to know how to -sort and organize the logs by timestamp. Although it's possible to parse -unstructured logs for analysis purposes, doing this may be more work than -switching to structured logging, such as via a standard logging framework in -your applications. +It is possible to store and analyze Unstructured logs in production, although +you may need to do substantial work to parse or otherwise pre-process them to be +machine-readable. For example, the above three logs will require a regular +expression to parse their timestamps and custom parsers to consitently extract +the bodies of the log message. This will typically be necessary for a logging +backend to know how to sort and organize the logs by timestamp. Although it's +possible to parse unstructured logs for analysis purposes, doing this may be +more work than switching to structured logging, such as via a standard logging +framework in your applications. ### Semistructured logs @@ -174,8 +173,8 @@ parsers to allow for analysis at scale. ## OpenTelemetry logging components -The following lists of concepts and components power OpenTelemetry's -logging support. +The following lists of concepts and components power OpenTelemetry's logging +support. ### Log Appender / Bridge