Restructure the logs page

content/en/docs/concepts/signals/logs.md

@@ -5,31 +5,167 @@ description: A recording of an event.
 ---
 
 A **log** is a timestamped text record, either structured (recommended) or
-unstructured, with metadata. Of all telemetry signals, logs have the biggest
-legacy. Most programming languages have built-in logging capabilities or
-well-known, widely used logging libraries. Although logs are an independent data
-source, they may also be attached to spans. In OpenTelemetry, any data that is
-not part of a distributed trace or a metric is a log. For example, _events_ are
-a specific type of log. Logs often contain detailed debugging/diagnostic info,
-such as inputs to an operation, the result of the operation, and any supporting
-metadata for that operation.
+unstructured, with optional metadata. Of all telemetry signals, logs have the
+biggest legacy. Most programming languages have built-in logging capabilities or
+well-known, widely used logging libraries.
 
-For traces and metrics, OpenTelemetry takes the approach of a clean-sheet
-design, specifies a new API, and provides full implementations of this API in
-multiple language SDKs.
+## OpenTelemetry Logs
 
-OpenTelemetry's approach with logs is different. Because existing logging
-solutions are widespread in language and operational ecosystems, OpenTelemetry
-acts as a "bridge" between those logs, the tracing and metrics signals, and
-other OpenTelemetry components. In fact, the API for logs is called the "Logs
-Bridge API" for this reason.
+OpenTelemetry does not define a bespoke API or SDK to create logs. Instead,
+OpenTelemetry logs are simply the existing logs you already have from a logging
+framework or infrastructure component. Under the covers, OpenTelemetry SDKs and
+autoinstrumentation utilize several components to automatically correlate logs
+with [traces](/docs/signals/traces).
 
-The [logs specification][] contains more details on this philosophy.
+OpenTelemetry's support for logs is designed to be maximally compatible with
+what you already have, providing capabilities to wrap those logs with additional
+context and a common toolkit to parse and manipulate logs into a common format
+across many different sources.
 
-To understand how logging in OpenTelemetry works, let's look at a list of
-components that will play a part in instrumenting our code.
+### OpenTelemetry Logs in the OpenTelemetry Collector
 
-## Log Appender / Bridge
+The [OpenTelemetry Collector](/docs/collector) provides several tools to work
+with logs:
+
+- Several receivers which parse logs from specific, known sources of log data
+- The filelogreceiver, which reads logs from any file and provides features to
+  parse them from different formats or use a regular expression
+- Processors like the transformprocessor which lets you parse nested data,
+  flatten nested structures, add/remove/update values, and more
+- Exporters that let you emit log data in a non-OpenTelemetry format
+
+Often times, the first step in adopting OpenTelemetry involves deploying a
+Collector as a general-purposes logging agent.
+
+### OpenTelemetry Logs for Applications
+
+In applications, OpenTelemetry logs are created with any logging library or
+built-in logging capabilities. When you add autoinstrumentation or activate an
+SDK, OpenTelemetry will automatically correlate your existing logs with any
+active trace and span, wrapping the log body with their IDs. In other words,
+OpenTelemetry will automatically correlate your logs!
+
+### Language Support
+
+Logs are a [stable](/docs/specs/otel/versioning-and-stability/#stable) signal in
+the OpenTelemetry specification. For the individual language specific
+implementations of the Logs API & SDK, the status is as follows:
+
+{{% signal-support-table "logs" %}}
+
+## Structured, Unstructured, and Semistructured Logs
+
+OpenTelemetry does not distinguish between structured and unstructured logs.
+However, it is vitally important to understand the different kinds of formats
+that log data can manifest in.
+
+### Structured logs
+
+A structured log is a log whose textual format follows a consistent,
+machine-readable format. Two of are JSON:
+
+```json
+{
+  "timestamp": "2024-08-04T12:34:56.789Z",
+  "level": "INFO",
+  "service": "user-authentication",
+  "environment": "production",
+  "message": "User login successful",
+  "context": {
+    "userId": "12345",
+    "username": "johndoe",
+    "ipAddress": "192.168.1.1",
+    "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
+  },
+  "transactionId": "abcd-efgh-ijkl-mnop",
+  "duration": 200,
+  "request": {
+    "method": "POST",
+    "url": "/api/v1/login",
+    "headers": {
+      "Content-Type": "application/json",
+      "Accept": "application/json"
+    },
+    "body": {
+      "username": "johndoe",
+      "password": "******"
+    }
+  },
+  "response": {
+    "statusCode": 200,
+    "body": {
+      "success": true,
+      "token": "jwt-token-here"
+    }
+  }
+}
+```
+
+and for infrastructure components, Common Log Format (CLF):
+
+```text
+127.0.0.1 - johndoe [04/Aug/2024:12:34:56 -0400] "POST /api/v1/login HTTP/1.1" 200 1234
+```
+
+It is also common to have different structured log formats mixed together. For
+example, and Extended Log Format (ELF) log mixes JSON with the
+whitespace-separated data in a CLF log.
+
+```text
+192.168.1.1 - johndoe [04/Aug/2024:12:34:56 -0400] "POST /api/v1/login HTTP/1.1" 200 1234 "http://example.com" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36" {"transactionId": "abcd-efgh-ijkl-mnop", "responseTime": 150, "requestBody": {"username": "johndoe"}, "responseHeaders": {"Content-Type": "application/json"}}
+```
+
+To make the most use of this log, it is recommended to parse bothe the JSON and
+the ELF-related pieces into a shared format to make analysis on an Observability
+backend easier.
+
+Structured logs are the preferred way to use logs. Because they are emitted in a
+consistent format, they are easier to parse, which makes them easier to
+preprocess in an OpenTelemetry Collector, correlate with other data, and
+ultimate analyze in an Observability backend. Use structured logs if you can.
+
+### Unstructured Logs
+
+Unstructured logs are logs that don't follow a consistent structure. They may be
+more human-readable, and are often used in development. However, it is not
+preferred to use unstructured logs to contain useful debugging data if you can
+help it.
+
+Examples of unstructured logs:
+
+```text
+[ERROR] 2024-08-04 12:45:23 - Failed to connect to database. Exception: java.sql.SQLException: Timeout expired. Attempted reconnect 3 times. Server: db.example.com, Port: 5432
+
+System reboot initiated at 2024-08-04 03:00:00 by user: admin. Reason: Scheduled maintenance. Services stopped: web-server, database, cache. Estimated downtime: 15 minutes.
+
+DEBUG - 2024-08-04 09:30:15 - User johndoe performed action: file_upload. Filename: report_Q3_2024.pdf, Size: 2.3 MB, Duration: 5.2 seconds. Result: Success
+```
+
+To reiterate, structured logs are fine for debugging applications locally, but
+are very difficult to use at scale when observing large systems. It is
+recommended that you use structured logging if possible.
+
+### Semistructured logs
+
+A semistructured log is a log that does use some self-consistent patterns to
+distinguish data so that it's machine-readable, but may not use the same
+formatting and delimiters between data across different systems.
+
+Example of a semistructured log:
+
+```text
+2024-08-04T12:45:23Z level=ERROR service=user-authentication userId=12345 action=login message="Failed login attempt" error="Invalid password" ipAddress=192.168.1.1 userAgent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
+```
+
+Although machine-readable, semistructured logs may require several different
+parsers to allow for analysis at scale.
+
+## OpenTelemetry Logging Componentry
+
+The following lists of concepts/components are what powers OpenTelemetry's
+logging support under the hood.
+
+### Log Appender / Bridge
 
 As an application developer, the **Logs Bridge API** should not be called by you
 directly, as it is provided for logging library authors to build log appenders /
@@ -39,7 +175,7 @@ OpenTelemetry LogRecordExporter.
 
 OpenTelemetry language SDKs offer this functionality.
 
-## Logger Provider
+### Logger Provider
 
 > Part of the **Logs Bridge API** and should only be used if you are the author
 > of a logging library.
@@ -49,20 +185,20 @@ A Logger Provider (sometimes called `LoggerProvider`) is a factory for
 lifecycle matches the application's lifecycle. Logger Provider initialization
 also includes Resource and Exporter initialization.
 
-## Logger
+### Logger
 
 > Part of the **Logs Bridge API** and should only be used if you are the author
 > of a logging library.
 
 A Logger creates log records. Loggers are created from Log Providers.
 
-## Log Record Exporter
+### Log Record Exporter
 
 Log Record Exporters send log records to a consumer. This consumer can be
 standard output for debugging and development-time, the OpenTelemetry Collector,
 or any open source or vendor backend of your choice.
 
-## Log Record
+### Log Record
 
 A log record represents the recording of an event. In OpenTelemetry a log record
 contains two kinds of fields:
@@ -89,15 +225,7 @@ The top-level fields are:
 For more details on log records and log fields, see
 [Logs Data Model](/docs/specs/otel/logs/data-model/).
 
-## Language Support
-
-Logs are a [stable](/docs/specs/otel/versioning-and-stability/#stable) signal in
-the OpenTelemetry specification. For the individual language specific
-implementations of the Logs API & SDK, the status is as follows:
-
-{{% signal-support-table "logs" %}}
-
-## Specification
+### Specification
 
 To learn more about logs in OpenTelemetry, see the [logs specification][].