stackdriver: amend special fields docs #1267
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -25,11 +25,12 @@ Before to get started with the plugin configuration, make sure to obtain the pro | |||||
| | resource | Set resource type of data. Supported resource types: _k8s\_container_, _k8s\_node_, _k8s\_pod_, _global_, _generic\_node_, _generic\_task_, and _gce\_instance_. | global, gce\_instance | | ||||||
| | k8s\_cluster\_name | The name of the cluster that the container \(node or pod based on the resource type\) is running in. If the resource type is one of the _k8s\_container_, _k8s\_node_ or _k8s\_pod_, then this field is required. | | | ||||||
| | k8s\_cluster\_location | The physical location of the cluster that contains \(node or pod based on the resource type\) the container. If the resource type is one of the _k8s\_container_, _k8s\_node_ or _k8s\_pod_, then this field is required. | | | ||||||
| | labels\_key | The value of this field is used by the Stackdriver output plugin to find the related labels from jsonPayload and then extract the value of it to set the LogEntry Labels. | `logging.googleapis.com/labels`. See [Stackdriver Special Fields](StackdriverSpecialFields) for more info. | | ||||||
| | labels | Optional list of comma separated of strings specifying `key=value` pairs. The resulting `labels` will be combined with the elements in obtained from `labels_key` to set the LogEntry Labels. Elements from `labels` will override duplicate values from `labels_key`.| | | ||||||
| | log\_name\_key | The value of this field is used by the Stackdriver output plugin to extract logName from jsonPayload and set the logName field. | `logging.googleapis.com/logName`. See [Stackdriver Special Fields](StackdriverSpecialFields) for more info. | | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
| | tag\_prefix | Set the tag\_prefix used to validate the tag of logs with k8s resource type. Without this option, the tag of the log must be in format of k8s\_container\(pod/node\).\* in order to use the k8s\_container resource type. Now the tag prefix is configurable by this option \(note the ending dot\). | k8s\_container., k8s\_pod., k8s\_node. | | ||||||
| | severity\_key | Specify the name of the key from the original record that contains the severity information. | `logging.googleapis.com/severity`. See [Stackdriver Special Fields](StackdriverSpecialFields) for more info. | | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
| | project_id_key | The value of this field is used by the Stackdriver output plugin to find the gcp project id from jsonPayload and then extract the value of it to set the PROJECT_ID within LogEntry logName, which controls the gcp project that should receive these logs. | `logging.googleapis.com/projectId`. See [Stackdriver Special Fields](StackdriverSpecialFields) for more info. | | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
| | autoformat\_stackdriver\_trace | Rewrite the _trace_ field to include the projectID and format it for use with Cloud Trace. When this flag is enabled, the user can get the correct result by printing only the traceID (usually 32 characters). | false | | ||||||
| | Workers | Enables dedicated thread(s) for this output. | 1 | | ||||||
| | custom\_k8s\_regex | Set a custom regex to extract field like pod\_name, namespace\_name, container\_name and docker\_id from the local\_resource\_id in logs. This is helpful if the value of pod or node name contains dots. | `(?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-(?<docker_id>[a-z0-9]{64})\.log$` | | ||||||
|
|
@@ -140,7 +141,7 @@ This will produce the following log: | |||||
| } | ||||||
| ``` | ||||||
|
|
||||||
| This makes the `resource_labels` API the recommended choice for supporting new or existing resource types that have all resource labels known before runtime or available on the payload during runtime. | ||||||
|
|
||||||
| For instance, for a K8s resource type, `resource_labels` can be used in tandem with the [Kubernetes filter](https://docs.fluentbit.io/manual/pipeline/filters/kubernetes) to pack all six resource labels. Below is an example of what this could look like for a `k8s_container` resource: | ||||||
|
|
||||||
|
|
@@ -189,3 +190,5 @@ Do following check: | |||||
| Stackdriver officially supports a [logging agent based on Fluentd](https://cloud.google.com/logging/docs/agent). | ||||||
|
|
||||||
| We plan to support some [special fields in structured payloads](https://cloud.google.com/logging/docs/agent/configuration#special-fields). Use cases of special fields is [here](./stackdriver_special_fields.md). | ||||||
|
|
||||||
| [StackdriverSpecialFields]: ./stackdriver_special_fields.md#log-entry-fields | ||||||
|
There was a problem hiding this comment. Hi @lecaros, this should make it so the links aren't broken. How did you test it? There was a problem hiding this comment. hi @braydonk. There was a problem hiding this comment. Should be fixed now. I had the syntax wrong for referring to the named link. |
||||||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,19 +1,29 @@ | ||
| # Stackdriver Special fields | ||
|
There was a problem hiding this comment. Why do we need to split this out into a separate doc @braydonk ? There was a problem hiding this comment. I don't know, this was done well before my time by another Google employee who doesn't work in this area anymore. I'll check with my team and see if anyone would mind merging it into the main doc There was a problem hiding this comment. Appreciate it mate, I was just thinking it might be easier to merge into one but if not I can merge |
||
|
|
||
| When the [google-logging-agent](https://cloud.google.com/logging/docs/agent) receives a structured log record, it treats [some fields](https://cloud.google.com/logging/docs/agent/configuration#special-fields) specially, allowing users to set specific fields in the LogEntry object that get written to the Logging API. | ||
|
|
||
| ## Log Entry Fields | ||
|
|
||
| Currently, we support some special fields in fluent-bit for setting fields on the LogEntry object: | ||
| | JSON log field | [LogEntry](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry) field | Description | | ||
| | :--- | :--- | :--- | | ||
| | logging.googleapis.com/operation | operation | Additional information about a potentially long-running operation | | ||
| | logging.googleapis.com/labels | labels | The value of this field should be a structured record | | ||
| | logging.googleapis.com/insertId | insertId | A unique identifier for the log entry. It is used to order logEntries | | ||
| | logging.googleapis.com/sourceLocation | sourceLocation | Additional information about the source code location that produced the log entry. | | ||
| | logging.googleapis.com/http_request | httpRequest | A common proto for logging HTTP requests. | | ||
| | logging.googleapis.com/trace | trace | Resource name of the trace associated with the log entry | | ||
| | logging.googleapis.com/traceSampled | traceSampled | The sampling decision associated with this log entry. | | ||
| | logging.googleapis.com/spanId | spanId | The ID of the trace span associated with this log entry. | | ||
| | timestamp | timestamp | An object including the seconds and nanos fields that represents the time | | ||
| | timestampSecond & timestampNanos | timestamp | The seconds and nanos that represents the time | | ||
|
|
||
| ## Other Special Fields | ||
|
|
||
| | JSON log field | Description | | ||
| | :--- | :--- | | ||
| | logging.googleapis.com/projectId | Changes the project ID that this log will be written to. Ensure that you are authenticated to write logs to this project. | | ||
|
|
||
| ## Operation | ||
| Operation field contains additional information about a potentially long-running operation with which a log entry is associated. | ||
|
|
||
|
|
@@ -239,7 +249,7 @@ the logEntry will be: | |
| ``` | ||
|
|
||
| ## httpRequest | ||
| HttpRequest field is a common proto for logging HTTP requests. | ||
|
|
||
| The JSON representation is as followed: | ||
| ```text | ||
|
|
@@ -268,20 +278,20 @@ Set the input log as followed: | |
| ```text | ||
| jsonPayload { | ||
| "logging.googleapis.com/http_request": { | ||
| "requestMethod":"GET", | ||
| "requestUrl":"logging.googleapis.com", | ||
| "requestSize":"12", | ||
| "status":200, | ||
| "responseSize":"12", | ||
| "userAgent":"Mozilla", | ||
| "remoteIp":"255.0.0.1", | ||
| "serverIp":"255.0.0.1", | ||
| "referer":"referer", | ||
| "latency":"1s", | ||
| "cacheLookup":true, | ||
| "cacheHit":true, | ||
| "cacheValidatedWithOriginServer":true, | ||
| "cacheFillBytes":"12", | ||
| "protocol":"HTTP/1.2" | ||
| } | ||
| ... | ||
|
|
@@ -294,20 +304,20 @@ the logEntry will be: | |
| ... | ||
| } | ||
| "httpRequest": { | ||
| "requestMethod":"GET", | ||
| "requestUrl":"logging.googleapis.com", | ||
| "requestSize":"12", | ||
| "status":200, | ||
| "responseSize":"12", | ||
| "userAgent":"Mozilla", | ||
| "remoteIp":"255.0.0.1", | ||
| "serverIp":"255.0.0.1", | ||
| "referer":"referer", | ||
| "latency":"1s", | ||
| "cacheLookup":true, | ||
| "cacheHit":true, | ||
| "cacheValidatedWithOriginServer":true, | ||
| "cacheFillBytes":"12", | ||
| "protocol":"HTTP/1.2" | ||
| } | ||
| ... | ||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.