Skip to content

Latest commit

 

History

History
105 lines (72 loc) · 6.02 KB

stackdriver.md

File metadata and controls

105 lines (72 loc) · 6.02 KB
Raw

Stackdriver

Stackdriver output plugin allows to ingest your records into Google Cloud Stackdriver Logging service.

Before to get started with the plugin configuration, make sure to obtain the proper credentials to get access to the service. We strongly recommend to use a common JSON credentials file, reference link:

Your goal is to obtain a credentials JSON file that will be used later by Fluent Bit Stackdriver output plugin.

Configuration Parameters

Key Description default
google_service_credentials Absolute path to a Google Cloud credentials JSON file Value of environment variable $GOOGLE_SERVICE_CREDENTIALS
service_account_email Account email associated to the service. Only available if no credentials file has been provided. Value of environment variable $SERVICE_ACCOUNT_EMAIL
service_account_secret Private key content associated with the service account. Only available if no credentials file has been provided. Value of environment variable $SERVICE_ACCOUNT_SECRET
resource Set resource type of data. Supported resource types: k8s_container, k8s_node, k8s_pod, global and gce_instance. global, gce_instance
k8s_cluster_name The name of the cluster that the container (node or pod based on the resource type) is running in. If the resource type is one of the k8s_container, k8s_node or k8s_pod, then this field is required.
k8s_cluster_location The physical location of the cluster that contains (node or pod based on the resource type) the container. If the resource type is one of the k8s_container, k8s_node or k8s_pod, then this field is required.
labels_key The value of this field is used by the Stackdriver output plugin to find the related labels from jsonPayload and then extract the value of it to set the LogEntry Labels. logging.googleapis.com/labels
tag_prefix Set the tag_prefix used to validate the tag of logs with k8s resource type. Without this option, the tag of the log must be in format of k8s_container(pod/node).* in order to use the k8s_container resource type. Now the tag prefix is configurable by this option (note the ending dot). k8s_container., k8s_pod., k8s_node.
severity_key Specify the name of the key from the original record that contains the severity information.
tag_prefix Set the tag_prefix used to validate the tag of logs with k8s resource type. Without this option, the tag of the log must be in format of k8s_container(pod/node).* in order to use the k8s_container resource type. Now the tag prefix is configurable by this option. k8s_container., k8s_pod., k8s_node.

Configuration File

If you are using a Google Cloud Credentials File, the following configuration is enough to get started:

[INPUT]
    Name  cpu
    Tag   cpu

[OUTPUT]
    Name        stackdriver
    Match       *

Example configuration file for k8s resource type:

local_resource_id is used by stackdriver output plugin to set the labels field for different k8s resource types. Stackdriver plugin will try to find the local_resource_id field in the log entry. If there is no field logging.googleapis.com/local_resource_id in the log, the plugin will then construct it by using the tag value of the log.

The local_resource_id should be in format:

  • k8s_container.<namespace_name>.<pod_name>.<container_name>
  • k8s_node.<node_name>
  • k8s_pod.<namespace_name>.<pod_name>

This implies that if there is no local_resource_id in the log entry then the tag of logs should match this format. Note that we have an option tag_prefix so it is not mandatory to use k8s_container(node/pod) as the prefix for tag.

[INPUT]
    Name               tail
    Tag_Regex          var.log.containers.(?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-(?<docker_id>[a-z0-9]{64})\.log$
    Tag                custom_tag.<namespace_name>.<pod_name>.<container_name>
    Path               /var/log/containers/*.log
    Parser             docker
    DB                 /var/log/fluent-bit-k8s-container.db

[OUTPUT]
    Name        stackdriver
    Match       custom_tag.*
    Resource    k8s_container
    k8s_cluster_name test_cluster_name
    k8s_cluster_location  test_cluster_location
    tag_prefix  custom_tag.

Troubleshooting Notes

Upstream connection error

Github reference: #761

An upstream connection error means Fluent Bit was not able to reach Google services, the error looks like this:

[2019/01/07 23:24:09] [error] [oauth2] could not get an upstream connection

This belongs to a network issue by the environment where Fluent Bit is running, make sure that from the Host, Container or Pod you can reach the following Google end-points:

Fail to process local_resource_id

The error looks like this:

[2020/08/04 14:43:03] [error] [output:stackdriver:stackdriver.0] fail to process local_resource_id from log entry for k8s_container

Do following check:

  • If the log entry does not contain the local_resource_id field, does the tag of the log match for format?

  • If tag_prefix is configured, does the prefix of tag specified in the input plugin match the tag_prefix?

    Other implementations

Stackdriver officially supports a logging agent based on Fluentd.

We plan to support some special fields in structured payloads. Use cases of special fields is here.