Recommended Cloudwatch_Logs Configuration

[matthewfala]

NOTICE: please see the main tracking ticket for multiple recently reported high impact issues in AWS for Fluent Bit: #542

Recommended Cloudwatch_Logs Configuration

Recently our team has received lots of inquires on tuning the cloudwatch_logs output plugin via it's configuration

Customers using out of tune cloudwatch configurations may experience log loss due to:

• Broken connection / network errors
• Lack of retries on batch failures
• Lack of immediate network retries on network failure

These issues can be resolved via appropriate configuration.

If you are configuring FireLens via a Fluent Bit config file, use the following cloudwatch_logs configuration:

[OUTPUT]
    # general cloudwatch_logs configuration (nothing special here, customize to fit your use case)
    Name                cloudwatch_logs
    Match               ApplicationLogs
    region              ${LOG_REGION}
    log_group_name      ${SERVICE_NAME}-ApplicationLogs
    log_stream_prefix   ApplicationLogs--${HOSTNAME}
    auto_create_group   On

    # if you want to only write the log string without container metadata fields
    log_key             log

    # from aws-for-fluent-bit v2.32.0 and on, to support higher throughput logging,
    # set workers to a high value such as 5 or the number of cores on your host
    workers             1

    # optimized cloudwatch_logs output configuration
    # delayed retries on error 
    retry_limit         5    
    # on is default
    net.keepalive On
    # CW uses 6s idle timeout, FLB has 1.5s timer to check conns.
    # 4s ensures FLB always closes the conn itself, which we found 
    # significantly reduces the rate of network error messages it outputs
    net.keepalive_idle_timeout 4s

If you are configuring FireLens via task definition logDriver configuration options:

"logConfiguration": {
	"logDriver":"awsfirelens",
	"options": {

// general cloudwatch_logs configuration (nothing special here, customize to fit your use case)
		"Name": "cloudwatch_logs",
		"region": "${LOG_REGION}",
		"log_group_name": "${SERVICE_NAME}-ApplicationLogs",
		"log_stream_prefix": "ApplicationLogs--${HOSTNAME}",
		"auto_create_group": "On",
		"log_key": "log",

// optimized cloudwatch_logs output configuration
		"workers": "1",
		"auto_retry_requests": "On",
		"retry_limit": "5"
	}
}

We may update the above configuration from time to time to reflect the cloudwatch_logs configuration that provides the best performance.

[PettitWesley]

These settings used to be in the example but are no longer since they are same as the defaults since 1.9.x Fluent Bit upstream version series:

    # create a separate thread for each cloudwatch_output (does not work with more than one worker per log stream due to cloudwatch_logs API concurrency limitations)
    # as of Fluent Bit 1.9, 1 worker is the default
    workers             1   
    # retry network requests immediately on failure
    # this setting also defaults to "On" in the 1.9 series. 
    auto_retry_requests On  

[Duplo-Yashwant]

Is there dual options is supported under logConfiguration? Like sending logs to cloudwatch as well as opensearch.