Fluent bit Cloudwatch plugin use instance role

[glnds]

Hi, I try to run Fluent bit on an Amazon Linux 2, everything works fine except that the Cloudwatch plugins seems unable to authenticate using the Instance Profile.

[2020/11/13 15:22:40] [ info] [engine] started (pid=7706)
[2020/11/13 15:22:40] [ info] [storage] version=1.0.6, initializing...
[2020/11/13 15:22:40] [ info] [storage] in-memory
[2020/11/13 15:22:40] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2020/11/13 15:22:40] [ warn] [aws_credentials] Failed to initialized profile provider: $HOME not set and AWS_SHARED_CREDENTIALS_FILE not set.
[2020/11/13 15:22:40] [ info] [sp] stream processor started

Any idea how I can tell the Cloudwatch plugin to use the IAM instance Profile to authenticate towards AWS?

[PettitWesley]

Hey @glnds sorry for the delayed response.

Fluent Bit can use instance profile. Please enable debug logging, the AWS credential library for the core plugins will print detailed information on where it tried to look for credentials.

Instance profile requires EC2 IMDS. It does not currently support IMDSv2... that's one thing I need to work on...

Make sure IMDS is enabled and reachable by Fluent Bit

[glnds]

The instance was still provisioned with IMDSv1. Currently, the issue seems to be gone, no clue why. It's a pity but I had to fall back to the fluentd client because I need a recursive wildcard for the input path and that's currently unsupported by fluentbit.

[smithdebug]

Hi, I try to run Fluent bit on a Windows server 2016, the Cloudwatch plugins seem unable to authenticate using the Instance Profile.

[PettitWesley]

@smithdebug Please share debug logs if you want help with the issue

[Dkairu]

@PettitWesley Currently having issues with authentication too. I am running a VM hosted in Azure not sure it makes a difference. Below are the debug logs. I have created a credentials file and placed it in .aws/credentials with a config as below

[default]
aws_access_key_id=AKIAMyAccessKey
aws_secret_access_key=0NlMySecretAccessKeyGoesHere

I have also set up env variables as below

AWS_SECRET_ACCESS_KEY=0NlMySecretAccessKeyGoesHere
AWS_DEFAULT_REGION=us-east-1
AWS_SHARED_CREDENTIALS_FILE=/root/.aws/credentials
AWS_ACCESS_KEY_ID=AKIAMyAccessKey
 
time="2021-05-19T00:57:35Z" level=warning msg="[aws_credentials] Failed to initialized profile provider: $HOME not set and AWS_SHARED_CREDENTIALS_FILE not set."
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [aws_credentials] Not initializing EKS provider because AWS_ROLE_ARN was not set"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [aws_credentials] Initialized EC2 Provider in standard chain"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [aws_credentials] Not initializing ECS Provider because AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is not set"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [fstore] [cio scan] opening path /tmp/fluent-bit/s3/MYBUCKET"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [fstore] [cio stream] new stream registered: 2021-05-19T00:05:08"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [fstore] [cio scan] opening stream 2021-05-19T00:05:08"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [fstore] 2021-05-19T00:05:08:6954047773678-8049021342258469240 mapped OK"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [fstore] [cio stream] new stream registered: 2021-05-19T00:44:44"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [fstore] [cio scan] opening stream 2021-05-19T00:44:44"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [fstore] 2021-05-19T00:44:44:6954047773678-10528202603601663098 mapped OK"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [fstore] [cio stream] new stream registered: 2021-05-19T00:56:10"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [fstore] [cio scan] opening stream 2021-05-19T00:56:10"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [fstore] 2021-05-19T00:56:10:6954047773678-10633147869138171180 mapped OK"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [fstore] [cio stream] new stream registered: multipart_upload_metadata"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [fstore] [cio scan] opening stream multipart_upload_metadata"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [fstore] [cio stream] new stream registered: 2021-05-18T23:44:15"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [fstore] [cio scan] opening stream 2021-05-18T23:44:15"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [fstore] 2021-05-18T23:44:15:14137865059283200653-3446987199572924012 mapped OK"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [fstore] 2021-05-18T23:44:15:6954047773678-7837171435068195840 mapped OK"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [fstore] 2021-05-18T23:44:15:6954047773678-10707450372515562884 mapped OK"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [fstore] 2021-05-18T23:44:15:6954047773678-3773947905818295394 mapped OK"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [fstore] [cio stream] new stream registered: 2021-05-19T00:33:22"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [fstore] [cio scan] opening stream 2021-05-19T00:33:22"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [fstore] 2021-05-19T00:33:22:6954047773678-7757044812502076368 mapped OK"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [fstore] [cio stream] new stream registered: 2021-05-19T00:21:56"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [fstore] [cio scan] opening stream 2021-05-19T00:21:56"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [fstore] 2021-05-19T00:21:56:6954047773678-6817026440691995176 mapped OK"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [fstore] [cio stream] new stream registered: 2021-05-19T00:12:23"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [fstore] [cio scan] opening stream 2021-05-19T00:12:23"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [fstore] 2021-05-19T00:12:23:6954047773678-6168765526643237532 mapped OK"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [fstore] [cio stream] new stream registered: 2021-05-19T00:17:32"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [fstore] [cio scan] opening stream 2021-05-19T00:17:32"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [fstore] 2021-05-19T00:17:32:6954047773678-1389950883665654562 mapped OK"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [fstore] created stream path /tmp/fluent-bit/s3/MYBUCKET/2021-05-19T00:57:35"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [fstore] [cio stream] new stream registered: 2021-05-19T00:57:35"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [aws_credentials] Sync called on the EC2 provider"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [aws_credentials] Init called on the env provider"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [aws_credentials] Init called on the EC2 IMDS provider"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [aws_credentials] requesting credentials from EC2 IMDS"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [imds] Using instance metadata V1"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [aws_client] (null): http_do=0, HTTP Status: 404"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [upstream] KA connection #35 to 169.254.169.254:80 is now available"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [ecs_imds] IMDS metadata response"
time="2021-05-19T00:57:35Z" level=debug msg="<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Strict//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd\">"
time="2021-05-19T00:57:35Z" level=debug msg="<html xmlns=\"http://www.w3.org/1999/xhtml\">"
time="2021-05-19T00:57:35Z" level=debug msg="<meta http-equiv=\"Content-Type\" content=\"text/html; charset=iso-8859-1\"/>"
time="2021-05-19T00:57:35Z" level=debug msg="<title>404 - File or directory not found.</title>"
time="2021-05-19T00:57:35Z" level=debug msg="<!--"
time="2021-05-19T00:57:35Z" level=debug msg="body{margin:0;font-size:.7em;font-family:Verdana, Arial, Helvetica, sans-serif;background:#EEEEEE;}"
time="2021-05-19T00:57:35Z" level=debug msg="fieldset{padding:0 15px 10px 15px;} "
time="2021-05-19T00:57:35Z" level=debug msg="h1{font-size:2.4em;margin:0;color:#FFF;}"
time="2021-05-19T00:57:35Z" level=debug msg="h2{font-size:1.7em;margin:0;color:#CC0000;} "
time="2021-05-19T00:57:35Z" level=debug msg="h3{font-size:1.2em;margin:10px 0 0 0;color:#000000;} "
time="2021-05-19T00:57:35Z" level=debug msg="#header{width:96%;margin:0 0 0 0;padding:6px 2% 6px 2%;font-family:\"trebuchet MS\", Verdana, sans-serif;color:#FFF;"
time="2021-05-19T00:57:35Z" level=debug msg="#content{margin:0 0 0 2%;position:relative;}"
time="2021-05-19T00:57:35Z" level=debug msg=".content-container{background:#FFF;width:96%;margin-top:8px;padding:10px;position:relative;}"
time="2021-05-19T00:57:35Z" level=debug msg="-->"
time="2021-05-19T00:57:35Z" level=debug msg="<div id=\"header\"><h1>Server Error</h1></div>"
time="2021-05-19T00:57:35Z" level=debug msg=" <div class=\"content-container\"><fieldset>"
time="2021-05-19T00:57:35Z" level=debug msg="  <h2>404 - File or directory not found.</h2>"
time="2021-05-19T00:57:35Z" level=debug msg="  <h3>The resource you are looking for might have been removed, had its name changed, or is temporarily unavailable.</h3>"
time="2021-05-19T00:57:35Z" level=debug msg="[output:s3:s3.1] Sending locally buffered data from previous executions to S3; buffer=/tmp/fluent-bit/s3/MYBUCKET"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [aws_credentials] Requesting credentials from the env provider.."
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [aws_credentials] Requesting credentials from the EC2 provider.."
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [aws_credentials] requesting credentials from EC2 IMDS"
time="2021-05-19T00:57:35Z" level=debug msg="[debug] [imds] Using instance metadata V1"

[kingsleykumar]

Hi, I try to run Fluent bit on a Windows server 2016, the Cloudwatch plugins seem unable to authenticate using the Instance Profile.

Have you managed to get it working on windows server?

[lado936]

@PettitWesley i have same issue, here is the debug logs, for me IMDS returns 401

2021-10-23 10:01:55
[2021/10/23 06:01:55] [debug] [aws_credentials] Reading shared credentials file.
2021-10-23 10:01:55
[2021/10/23 06:01:55] [debug] [aws_credentials] Shared credentials file /root/.aws/credentials does not exist
2021-10-23 10:01:55
[2021/10/23 06:01:55] [debug] [aws_credentials] Init called on the EC2 IMDS provider
2021-10-23 10:01:55
[2021/10/23 06:01:55] [debug] [aws_credentials] requesting credentials from EC2 IMDS
2021-10-23 10:01:55
[2021/10/23 06:01:55] [debug] [http_client] not using http_proxy for header
2021-10-23 10:01:55
[2021/10/23 06:01:55] [debug] [http_client] server 169.254.169.254:80 will close connection #32
2021-10-23 10:01:55
[2021/10/23 06:01:55] [debug] [aws_client] (null): http_do=0, HTTP Status: 401
2021-10-23 10:01:55
[2021/10/23 06:01:55] [debug] [http_client] not using http_proxy for header
2021-10-23 10:01:55
[2021/10/23 06:01:55] [debug] [aws_credentials] Initialized Env Provider in standard chain
2021-10-23 10:01:55
[2021/10/23 06:01:55] [debug] [aws_credentials] Initialized AWS Profile Provider in standard chain
2021-10-23 10:01:55
[2021/10/23 06:01:55] [debug] [aws_credentials] Not initializing EKS provider because AWS_ROLE_ARN was not set
2021-10-23 10:01:55
[2021/10/23 06:01:55] [debug] [aws_credentials] Not initializing ECS Provider because AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is not set
2021-10-23 10:01:55
[2021/10/23 06:01:55] [debug] [aws_credentials] Initialized EC2 Provider in standard chain
2021-10-23 10:01:55
[2021/10/23 06:01:55] [debug] [aws_credentials] Sync called on the EC2 provider
2021-10-23 10:01:55
[2021/10/23 06:01:55] [debug] [aws_credentials] Init called on the env provider
2021-10-23 10:01:55
[2021/10/23 06:01:55] [debug] [aws_credentials] Init called on the profile provider
2021-10-23 10:01:55
[2021/10/23 06:01:55] [debug] [aws_credentials] Reading shared config file.
2021-10-23 10:01:55
[2021/10/23 06:01:55] [debug] [out_es] Enabled AWS Auth```

[PettitWesley]

@lado936 What version of FB are you using? I think this might be an IMDSv2 issue which is fixed by the latest version- do you know if you are using IMDSv2?

@matthewfala Is this an IMDSv2 related thing?

[lado936]

@PettitWesley Yep, after one day of debugging i found out that you pushed breaking changes and i was using latest tag for my fluentbit, set it to correct version and fixed it.
P.S i left message on that changes too, but will leave here too, that kind of changes must be at least notified in AWS console or somewhere, as a lot of people is using it

[matthewfala]

Thank you @lado936.

This happens when hop limit is set to 1 since the new image uses IMDSv2 by default and the fallback to the old system on hop count 1 has a bug. Working on a fix to allow for a fallback to IMDSv1 if IMDSv2 is not available due to the hop count.

Here's some information for others to help:
#259

[lado936]

Yes i figured it out, just you have to notify people when releasing changes like that, but yeah, using latest tag is quite an unproffesional from me 😁