Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for EKS Pod Identity #784

Open
askulkarni2 opened this issue Jan 26, 2024 · 11 comments
Open

Support for EKS Pod Identity #784

askulkarni2 opened this issue Jan 26, 2024 · 11 comments
Assignees
Labels
enhancement Feature request or enhancement on existing features help wanted Extra attention is needed

Comments

@askulkarni2
Copy link

askulkarni2 commented Jan 26, 2024

Describe the question/issue

Amazon EKS launched Pod Identity to simplify IAM configuration for pods. In order for AWS for FluentBit to support this, it will need to use an AWS SDK version that provides the new credential provider logic available >= v1.47.11.

@arvind-v
Copy link

arvind-v commented Jan 31, 2024

To build on @askulkarni2's comment above: Both EKS pod identity and IAM role for Service Account (IRSA) are not currently working with AWS for FluentBit. I tested an OpenSearch Ingestion Pipeline with AWS for FluentBit following the instructions at https://docs.aws.amazon.com/opensearch-service/latest/developerguide/configure-client-fluentbit.html.

For http output, FluentBit seems to build the SIGv4 request from scratch. IRSA and Pod Identity are ignored and the code relies on aws_role_arn to build the request. Further, the node group instance role is used for signing the SIGv4 request instead of the IAM role associated with the EKS pod identity or IRSA. See debug output from fluenbit pod below:

[2024/01/30 23:40:33] [debug] [aws_client] Unable to parse API response- response is not valid JSON.
[2024/01/30 23:40:33] [debug] [aws_credentials] STS raw response: 
<ErrorResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
  <Error>
    <Type>Sender</Type>
    <Code>AccessDenied</Code>
    <Message>User: arn:aws:sts::nnnnnnnn:assumed-role/eksctl-eks-cluster-name-nodegroup-NodeInstanceRole-abcdef/i-nnnnn is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::nnnnnnnn:role/ingestion-role</Message>
  </Error>
  <RequestId>afd30be4-0342-47fc-ba2d-78144185b411</RequestId>
</ErrorResponse>

[2024/01/30 23:40:33] [error] [aws_credentials] STS assume role request failed

A temporary workaround was to setup the aws_role_arn, which is named ingestion-role in my example, with a trust policy that enables the EKS cluster's NodeInstanceRole to assume the role.

A change in upstream flb_aws_credentials_sts.c may be needed to build the SIGv4 request differently and support EKS pod identity and IRSA.

@PettitWesley PettitWesley self-assigned this Mar 12, 2024
@PettitWesley PettitWesley added enhancement Feature request or enhancement on existing features help wanted Extra attention is needed labels Mar 12, 2024
@PettitWesley
Copy link
Contributor

I created the AWS credential providers for the Fluent Bit C plugins and docs: https://github.com/aws/aws-for-fluent-bit/blob/mainline/troubleshooting/debugging.md#credential-chain-resolution-issues

These are our different plugins: https://github.com/aws/aws-for-fluent-bit/blob/mainline/troubleshooting/debugging.md#aws-go-plugins-vs-aws-core-c-plugins

I'm new to EKS Pod identities. It looks like the credentials are vended via a local agent that has an http cred endpoint on a link local IP: https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html

This is exacty the same as how ECS cred vending works AFAICT: https://aws.amazon.com/blogs/compute/under-the-hood-task-networking-for-amazon-ecs/

So it should be pretty simple to modify the http credentials provider to support it: https://github.com/fluent/fluent-bit/blob/master/src/aws/flb_aws_credentials_http.c

I can't promise I'll work on this right now but I agree that Fluent Bit should support all new EKS features.

@PettitWesley
Copy link
Contributor

Possibly related: aws/amazon-eks-pod-identity-webhook#216 (comment)

@jefchien
Copy link

jefchien commented Apr 4, 2024

Took a look at the AWS SDK Go implementation and it's similar to the ECS credential provider with a few differences. It is in front of the ECS credential provider in the chain and selected based on the presence of AWS_CONTAINER_CREDENTIALS_FULL_URI (which contains the URI to make the request to). EKS Pod Identities support requires authentication token providers, which are also configured via environment variable. The rest of the HTTP request and response parsing is the same.

@TreeKat71
Copy link

Any update? Seems like IRSAv2 is not working right now

@PettitWesley
Copy link
Contributor

I am working on implementing support for EKS Pod identity actively. I can not promise a release date for it.

@georgejohnis
Copy link

Thank you @PettitWesley for working on it.

@iamahgoub
Copy link

@PettitWesley -- any update on this work? Thanks!

@trmbdevops
Copy link

I am working on implementing support for EKS Pod identity actively. I can not promise a release date for it.

@PettitWesley Any update on this yet?

@voidlily
Copy link

I'm also interested in this in order to get the container insights addon off of needing cloudwatch permissions on the instance profile and being able to use pod identity instead.

This may require an update to fluent-bit upstream, see these linked issues

fluent/fluent-bit#8550
fluent/fluent-bit#9206

@AzySir
Copy link

AzySir commented Nov 26, 2024

I am working on implementing support for EKS Pod identity actively. I can not promise a release date for it.

Thanks for that! Do you think it's worth putting a giant label on the READ ME - so newbies like me don't spend hours on it :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Feature request or enhancement on existing features help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

10 participants