Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Please support AWS IAM Instance profiles! #12464

Closed
davidmco65 opened this issue Jun 6, 2019 · 17 comments · Fixed by #17658 or #23344
Closed

Please support AWS IAM Instance profiles! #12464

davidmco65 opened this issue Jun 6, 2019 · 17 comments · Fixed by #17658 or #23344
Assignees
Labels

Comments

@davidmco65
Copy link

I can't use the Functionbeat on my EC2 server because it doesn't appear to use the default credential provider chain. Meaning that I have a Role assigned to me EC2 instance that gives it permission to access the bucket specified in the configuration, but the deploy function doesn't see use it and therefore has no permissions to do anything with S3.

I'm trying to deploy the functionbeat from our EC2 server and it's not possible given this missing functionality. I've got my entire ELK stack deployment scripted except for this functionbeat.

@exekias
Copy link
Contributor

exekias commented Jul 16, 2019

I think #12727 would fix this if used in Functionbeat? cc @kvch

@kvch
Copy link
Contributor

kvch commented Jul 16, 2019

Yes, I think so. But still, it needs a bit of work in Functionbeat. :)

@kaiyan-sheng
Copy link
Contributor

@davidmco65 Thanks for creating this issue. When you mentioned default credential provider chain, do you mean the aws credential file under ~/.aws/credentials?

@davidmco65
Copy link
Author

@kaiyan-sheng No, the default credential provider chain runs through a set of potential credential providers. One of which is the ~/.aws/credentials file, but others include the EC2 Instance Profile - a role that's assigned to the EC2 instance running the beat and would give the beat the permissions it needs to access the resources it needs, i.e., S3 bucket. In our environment, we literally cannot use access key id/secret key, so there is no way to configure the function beat that will allow it to install properly.

Thanks!

@jderose9
Copy link

jderose9 commented Oct 31, 2019

I need to be able to specify a role to assume in a child account via a named profile present in the ~/.aws/config file. This is typically passed as --profile parameter on aws CLI commands and many other tools that interact with AWS.

https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html

@Thiago-Dantas
Copy link

I also need this but in Metricbeat, should I open a new issue?

@kaiyan-sheng
Copy link
Contributor

@Thiago-Dantas I'm planning to use this same issue to track for Metricbeat as well! Thanks!

@kaiyan-sheng kaiyan-sheng changed the title Please support AWS Instance profiles! Please support AWS IAM Instance profiles! Apr 9, 2020
@jeffvestal
Copy link

jeffvestal commented Apr 9, 2020

I have a user who is interested in this feature so they don't have to store creds locally on the ec2 instance.
@kaiyan-sheng ping me if you need more details.
Thanks!

@kvch
Copy link
Contributor

kvch commented Apr 15, 2020

@kaiyan-sheng Are you sure this issue can be closed? The original author requested the feature for Functionbeat. However, your PR only adds support in metrcibeat and filebeat.

@zube zube bot assigned kvch Apr 15, 2020
@zube zube bot closed this as completed Apr 15, 2020
@zube zube bot added [zube]: Done and removed [zube]: Inbox labels Apr 15, 2020
@zube zube bot unassigned kvch Apr 15, 2020
@zube zube bot reopened this Apr 15, 2020
@zube zube bot added [zube]: Inbox and removed [zube]: Done labels Apr 15, 2020
@kvch
Copy link
Contributor

kvch commented Apr 16, 2020

No, it does not. Functionbeat depends on the AWS SDK to handle credentials. Thus, it only supports passing those via environment variables. Thank you for reopening.

@andresrc andresrc added [zube]: Backlog and removed Team:Platforms Label for the Integrations - Platforms team [zube]: Inbox labels Apr 19, 2020
@prabaharang
Copy link

If Functionbeat uses AWS SDK, what is the credential provider chain is being used. Wouldn't AWS SDK allow to use Instance profile credentials by default (in case of Default Credential Provider Chain)?

@kvch
Copy link
Contributor

kvch commented Apr 27, 2020

I guess yes, but we haven't tested it yet. Also, all of the information I was able to found on the subject is for the v1 of aws-sdk-go. However, in Beats we use aws-sdk-go-v2, so there migth be differences between the docs of v1 and reality of v2.

@kaiyan-sheng
Copy link
Contributor

Hi @kvch, since this issue is related to Functionbeat, is it ok if I assign it to you instead?

@kvch kvch assigned kvch and unassigned kaiyan-sheng Jun 24, 2020
@kvch
Copy link
Contributor

kvch commented Jun 24, 2020

@kaiyan-sheng I've just assinged myself. Thanks for the ping.

@s-t-ev
Copy link

s-t-ev commented Aug 18, 2020

@kvch Any luck with implementing this for functionbeat? Thanks!

@BrunoMCBraga
Copy link

Fellas, i come from the future and this problem still persists. Functionbeat seems to still be using AWS env variable credentials. I am currently playing with Elastic stack using one single node on as private network and i cannot afford to put my credentials there, specially when the list of permissions necessary: https://www.elastic.co/guide/en/beats/functionbeat/current/iam-permissions.html#iam-permissions-cloudwatch

Is quite big. Looks like a disaster ready to happen.

kvch added a commit that referenced this issue Jan 7, 2021
## What does this PR do?

This PR makes credential settings when deploying Lambdas to AWS more flexible. New options are introduced:

1. `access_key_id`, `secret_access_key` and/or `session_token` for tokens

```yaml
functionbeat.provider.aws.access_key_id: '${AWS_ACCESS_KEY_ID:""}'
functionbeat.provider.aws.secret_access_key: '${AWS_SECRET_ACCESS_KEY:""}'
functionbeat.provider.aws.session_token: '${AWS_SESSION_TOKEN:""}'
``` 
2. `role_arn` for assuming IAM roles
```yaml
functionbeat.provider.aws.role_arn: arn:aws:iam::123456789012:role/test-fnb
```

3. `credential_profile_name` and/or `shared_credential_file` for credential files
```yaml
functionbeat.provider.aws.credential_profile_name: fnb-aws
functionbeat.provider.aws.shared_credential_file: /etc/functionbeat/aws_credentials
```

## Why is it important?

Credential configuration becomes more flexible and follows the same pattern as in Filebeat and Metricbeat.

## Related issues

Based on #17658
Closes #12464

Co-authored-by: Brandon Morelli <[email protected]>
kvch added a commit to kvch/beats that referenced this issue Jan 7, 2021
This PR makes credential settings when deploying Lambdas to AWS more flexible. New options are introduced:

1. `access_key_id`, `secret_access_key` and/or `session_token` for tokens

```yaml
functionbeat.provider.aws.access_key_id: '${AWS_ACCESS_KEY_ID:""}'
functionbeat.provider.aws.secret_access_key: '${AWS_SECRET_ACCESS_KEY:""}'
functionbeat.provider.aws.session_token: '${AWS_SESSION_TOKEN:""}'
```
2. `role_arn` for assuming IAM roles
```yaml
functionbeat.provider.aws.role_arn: arn:aws:iam::123456789012:role/test-fnb
```

3. `credential_profile_name` and/or `shared_credential_file` for credential files
```yaml
functionbeat.provider.aws.credential_profile_name: fnb-aws
functionbeat.provider.aws.shared_credential_file: /etc/functionbeat/aws_credentials
```

Credential configuration becomes more flexible and follows the same pattern as in Filebeat and Metricbeat.

Based on elastic#17658
Closes elastic#12464

Co-authored-by: Brandon Morelli <[email protected]>
(cherry picked from commit 5e6558b)
kvch added a commit that referenced this issue Jan 7, 2021
…3386)

This PR makes credential settings when deploying Lambdas to AWS more flexible. New options are introduced:

1. `access_key_id`, `secret_access_key` and/or `session_token` for tokens

```yaml
functionbeat.provider.aws.access_key_id: '${AWS_ACCESS_KEY_ID:""}'
functionbeat.provider.aws.secret_access_key: '${AWS_SECRET_ACCESS_KEY:""}'
functionbeat.provider.aws.session_token: '${AWS_SESSION_TOKEN:""}'
```
2. `role_arn` for assuming IAM roles
```yaml
functionbeat.provider.aws.role_arn: arn:aws:iam::123456789012:role/test-fnb
```

3. `credential_profile_name` and/or `shared_credential_file` for credential files
```yaml
functionbeat.provider.aws.credential_profile_name: fnb-aws
functionbeat.provider.aws.shared_credential_file: /etc/functionbeat/aws_credentials
```

Credential configuration becomes more flexible and follows the same pattern as in Filebeat and Metricbeat.

Based on #17658
Closes #12464

Co-authored-by: Brandon Morelli <[email protected]>
(cherry picked from commit 5e6558b)
@zube zube bot removed the [zube]: Done label Apr 7, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet