AWS permission

[Ilurusheshasai]

Hi everyone, I am getting the following error can anyone help me with the next steps?

Here is what I have done, I have created an EC2 instance, created virtual env, git cloned repo, then updated the aws_credentials.json and global configuration.json files with correct values and installed the requirements as described here. I am getting this error when I try to run the "launch_script.py -help-setup-new-environment" which is the next step.

[ec2-user@ip---- cluster_management]$ python launch_script.py -help-setup-new-environment
Enter the name of the environment you want to create:
Names must be 4 to 40 characters in length.
Names can only contain letters, numbers, and hyphens, and cannot start or end with a hyphen.

test123
Traceback (most recent call last):
File "launch_script.py", line 631, in
do_help_setup_new_environment()
File "launch_script.py", line 424, in do_help_setup_new_environment
do_fail_if_environment_exists(name)
File "launch_script.py", line 290, in do_fail_if_environment_exists
environment_exists = check_if_eb_environment_exists(name)
File "/home/ec2-user/Beiwe-UMBC/beiwe-backend/cluster_management/deployment_helpers/aws/elastic_beanstalk.py", line 335, in check_if_eb_environment_exists
extant_environments = get_environments_list()
File "/home/ec2-user/Beiwe-UMBC/beiwe-backend/cluster_management/deployment_helpers/aws/elastic_beanstalk.py", line 102, in get_environments_list
environments = create_eb_client().describe_environments()['Environments']
File "/home/ec2-user/.pyenv/versions/3.8.18/lib/python3.8/site-packages/botocore/client.py", line 553, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/ec2-user/.pyenv/versions/3.8.18/lib/python3.8/site-packages/botocore/client.py", line 1009, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidClientTokenId) when calling the DescribeEnvironments operation: The security token included in the request is invalid.

[biblicabeebli]

What are the permissions (IAM policy) assigned to the user that your AWS keys are attached to?

[Ilurusheshasai]

The issue is there is another team that gives access and they are not sharing the policy. If I can get a copy from you, I will share it with the admin team to get the required permissions.

[biblicabeebli]

Unfortunately I never sat down and came up with the minimum list of permissions for deploying a server, or for running the server. To build that list someone needs to go through every call made using Boto in the cluster_management code.

I think we've just been using administrator access credentials. This is one of those things that is quite bad practice and we really should deal with, I apologize that it is not.

(for running a server it should just be read and write and list and maybe head access to AWS S3.)

[Ilurusheshasai]

Hi Eli,

This is the policy attached to my role:

https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AdministratorAccess-AWSElasticBeanstalk.html

[Ilurusheshasai]

Hi Eli,
I have managed to get the required AdministratorAccess for my credentials. Still facing the same error. Can you please suggest how I can overcome this error? I am using proper credentials and edited aws_credentials.json and global_configuration.json as directed on the git page.

Note: I was able to use CLI to describe the Beanstalk environments but it failed with the Python script.

ERROR:
Python 3.8.0 (default, Apr 17 2024, 23:58:37)
[GCC 11.4.1 20230605 (Red Hat 11.4.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.

import boto3
def _get_client(client_type):
""" connect to a boto3 CLIENT in the appropriate type and region. """
return boto3.client(
'elasticbeanstalk',
aws_access_key_id="xyz",
aws_secret_access_key="xyz",
region_name="us-east-2",
)
...
def create_eb_client():
return _get_client('elasticbeanstalk')
...
### environments = create_eb_client().describe_environments()
Traceback (most recent call last):
File "", line 1, in
File "/home/ec2-user/.pyenv/versions/3.8.0/lib/python3.8/site-packages/botocore/client.py", line 565, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/ec2-user/.pyenv/versions/3.8.0/lib/python3.8/site-packages/botocore/client.py", line 1021, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidClientTokenId) when calling the DescribeEnvironments operation: The security token included in the request is invalid.

[Ilurusheshasai]

Hi Eli, Please confirm if this is accurate.

The reason for this error is that my credentials are SSO-generated. If your user is an SSO user (has temporary credential keys, and an extra variable access token which needs to be passed to access the service) instead of an IAM user (has permanent keys( access key ID and access key)). We need to modify the code as shown below:

Change 1: This is the first file you need to change cluster_management/general_configuration/aws_credentials.json include AWS_SESSION_TOKEN variable

{
"AWS_ACCESS_KEY_ID" : "FILL ME IN!",
"AWS_SECRET_ACCESS_KEY" : "AND ME TOO!"
"AWS_SESSION_TOKEN" : "ADD THIS IF YOUR CREDENTIALS ARE CREATED USING SSO, INSTEAD OF IAM"
}

Change 2:
cluster_management/deployment_helpers/boto_helpers.py
Function _get_client needs to be modified to pick aws_session_token, need to update _get_resource function as well similarly

def _get_client(client_type):
""" connect to a boto3 CLIENT in the appropriate type and region. """
return boto3.client(
client_type,
aws_access_key_id=AWS_CREDENTIALS["AWS_ACCESS_KEY_ID"],
aws_secret_access_key=AWS_CREDENTIALS["AWS_SECRET_ACCESS_KEY"],
aws_session_token = AWS_CREDENTIALS["AWS_SESSION_TOKEN"],
region_name=GLOBAL_CONFIGURATION["AWS_REGION"],
)

Change 3: We need to add "AWS_SESSION_TOKEN" to the AWS_CREDENTIALS_FILE_KEYS list in cluster_management/deployment_helpers/constants.py
AWS_CREDENTIALS_FILE_KEYS = ["AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY", "AWS_SESSION_TOKEN"]

[biblicabeebli]

@Ilurusheshasai Apologies for the delay, I am reviewing your issues and emails now.

This should work for operations locally, the question is whether it continues to work in the future, I don't know exactly how to quantify "temporary", and I'm not a world-class expert on all the credentialing types availabe within AWS (there's a lot).

There is another function, _get_resource that is virtually identical and I assume needs the same change.
I've changed it up in order to preserve compatibility with existing user accounts.

I've made a branch for this, improve-launch-creds would you mind testing it? all you have to do is swap to the branch and run something like python manage_shell.py (ipython -i manage_shell.py if you have ipython installed, is the expected usage). This will load the debugging/testing shell, which automatically instantiates one of each of the boto3 clients and a boto3 resource. It should error immediately if the parameters passed in to those are wrong.

The new code looks like this:

def _prepare_credentials() -> dict:
    # general parameters for boto3 clients and resources.
    params = {
        "aws_access_key_id": AWS_CREDENTIALS["AWS_ACCESS_KEY_ID"],
        "aws_secret_access_key": AWS_CREDENTIALS["AWS_SECRET_ACCESS_KEY"],
        "region_name": GLOBAL_CONFIGURATION["AWS_REGION"],
    }
    
    # if the configuration info contains AWS_SESSION_TOKEN (required for an SSO credentials) we need
    # to add the session token to the client.
    if "AWS_SESSION_TOKEN" in AWS_CREDENTIALS:
        params["aws_session_token"] = AWS_CREDENTIALS["AWS_SESSION_TOKEN"]
    
    return params


def _get_client(client_type):
    """ connect to a boto3 CLIENT in the appropriate type and region. """
    return boto3.client(client_type, **_prepare_credentials())


def _get_resource(client_type):
    """ connect to a boto3 RESOURCE in the appropriate type and region. """
    return boto3.resource(client_type, **_prepare_credentials())

[biblicabeebli]

(reminder to self to not close this issue. I need to put together some documentation, or maybe wrap those boto3 client instantiations in an error catch that prints something helpful.)

[Ilurusheshasai]

Hi @biblicabeebli, Yes, things are working well now.

Please note, I have switched to the new branch and added "AWS_SESSION_TOKEN" to AWS_CREDENTIALS_FILE_KEYS = ["AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY"] in constants.py. If I do not do this I get the below error.

python launch_script.py -help-setup-new-environment
ERROR cluster-management: a key 'AWS_SESSION_TOKEN' is present in general_configuration/aws_credentials.json, but was not expected.
ERROR cluster-management: before you can take any action with this tool you must fill out the contents of the aws_credentials.json file in the general_configuration folder.

To avoid the above error,
I think we need to update the list with the variable name AWS_CREDENTIALS_FILE_KEYS in constant.py to help SSO credentials. But not sure if we add this and not have this AWS_SESSION_TOKEN variable in JSON for regular IAM users throws an error. I am not able to check it as I don't have a regular IAM user.

[biblicabeebli]

And that's why we test!
I'll fiddle with the credential validation a bit.

This does raise the question - did you just do it ~manually, not through the credential file?

[biblicabeebli]

Ok I did a possible update fix on that branch, but I'm on the wrong computer to do a test of it, could you pull and see if I screwed it up?