switch server mode to proxies

[thehesiod]

botocore just added support for specifying proxy servers. It seems like it would be better to switch to using a single proxy, then specifying the endpoint_url, as it would require fewer changes on the client, and could support multiple services from a single endpoint. Could also solve the url parsing issue with aws region(s).

thoughts?

[JackDanger]

@thehesiod This sounds like a great idea to me.

[thehesiod]

based on research in #1317 it seems like one could either support this on the client side via Config.proxies, or http[s]_proxy environment variables.

[terricain]

Hmm, I "think" if we do https_proxy, we'd need to serve moto on https, and also trust whatever CA we're using in lambdas. If we only listen on http, we'll need to patch whatever library in use to default use_ssl=False

[jacobeatsspam]

I ended up here from #1317, and getting an error when trying to contact services inside lambda:

Unit test

    @moto.mock_lambda
    @moto.mock_iam
    @moto.mock_secretsmanager
    def test_lambda(self):
        # prep the environment
        self.config_path = '/HelloWorld'
        secretsmanager = boto3.client('secretsmanager')
        secretsmanager.create_secret(
            Name=self.config_path,
            SecretString='{}',
        )
        iam = boto3.client('iam')
        role = iam.create_role(
            RoleName='PythonExampleRole',
            AssumeRolePolicyDocument='{}',
            Path='/',
        )
        lambdaa = boto3.client('lambda')
        lambdaa.create_function(
            FunctionName='HelloWorld',
            Description='blurg',
            Runtime='python3.8',
            Role=role['Role']['Arn'],
            Code={ 'ZipFile': self._build() },
            Handler='src.hello_world.HelloWorldHandler',
            Environment={'Variables': {'CONFIG_PATH': self.config_path}},
            Timeout=3,
            MemorySize=128,
            Publish=True,
        )
        lambdaa.invoke(
            FunctionName='HelloWorld',
            InvocationType='RequestResponse',
            Payload='',
        )

and the error

The security token included in the request is invalid

My $0.02; If moto exposes the ability to invoke a lambda locally, it should automatically proxy the container. It's pretty much a guarantee that the user will be attempting to contact mocked services from within the lambda container.

I think this annoyance would be solved by this issue.

[bblommers]

moto >= 4.2.5.dev12 now contains a proxy!

It contains a SSL certificate generator, but for SDK's to trust these certificates, the SDK needs to be configured to use the certificate bundle that comes with Moto.
Alternatively, most SDK's support some version of verify=False.

Documentation on how to get started and configure this all:
http://docs.getmoto.org/en/latest/docs/proxy_mode.html

[b-guerra]

@bblommers , that's great news!

I am trying to invoke a Lambda function to access my mock dynamoDB, so in order to do so, I am using the moto_proxy
It is starting without any problems:

$ moto_proxy -H 127.0.0.1
Call `moto_proxy -h` for example invocations
Serving HTTP Proxy on 127.0.0.1:5005 ..

And whenever I call my mock tests, it creates the dynamoDB and lambda function without issues, however, whenever I try to reach the dynamoDB via the proxy server, it just keeps hanging and hanging, and nothing ever reaches my server. Code snippet below:

session = boto3.session.Session(
    aws_access_key_id='mock',
    aws_secret_access_key='mock',
    aws_session_token='us-east-1',
)

url = "http://127.0.0.1:5005"
config = Config(proxies={"https": url})
dynamodb = session.resource('dynamodb', config=config, verify=False)
table = dynamodb.Table('MyTable')

From my understanding, the lambda function, when invoked, it runs within a container (i.e. not being able to reach the localhost that started it), but I thought the moto proxy would be helping specifically those scenarios. Do you have any tips? Thanks

[bblommers]

Hi @b-guerra, the proxy would have to be run on 0.0.0.0 in order for it to be reachable from inside Docker:
moto_proxy -H 0.0.0.0

[b-guerra]

Thanks for the quick reply @bblommers!!

$ moto_proxy -H 0.0.0.0
Call `moto_proxy -h` for example invocations
Serving HTTP Proxy on 0.0.0.0:5005 ...

I changed the proxy inside the code to 0.0.0.0,
However, I am still having the same error :(

Here is the script invoking the lambda

        url = "http://0.0.0.0:5005"
        config = Config(proxies={"https": url})
        lambda_client = session.client('lambda', region_name='us-east-1', config=config, verify=False)
        lambda_client.update_function_code(FunctionName='MyLambda', ZipFile=zip_buffer.read())

        # JSON data you want to send to the Lambda function
        json_data = {
            'id': 'value',
            'another_key': 'another_value'
        }

        # Invoke the Lambda function with the JSON data
        lambda_response = lambda_client.invoke(
            FunctionName='MyLambda',
            InvocationType='RequestResponse',
            Payload=json.dumps({'body': json.dumps(json_data)})
        )

And here is the lambda function

def lambda_handler(event, context):
    try:

        session = boto3.session.Session(
            aws_access_key_id='mock',
            aws_secret_access_key='mock',
            aws_session_token='us-east-1',
        )

        url = "http://0.0.0.0:5005"
        config = Config(proxies={"https": url})
        dynamodb = session.resource('dynamodb', config=config, verify=False)
        table = dynamodb.Table('MyTable')

        # Parse the JSON input from the event
        json_data = json.loads(event.get('body'))
        response = table.put_item(Item=json_data)

        # Check the response to confirm successful update
        if response['ResponseMetadata']['HTTPStatusCode'] == 200:
            return {
                'statusCode': 200,
                'body': json.dumps('Data updated successfully')
            }
        else:
            return {
                'statusCode': 500,
                'body': json.dumps('Error updating data')
            }

    except Exception as e:
        response = {
            'statusCode': 500,
            'body': json.dumps(f'Error: {str(e)}')
        }

    return response

However, the error I am getting are still the same one:

{'errorMessage': '2023-10-05T13:08:34.657Z f49e6cb3-0e54-1f91-7e9b-60c83bfcba98 Task timed out after 3.00 seconds'}

Do you have any idea why?

[bblommers]

That sounds like you're connecting to the real AWS, @b-guerra. Moto only times out after 5 minutes.

I've created an example test repro to show that it does work.
The test file, based both on our internal tests and on your example: https://github.com/bblommers/moto-proxy-example/blob/main/test/test.py
The (straight-forward) configuration: https://github.com/bblommers/moto-proxy-example/blob/main/.github/workflows/test.yml
And the successful invocation: https://github.com/bblommers/moto-proxy-example/actions/runs/6423279685

The biggest difference, from what I can see, is that I didn't configure the proxy inside the Lambda handler. Because Moto already knows that it's running the Proxy, it automatically sets the required environment variables for every Lambda-invocation. So no further configuration is necessary.

[b-guerra]

First of all, thank you so much for the help @bblommers!

That is amazing man!! it is working now like a charm.
The real problem was that I was running everything under a VPN (which weirdly were blocking the moto server connection).

I wanted to ask a question, is it possible to create a API Gateway that has Cognito authentication, and invoke a lambda function making a call to the API all under moto?

Thank you!

[bblommers]

Happy to help @b-guerra!

Invoking a Lambda via the API isn't supported yet, and authorizers aren't invoked either. I'm not against the idea of implementing this in Moto though, so feel free to open a feature request is this is something you want.
There are a lot of parameters and code-paths to consider here, so if you do open a new ticket, please add as many details as possible - ideally even with a minimally reproducible example.