Issue with AWS API Gateway signed requests in k6 tests

[rgrygorovych]

Dear Team,

We are currently working on integrating k6 tests with our private AWS API Gateway. While doing so, we encountered an issue with performing signed requests using k6 internal libraries.

After some investigation, we found a workaround by modifying the signature implementation in the signature.ts file. Specifically, we changed the request header to accommodate a different hostname value compared to the host value in the header.

The original line in signature.ts:

request.headers[constants.HOST_HEADER] = request.endpoint.hostname

was replaced with:

request.headers[constants.HOST_HEADER] = request.headers.host

This change allowed us to successfully perform requests from the open-source k6 tool to our API. Currently, we have forked the k6-jslib-aws repository. However, we would like to seek clarification on whether this modification indicates an issue in the implementation. Should the change be made on your end, or is there something we may be doing incorrectly on our side?

Your guidance on this matter would be greatly appreciated.

[joanlopez]

Hey @rgrygorovych,

First of all, thanks so much for bringing this topic! 🙏🏻

The original line in signature.ts:

request.headers[constants.HOST_HEADER] = request.endpoint.hostname

was replaced with:

request.headers[constants.HOST_HEADER] = request.headers.host

Looking at the line of code you pointed out, and comparing it to the one you provided, I suspect the problem, in your case, is that the original value of the Host header is being overwritten by request.endpoint.hostname, because I think the line you suggest does nothing: aren't request.headers[constants.HOST_HEADER] and request.headers.host the same thing?

If so, can you confirm that the header value (request.headers[constants.HOST_HEADER]) and the endpoint's hostname (request.endpoint.hostname) are different in your case? Is that expected for some reason? Can you bring further details?

In such case, I'm wondering whether would make sense to slightly modify the current code to only set the value of request.headers[constants.HOST_HEADER] when empty (not set), otherwise just leave the existing value.

So, something like:

if (!request.headers[constants.HOST_HEADER]) {
    request.headers[constants.HOST_HEADER] = request.endpoint.hostname;
}

Do you think that would make sense? Could you verify that would work in your case?
cc/ @oleiade, as you may have more context.

Thanks! 🙇🏻

[oleiade]

Out of the blue with my "cache" cold, I think the solution you proposed @joanlopez makes sense indeed. I'd be interested in learning wether that addresses your problem indeed @rgrygorovych.

For context the sign and presign methods were heavily inspired by the implementation the aws-sdk-js v3 had at the time. Unfortunately, it looks like AWS switched to code generation in the meantime, and I can't find that reference code anymore.

Furthermore, we adjusted it to use the request.endpoint instead, in order to cater to users with an "alternative" backend (minIO, digital ocean spaces, etc).

I think we should be open to making changes to it indeed, but we should provide tests ensuring it passes the signature test suite (also ported from the initial AWS implementation), because the SignatureV4 class is key to all our client classes 👍🏻

[rgrygorovych]

Hello @oleiade and @joanlopez here is more details related to this, hope you find it helpful.

Context: The team is working on accessing API endpoints that require a signature through a VPC endpoint. The following log outputs illustrate the issue and the suggested change:

Code Before Override:


        console.log("BEFORE")
        console.log("request.headers[constants.HOST_HEADER] = ", request.headers[constants.HOST_HEADER]);
        console.log("request.endpoint.hostname = ", request.endpoint.hostname);
        console.log("request.headers.host = ", request.headers.host);
        
        request.headers[constants.HOST_HEADER] = request.endpoint.hostname

        console.log("AFTER")
        console.log("request.headers[constants.HOST_HEADER] = ", request.headers[constants.HOST_HEADER]);
        console.log("request.endpoint.hostname = ", request.endpoint.hostname);
        console.log("request.headers.host = ", request.headers.host);

...

Result with failing POST Requests:

time="2024-03-26T19:09:47Z" level=info msg=BEFORE source=console
time="2024-03-26T19:09:47Z" level=info msg="request.headers[constants.HOST_HEADER] =  xxxxxxxxxx.execute-api.us-east-1.amazonaws.com" source=console
time="2024-03-26T19:09:47Z" level=info msg="request.endpoint.hostname =  vpce-yyyyyyyyy-zzzzzzz.execute-api.us-east-1.vpce.amazonaws.com" source=console
time="2024-03-26T19:09:47Z" level=info msg="request.headers.host =  xxxxxxxxxx.execute-api.us-east-1.amazonaws.com" source=console
time="2024-03-26T19:09:47Z" level=info msg=AFTER source=console
time="2024-03-26T19:09:47Z" level=info msg="request.headers[constants.HOST_HEADER] =  vpce-yyyyyyyyy-zzzzzzz.execute-api.us-east-1.vpce.amazonaws.com" source=console
time="2024-03-26T19:09:47Z" level=info msg="request.endpoint.hostname =  vpce-yyyyyyyyy-zzzzzzz.execute-api.us-east-1.vpce.amazonaws.com" source=console
time="2024-03-26T19:09:47Z" level=info msg="request.headers.host =  vpce-yyyyyyyyy-zzzzzzz.execute-api.us-east-1.vpce.amazonaws.com" source=console

Another execution with the proposed change:

Code After Override:

        console.log("BEFORE")
        console.log("request.headers[constants.HOST_HEADER] = ", request.headers[constants.HOST_HEADER]);
        console.log("request.endpoint.hostname = ", request.endpoint.hostname);
        console.log("request.headers.host = ", request.headers.host);
        
        // Change to accommodate for hostname to be different from host value in header 
        if (!request.headers[constants.HOST_HEADER]) {
            request.headers[constants.HOST_HEADER] = request.endpoint.hostname;
        }
        
        console.log("AFTER")
        console.log("request.headers[constants.HOST_HEADER] = ", request.headers[constants.HOST_HEADER]);
        console.log("request.endpoint.hostname = ", request.endpoint.hostname);
        console.log("request.headers.host = ", request.headers.host);
...

Result with passing POST Requests:

time="2024-03-26T19:05:21Z" level=info msg=BEFORE source=console
time="2024-03-26T19:05:21Z" level=info msg="request.headers[constants.HOST_HEADER] =  xxxxxxxxxx.execute-api.us-east-1.amazonaws.com" source=console
time="2024-03-26T19:05:21Z" level=info msg="request.endpoint.hostname =  vpce-yyyyyyyyy-zzzzzzz.execute-api.us-east-1.vpce.amazonaws.com" source=console
time="2024-03-26T19:05:21Z" level=info msg="request.headers.host =  xxxxxxxxxx.execute-api.us-east-1.amazonaws.com" source=console
time="2024-03-26T19:05:21Z" level=info msg=AFTER source=console
time="2024-03-26T19:05:21Z" level=info msg="request.headers[constants.HOST_HEADER] =  xxxxxxxxxx.execute-api.us-east-1.amazonaws.com" source=console
time="2024-03-26T19:05:21Z" level=info msg="request.endpoint.hostname =  vpce-yyyyyyyyy-zzzzzzz.execute-api.us-east-1.vpce.amazonaws.com" source=console
time="2024-03-26T19:05:21Z" level=info msg="request.headers.host =  xxxxxxxxxx.execute-api.us-east-1.amazonaws.com" source=console

Summary: Requests were failing before the change but succeeded after the change was applied.

We were inspired by the following reference materials:

• The fix proposed in a similar issue on the AWS SDK for Ruby GitHub repository: PR #2345.
• The documentation on testing invoke URLs for private APIs in the AWS API Gateway Developer Guide: AWS API - Gateway Private API Test Invoke URL.

[joanlopez]

Great! Thanks for such a detailed summary @rgrygorovych!

Do you want to open the pull request with the change agreed? I think you did the hard work, so you definitely deserve the credits for that contribution! 🙇🏻

[rgrygorovych]

Great! Thanks for such a detailed summary @rgrygorovych!

Do you want to open the pull request with the change agreed? I think you did the hard work, so you definitely deserve the credits for that contribution! 🙇🏻

sure, just need permissions :).

[joanlopez]

sure, just need permissions :).

Can you fork it and create the PR from your own fork, please? 🙏🏻

[rgrygorovych]

sure, just need permissions :).

Can you fork it and create the PR from your own fork, please? 🙏🏻

Here is link #97