Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pre-signed URL issues (generate_presigned_post and generate_presigned_url) #1982

Closed
pcraciunoiu opened this issue May 29, 2019 · 18 comments
Closed
Assignees
Labels

Comments

@pcraciunoiu
Copy link

pcraciunoiu commented May 29, 2019

Hi there,

I'm having two issues which I believe are related. I've already spent a few hours trying to figure this out and it looks like a bug. I finally stumbled on a solution that works, but it's not ideal.

Here is the setup:

Doesn't work

s3 = boto3.client(
    "s3",
    aws_access_key_id=self.aws_access_key_id,
    aws_secret_access_key=self.aws_secret_access_key,
)
signature = s3.generate_presigned_post(
    Bucket=bucket,
    Key=key,
    Fields={"acl": "private", "Content-Type": file_type},
    Conditions=[{"acl": "private"}, {"Content-Type": file_type}],
    ExpiresIn=expires_in,
)

Results in signature["url"] = "https://{bucket}.s3.amazonaws.com/"
Should instead be "https://{bucket}.s3.{region}.amazonaws.com/"

Without region, there is a redirect on the frontend which breaks CORS. See also #421.

url = s3.generate_presigned_url(
    'get_object',
    Params={
        'Bucket': bucket,
        'Key': key,
    },
    HttpMethod="GET",
)

This URL always results in SignatureDoesNotMatch. Presumably also because it is missing the region, and it also gets redirected.

For what it's worth, the region I'm testing with is 'us-east-2'

Works

# This needs to be changed for the frontend to upload
signature["url"] = f"https://{self._bucket}.s3.{self._region}.amazonaws.com/"

# This client init config makes the signature work for download.
s3 = boto3.client(
    "s3",
    "us-east-2",  # Not specifying this breaks.
    config=Config(s3={'addressing_style': 'path'})
    aws_access_key_id=self.aws_access_key_id,
    aws_secret_access_key=self.aws_secret_access_key,
)

This is not an acceptable path forward because AWS is retiring the path addressing_style.

Remainder of the code the same.

Any help greatly appreciated! Thank you for your hard work maintaining this library.

@pcraciunoiu
Copy link
Author

FWIW I've also looked at #1644 (this comment helped) and #923.

@swetashre
Copy link
Contributor

@pcraciunoiu - Thank you for your post. As per the documentation the response url doesn't include region. This is the expected url signature["url"] = "https://{bucket}.s3.amazonaws.com/"

And for the second example when i run your code i am not getting any error with the latest version of boto3.
It is possible that switching from virtual to path addressing style could fix pre-signed URLs signed using SigV4. For SigV4 the host is signed as part of the signature, which can cause problems for newly created buckets when virtual addressing style is being used. S3 will redirect to a different host for buckets that DNS haven't propagated for, which leads signature mismatch errors as the host is no longer correct.

Hope it helps ad let me know if you have any further questions.

@swetashre swetashre added s3 closing-soon This issue will automatically close in 4 days unless further comments are made. labels May 30, 2019
@pcraciunoiu
Copy link
Author

@swetashre but the URL always gets redirected, what's the point of generating it without the region?

So you were able to run it with virtual and the file loads in your browser for the presigned URL? I am using the code above with defaults and I'm not sure why it's not working. Can you post a sample of what you get... I tried this a bunch of different times and was not able to get a URL that actually downloads the file for virtual.

@no-response no-response bot removed the closing-soon This issue will automatically close in 4 days unless further comments are made. label May 30, 2019
@pcraciunoiu
Copy link
Author

Also thank you for your prompt response. I really appreciate it!

@swetashre
Copy link
Contributor

@pcraciunoiu - Thank you for your reply. Here is the code snippet with response from generate_presigned_url

import boto3

s3 = boto3.client('s3')
url = s3.generate_presigned_url(
    'get_object',
     Params={
        'Bucket': 'testbucket',
        'Key': 'outfile.txt',
    },
    HttpMethod="GET",
)

Response url:
https://testbucket.s3.amazonaws.com/outfile.txt?AWSAccessKeyId=SRQPO4NMLKJIHNGFEDCB&Expires=1559248814&Signature=JPaP9naftETY516%2BQvcLb4SipVM%3D

@swetashre swetashre added the closing-soon This issue will automatically close in 4 days unless further comments are made. label May 30, 2019
@pcraciunoiu
Copy link
Author

Yeah, looks like only v4 is supported for this bucket, in us-east-2. I get this:

<Error>
<Code>InvalidRequest</Code>
<Message>
The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.
</Message>
<RequestId>9691BDEC22469DFC</RequestId>
<HostId>
JNbnKZ/bUKph5kHTSMMx6FRlkJPkrxj09hOalqRl0MA/BeaO+EJgoECXNeuIJRew98/TvglY2Lk=
</HostId>
</Error>

@no-response no-response bot removed the closing-soon This issue will automatically close in 4 days unless further comments are made. label May 30, 2019
@pcraciunoiu
Copy link
Author

If I do this instead, I get a different error:

import boto3
from botocore.client import Config

s3 = boto3.client(
    's3',
    config=Config(signature_version='s3v4')
)
url = s3.generate_presigned_url(
    'get_object',
     Params={
        'Bucket': 'testbucket',
        'Key': 'outfile.txt',
    },
    HttpMethod="GET",
)
<Error>
<script/>
<Code>AuthorizationQueryParametersError</Code>
<Message>
Error parsing the X-Amz-Credential parameter; the region 'us-east-1' is wrong; expecting 'us-east-2'
</Message>
<Region>us-east-2</Region>
<RequestId>BFD0841A02020B64</RequestId>
<HostId>
tQs0P8RtoCctKzyaIxGe6BpiiWN/yLBz1RheMKb2lqRe6UwPbXI1G5GqZxCdzB4zVrtvq6VurY0=
</HostId>
</Error>

And if I specify the region, then it works. So I may have stumbled upon a better option than doing path access. I'll try that out in the code path and see if it works.

For the URL in the signature, it seems to me like there should be a way to have it include the region to avoid the redirect...

@pcraciunoiu
Copy link
Author

OK, that worked. So it seems like for certain regions the region AND the config v4 need to both be specified. Otherwise the signed URLs don't work. Might need to update the docs for generate_presigned_url, or if that's already there I missed it...

As far as the URL signature, I am doing that string replace, but I'm open to better options. I'm curious why you wouldn't want to include it in the signature URL though.

@swetashre swetashre self-assigned this May 30, 2019
@pcraciunoiu
Copy link
Author

Just wanted to follow up that this doesn't work again. I'm not sure what changed, but now I get:

The request signature we calculated does not match the signature you provided. Check your key and signing method.

Again, switching to the path style fixes the issue. So the diff is just:

-                config=Config(signature_version="s3v4"),
+                config=Config(s3={'addressing_style': 'path'}),

@swetashre
Copy link
Contributor

@pcraciunoiu - Thanks for the feedback. Are you still getting the error ? There are certain regions which does not support signature version 2 and for that you have to use v4. Here is the link to documentation:
https://docs.aws.amazon.com/general/latest/gr/signature-version-2.html#signature-2-regions-services

It is possible that switching from virtual to path addressing style could fix pre-signed URLs signed using SigV4. For SigV4 the host is signed as part of the signature, which can cause problems for newly created buckets when virtual addressing style is being used. S3 will redirect to a different host for buckets that DNS haven't propagated for, which leads signature mismatch errors as the host is no longer correct.

Are you still getting error when you specify both signature version and region ? Can you please provide the debug log ?

@swetashre swetashre added the closing-soon This issue will automatically close in 4 days unless further comments are made. label Jun 6, 2019
@pcraciunoiu
Copy link
Author

@swetashre yes I got the error when I specify signature version and region. I wasn't able to reproduce it, so I'm not sure what's going on :) Maybe I'm crazy.

@no-response no-response bot removed the closing-soon This issue will automatically close in 4 days unless further comments are made. label Jun 6, 2019
@swetashre
Copy link
Contributor

@pcraciunoiu - Thanks for the reply. I am not able to reproduce your issue. It would have been very useful if we had gotten the debug log. Please make sure you are using correct region where the bucket exists and that region supports the specified signature version.

At any way it is working fine for you now.

@swetashre swetashre added the closing-soon This issue will automatically close in 4 days unless further comments are made. label Jun 6, 2019
@arcivanov
Copy link

Again, switching to the path style fixes the issue. So the diff is just:

-                config=Config(signature_version="s3v4"),
+                config=Config(s3={'addressing_style': 'path'}),

@pcraciunoiu Thing is, s3v4 is going to be the only mechanism starting June 20th as s3v2 is being decommissioned.

@pcraciunoiu
Copy link
Author

Thanks, not so relevant to the issue posted. I think the problem is caused by addressing_style sometimes, and specifying region and or signature v4 seems to fix the URL for us-east-2. This is not documented that I can tell, and it's not obvious what to do to fix when you encounter it.

@no-response no-response bot removed the closing-soon This issue will automatically close in 4 days unless further comments are made. label Jun 9, 2019
@swetashre
Copy link
Contributor

@pcraciunoiu - Thanks for all the feedback. I assume this issue has been solved for you. So i am closing this. Please reopen if you have any questions.

@ConnorWhalen
Copy link

fwiw i am also seeing this issue. Until a newly created bucket's global DNS gets set up, presigned URLs generated with generate_presigned_url return a redirect and fail CORS. Specifying the region and s3v4 don't fix this but path addressing does, though path addressing will be retired for new buckets next september. Returning the region-specific virtual address would fix this problem, and there doesn't seem to be a way to do that currently through this api. It's also only a problem until the bucket's global DNS gets set up, however long that may take (for me today in ca-central-1 it's ~2 hours)

@revmischa
Copy link

Does this work?

s3 = boto_session.client("s3", endpoint_url=f"https://s3.{region}.amazonaws.com")

@getup8
Copy link

getup8 commented Nov 28, 2019

@revmischa, I had a similar issue (I wanted virtual, regional URLs in my presigned URL), and used your setting but also needed to explicitly specify virtual paths in the config.

import boto3
from botocore.client import Config

region = 'us-east-1'

s3 = boto_session.client(
    's3',
    endpoint_url=f'https://s3.{region}.amazonaws.com',
    config=Config(s3={'addressing_style': 'virtual'}))

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

6 participants