Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Client Side Encryption to S3 client #3395

Open
2 tasks
frittentheke opened this issue Aug 18, 2022 · 13 comments
Open
2 tasks

Add Client Side Encryption to S3 client #3395

frittentheke opened this issue Aug 18, 2022 · 13 comments
Labels
feature-request This issue requests a feature. p2 This is a standard priority issue s3

Comments

@frittentheke
Copy link

frittentheke commented Aug 18, 2022

Describe the feature

While there is support for client-side encryption in other popular language SDKs like Java, Go or C++ (https://docs.aws.amazon.com/general/latest/gr/aws_sdk_cryptography.html) for encrypting data before it's sent to AWS S3 this is not supported by boto3 (Python3).

The documentation does talk about using the Encryption SDK (https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingClientSideEncryption.html), but this is produces a different data format:

The AWS Encryption SDK and the Amazon S3 encryption clients are not compatible because they produce ciphertexts with different data formats. For more information about the AWS Encryption SDK, see the AWS Encryption SDK Developer Guide.

(https://docs.aws.amazon.com/general/latest/gr/aws_sdk_cryptography.html)

Use Case

With no support for the otherwise common format for client side encryption, that is used for other language SDKs, it's not possible to e.g. write encrypted objects via Java and to read them back via a Python3 client using boto3.

Also there are quite a few tools using boto3 as S3 client library which would then allow for client side encryption to be used:

Proposed Solution

There have been attempts to wrap the boto3 client and to add client encryption externally, like

  1. @boldfield implemented via https://github.com/boldfield/s3-encryption, continued by @StephenSorriaux (https://github.com/StephenSorriaux/s3-encryption)
  2. @terrycain added this for aioboto3, see https://github.com/terrycain/aioboto3/blob/07d6cea1214a9bb6529f334e375134554ad7ab58/README.rst#amazon-s3-client-side-encryption, unfortunately only for KMS managed keys yet
  3. ....

Proposed solution would be to have this feature natively provided by boto3 and to make the SDKs compatible in this regard.

Other Information

No response

Acknowledgements

  • I may be able to implement this feature request
  • This feature might incur a breaking change

SDK version used

any

Environment details (OS name and version, etc.)

Linux

@frittentheke frittentheke added feature-request This issue requests a feature. needs-triage This issue or PR still needs to be triaged. labels Aug 18, 2022
@tim-finnigan tim-finnigan added investigating This issue is being investigated and/or work is in progress to resolve the issue. and removed needs-triage This issue or PR still needs to be triaged. labels Aug 18, 2022
@StephenSorriaux
Copy link

Hey,

Thanks for citing my work here. FWIW my fork also has an (undocumented) async client available, and I was able use the Java SDK to decrypt what my Python lib encrypted (which was awesome to see).

@tim-finnigan
Copy link
Contributor

Thanks @frittentheke for the feature request. The cryptography team is working towards taking ownership of the S3 Encryption Client implementations in various languages. Supporting boto3 is on their roadmap but they don’t have a concrete timeline as of yet.

@frittentheke
Copy link
Author

Thanks @frittentheke for the feature request. The cryptography team is working towards taking ownership of the S3 Encryption Client implementations in various languages. Supporting boto3 is on their roadmap but they don’t have a concrete timeline as of yet.

Thanks for the quick and positive response!
Is the roadmap accessible anywhere or is there another issue that could be tracked for this feature?

@tim-finnigan
Copy link
Contributor

Hi @frittentheke thanks for following up. There is no public roadmap for this to my knowledge. But the Cryptography team said this feature isn't likely to be implemented in the immediate future. I recommend reaching out through AWS Support to seek more information as needed.

@frittentheke
Copy link
Author

Hi @frittentheke thanks for following up. There is no public roadmap for this to my knowledge. But the Cryptography team said this feature isn't likely to be implemented in the immediate future. I recommend reaching out through AWS Support to seek more information as needed.

How would support be able to help here @tim-finnigan?
Could you kindly mention anybody from that particular team to maybe get a feel for how far up in the queue the client-side-encryption feature is?

@tim-finnigan
Copy link
Contributor

Hi @frittentheke - I suggested reaching out to AWS Support as that could help with establishing a more direct correspondence regarding this particular feature request going forward. The information passed on to me by the cryptography team was that this is on their roadmap but they don't yet have an official timeline, and as I mentioned this isn't something that is likely to prioritized in the near future. But if others share their use cases and how the lack of this feature is a blocker for them then it could help with escalating the priority.

@tim-finnigan tim-finnigan added s3 and removed investigating This issue is being investigated and/or work is in progress to resolve the issue. labels Oct 31, 2022
@aBurmeseDev aBurmeseDev added the p2 This is a standard priority issue label Nov 9, 2022
@timetxt
Copy link

timetxt commented Aug 12, 2023

tim-finnigan

AWS Support has the same or actually less visibility and connection with cryptography team comparing with anyone this ticket. Reaching out AWS Support will not give the information that you are looking for.

@timetxt
Copy link

timetxt commented Aug 12, 2023

also for anyone found this issue post, s3crypto is supported in language like Go. AWS SDK for Go has two versions, V1 and V2. V1 is still supporting s3crypto while s3crypto is removed from V2 at the moment

V1: https://docs.aws.amazon.com/sdk-for-go/api/service/s3/s3crypto/#pkg-overview

V2 document when s3crypto was supported in V2: https://pkg.go.dev/github.com/aws/[email protected]/service/s3/s3crypto

@ristosaari
Copy link

I would appreciate if Python boto3 support would be given out of the box fox SDK.

@medmunds
Copy link

medmunds commented Sep 2, 2024

Another use case: Amazon SES's email receiving has a "message encryption" option for its "Deliver to S3 bucket" action. But it requires using an S3 encryption client to decrypt the S3 objects: https://docs.aws.amazon.com/ses/latest/dg/receiving-email-action-s3.html.

There is currently no AWS-supported Python option for using SES email receiving with S3 message encryption enabled. (And the third-party Python packages seem to be outdated.)

@briveira
Copy link

briveira commented Sep 4, 2024

Two years later, AWS Python SDK still lacks client side encryption with S3. Can't believe it! :-o

@simmosn
Copy link

simmosn commented Oct 4, 2024

FYI, my org has raised a feature request for this with our AWS rep

@Hoeze
Copy link

Hoeze commented Dec 24, 2024

any updates on this topic?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature-request This issue requests a feature. p2 This is a standard priority issue s3
Projects
None yet
Development

No branches or pull requests

10 participants