-
Notifications
You must be signed in to change notification settings - Fork 24.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Packet-based enc/dec cipher streams #49896
Merged
albertzaharovits
merged 63 commits into
elastic:repository-encrypted-client-side
from
albertzaharovits:packet-based-cipherstream-2
Jan 6, 2020
Merged
Packet-based enc/dec cipher streams #49896
albertzaharovits
merged 63 commits into
elastic:repository-encrypted-client-side
from
albertzaharovits:packet-based-cipherstream-2
Jan 6, 2020
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
albertzaharovits
added
>feature
:Security/Security
Security issues without another label
labels
Dec 6, 2019
Pinging @elastic/es-security (:Security/Security) |
…arch/repositories/encrypted/DecryptionPacketsInputStream.java Co-Authored-By: Tim Vernum <[email protected]>
Thank you for another thorough review @tvernum ! |
tvernum
approved these changes
Jan 6, 2020
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
@elasticmachine update branch |
albertzaharovits
added a commit
that referenced
this pull request
Nov 25, 2020
albertzaharovits
added a commit
to albertzaharovits/elasticsearch
that referenced
this pull request
Nov 30, 2020
This adds a new bare snapshot repository project which contains the classes implementing encryption (and decryption) input stream decorators that support mark and reset. Relates elastic#48221 , elastic#46170
albertzaharovits
added a commit
that referenced
this pull request
Dec 2, 2020
This builds upon the data encryption streams from #49896 to create an encrypted snapshot repository. The repository encryption works with the following existing repository types: FS, Azure, S3, GCS (possibly works with HDFS and URL, but these are not tested). The encrypted repository is protected by a password stored on every node's keystore. The repository keys (KEK - key encryption key) are generated from the password using the PBKDF2 function, and are used to encrypt (using the AES Wrap algorithm) other symmetric keys (referred to as DEK - data encryption keys) which are themselves used to encrypt the blobs of the regular snapshot. The platinum or enterprise licenses are required to snapshot to the encrypted repository, but no license is required to list or restore already encrypted snapshots.
albertzaharovits
added a commit
that referenced
this pull request
Dec 23, 2020
The client-side encrypted repository is a new type of snapshot repository that internally delegates to the regular variants of snapshot repositories (of types Azure, S3, GCS, FS, and maybe others but not yet tested). After the encrypted repository is set up, it is transparent to the snapshot and restore APIs (i.e. all snapshots stored in the encrypted repository are encrypted, no other parameters required). The encrypted repository is protected by a password stored on every node's keystore (which must be the same across the nodes). The password is used to generate a key encrytion key (KEK), using the PBKDF2 function, which is used to encrypt (using the AES Wrap algorithm) other symmetric keys (referred to as DEK - data encryption keys), which themselves are generated randomly, and which are ultimately used to encrypt the snapshot blobs. For example, here is how to set up an encrypted FS repository: ------ 1) make sure that the cluster runs under at least a "platinum" license (simplest test configuration is to put `xpack.license.self_generated.type: "trial"` in the elasticsearch.yml file) 2) identical to the un-encrypted FS repository, specify the mount point of the shared FS in the elasticsearch.yml conf file (on all the cluster nodes), e.g. `path.repo: ["/tmp/repo"]` 3) store the repository password inside the elasticsearch.keystore, *on every cluster node*. In order to support changing password on existing repository (implemented in a follow-up), the password itself must be names, e.g. for the "test_enc_key" repository password name: `./bin/elasticsearch-keystore add repository.encrypted.test_enc_pass.password` *type in the password* 4) start up the cluster and create the new encrypted FS repository, named "test_enc", by calling: ` curl -X PUT "localhost:9200/_snapshot/test_enc?pretty" -H 'Content-Type: application/json' -d' { "type": "encrypted", "settings": { "location": "/tmp/repo/enc", "delegate_type": "fs", "password_name": "test_enc_pass" } } ' ` 5) the snapshot and restore APIs work unmodified when they refer to this new repository, e.g. ` curl -X PUT "localhost:9200/_snapshot/test_enc/snapshot_1?wait_for_completion=true"` Related: #49896 #41910 #50846 #48221 #65768
albertzaharovits
added a commit
to albertzaharovits/elasticsearch
that referenced
this pull request
Dec 23, 2020
The client-side encrypted repository is a new type of snapshot repository that internally delegates to the regular variants of snapshot repositories (of types Azure, S3, GCS, FS, and maybe others but not yet tested). After the encrypted repository is set up, it is transparent to the snapshot and restore APIs (i.e. all snapshots stored in the encrypted repository are encrypted, no other parameters required). The encrypted repository is protected by a password stored on every node's keystore (which must be the same across the nodes). The password is used to generate a key encrytion key (KEK), using the PBKDF2 function, which is used to encrypt (using the AES Wrap algorithm) other symmetric keys (referred to as DEK - data encryption keys), which themselves are generated randomly, and which are ultimately used to encrypt the snapshot blobs. For example, here is how to set up an encrypted FS repository: ------ 1) make sure that the cluster runs under at least a "platinum" license (simplest test configuration is to put `xpack.license.self_generated.type: "trial"` in the elasticsearch.yml file) 2) identical to the un-encrypted FS repository, specify the mount point of the shared FS in the elasticsearch.yml conf file (on all the cluster nodes), e.g. `path.repo: ["/tmp/repo"]` 3) store the repository password inside the elasticsearch.keystore, *on every cluster node*. In order to support changing password on existing repository (implemented in a follow-up), the password itself must be names, e.g. for the "test_enc_key" repository password name: `./bin/elasticsearch-keystore add repository.encrypted.test_enc_pass.password` *type in the password* 4) start up the cluster and create the new encrypted FS repository, named "test_enc", by calling: ` curl -X PUT "localhost:9200/_snapshot/test_enc?pretty" -H 'Content-Type: application/json' -d' { "type": "encrypted", "settings": { "location": "/tmp/repo/enc", "delegate_type": "fs", "password_name": "test_enc_pass" } } ' ` 5) the snapshot and restore APIs work unmodified when they refer to this new repository, e.g. ` curl -X PUT "localhost:9200/_snapshot/test_enc/snapshot_1?wait_for_completion=true"` Related: elastic#49896 elastic#41910 elastic#50846 elastic#48221 elastic#65768
albertzaharovits
added a commit
that referenced
this pull request
Dec 28, 2020
The client-side encrypted repository is a new type of snapshot repository that internally delegates to the regular variants of snapshot repositories (of types Azure, S3, GCS, FS, and maybe others but not yet tested). After the encrypted repository is set up, it is transparent to the snapshot and restore APIs (i.e. all snapshots stored in the encrypted repository are encrypted, no other parameters required). The encrypted repository is protected by a password stored on every node's keystore (which must be the same across the nodes). The password is used to generate a key encrytion key (KEK), using the PBKDF2 function, which is used to encrypt (using the AES Wrap algorithm) other symmetric keys (referred to as DEK - data encryption keys), which themselves are generated randomly, and which are ultimately used to encrypt the snapshot blobs. For example, here is how to set up an encrypted FS repository: ------ 1) make sure that the cluster runs under at least a "platinum" license (simplest test configuration is to put `xpack.license.self_generated.type: "trial"` in the elasticsearch.yml file) 2) identical to the un-encrypted FS repository, specify the mount point of the shared FS in the elasticsearch.yml conf file (on all the cluster nodes), e.g. `path.repo: ["/tmp/repo"]` 3) store the repository password inside the elasticsearch.keystore, *on every cluster node*. In order to support changing password on existing repository (implemented in a follow-up), the password itself must be names, e.g. for the "test_enc_key" repository password name: `./bin/elasticsearch-keystore add repository.encrypted.test_enc_pass.password` *type in the password* 4) start up the cluster and create the new encrypted FS repository, named "test_enc", by calling: ` curl -X PUT "localhost:9200/_snapshot/test_enc?pretty" -H 'Content-Type: application/json' -d' { "type": "encrypted", "settings": { "location": "/tmp/repo/enc", "delegate_type": "fs", "password_name": "test_enc_pass" } } ' ` 5) the snapshot and restore APIs work unmodified when they refer to this new repository, e.g. ` curl -X PUT "localhost:9200/_snapshot/test_enc/snapshot_1?wait_for_completion=true"` Related: #49896 #41910 #50846 #48221 #65768
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds a new bare snapshot repository project which contains the classes implementing encryption (and decryption) input stream decorators that support mark and reset.
Relates #48221 , #46170
Edit:
Extract from javadocs explaining how encryption works:
and how decryption works: