Bug 27755 - Using the Subtle Crypto Interface with Streams #73
Comments
|
After listening to Ryan rage about the use of BER encoding for ASN.1 objects, I have a feeling that this should be closed as won't fix because it presents a security issue. When one looks at the encrypt/decrypt APIs for authenticated encryption, it is required that the entire stream be observed on the decrypt side and could be argued that it needs to be observed on the encrypt side prior to emitting the processed stream. This is due to the fact that if the decryption process does not validate then no output is to be produced for consumption. Allowing this to be done in a streaming fashion means that the browser potentially needs to have an infinite size buffer to hold the intermediate result to be returned to the client. Similar issues hold for processing of signature values for the new X448 EdDSA algorithm where the message M is hashed twice. Allowing for an indefinite length input means that there are potential buffer overrun problems. |
|
Node.js has a streaming crypto API without any security issues: const crypto = require('crypto');
const hash = crypto.createHash('sha256');
hash.update('some data to hash');
hash.update('more data');
hash.update('even more data');
console.log(hash.digest('hex'));Why can't the web platform? |
|
I absolutely agree with @feross on this. Most (if not all) of the APIs can work in a streaming mode without any security issues. In fact, this is how these APIs are exposed in OpenSSL, so they always work in a streaming mode under the hood anyway, regardless of what high-level API may look like. |
|
All of the current hash functions that I am familiar with will allow for streaming APIs because they are built using a Merkle–Damgård construction. This means that they are processed on a block by block basis. However there are algorithms for which this is not doable. For example, the EdDSA algorithm that I mentioned above computes: R = fn( SHAKE256(dom(F, C) || prefix || M, 114) ) as you can see, you need to all of the message M to compute R before you can start doing the computation of k. This means that the entire message needs to be buffered unlike the hash example you gave above. Note also the comment that I made on authenticated decryption where the entire message needs to be kept before doing the validation step at the end. |
|
@jimsch in your description Authenticated decryption should work as well, as far as I can tell... Though, the fact that the integrity is checked only at the end of decryption process means that the API will be kind of awkward. I don't think that there are much pros of using streams for authenticated decryption. |
|
@indutny please re-read my previous post and look at the requirements to finish R before using M for k |
|
@jimsch oh I see it now. Sorry about that! Yeah, streaming won't work for this kind of encryption/decryption schemes indeed. Still many hashes and ciphers work just fine with streams. |
|
A native streaming api would indeed be great. Our use-case would be large file encryption in OpenPGP.js. |
|
If we address this, I think it will not be in this version since it requires substantial work. |
|
I imagine we can close this as won't fix, but when streaming stabilizes we can then revisit as part of maintenance of the spec since as @jimsch correctly points out, it won't work for quite a few algorithms. We could also try to test to see if anyone supports streaming - any ideas? |
|
v.Next. |
|
Is there any update on this topic? |
|
+1 |
|
If streaming/progressive encryption isn't implemented, it's going to hugely limit the scope of usage of the API. I really need that kind of functionality for the software I work on. |
|
+1! |
|
Privacy is a groing concern. Being able to decrypt locally without consuming too much memory is a must i think. |
|
I think digest maybe a good point to start with. |
|
+1000 |
|
Hi all, bringing this issue back up. Any updates or recent discussion on it? I believe that the security considerations do not hold and it what it promotes is for users to find other ways to encrypt their files as the usage of browsers to share large documents grows. Possibly by having to shim their own encryption streaming API which will be considerably slower than a native one through WebCrypto. |
|
+1 |
|
100% agreed with @ericmackrodt & @neckaros & @diasdavid. With GDPR on the horizon this would make things a lot more easier for European establishments. |
|
@jimsch By any chance, could a streaming API be provided for those encryption schemes that could be streamed? Just because it's not possible for some doesn't make it impossible for all (and there's different tradeoffs for each). And one good example of this is with client decryption of large files on mobile (only high end phones/tablets have the RAM available to reliably decrypt a 750MB video download in-memory). |
|
It could, on the other hand there may be other things that could be done as well. For example one could do chunked encryption of large objects such as video which is designed to be streamed so that each chunk can be independently decrypted and streamed. The world is moving towards only using authenticated encrypted algorithms and doing streaming such as you suggest means that you are willing to use a decrypted stream that may have been corrupted w/o being able to detect this. Additionally, one would need to get a group of people together at the W3C who are interested in doing an update to the document and then decide which algorithms could/should be streamable and which should not. |
|
Since this API hasn't been completed yet I would like to point out that saving state would be useful. The web is transient in nature and would benefit from being able to save the state. For example the Python hashlib api did not provide saving the state which resulted in people reverting to C. |
This comment was marked as off-topic.
This comment was marked as off-topic.
A quick search revealed that there is indeed an active draft exploring this: https://webcrypto-streams.proposal.wintercg.org/#encryptionstream I do not support the WhatWG in any meaningful way but since you are rather rudely criticizing there lack of progress in their non-funded pursuit, I assume you do know more here. Would love to get some insights on why you think this is simpler to do than it looks! |
- parsing/creating PEM/pkix/pkcs1 files is now done by asn1.js - Streaming AES-CTR ciphers are now in @libp2p/crypto-aes-ctr - RSA encryption/decryption and PEM import/export are now in @libp2p/crypto-rsa WebCrypto [doesn't support streaming ciphers](w3c/webcrypto#73). We have a node-forge-backed shim that allows using streaming AES-CTR in browsers but we don't use it anywhere, so this has been split out into it's own module as `@libp2p/aes-ctr`. This was added to `@libp2p/crypto` to [support webrtc-stardust](libp2p/js-libp2p-crypto#125 (comment)) but that effort didn't go anywhere and we don't use these methods anywhere else in the stack. For reasons lost to the mists of time, we chose to require a padding algorithm that WebCrypto doesn't support so node-forge (or some other userland implemenation) will always be necessary in browsers, so these ops have been pull out into @libp2p/crypto-rsa which people can use if they need it. This is now done by manipulating the asn1 structures directly. The previous PEM import/export is also ported to `@libp2p/crypto-rsa` because it seems to handle more weird edge cases introduced by OpenSSL. These could be handled in `@libp2p/crypto` eventually but for now it at least supports round-tripping it's own PEM files. BREAKING CHANGE: Legacy RSA operations are now in @libp2p/crypto-rsa, streaming AES-CTR ciphers are in @libp2p/crypto-aes-ctr
TLDR: the bundle size has been reduced by ~50KB - parsing/creating PEM/pkix/pkcs1 files is now done by asn1.js - Streaming AES-CTR ciphers are now in @libp2p/crypto-aes-ctr - RSA encryption/decryption and PEM import/export are now in @libp2p/crypto-rsa WebCrypto [doesn't support streaming ciphers](w3c/webcrypto#73). We have a node-forge-backed shim that allows using streaming AES-CTR in browsers but we don't use it anywhere, so this has been split out into it's own module as `@libp2p/aes-ctr`. This was added to `@libp2p/crypto` to [support webrtc-stardust](libp2p/js-libp2p-crypto#125 (comment)) but that effort didn't go anywhere and we don't use these methods anywhere else in the stack. For reasons lost to the mists of time, we chose to require a padding algorithm that WebCrypto doesn't support so node-forge (or some other userland implemenation) will always be necessary in browsers, so these ops have been pull out into @libp2p/crypto-rsa which people can use if they need it. This is now done by manipulating the asn1 structures directly. The previous PEM import/export is also ported to `@libp2p/crypto-rsa` because it seems to handle more weird edge cases introduced by OpenSSL. These could be handled in `@libp2p/crypto` eventually but for now it at least supports round-tripping it's own PEM files. BREAKING CHANGE: Legacy RSA operations are now in @libp2p/crypto-rsa, streaming AES-CTR ciphers are in @libp2p/crypto-aes-ctr
TLDR: the bundle size has been reduced by ~50KB - parsing/creating PEM/pkix/pkcs1 files is now done by asn1.js - Streaming AES-CTR ciphers are now in @libp2p/crypto-aes-ctr - RSA encryption/decryption and PEM import/export are now in @libp2p/crypto-rsa WebCrypto [doesn't support streaming ciphers](w3c/webcrypto#73). We have a node-forge-backed shim that allows using streaming AES-CTR in browsers but we don't use it anywhere, so this has been split out into it's own module as `@libp2p/aes-ctr`. This was added to `@libp2p/crypto` to [support webrtc-stardust](libp2p/js-libp2p-crypto#125 (comment)) but that effort didn't go anywhere and we don't use these methods anywhere else in the stack. For reasons lost to the mists of time, we chose to require a padding algorithm that WebCrypto doesn't support so node-forge (or some other userland implemenation) will always be necessary in browsers, so these ops have been pull out into @libp2p/crypto-rsa which people can use if they need it. This is now done by manipulating the asn1 structures directly. The previous PEM import/export is also ported to `@libp2p/crypto-rsa` because it seems to handle more weird edge cases introduced by OpenSSL. These could be handled in `@libp2p/crypto` eventually but for now it at least supports round-tripping it's own PEM files. BREAKING CHANGE: Legacy RSA operations are now in @libp2p/crypto-rsa, streaming AES-CTR ciphers are in @libp2p/crypto-aes-ctr
TLDR: the bundle size has been reduced by ~50KB - parsing/creating PEM/pkix/pkcs1 files is now done by asn1.js - Streaming AES-CTR ciphers are now in @libp2p/crypto-aes-ctr - RSA encryption/decryption and PEM import/export are now in @libp2p/crypto-rsa WebCrypto [doesn't support streaming ciphers](w3c/webcrypto#73). We have a node-forge-backed shim that allows using streaming AES-CTR in browsers but we don't use it anywhere, so this has been split out into it's own module as `@libp2p/aes-ctr`. This was added to `@libp2p/crypto` to [support webrtc-stardust](libp2p/js-libp2p-crypto#125 (comment)) but that effort didn't go anywhere and we don't use these methods anywhere else in the stack. For reasons lost to the mists of time, we chose to require a padding algorithm that WebCrypto doesn't support so node-forge (or some other userland implemenation) will always be necessary in browsers, so these ops have been pull out into @libp2p/crypto-rsa which people can use if they need it. This is now done by manipulating the asn1 structures directly. The previous PEM import/export is also ported to `@libp2p/crypto-rsa` because it seems to handle more weird edge cases introduced by OpenSSL. These could be handled in `@libp2p/crypto` eventually but for now it at least supports round-tripping it's own PEM files. BREAKING CHANGE: Legacy RSA operations are now in @libp2p/crypto-rsa, streaming AES-CTR ciphers are in @libp2p/crypto-aes-ctr
TLDR: the bundle size has been reduced by about 1/3rd - parsing/creating PEM/pkix/pkcs1 files is now done by asn1.js - Streaming AES-CTR ciphers are now in [@libp2p/aes-ctr](https://github.com/libp2p/js-libp2p-aes-ctr) - RSA encryption/decryption and PEM import/export are now in [@libp2p/rsa](https://github.com/libp2p/js-libp2p-rsa) ## AES-CTR WebCrypto [doesn't support streaming ciphers](w3c/webcrypto#73). We have a node-forge-backed shim that allows using streaming AES-CTR in browsers but we don't use it anywhere, so this has been split out into it's own module as `@libp2p/aes-ctr`. ## RSA encrypt/decrypt This was added to `@libp2p/crypto` to [support webrtc-stardust](libp2p/js-libp2p-crypto#125 (comment)) but that effort didn't go anywhere and we don't use these methods anywhere else in the stack. For reasons lost to the mists of time, we chose to use a [padding algorithm](https://github.com/libp2p/js-libp2p-crypto/blob/3d0fd234deb73984ddf0f7c9959bbca92194926a/src/keys/rsa.ts#L59) that WebCrypto doesn't support so node-forge (or some other userland implemenation) will always be necessary in browsers, so these ops have been pulled out into `@libp2p/rsa` which people can use if they need it. This is now done by manipulating the asn1 structures directly. ## PEM/pkix/pkcs1 The previous PEM import/export is also ported to `@libp2p/crypto-rsa` because it seems to handle more weird edge cases introduced by OpenSSL. These could be handled in `@libp2p/crypto` eventually but for now it at least supports round-tripping it's own PEM files. Fixes #2086 BREAKING CHANGE: Legacy RSA operations are now in @libp2p/rsa, streaming AES-CTR ciphers are in @libp2p/aes-ctr
|
+1 |
|
@jasnell @tniessen wdyt of adding DigestStream to https://github.com/wintercg/proposal-webcrypto-streams ? |
|
There's an issue about moving the Web Crypto Streams draft to WICG here: WICG/proposals#185. Implementers: please voice your support there if you're in favor of this work 😊 |
|
This seems to go the route of automatically streaming every algorithm, which is not the way to do this. Cryptographic algorithms being streamable is the exception rather than the rule. In particular, modern symmetric cryptography uses AEADs, which are not streamable. There are some libraries which make them streamable, but this is a mistake. See https://www.imperialviolet.org/2015/05/16/aeads.html#:~:text=AEADs%20with%20large%20plaintexts |
|
@davidben To be fair, this is much simpler to spec and implement. And OpenSSL and its derivatives provide exactly this same kind of API, so browsers might not even be managing this buffer directly. |
|
@davidben The issue brought up there mainly affects decryption, not encryption. Though, you might (reasonably) argue that if we shouldn't have streaming decryption, we shouldn't have streaming encryption either, but there are plausible scenarios where it might still be useful, e.g. if the encrypting client might be memory-constrained but the decrypting client isn't (think SecureDrop or so). I would be cautious to specify streaming encryption and/or decryption of only the non-AEAD modes, as it might incentivize using those, while we probably want to incentivize using AEAD. So if we don't want to have streaming encryption for AEAD, perhaps we shouldn't have streaming encryption at all, and just recommend chunking the plaintext (although this of course won't be possible for legacy protocols). So, I think that starting with streaming hashing, for example as proposed in #73 (comment), might be better (and simpler). However, having a draft in the WICG where we can work on that & iterate on the API still wouldn't hurt, IMHO 😊 |
|
@gobengo ... sorry I just spotted your question here:
Definitely +1 |
Yep, this has come up in conversations a few times and I would consider the initial explainer / draft that we put together in wintercg as just an initial starting point. I think there's a TON of refinement and clarification that is needed. We wouldn't intend for "automatically streaming every algorithm" to be the actual approach we take. |
|
Given the position that OpenSSL has taken, It doesn't seem like it's your job to protect us from ourselves or dictate system design. |
|
Well - with no offense to the OpenSSL team, I don't think we should take their API as the end-all and be-all of crypto APIs. AEAD modes are typically defined as single-shot functions, that don't allow passing a stream. Deviating from that would need a stronger justification than "OpenSSL does it", IMHO. |
|
Indeed many libraries do not expose streaming AEADs, including some of those used by browsers. OpenSSL's streaming API is an artifact of them trying to mix multiple, unrelated algorithms into the same API. (A similar mistake that has been made here.) Any proposal with a streaming AEAD is a non-starter here. AEADs are simply not streaming primitives. |
|
I don't mean to stir trouble, but you didn't address my point. It was not "OpenSSL does it." It was "it's not your decision to make." |
|
How much more work would it create to split hashing from encryption here? It seems that the latter is contentious but the former is not. |
|
Also, due to the current state of cryptography algorithms and their usage, stream input to hashes is a bit more critical.
|
|
Unfortunately, the authors and core proponents of this spec couldn't care less about hashing use cases outside of crypto. We've hashed over this (heh) on previous threads while trying to get something as simple as SHA-1 on "insecure" origins:
The key comment from @twiss:
Therefore, I doubt we'll get built-in hashing of streams without the associated crypto side... but who knows, maybe someone will prove me wrong. We're stuck with DIY in the meantime. At least it's possible. |
|
@bradisbell Bringing in unrelated issues isn't really relevant, and also not really helpful while we're actively discussing the addition of streaming to Web Crypto. @dead-claudia and @jpsugar-flow I tend to agree. Taking a step back, and perhaps contrary to what I said before in this thread; if we want to focus on adding streaming hashing first since it's least controversial, and if everyone is happy with the proposal in #73 (comment) (i.e. extending the existing function rather than adding a new If folks are OK with that, I can write something up. |
@jakearchibald's proposal looks good to me. Workers will need to keep I imagine that |
|
Oh, I missed that proposal. Very elegant. +1 from me. (I'm jpsugar-flow, but this is not work-related for me.) |
and others
Bug 27755:
Though the StreamsAPI is referenced in Informative Reference, the functions under window.crypto.subtle are specified with only one-shot data inputs.
Use-cases: Data may not be available at once. Data may be too huge to keep in memory.
For encrypt()/decrypt() it would make sense to have a streaming readable output if the input is a readable stream.
The text was updated successfully, but these errors were encountered: