Refactor Key Management in @web5/crypto

[frankhinek]

Design Guidelines

• The key manager interface must be passed as an argument to all public API methods that require key material. e.g.
  • DID Creation
  • Data Signing
• MVP should include an in-memory Key Manager implementation.
• Consumers of our SDKs should be able to provide their own KeyManager implementations if desired.

Feature Proposal Tasks:

• Propose API surface design prior to implementation. This can be done by creating a draft PR for a respective feature and providing prototypal code or proposing the design in the comments
• Approval Required from a minimum of 2 people

Implementation Tasks:

• Produce/update API reference documentation for any/all methods added to the public API surface. documenting private API methods is optional
• Produce/update relevant example usage
• Test coverage must be provided for all public API methods.
• Tests that consume shared test vectors must all be passing

[frankhinek]

Proposal

General Design Choices

• Use JSON web key (JWK) as the key format.
  • Provide conversion utilities to:
    • convert between JWK and Web Crypto API's CryptoKey (used by web browsers)
    • Encode raw Uint8Array keys to JWK and decode JWKs to raw Uint8Array keys
• Use IANA JSON Object Signing and Encryption (JOSE) naming conventions for algorithm, curves, etc. whenever they exist.
  • If algorithms used are not currently registered, TBD should either signal support for an existing draft, if one exists, or propose a new algorithm name if no relevant proposal is found.
  • Examples:
    • algorithm: { name: 'ES256K' }: ECDSA (Elliptic Curve Digital Signature Algorithm) using the secp256k1 curve and SHA-256 hashing
    • algorithm: { name: 'EdDSA', curve: 'Ed25519' }: EdDSA (Edwards-curve Digital Signature Algorithm) using the Ed25519 curve
    • algorithm: { name: 'A256CTR' }: AES (Advanced Encryption Standard) in CTR (Counter) mode using a 256-bit key
    • algorithm: { name: 'HS512' }: HMAC (Hash-Based Message Authentication Code) using the SHA-512 hashing
• All I/O that interacts with private or secret keys is done via reference using a KeyIdentifier type, which is a string value. Concrete implementations can use any string as the key identifier (e.g. JWK thumbprint, UUID generated by hosted KMS, etc.).
• Provide a base crypto interface definition that concrete implementations conform to.
• Any implementation of the crypto interface can be passed as an argument to all public API methods that involve key material (e.g., DID creation, VC signing, general data signing/verification, etc.).

Base Interface

Define a CryptoApi interface that all concrete implementations implement/extend:

export interface CryptoApi {

  // Key Generation Operations
  generateKey(options: GenerateKeyOptions): Promise<KeyIdentifier>;

  // Key Derivation Operations
  deriveBits(options: DeriveBitsOptions): Promise<Uint8Array>;
  deriveKey(options: DeriveKeyOptions): Promise<KeyIdentifier>;

  // Key Wrapping Operations
  wrapKey(options: WrapKeyOptions): Promise<Uint8Array>;
  unwrapKey(options: UnwrapKeyOptions): Promise<JsonWebKey>;

  // Signature Operations
  sign(options: SignOptions): Promise<Uint8Array>;
  verify(options: VerifyOptions): Promise<boolean>;

  // Hashing Operations
  digest(options: DigestOptions): Promise<Uint8Array>;

  // Cipher Operations
  encrypt(options: EncryptOptions): Promise<Uint8Array>;
  decrypt(options: DecryptOptions): Promise<Uint8Array>;

  // Import/Export Operations
  importKey(options: ImportKeyOptions): Promise<KeyIdentifier>;
  exportKey(options: ExportKeyOptions): Promise<JsonWebKey>;
}

Concrete Implementations

At a minimum, develop and publish two implementations of the CryptoApi interface:

• InMemoryCrypto:

  • isomorphic implementation (Node.js, web browser, and React Native)
  • prefers the Web Crypto API cryptographic algorithm implementations but substitutes @noble or node:crypto libs if Web Crypto lacks the required functionality
  • uses an in-memory key store (Map)

• AwsKmsCrypto:

  • implementation for back-end development in a Node.js runtime environment
  • limited to the algorithms made available by existing AWS services (initially only AWS KMS).
  • all private / secret key material is secured by AWS infrastructure

Consumers of our SDKs can develop their own CryptoApi implementations, if desired. Concrete examples include an implementation for Google Cloud KMS, one that persists un-extractable keys in a web browser, or one that offers both in-memory and local key stores depending on whether keys are ephemeral or long-lived.

Usage Examples

Initialization

In-memory:

import { InMemoryCrypto } from `@web5/crypto`;

const crypto = new InMemoryCrypto();

AWS KMS:

import { AwsKmsCrypto } from `@web5/crypto`;
import { KMSClient } from '@aws-sdk/client-kms';

const kmsClient = new KMSClient({ region: 'us-east-1' })
const crypto = new AwsKmsCrypto({ kmsClient });

Key Generation

const dataEncKeyId = await crypto.generateKey({ algorithm: { name: 'A256CTR' }});
console.log(dataEncKeyId) // vDNSXebfrWOE2fzXhJXbRH28tZhuv2FEd9XYow-SpRA

const credSignKeyId = await crypto.generateKey({ algorithm: { name: 'ES256K' }});
console.log(credSignKeyId) // riJYyUykKbIU9hgO27p-smjDncj082aRcmknKS7XQ-w

Signing

const data = new Uint8Array(32);
const signature = await crypto.sign({ algorithm: { name: 'ES256K' }, data, keyId: credSignKeyId });

Verification

const isValid = await crypto.verify({ algorithm: { name: 'ES256K' }, data, keyId: credSignKeyId, signature });

Encryption

const plaintext = new Uint8Array(1024);
const ciphertext = await crypto.encrypt({ algorithm: { name: 'A256CTR', data: plaintext, keyId: dataEncKeyId });

Decryption

const plaintextOutput = await crypto.decrypt({ algorithm: { name: 'A256CTR', data: ciphertext, keyId: dataEncKeyId });

DID Creation

By passing a concrete CryptoApi implementation to the DID method, private keys are generated and stored in the underlying key store (e.g., in-memory, AWS HSM, etc.). For some DID methods like did:dht and did:jwk a single key is generated, whereas others like did:ion generate three or four keys.

import { DidDhtMethod, DidIonMethod, DidJwkMethod, DidKeyMethod } from '@web5/dids';

const didA = await DidDhtMethod.create({ crypto });
const didB = await DidIonMethod.create({ crypto });
const didB = await DidJwkMethod.create({ crypto });
const didC = await DidKeyMethod.create({ crypto });

Appendix

General Type Definitions

export interface AlgorithmIdentifier {
  name: string;
}

export type KeyIdentifier = string;

CryptoApi Operation Type Definitions

export interface DecryptOptions {
  algorithm: AlgorithmIdentifier;
  keyId: KeyIdentifier;
  data: Uint8Array;
}

export interface DeriveBitsOptions {
  algorithm: AlgorithmIdentifier;
  baseKeyId: KeyIdentifier;
  length: number;
}

export interface DeriveKeyOptions {
  algorithm: AlgorithmIdentifier;
  baseKeyId: KeyIdentifier;
  derivedKeyType: AlgorithmIdentifier;
}

export interface DigestOptions {
  algorithm: AlgorithmIdentifier;
  data: Uint8Array;
}

export interface EncryptOptions {
  algorithm: AlgorithmIdentifier;
  keyId: KeyIdentifier;
  data: Uint8Array;
}

export interface ExportKeyOptions {
  keyId: KeyIdentifier;
}

export interface GenerateKeyOptions {
  algorithm: AlgorithmIdentifier;
}

export interface ImportKeyOptions {
  algorithm: AlgorithmIdentifier;
  key: JsonWebKey;
}

export interface SignOptions {
  algorithm: AlgorithmIdentifier;
  keyId: KeyIdentifier;
  data: Uint8Array;
}

export interface VerifyOptions {
  algorithm: AlgorithmIdentifier;
  keyId: KeyIdentifier;
  signature: Uint8Array;
  data: Uint8Array;
}

export interface WrapKeyOptions {
  key: JsonWebKey;
  wrappingKeyId: KeyIdentifier;
  wrapAlgorithm: AlgorithmIdentifier;
}

export interface UnwrapKeyOptions {
  wrappedKey: Uint8Array;
  unwrappingKeyId: KeyIdentifier;
  unwrapAlgorithm: AlgorithmIdentifier;
}

Crypto Algorithm Type Definitions

These are examples and it is not intended to be an exhaustive list. Concrete implementations will likely override these definitions and include only those algorithms that are useful for a given use case or supported by a particular HSM (e.g., AWS KMS only supports certain algorithms).

export interface AesAlgorithm extends AlgorithmIdentifier {
  name: 'A128CBC' | 'A192CBC' | 'A256CBC' | 'A128CTR' | 'A192CTR' | 'A256CTR' | 'A128GCM' | 'A192GCM' | 'A256GCM';
}

export interface ChaChaAlgorithm extends AlgorithmIdentifier {
  name: 'XC20' | 'XC20P';
}

export interface EcdsaAlgorithm extends AlgorithmIdentifier {
  name: 'ES256' | 'ES256K';
 }

export interface EddsaAlgorithm extends AlgorithmIdentifier {
  name: 'EdDSA';
  curve: 'Ed25519' | 'Ed448';
 }

export interface EcdhAlgorithm extends AlgorithmIdentifier {
  name: 'ECDH' | 'ECDH-ES';
  curve: 'secp256k1' | 'X25519';
 }

AES Type Definitions

export interface AesCbcOptions extends AlgorithmIdentifier {
  name: 'A128CBC' | 'A192CBC' | 'A256CBC';
  iv: Uint8Array;
}

export interface AesCtrOptions extends AlgorithmIdentifier {
  name: 'A128CTR' | 'A192CTR' | 'A256CTR';
  counter: Uint8Array;
}

export interface AesGcmOptions extends AlgorithmIdentifier {
  name: 'A128GCM' | 'A192GCM' | 'A256GCM';
  additionalData?: Uint8Array;
  iv: Uint8Array;
  tagLength?: number;
}

ChaCha Type Definitions

export interface XChaCha20Options extends AlgorithmIdentifier {
  name: 'XC20';
  nonce: Uint8Array;
}

export interface XChaCha20Poly1305Options extends AlgorithmIdentifier {
  name: 'XC20P';
  nonce: Uint8Array;
  additionalData?: Uint8Array;
}

HKDF Type Definitions

export interface HkdfOptions extends AlgorithmIdentifier {
  hash: string;
  info: Uint8Array;
  salt: Uint8Array;
}

[andresuribe87]

Overall awesome stuff!

Major pieces of feedback/questions:

First, related to

Encode raw Uint8Array keys to JWK and decode JWKs to raw Uint8Array keys

What's the motivation for supporting raw keys vs only using JWKs?

Related to encryption. I think it's important to make key rotation first class. Cloud managers typically provide some form of automatically doing this, but it really depends on the use case. How do you envision key rotation being facilitated by the CryptoAPI?

Related to naming, I'm curious why CryptoAPI? Did you consider smaller and more focused interfaces?

Lastly, should there be special provisions in order to handle large streams of data (or large files) that need to be signed or encrypted?

[tomdaffurn]

Does digest do just hashing, or HMAC? Will there be AlgorithmIdentifiers for hashing functions? Those don't need keys or cloud KMS, so it stands out amongst the other key using operations

[tomdaffurn]

web5-kt had separate Crypto, KeyManager, and algorithm classes (e.g. Secp256k1). Seems like CryptoApi combines all 3 into one? I like the simplicity of it, but I think there'll be a lot of duplicate code in CryptoApi implementations. Will there be any effort to extract JWK/JOSE, algorithms, or validation to common code?

[tomdaffurn]

Having one AlgorithmIdentifier type to cover all the different and incompatible algs feels like we're dooming implementers to have validation at the top of every function. At least KDF could be separate from the encrypt/sign algs

[frankhinek]

Great questions and feedback @andresuribe87 and @tomdaffurn.

First, related to

Encode raw Uint8Array keys to JWK and decode JWKs to raw Uint8Array keys

What's the motivation for supporting raw keys vs only using JWKs?

The proposal is to only use the JWK key format in the CryptoApi methods. The suggested "conversion utilities" would make it easy to convert other key formats (e.g., CryptoKey, Uint8Array, etc.) to/from JWK. These will likely be needed internally when the underlying KMS or cryptographic algorithm implementations use another format. For example, an implementation that uses AWS KMS will have to convert ASN.1 DER keys to JWK, like web5-kt does in AwsKeyManager. Another example would be an in-memory or local key store implementation that uses the Web Crypto API for signing, verification, encryption, etc. will have to convert CryptoKey to JWK.

Related to encryption. I think it's important to make key rotation first class. Cloud managers typically provide some form of automatically doing this, but it really depends on the use case. How do you envision key rotation being facilitated by the CryptoAPI?

Good question and a topic that deserves more consideration. How do we balance the simplicity and consistency of an API that provides a set of common crypto operations with more fine-grained configuration and advanced functionality that is only available in some scenarios (e.g., HSM backed)? What should be considered a "common" operation? Will consumers of our SDKs use them to manage key rotation or will they implement these policies directly in the underlying KMS?

I used the Web Crypto API as inspiration, but it doesn't account for features like key rotation. Additionally, if something like JWK thumbprint is used as a key identifier, how would that work post-key rotation? We should consider this and other questions to arrive at a consensus.

Related to naming, I'm curious why CryptoAPI? Did you consider smaller and more focused interfaces?

This design choice was inspired by the Web Crypto API since it is supported by 98%+ of desktop and mobile browser installations (billions of devices). Yes, other options were considered and I think both @mistermoe and I have prototyped half a dozen versions over the past few months. In general, the approach was to provide a single API that exposed a common set of cryptographic algorithms including digital signature, cipher, hashing, and key derivation.

A few mentions of other options considered:

• Multiple implementations that are operation specific (KeyGenerator, Signer, Cipher, Hasher, etc.) which would be used depending on the need. For example, a use case that involves JWS might only need the Signer for creating and then later verifying a signature whereas JWE might need both a Signer and Cipher. Does this improve the developer experience or is easier/intuitive to use a single API to access all of the common crypto operations?
• A single API that implements KeyGenerator, Signer, Cipher, Hasher, etc. interfaces such that the API could be used in a unified way or a developer could pull in one of the smaller modules for a specific need. This would be straightforward to implement in TypeScript but not sure if its possible in Kotlin.
• Use a dot-delimited approach (crypto.aes.encrypt(), crypto.secp256k1.sign(), crypto.hmac.sign()). You still end up having to pass properties to these methods (curves, length, initialization vector, nonce, salt, etc.), so it isn't clear that this buys you much when IDE code-completion (e.g., IntelliSense) informs developers what algorithm and parameters are supported by a particular implementation.

Lastly, should there be special provisions in order to handle large streams of data (or large files) that need to be signed or encrypted?

That's an insightful question and one I've been thinking about a lot lately. The current Web Crypto API supports only one-shot encryption and hashing. Good read on the subject here and here. A draft community report still under consideration by W3C and browser vendors here.

Does digest do just hashing, or HMAC? Will there be AlgorithmIdentifiers for hashing functions? Those don't need keys or cloud KMS, so it stands out amongst the other key using operations

The digest() method is focused purely on hashing. For anyone reading along that is less familiar with the subject, a digest is a fixed-size output generated by a hash function from input data. Ideally, a digest is quick to calculate, irreversible, and unpredictable, and therefore indicates whether someone has tampered with a given message.

HMAC is primarily considered a signing algorithm, and is therefore, accessed through the sign() operation. For anyone that isn't familiar, HMAC combines a hash function with a secret key, providing both data integrity and authentication of the message/data. HMAC isn't just a digest algorithm because it verifies that a message/data hasn't been altered and confirms its authenticity using the shared secret key.

web5-kt had separate Crypto, KeyManager, and algorithm classes (e.g. Secp256k1). Seems like CryptoApi combines all 3 into one? I like the simplicity of it, but I think there'll be a lot of duplicate code in CryptoApi implementations. Will there be any effort to extract JWK/JOSE, algorithms, or validation to common code?

Yes, that seems like a good approach and is the direction web5-js is already headed down -- but more work to be done and this code will be cleaned up quite a bit once the switch is made to use JWK throughout instead of CryptoKey:

• JOSE functionality
• Crypto algorithms that can be used by in-memory, local key store, etc. implementations.

The core methods of the prototype I created for InMemoryCrypto using this approach is only about 125 lines (w/o comments) and mostly calls the underlying crypto algorithm implementations.

In contrast, an AWS or GCP KMS implementation would involve constructing and executing commands to the KMS client and handling any format/encoding conversion.

Having one AlgorithmIdentifier type to cover all the different and incompatible algs feels like we're dooming implementers to have validation at the top of every function. At least KDF could be separate from the encrypt/sign algs

Apologies for the confusion. I should have explained that better. That's only present in the base type definitions. Concrete implementations would override these with the algorithms that they support. For example, an AWS implementation might initially only support signatures with ES256 and ES256K whereas an in-memory implementation might include ES256K and EdDSA/Ed25519.

An example would be the base GenerateKeyOptions and SignOptions types:

export namespace CryptoApi {
  export interface GenerateKeyOptions {
    algorithm: AlgorithmIdentifier;
  }

  export interface SignOptions {
    algorithm: AlgorithmIdentifier;
    keyId: KeyIdentifier;
    data: Uint8Array;
  }
}

And the specific implementation choices made for an InMemoryCrypto implementation which supports generating AES, ChaCha, ECDSA/EdDSA, HMAC, and PBKDF2 keys and signing with ECDSA, EdDSA, and HMAC.

namespace InMemoryCrypto {
  export interface GenerateKeyOptions extends CryptoApi.GenerateKeyOptions {
    algorithm:
      | CryptoApi.AesOptions
      | CryptoApi.ChaChaOptions
      | CryptoApi.EcOptions
      | CryptoApi.HmacOptions
      | CryptoApi.Pbkdf2Options;
  }

  export interface SignOptions extends CryptoApi.SignOptions {
    algorithm:
      | CryptoApi.EcdsaOptions
      | CryptoApi.EdDsaOptions
      | CryptoApi.HmacOptions;
  }
}

Taking this approach enables IDE/IntelliSense hints to automatically list only those algorithms that are available:

[andresuribe87]

The suggested "conversion utilities" would make it easy to convert other key formats (e.g., CryptoKey, Uint8Array, etc.) to/from JWK.

Is this something that needs to be exposed as part of the public API? It seems like it's a lower-level implementation detail. In general, I'm a big fan of making public APIs as slim as possible. Smaller public APIs means a smaller surface that needs to be maintained, implemented, and less choice for developers (which translates into less time learning how to use the API). Going from private -> public is much easier than going from public -> private, in case it's something that we really want to expose to the public.

A single API that implements KeyGenerator, Signer, Cipher, Hasher, etc. interfaces such that the API could be used in a unified way or a developer could pull in one of the smaller modules for a specific need. This would be straightforward to implement in TypeScript but not sure if its possible in Kotlin.

Just noting that my strong preference is to have smaller interfaces like the ones mentioned above. It's possible in most languages that I'm familiar with, including Kotlin, golang and C++. I would also slims down the dependencies that are needed, as devs can pull in only what they need, instead of the whole bundled API. Another benefit is that this conversation can be separated into smaller pieces. For instance, I think we should include encryption until we have a rock solid story of how we're going to support key-rotation (I believe it's crucial that we make sure our design promotes the correct way of doing encryption, which include key-rotation).

[andresuribe87]

@frankhinek a couple of additional use cases that I think are important to address.

There is often times where applications need to know whether the private key associated with a public key is available in the instance that implements CryptoAPI. For such cases, it would be important to add a method that can address that. One possible solution is to include containsKey(keyAlias) as part of the API. This assumes that users have a way of going from public key -> key alias. So you would need a way to either:

1. tell CryptoAPI which keyAlias to use when generating or
2. have another method in CryptoAPI that does public key -> key alias.

(2) is what we've implemented in kotlin as shown in https://github.com/TBD54566975/web5-kt/blob/bbef724240451e143e5ba72d829d25c371e440dd/crypto/src/main/kotlin/web5/sdk/crypto/KeyManager.kt#L64

But I actually think we should implement (1). Let me elaborate why below.

A second, more advanced, use case that's specific to DidIon (and possibly DidWeb). When a user boots up an application for the n-th time, they will likely want to know whether their instance of CryptoApi contains all the keys necessary to sign and manage a given did. Let's focus on ION for a second. This means that the application will need to know whether the latest recovery and latest update keys are present. The problem here arises when I do updates. An update to the DID document changes the update key. But, anchoring the update might fail. This introduces race conditions that aren't fun to deal with. From the perspective of a user of the CryptoAPI, I believe that this can be fixed if we can pass in arbitrary keyAliases to use (option (1) above), and additionally having a method to rename a keyAlias atomically (or have the ability to have multiple aliases for a single key).

An alternative is to expose transaction functionality with commits and rollbacks, but that sounds very nasty.

[mistermoe]

@frankhinek many props for always taking the time to thoroughly explain your proposals and the rationale behind decsions.

Re:

ah ok so, if AwsKmsCrypto wanted its verify to work the same way as InMemoryCrypto.verify works, since verifying doesn't necessitate calls to a KMS, i suppose we could just create an instance of InMemoryCrypto inside AwsKmsCrypto and call that instance's verify function?

[frankhinek]

ah ok so, if AwsKmsCrypto wanted its verify to work the same way as InMemoryCrypto.verify works, since verifying doesn't necessitate calls to a KMS, i suppose we could just create an instance of InMemoryCrypto inside AwsKmsCrypto and call that instance's verify function?

Yes, that's certainly an approach that could be taken. Ideally implementers have enough of the basic structure and reference implementations in place that they can extend or modify to meet specific use case requirements.

[decentralgabe]

A few thoughts:

• byte encoding is not specific enough. for ecdsa keys there are compressed/uncompressed serialization formats, I know RSA has a few different types too. we should provide encoding format recommendations for each key type (or a default practice)
• verification needs to add audience as an optional field to make use of the aud claim JWTs have
• I would recommend explicitly listing the keys/algorithms we support, and provide guidance for -- you can see such a list for the ssi sdk here
• what do you think about advocating for a specific library (or set of libs) that can be used like tink or libsodium?