diff --git a/G1-true-binary-metadata.md b/G1-true-binary-metadata.md new file mode 100644 index 000000000..7479232df --- /dev/null +++ b/G1-true-binary-metadata.md @@ -0,0 +1,89 @@ +True Binary Metadata +---- +* Author(s): ctiller +* Approver: ejona +* Status: Draft +* Implemented in: n/a +* Last updated: 3/29/17 +* Discussion at: https://groups.google.com/forum/#!topic/grpc-io/Ww8waYz_Nes + +## Abstract + +Add a HTTP2 extension to allow binary metadata (those suffixed with -bin) to be +sent without the base64 encode/decode step. + +## Background + +gRPC allows binary metadata to be sent by applications. These metadata elements +are keyed with a -bin suffix. When transmitted on the wire, since HTTP2 does not +allow binary headers, we base64 encode these elements and Huffman compress them. +On receipt, we transparently reverse the transformation. + +This transformation is costly in terms of CPU, and there exist use cases where +this transformation can become the CPU bottleneck for gRPC. + +### Related Proposals: + +n/a + +## Proposal + +### New setting + +Expose a custom setting in our HTTP2 settings exchange: +GRPC_ALLOW_TRUE_BINARY_METADATA = 0xfe03. + +This setting is randomly chosen (to avoid conflicts with other extensions), and +within the experimental range of HTTP extensions (see https://tools.ietf.org/html/rfc7540#section-11.3). + +The setting can have the values 0 (default) or 1. If the setting is 1, then +peers MAY use a 'true binary' encoding (described below), instead of the current +base64 encoding for -bin metadata. + +Implementations SHOULD transmit this setting only once, and as part of the first +settings frame. + +### 'True binary' encoding + +When transmitting metadata on a connection where the peer has specified +GRPC_ALLOW_TRUE_BINARY_METADATA, instead of encoding using base64, an +implementation MAY instead prefix a NUL byte to the metadata and transmit the +metadata value in binary form. + +Since this is a HTTP2 extension and other extensions might alias this extension +id, it's possible that this becomes misconfigured. In that case, peers are +required to RST_STREAM with http error PROTOCOL_ERROR (as required by https://tools.ietf.org/html/rfc7540#section-10.3). +If a binary encoding was attempted and such a RST_STREAM is received without any other headers, +implementations SHOULD retry the request with base64 encoding, and disable +binary encoding for future requests. Verbosely logging this condition is +encouraged. + +### Examples + +Suppose we wanted to send metadata element 'foo-bin: 0x01' (ie a single byte +containing '1'). + +Under base64, we'd send a http header 'foo-bin: AQ'. +Under binary, we'd send 'foo-bin: 0x00 0x01' (ie prefixing a NUL byte and then +sending the binary metadata value.) + +## Rationale + +Binary metadata transmission performance is critical for a number of +applications. + +Various workarounds were considered: +1. Switching to base16 - this would require a backwards incompatible protocol + change for gRPC, and has the disadvantage of bloating wire size, which + additionally interacts badly with hpack. +2. Adding a new suffix (-raw): this leaks further implementation details to + application developers, and likely would still need a base64 workaround + +## Implementation + +A trial implementation in gRPC C core is underway and expected to be ready in +coming weeks. + +## Open issues (if applicable) + +n/a