http json_to_metadata filter

[JuniorHsu]

Title: support parse json value and set to metadata

Description:
We propose the addition of a new filter, the json_to_metadata filter, to Envoy, inspired by the thrift payload_to_metadata filter. This new filter addresses a use case where rate limiting is required based on a predefined set of key-value pairs in the request body. Here are the key features and considerations for the filter:

1. Selective Rule Support: Each rule can be configured to apply to request only, response only, or both, depending on the requirements.
2. Blocking Data Flow: The filter can block the data flow to subsequent filters until all desired key-value pairs are obtained. This feature is especially useful when metadata plays a crucial role during the decodeHeaders stage, facilitating tasks such as subset load balancing and rate limiting. However, while request/response size is large, memory is something to worry about.
3. Streamlined JSON Parsing: Although Envoy currently supports only whole message decoding, the json_to_metadata filter aims to support streaming JSON decoding in the future. This enhancement would enable efficient processing of large JSON payloads.
4. Conditional Parsing: The filter can be configured to parse JSON only when a specific HTTP header is present or when the header's value conforms to a defined regular expression. This selective parsing capability allows for granular control over when to perform JSON parsing and optimizing performance.
5. Initial Support for Simple Values: In its initial release, the json_to_metadata filter will focus on supporting simple values(non object/non array), ensuring a straightforward implementation and ease of use.

Here's an example json body:

{
  "foo": "bar",
  "baz": {
  {
    "key": "value",
    "key2", "value2"
   },
   "qux": ["a", "b", "c"]
}

Support key list to parse:

["foo"]
["baz", "key"]

Unsupported key list to parse, which triggers the on_missing clause

["unknown"]
["baz"]
["qux"]
["bar"]
["foo", "bar"]

[wbpcode]

Hi, thanks for your suggesion. Please check the https://github.com/envoyproxy/envoy/blob/main/CONTRIBUTING.md for basic requirement for new extension adding.

[JuniorHsu]

We have implemented one and worked in a private build. Here's the proposed proto, and we're looking for a maintainer as the code owner of new component. Thanks!

// [#next-free-field: 5]
message JsonToMetadata {
  enum ValueType {
    // The value is a serialized `protobuf.Value
    // <https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/struct.proto#L62>`_.
    PROTOBUF_VALUE = 0;
    STRING = 1;
    NUMBER = 2;
  }

  // [#next-free-field: 6]
  message KeyValuePair {
    // The namespace — if this is empty, the filter's namespace will be used.
    string metadata_namespace = 1;

    // The key to use within the namespace.
    string key = 2 [(validate.rules).string = {min_len: 1}];

    oneof value_type {
      // The value to pair with the given key.
      //
      // When used for on_present case, if value is non-empty it'll be used instead
      // of the attribute value. If both are empty, the attribute value is used as-is.
      //
      // When used for on_missing/on_error case, a non-empty value
      // must be provided.
      //
      // It ignores ValueType, i.e., not type casting.
      google.protobuf.Value value = 3;
    }

    // The value's type — defaults to protobuf.Value.
    ValueType type = 4 [(validate.rules).enum = {defined_only: true}];

    // False if we want to overwrite the existing metadata value. Default to false.
    bool preserve_existing_metadata_value = 5;
  }

  message Selector {
    oneof selector {
      // key to match
      string key = 1 [(validate.rules).string = {min_len: 1}];
    }
  }

  // A Rule defines what metadata to apply when a key-value is present, missing in the json
  // or fail to parse the payload.
  // [#next-free-field: 5]
  message Rule {
    // Specifies that a match will be performed on the value of a property.
    // Here's an example to match on 1 in {"foo": {"bar": 1}, "bar": 2}
    //
    // selectors:
    // - key: foo
    // - key: bar
    repeated Selector selectors = 1 [(validate.rules).repeated = {min_items: 1}];

    // If the attribute is present, apply this metadata KeyValuePair.
    KeyValuePair on_present = 2;

    // If the attribute is missing, apply this metadata KeyValuePair.
    //
    // The value in the KeyValuePair must be set.
    KeyValuePair on_missing = 3;

    // If the body is too large or fail to parse, apply this metadata KeyValuePair.
    //
    // The value in the KeyValuePair must be set.
    KeyValuePair on_error = 4;
  }

  // the list of rules to apply to requests
  repeated Rule request_rules = 1 [(validate.rules).repeated = {min_items: 1}];

  // Soft limit on size of the read buffers.
  // If the body larger than the limit, simply do not parse the body and apply on_error policy.
  //
  // The limit is bounds by
  // ref:`per_connection_buffer_limit_bytes <envoy_v3_api_field_config.listener.v3.Listener.per_connection_buffer_limit_bytes>`
  // That is, if the body buffer size is larger than the connection limit,
  // the listener manager would reply 413 for requests and 500 for responses before
  // filters get chance to manipulate.
  google.protobuf.UInt32Value request_buffer_limit_bytes = 2;

  // allowed content-type for json to metadata transformation
  // default to {"application/json"}
  //
  // Set `request_allow_empty_content_type` if empty/missing content-type header
  // is allowed.
  repeated string request_allow_content_types = 3
    [(validate.rules).repeated = {items {string {min_len: 1}}}];

  // allowed empty content-type for json to metadata transformation
  // default to false
  bool request_allow_empty_content_type = 4;
}

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.