Vulnerability type
Uncontrolled Resource Consumption
Attack type
Remote
Impact
Denial-of-service, Resource consumption (memory)
Discoverer(s)/Credits
An OOM issue was initially reported by Bartosz Borkowski (#9891). After investigating the reported issue, Matt Klein discovered the (assumed) root cause.
Description
Envoy employs HTTP/2 stream flow control as configured by the initial_stream_window_size setting that can be configured both on downstream HTTP listeners as well as upstream HTTP clusters. Envoy also limits the number of simultaneous downstream streams as configured by the max_concurrent_streams setting.
Envoy can consume and never release stream memory in the following case:
Untrusted client opens streams up to the max_concurrent_streams limit.
Untrusted client requests a resource that will generate a large response.
Depending on the configuration of initial_stream_window_size for the upstream cluster, Envoy will buffer approximately that amount of memory on behalf of the stream.
If the downstream client does not open receive window, Envoy will start flow control and stop allowing data from upstream.
However, there existed a case in which Envoy would receive the final data of the stream from upstream (pending window), but not have enough open window to send the data to downstream. This data would be buffered forever and not covered by any timeout.
The effectiveness of this attack is limited by both the max_concurrent_streams and initial_stream_window_size settings. However, a malicious client with knowledge of backend configuration may be able to craft a sequence of connections and streams that would cause Envoy to use substantial memory and never free it. (By opening up max_concurrent_streams - 1 streams per connection and using the final stream for normal periodic requests, the connection idle timer would never fire.)
The fix for this CVE is that Envoy will now use the stream_idle_timeout to cover the final data flush in which data may be buffered pending available downstream window. If downstream window is never opened, Envoy will reset the stream and release the buffered data once the idle timeout fires.
Vulnerability type
Uncontrolled Resource Consumption
Attack type
Remote
Impact
Denial-of-service, Resource consumption (memory)
Discoverer(s)/Credits
An OOM issue was initially reported by Bartosz Borkowski (#9891). After investigating the reported issue, Matt Klein discovered the (assumed) root cause.
Description
Envoy employs HTTP/2 stream flow control as configured by the
initial_stream_window_sizesetting that can be configured both on downstream HTTP listeners as well as upstream HTTP clusters. Envoy also limits the number of simultaneous downstream streams as configured by themax_concurrent_streamssetting.Envoy can consume and never release stream memory in the following case:
Untrusted client opens streams up to the
max_concurrent_streamslimit.Untrusted client requests a resource that will generate a large response.
Depending on the configuration of
initial_stream_window_sizefor the upstream cluster, Envoy will buffer approximately that amount of memory on behalf of the stream.If the downstream client does not open receive window, Envoy will start flow control and stop allowing data from upstream.
However, there existed a case in which Envoy would receive the final data of the stream from upstream (pending window), but not have enough open window to send the data to downstream. This data would be buffered forever and not covered by any timeout.
The effectiveness of this attack is limited by both the
max_concurrent_streamsandinitial_stream_window_sizesettings. However, a malicious client with knowledge of backend configuration may be able to craft a sequence of connections and streams that would cause Envoy to use substantial memory and never free it. (By opening upmax_concurrent_streams- 1 streams per connection and using the final stream for normal periodic requests, the connection idle timer would never fire.)The fix for this CVE is that Envoy will now use the
stream_idle_timeoutto cover the final data flush in which data may be buffered pending available downstream window. If downstream window is never opened, Envoy will reset the stream and release the buffered data once the idle timeout fires.