Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

connection: Remember transport socket read resumption requests and replay them when re-enabling read. #13772

Merged

Conversation

antoniovicente
Copy link
Contributor

Commit Message: connection: Remember transport socket read resumption requests and replay them when re-enabling read.
Additional Description: Fixes SslSocket read resumption after readDisable when processing the SSL record that contains the last bytes of the HTTP message
Risk Level: low
Testing: new unit and integration tests
Docs Changes: n/a
Release Notes: added
Platform Specific Features: n/a
Fixes #12304

…play them when re-enabling read.

Signed-off-by: Antonio Vicente <[email protected]>
Copy link
Contributor

@alyssawilk alyssawilk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, I like this way better.
High level thought - do we want to runtime guard this just because it's high risk? Mild concerns about the complexity here opening up a spin loop bug.

@@ -554,6 +558,7 @@ void ConnectionImpl::onReadReady() {
return;
}

want_read_ = false;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we also set this false for the early return onRead above? If not please comment.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commented.

MockTransportConnectionImplTest.ResumeWhileAndAfterReadDisable covers the early return case which requires us to clear want_read_ just before doRead.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry how is want_read_ cleared before onRead above? Maybe update the comment above also as to why we don't need to clear or how it's already cleared?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added comment.

@mattklein123 mattklein123 self-assigned this Oct 27, 2020
Signed-off-by: Antonio Vicente <[email protected]>
Copy link
Contributor Author

@antoniovicente antoniovicente left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, I like this way better.
High level thought - do we want to runtime guard this just because it's high risk? Mild concerns about the complexity here opening up a spin loop bug.

Let's talk about the risk level of this PR. The case I'm adding resumption for is after a call to readDisable(false). That can happen when either:

  • High watermark trigger cleared from read buffer as bytes are drained
  • High watermark trigger cleared from write buffer as bytes are drained
  • Finished processing of an H1 request from the client

All of these count as proofs of work. It doesn't seem likely that the connection will continuously wake up when there's no available work.

If for instance we were to not clear want_read_ when we should, we'ld get a spurious wakeup on readDisable(false), but only once unless the connection is able to successfully read additional bytes from the socket and push them to the filter pipeline.

I agree that this PR increases complexity as it requires that the connection track additional state about the state of the transport and increases coupling between the connection and transport implementation.

I think the risk level of this PR is relatively low. That said, I have no objections to runtime guarding this change as an extra precaution.

Thanks for the prompt review.

@@ -554,6 +558,7 @@ void ConnectionImpl::onReadReady() {
return;
}

want_read_ = false;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commented.

MockTransportConnectionImplTest.ResumeWhileAndAfterReadDisable covers the early return case which requires us to clear want_read_ just before doRead.

Copy link
Member

@mattklein123 mattklein123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks this is much better agreed! I haven't looked at the tests but the change looks like the right shape. Happy to take another pass once the comments are fleshed out a bit.

/wait

Comment on lines 358 to 362
setReadBufferReady();
file_event_->activate(Event::FileReadyType::Read);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For clarity I might revert this change, even though it redundantly sets want_read_, but nbd either way. Up to you.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted.

@@ -554,6 +558,7 @@ void ConnectionImpl::onReadReady() {
return;
}

want_read_ = false;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry how is want_read_ cleared before onRead above? Maybe update the comment above also as to why we don't need to clear or how it's already cleared?

Signed-off-by: Antonio Vicente <[email protected]>
Signed-off-by: Antonio Vicente <[email protected]>
Signed-off-by: Antonio Vicente <[email protected]>
mattklein123
mattklein123 previously approved these changes Oct 27, 2020
Copy link
Member

@mattklein123 mattklein123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks this LGTM modulo potential comment on naming. I will defer to @alyssawilk for final review. Thank you for iteration this is much cleaner!

// be read disabled. To handle these cases, fake an event to make sure the buffered data gets
// processed regardless and ensure that we dispatch it via onRead.
if ((consumerWantsToRead() && read_buffer_.length() > 0) ||
(read_disable_count_ == 0 && want_read_)) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry one other question from another look: is it possible to somehow fold all of this logic into consumerWantsToRead()? It seems kind of odd that we have a function that is tracking the consumer wants to read and then a different wants to read boolean. If it's too hard to merge them maybe some better names for both the function and the boolean?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tried to improve on names and the if predicate.

@yanavlasov yanavlasov self-assigned this Oct 28, 2020
@yanavlasov
Copy link
Contributor

The change looks good and I think is safe. My paranoid self tells me the flag protecting it is a good idea though. I think we just need to flag protect setting the transport_wants_read_ = true in the setReadBufferReady() method. Up to you though.

EXPECT_CALL(*file_event_, activate(Event::FileReadyType::Read));
connection_->readDisable(false);

// Do a read to clear the want_read_ flag, verify that no read activation is scheduled.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

want_read_ -> transport_wants_read_ here and other places below.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Nice catch.

alyssawilk
alyssawilk previously approved these changes Oct 28, 2020
Signed-off-by: Antonio Vicente <[email protected]>
Copy link
Member

@mattklein123 mattklein123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks LGTM with one small question.

Comment on lines +364 to +365
// Sanity check: resumption with read_disable_count_ > 0 should only happen if the read
// buffer's high watermark has triggered.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stupid question: why do we bother with this extra logic to read when the disable count is still 1 and high watermark triggered? Is the idea that we already have the data so we might as well flush it? If the high watermark triggered it's probably going to be stuck on the other side, so mostly just wondering if this extra logic is actually worth it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is part of the implementation of high-watermarks on the read buffer introduced in PR #11170

The parser is able to consume input from the read buffer, we just don't want to read additional bytes from the transport into the read buffer.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the idea here is that we allow reading data out in the hope that it will get us below the low watermark? OK thanks that makes sense.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes.

The common scenario is when the H1 parser stops consuming bytes from the read buffer and calls readDisable(true) once it detects the end of the current request message. When request processing is complete, the H1 parser re-enables read. When H1 pipelining is in use, a full second request message could be in the read buffer and the read buffer may have triggered the high-watermark.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sense, though note that we don't do pipelining.

@antoniovicente
Copy link
Contributor Author

/retest

@repokitteh-read-only
Copy link

Retrying Azure Pipelines:
Retried failed jobs in: envoy-presubmit

🐱

Caused by: a #13772 (comment) was created by @antoniovicente.

see: more, trace.

@antoniovicente
Copy link
Contributor Author

/retest

giving macos one last chance.

@repokitteh-read-only
Copy link

Retrying Azure Pipelines:
Retried failed jobs in: envoy-presubmit

🐱

Caused by: a #13772 (comment) was created by @antoniovicente.

see: more, trace.

@antoniovicente antoniovicente merged commit a625c80 into envoyproxy:master Oct 28, 2020
mpuncel added a commit to mpuncel/envoy that referenced this pull request Oct 30, 2020
* master: (83 commits)
  tls: Typesafe tls slots (envoyproxy#13789)
  docs(example): Correct URL for caching example page (envoyproxy#13810)
  [fuzz] Made health check fuzz more efficient (envoyproxy#13747)
  rtds: properly scope rtds stats (envoyproxy#13764)
  http: fixing a bug with IPv6 hosts (envoyproxy#13798)
  connection: Remember transport socket read resumption requests and replay them when re-enabling read. (envoyproxy#13772)
  network: adding some accessors for ALPN work. (envoyproxy#13785)
  docs: added a step about how to handle platform specific extensions (envoyproxy#13759)
  Fix identation in ip transparency code snippet (envoyproxy#13743)
  wasm: enable WAVM's stack unwinding feature (envoyproxy#13792)
  log: set route name for direct response (envoyproxy#13683)
  Use nghttp2 as external dependsncy in protocol_constraints_lib (envoyproxy#13763)
  [Windows] Update windows dev docs (envoyproxy#13741)
  cel: patch thread safety issue (envoyproxy#13739)
  Windows: Fix ssl_socket_test (envoyproxy#13264)
  apple dns: add fake api test suite (envoyproxy#13780)
  overload: scale selected timers in response to load (envoyproxy#13475)
  examples: Add dynamic configuration (control plane) sandbox (envoyproxy#13746)
  Removed exception in getResponseStatus() (envoyproxy#13314)
  network: add timeout for transport connect (envoyproxy#13610)
  ...

Signed-off-by: Michael Puncel <[email protected]>
@PiotrSikora
Copy link
Contributor

/backport

(carried from the issue #12304)

@repokitteh-read-only repokitteh-read-only bot added the backport/review Request to backport to stable releases label Nov 3, 2020
@PiotrSikora
Copy link
Contributor

@cpakulski anything with backport/review label needs to be evaluated for backporting and if approved, that label should be changed to backport/approved and PRs sent to stable releases.

@cpakulski cpakulski added backport/approved Approved backports to stable releases and removed backport/review Request to backport to stable releases labels Nov 13, 2020
cpakulski pushed a commit to cpakulski/envoy that referenced this pull request Nov 13, 2020
…sts and replay them when re-enabling read. (envoyproxy#13772)

Fixes SslSocket read resumption after readDisable when processing the SSL record that contains the last bytes of the HTTP message

Signed-off-by: Antonio Vicente <[email protected]>

Signed-off-by: antonio <[email protected]>
Signed-off-by: Christoph Pakulski <[email protected]>
lizan pushed a commit that referenced this pull request Nov 16, 2020
…on requests and replay them when re-enabling read. (#13772) (#14017)

Fixes SslSocket read resumption after readDisable when processing the SSL record that contains the last bytes of the HTTP message

Risk Level: low
Testing: new unit and integration tests
Docs Changes: n/a
Release Notes: added
Platform Specific Features: n/a
Fixes #12304

Signed-off-by: Antonio Vicente <[email protected]>
Signed-off-by: Christoph Pakulski <[email protected]>
cpakulski pushed a commit to cpakulski/envoy that referenced this pull request Nov 18, 2020
…play them when re-enabling read. (envoyproxy#13772)

Fixes SslSocket read resumption after readDisable when processing the SSL record that contains the last bytes of the HTTP message

Signed-off-by: Antonio Vicente <[email protected]>

Signed-off-by: antonio <[email protected]>
Signed-off-by: Christoph Pakulski <[email protected]>
antoniovicente added a commit to antoniovicente/envoy that referenced this pull request Nov 24, 2020
…play them when re-enabling read. (envoyproxy#13772)

Fixes SslSocket read resumption after readDisable when processing the SSL record that contains the last bytes of the HTTP message

Signed-off-by: Antonio Vicente <[email protected]>
lizan pushed a commit that referenced this pull request Dec 1, 2020
…ion requests and replay them when re-enabling read. (#13772) (#14173)

* connection: Remember transport socket read resumption requests and replay them when re-enabling read. (#13772)

Fixes SslSocket read resumption after readDisable when processing the SSL record that contains the last bytes of the HTTP message

Signed-off-by: Antonio Vicente <[email protected]>
antoniovicente added a commit to antoniovicente/envoy that referenced this pull request Dec 3, 2020
…ion requests and replay them when re-enabling read. (envoyproxy#13772) (envoyproxy#14173)

Fixes SslSocket read resumption after readDisable when processing the SSL record that contains the last bytes of the HTTP message

Signed-off-by: Antonio Vicente <[email protected]>
antoniovicente added a commit to antoniovicente/envoy that referenced this pull request Dec 3, 2020
…ion requests and replay them when re-enabling read. (envoyproxy#13772) (envoyproxy#14173)

Fixes SslSocket read resumption after readDisable when processing the SSL record that contains the last bytes of the HTTP message

Signed-off-by: Antonio Vicente <[email protected]>
antoniovicente added a commit that referenced this pull request Dec 4, 2020
…ion requests and replay them when re-enabling read. (#13772) (#14173) (#14254)

Fixes SslSocket read resumption after readDisable when processing the SSL record that contains the last bytes of the HTTP message

Signed-off-by: Antonio Vicente <[email protected]>
antoniovicente added a commit that referenced this pull request Dec 5, 2020
…ion requests and replay them when re-enabling read. (#13772) (#14173) (#14255)

Fixes SslSocket read resumption after readDisable when processing the SSL record that contains the last bytes of the HTTP message

Signed-off-by: Antonio Vicente <[email protected]>
@cpakulski
Copy link
Contributor

Backported to releases 1.13, 1.14, 1.15 and 1.16. Removing backport/approved label.

@cpakulski cpakulski removed the backport/approved Approved backports to stable releases label Dec 16, 2020
istio-testing pushed a commit to istio/envoy that referenced this pull request Jan 8, 2021
* docs: kick-off 1.15.1 release. (envoyproxy#12166)

Signed-off-by: Piotr Sikora <[email protected]>

* tls: update BoringSSL-FIPS to 20190808. (envoyproxy#12170)

Signed-off-by: Piotr Sikora <[email protected]>

* test: Exclude wasm_vm_test from CI by making it a "manual" test. (#207)

The wee v8 build times out in CI under --config=asan because the machine the job is scheduled on is too small.

Signed-off-by: Antonio Vicente <[email protected]>

* [v1.15] http: header map security fixes for duplicate headers (#197) (#200)

Previously header matching did not match on all headers for
non-inline headers. This patch changes the default behavior to
always logically match on all headers. Multiple individual
headers will be logically concatenated with ',' similar to what
is done with inline headers. This makes the behavior effectively
consistent. This behavior can be temporary reverted by setting
the runtime value "envoy.reloadable_features.header_match_on_all_headers"
to "false".

Targeted fixes have been additionally performed on the following
extensions which make them consider all duplicate headers by default as
a comma concatenated list:
1) Any extension using CEL matching on headers.
2) The header to metadata filter.
3) The JWT filter.
4) The Lua filter.
Like primary header matching used in routing, RBAC, etc. this behavior
can be disabled by setting the runtime value
"envoy.reloadable_features.header_match_on_all_headers" to false.

Finally, the setCopy() header map API previously only set the first
header in the case of duplicate non-inline headers. setCopy() now
behaves similiarly to the other set*() APIs and replaces all found
headers with a single value. This may have had security implications
in the extauth filter which uses this API. This behavior can be disabled
by setting the runtime value
"envoy.reloadable_features.http_set_copy_replace_all_headers" to false.

Fixes https://github.com/envoyproxy/envoy-setec/issues/188

Signed-off-by: Matt Klein <[email protected]>

* backport to v1.15: Fix Kafka Repository Location (#223)

Update mirror used to fetch kafka dependency to a valid, working mirror.

Based on envoyproxy#13025
Resolves envoyproxy#13011

Signed-off-by: Antonio Vicente <[email protected]>

* release: cutting 1.15.1 (#217)

Signed-off-by: Antonio Vicente <[email protected]>

* docs: Fix release notes for v1.15.1 release. (envoyproxy#13318)

Signed-off-by: Antonio Vicente <[email protected]>

* Backport flaky test and tsan fixes to releases/v1.15 branch (envoyproxy#13337)

* hds: fix integration test flakes (envoyproxy#12214)

Part of envoyproxy#12184

Signed-off-by: Matt Klein <[email protected]>
Signed-off-by: Antonio Vicente <[email protected]>

* Switch to a tsan-instrumented libc++ for tsan tests (envoyproxy#12134)

This fixes envoyproxy#9784 and re-enables vhds_integration_test

Risk Level: Low, but will most likely increase memory usage

Signed-off-by: Dmitri Dolguikh <[email protected]>

Signed-off-by: Antonio Vicente <[email protected]>

* test: shard hds_integration_test (envoyproxy#12482)

This should avoid TSAN timeout flakes.

Signed-off-by: Matt Klein <[email protected]>
Signed-off-by: Antonio Vicente <[email protected]>

* test: shard http2_integration_test (envoyproxy#11939)

This should mitigate TSAN timeout.

Signed-off-by: Lizan Zhou <[email protected]>
Signed-off-by: Antonio Vicente <[email protected]>

* test: fix http2_integration_test flake (envoyproxy#12450)

Fixes envoyproxy#12442

Signed-off-by: Matt Klein <[email protected]>
Signed-off-by: Antonio Vicente <[email protected]>

* Kick CI

Signed-off-by: Antonio Vicente <[email protected]>

Co-authored-by: Matt Klein <[email protected]>
Co-authored-by: Dmitri Dolguikh <[email protected]>
Co-authored-by: Lizan Zhou <[email protected]>

* docs: kick off v1.15.3-dev (envoyproxy#13695)

Signed-off-by: Christoph Pakulski <[email protected]>

* 1.15: CI fixes backport (envoyproxy#13697)

Backport following commits to 1.15:
748b2ab (mac ci: try ignoring update failure (envoyproxy#13658), 2020-10-20)
f95f539 (ci: various improvements (envoyproxy#13660), 2020-10-20)
73d78f8 (ci: use multiple stage (envoyproxy#13557), 2020-10-15)
b7a4756 (ci: use azp for api and go-control-plane sync (envoyproxy#13550), 2020-10-14)
876a6bb (ci use azp to sync filter example (envoyproxy#13501), 2020-10-12)
a0f31ee (ci: use azp to generate docs (envoyproxy#13481), 2020-10-12)

Signed-off-by: Lizan Zhou <[email protected]>
Co-authored-by: asraa <[email protected]>

* 1.15: fix CI script (envoyproxy#13724)

Signed-off-by: Lizan Zhou <[email protected]>

* Prevent SEGFAULT when disabling listener (envoyproxy#13515) (envoyproxy#13903)

This prevents the stop_listening overload action from causing
segmentation faults that can occur if the action is enabled after the
listener has already shut down.

Signed-off-by: Alex Konradi <[email protected]>
Signed-off-by: Christoph Pakulski <[email protected]>

* proxy protocol: set downstreamRemoteAddress on StreamInfo (envoyproxy#14131) (envoyproxy#14169)

This fixes a regression which resulted in the downstreamRemoteAddress
on the StreamInfo for a connection not having the address supplied by
the proxy protocol filter, but instead having the address of the
directly connected peer.

This issue does not affect HTTP filters.

Fixes envoyproxy#14087

Signed-off-by: Greg Greenway <[email protected]>
Signed-off-by: Christoph Pakulski <[email protected]>

* ci: temproray disable vhds_integration_test in TSAN (envoyproxy#12067) (envoyproxy#14217)

Signed-off-by: Lizan Zhou <[email protected]>

* tcmalloc changed and the data coming out of tcmalloc::MallocExtension::GetNumericProperty("generic.current_allocated_bytes") (envoyproxy#14165)

Commit Message: tcmalloc changed and the data coming out of tcmalloc::MallocExtension::GetNumericProperty("generic.current_allocated_bytes") no longer appears to be deterministic, even in unthreaded tests. So disable exact mem checks till we sort that out
Additional Description:
Risk Level: low
Testing: just thread_local_store_test
Docs Changes: n/a
Release Notes: n/a

no longer appears to be deterministic, even in unthreaded tests. So disable exact mem checks till we sort that out

Signed-off-by: Joshua Marantz <[email protected]>
Signed-off-by: Christoph Pakulski <[email protected]>

Co-authored-by: Joshua Marantz <[email protected]>

* backport to v1.15: connection: Remember transport socket read resumption requests and replay them when re-enabling read. (envoyproxy#13772) (envoyproxy#14173)

* connection: Remember transport socket read resumption requests and replay them when re-enabling read. (envoyproxy#13772)

Fixes SslSocket read resumption after readDisable when processing the SSL record that contains the last bytes of the HTTP message

Signed-off-by: Antonio Vicente <[email protected]>

* backport to 1.15: udp: properly handle truncated/dropped datagrams (envoyproxy#14122) (envoyproxy#14166)

Signed-off-by: Matt Klein <[email protected]>
Signed-off-by: Christoph Pakulski <[email protected]>
Co-authored-by: Matt Klein <[email protected]>
Co-authored-by: Christoph Pakulski <[email protected]>

* backport to 1.15: vrp: allow supervisord to open its log file (envoyproxy#14066) (envoyproxy#14280)

Commit Message: Allow supervisord to open its log file
Additional Description:
Change the default location of the log file and give supervisord
permissions to write to it.

Risk Level: low
Testing: built image locally
Docs Changes: n/a
Release Notes: n/a
Platform Specific Features: n/a

Signed-off-by: Alex Konradi <[email protected]>
Signed-off-by: Christoph Pakulski <[email protected]>

* rel 1.15: close release 1.15.3 (envoyproxy#14303)

Signed-off-by: Christoph Pakulski <[email protected]>

* Kick off rel 1.15.4. (envoyproxy#14323)

Signed-off-by: Christoph Pakulski <[email protected]>

* backport to 1.15: http: fix datadog and squash handling of responses without body (envoyproxy#13328) (envoyproxy#14458)

Commit Message: Fixing bugs in datadog and sqaush where unexpected bodyless responses would crash Envoy
Risk Level: low
Testing: new unit tests, updated certs
Docs Changes: n/a
Release Notes: inline
Signed-off-by: Christoph Pakulski <[email protected]>
Co-authored-by: alyssawilk <[email protected]>

* backport 1.15: http: fixing a bug with IPv6 hosts (envoyproxy#14273)

Fixing a bug where HTTP parser offsets for IPv6 hosts did not include [] and Envoy assumed it did.
This results in mis-parsing addresses for IPv6 CONNECT requests and IPv6 hosts in fully URLs over HTTP/1.1

Risk Level: low
Testing: new unit, integration tests
Docs Changes: n/a
Release Notes: inline

Signed-off-by: Shikugawa <[email protected]>
Co-authored-by: alyssawilk <[email protected]>

* backport to 1.15: tls: fix detection of the upstream connection close event. (envoyproxy#13858) (envoyproxy#14568)

Fixes envoyproxy#13856.

Signed-off-by: Piotr Sikora <[email protected]>
Signed-off-by: Christoph Pakulski <[email protected]>

Co-authored-by: Piotr Sikora <[email protected]>
Co-authored-by: antonio <[email protected]>
Co-authored-by: Matt Klein <[email protected]>
Co-authored-by: Dmitri Dolguikh <[email protected]>
Co-authored-by: Lizan Zhou <[email protected]>
Co-authored-by: Christoph Pakulski <[email protected]>
Co-authored-by: asraa <[email protected]>
Co-authored-by: Joshua Marantz <[email protected]>
Co-authored-by: Rei Shimizu <[email protected]>
Co-authored-by: alyssawilk <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

shouldDrainReadBuffer followed up by readDisable(true) and readDisable(false) strange behaviour
6 participants