Loglevel Trace causes Router to crash

[lleadbet]

Describe the bug

Reporting on behalf of a customer. They're reporting these issues (whenever loglevel is set to trace):

• If you've enabled the TLS configuration with loglevel as trace, the Router will crash after some time, or potentially immediately
• If you don't have any TLS configuration and view the sandbox and refresh, it crashes after a minute or so
• if you've enabled the auth plugin and configured subgraph TLS along with --log=trace, the Router crashes on start

To Reproduce

Use this config:

authentication:
 jwt:
   jwks: # This key is required.
     - url: https://dev-zzp5enui.us.auth0.com/.well-known/jwks.json

   # These keys are optional. Default values are shown.
   header_name: Authorization
   header_value_prefix: Bearer

authorization:
  require_authentication: true

tls:
  subgraph:
    subgraphs:
      subgraph_name:
        certificate_authorities: "${file.config/cer.crt}"

Start the router with ./router --config router.yaml --log=trace and it should crash.

Expected behavior

Router to not crash.

Output

Last log before crashing over time (TLS only):

thread panicked while processing panic. aborting.
Abort trap: 6

Last logs before crashing on start (JWT + TLS)

{"timestamp":"2023-07-18T15:41:07.719052Z","level":"DEBUG","message":"loading root certificates","target":"apollo_router::router_factory"}
{"timestamp":"2023-07-18T15:41:07.719069Z","level":"TRACE","connection.state":"Open","target":"h2::proto::connection","span":{"field_error":"EOF while parsing a value at line 1 column 0","name":"poll"},"spans":[{"field_error":"EOF while parsing a value at line 1 column 0","name":"Connection"},{"field_error":"EOF while parsing a value at line 1 column 0","name":"poll"}]}
{"timestamp":"2023-07-18T15:41:07.719126Z","level":"TRACE","message":"poll","target":"h2::codec::framed_read","span":{"field_error":"EOF while parsing a value at line 1 column 0","name":"FramedRead::poll_next"},"spans":[{"field_error":"EOF while parsing a value at line 1 column 0","name":"Connection"},{"field_error":"EOF while parsing a value at line 1 column 0","name":"poll"},{"field_error":"EOF while parsing a value at line 1 column 0","name":"FramedRead::poll_next"}]}
{"timestamp":"2023-07-18T15:41:07.719530Z","level":"INFO","message":"Health check endpoint exposed at http://127.0.0.1:8088/health"}

Desktop (please complete the following information):

• OS: macOS
• Version v1.24.0

Additional context

The affected user also has been experiencing the issue noted in #3124, which may or may not be related. I (personally) have been unable to reproduce either issue, however.

[garypen]

It looks like the error is in the h2 crate. You can use a log filter to set h2 loglevel to info as follows:

--log=trace,h2=info

This will give trace logging for all crates apart from h2 which will log at info level.