tls context manager: failing assertion in destructor

[goaway]

Title: The assertion in Tls::ContextManagerImpl's destructor may sometimes fail.

Description:
When a Tls::ContextManagerImpl is destructed due to an Envoy Server::InstanceImpl being destructed, the ASSERT in the destructor may fail. There's a call immediately above to removeEmptyContexts() which seems to imply that either some contexts are not yet empty and/or removeEmptyContexts() is not performing as expected. Whether this represents an actual bug in the code, or simply an unexpected but benign set of circumstances is unknown.

[Relevant Links:]
For a detailed stack trace, see this Envoy Mobile issue:
envoyproxy/envoy-mobile#572

[asraa]

related, a fuzz bug caught this a while back, and i gave it a stab to fix: #8553

[goaway]

Thank you @asraa!

[mattklein123]

I think this is fixed.

[jamesmulcahy]

The codebase still contains a KNOWN_ISSUE_ASSERT that points here, but Matt's comment above suggests this is fixed. Any pointers on how to reconcile these two?

In the code, I see:

ContextManagerImpl::~ContextManagerImpl() {
  removeEmptyContexts();
  KNOWN_ISSUE_ASSERT(contexts_.empty(), "https://github.com/envoyproxy/envoy/issues/10030");
}

And it manifests as:

#35 20.45 [2021-05-01 01:07:59.085][1446][critical][assert] [external/envoy/source/extensions/transport_sockets/tls/context_manager_impl.cc:20] assert failure: contexts_.empty(). Details: https://github.com/envoyproxy/envoy/issues/10030
#35 20.45 [2021-05-01 01:07:59.085][1446][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] Caught Aborted, suspect faulting address 0x5a6
#35 20.45 [2021-05-01 01:07:59.085][1446][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
#35 20.45 [2021-05-01 01:07:59.085][1446][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:92] Envoy version: 323466f8284d273f783b865bead3a2c11b02f0c4/1.19.0-dev/Distribution/DEBUG/BoringSSL
#35 20.45 [symbolize_elf.inc : 1266] RAW: /bin/proxyd: open failed: errno=2
#35 20.45 [2021-05-01 01:07:59.085][1446][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #0: [0x55d9d351cd8c]
#35 20.45 [2021-05-01 01:07:59.085][1446][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #1: __restore_rt [0x7fcec0d7d980]
#35 20.45 [symbolize_elf.inc : 1266] RAW: /bin/proxyd: open failed: errno=2
#35 20.45 [2021-05-01 01:07:59.085][1446][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #2: [0x55d9d334165c]
#35 20.45 [symbolize_elf.inc : 1266] RAW: /bin/proxyd: open failed: errno=2
#35 20.45 [2021-05-01 01:07:59.085][1446][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #3: [0x55d9d1e1f10f]
#35 20.45 [symbolize_elf.inc : 1266] RAW: /bin/proxyd: open failed: errno=2
#35 20.45 [2021-05-01 01:07:59.085][1446][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #4: [0x55d9d1e1f08f]
#35 20.45 [symbolize_elf.inc : 1266] RAW: /bin/proxyd: open failed: errno=2
#35 20.45 [2021-05-01 01:07:59.085][1446][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #5: [0x55d9d1e121a9]
#35 20.45 [symbolize_elf.inc : 1266] RAW: /bin/proxyd: open failed: errno=2
#35 20.45 [2021-05-01 01:07:59.085][1446][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #6: [0x55d9d1e10f33]
#35 20.45 [symbolize_elf.inc : 1266] RAW: /bin/proxyd: open failed: errno=2
#35 20.45 [2021-05-01 01:07:59.085][1446][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #7: [0x55d9d1e0d5b8]
#35 20.45 [symbolize_elf.inc : 1266] RAW: /bin/proxyd: open failed: errno=2
#35 20.45 [2021-05-01 01:07:59.085][1446][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #8: [0x55d9d16e87fb]
#35 20.45 [symbolize_elf.inc : 1266] RAW: /bin/proxyd: open failed: errno=2
#35 20.45 [2021-05-01 01:07:59.085][1446][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #9: [0x55d9d16eda7e]
#35 20.45 [symbolize_elf.inc : 1266] RAW: /bin/proxyd: open failed: errno=2
#35 20.45 [2021-05-01 01:07:59.085][1446][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #10: [0x55d9d16e9a81]
#35 20.45 [symbolize_elf.inc : 1266] RAW: /bin/proxyd: open failed: errno=2
#35 20.45 [2021-05-01 01:07:59.085][1446][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #11: [0x55d9d16e5e1b]
#35 20.45 [2021-05-01 01:07:59.085][1446][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #12: __libc_start_main [0x7fcec099bbf7]

Is there an actual known issue still, or is this an unrelated/new bug, and that KNOWN_ISSUE_ASSERT should be updated to not point at this issue?

[akonradi]

@mattklein123 @goaway thoughts on the above?

[mattklein123]

I think this is probably a different issue. If you can reproduce it can we debug it directly?

[doujiang24]

I reproduced this error when the certificate pem file path is missing:

[2022-04-13 15:40:24.825][49660][critical][main] [source/server/server.cc:117] error initializing configuration '/path/to/envoy-demo-tls.yaml': Invalid path: certs/servercert.pem
...
[2022-04-13 15:40:24.828][49660][critical][assert] [source/extensions/transport_sockets/tls/context_manager_impl.cc:20] assert failure: contexts_.empty(). Details: https://github.com/envoyproxy/envoy/issues/10030
./start.sh: line 14: 49660 Aborted                 $envoy -c /path/to/envoy-demo-tls.yaml -l trace --component-log-level upstream:info --base-id 9

with such a YAML:

      transport_socket:
        name: envoy.transport_sockets.tls
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
          common_tls_context:
            tls_certificates:
            - certificate_chain:
                filename: certs/servercert.pem
              private_key:
                filename: certs/serverkey.pem

I'm not sure if it's related, hope it helps.