ca-certificates issue on the 1.4.0 version's docker image

[eole1712]

Describe the bug
Hello !
Since the latest version of the docker image (1.4.0), the router does not launch and quits panicking in hyper-rustls because of the lack of CA certificates.

I have added the trace of the error.

To Reproduce

1. Build the docker image with --platform=linux/amd64 parameter
2. Launch the router with the APOLLO_KEY and APOLLO_GRAPH_REF environment variables to use the Uplink and fetch the scheme from Apollo Studio

Expected behavior
The router launches correctly.

Output

Default output :

2022-11-23T08:51:32.159934Z  INFO Apollo Router v1.4.0 // (c) Apollo Graph, Inc. // Licensed as ELv2 (https://go.apollo.dev/elv2)
2022-11-23T08:51:32.161786Z  INFO Anonymous usage data is gathered to inform Apollo product development.  See https://go.apollo.dev/o/privacy for more info.
2022-11-23T08:51:39.180584Z  INFO Apollo Studio usage reporting is enabled. See https://go.apollo.dev/o/data for details
2022-11-23T08:51:39.182924Z  INFO creating apollo exporter spaceport_endpoint=https://127.0.0.1:45471/
2022-11-23T08:51:39.187163Z  INFO creating apollo exporter spaceport_endpoint=https://127.0.0.1:45471/
thread 'tokio-runtime-worker' panicked at 'no CA certificates found', /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/hyper-rustls-0.23.1/src/config.rs:48:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2022-11-23T08:51:39.212747Z apollo-router/src/plugins/telemetry/apollo_exporter.rs:133  INFO apollo_router::plugins::telemetry::apollo_exporter: terminating apollo exporter spaceport_endpoint=https://127.0.0.1:45471/
2022-11-23T08:51:39.219213Z apollo-router/src/plugins/telemetry/apollo_exporter.rs:133  INFO apollo_router::plugins::telemetry::apollo_exporter: terminating apollo exporter spaceport_endpoint=https://127.0.0.1:45471/
2022-11-23T08:51:39.222211Z ERROR task 11 panicked
2022-11-23T08:51:39.222623Z ERROR failed to start server

The output with RUST_BACKTRACE=full :

2022-11-23T09:05:09.160763Z  INFO Apollo Router v1.4.0 // (c) Apollo Graph, Inc. // Licensed as ELv2 (https://go.apollo.dev/elv2)
2022-11-23T09:05:09.162174Z  INFO Anonymous usage data is gathered to inform Apollo product development.  See https://go.apollo.dev/o/privacy for more info.
2022-11-23T09:05:16.582771Z  INFO Apollo Studio usage reporting is enabled. See https://go.apollo.dev/o/data for details
2022-11-23T09:05:16.584891Z  INFO creating apollo exporter spaceport_endpoint=https://127.0.0.1:45543/
2022-11-23T09:05:16.590531Z  INFO creating apollo exporter spaceport_endpoint=https://127.0.0.1:45543/
thread 'tokio-runtime-worker' panicked at 'no CA certificates found', /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/hyper-rustls-0.23.1/src/config.rs:48:9
stack backtrace:
   0:       0x400241480d - std::backtrace_rs::backtrace::libunwind::trace::h8217d0a8f3fd2f41
                               at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:       0x400241480d - std::backtrace_rs::backtrace::trace_unsynchronized::h308103876b3af410
                               at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:       0x400241480d - std::sys_common::backtrace::_print_fmt::hc208018c6153605e
                               at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/sys_common/backtrace.rs:66:5
   3:       0x400241480d - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hf89a7ed694dfb585
                               at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/sys_common/backtrace.rs:45:22
   4:       0x400243a71c - core::fmt::write::h21038c1382fe4264
                               at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/fmt/mod.rs:1197:17
   5:       0x400240da81 - std::io::Write::write_fmt::h7dbb1c9a3c254aef
                               at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/io/mod.rs:1672:15
   6:       0x4002416275 - std::sys_common::backtrace::_print::h4e8889719c9ddeb8
                               at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/sys_common/backtrace.rs:48:5
   7:       0x4002416275 - std::sys_common::backtrace::print::h1506fe2cb3022667
                               at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/sys_common/backtrace.rs:35:9
   8:       0x4002416275 - std::panicking::default_hook::{{closure}}::hd9d7ce2a8a782440
                               at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/panicking.rs:295:22
   9:       0x4002415f96 - std::panicking::default_hook::h5b16ec25444b1b5d
                               at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/panicking.rs:314:9
  10:       0x4002416806 - std::panicking::rust_panic_with_hook::hb0138cb6e6fea3e4
                               at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/panicking.rs:698:17
  11:       0x4000e19c7b - std::panicking::begin_panic::{{closure}}::hf1d75c233e96dfcb
  12:       0x4000e19c44 - std::sys_common::backtrace::__rust_end_short_backtrace::h1d7b4943490e9b11
  13:       0x400026dd9a - std::panicking::begin_panic::h0a170535f6de683c
  14:       0x4000e1a559 - <rustls::builder::ConfigBuilder<rustls::client::client_conn::ClientConfig,rustls::builder::WantsVerifier> as hyper_rustls::config::ConfigBuilderExt>::with_native_roots::h33b4fd15b70dcdf1
  15:       0x4000e19886 - hyper_rustls::connector::builder::ConnectorBuilder<hyper_rustls::connector::builder::WantsTlsConfig>::with_native_roots::h721841ea76c08c37
  16:       0x4000663e8a - apollo_router::services::subgraph_service::SubgraphService::new::h5f42886bd1c88438
  17:       0x40007f355e - <apollo_router::router_factory::YamlSupergraphServiceFactory as apollo_router::router_factory::SupergraphServiceConfigurator>::create::{{closure}}::h1455dc087792a5ec
  18:       0x40007903c2 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::hdf21c519f3964de5
  19:       0x4000795057 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::heb77c69a5f91d85b
  20:       0x40007db4ea - apollo_router::state_machine::StateMachine<S,FA>::process_events::{{closure}}::hc034b086e13ae8e8
  21:       0x400074720c - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h1a11f7b4ac1e2fd9
  22:       0x400061e31a - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h094b1d1efd1deb6c
  23:       0x40008e13fa - tokio::runtime::task::core::Core<T,S>::poll::h57afd211f05e6a83
  24:       0x40005c2ef2 - tokio::runtime::task::harness::Harness<T,S>::poll::h8ad47af927e00a10
  25:       0x4002329bde - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::h740826a2a12ef76c
  26:       0x40023292cf - tokio::runtime::scheduler::multi_thread::worker::Context::run::he6e6b717918b53a4
  27:       0x4002320ec7 - tokio::macros::scoped_tls::ScopedKey<T>::set::hb793ef04c07affcf
  28:       0x4002328da4 - tokio::runtime::scheduler::multi_thread::worker::run::h0df3ba70e2875049
  29:       0x400230a7ce - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::hb4d8d10612111d03
  30:       0x400231e8b5 - tokio::runtime::task::core::Core<T,S>::poll::h9ae8e48ea7f52501
  31:       0x40023180cf - tokio::runtime::task::harness::Harness<T,S>::poll::h479887723773f632
  32:       0x4002317471 - tokio::runtime::blocking::pool::Inner::run::hb522249ed9ae68c2
  33:       0x400230b983 - std::sys_common::backtrace::__rust_begin_short_backtrace::hb9dca60c7b65a48e
  34:       0x400230c0bf - core::ops::function::FnOnce::call_once{{vtable.shim}}::had1a811870d0ef5b
  35:       0x4002419d13 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h1680342795a2dc08
                               at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/alloc/src/boxed.rs:1951:9
  36:       0x4002419d13 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h45204a69827b0e83
                               at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/alloc/src/boxed.rs:1951:9
  37:       0x4002419d13 - std::sys::unix::thread::Thread::new::thread_start::h5d4e11bbda4161c8
                               at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/sys/unix/thread.rs:108:17
  38:       0x40055a7ea7 - start_thread
  39:       0x4005808a2f - clone
  40:                0x0 - <unknown>
2022-11-23T09:05:16.752524Z apollo-router/src/plugins/telemetry/apollo_exporter.rs:133  INFO apollo_router::plugins::telemetry::apollo_exporter: terminating apollo exporter spaceport_endpoint=https://127.0.0.1:45543/
2022-11-23T09:05:16.765523Z apollo-router/src/plugins/telemetry/apollo_exporter.rs:133  INFO apollo_router::plugins::telemetry::apollo_exporter: terminating apollo exporter spaceport_endpoint=https://127.0.0.1:45543/
2022-11-23T09:05:16.766491Z ERROR task 11 panicked
2022-11-23T09:05:16.767160Z ERROR failed to start server
failed to start server

Additional context
Add any other context about the problem here.

[eole1712]

According to my tests it's because the ssl certificates are missing in the run image debian:bullseye-slim introduced by #2085 while they were present in gcr.io/distroless/cc-debian11

[garypen]

This is confusing me. I've used our docker image a number of times with no issues.

The required certificates are present be cause we install curl and use curl to do the following:

RUN curl -sSL https://router.apollo.dev/download/nix/${ROUTER_RELEASE}/ | sh

which wouldn't work if the required certificates were not present.

You mention building your own docker image. Could that be the source of the problem?

[garypen]

Ok. I know what the problem is now. You are using build_docker_image.sh to build your own image and that doesn't have the certificates in place since 1.4.0. I already raised another PR, #2142 , which address this. I'll close your PR and mark my PR as resolving this issue.