Grpc timeout exception

[wisamidris7]

Issue with TLS Configuration in Docker Networking

Description

I am experiencing issues with TLS configuration for inter-container communication in Docker Compose. The issue arises when one container attempts to communicate with another using HTTPS, and both configurations I’ve tried have not resolved the problem.
And when the request coming from outside docker compose it worked.

Docker Environment Variables

For EditionsService Container

The following environment variables are set:

ASPNETCORE_Kestrel__Endpoints__Https__Url="https://example.com:10004"
ASPNETCORE_Kestrel__Certificates__Default__Path="/app/certs/{private}.pfx"
ASPNETCORE_Kestrel__Certificates__Default__Password="{private}"
ASPNETCORE_Kestrel__EndpointDefaults__Protocols="Http1AndHttp2"

Issue Details

When IdentityService tries to make a gRPC call to EditionsService, two configurations have been tested, each resulting in different issues:

1. External URL Configuration

   services__editions-api__https__0="https://example.com:10004/"

   • Problem: This configuration works for external access but times out when used for internal Docker communication between containers.

2. Internal Network URL Configuration

   services__editions-api__https__0="https://common-api:10004/"

   • Problem: This configuration results in SSL certificate validation issues because common-api is not recognized by the certificate issued for example.com.

Request for Assistance

I am seeking guidance on how to properly configure TLS for secure inter-container communication in Docker Compose. Specifically, I need a solution that addresses both:

1. Ensuring valid SSL/TLS certificates for internal communication.
2. Avoiding timeout issues when using external URLs for internal Docker network communication.

Any advice or suggestions on how to resolve these issues would be greatly appreciated.

[behdad088]

Hey @wisamidris7, so I’ve been looking into the TLS issue with Docker Compose, and here’s what’s going on and how we can fix it.

The Problem:

1. External URL Configuration (example.com):

   • This works fine when we're accessing the service from outside Docker Compose.
   • But when one container (like IdentityService) tries to hit another (EditionsService) using the external URL (example.com), it times out because Docker containers are trying to talk to each other over the external network, which isn’t the best way to do it.

2. Internal Network URL Configuration (common-api):

   • If we try to use the internal Docker network (common-api) to connect, we hit SSL certificate issues. That’s because the SSL cert is issued for example.com, but we’re using common-api inside Docker, which doesn’t match, so it fails the validation.

Possible Fixes:

1. Use a Wildcard or SAN Certificate:

• One option is to get a SAN certificate or a wildcard certificate that covers both example.com and common-api (or something like *.example.com).
• This way, we can validate the SSL/TLS both internally (via common-api) and externally (via example.com).

So we’d update the cert, mount it in the container, and change the Compose setup to point to the updated cert.

2. Skip SSL Validation for Internal Requests:

• Another option is to tell the internal services (like IdentityService) to skip SSL validation when making requests inside Docker.

• In .NET, for example, we can modify the HTTP client to ignore SSL issues for internal Docker calls like this:

  var httpClientHandler = new HttpClientHandler();
  httpClientHandler.ServerCertificateCustomValidationCallback = 
      (message, cert, chain, sslPolicyErrors) => { return true; };
  
  var httpClient = new HttpClient(httpClientHandler);

• This is a bit of a shortcut but works well when it’s just internal communication.

3. Set Up DNS for Internal Services:

• We could set up a custom DNS inside Docker so that common-api resolves to example.com. This way, the certificate would match, and we won’t get SSL validation errors.
• This approach is a bit more complex but can work if we want a cleaner solution without skipping SSL validation.

4. Use an Nginx Reverse Proxy:

• Another way is to use Nginx inside Docker as a reverse proxy. This lets us handle the SSL certs at the proxy level and forward the requests to the right container.
• Basically, Nginx will handle requests to example.com and route them to common-api, and we can keep the SSL cert valid for example.com.

Here’s how that might look:

• Nginx gets the external requests, does the SSL handshake using example.com, and then internally forwards the request to common-api.

This keeps everything secure and avoids those annoying SSL validation issues.

TL;DR:

We have a few options:

1. Get a SAN or wildcard cert to cover both the external and internal service names.
2. Bypass SSL validation for internal Docker calls (quick and dirty, but it works).
3. Use a custom DNS inside Docker to resolve internal services to the external domain.
4. Set up Nginx as a reverse proxy to handle the SSL certs and forward requests.

Let me know which direction you think makes sense for our setup!