Unable to setup stack behind proxy

[grizzlycode]

Problem description

I'm trying to setup this stack behind a proxy. However, it fails due to timeout.

I have my docker daemon already configured to use the proxy and I use if for my other projects. However, it appears this is not considered when the stack gets built. Upon a cursory search it appears that additonal commands are required when a container is "built" like these are. I've tried various methods provided by the Docker docs , but I don't have a working solution.

Are there additional proxy commands or config changes that must be used to make this stack work behind a proxy?

If so, can you share them and possibly make a guide and an optional variable in the .env so if a person populates for instance HTTP_PROXY=my.proxy.com:3128 it will utilize that in the build process? I can't be the first person that has tried to put this up behind a proxy.

Extra information

Stack configuration

Docker setup

$ docker version

[Version:           24.0.2

API version:       1.43

Go version:        go1.20.4

Git commit:        cb74dfc

Built:             Thu May 25 21:51:00 2023

OS/Arch:           linux/amd64

Context:           default

 

Server: Docker Engine - Community

Engine:

  Version:          24.0.2

  API version:      1.43 (minimum version 1.12)

  Go version:       go1.20.4

  Git commit:       659604f

  Built:            Thu May 25 21:51:00 2023

  OS/Arch:          linux/amd64

  Experimental:     false

containerd:

  Version:          1.6.21

  GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8

runc:

  Version:          1.1.7

  GitCommit:        v1.1.7-0-g860f061

docker-init:

  Version:          0.19.0

  GitCommit:        de40ad0

]

$ docker-compose version

[This is now built-in to newer versions of Docker and no longer applicable.]

Container logs

$ docker-compose logs

[[+] Building 30.3s (3/3) FINISHED

=> [tls internal] load build definition from Dockerfile                                                           0.0s

=> => transferring dockerfile: 193B                                                                               0.0s

=> [tls internal] load .dockerignore                                                                              0.0s

=> => transferring context: 129B                                                                                  0.0s

=> ERROR [tls internal] load metadata for [docker.elastic.co/elasticsearch/elasticsearch:8.7.1](http://docker.elastic.co/elasticsearch/elasticsearch:8.7.1)                    30.3s

------

> [tls internal] load metadata for [docker.elastic.co/elasticsearch/elasticsearch:8.7.1](http://docker.elastic.co/elasticsearch/elasticsearch:8.7.1):

------

failed to solve: DeadlineExceeded: DeadlineExceeded: DeadlineExceeded: [docker.elastic.co/elasticsearch/elasticsearch:8.7.1](http://docker.elastic.co/elasticsearch/elasticsearch:8.7.1): failed to authorize: DeadlineExceeded: failed to fetch anonymous token: Get https://docker-auth.elastic.co/auth?scope=repository%3Aelasticsearch%2Felasticsearch%3Apull&service=token-service: dial tcp [34.68.230.202:443](http://34.68.230.202:443/): i/o timeout]

[antoineco]

Nowadays Docker image builds are performed by docker buildx under the hood.

According to docker/buildx#170 (comment), these options need to be passed as build args. Compose does this automatically since v2.16.0 (docker/compose#10159) by reading your ~/.docker/config.json file.

{ 
  "proxies":
    { 
      "default": 
        { 
          "httpProxy": "http://<ip-address>:<port>", 
          "httpsProxy": "https://<ip-address>:<port>", 
          "noProxy": "*.<domain>,127.0.0.0/8" 
        } 
    } 
}

Ref. https://docs.docker.com/network/proxy/#configure-the-docker-client

[grizzlycode]

I have configured and tried the client config as well and I still get the error.

[antoineco]

If I were you I would check what version of Compose ships with your Docker distribution.

You mentioned that Compose was now part of Docker, and that its version is irrelevant. Both are wrong:

• Compose ships with Docker Desktop (it's inside ~/.docker/cli-plugins/)
• the version is very relevant because, as you can see, support for proxy settings was added very recently.

docker compose version will return the version.

If the version is greater than 2.16.0, it's a Compose bug that needs to be reported upstream. docker-elk is only an application running on top of it.

If the version is less than 2.16.0, an upgrade is required for the proxy client config to be supported.

[grizzlycode]

Understood, I guess I have never not have it update since its been included. I understand though that some may not be able to update their software. According to Docker Compose release notes I have the lastest version.

Compose version:

Docker Compose version v2.18.1

So far I have the docker client config and docker daemon configured to use the proxy.

[antoineco]

@grizzlycode I did my little investigation in #868. It turns out that the proxy settings work as expected, so there is no bug in Compose. I configured a dummy proxy URL, and you can see HTTP requests failing inside containers at https://github.com/deviantony/docker-elk/actions/runs/5104016989/jobs/9174648224

You issue is different and not directly related to Compose.

The message failed to solve: DeadlineExceeded: DeadlineExceeded: DeadlineExceeded: docker.elastic.co/elasticsearch/elasticsearch:8.7.1: failed to authorize: DeadlineExceeded: failed to fetch anonymous token: Get https://docker-auth.elastic.co/auth?scope=repository%3Aelasticsearch%2Felasticsearch%3Apull&service=token-service: dial tcp 34.68.230.202:443: i/o timeout indicates that requests to docker-auth.elastic.co — issued by the Docker Daemon — are being dropped.

Could you please share the output of docker image pull docker.elastic.co/elasticsearch/elasticsearch:8.7.1?

[grizzlycode]

Per your request here is the ouput of docker pull.

Output:

8.7.1: Pulling from elasticsearch/elasticsearch

8d2cded47a25: Pull complete

4eb7c3ccec8d: Pull complete

0f40ad4df92c: Pull complete

89732bc75041: Pull complete

84c1a116837f: Pull complete

8f67ce1599b4: Pull complete

22b3e1bb1f4e: Pull complete

aa7894424d88: Pull complete

68025dcc9227: Pull complete

2a782ddcb48b: Pull complete

Digest: sha256:bfd17b9ba48b247c7c93f10ed18350d289ebf4a06bc10b928357a1bb8f923e08

Status: Downloaded newer image for [docker.elastic.co/elasticsearch/elasticsearch:8.7.1](http://docker.elastic.co/elasticsearch/elasticsearch:8.7.1)

[docker.elastic.co/elasticsearch/elasticsearch:8.7.1](http://docker.elastic.co/elasticsearch/elasticsearch:8.7.1)

[antoineco]

[grizzlycode]

So after manually pulling down the elasticsearch image I tried docker compose up tls and it worked this time. However, when I tried docker compose up setup it failed.

error:
:' Connection to Elasticsearch failed. Exit code: 60

[antoineco]

Probably because the traffic that is supposed to stay internal to the elk local network is actually going over the proxy.

See my example at #865 (comment) to exclude Docker's networks with the noProxy option (172.16.0.0/12 should work).

[grizzlycode]

I have no_proxy setup for both client/daemon and it didn't work same error, "Exit code 60" I tried adding the proxy to the docker compose via args, it failed with same code. I also tried adding to the Dockerfile and that didn't work.

[antoineco]

It's noProxy not no_proxy, and only the client configuration is relevant for the issue you're facing.

Anyway, without the complete picture I doubt I can help any further. The question is about the usage of Docker and of some corporate proxy which I have no knowledge of. In both cases, docker-elk is out of the picture.

I also did some experiments locally, and both httpsProxy and noProxy were correctly propagated, so I would suggest troubleshooting at a deeper level (your Docker network, your IT support, ...).

[grizzlycode]

Ok so for my sitauton the config.json doesn't work. However, when I set the bash variables for http_proxy & https_proxy everything is now working. Closing issue as its now resolved.

[Bloopps]

@grizzlycode

Can you share your configuration? I'm in the same situation and I can't find a solution for using my proxy.
Thanks

[grizzlycode]

@Bloopps

I don't have access to that specific config currently. However, there are some components that need to be aware of a proxy if one is used in your environemnt. Such as:

• Docker
• Kibana (Fleet integrations)
• Elastic Agents

This can be difficult to troubleshoot without local IT support assisting as each network is different. Beware some networks have two proxies one for either http/https or perhaps just uses one for both.

Start with creating the proxy variables in Docker config and try to reach out to the internet. This can be done with any container that can reach out to the internet for testing purposes. If your other containers can reach the internet then your docker instance should be configured correctly. By setting at the Docker level that should allow any container to reach the internet now.

However, I have had issues specfiically with Fleet where I still had to set the proxy in Kibana.yml for it to work. So always check documentation and look to the individual config files to ensure the proxy is defined there as well if applicable.