Engine not able to connect to elasticsearch

[seihtam]

The engine container is not able to connect to the elasticsearch container.
I confirmed that the problem was not any of the bugs described in https://github.com/maliceio/malice/blob/master/docs/KnownBugs.md.

Connecting to elasticsearch from my host OS works:

>> curl http://localhost:9200/
{
  "name" : "Qd_Kpeg",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "_TRWXiqyTWi7tVGwpV01oQ",
  "version" : {
    "number" : "6.5.0",
    "build_flavor" : "oss",
    "build_type" : "tar",
    "build_hash" : "816e6f6",
    "build_date" : "2018-11-09T18:58:36.352602Z",
    "build_snapshot" : false,
    "lucene_version" : "7.5.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

Following command fails with connect: connection refused:

>> docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v `pwd`:/malice/samples malice/engine scan putty.exe
#### File
| Field  | Value                                                            |
| ------ | ---------------------------------------------------------------- |
| Name   | putty.exe                                                        |
| Path   | putty.exe                                                        |
| Size   | 854.1kB                                                          |
| MD5    | 54cb91395cdaad9d47882533c21fc0e9                                 |
| SHA1   | 3b1333f826e5fe36395042fe0f1b895f4a373f1b                         |
| SHA256 | 7afb56dd48565c3c9804f683c80ef47e5333f847f2d3211ec11ed13ad36061e1 |
FATA[0001] scan cmd failed to store file info: failed to connect to database: failed to ping elasticsearch: Get http://localhost:9200/: dial tcp 127.0.0.1:9200: connect: connection refused

As a workaround adding --network="host" to the command above makes it work as expected.

Docker version:

Docker version:
Client:
 Version:           18.09.0
 API version:       1.39
 Go version:        go1.10.4
 Git commit:        4d60db4
 Built:             Wed Nov  7 00:49:01 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.0
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       4d60db4
  Built:            Wed Nov  7 00:16:44 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Docker info (with some info removed):

Docker info:
Containers: 6
 Running: 1
 Paused: 0
 Stopped: 5
Images: 26
Server Version: 18.09.0
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: c4446665cb9c30056f4998ed953e6d4ff22c7c39
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-39-generic
Operating System: Linux Mint 19
OSType: linux
Architecture: x86_64
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

WARNING: No swap limit support

[blacktop]

If I understand this correctly, this was fixed when you added: --network="host" ?

I could be that things have changed with the 🆕 docker that was just released?

[seihtam]

Yes that is correct. From my host OS curl http://localhost:9200/ returned elasticsearch information and to confirm that the engine docker couldn't connect to elasticsearch i opened a shell inside that container and ran wget http://localhost:9200/ which resulted in connection refused.
So i attempted giving the engine docker access to the "host" network and that fixed the problem. Though this is a workaround since I would rather not give the docker container access to the "host" network.

[blacktop]

So docker shouldn't talk to elasticsearch on localhost it should talk to it on http://elasticsearch:9200 because it is linked into the container with --link ?

I'll try this in docker as well and check.

[seihtam]

Yes that makes sense.
By the way I will follow up on all the other issues i opened this Thursday. Thanks for the quick responses :)

[seihtam]

I forgot to mention this, but it doesn't change anything about the way the issue should be fixed. For the workaround to work the elk stack will also have to be started with --network="host":
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock --network="host" malice/engine elk

[ziqiuwendan]

I got same problem，add --network="host", it work as expected

[blacktop]

I'll add this to the docs in the README thank you!