Elastalert, Elasticsearch and Kibana in Docker

[MinnMoto]

Greetings,

I'm working on some older RHEL servers that don't support more advanced Docker commands. I have Kibana and Elastalert working fine in docker, communicating to each other's containers. I am trying to load Elastalert on the same "network" in Docker.

My Elasticsearch install is called es01, Kibana kb01. For my Elastalert (el01) container, I have es_host=es01 and ex_port=9200 as environment parameters in my docker_compose.yaml, but also set this way in my elastalert.yaml. Unfortunately, Elastalert keeps using the hostname of my server that docker is hosted on to find Elasticsearch. I can see es01 from inside the el01 container, and query the es01:9200 endpoint successfully.

I was using jertel/elastalert-docker:latest for this test. I've tried jertel/elastalert2:2.1.2 now. Are there big differences?

Suggestions?

[MinnMoto]

Seems I've made some traction. Probably posting this got me thinking the right way. Rubber ducking?

Regardless, I am using the new jertel/elastalert2:2.1.2. I have setup in my docker_compose.yaml to find my config file and the rules folder.

elastalert@e72ca5656678:/opt/elastalert$ elastalert-test-rule rules/stage_db_errs.yaml
INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent.
            To send them but remain verbose, use --verbose instead.
**WARNING:elasticsearch:GET http://msplwb400:9200/ [status:N/A request:0.031s]**
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 169, in _new_conn
    conn = connection.create_connection(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/lib/python3.9/http/client.py", line 1257, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1303, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1252, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1012, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.9/http/client.py", line 952, in send
    self.connect()
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 200, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 181, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f78d69c5790>: Failed to establish a new connection: [Errno 111] Connection refused

It's still finding my docker host server msplwb400. Below is a my docker-compose.yaml file.

version: '3'

networks:
  elknet:
    driver: bridge
#    internal: true

services:
  el01:
    container_name: el01
    image: case.artifacts.company.com/docker-dev-virtual/jertel/elastalert2:2.1.2
    depends_on:
      - es01
    links:
      - es01
    networks:
      - elknet
    restart: unless-stopped
    logging:
      driver: "json-file"
      options:
        max-size: "2m"
        max-file: "2"
    environment:
      - es_host=es01
      - es_port=9200
    volumes:
      - /teneo_elk/docker/elastalert2/config.yaml:/opt/elastalert/config.yaml
      - /teneo_elk/docker/elastalert2/rules:/opt/elastalert/rules

  es01:
    image: case.artifacts.company.com/docker-dev-virtual/elastic/elasticsearch:7.6.2
    container_name: es01
    # uid=983(elasticsearch) gid=206(elasticsearch) groups=206(elasticsearch)
    user: "983"
    environment:
      #- cluster.initial_master_nodes=es01
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.type=single-node
      - bootstrap.memory_lock=true
      #- "ES_JAVA_OPTS=-Xms512m -Xmx512m -XX:-HeapDumpOnOutOfMemoryError"
      - "ES_JAVA_OPTS=-Xms4g -Xmx4g -XX:-HeapDumpOnOutOfMemoryError"
    ulimits:
      memlock:
        soft: "-1"
        hard: "-1"
    ports:
      - "9211:9200"
      - "9311:9300"
    expose:
      - 9200
    restart: unless-stopped
    logging:
      driver: "json-file"
      options:
        max-size: "2m"
        max-file: "2"
    volumes:
      - /teneo_elk/elk_data/elk762/elasticsearch/data:/usr/share/elasticsearch/data
      - /teneo_elk/elk_data/elk762/elasticsearch/logs:/usr/share/elasticsearch/logs
    networks:
      - elknet

and my config file

# The Elasticsearch hostname for metadata writeback
# Note that every rule can have its own Elasticsearch host
#es_host: elasticsearch.example.com
es_host: es01

# The Elasticsearch port
es_port: 9200

[jertel]

Does your rules/stage_db_errs.yaml file override the es_host value?

[MinnMoto]

Good call - yes. Prob don't need to define server in a rule, correct?

[jertel]

Usually not, unless you have one rule out of your collection that needs to monitor a different cluster from the others. My thought was perhaps you had es_host defined in that rule file and pointing to the wrong host.

[MinnMoto]

Thank you for your help. I am up and running now. Cheers! Dan

…

On Thu, Aug 12, 2021, 12:01 PM Jason Ertel ***@***.***> wrote: Usually not, unless you have one rule out of your collection that needs to monitor a different cluster from the others. My thought was perhaps you had es_host defined in that rule file and pointing to the wrong host. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#398 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACTR2PSEJ3FQU6ITGNFNH4LT4P45NANCNFSM5B7RE5FA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .