jaegertracing/jaeger-es-index-cleaner TIMEOUT variable ignored

[ykuksenko]

Describe the bug
Elasticsearch index cleaner times out at 10 seconds regardless of what the TIMEOUT environment variable is set to for the container.

This may also affect #2938 as that PR seems to copy what the non rollover cleaner is doing.

To Reproduce
Steps to reproduce the behavior:
The operation has to take more than 10 seconds for this to trigger. If I clean less data the error goes away.

$ docker run -it --rm --net=host -e ROLLOVER=false -e TIMEOUT=120 jaegertracing/jaeger-es-index-cleaner:latest 60 http://192.168.0.5:9200  
Removing jaeger-service-2021-03-02
Removing jaeger-service-2021-03-05
Removing jaeger-span-2021-03-03
Removing jaeger-service-2021-03-08
Removing jaeger-span-2021-03-05
Removing jaeger-span-2021-03-02
Removing jaeger-span-2021-03-06
Removing jaeger-service-2021-03-07
Removing jaeger-span-2021-03-08
Removing jaeger-span-2021-03-04
Removing jaeger-service-2021-03-10
Removing jaeger-span-2021-03-09
Removing jaeger-service-2021-03-03
Removing jaeger-service-2021-03-09
Removing jaeger-service-2021-03-04
Removing jaeger-service-2021-03-06
Removing jaeger-span-2021-03-10
Removing jaeger-service-2021-03-11
Removing jaeger-span-2021-03-07
Removing jaeger-span-2021-03-11
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/lib/python3.8/http/client.py", line 1347, in getresponse
    response.begin()
  File "/usr/local/lib/python3.8/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.8/http/client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/elasticsearch/connection/http_urllib3.py", line 251, in perform_request
    response = self.pool.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 637, in urlopen
    retries = retries.increment(method, url, error=e, _pool=self,
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 344, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.8/site-packages/urllib3/packages/six.py", line 686, in reraise
    raise value
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 597, in urlopen
    httplib_response = self._make_request(conn, method, url,
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 306, in _raise_timeout
    raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='192.168.0.5', port=9200): Read timed out. (read timeout=10)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/curator/actions.py", line 678, in do_action
    self.__chunk_loop(lst)
  File "/usr/local/lib/python3.8/site-packages/curator/actions.py", line 647, in __chunk_loop
    self.client.indices.delete(
  File "/usr/local/lib/python3.8/site-packages/elasticsearch/client/utils.py", line 153, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/elasticsearch/client/indices.py", line 296, in delete
    return self.transport.perform_request(
  File "/usr/local/lib/python3.8/site-packages/elasticsearch/transport.py", line 415, in perform_request
    raise e
  File "/usr/local/lib/python3.8/site-packages/elasticsearch/transport.py", line 381, in perform_request
    status, headers_response, data = connection.perform_request(
  File "/usr/local/lib/python3.8/site-packages/elasticsearch/connection/http_urllib3.py", line 263, in perform_request
    raise ConnectionTimeout("TIMEOUT", str(e), e)
elasticsearch.exceptions.ConnectionTimeout: ConnectionTimeout caused by - ReadTimeoutError(HTTPConnectionPool(host='192.168.0.5', port=9200): Read timed out. (read timeout=10))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/es-index-cleaner/esCleaner.py", line 117, in <module>
    main()
  File "/es-index-cleaner/esCleaner.py", line 53, in main
    delete_indices.do_action()
  File "/usr/local/lib/python3.8/site-packages/curator/actions.py", line 680, in do_action
    utils.report_failure(err)
  File "/usr/local/lib/python3.8/site-packages/curator/utils.py", line 181, in report_failure
    raise exceptions.FailedExecution(
curator.exceptions.FailedExecution: Exception encountered.  Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: ConnectionTimeout caused by - ReadTimeoutError(HTTPConnectionPool(host='192.168.0.5', port=9200): Read timed out. (read timeout=10))

Expected behavior
Indexes deleted without error.

Actual behavior
Currently the indexes are deleted correctly but the timeout error shows up.
Rerunning the same command shows no indexes are to be deleted.
So this is mostly a cosmetic error.

Screenshots

Version (please complete the following information):

• OS: Linux
• Jaeger version: 1.16.0
• Deployment: Docker

What troubleshooting steps did you try?
Tried multiple version of the jaeger-es-index-cleaner:

jaegertracing/jaeger-es-index-cleaner               latest         4dcabe316894   12 hours ago    126MB
jaegertracing/jaeger-es-index-cleaner               <none>         10585b89206d   6 days ago      126MB
jaegertracing/jaeger-es-index-cleaner               1.22.0         f7cd0bdf56a9   2 months ago    122MB

Additional context
Elasticsearch 6.8.7 is being used here. The cleaner version in theory should not matter as it is just an interface to curator.

[pavolloffay]

Any suggestion how to fix this?

The TIMEOUT is passed to https://curator.readthedocs.io/en/latest/actionclasses.html#deleteindices correctly

jaeger/plugin/storage/es/esCleaner.py

Line 52 in 7636530

delete_indices = curator.DeleteIndices(ilo, master_timeout=timeout)

[pavolloffay]

please reopen if the issue is still there. Now we have new index cleaner implementation based on golang.