Curator shouldn't die when there is no data to delete

[IngaFeick]

To submit a bug or report an issue

I am running curator in Docker. The first action in my config is trying to delete data for which there are currently no indices in my elasticsearch. Curator considers this an error and executes the same action repeatedly, then dies without processing the rest of my actions.yml.

Expected Behavior

I expect curator to log that the action didn't delete data, and then proceed with the other actions.

Actual Behavior

Curator repeatedly tried to perform the "faulty" action no. 1 and then exited, without processing the remaining actions (actions 2 + 3).

Steps to Reproduce the Problem

actions.yaml:

actions:
  1:
    action: delete_indices
    description: delete indices older than 30 days (based on index name)
    options:
      timeout_override:
      continue_if_exception: False
    filters:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 30
      exclude:
    - filtertype: pattern 
      kind: prefix 
      value: some-index-i-want-to-keep
      exclude: True

  2:
    action: delete_indices
    description: delete indices older than 2 months. This one's for the indices with month based names. 
    options:
      timeout_override:
      continue_if_exception: False
    filters:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m'
      unit: months
      unit_count: 2
  3:
    action: delete_indices
    description: delete foobar indices older than 3 days.
    options:
      timeout_override:
      continue_if_exception: False
    filters:
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 3
    - filtertype: pattern 
      kind: prefix 
      value: foobar

config.yml:

client:
  hosts:
    - 10.2.3.226
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

Specifications

• Version: 5.5.4
• Platform: Linux
• Subsystem: Debian

Detailed Description

Curator got stuck in this loop:

2018-11-12 12:52:05,702 INFO Preparing Action ID: 1, "delete_indices" 
2018-11-12 12:52:05,712 INFO Trying Action ID: 1, "delete_indices": delete indices older than 30 days (based on index name) 2018-11-12 12:52:17,250 ERROR Unable to complete action "delete_indices". No actionable items in list: <class 'curator.exceptions.NoIndices'> 
2018-11-12 12:52:41,261 INFO Preparing Action ID: 1, "delete_indices" 
2018-11-12 12:52:41,271 INFO Trying Action ID: 1, "delete_indices": delete indices older than 30 days (based on index name) 2018-11-12 12:52:48,612 ERROR Unable to complete action "delete_indices". No actionable items in list: <class 'curator.exceptions.NoIndices'> 
2018-11-12 12:53:10,402 INFO Preparing Action ID: 1, "delete_indices" 
2018-11-12 12:53:10,412 INFO Trying Action ID: 1, "delete_indices": delete indices older than 30 days (based on index name) 
2018-11-12 12:53:18,500 ERROR Unable to complete action "delete_indices". No actionable items in list: <class 'curator.exceptions.NoIndices'> 
2018-11-12 12:53:42,567 INFO Preparing Action ID: 1, "delete_indices" 
2018-11-12 12:53:42,576 INFO Trying Action ID: 1, "delete_indices": delete indices older than 30 days (based on index name) 
2018-11-12 12:53:49,621 ERROR Unable to complete action "delete_indices". No actionable items in list: <class 'curator.exceptions.NoIndices'> 

and then died without any further logs.

[untergeek]

This is extremely unusual. I wonder if it is Docker-specific. It should terminate. It should not loop. It has never looped in any other environment I’ve run Curator in. In fact, my CI tests would have caught that. Do you have a wrapper script that is simply calling Curator repeatedly?

There is already a flag that should address the empty list issue, and that is ignore_empty_list

[IngaFeick]

I can imagine that it is specific to Docker or Nomad rather, yes, as I have not seen this behaviour before either. My Dockerfile is simply this:

FROM "artifactory.internal:9090/perf/curator:5.5.4"

COPY "conf/foo/config.yml" /curator/config.yml
COPY "conf/foo/actions.yml" /curator/actions.yml

CMD ["--config", "/curator/config.yml", "/curator/actions.yml"]

I have seen that Nomad has restarted the image 2 times, making it 3 runs in total and I was wondering if that messed up the logs but I see the error message 4 times in there, so the numbers don't match (assuming that our Nomad is not missing logs - I'm suspecting something like this).

I will try that ignore_empty_list option, thank you. Have you considered making this the default behaviour? I think it might be more intuitive.

Thank you lots!

[untergeek]

I considered making it the default behavior, but there are times when an empty list is not expected. It should be an error in such cases. The flag allows you to bypass it.

[IngaFeick]

I'm closing the ticket because I assume that the repetetiveness of the logs was maybe/probably caused by Nomad, and the flag is going to solve the other issue. Thank you!