Remove AWS ES indexes that exceed a configurable threshold.
If you are using something like logstash or zipkin that makes daily indexes then aws-es-index-reaper can be used to keep the index count in check.
The index list is retreived from elasticsearch using the GET settings API (https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-settings.html#indices-get-settings).
For example if the env var INDEX_PREFIX has the value logstash the query would be.
GET /logstash*/_settings
The indexes are sorted using the CreationDate attribute on each index and the oldest indexes above the threshold MAX_COUNT are removed.
Control the configuration by setting the following environmental variables
INDEX_PREFIX = Index name prefix used to locate matching indexes.
ES_URL = Url to interact with AWS ES domain. Do not include trailing slash.
MAX_COUNT = Any matching indexes over this count will be removed.
To deploy aws-es-index-reaper you can modify the reaper-scheduledjob.yaml to your needs and then run
kubectl apply -f reaper-scheduledjob.yaml
You can find the docker image here.