TLS CA too many open files - decisionlog upload

[iamatwork]

OPA v0.37.2
config:

services:
decisionlogger:
url: https://XXX
tls:
ca_cert: /opt/XXX/certs/myCA.crt
decision_logs:
reporting:
min_delay_seconds: 15
service: decisionlogger

When uploading decisionlogs fails (with authorization error), after a few tries, OPA goes bonkers and tries to upload the logs without any delay (configured minimum is 15s). This fails with " /opt/XXX/certs/myCA.crt: too many open files"

{"level":"error","msg":"log upload failed: Post "https://XXX:9200/opa_log/_doc\": dial tcp: lookup XXX on XXX: no such host.","plugin":"decision_logs","time":"2022-03-25T13:24:54Z"}
{"level":"debug","msg":"Waiting 0s before next upload/retry.","plugin":"decision_logs","time":"2022-03-25T13:24:54Z"}
{"headers":{"Content-Encoding":["gzip"],"Content-Type":["application/json"],"User-Agent":["Open Policy Agent/0.37.2 (linux, amd64)"]},"level":"debug","method":"POST","msg":"Sending request.","time":"2022-03-25T13:24:54Z","url":"https://XXXX"}
{"headers":{"Content-Length":["544"],"Content-Type":["application/json; charset=UTF-8"],"Www-Authenticate":["Basic realm="security" charset="UTF-8"","Bearer realm="security"","ApiKey"]},"level":"debug","method":"POST","msg":"Received response.","status":"401 Unauthorized","time":"2022-03-25T13:24:54Z","url":"https://XXX:9200"}
{"level":"error","msg":"log upload failed, server replied with not authorized.","plugin":"decision_logs","time":"2022-03-25T13:24:54Z"}
{"level":"debug","msg":"Waiting 0s before next upload/retry.","plugin":"decision_logs","time":"2022-03-25T13:24:54Z"}
{"level":"error","msg":"log upload failed: open /opt/XXX/certs/myCA.crt: too many open files.","plugin":"decision_logs","time":"2022-03-25T13:24:54Z"}
{"level":"debug","msg":"Waiting 0s before next upload/retry.","plugin":"decision_logs","time":"2022-03-25T13:24:54Z"}
{"level":"error","msg":"log upload failed: open /opt/XXX/certs/myCA.crt: too many open files.","plugin":"decision_logs","time":"2022-03-25T13:24:54Z"}

[ashutosh-narkar]

Looking at your config I would have first expected an error like

error: config error: reporting configuration missing 'max_delay_seconds' in decision_logs

Looks like we are not properly setting the retry value in the delay calculation which causes this. We'll add a fix soon. Thanks for reporting.