[inputs.mongodb] Telegraf crash on init if conn to MongoDB fails

[papey]

Relevent telegraf.conf

/etc/telegraf/telegraf.d/mongo.conf

[[inputs.mongodb]]
  servers = ["mongodb://127.0.0.1:27017/?connect=direct"]
  gather_perdb_stats = false

  fieldpass = ["deletes_per_sec", "flushes_per_sec", "getmores_per_sec", "inserts_per_sec", "updates_per_sec", "queries_per_sec", "open_connections", "net_in_bytes", "net_out_bytes", "repl_lag", "state"]

System info

Telegraf 1.20.2 (git: HEAD f721f53), Debian 10.11

Docker

No response

Steps to reproduce

1. Add a dedicated conf for MongoDB like the one above (to simulate a failed conn, just change the port number, for example)
2. Restart Telegraf

Expected behavior

At least a log message to STDERR, on init, about a failing conn to a MongoDB server (with no retry mechanism)

Actual behavior

Telegraf crashes :

Nov 08 17:32:09 test-mongo-telegraf telegraf[3423]: 2021-11-08T16:32:09Z E! [telegraf] Error running agent: could not initialize input inputs.mongodb: unable to connect to MongoDB: server selection error: context deadline exceeded, current topology: { Type: Single,
Nov 08 17:32:09 test-mongo-telegraf systemd[1]: telegraf.service: Main process exited, code=exited, status=1/FAILURE

Additional info

My hypothesis is that we need to rethink some logic in the Init function in the mongodb input code

[powersj]

Hi,

I want to be certain I understand your desire for this bug. Are you after more log messages? The message you provided shows that it failed to connect to MongoDB already.

Are you hoping for some sort of retry?

Or something else?

Thanks!

[papey]

Thanks for your feedback and sorry if it was unclear,

I want to be certain I understand your desire for this bug. Are you after more log messages? The message you provided shows that it failed to connect to MongoDB already.

Yes, but since the error is returned in the Init() func of the mongodb package, Telegraf just exit with an error exit status (equal to 1, see the log above).

From that state, lets imagine I have other metrics collected (lets say from other inputs for example). If one of the MongoDB servers listed is not reachable for some reason, I want Telegraf to continue running and collect metrics from other inputs.

I see to solution (the simple one [1] and the complex one [2]):

1. just log the error without failing and continue running

or

2. log error and add a recheable state in the Server struct, and implement a retry mechanism

(I am currently working on code for solution [2])

[papey]

See #10086 for implementation details and discussion.

[papey]

Can you please provide some feedback about the PR ? Thanks ❤️