Fix SSL support for MongoDB and RabbitMQ under Python 3.x

[Kami]

This pull request fixes #4832 SSL support for MongoD and RabbitMQ when running under Python 3.6.

Background / Details

When running under Python 3 and enabling SSL support for MongoDB, st2api and st2auth would fail with a cryptic error:

2019-12-15 08:08:45,891 140651884441824 INFO (unknown file) [-] Connecting to database "somedatabase" @ "mongodb-host:27017" as user "someuser".
2019-12-15 08:08:45,892 140651884441824 WARNING (unknown file) [-] Retry on ConnectionError - Cannot connect to database default :
maximum recursion depth exceeded while calling a Python object

After some more digging in and manual instrumentation, I managed to track down the original exception / root cause - https://gist.github.com/Kami/ea8e63cdc539fd879fff41271969d650.

The root cause was an SSL error which happened because st2api and st2auth didn't perform eventlet monkey patching early enough. They performed monkey patching after some other module (likely mongoengine or pymongo) already imported ssl which has undefined behavior and won't work.

Proposed Fix

This pull request fixes the issue by making sure we perform eventlet monkey patching as early as possible (before any other modules are imported) inside st2api and st2auth service.

In addition to that, I made another change by setting serverSelectionTimeoutMS MongoClient option to 5 seconds.

It defaults to 30 seconds which means if there is a SSL related connection error (e.g. handshake failed), exception won't be raised until 30 seconds has passed.

Keep in mind though that in such scenarios, only "connection closed" exception will be returned to the user and for the actual root cause, user will need to check the mongo server logs (that's the limitation of the client / server and nothing we can do).

This will also provide a better user-experience because previously in many scenarios our code would wait 30 seconds for this timeout to be reached before propagating the connection error.

Affected Components / Services

Based on my digging it, this issue only affected st2api and st2auth because other services already performed eventlet monkey patching as early as possible.

This also explains why user reported that st2-register-content worked in #4832 (we intentionally don't perform any monkey patching there so it works fine).

Another thing worth keeping in mind is that we have two entry points for st2api and st2auth - WSGI one (for production gunicorn deployments) and the direct eventlet WSGI server one (for local testing and development).

Both entry points were affected so I needed to fix both.

Configuration

For completeness, here is the st2.conf and mongod.conf configuration I used:

st2.conf:

...
[database]
ssl = true
host = 127.0.0.1
ssl_cert_reqs = none
username = stackstorm
password = ...
...

mongod.conf:

...
net:
  port: 27017
  bindIp: 127.0.0.1
  ssl:
    mode: requireSSL
    PEMKeyFile: /home/ubuntu/mongodb.pem
    CAFile: /home/ubuntu/rootCA.pem
    allowConnectionsWithoutCertificates: true  # NOTE: If this option is not set, client also needs to specify client server otherwise connection / auth will fail
...

[Kami]

For "just in case", I will also make this configurable via st2.conf.

[arm4b]

Thanks for investigating and fixing this! 👍