rest: logging does not work correctly in production mode

[audrium]

Currently on production mode installation there are some issues with rest-api logging:

1. It is only working for limited amount of endpoints. E.g. /ping is not being logged while /api/you logged as expected

2. There are lots of deprecation warnings in the logs like this:

<unknown>:1: DeprecationWarning: invalid escape sequence \[

2021-11-12 16:01:59,147 | py.warnings | uWSGIWorker2Core0 | WARNING | <unknown>:1: DeprecationWarning: invalid escape sequence \[

which are repeated for almost 1.5k lines after application starts. This could be due to outdated invenio dependencies. However it's not trivial to upgrade it because we have a conflicting celery dependency (from outdated kombu version which requires celery<5.0.0 pinning).

3. Logging level set in reana-commons is not respected: logger: basic configuration does not work because of the invenio-logging handlers #298

4. It seems that factory.py and create_app() are only being used in reana-admin and scheduler (defined here). Needs a deeper investigation, but if that's the case there is no point in registering the blueprints there. Looks like invenio-base is taking care for starting up the application. To reproduce try to:

   • comment/remove factory.py and app.py
   • reana-dev docker-build -c . -b DEBUG=1
   • reana-dev kind-load-docker-image -c .
   • kill the old pod and inspect the new one - works as expected
   • it does not work while redeploying the cluster because it can't start flask reana-admin create-admin-user

[mvidalgarcia]

After some research, I've discovered that we're only printing 4xx and 5xx logs when using uwsgi. IIRC this was our decision, perhaps to avoid too much output and thus be able to detect errors easier. Should we keep it like this? @tiborsimko

1. It is only working for limited amount of endpoints. E.g. /ping is not being logged while /api/you logged as expected

Probably you were not logged in so the HTTP code was 401.

[tiborsimko]

Good catch! I think printing everything is a better default, and if that is really too verbose, let's allow admins to change it easily.
This calls for making the behaviour configurable via Helm values...