Use redis as Airflow broker

[GiancarloFusiello]

In reference to libero/publisher#247

Adds redis image to docker-compose.yml and update config files to use redis broker.

[diversemix]

Intrigued to know why we are using redis as a message bus? I read an interesting article here: https://medium.com/wbaa/airflow-ha-environment-c60ddca825a9 .... I'm guessing its because its quick and easy to setup?

[giorgiosironi]

The name of the docker-compose service needs changing as a Message Bus enables separate applications to integrate, whereas this broker is internal to the jats-ingester application to provide a queue of jobs for workers to pick up.

[giorgiosironi]

Intrigued to know why we are using redis as a message bus?

(the answer then being, we should not be using redis as a message bus)

[giorgiosironi]

I assume this should be private to Celery, as we don't write or read values from the DAGs code for example. Possibilities for a service name?

• celery-broker (not very precise because brokers usually have their own protocol like AMQP, this is just part of the broker concept)
• celery-broker- backend (matches db as that doesn't mention postgres, and Celery's terminology)
• celery-broker-redis (to communicate what this is)

[GiancarloFusiello]

name updated to airflow-broker

[giorgiosironi]

https://medium.com/wbaa/airflow-ha-environment-c60ddca825a9

Not targeting High Availability at the moment as that would require redundancy for all containers and multiple nodes, plus unclear if it's a requirement of Libero demo deployments.

[giorgiosironi]

Trying to verify how values are cleared from Redis, not sure if we need to setup a TTL for them.

[diversemix]

Trying to verify how values are cleared from Redis, not sure if we need to setup a TTL for them.

@petereast is the in-house redis expert... my understanding is that its up to the app adding the data to set the TTL

[giorgiosironi]

My current understanding is that this PR should stop the kombu_* tables in the Postgres database from growing as Redis would substitute for that. Collecting data while running locally (very valuable to have the fake S3 implementation which allows this).

[GiancarloFusiello]

@giorgiosironi @diversemix My original thought was to use RabbitMQ as it is likely (but not defined yet) we may need to implement a message bus for sending messages/events/commands between Libero Publisher services. Therefore, why not use the same message bus to handle messages for celery.

My discussion with @giorgiosironi before actioning this task lead me to believe that this is an area of uncertainty and, as a result, the decision to use redis at this time was based on the ease of setup and maintenance.

Perhaps the task needs further discussion and a separate issue/ticket?

[diversemix]

@GiancarloFusiello @giorgiosironi ... my preference is to raise a ticket too... and keep this ticket moving forward with what was intended 😉

[giorgiosironi]

Just verified that this is working well.

On the db side, the kombu_* tables do not appear in a brand new airflow-db database. However in an existing database they are not removed so they will need an explicit deletion in the environments like demo.

On the Redis side, there is a small space of Redis keys (4 to 6) such as _kombu.binding.celery.pidbox and unacked that is maintained by the scheduler and the workers. The naming pattern (e.g. unacked) suggests that this Redis instance will be private to Celery and can't be used by anything else (which is fine as long as we know it). The memory being used by Redis here is in the order of a few MBs so incredibly low.

[GiancarloFusiello]

Adding more information. If I'm reading the following documents correctly, it seems that Kombu uses redis pub/sub by default. If this is the case then data sent (published) to redis will not be retained when pushed to subscribers:
https://docs.celeryproject.org/projects/kombu/en/stable/introduction.html?highlight=ttl#transport-comparison
https://docs.celeryproject.org/projects/kombu/en/stable/reference/kombu.transport.redis.html?highlight=redis#kombu.transport.redis.Channel.fanout_patterns

[giorgiosironi]

raise a ticket too

Not sure what would be the title of a follow-up, can you suggest one? 🤔

[giorgiosironi]

Empirically works as intended, sorting out the naming before merge

[diversemix]

raise a ticket too

Not sure what would be the title of a follow-up, can you suggest one?

"Decide what Message Bus to use in Libero"

[giorgiosironi]

Is that libero/community#10 (outdated in requirements but not in technology understanding)?

[giorgiosironi]

do we want to change the result_backend setting as well? Not sure what the impact would be, and The use of a database is highly recommended

[GiancarloFusiello]

No. Using a relational database is best for this.

[GiancarloFusiello]

Some log messages I've seen:

airflow-broker_1     | 1:M 16 Sep 2019 08:55:18.672 # Server initialized
airflow-broker_1     | 1:M 16 Sep 2019 08:55:18.672 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
airflow-broker_1     | 1:M 16 Sep 2019 08:55:18.672 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.

[GiancarloFusiello]

Just seen these also... will address separately:

airflow_worker_1     |  * Environment: production
airflow_worker_1     |    WARNING: This is a development server. Do not use it in a production deployment.
airflow_worker_1     |    Use a production WSGI server instead.

[diversemix]

Is that libero/community#10 (outdated in requirements but not in technology understanding)?

I got Peter to comment on this with the state of where we are with PubSweet/reviewer 😄

[giorgiosironi]

Feel free to open tickets and tag them with jats-ingester for issues to be dealt with separately. Approved.
https://github.com/libero/sample-configuration needs a similar change.

[GiancarloFusiello]

Feel free to open tickets and tag them with jats-ingester for issues to be dealt with separately.

Will create a new ticket regarding the airflow worker logs.

https://github.com/libero/sample-configuration needs a similar change.

As it's related, i'll use the same pattern of linking the main issue to the PR.