TelegraMQ

Simple RabbitMQ and Telegram Integration with Python

What is Message Queueing?

The use of Message Queues provides a way for parts of the application to push messages to a queue asynchronously and ensure they are delivered to the correct destination. To implement message queuing, a message broker like RabbitMQ is a good option. The message broker works as a Middleware providing temporary message storage when the receiving service is busy or disconnected.

The basic flow in a Message Queue is:

Producer - Generates and send the message to the Broker

Broker (Message Queue) - Receive and store the message, waiting to being consumed.

Consumers - One or more applications querying messages in one or more queues stored in the broker

Handling communication with brokers

A message broker acts as a middleman for the microservices, receiving messages from one application (producers) and handing them over to others (consumers) to do the job. For example; with RabbitMQ message broker, messages are not published directly to a queue. Instead, the producer sends a message to an exchange. The job of an exchange is to accept messages from the producer applications and route them to the correct message queues. The messages stay in the queue until the consumer handles them and removes them.

There are a couple of different message brokers to choose from. When choosing between brokers, you should try to nail down your requirements. RabbitMQ and Apache Kafka are two open-source message brokers. You can read about the main difference between them in this comparison: "When to use RabbitMQ or Apache Kafka" https://www.cloudamqp.com/blog/when-to-use-rabbitmq-or-apache-kafka.html.

RabbitMQ enables asynchronous processing, meaning that it allows you to put a message in a queue without processing it immediately.

RabbitMQ is therefore ideal for long-running tasks or blocking tasks, allowing web servers to respond quickly to requests instead of being forced to perform computationally intensive tasks on the spot. RabbitMQ simply stores messages and passes them to consumers when ready.

RabbitMQ is a reliable open source message broker. It has been on the market since 2007 and became a part of Pivotal software 2013. It's continuously updated and improved upon. RabbitMQ has a strong community and highly active core team that produce additional features, improvements and handy plugins. The license of RabbitMQ has never changed (Nov 2019).

RabbitMQ supports several standardized protocols such as AMQP, MQTT, STOMP, etc. where it natively implements AMQP 0.9.1. The ability of RabbitMQ to support different standardized message protocols means that it can be used in many different scenarios and it allows you to replace your RabbitMQ broker with any AMQP based broker.

RabbitMQ is used by a large number of companies within various industries and is used and trusted by large companies (Zalando, WeWork, Wunderlist, Bloomberg, and more). All relying on a microservice based architecture.

RabbitMQ is user-friendly, and by following these RabbitMQ best practices, it is easy to tweak the configurations to suit the intended purpose. RabbitMQ is written in Erlang and is the world’s most deployed open-source message broker, meaning that it’s a well-tested, robust broker.

The RabbitMQ broker is scalable and flexible. Your team only needs to maintain the producers and the consumers sending and receiveing messages to/from the queue. Under heavy load, if the queue grows larger, the standard reaction is to add more consumers and parallelize the work. This is a simple and effective method of scaling.

Benefits of Message Queues

In modern cloud architecture, applications are decoupled into smaller, independent building blocks that are easier to develop, deploy and maintain. Message queues provide communication and coordination for these distributed applications.

Message queues can significantly simplify coding of decoupled applications, while improving performance, reliability and scalability. You can also combine message queues with Pub/Sub messaging in a fanout design pattern.

Better Performance

Message queues enable asynchronous communication, which means that the endpoints that are producing and consuming messages interact with the queue, not each other. Producers can add requests to the queue without waiting for them to be processed. Consumers process messages only when they are available. No component in the system is ever stalled waiting for another, optimizing data flow.

Increased Reliability

Queues make your data persistent, and reduce the errors that happen when different parts of your system go offline. By separating different components with message queues, you create more fault tolerance. If one part of the system is ever unreachable, the other can still continue to interact with the queue. The queue itself can also be mirrored for even more availability.

Granular Scalability

Message queues make it possible to scale precisely where you need to. When workloads peak, multiple instances of your application can all add requests to the queue without risk of collision. As your queues get longer with these incoming requests, you can distribute the workload across a fleet of consumers. Producers, consumers and the queue itself can all grow and shrink on demand.

Simplifed Decoupling

Message queues remove dependencies between components and significantly simplify the coding of decoupled applications. Software components aren’t weighed down with communications code and can instead be designed to perform a discrete business function.

Message queues are an elegantly simple way to decouple distributed systems, whether you're using monolithic applications, microservices or serverless architectures.

Now that you know more about Message Queues let's do a simple example using Python, Python Pika, RabbitMQ and Telegram.

In this POC we gonna follow the steps bellow:

Create 2 Ubuntu 20.04 VPcs

Install Python, Pika and RabbitMQ in the Server 2, then create our receiver for queue the message then send to the telegram bot.

Install Python and Pika in the Server 1 (Sender), then create our application for sending message to the queue.

In the fisrt step we gonna use server 2 as a Broker and Consumer, sending our message locally via localhost to our Telegram Bot.

In the second step, we gonna allow our guest user for sending our message remotely to our Telegram Bot.

Let's start updating our Ubuntu OS.

root@MQBroker:~# sudo apt update && sudo apt upgrade

Our installation already have python3.8 So let's install python-pika.

Upload rabbitmq.conf, rabbitmq-install.sh, receive-message.py and send-message_localhost.py to your server.
Let's install RabbitMQ now. Just execute the bash script rabbitmq-install.sh.

root@MQBroker:~# sh rabbitmq-install.sh

After the script finishes, our RabbitMQ Server is already installed and running.

You can check running the comand:

root@MQBroker:~# sudo systemctl status rabbitmq-server

If you run receive-message.py now, you gonna start our consumer, and will connect to our queue waiting for some message.

You can test if is working, opening another ssh session then running send-message_localhost.py.

You can also stop the receiver application and send some messages to the broker.

Here we've sent 3 messages with the receiver down.

Now if you run again the receiver, it will print all the 3 messages stored in the queue.

Ok, we have our Broker working locally, but doesn't make sense right? We want it receiving messages from one application in another server and for this, we need to create the rabbitmq.conf setting some permission for our user guest connecting the broker remotely.

Let's move to the next step installing Python and Pika in the server MQSender. You can repeat the same steps we followed for the Broker, we just don't need to install RabbitMQ in the MQSender.

Now you can copy the python script send-message_remote_host.py to your MQSender server and put your MQBroker Server IP in my case is 78.111.85.11.

If you run the script now you'll get an ERROR. pika.exceptions.ProbableAuthenticationError: (403, 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.')

This is because we don't have the RabbitMQ config file with the Access Control configuration in the MQBroker server, permiting guest user to connect outside the loopback (localhost).

You can upload the rabbitmq.conf here to your MQBroker server in the directory /etc/rabbitmq

Now restart the RabbitMQ service.

Run the receive-message.py again to start our consumer application.

Go to MQSender and run send-message_remote_host.py again and see what's happen in your MQBroker/Consumer.

Our RabbitMQ is working queueing the messages and being consumed by our application.

The advantage here, is that doesn't matter if you have the sender application in Python, Php, Java, etc and the consumer in different language from the sender, as long as they are connected to the message broker, they will comunicate with each other sending and receiving the message.

There's a lot of use for Message Queue, like avoiding transactions being made in the Database, saving processing power and storage in your Database Server. With some configurations the Message Broker will manage all the message life cycle, saving the message to be consumed another time following your TTL configuration, or just deleting. That's your call.

Now, what about receiving our message in Telegram? You can also use Telegram Api Webhook, then users can send one command in Telegram Bot, Telegram Webhook send the message to Message Queue, then the consumer application receive the message. Remembering, when we say message, can be commands, files, etc. The sky is the limit.

Let's just use a Telegram Bot receiving our message from our Broker/Consumer.

Create your Telegram Bot and get the TOKEN and your personal chatid. (Sorry, I'll not explain these steps here)

Go to your MQBroker server and make sure you have Pip and Python requests module.

Edit receive-message.py and uncomment line 22 and replace the token with your Bot Token and chatid with your personal chatID.

PS: Our body here is byte so you need to convert to str. body.decode('utf-8')

Save receive-message.py, start your Telegram Bot to receive messages, run receive-message.py to start consuming and routing the messages in the queue. Now go to your MQSender server and run send-message_remote_host.py and watch the magic! :))