MQTT is a communication protocol based on Publish/Subscribe model. It has the following features:
- Simple and easy to implement
- QoS support for complex device network environment.
- Lightweight and designed specially for unstable network connections and bandwidth saving
- Data irrelevant
- Continuous session awareness
Those characteristics make it suited for IoT applications, and become one of the most popular protocols in IoT domain.
In a typical MQTT Publish/Subscribe architecture, there are publishers, subscribers, and a broker in the system. A publisher sends a message to the broker on a specific topic and then the broker sends that message to all subscribers which are subscribed to the same topic.
As a broker is the central component in an MQTT based IoT application, it plays an important role in the scalability and availability of the application. There have been many research papers and technical writeups on the performance evaluation and comparison of various MQTT brokers. However, the existing benchmarks often measure particular factors only, with scenarios not matching the realistic large-scale IoT applications. Through development and customer support experience of the past decade, we have learnt that many factors impact MQTT broker performance.
For one thing, MQTT protocol states many features improving reliability and security, which bring overhead cost on broker side as well. Some of the features are:
- QoS
- Keep Alive
- Clean Session
- Retain
- Topic wildcards
- Authentication method
- TLS authentication enablement
In addition, different MQTT versions suggest different functions. For example, shared subscription is supported in MQTT 5.0 but not 3.1.1 or 3.1.
For another thing, MQTT brokers deployment ways influence the performance evaluation. The brokers can be deployed on single node, or in a cluster way. If it is deployed in a cluster way, it can be put behind a load balancer or not. To be comparable, the brokers should be deployed in the similar ways.
What is more, environment conditions like hardware, OS, system and network configuration also impact the evaluation result. Some of the important conditions are:
- receive buffer size
- network round trip time
The rapid development of IoT applications brings new challenges. As a hub of devices and data, the capability of the MQTT broker to support massive device connectivity and data throughput is critical to the IoT business. Consequently, a thorough evaluation on its performance is important in the full IoT solution. The intent of this work is suggesting metrics and evaluation criteria, presenting real-world scenarios and use cases, and in the future, providing a tool that fully compatible with the benchmark suite. To this end, we hope to establish a practical base for benchmarking MQTT brokers, iteratively optimize its coverage, and help users find the most suitable MQTT broker for their needs and workloads.
To measure performance of an MQTT broker, we need to define metrics, which can be looked at in a variety of use cases. Two primary perspectives of measurement are considered. One focuses on the computing resources perspective. The other focuses on the capabilities provided by the MQTT broker itself, and divides into three detailed aspects - scalability, availability, and latency, as suggested in Google “Cloud Pub/Sub” product guide. It should be noticed that different aspects may contradict each other under different circumstances. To achieve different evaluation purposes, user needs to pick the suitable metrics.
| Computing resources metrics | Broker capability metrics | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Scalability the ability to handle increases in load |
Availability the ability to handle problems and failures |
Latency the amount of time to deliver a published message to a subscriber |
|||||||||||||
| CPU load and usage | Memory usage | Disk I/O rate | Packets receiving and sending rate | Rate of new established connections | Number of concurrent connections | Number of publishers | Number of subscribers | Number of topics | Size of messages | Rate of messages published | Rate of messages subscribed | Success rate | Reconnection delay | Average latency time | 90th percentile latency time |
A benchmark tool then is used to simulate publishers and/or subscribers based on use cases. It first generates dataset for benchmark, and loads the dataset to construct a series of MQTT requests, then executes to perform publishing and subscribing. The tool should work in a consistent way regardless of what the MQTT broker is. As many practical use cases involve large number of publishers/subscribers, the tool should also be implemented in a sophisticated way in order to generate desired large-scale load.
As part of the benchmark submisison, a full disclosure report is suggested. The intent of the report is to simplify comparison between results of different brokers, and to provide enough information so other users can replicate the results of this benchmark. The report should identify the benchmark tool it uses, explain the detailed use cases, and list the hardware, OS and network conditions to perform the benchmark. It should also cover comparison data regarding the measurement metrics, both including computing resources perspective and broker capability perspective.
By analyzing realistic requirements from our customers, we categorize the benchmark into the following scenarios:
- Connection
- Fan-out
- Point-to-point
- Fan-in
- Composite
Next, we will describe each scenario and give detailed use cases for each scenario. To meet different user needs, different use cases are demanded. For example, some of them can be used for basic verification purpose, while some of them aim for enterprise level verification. Therefore, we group use cases of the same scalability level into one set, to help user pick up the use cases they need more easily. The use cases in each set will be introduced in details immediately after.
This set is for small-scale. It consists of the following use cases:
| Use case | Description |
|---|---|
| singlenode-conn-tcp-10K-100 | Connection scenario for 1,000 clients |
| singlenode-fanout-1-1K-1-1K | Fan-out scenario for 1 pub message per second and 1,000 sub messages per second |
| singlenode-p2p-1K-1K-1K-1K | Point-to-point scenario for 1,000 pub messages per second and 1,000 sub messages per second |
| singlenode-sharedsub-1K-5-1K-1K | Fan-in scenario for 1,000 pub messages per second and 1,000 sub messages per second in shared subscription way |
This set is for large-scale. It consists of the following use cases:
| Use case | Description |
|---|---|
| singlenode-conn-tcp-1M-5K | Connection scenario for 1,000,000 clients |
| singlenode-fanout-5-1000-5-250K | Fan-out scenario for 5 pub messages per second and 250,000 sub messages per second |
| singlenode-p2p-50K-50K-50K-50K | Point-to-point scenario for 50,000 pub messages per second and 50,000 sub messages per second |
| singlenode-sharedsub-50K-500-50K-50K | Fan-in scenario for 50,000 pub messages per second and 50,000 sub messages per second in shared subscription way |
Here comes the detailed use cases. The primary use cases focus on those executed on single-node-brokers, and in the future, we will add more supplemental use cases for clustering-brokers.
In a connection scenario, a batch of clients connect to the broker within a period of time, and keep the connections with the broker for quite a while.
2 use cases of this scenario for brokers deployed on single node are presented as:
| Use case | singlenode-conn-tcp-10K-100 |
| Description | 10,000 clients simultaneously connect to broker |
| Details |
|
| Computing resource metrics of interest |
|
| Broker capability metrics of interest |
|
| Use case | singlenode-conn-tcp-1M-5K |
| Description | 1,000,000 clients simultaneously connect to broker |
| Details |
|
| Computing resource metrics of interest |
|
| Broker capability metrics of interest |
|
In a fan-out scenario, a large number of clients act as subscribers, with only a few or a single publisher.
2 use cases of this scenario for brokers deployed on single node are presented as:
| Use case | singlenode-fanout-1-1K-1-1K |
| Description | 1 publisher publishes messages to 1 topic which are subscribed by 1,000 subscribers |
| Details |
|
| Computing resource metrics of interest |
|
| Broker capability metrics of interest |
|
| Use case | singlenode-fanout-5-1000-5-250K |
| Description | 50 publishers publish messages to 5 topics which are subscribed by 1,000 subscribers |
| Details |
|
| Computing resource metrics of interest |
|
| Broker capability metrics of interest |
|
In a point-to-point scenario, the equal number of clients act as publishers and subscribers respectively.
2 use cases of this scenario for brokers deployed on single node are presented as:
| Use case | singlenode-p2p-1K-1K-1K-1K |
| Description | 1,000 publishers publish messages to 1,000 topics which are subscribed by 1,000 subscribers |
| Details |
|
| Computing resource metrics of interest |
|
| Broker capability metrics of interest |
|
| Use case | singlenode-p2p-50K-50K-50K-50K |
| Description | 50,000 publishers publish messages to 50,000 topics which are subscribed by 50,000 subscribers |
| Details |
|
| Computing resource metrics of interest |
|
| Broker capability metrics of interest |
|
A fan-in scenario is the opposite of fan-out. In a fan-in scenario, a large number of clients act as publishers, with only a few or a single subscriber.
2 use cases of this scenario for brokers deployed on single node are presented as:
| Use case | singlenode-sharedsub-1K-5-1K-1K |
| Description | 1,000 publishers publish messages to 1,000 topics which are subscribed in shared subscription way by 5 subscribers |
| Details |
|
| Computing resource metrics of interest |
|
| Broker capability metrics of interest |
|
| Use case | singlenode-sharedsub-50K-500-50K-50K |
| Description | 50,000 publishers publish messages to 50,000 topics which are subscribed in shared subscription way by 500 subscribers |
| Details |
|
| Computing resource metrics of interest |
|
| Broker capability metrics of interest |
|
A composite scenario combines scenarios involving connection/publish/subscribe. Usually, a large number of clients connect to the broker, with most of them playing roles of background connections, and the rest of them perform fan-out, point-to-point, or fan-in scenario.
We welcome contributions from the community to enrich use cases!
We execute benchmarking on several MQTT brokers (including EMQX, Mosquitto, NanoMQ, etc.), using XMeter as the benchmark tool. For each use case involved, XMeter simulates a huge number of MQTT clients, executes expected publishing/subscription, and generates reports covering the primary performance metrics.
For detailed benchmarking information and result, please refer to the following pages:
Benchmarking results of concurrent connection
Benchmarking results of fan-out