Unlock the potential of the Internet of Things (IoT) with this cutting-edge Event Streaming Architecture. Leveraging the power of Apache Kafka and MQTT, this project is designed to handle IoT data at scale. 🚀
Ingest real-time data from IoT sensors across diverse locations, analyze metrics like temperature, humidity, pressure, and luminosity, and store them efficiently in a high-performance MongoDB database. 🌡️💧📈
The technology stack includes Spring Boot, Kafka Streams, Micrometer, and Grafana for real-time monitoring, making this architecture a powerhouse for IoT solutions. 💻📈🔍
With Docker containerization, deployment is a breeze. Explore this project, visualize IoT data, and gain insights into your sensor networks. 🐳🚀📊
Check out the detailed Medium article for a comprehensive overview. 📖
Thank you for visiting the Smart IoT Event Streaming GitHub repository! Empower your IoT endeavors and scale with confidence. 🌐📈💡
Project developed to practice what I have learned in the Udemy course Apache Kafka Series - Kafka Connect Hands-on Learning and Apache Kafka Series - Kafka Streams for Data Processing.
The main goals of this architecture are the following:
- Ingest and store real-time data from IoT sensors located in various locations.
- Analyze and make aggregations through rotating time windows to know average temperature by sensor or place, in addition to humidity, pressure and luminosity.
- Store data after processing for subsequent monitoring in a NOSQL database with good performance for frequent writes for relatively homogeneous document sizes such as MongoDB.
- Visualization of the metrics of the sensors in real time and of the performance of the components of the architecture through Grafana and the consolidated documents in MongoDB through Mongo-Express.
- Spring Boot 2.3.3 / Apache Maven 3.6.3.
- Spring Boot Starter Actuator.
- Kafka Streams.
- Spring Kafka.
- Micrometer Registry Prometheus.
- Eclipse Paho MQTT Client.
- Kafka Connect.
- Kafka Rest Proxy
- lombok.
- Jackson.
- NodeExporter (Exporter for machine metrics).
- Prometheus.
- Grafana.
- Eclipse Mosquitto.
- MongoDB.
- Mongo DB Express (Web-based MongoDB admin interface, written with Node.js and express).
- Cadvisor (Analyzes resource usage and performance characteristics of running containers).
- kafka-exporter (Kafka exporter for Prometheus).
The available tasks are detailed below (rake --task)
| Task | Description |
|---|---|
| check_deployment_file_task | Check Deployment File |
| check_docker_task | Check Docker and Docker Compose Task |
| cleaning_environment_task | Cleaning Evironment Task |
| deploy | Deploys the IoT Event Streaming Architecture and laun... |
| login | Authenticating with existing credentials |
| start | Start Containers |
| status | Status Containers |
| stop | Stop Containers |
| undeploy | UnDeploy IoT Event Streaming Architecture |
To start the platform make sure you have Ruby installed, go to the root directory of the project and run the rake deploy task, this task will carry out a series of preliminary checks, discard images and volumes that are no longer necessary and also proceed to download all the images and the initialization of the containers.
| Container | Port |
|---|---|
| kafka-topics-ui | localhost:8081 |
| kafka-connect-ui | localhost:8082 |
| zoonavigator-web | localhost:8083 |
| mongo-express | localhost:8084 |
| grafana | localhost:8085 |
| prometheus | localhost:8086 |
| kafka-rest-proxy | localhost:9999 |
