Rest Gateway to publish plans to Kafka and then process asynchronously

[vishalag001]

Hi @wajda,

Is this possible that rest-gateway publishes the plan to a Kafka topic and then we process the plan asynchronously ?

In the current setup, each HTTP call makes a database entry. Ideally HTTP calls and database actions should be decoupled. If the database is loaded with too many requests, then subsequent HTTP calls gets timed out as it could get any DB resources and this way we loose on the data due to Request timeout.

But with kafka in place, HTTP rest calls will publish the mesaages onto a topic and then exit. Then another thread/process can consume this plan message and make DB inserts, which in this case will be a kafkaConsumer.

Do let me know your thought !

[wajda]

There is a Kafka module n Spline specifically created fie that purpose - https://github.com/AbsaOSS/spline/tree/develop/kafka-gateway

The docker - https://hub.docker.com/r/absaoss/spline-kafka-server

[wajda]

If you want to maintain REST API for the agents, but using Kafka behind it, you'll need to create your own REST API bridge service. This pattern has a lot of benefits in my opinion compared to the vanilla "agent -> Kafka -> Spline", so I would go with that as well.