Do consumers/subscriptions get lost between HTTP bridge restarts?

[noamichael]

Hello all,

I've been playing around with deploying HTTP Bridge with Strimzi as an alternative to connecting via the Kafka protocol, and I noticed some behavior I wanted to confirm. It would seem that if the HTTP bridge pod goes down, it loses all the created consumers/subscriptions and they must be recreated.

The consumers are created and stored in an in-memory map called httpSinkEndpoints. Here is the definition of the map:

strimzi-kafka-bridge/src/main/java/io/strimzi/kafka/bridge/http/HttpBridgeContext.java

Line 23 in 5bcb731

private Map<ConsumerInstanceId, HttpSinkBridgeEndpoint<K, V>> httpSinkEndpoints = new HashMap<>();

and here is the place in the createConsumer method where consumers are added to the map:

strimzi-kafka-bridge/src/main/java/io/strimzi/kafka/bridge/http/HttpBridge.java

Line 313 in 5bcb731

httpBridgeContext.getHttpSinkEndpoints().put(httpEndpoint.consumerInstanceId(), httpEndpoint);

If I'm understanding this correctly, there is nothing in place to repopulate that map if the bridge pods goes down. Does that mean the client applications should have logic for recreating both the consumers and the subscriptions in the case of pod restarts? I didn't find anything in the docs explaining if this was the case either, so I appreciate any insight you can offer!

[scholzj]

That is correct. The consumers and subscriptions are lost when the bridge restarts. That is expected behavior.

---

as an alternative to connecting via the Kafka protocol

I do not know anything about your project - but in general, I think whenever you have a good Kafka client available, you should use it. The HTTP Bridge makes sense in situations when you do not have a Kafka client around or it would need too many resources to use it (e.g. IoT use-cases) or for example when you want to better isolate your Kafka cluster from the outside world (e.g. using HTTP from the remote application over open internet instead of the Kafka protocol.).

[noamichael]

Thank you for the quick reply! I was just exploring the different methods of connecting to Kafka and I'll prefer the clients when possible. Do you think this behavior should be documented somewhere since it might require applications using the bridge to add in some level of retry logic if the bridge restarts? Perhaps I can make a PR if so.

[scholzj]

TBH, I thought that this would be documented. But if it isn't then PR would be great. The documentation sources for the Bridge docs should be in this repo: https://github.com/strimzi/strimzi-kafka-bridge/tree/main/documentation (apart from the api part is IIRC auto.generated from the OpenAPI spec - so that cannot be changed directly).