gRPC-Kafka backpressure support?

[Sumit2202]

My team wanted to use Zilla proxy for inter-service gRPC calls, and we had a question on whether Zilla has capabilities for pausing and resuming the reply topic in kafka to handle backpressure across services, currently we are using gRPC-Kafka and Kafka-gRPC in our zilla config file, do we use guard topics? Can we dynamically adjust the in/out rate from topics, across gRPC services through Zilla?

[vordimous]

Currently, Zilla doesn't support rate-limiting, but it is on our feature roadmap. This enhancement would cover both inbound and outbound messages to a Kafka topic.

We may be able to help you work through what options you have by using Zilla and Kafka to send messages between your gRPC services. Can you describe what is breaking in your current setup and what a rate-limited stream of messages would help fix?

[vordimous]

@Sumit2202 Let me see if I understand your setup correctly.

1. You are producing messages onto a Kafka topic from gRPC services using the grpc-kafka binding
2. You would like to consume messages from a Kafka topic and call a gRPC service using the kafka-grpc binding
3. There are too many messages being published at one time for the receiving gRPC service to handle
4. You have implemented a workaround to Fetch a measured number of messages from the topic using a direct connection to Kafka from the server

[Sumit2202]

💯

[vordimous]

@Sumit2202 can the server scale to have multiple gRPC services be able to process the messages? Zilla uses consumer groups so Kafka can distribute the load over more than one gRPC service.

[Sumit2202]

Yes,servers are running inside K8s, but we still needed a Manual circuit breaker and way to unsubscribe from the kafka topic in events when server had some downtime issues and the continuos stream of upstream requests were only making it worse.

[vordimous]

@Sumit2202 I see. So, under normal operation, the rate of messages isn't as big of an issue, but when there is a large backlog of messages, Zilla delivers them to the gRPC service very fast, causing race conditions?