what is this exception? should i catch it??

[chusteven]

1/ this one

WARNING:kafka.coordinator:Heartbeat session expired, marking coordinator dead
WARNING:kafka.coordinator:Marking the coordinator dead (node coordinator-0) for group consumer-group: Heartbeat session expired.
INFO:kafka.cluster:Group coordinator for consumer-group is BrokerMetadata(nodeId='coordinator-0', host='raspberrypi', port=9092, rack=
None)
INFO:kafka.coordinator:Discovered coordinator coordinator-0 for group consumer-group
WARNING:kafka.coordinator.consumer:Auto offset commit failed for group consumer-group: NodeNotReadyError: coordinator-0
WARNING:kafka.coordinator.consumer:Auto offset commit failed for group consumer-group: NodeNotReadyError: coordinator-0
WARNING:kafka.coordinator.consumer:Auto offset commit failed for group consumer-group: NodeNotReadyError: coordinator-0
...
WARNING:kafka.coordinator:Heartbeat session expired, marking coordinator dead
WARNING:kafka.coordinator:Marking the coordinator dead (node coordinator-0) for group consumer-group: Heartbeat session expired.
INFO:kafka.cluster:Group coordinator for consumer-group is BrokerMetadata(nodeId='coordinator-0', host='raspberrypi', port=9092, rack=
None)
INFO:kafka.coordinator:Discovered coordinator coordinator-0 for group consumer-group
INFO:root:No need to flush buffer, sleeping...
WARNING:kafka.coordinator.consumer:Auto offset commit failed for group consumer-group: NodeNotReadyError: coordinator-0
WARNING:kafka.coordinator.consumer:Auto offset commit failed for group consumer-group: NodeNotReadyError: coordinator-0
WARNING:kafka.coordinator:Heartbeat session expired, marking coordinator dead
WARNING:kafka.coordinator:Marking the coordinator dead (node coordinator-0) for group consumer-group: Heartbeat session expired.
INFO:kafka.cluster:Group coordinator for consumer-group is BrokerMetadata(nodeId='coordinator-0', host='raspberrypi', port=9092, rack=
None)
INFO:kafka.coordinator:Discovered coordinator coordinator-0 for group consumer-group
INFO:root:No need to flush buffer, sleeping...
INFO:root:No need to flush buffer, sleeping...

2/ and another one

INFO:kafka.coordinator:Discovered coordinator coordinator-0 for group consumer-group
WARNING:kafka.coordinator.consumer:Auto offset commit failed for group consumer-group: NodeNotReadyError: coordinator-0
WARNING:kafka.coordinator.consumer:Auto offset commit failed for group consumer-group: NodeNotReadyError: coordinator-0
WARNING:kafka.coordinator.consumer:Auto offset commit failed for group consumer-group: NodeNotReadyError: coordinator-0
WARNING:kafka.coordinator.consumer:Auto offset commit failed for group consumer-group: NodeNotReadyError: coordinator-0
WARNING:kafka.coordinator.consumer:Auto offset commit failed for group consumer-group: NodeNotReadyError: coordinator-0
WARNING:kafka.coordinator:Heartbeat: local member_id was not recognized; this consumer needs to re-join
WARNING:kafka.coordinator:Heartbeat: local member_id was not recognized; this consumer needs to re-join
WARNING:kafka.coordinator:Heartbeat: local member_id was not recognized; this consumer needs to re-join
ERROR:kafka.coordinator.consumer:Offset commit failed: This is likely to cause duplicate message delivery
Traceback (most recent call last):
  File "/home/pi/.local/lib/python3.7/site-packages/kafka/coordinator/consumer.py", line 525, in _maybe_auto_commit_offsets_sync
    self.commit_offsets_sync(self._subscription.all_consumed_offsets())
  File "/home/pi/.local/lib/python3.7/site-packages/kafka/coordinator/consumer.py", line 518, in commit_offsets_sync
    raise future.exception # pylint: disable-msg=raising-bad-type
kafka.errors.CommitFailedError: CommitFailedError: Commit cannot be completed since the group has already
            rebalanced and assigned the partitions to another member.
            This means that the time between subsequent calls to poll()
            was longer than the configured max_poll_interval_ms, which
            typically implies that the poll loop is spending too much
            time message processing. You can address this either by
            increasing the rebalance timeout with max_poll_interval_ms,
            or by reducing the maximum size of batches returned in poll()
            with max_poll_records.

INFO:root:No need to flush buffer, sleeping...
INFO:kafka.coordinator.consumer:Revoking previously assigned partitions {TopicPartition(topic='tweets-stream', partition=0)} for group
 consumer-group
INFO:kafka.coordinator:(Re-)joining group consumer-group
INFO:kafka.coordinator:Elected group leader -- performing partition assignments using range
INFO:kafka.coordinator:Successfully joined group consumer-group with generation 8
INFO:kafka.consumer.subscription_state:Updated partition assignment: [TopicPartition(topic='tweets-stream', partition=0)]
INFO:kafka.coordinator.consumer:Setting newly assigned partitions {TopicPartition(topic='tweets-stream', partition=0)} for group consu
mer-group
WARNING:kafka.coordinator.consumer:Auto offset commit failed for group consumer-group: CommitFailedError: Commit cannot be completed s
ince the group has already
            rebalanced and assigned the partitions to another member.
            This means that the time between subsequent calls to poll()
            was longer than the configured max_poll_interval_ms, which
            typically implies that the poll loop is spending too much
            time message processing. You can address this either by
            increasing the rebalance timeout with max_poll_interval_ms,
            or by reducing the maximum size of batches returned in poll()
            with max_poll_records.

[chusteven]

I mean... it looks like something recovered on its own because both the daemon thread and the main one are still alive...

Another thing to note is that my producer went down and maybe there's something here re: the heartbeat... like "hey there's nothing here in this topic!" that may be responsible for this failure (thinking out loud... producer failures shouldn't result in consumer failures; who is responsible for monitoring??)

Anyway, I'm gonna wait until 630 just to make extra sure the main thread is still alive

---

630 came and went and the main thread is still alive!

INFO:root:No need to flush buffer, sleeping...
INFO:root:No need to flush buffer, sleeping...
INFO:root:No need to flush buffer, sleeping...
INFO:root:No need to flush buffer, sleeping...
INFO:root:No need to flush buffer, sleeping...
INFO:root:No need to flush buffer, sleeping...
INFO:root:About to start writing 12000 messages into S3
INFO:root:Finished writing into S3 into filepath sample-stream/dt=2022-02-13/18747061.json.gz; took 31.23 seconds