performance improvements

[tzachshabtay]

• Performance improvements:

  • Removed logging by default from server-side fetches to improve performance by x4 for the normal case
  • Using kafkajs per batch processing instead of per message processing, which gives us greater control to squeeze some more performance (we can discard complete batches for wrong partitions, and we can discard heartbeats as we don't intend to keep the consumer group alove)
  • Subscribing latest only instead of oldest only so for the scenario where the topic is stale we won't get unneeded messages from the time we subscribe until we seek to our desired offset
  • Added server timings for the message fetch request to help with profiling

• Transactional topic support: for transactional topics the offsets are not consecutive (apparently kafka transactional producers write control messages which are hidden), and the highest offset might not even exist. By moving to per batch processing it seems we're able to account for that as the batch's high offset from kafkajs will include the missing offset so we can check it and stop processing on time, avoiding a timeout which was the previous status quo

• Fixed a bug where we didn't show the partition column when selecting all partitions

• Fixed a bug where searching newest offsets with a limit bigger than the highest offset resulted in an error