Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revamp Kafka consumer check #13918

Merged
merged 88 commits into from
Apr 14, 2023
Merged
Changes from 1 commit
Commits
Show all changes
88 commits
Select commit Hold shift + click to select a range
99d8545
Remove deprecated implementation of kafka_consumer (#13915)
yzhan289 Feb 9, 2023
5fad2df
Remove DSM (#13914)
fanny-jiang Feb 9, 2023
64cfd17
Remove more unused code (#13922)
fanny-jiang Feb 10, 2023
d656496
Flatten kafka consumer check (#13929)
yzhan289 Feb 10, 2023
8cbbb42
Add more tests to increase code coverage (#13921)
fanny-jiang Feb 10, 2023
7661573
Flatten the check structure
FlorentClarret Feb 15, 2023
ff2adfa
Revert "Flatten the check structure"
FlorentClarret Feb 15, 2023
dcaf7a4
Refactor Kafka Consumer (#13931)
yzhan289 Feb 15, 2023
ea70817
Remove KafkaClient and refactor tests (#13967)
yzhan289 Feb 15, 2023
d1e23e6
Pass in config to client (#13970)
yzhan289 Feb 15, 2023
71925f4
Move metric reporting back into main check (#13973)
fanny-jiang Feb 15, 2023
1efecf5
Drop Python 2 support (#13961)
FlorentClarret Feb 16, 2023
4bc5d72
Fix agent deps (#13979)
FlorentClarret Feb 16, 2023
54270d1
Split the tests (#13983)
FlorentClarret Feb 16, 2023
9445ce6
Add missing license headers (#13985)
FlorentClarret Feb 16, 2023
464cedb
Separate config logic (#13989)
yzhan289 Feb 16, 2023
9d6bc06
move tls_context back into check (#13987)
fanny-jiang Feb 16, 2023
f3d491f
Merge branch 'master' into AI-2904/kafka-consumer-revamp
FlorentClarret Feb 17, 2023
c7c5f23
Fix license headers (#13993)
FlorentClarret Feb 17, 2023
cbc9499
Add healthchecks to zookeeper (#13998)
FlorentClarret Feb 17, 2023
d73b7cf
Refactor the tests (#13997)
FlorentClarret Feb 17, 2023
9ec25cb
Remove self.check and cleanup (#13992)
yzhan289 Feb 17, 2023
29d72d4
Rename variables to be consistent
yzhan289 Feb 17, 2023
ef4d252
Merge branch 'master' into AI-2904/kafka-consumer-revamp
FlorentClarret Feb 23, 2023
cf4e083
Refactor and fix tests (#14019)
fanny-jiang Feb 23, 2023
8613a9a
Disable one unit test (#14025)
FlorentClarret Feb 24, 2023
47b3e79
Merge branch 'master' into AI-2904/kafka-consumer-revamp
FlorentClarret Feb 24, 2023
c122a42
Create environments for the new kafka client (#14022)
FlorentClarret Feb 24, 2023
734439b
Increase test coverage (#14021)
yzhan289 Mar 1, 2023
8dab1dd
Merge branch 'master' into AI-2904/kafka-consumer-revamp
yzhan289 Mar 1, 2023
0a84c96
Add dependency (#14076)
yzhan289 Mar 2, 2023
06880c7
Merge branch 'master' into AI-2904/kafka-consumer-revamp
FlorentClarret Mar 2, 2023
07cedd3
Pass consumer offsets into highwater offsets (#14077)
yzhan289 Mar 3, 2023
74e3b19
Create Kafka client for confluent lib (#14078)
yzhan289 Mar 3, 2023
feaf069
Remove collect_broker_version (#14095)
yzhan289 Mar 3, 2023
eff52d1
Merge branch 'master' into AI-2904/kafka-consumer-revamp
yzhan289 Mar 3, 2023
03cd475
Implement reset offsets (#14103)
yzhan289 Mar 6, 2023
fda2758
Implement get_partitions_for_topic (#14079)
yzhan289 Mar 6, 2023
18cf334
Merge branch 'master' into AI-2904/kafka-consumer-revamp
FlorentClarret Mar 8, 2023
a6fd4d9
Implement consumer offsets (#14080)
yzhan289 Mar 8, 2023
7f6409f
Merge branch 'master' into AI-2904/kafka-consumer-revamp
yzhan289 Mar 8, 2023
4be6ab7
Use confluent-kafka during the test setup (#14122)
FlorentClarret Mar 10, 2023
e8e800c
Implement get_highwater_offsets and get_highwater_offsets_dict (#14094)
yzhan289 Mar 13, 2023
7a6a962
Modify the hatch environment to support several authentication method…
FlorentClarret Mar 13, 2023
c258d73
Merge branch 'master' into AI-2904/kafka-consumer-revamp
FlorentClarret Mar 13, 2023
24c5cf7
Create the topics from the python code instead of the docker image
FlorentClarret Mar 10, 2023
46b6745
drop KAFKA_VERSION
FlorentClarret Mar 13, 2023
35f2639
Remove some unused functions (#14145)
FlorentClarret Mar 14, 2023
926a1d8
Merge branch 'master' into AI-2904/kafka-consumer-revamp
FlorentClarret Mar 14, 2023
13a8fd3
Update all the tests to use the `kafka_instance` instead of a custom …
FlorentClarret Mar 14, 2023
df4e4f1
Implement the `request_metadata_update` method (#14152)
FlorentClarret Mar 14, 2023
65751ea
Merge branch 'master' into AI-2904/kafka-consumer-revamp
FlorentClarret Mar 15, 2023
11aae44
Remove the `get_dict` methods from the clients (#14149)
FlorentClarret Mar 15, 2023
f39cb7c
Merge branch 'master' into AI-2904/kafka-consumer-revamp
FlorentClarret Mar 16, 2023
26b607d
Manually build confluent-kafka in the test env (#14173)
FlorentClarret Mar 16, 2023
0902b3c
Merge branch 'master' into AI-2904/kafka-consumer-revamp
FlorentClarret Mar 17, 2023
682c703
Refactor the confluent kafka client (#14158)
FlorentClarret Mar 17, 2023
7d53dd4
Merge branch 'master' into AI-2904/kafka-consumer-revamp
FlorentClarret Mar 20, 2023
8c4bb33
Merge branch 'master' into AI-2904/kafka-consumer-revamp
FlorentClarret Mar 21, 2023
206b544
Merge branch 'master' into AI-2904/kafka-consumer-revamp
FlorentClarret Mar 22, 2023
ac7e5ac
Merge branch 'master' into AI-2904/kafka-consumer-revamp
FlorentClarret Mar 23, 2023
674db3d
Merge branch 'master' into AI-2904/kafka-consumer-revamp
yzhan289 Mar 23, 2023
2d23ca1
Add a tls e2e env and implement it (#14137)
FlorentClarret Mar 24, 2023
8b61e65
Merge branch 'master' into AI-2904/kafka-consumer-revamp
FlorentClarret Mar 24, 2023
01a7d1b
Add a kerberos e2e env and implement it (#14120)
FlorentClarret Mar 24, 2023
c878425
Merge branch 'master' into AI-2904/kafka-consumer-revamp
FlorentClarret Mar 24, 2023
bbc8cdb
Merge branch 'master' into AI-2904/kafka-consumer-revamp
FlorentClarret Mar 27, 2023
58a69f9
Merge branch 'master' into AI-2904/kafka-consumer-revamp
FlorentClarret Mar 27, 2023
9613334
Merge branch 'master' into AI-2904/kafka-consumer-revamp
FlorentClarret Mar 27, 2023
493a4d2
Merge branch 'master' into AI-2904/kafka-consumer-revamp
yzhan289 Mar 27, 2023
9f13a59
Merge branch 'master' into AI-2904/kafka-consumer-revamp
FlorentClarret Mar 28, 2023
9093016
Add a krb5 config file to run the tests locally (#14251)
FlorentClarret Mar 28, 2023
1c70ff9
Merge branch 'master' into AI-2904/kafka-consumer-revamp
yzhan289 Mar 29, 2023
4dc1e1c
Implement OAuth config (#14247)
yzhan289 Mar 30, 2023
3fd3166
Merge branch 'master' into AI-2904/kafka-consumer-revamp
yzhan289 Mar 30, 2023
bd21f05
Merge branch 'master' into AI-2904/kafka-consumer-revamp
yzhan289 Mar 31, 2023
3e4b54d
Merge branch 'master' into AI-2904/kafka-consumer-revamp
FlorentClarret Apr 3, 2023
53a74b4
Drop the legacy client (#14243)
FlorentClarret Apr 3, 2023
80e2eb7
Merge branch 'master' into AI-2904/kafka-consumer-revamp
yzhan289 Apr 12, 2023
da6666e
Merge branch 'master' into AI-2904/kafka-consumer-revamp
yzhan289 Apr 12, 2023
cb01ec3
Fix style
yzhan289 Apr 12, 2023
79b8927
Apply suggestions
yzhan289 Apr 13, 2023
5d6e062
Make try-except smaller
yzhan289 Apr 13, 2023
4a33136
Change asserts into config errors
yzhan289 Apr 13, 2023
a093a2b
Merge branch 'master' into AI-2904/kafka-consumer-revamp
yzhan289 Apr 13, 2023
c5e6c6b
Add back disable e2e for kerberos
yzhan289 Apr 13, 2023
5ccdc25
Remove licenses for removed dependencies
yzhan289 Apr 13, 2023
558307c
Merge branch 'master' into AI-2904/kafka-consumer-revamp
yzhan289 Apr 13, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 8 additions & 5 deletions kafka_consumer/datadog_checks/kafka_consumer/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -104,12 +104,13 @@ def get_highwater_offsets(self, consumer_offsets):
def get_partitions_for_topic(self, topic):
try:
cluster_metadata = self.kafka_client.list_topics(topic, timeout=self.config._request_timeout)
topic_metadata = cluster_metadata.topics[topic]
partitions = list(topic_metadata.partitions.keys())
return partitions
except KafkaException as e:
yzhan289 marked this conversation as resolved.
Show resolved Hide resolved
self.log.error("Received exception when getting partitions for topic %s: %s", topic, e)
return None
else:
topic_metadata = cluster_metadata.topics[topic]
partitions = list(topic_metadata.partitions.keys())
return partitions
Comment on lines +110 to +113
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can leave this for later, but we don't really need the else block here anymore since the except clause already returns early. Removing the else makes it more clear what the main ("happy") path of the function is.


def request_metadata_update(self):
# https://github.com/confluentinc/confluent-kafka-python/issues/594
Expand Down Expand Up @@ -147,6 +148,9 @@ def get_consumer_offsets(self):
for future in self._get_consumer_offset_futures(consumer_groups):
try:
response_offset_info = future.result()
except KafkaException as e:
self.log.debug("Failed to read consumer offsets for %s: %s", consumer_group, e)
else:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here, we can reduce the extra nesting by continueing when we catch the exception and keeping the main code out of the try-except.

self.log.debug('FUTURE RESULT: %s', response_offset_info)
consumer_group = response_offset_info.group_id
topic_partitions = response_offset_info.topic_partitions
Expand All @@ -171,8 +175,7 @@ def get_consumer_offsets(self):
str(topic_partition.partition),
)
consumer_offsets[(consumer_group, topic, partition)] = offset
except KafkaException as e:
self.log.debug("Failed to read consumer offsets for %s: %s", consumer_group, e)


return consumer_offsets

Expand Down