-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add database query utilities #5045
Conversation
Codecov Report
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good, some minor comments.
Have you looked how we can adapt this to oracle and others?
self.query = query | ||
self.columns = tuple(column_data) | ||
self.tags = tags | ||
del self.query_data |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Any reason to do that instead of setting it to None?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not really, goal is just to not use memory needlessly
Yes others can easily be adapted, though may need things in |
What does this PR do?
This implements a standard way to define and collect data from arbitrary queries. 2 new classes are introduced under
datadog_checks.base.utils.db
:Query
- This class accepts a singledict
argument which is the necessary data to run the query. The representation is based on ourcustom_queries
format originally designed and implemented way back in Support custom queries #1528. It is now part of all our database integrations and other products have since adopted this idea e.g. https://cloud.google.com/solutions/sap/docs/sap-hana-monitoring-agent-user-guide#defining_custom_queries.QueryManager
- This class is in charge of running any number ofQuery
instances for a singleAgentCheck
instance given anexecutor
. The executor must accept a single querystr
and return a sequence of results.Before queries can be executed, they must be compiled once via
query_manager.compile_queries()
to validate the input data. After compilation, there is no longer need for error checking nortype
look-ups, so we transform every column type to a stateful closure.We support the standard submission method types, with the addition of:
tag
- This was already supported, but there is now aboolean
argument which will convert values totrue
orfalse
as astr
monotonic_gauge
- Given a column namedfoo
, this will submitfoo.total
asgauge
andfoo.count
asmonotonic_count
temporal_percent
- This maps to Add total_time_to_temporal_percent utility #4924 and will submit of course asrate
. For convenience, thescale
argument can also be eithersecond
,millisecond
,microsecond
, ornanosecond
match
- This requires anitems
argument which is a map ofsource
column values to other types. This is most useful for columnar-oriented databases, see e.g. https://clickhouse.yandex/docs/en/operations/system_tables/#system_tables-metricsAny arguments not picked up by a transformer will be passed directly to the metric submission methods for overriding tags, setting the hostname, etc.
Motivation
Code reuse
Additional Notes
This satisfies all current use cases for ClickHouse, but we'll need to eventually implement:
metadata
submissionextra_metrics
that would be guaranteed to run after every column has been processed to compute things such as percentagesQuery
to do extra complicated things