-
Notifications
You must be signed in to change notification settings - Fork 314
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Expose HTTP endpoint SQL queries, queries count and execution time via Prometheus #2828
Comments
Thanks for opening your first issue in the Marquez project! Please be sure to follow the issue template! |
At the moment, we can see the data of interest using the Java method, but we desire this feature to allow us to see it from an HTTP endpoint perspective as well (e.g., |
@mobuchowski 's suggest: add the HTTP verb and the endpoint path We need to investigate if and how this could be accomplished, and if there are better ways |
@JDarDagran is implementing this in #2850 |
Closed by: #2850 |
Context
Since the 0.7.0 release (#1906), Marquez supports pushing metrics to Prometheus.
This task proposes extending the current capability to give visibility to Marquez's SQL queries. Some of the questions we'd like to be answered:
By identifying potential bottlenecks in Marquez queries and the database, this extension could facilitate the provisioning of adequate resources. This, in turn, could lead to improved performance and efficiency of the database and Marquez itself.
Implementation
If possible, we could give visibility of frequency (count) and duration (gauge) for all queries Marquez runs. There is a possibility this could be done close to
jdbi
: https://metrics.dropwizard.io/4.2.0/manual/jdbi.htmlIf this is not possible, we could add the instrumentation to specific write and read endpoints, covering at least the SQL queries triggered by the following endpoints:
api/v1/lineage
(*)api/v1/namespaces/{namespace}
(*)api/v1/namespaces
api/v1/namespaces/{namespace}/jobs/{job}
api/v1/namespaces/{namespace}/datasets
api/v1/column-lineage
The most critical are (*)
The text was updated successfully, but these errors were encountered: