Two (supposedly identical) sets of Cloud Monitoring queries (with MQF and MQL) return (vastly) different results #289
Assignees
Labels
backend
bug
Something isn't working
p/cloud_monitoring_mql
p/cloud_monitoring
Cloud Monitoring provider issue
Comments
lvaylet
added
bug
|
A standalone test script saved as """Make sure requests to Cloud Monitoring with MQF and MQL return the same results."""
from datetime import datetime
from slo_generator.backends.cloud_monitoring import CloudMonitoringBackend
from slo_generator.backends.cloud_monitoring_mql import CloudMonitoringMqlBackend
PROJECT_ID: str = "slo-generator-demo"
mqf_backend = CloudMonitoringBackend(PROJECT_ID)
mql_backend = CloudMonitoringMqlBackend(PROJECT_ID)
# Use a specific, fixed timestamp for both queries and the same window.
# Python uses floating point numbers to represent time in seconds since the epoch, in
# UTC, with decimal part representing nanoseconds.
end_time: float = 1666995015.5144777
# MQL expects dates formatted like "%Y/%m/%d %H:%M:%S" or "%Y/%m/%d-%H:%M:%S".
# Reference: https://cloud.google.com/monitoring/mql/reference#lexical-elements
end_time_str: str = datetime.fromtimestamp(end_time).strftime("%Y/%m/%d %H:%M:%S")
assert end_time_str == "2022/10/28 22:10:15" # MQL ignores nanoseconds
mqf_query: str = """resource.type="gae_app"
metric.type="appengine.googleapis.com/http/server/response_count"
resource.labels.project_id="slo-generator-demo"
resource.labels.module_id="ratingservice"
( metric.labels.response_code = 429 OR
metric.labels.response_code = 200 OR
metric.labels.response_code = 201 OR
metric.labels.response_code = 202 OR
metric.labels.response_code = 203 OR
metric.labels.response_code = 204 OR
metric.labels.response_code = 205 OR
metric.labels.response_code = 206 OR
metric.labels.response_code = 207 OR
metric.labels.response_code = 208 OR
metric.labels.response_code = 226 OR
metric.labels.response_code = 304 )
"""
mql_query: str = """fetch gae_app
| metric 'appengine.googleapis.com/http/server/response_count'
| filter resource.project_id == 'slo-generator-demo'
| filter resource.module_id == 'ratingservice'
| filter
metric.response_code == 429
|| metric.response_code == 200
|| metric.response_code == 201
|| metric.response_code == 202
|| metric.response_code == 203
|| metric.response_code == 204
|| metric.response_code == 205
|| metric.response_code == 206
|| metric.response_code == 207
|| metric.response_code == 208
|| metric.response_code == 226
|| metric.response_code == 304
"""
for window in [3600, 24 * 3600, 7 * 24 * 3600, 28 * 24 * 3600, 6 * 28 * 24 * 3600]:
mqf_count = mqf_backend.count(list(mqf_backend.query(end_time, window, mqf_query)))
mql_count = mql_backend.count(list(mql_backend.query(mql_query, window)))
print(f"window = {window}s | MQF: {mqf_count} | MQL: {mql_count}")The output is: |
|
The changes from #290 seem to fix the issue. The small discrepancies observed with 7-day and 6-month windows (1 event and 5 events, respectively) represent a negligible percentage of the actual number of events. (venv) user@workstation-l9pyhi6x:~/workspace/github/google/slo-generator$ python test_mqf_vs_mql.py
window = 3600s | MQF: 1124 | MQL: 1124
window = 86400s | MQF: 27371 | MQL: 27371
window = 604800s | MQF: 191674 | MQL: 191675
window = 2419200s | MQF: 767616 | MQL: 767616
window = 14515200s | MQF: 1401063 | MQL: 1401068 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
slo-generator2.3.2python3.10.7.envfile:The MQF queries return:
The MQL queries (with the same timestamp) return:
The SLO configs look identical though:
The same discrepancies can be observed on a different project (
slo-generator-demofor example) and/or a different GAE service.The text was updated successfully, but these errors were encountered: