Fill missing metrics labels with empty string to avoid label dimensions inconsistent failure

[everesio]

The PR fixes #19 by adding missing labels keys with empty string values.
The feature (fix) is enabled with "collector.fill-missing-labels" flag.

[olvesh]

I tested your branch, but it still gives me:

4 error(s) occurred:
* collected metric stackdriver_consumed_api_serviceruntime_googleapis_com_api_request_count label:<name:"grpc_status_code" value:"0" > label:<name:"location" value:"europe-west1" > label:<name:"method" value:"google.api.servicecontrol.v1.ServiceController.Check" > label:<name:"project_id" value:"<redacted>" > label:<name:"protocol" value:"" > label:<name:"response_code" value:"200" > label:<name:"response_code_class" value:"2xx" > label:<name:"service" value:"servicecontrol.googleapis.com" > label:<name:"unit" value:"1" > label:<name:"version" value:"v1" > gauge:<value:1 >  has label dimensions inconsistent with previously collected metrics in the same metric family
* collected metric stackdriver_consumed_api_serviceruntime_googleapis_com_api_request_count label:<name:"grpc_status_code" value:"0" > label:<name:"location" value:"europe-west1" > label:<name:"method" value:"google.api.servicecontrol.v1.ServiceController.Report" > label:<name:"project_id" value:"<redacted>" > label:<name:"protocol" value:"" > label:<name:"response_code" value:"200" > label:<name:"response_code_class" value:"2xx" > label:<name:"service" value:"servicecontrol.googleapis.com" > label:<name:"unit" value:"1" > label:<name:"version" value:"v1" > gauge:<value:2 >  has label dimensions inconsistent with previously collected metrics in the same metric family
* collected metric stackdriver_consumed_api_serviceruntime_googleapis_com_api_request_latencies label:<name:"location" value:"europe-west1" > label:<name:"method" value:"google.api.servicecontrol.v1.ServiceController.Check" > label:<name:"project_id" value:"<redacted>" > label:<name:"service" value:"servicecontrol.googleapis.com" > label:<name:"unit" value:"s" > label:<name:"version" value:"v1" > histogram:<sample_count:1 sample_sum:0.00061123 bucket:<cumulative_count:0 upper_bound:1e-06 > bucket:<cumulative_count:0 upper_bound:9.999999999999999e-06 > bucket:<cumulative_count:0 upper_bound:9.999999999999999e-05 > bucket:<cumulative_count:1 upper_bound:0.001 > bucket:<cumulative_count:1 upper_bound:0.01 > bucket:<cumulative_count:1 upper_bound:0.09999999999999999 > bucket:<cumulative_count:1 upper_bound:1 > bucket:<cumulative_count:1 upper_bound:10 > bucket:<cumulative_count:1 upper_bound:100 > bucket:<cumulative_count:1 upper_bound:inf > >  has label dimensions inconsistent with previously collected metrics in the same metric family
* collected metric stackdriver_consumed_api_serviceruntime_googleapis_com_api_request_latencies label:<name:"location" value:"europe-west1" > label:<name:"method" value:"google.api.servicecontrol.v1.ServiceController.Report" > label:<name:"project_id" value:"<redacted>" > label:<name:"service" value:"servicecontrol.googleapis.com" > label:<name:"unit" value:"s" > label:<name:"version" value:"v1" > histogram:<sample_count:2 sample_sum:0.001260585 bucket:<cumulative_count:0 upper_bound:1e-06 > bucket:<cumulative_count:0 upper_bound:9.999999999999999e-06 > bucket:<cumulative_count:0 upper_bound:9.999999999999999e-05 > bucket:<cumulative_count:2 upper_bound:0.001 > bucket:<cumulative_count:2 upper_bound:0.01 > bucket:<cumulative_count:2 upper_bound:0.09999999999999999 > bucket:<cumulative_count:2 upper_bound:1 > bucket:<cumulative_count:2 upper_bound:10 > bucket:<cumulative_count:2 upper_bound:100 > bucket:<cumulative_count:2 upper_bound:inf > >  has label dimensions inconsistent with previously collected metrics in the same metric family

using the following metrics:

cloudsql.googleapis.com/,datastore.googleapis.com/,pubsub.googleapis.com/,serviceruntime.googleapis.com/

[everesio]

have you used a new flag which enables the fix?

[olvesh]

Oops, I might have missed the STACKDRIVER_EXPORTER_COLLECTOR_FILL_MISSING_LABELS flag :-)

Will test again next week...

[wjoel]

Thanks for the patch! It seems to work almost perfectly, but not quite. We're still seeing sporadic errors for some metrics from loadbalancing.googleapis.com, but no errors if we remove that type prefix.

On the other hand, the issues we saw before applying your changes were also about loadbalancing metrics, so I'm not 100% sure if it made much of a difference. We have set STACKDRIVER_EXPORTER_COLLECTOR_FILL_MISSING_LABEL=true.

Perhaps the remaining issues have something to do with histograms, specifically?

* collected metric stackdriver_https_lb_rule_loadbalancing_googleapis_com_https_total_latencies
 label:<name:"backend_name" value:"k8s-ig--635e72c38338af7e" >
 label:<name:"backend_scope" value:"europe-west1-c" >
 label:<name:"backend_scope_type" value:"ZONE" >
 label:<name:"backend_target_name" value:"k8s-be-31156--635e72c38338af7e" >
 label:<name:"backend_target_type" value:"BACKEND_SERVICE" >
 label:<name:"backend_type" value:"INSTANCE_GROUP" >
 label:<name:"cache_result" value:"DISABLED" >
 label:<name:"client_country" value:"Mexico" >
 label:<name:"forwarding_rule_name" value:"k8s-fw-default-identity-aware-proxy--635e72c38338af7e" >
 label:<name:"matched_url_path_rule" value:"UNMATCHED" >
 label:<name:"project_id" value:"cognitedata" >
 label:<name:"protocol" value:"HTTP/1.1" >
 label:<name:"proxy_continent" value:"America" >
 label:<name:"region" value:"global" >
 label:<name:"response_code" value:"404" >
 label:<name:"response_code_class" value:"400" >
 label:<name:"target_proxy_name" value:"k8s-tp-default-identity-aware-proxy--635e72c38338af7e" >
 label:<name:"unit" value:"ms" >
 label:<name:"url_map_name" value:"k8s-um-default-identity-aware-proxy--635e72c38338af7e" >
 histogram:<sample_count:1 sample_sum:196
  bucket:<cumulative_count:0 upper_bound:1 >
  bucket:<cumulative_count:0 upper_bound:1.4 >
  bucket:<cumulative_count:0 upper_bound:1.9599999999999997 >
  bucket:<cumulative_count:0 upper_bound:2.7439999999999993 >
  bucket:<cumulative_count:0 upper_bound:3.841599999999999 >
  bucket:<cumulative_count:0 upper_bound:5.378239999999998 >
  bucket:<cumulative_count:0 upper_bound:7.529535999999997 >
  bucket:<cumulative_count:0 upper_bound:10.541350399999994 >
  bucket:<cumulative_count:0 upper_bound:14.757890559999991 >
  bucket:<cumulative_count:0 upper_bound:20.661046783999986 >
  bucket:<cumulative_count:0 upper_bound:28.92546549759998 >
  bucket:<cumulative_count:0 upper_bound:40.49565169663997 >
  bucket:<cumulative_count:0 upper_bound:56.693912375295945 >
  bucket:<cumulative_count:0 upper_bound:79.37147732541432 >
  bucket:<cumulative_count:0 upper_bound:111.12006825558004 >
  bucket:<cumulative_count:0 upper_bound:155.56809555781203 >
  bucket:<cumulative_count:1 upper_bound:217.79533378093686 >
  bucket:<cumulative_count:1 upper_bound:304.91346729331156 >
  bucket:<cumulative_count:1 upper_bound:426.8788542106362 >
  bucket:<cumulative_count:1 upper_bound:597.6303958948906 >
  bucket:<cumulative_count:1 upper_bound:836.6825542528468 >
  bucket:<cumulative_count:1 upper_bound:1171.3555759539854 >
  bucket:<cumulative_count:1 upper_bound:1639.8978063355794 >
  bucket:<cumulative_count:1 upper_bound:2295.856928869811 >
  bucket:<cumulative_count:1 upper_bound:3214.199700417735 >
  bucket:<cumulative_count:1 upper_bound:4499.879580584829 >
  bucket:<cumulative_count:1 upper_bound:6299.83141281876 >
  bucket:<cumulative_count:1 upper_bound:8819.763977946264 >
  bucket:<cumulative_count:1 upper_bound:12347.669569124768 >
  bucket:<cumulative_count:1 upper_bound:17286.73739677467 >
  bucket:<cumulative_count:1 upper_bound:24201.43235548454 >
  bucket:<cumulative_count:1 upper_bound:33882.00529767835 >
  bucket:<cumulative_count:1 upper_bound:47434.807416749696 >
  bucket:<cumulative_count:1 upper_bound:66408.73038344957 >
  bucket:<cumulative_count:1 upper_bound:92972.22253682939 >
  bucket:<cumulative_count:1 upper_bound:130161.11155156113 >
  bucket:<cumulative_count:1 upper_bound:182225.55617218558 >
  bucket:<cumulative_count:1 upper_bound:255115.7786410598 >
  bucket:<cumulative_count:1 upper_bound:357162.09009748365 >
  bucket:<cumulative_count:1 upper_bound:500026.9261364771 >
  bucket:<cumulative_count:1 upper_bound:700037.696591068 >
  bucket:<cumulative_count:1 upper_bound:980052.775227495 >
  bucket:<cumulative_count:1 upper_bound:1.372073885318493e+06 >
  bucket:<cumulative_count:1 upper_bound:1.92090343944589e+06 >
  bucket:<cumulative_count:1 upper_bound:2.6892648152242457e+06 >
  bucket:<cumulative_count:1 upper_bound:3.7649707413139436e+06 >
  bucket:<cumulative_count:1 upper_bound:5.270959037839521e+06 >
  bucket:<cumulative_count:1 upper_bound:7.379342652975327e+06 >
  bucket:<cumulative_count:1 upper_bound:1.0331079714165458e+07 >
  bucket:<cumulative_count:1 upper_bound:1.446351159983164e+07 >
  bucket:<cumulative_count:1 upper_bound:2.0248916239764296e+07 >
  bucket:<cumulative_count:1 upper_bound:2.8348482735670015e+07 >
  bucket:<cumulative_count:1 upper_bound:3.968787582993802e+07 >
  bucket:<cumulative_count:1 upper_bound:5.5563026161913216e+07 >
  bucket:<cumulative_count:1 upper_bound:7.77882366266785e+07 >
  bucket:<cumulative_count:1 upper_bound:1.0890353127734989e+08 >
  bucket:<cumulative_count:1 upper_bound:1.5246494378828984e+08 >
  bucket:<cumulative_count:1 upper_bound:2.1345092130360574e+08 >
  bucket:<cumulative_count:1 upper_bound:2.98831289825048e+08 >
  bucket:<cumulative_count:1 upper_bound:4.183638057550673e+08 >
  bucket:<cumulative_count:1 upper_bound:5.857093280570941e+08 >
  bucket:<cumulative_count:1 upper_bound:8.199930592799315e+08 >
  bucket:<cumulative_count:1 upper_bound:1.147990282991904e+09 >
  bucket:<cumulative_count:1 upper_bound:1.6071863961886654e+09 >
  bucket:<cumulative_count:1 upper_bound:2.250060954664132e+09 >
  bucket:<cumulative_count:1 upper_bound:3.1500853365297847e+09 >
  bucket:<cumulative_count:1 upper_bound:4.410119471141699e+09 >
  bucket:<cumulative_count:1 upper_bound:inf > >
has label dimensions inconsistent with previously collected metrics in the same metric family
* collected metric stackdriver_https_lb_rule_loadbalancing_googleapis_com_https_total_latencies
 label:<name:"backend_name" value:"k8s-ig--ecf3406460733d56" >
 label:<name:"backend_scope" value:"europe-west1-c" >
 label:<name:"backend_scope_type" value:"ZONE" >
 label:<name:"backend_target_name" value:"k8s-be-31785--ecf3406460733d56" >
 label:<name:"backend_target_type" value:"BACKEND_SERVICE" >
 label:<name:"backend_type" value:"INSTANCE_GROUP" >
 label:<name:"cache_result" value:"DISABLED" >
 label:<name:"client_country" value:"Mexico" >
 label:<name:"forwarding_rule_name" value:"k8s-fw-default-identity-aware-proxy--ecf3406460733d56" >
 label:<name:"matched_url_path_rule" value:"UNMATCHED" >
 label:<name:"project_id" value:"cognitedata" >
 label:<name:"protocol" value:"HTTP/1.1" >
 label:<name:"proxy_continent" value:"America" >
 label:<name:"region" value:"global" >
 label:<name:"response_code" value:"404" >
 label:<name:"response_code_class" value:"400" >
 label:<name:"target_proxy_name" value:"k8s-tp-default-identity-aware-proxy--ecf3406460733d56" >
 label:<name:"unit" value:"ms" >
 label:<name:"url_map_name" value:"k8s-um-default-identity-aware-proxy--ecf3406460733d56" >
 histogram:<sample_count:1 sample_sum:185
  bucket:<cumulative_count:0 upper_bound:1 >
  bucket:<cumulative_count:0 upper_bound:1.4 >
  bucket:<cumulative_count:0 upper_bound:1.9599999999999997 >
  bucket:<cumulative_count:0 upper_bound:2.7439999999999993 >
  bucket:<cumulative_count:0 upper_bound:3.841599999999999 >
  bucket:<cumulative_count:0 upper_bound:5.378239999999998 >
  bucket:<cumulative_count:0 upper_bound:7.529535999999997 >
  bucket:<cumulative_count:0 upper_bound:10.541350399999994 >
  bucket:<cumulative_count:0 upper_bound:14.757890559999991 >
  bucket :<cumulative_count:0 upper_bound:20.661046783999986 >
  bucket :<cumulative_count:0 upper_bound:28.92546549759998 >
  bucket :<cumulative_count:0 upper_bound:40.49565169663997 >
  bucket :<cumulative_count:0 upper_bound:56.693912375295945 >
  bucket :<cumulative_count:0 upper_bound:79.37147732541432 >
  bucket :<cumulative_count:0 upper_bound:111.12006825558004 >
  bucket :<cumulative_count:0 upper_bound:155.56809555781203 >
  bucket :<cumulative_count:1 upper_bound:217.79533378093686 >
  bucket :<cumulative_count:1 upper_bound:304.91346729331156 >
  bucket :<cumulative_count:1 upper_bound:426.8788542106362 >
  bucket :<cumulative_count:1 upper_bound:597.6303958948906 >
  bucket :<cumulative_count:1 upper_bound:836.6825542528468 >
  bucket :<cumulative_count:1 upper_bound:1171.3555759539854 >
  bucket :<cumulative_count:1 upper_bound:1639.8978063355794 >
  bucket :<cumulative_count:1 upper_bound:2295.856928869811 >
  bucket :<cumulative_count:1 upper_bound:3214.199700417735 >
  bucket :<cumulative_count:1 upper_bound:4499.879580584829 >
  bucket :<cumulative_count:1 upper_bound:6299.83141281876 >
  bucket :<cumulative_count:1 upper_bound:8819.763977946264 >
  bucket :<cumulative_count:1 upper_bound:12347.669569124768 >
  bucket :<cumulative_count:1 upper_bound:17286.73739677467 >
  bucket :<cumulative_count:1 upper_bound:24201.43235548454 >
  bucket :<cumulative_count:1 upper_bound:33882.00529767835 >
  bucket :<cumulative_count:1 upper_bound:47434.807416749696 >
  bucket :<cumulative_count:1 upper_bound:66408.73038344957 >
  bucket :<cumulative_count:1 upper_bound:92972.22253682939 >
  bucket :<cumulative_count:1 upper_bound:130161.11155156113 >
  bucket :<cumulative_count:1 upper_bound:182225.55617218558 >
  bucket :<cumulative_count:1 upper_bound:255115.7786410598 >
  bucket :<cumulative_count:1 upper_bound:357162.09009748365 >
  bucket :<cumulative_count:1 upper_bound:500026.9261364771 >
  bucket :<cumulative_count:1 upper_bound:700037.696591068 >
  bucket :<cumulative_count:1 upper_bound:980052.775227495 >
  bucket :<cumulative_count:1 upper_bound:1.372073885318493e+06 >
  bucket :<cumulative_count:1 upper_bound:1.92090343944589e+06 >
  bucket :<cumulative_count:1 upper_bound:2.6892648152242457e+06 >
  bucket :<cumulative_count:1 upper_bound:3.7649707413139436e+06 >
  bucket :<cumulative_count:1 upper_bound:5.270959037839521e+06 >
  bucket :<cumulative_count:1 upper_bound:7.379342652975327e+06 >
  bucket :<cumulative_count:1 upper_bound:1.0331079714165458e+07 >
  bucket :<cumulative_count:1 upper_bound:1.446351159983164e+07 >
  bucket :<cumulative_count:1 upper_bound:2.0248916239764296e+07 >
  bucket :<cumulative_count:1 upper_bound:2.8348482735670015e+07 >
  bucket :<cumulative_count:1 upper_bound:3.968787582993802e+07 >
  bucket :<cumulative_count:1 upper_bound:5.5563026161913216e+07 >
  bucket :<cumulative_count:1 upper_bound:7.77882366266785e+07 >
  bucket :<cumulative_count:1 upper_bound:1.0890353127734989e+08 >
  bucket :<cumulative_count:1 upper_bound:1.5246494378828984e+08 >
  bucket :<cumulative_count:1 upper_bound:2.1345092130360574e+08 >
  bucket :<cumulative_count:1 upper_bound:2.98831289825048e+08 >
  bucket :<cumulative_count:1 upper_bound:4.183638057550673e+08 >
  bucket :<cumulative_count:1 upper_bound:5.857093280570941e+08 >
  bucket :<cumulative_count:1 upper_bound:8.199930592799315e+08 >
  bucket :<cumulative_count:1 upper_bound:1.147990282991904e+09 >
  bucket :<cumulative_count:1 upper_bound:1.6071863961886654e+09 >
  bucket :<cumulative_count:1 upper_bound:2.250060954664132e+09 >
  bucket :<cumulative_count:1 upper_bound:3.1500853365297847e+09 >
  bucket :<cumulative_count:1 upper_bound:4.410119471141699e+09 >
  bucket :<cumulative_count:1 upper_bound:inf > >
has label dimensions inconsistent with previously collected metrics in the same metric family

[everesio]

I checked the histograms and everything seems to be working as expected (assuming that the mentioned metric is always DISTRIBUTION - as it should be due to https://cloud.google.com/monitoring/api/metrics_gcp#gcp-loadbalancing)

Could you build in some traces and provide some raw data for "stackdriver_https_lb_rule_loadbalancing_googleapis_com_https_total_latencies" ?

[olvesh]

After enabling the fill missing flag, I still get some errors, although much less than before. I also hit some rate limits. We have quite a few topics. Would it help to run in debug log level for some time?

time="2018-06-12T19:22:56Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/snapshot/oldest_message_age`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
time="2018-06-12T19:22:56Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/streaming_pull_mod_ack_deadline_request_count`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
time="2018-06-12T19:22:56Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/streaming_pull_ack_message_operation_count`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
time="2018-06-12T19:22:56Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/streaming_pull_response_count`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
time="2018-06-12T19:22:56Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/message_sizes`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
time="2018-06-12T19:22:56Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/num_unacked_messages_by_region`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
time="2018-06-12T19:22:56Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/byte_cost`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
time="2018-06-12T19:22:56Z" level=error msg="Error while getting Google Stackdriver Monitoring metrics: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:132"
time="2018-06-12T19:38:54Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/oldest_retained_acked_message_age_by_region`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
time="2018-06-12T19:38:54Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/num_undelivered_messages`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
time="2018-06-12T19:38:54Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/snapshot/num_messages`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
time="2018-06-12T19:38:54Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/streaming_pull_response_count`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
time="2018-06-12T19:38:54Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/mod_ack_deadline_request_count`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
time="2018-06-12T19:38:54Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/retained_acked_bytes_by_region`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
time="2018-06-12T19:38:54Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/streaming_pull_ack_message_operation_count`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
time="2018-06-12T19:38:54Z" level=error msg="Error while getting Google Stackdriver Monitoring metrics: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:132"
time="2018-06-12T19:41:56Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/snapshot/num_messages`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
time="2018-06-12T19:41:56Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/num_retained_acked_messages_by_region`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
time="2018-06-12T19:41:56Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/send_request_count`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
time="2018-06-12T19:41:56Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/snapshot/num_messages_by_region`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
time="2018-06-12T19:41:56Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/snapshot/backlog_bytes_by_region`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
time="2018-06-12T19:41:56Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/num_undelivered_messages`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
time="2018-06-12T19:41:56Z" level=error msg="Error while getting Google Stackdriver Monitoring metrics: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:132"
time="2018-06-12T19:42:56Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/oldest_retained_acked_message_age`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
time="2018-06-12T19:42:56Z" level=error msg="Error while getting Google Stackdriver Monitoring metrics: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:132"
time="2018-06-12T19:44:56Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/unacked_bytes_by_region`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
time="2018-06-12T19:44:56Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/byte_cost`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
time="2018-06-12T19:44:56Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/streaming_pull_ack_message_operation_count`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
time="2018-06-12T19:44:56Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/num_unacked_messages_by_region`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
time="2018-06-12T19:44:56Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/snapshot/oldest_message_age`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
time="2018-06-12T19:44:56Z" level=error msg="Error while getting Google Stackdriver Monitoring metrics: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:132"
time="2018-06-12T22:42:26Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/mod_ack_deadline_message_operation_count`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
time="2018-06-12T22:42:26Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/message_sizes`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
time="2018-06-12T22:42:26Z" level=error msg="Error while getting Google Stackdriver Monitoring metrics: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:132"
time="2018-06-12T22:46:26Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/snapshot/oldest_message_age_by_region`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
time="2018-06-12T22:46:26Z" level=error msg="Error while getting Google Stackdriver Monitoring metrics: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:132"
time="2018-06-12T22:56:26Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/unacked_bytes_by_region`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
time="2018-06-12T22:56:26Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/byte_cost`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
time="2018-06-12T22:56:26Z" level=error msg="Error while getting Google Stackdriver Monitoring metrics: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:132"
time="2018-06-12T23:07:24Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/snapshot/num_messages_by_region`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
time="2018-06-12T23:07:24Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/snapshot/oldest_message_age`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
time="2018-06-12T23:07:24Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/unacked_bytes_by_region`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
time="2018-06-12T23:07:24Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/num_retained_acked_messages_by_region`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
time="2018-06-12T23:07:24Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/message_sizes`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
time="2018-06-12T23:07:24Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/snapshot/config_updates_count`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
time="2018-06-12T23:07:24Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/snapshot/backlog_bytes`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
time="2018-06-12T23:07:24Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/oldest_unacked_message_age_by_region`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
time="2018-06-12T23:07:24Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/streaming_pull_message_operation_count`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
time="2018-06-12T23:07:24Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/streaming_pull_response_count`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
time="2018-06-12T23:07:24Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/unacked_bytes_by_region`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
time="2018-06-12T23:07:24Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/streaming_pull_ack_message_operation_count`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
time="2018-06-12T23:07:24Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/oldest_retained_acked_message_age_by_region`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
time="2018-06-12T23:07:24Z" level=error msg="Error while getting Google Stackdriver Monitoring metrics: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:132"
time="2018-06-12T23:07:26Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/num_unacked_messages_by_region`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
time="2018-06-12T23:07:26Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/push_request_latencies`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
time="2018-06-12T23:07:26Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/num_outstanding_messages`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
time="2018-06-12T23:07:26Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/push_request_count`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
time="2018-06-12T23:07:26Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/snapshot/oldest_message_age_by_region`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
time="2018-06-12T23:07:26Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/streaming_pull_mod_ack_deadline_message_operation_count`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
time="2018-06-12T23:07:26Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/mod_ack_deadline_message_operation_count`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
time="2018-06-12T23:07:26Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/streaming_pull_message_operation_count`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
time="2018-06-12T23:07:26Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/oldest_retained_acked_message_age_by_region`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
time="2018-06-12T23:07:26Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/backlog_bytes`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
time="2018-06-12T23:07:26Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/num_retained_acked_messages`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
time="2018-06-12T23:07:26Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/num_retained_acked_messages_by_region`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
time="2018-06-12T23:07:26Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/unacked_bytes_by_region`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
time="2018-06-12T23:07:26Z" level=error msg="Error while getting Google Stackdriver Monitoring metrics: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:132"
time="2018-06-12T23:17:24Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/oldest_unacked_message_age_by_region`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
time="2018-06-12T23:17:24Z" level=error msg="Error while getting Google Stackdriver Monitoring metrics: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:132"
time="2018-06-12T23:28:26Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/streaming_pull_ack_message_operation_count`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
time="2018-06-12T23:28:26Z" level=error msg="Error while getting Google Stackdriver Monitoring metrics: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:132"
time="2018-06-12T23:34:24Z" level=error msg="Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/mod_ack_deadline_request_count`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
time="2018-06-12T23:34:24Z" level=error msg="Error while getting Google Stackdriver Monitoring metrics: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:132"

[olvesh]

To get a sense of what is failing the most (logs from 25. May to 28 of June):

   1 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/snapshot/backlog_bytes_by_region`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
   1 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/snapshot/backlog_bytes_by_region`: googleapi: Error 500: An internal error occurred., backendError" source="monitoring_collector.go:176"
   1 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/snapshot/num_messages_by_region`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
   1 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/snapshot/num_messages`: googleapi: Error 500: Authentication backend unknown error., backendError" source="monitoring_collector.go:176"
   1 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/snapshot/oldest_message_age_by_region`: googleapi: Error 500: An internal error occurred., backendError" source="monitoring_collector.go:176"
   1 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/backlog_bytes`: googleapi: Error 500: An internal error occurred., backendError" source="monitoring_collector.go:176"
   1 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/config_updates_count`: googleapi: Error 500: Authentication backend unknown error., backendError" source="monitoring_collector.go:176"
   1 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/mod_ack_deadline_request_count`: googleapi: Error 500: An internal error occurred., backendError" source="monitoring_collector.go:176"
   1 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/num_outstanding_messages`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
   1 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/num_unacked_messages_by_region`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
   1 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/oldest_retained_acked_message_age`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
   1 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/oldest_unacked_message_age`: googleapi: Error 500: Authentication backend unknown error., backendError" source="monitoring_collector.go:176"
   1 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/pull_ack_message_operation_count`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
   1 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/pull_ack_message_operation_count`: googleapi: Error 500: Authentication backend unknown error., backendError" source="monitoring_collector.go:176"
   1 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/pull_ack_request_count`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
   1 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/retained_acked_bytes_by_region`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
   1 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/retained_acked_bytes`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
   1 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/streaming_pull_message_operation_count`: googleapi: Error 500: An internal error occurred., backendError" source="monitoring_collector.go:176"
   1 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/streaming_pull_mod_ack_deadline_message_operation_count`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
   1 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/streaming_pull_mod_ack_deadline_message_operation_count`: googleapi: Error 500: An internal error occurred., backendError" source="monitoring_collector.go:176"
   1 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/unacked_bytes_by_region`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
   1 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/byte_cost`: googleapi: Error 500: Authentication backend unknown error., backendError" source="monitoring_collector.go:176"
   1 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/message_sizes`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
   1 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/num_retained_acked_messages_by_region`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
   1 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/num_retained_acked_messages_by_region`: googleapi: Error 500: An internal error occurred., backendError" source="monitoring_collector.go:176"
   1 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/num_unacked_messages_by_region`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
   1 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/oldest_retained_acked_message_age_by_region`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
   1 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/retained_acked_bytes_by_region`: googleapi: Error 500: Authentication backend unknown error., backendError" source="monitoring_collector.go:176"
   1 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/send_request_count`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
   1 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/unacked_bytes_by_region`: googleapi: Error 500: An internal error occurred., backendError" source="monitoring_collector.go:176"
   2 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/snapshot/num_messages`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
   2 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/config_updates_count`: googleapi: Error 500: An internal error occurred., backendError" source="monitoring_collector.go:176"
   2 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/mod_ack_deadline_request_count`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
   2 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/num_undelivered_messages`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
   2 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/oldest_unacked_message_age_by_region`: googleapi: Error 500: Authentication backend unknown error., backendError" source="monitoring_collector.go:176"
   2 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/push_request_latencies`: googleapi: Error 500: An internal error occurred., backendError" source="monitoring_collector.go:176"
   2 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/streaming_pull_mod_ack_deadline_request_count`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
   2 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/streaming_pull_response_count`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
   2 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/byte_cost`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
   2 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/config_updates_count`: googleapi: Error 500: An internal error occurred., backendError" source="monitoring_collector.go:176"
   2 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/retained_acked_bytes_by_region`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   2 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/send_message_operation_count`: googleapi: Error 500: An internal error occurred., backendError" source="monitoring_collector.go:176"
   3 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/snapshot/config_updates_count`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   3 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/snapshot/oldest_message_age`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
   3 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/num_retained_acked_messages`: googleapi: Error 500: An internal error occurred., backendError" source="monitoring_collector.go:176"
   3 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/num_undelivered_messages`: googleapi: Error 500: An internal error occurred., backendError" source="monitoring_collector.go:176"
   3 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/pull_ack_message_operation_count`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   3 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/streaming_pull_ack_message_operation_count`: googleapi: Error 429: The query rate is too high., rateLimitExceeded" source="monitoring_collector.go:176"
   3 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/unacked_bytes_by_region`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   4 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/num_unacked_messages_by_region`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   4 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/oldest_retained_acked_message_age_by_region`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   4 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/byte_cost`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   4 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/oldest_unacked_message_age_by_region`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   4 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/send_request_count`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   5 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/pull_ack_request_count`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   5 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/streaming_pull_ack_message_operation_count`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   5 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/config_updates_count`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   5 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/num_retained_acked_messages_by_region`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   5 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/num_unacked_messages_by_region`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   5 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/oldest_retained_acked_message_age_by_region`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   6 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/snapshot/backlog_bytes`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   6 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/snapshot/num_messages`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   6 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/byte_cost`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   6 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/config_updates_count`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   6 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/num_retained_acked_messages_by_region`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   6 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/oldest_retained_acked_message_age`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   6 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/pull_request_count`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   6 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/retained_acked_bytes_by_region`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   6 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/streaming_pull_ack_request_count`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   6 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/streaming_pull_mod_ack_deadline_message_operation_count`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   7 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/snapshot/oldest_message_age_by_region`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   7 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/backlog_bytes`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   7 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/num_retained_acked_messages`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   7 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/oldest_unacked_message_age_by_region`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   7 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/pull_message_operation_count`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   7 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/streaming_pull_mod_ack_deadline_request_count`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   7 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/streaming_pull_response_count`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   7 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/send_message_operation_count`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   8 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/snapshot/num_messages_by_region`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   8 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/num_undelivered_messages`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   8 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/oldest_unacked_message_age`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   8 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/push_request_latencies`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   8 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/message_sizes`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   8 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/topic/unacked_bytes_by_region`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   9 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/snapshot/backlog_bytes_by_region`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
   9 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/push_request_count`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
  10 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/mod_ack_deadline_message_operation_count`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
  10 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/mod_ack_deadline_request_count`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
  10 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/streaming_pull_message_operation_count`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
  11 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/retained_acked_bytes`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
  12 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/snapshot/oldest_message_age`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"
  12 Error retrieving Time Series metrics for descriptor `pubsub.googleapis.com/subscription/num_outstanding_messages`: googleapi: Error 503: The service is currently unavailable., backendError" source="monitoring_collector.go:176"```

[olvesh]

It just struck me that these are 503s we are getting from the Google APIs. Not 503s from stackdriver_exporter...

[olvesh]

I have used this branch in production many months now, and it looks ok.
To further fix my problem above regarding 503 errors, consider merging my pr #35