Metric sink does not write timestamps for Prometheus

[djsunn]

When metricSinks.prometheusScrapingEndpoint.enableMetricTimestamps is set to true in the runtime config, the timestamp of the scrape is not reflected when calling /metrics.

Expected Behavior

# HELP azure_network_interface_bytes_received_rate Number of bytes the network interface received
# TYPE azure_network_interface_bytes_received_rate gauge
azure_network_interface_bytes_received_rate{resource_group="dsunnerer-dp200-1",subscription_id="xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",resource_uri="subscriptions/xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/dsunnerer-dp200-1/providers/Microsoft.Network/networkInterfaces/test-dsunnerer913",instance_name="test-dsunnerer913"} 174316.6 1597405378
# HELP promitor_ratelimit_arm Indication how many calls are still available before Azure Resource Manager is going to throttle us.
# TYPE promitor_ratelimit_arm gauge
promitor_ratelimit_arm{tenant_id="xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",subscription_id="xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",app_id="xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"} 11976 1597405378

Actual Behavior

# HELP azure_network_interface_bytes_received_rate Number of bytes the network interface received
# TYPE azure_network_interface_bytes_received_rate gauge
azure_network_interface_bytes_received_rate{resource_group="dsunnerer-dp200-1",subscription_id="xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",resource_uri="subscriptions/xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/dsunnerer-dp200-1/providers/Microsoft.Network/networkInterfaces/test-dsunnerer913",instance_name="test-dsunnerer913"} 174316.6
# HELP promitor_ratelimit_arm Indication how many calls are still available before Azure Resource Manager is going to throttle us.
# TYPE promitor_ratelimit_arm gauge
promitor_ratelimit_arm{tenant_id="xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",subscription_id="xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",app_id="xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"} 11976

Steps to Reproduce the Problem

1. Set metricSinks.prometheusScrapingEndpoint.enableMetricTimestamps in the runtime configuration
2. Start the container
3. Call /metrics

Configuration

Provide insights in the configuration that you are using:

Used runtime configuration

server:
  httpPort: 80 # Optional. Default: 80
metricSinks:
  prometheusScrapingEndpoint:
    metricUnavailableValue: NaN # Optional. Default: NaN
    enableMetricTimestamps: true # Optional. Default: true
    baseUriPath: /metrics # Optional. Default: /metrics
metricsConfiguration:
  absolutePath: /config/metrics-declaration.yaml # Optional. Default: /config/metrics-declaration.yaml
telemetry:
  applicationInsights:
    instrumentationKey: ABC # Optional. Note: Required to be specified when turned on
    isEnabled: false # Optional. Default: false
    verbosity: trace # Optional. Default: N/A
  containerLogs:
    isEnabled: true # Optional. Default: true
    verbosity: error # Optional. Default: N/A
  defaultVerbosity: error # Optional. Default: error
azureMonitor:
  logging:
    informationalLevel: BodyAndHeaders
    isEnabled: true
resourceDiscovery:
  #host:
  #port:

Output when calling `/api/v1/configuration/runtime`

{
  "azureMonitor": {
    "logging": {
      "informationLevel": "Basic",
      "isEnabled": true
    }
  },
  "metricsConfiguration": {
    "absolutePath": "/config/metrics-declaration.yaml"
  },
  "metricSinks": {
    "prometheusScrapingEndpoint": {
      "baseUriPath": "/metrics",
      "metricUnavailableValue": "NaN",
      "enableMetricTimestamps": true
    }
  },
  "server": {
    "httpPort": 80
  },
  "telemetry": {
    "defaultVerbosity": "Error",
    "containerLogs": {
      "verbosity": "Error",
      "isEnabled": true
    },
    "applicationInsights": {
      "verbosity": "Trace",
      "isEnabled": false,
      "instrumentationKey": "***"
    }
  }
}

Used metrics configuration

---
version: v1
azureMetadata:
  tenantId: "xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
  subscriptionId: "xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
  resourceGroupName: dsunnerer-dp200-1
metricDefaults:
  aggregation:
    interval: 00:05:00
  scraping:
    # Every minute
    schedule: "0 * * ? * *"
metrics:
  - name: azure_network_interface_bytes_received_rate
    description: "Number of bytes the network interface received"
    resourceType: NetworkInterface
    azureMetricConfiguration:
      metricName: BytesReceivedRate
      aggregation:
        type: Average
    resources:
      - networkInterfaceName: test-dsunnerer913
        resourceGroupName: dsunnerer-dp200-1

Specifications

• Version: 1.6.1 and 2.0.0-preview-2-linux

[tomkerkhove]

Dammit, sorry about that :/

[tomkerkhove]

Will be shipped in 2.0 and you can use tomkerkhove/promitor-agent-scraper-ci:pr1218-linux for now!

[tomkerkhove]

Sorry for the inconvenience.