Skip to content

Commit

Permalink
Provide support for multi-dimensional metrics (#807)
Browse files Browse the repository at this point in the history
* Document new feature

Signed-off-by: Tom Kerkhove <[email protected]>

* Update changelog

Signed-off-by: Tom Kerkhove <[email protected]>

* Allow customers to configure a dimension

Signed-off-by: Tom Kerkhove <[email protected]>

* Provide support for multi-dimensional metrics

Signed-off-by: Tom Kerkhove <[email protected]>

* More local tracing

Signed-off-by: Tom Kerkhove <[email protected]>

* Use new doc endpoint

Signed-off-by: Tom Kerkhove <[email protected]>

* Improve logging

Signed-off-by: Tom Kerkhove <[email protected]>

* Block EntityPath dimension for Service Bus for now

Signed-off-by: Tom Kerkhove <[email protected]>

* Add ACI to local testing

* Remove usings

Signed-off-by: Tom Kerkhove <[email protected]>

* Markdown linting

Signed-off-by: Tom Kerkhove <[email protected]>

* Markdown linting

Signed-off-by: Tom Kerkhove <[email protected]>

* Update docs

Signed-off-by: Tom Kerkhove <[email protected]>

* Uncomment sample dimension

* Move azureMetricConfiguration.dimensionName to azureMetricConfiguration.dimension.Name

Signed-off-by: Tom Kerkhove <[email protected]>
  • Loading branch information
tomkerkhove authored Jan 5, 2020
1 parent 42959a4 commit 6c390c9
Show file tree
Hide file tree
Showing 39 changed files with 482 additions and 84 deletions.
2 changes: 2 additions & 0 deletions changelog/content/experimental/unreleased.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,11 @@ weight: 1
version:
---

- {{% tag added %}} Multi-dimensional metric support ([docs](https://promitor.io/configuration/v1.x/metrics/#metrics) | [#81](https://github.com/tomkerkhove/promitor/issues/81))
- {{% tag added %}} Azure SQL Database Scraper ([docs](https://promitor.io/configuration/v1.x/metrics/sql-database) | [#317](https://github.com/tomkerkhove/promitor/issues/317))
- {{% tag added %}} OpenAPI v3.0 support (`/api/v1/docs.json` | [docs](ttps://promitor.io/operations/#exploring-our-rest-apis) | [#734](https://github.com/tomkerkhove/promitor/issues/734))
- {{% tag added %}} OpenAPI UI based on OpenAPI v3.0 (`/api/docs/` | [docs](ttps://promitor.io/operations/#exploring-our-rest-apis) | [#734](https://github.com/tomkerkhove/promitor/issues/734))
- {{% tag added %}} Provide traces in Azure Application Insights ([docs](https://promitor.io/configuration/v1.x/runtime#azure-application-insights) | [#29](https://github.com/tomkerkhove/promitor/issues/29))
- {{% tag added %}} Introduce Frequently asked questions (FAQs) in documentation ([FAQs](https://promitor.io/faq))
- {{% tag changed %}} Helm Chart creates apps/v1 Deployment instead of v1beta1 ([#669](https://github.com/tomkerkhove/promitor/issues/669))
- {{% tag changed %}} Provide exceptions in Azure Application Insights for all exceptions, not just scraping ([docs](https://promitor.io/configuration/v1.x/runtime#azure-application-insights) | [#29](https://github.com/tomkerkhove/promitor/issues/29))
7 changes: 7 additions & 0 deletions docs/configuration/v1.x/metrics/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,11 @@ Every metric that is being declared needs to define the following fields:

Additionally, the following fields are optional:

- `azureMetricConfiguration.dimension.Name` - The name of the dimension that should
be used to scrape a multi-dimensional metric in Azure Monitor.
-*Promitor simply acts as a proxy and will not validate if it's supported or
not, we recommend verifying that the dimension is supported in the
[official documentation](https://docs.microsoft.com/en-us/azure/azure-monitor/platform/metrics-supported)*
- `scraping.schedule` - A scraping schedule for the individual metric; overrides
the the one specified in `metricDefaults`

Expand Down Expand Up @@ -83,6 +88,8 @@ metrics:
schedule: "0 */2 * ? * *"
azureMetricConfiguration:
metricName: ActiveMessages
dimension:
name: <dimension-name>
aggregation:
type: Total
interval: 00:15:00
Expand Down
4 changes: 4 additions & 0 deletions docs/configuration/v1.x/metrics/service-bus-queue.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,10 @@ The following scraper-specific metric label will be added:

- `entity_name` - Name of the queue

Notes:

- We currently do not support `EntityPath` as a dimension due to internal limitations.

Example:

```yaml
Expand Down
19 changes: 19 additions & 0 deletions docs/faq.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
---
layout: default
title: Frequently asked questions (FAQs)
---

## Are multi-dimensional metrics supported?

Yes, every scraper supports scraping multi-dimensional metrics except for
Azure Storage queues.

You can configure the dimension you are interested in via
`azureMetricConfiguration.dimension.Name`, for more information see
our ['Metric Configuration' page](/configuration/v1.x/metrics/#metrics).

However, you can only use it with metrics in Azure Monitor that support this,
for a complete overview we recommend reading the
[official documentation](https://docs.microsoft.com/en-us/azure/azure-monitor/platform/metrics-supported).

[&larr; back](/)
3 changes: 2 additions & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ Docker image is available on [Docker Hub](https://hub.docker.com/r/tomkerkhove/p
## Features

- Provides scraping endpoint for Prometheus
- Automatically scrapes Azure Monitor metrics
- Automatically scrapes Azure Monitor metrics (single and multi-dimensional)
- Built-in support for a variety of Azure services ([overview](configuration/v1.x/metrics#supported-azure-services))
- Easy to declare metrics to scrape via YAML & APIs
- Easily deployable via Docker & Kubernetes
Expand Down Expand Up @@ -63,6 +63,7 @@ and vote for features!
- [Health](operations#health)
- **Walkthroughs**
- [Deploying Promitor, Prometheus, and Grafana on an AKS Cluster](/walkthrough)
- [**Frequently asked questions (FAQs)**](/faq)

## Support

Expand Down
8 changes: 8 additions & 0 deletions docs/metrics/labels.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ Here is an overview of how we label metrics.
There are a couple of scenarios where labels are being added:

- Built-in labels
- Metric dimension labels
- Scaler-specific labels
- Bring-your-own labels

Expand All @@ -23,6 +24,13 @@ built-in labels:
- `resource_group` - Name of the resource group.
- `instance_name` - Name of the instance, if applicable.

## Metric dimension labels

Metrics support specifying a dimension which will be scraped in Azure Monitor.

Every metric value will be reported under the configured metric name, but a
label for the dimension and it's respective value will be added.

## Scraper-specific labels

Every scraper can provide additional labels to provide more information.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,11 @@ public class AzureMetricConfiguration
/// </summary>
public string MetricName { get; set; }

/// <summary>
/// Information about the dimension of an Azure Monitor metric
/// </summary>
public MetricDimension Dimension { get; set; }

/// <summary>
/// Configuration on how to aggregate the metric
/// </summary>
Expand Down
13 changes: 13 additions & 0 deletions src/Promitor.Core.Scraping/Configuration/Model/MetricDimension.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
namespace Promitor.Core.Scraping.Configuration.Model
{
/// <summary>
/// Information about the dimension of an Azure Monitor metric
/// </summary>
public class MetricDimension
{
/// <summary>
/// Name of the dimension
/// </summary>
public string Name { get; set; }
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,15 @@ namespace Promitor.Core.Scraping.Configuration.Serialization.v1.Core
public class AzureMetricConfigurationDeserializer : Deserializer<AzureMetricConfigurationV1>
{
private const string MetricNameTag = "metricName";
private const string DimensionTag = "dimension";
private const string AggregationTag = "aggregation";
private readonly IDeserializer<MetricDimensionV1> _dimensionDeserializer;
private readonly IDeserializer<MetricAggregationV1> _aggregationDeserializer;

public AzureMetricConfigurationDeserializer(IDeserializer<MetricAggregationV1> aggregationDeserializer, ILogger<AzureMetricConfigurationDeserializer> logger)
public AzureMetricConfigurationDeserializer(IDeserializer<MetricDimensionV1> dimensionDeserializer, IDeserializer<MetricAggregationV1> aggregationDeserializer, ILogger<AzureMetricConfigurationDeserializer> logger)
: base(logger)
{
_dimensionDeserializer = dimensionDeserializer;
_aggregationDeserializer = aggregationDeserializer;
}

Expand All @@ -21,6 +24,7 @@ public override AzureMetricConfigurationV1 Deserialize(YamlMappingNode node)
return new AzureMetricConfigurationV1
{
MetricName = node.GetString(MetricNameTag),
Dimension = DeserializeDimension(node),
Aggregation = DeserializeAggregation(node)
};
}
Expand All @@ -29,7 +33,17 @@ private MetricAggregationV1 DeserializeAggregation(YamlMappingNode node)
{
if (node.Children.TryGetValue(AggregationTag, out var aggregationNode))
{
return _aggregationDeserializer.Deserialize((YamlMappingNode) aggregationNode);
return _aggregationDeserializer.Deserialize((YamlMappingNode)aggregationNode);
}

return null;
}

private MetricDimensionV1 DeserializeDimension(YamlMappingNode node)
{
if (node.Children.TryGetValue(DimensionTag, out var aggregationNode))
{
return _dimensionDeserializer.Deserialize((YamlMappingNode)aggregationNode);
}

return null;
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
using Microsoft.Extensions.Logging;
using Promitor.Core.Scraping.Configuration.Serialization.v1.Model;
using YamlDotNet.RepresentationModel;

namespace Promitor.Core.Scraping.Configuration.Serialization.v1.Core
{
public class MetricDimensionDeserializer : Deserializer<MetricDimensionV1>
{
private const string NameTag = "name";

public MetricDimensionDeserializer(ILogger<MetricDimensionDeserializer> logger)
: base(logger)
{
}

public override MetricDimensionV1 Deserialize(YamlMappingNode node)
{
return new MetricDimensionV1
{
Name = node.GetString(NameTag),
};
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ public V1MappingProfile()
CreateMap<AzureMetadataV1, AzureMetadata>();
CreateMap<MetricDefaultsV1, MetricDefaults>();
CreateMap<AggregationV1, Aggregation>();
CreateMap<MetricDimensionV1, MetricDimension>();
CreateMap<ScrapingV1, Configuration.Model.Scraping>();
CreateMap<AzureMetricConfigurationV1, AzureMetricConfiguration>();
CreateMap<MetricAggregationV1, MetricAggregation>();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,11 @@ public class AzureMetricConfigurationV1
/// </summary>
public string MetricName { get; set; }

/// <summary>
/// Information about the dimension of an Azure Monitor metric
/// </summary>
public MetricDimensionV1 Dimension { get; set; }

/// <summary>
/// The settings for how the metric should be aggregated before being returned from Azure.
/// </summary>
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
namespace Promitor.Core.Scraping.Configuration.Serialization.v1.Model
{
/// <summary>
/// Information about the dimension of an Azure Monitor metric
/// </summary>
public class MetricDimensionV1
{
/// <summary>
/// Name of the dimension
/// </summary>
public string Name { get; set; }
}
}
24 changes: 17 additions & 7 deletions src/Promitor.Core.Scraping/Prometheus/PrometheusMetricWriter.cs
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
using Promitor.Core.Configuration.Model.Prometheus;
using Promitor.Core.Scraping.Configuration.Model.Metrics;
using Promitor.Core.Scraping.Prometheus.Interfaces;
using Promitor.Integrations.AzureMonitor;

namespace Promitor.Core.Scraping.Prometheus
{
Expand All @@ -28,23 +29,32 @@ public PrometheusMetricWriter(IOptionsMonitor<PrometheusConfiguration> prometheu
public void ReportMetric(PrometheusMetricDefinition metricDefinition, ScrapeResult scrapedMetricResult)
{
var enableMetricTimestamps = _prometheusConfiguration.CurrentValue.EnableMetricTimestamps;
var labels = DetermineLabels(metricDefinition, scrapedMetricResult);

var gauge = Metrics.CreateGauge(metricDefinition.Name, metricDefinition.Description, includeTimestamp: enableMetricTimestamps, labelNames: labels.Names);
var metricValue = DetermineMetricMeasurement(scrapedMetricResult);
gauge.WithLabels(labels.Values).Set(metricValue);
foreach (var measuredMetric in scrapedMetricResult.MetricValues)
{
var measuredMetricValue = DetermineMetricMeasurement(measuredMetric);
var labels = DetermineLabels(metricDefinition, scrapedMetricResult, measuredMetric);

var gauge = Metrics.CreateGauge(metricDefinition.Name, metricDefinition.Description, includeTimestamp: enableMetricTimestamps, labelNames: labels.Names);
gauge.WithLabels(labels.Values).Set(measuredMetricValue);
}
}

private double DetermineMetricMeasurement(ScrapeResult scrapedMetricResult)
private double DetermineMetricMeasurement(MeasuredMetric scrapedMetricResult)
{
var metricUnavailableValue = _prometheusConfiguration.CurrentValue?.MetricUnavailableValue ?? Defaults.Prometheus.MetricUnavailableValue;
return scrapedMetricResult.MetricValue ?? metricUnavailableValue;
return scrapedMetricResult.Value ?? metricUnavailableValue;
}

private (string[] Names, string[] Values) DetermineLabels(PrometheusMetricDefinition metricDefinition, ScrapeResult scrapeResult)
private (string[] Names, string[] Values) DetermineLabels(PrometheusMetricDefinition metricDefinition, ScrapeResult scrapeResult, MeasuredMetric measuredMetric)
{
var labels = new Dictionary<string, string>(scrapeResult.Labels);

if (measuredMetric.IsDimensional)
{
labels.Add(measuredMetric.DimensionName, measuredMetric.DimensionValue);
}

if (metricDefinition?.Labels?.Any() == true)
{
foreach (var customLabel in metricDefinition.Labels)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@ protected override async Task<ScrapeResult> ScrapeResourceAsync(string subscript
var resourceUri = string.Format(ResourceUriTemplate, AzureMetadata.SubscriptionId, scrapeDefinition.ResourceGroupName, resource.ContainerGroup);

var metricName = scrapeDefinition.AzureMetricConfiguration.MetricName;
var foundMetricValue = await AzureMonitorClient.QueryMetricAsync(metricName, aggregationType, aggregationInterval, resourceUri);
var dimensionName = scrapeDefinition.AzureMetricConfiguration.Dimension?.Name;
var foundMetricValue = await AzureMonitorClient.QueryMetricAsync(metricName,dimensionName, aggregationType, aggregationInterval, resourceUri);

return new ScrapeResult(subscriptionId, scrapeDefinition.ResourceGroupName, resource.ContainerGroup, resourceUri, foundMetricValue);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@ protected override async Task<ScrapeResult> ScrapeResourceAsync(string subscript
var resourceUri = string.Format(ResourceUriTemplate, subscriptionId, scrapeDefinition.ResourceGroupName, resource.RegistryName);

var metricName = scrapeDefinition.AzureMetricConfiguration.MetricName;
var foundMetricValue = await AzureMonitorClient.QueryMetricAsync(metricName, aggregationType, aggregationInterval, resourceUri);
var dimensionName = scrapeDefinition.AzureMetricConfiguration.Dimension?.Name;
var foundMetricValue = await AzureMonitorClient.QueryMetricAsync(metricName,dimensionName, aggregationType, aggregationInterval, resourceUri);

return new ScrapeResult(subscriptionId, scrapeDefinition.ResourceGroupName, resource.RegistryName, resourceUri, foundMetricValue);
}
Expand Down
3 changes: 2 additions & 1 deletion src/Promitor.Core.Scraping/ResourceTypes/CosmosDbScraper.cs
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@ protected override async Task<ScrapeResult> ScrapeResourceAsync(string subscript
var resourceUri = string.Format(ResourceUriTemplate, subscriptionId, scrapeDefinition.ResourceGroupName, resource.DbName);

var metricName = scrapeDefinition.AzureMetricConfiguration.MetricName;
var foundMetricValue = await AzureMonitorClient.QueryMetricAsync(metricName, aggregationType, aggregationInterval, resourceUri);
var dimensionName = scrapeDefinition.AzureMetricConfiguration.Dimension?.Name;
var foundMetricValue = await AzureMonitorClient.QueryMetricAsync(metricName, dimensionName, aggregationType, aggregationInterval, resourceUri);

return new ScrapeResult(subscriptionId, scrapeDefinition.ResourceGroupName, resource.DbName, resourceUri, foundMetricValue);
}
Expand Down
3 changes: 2 additions & 1 deletion src/Promitor.Core.Scraping/ResourceTypes/GenericScraper.cs
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@ protected override async Task<ScrapeResult> ScrapeResourceAsync(string subscript
{
var resourceUri = string.Format(ResourceUriTemplate, subscriptionId, scrapeDefinition.ResourceGroupName, resource.ResourceUri);
var metricName = scrapeDefinition.AzureMetricConfiguration.MetricName;
var foundMetricValue = await AzureMonitorClient.QueryMetricAsync(metricName, aggregationType, aggregationInterval, resourceUri, resource.Filter);
var dimensionName = scrapeDefinition.AzureMetricConfiguration.Dimension?.Name;
var foundMetricValue = await AzureMonitorClient.QueryMetricAsync(metricName,dimensionName, aggregationType, aggregationInterval, resourceUri, resource.Filter);

return new ScrapeResult(subscriptionId, scrapeDefinition.ResourceGroupName, resourceUri, foundMetricValue);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@ protected override async Task<ScrapeResult> ScrapeResourceAsync(string subscript
var resourceUri = string.Format(ResourceUriTemplate, AzureMetadata.SubscriptionId, scrapeDefinition.ResourceGroupName, resource.NetworkInterfaceName);

var metricName = scrapeDefinition.AzureMetricConfiguration.MetricName;
var foundMetricValue = await AzureMonitorClient.QueryMetricAsync(metricName, aggregationType, aggregationInterval, resourceUri);
var dimensionName = scrapeDefinition.AzureMetricConfiguration.Dimension?.Name;
var foundMetricValue = await AzureMonitorClient.QueryMetricAsync(metricName, dimensionName, aggregationType, aggregationInterval, resourceUri);

return new ScrapeResult(subscriptionId, scrapeDefinition.ResourceGroupName, resource.NetworkInterfaceName, resourceUri, foundMetricValue);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@ protected override async Task<ScrapeResult> ScrapeResourceAsync(string subscript
var resourceUri = string.Format(ResourceUriTemplate, subscriptionId, scrapeDefinition.ResourceGroupName, resource.ServerName);

var metricName = scrapeDefinition.AzureMetricConfiguration.MetricName;
var foundMetricValue = await AzureMonitorClient.QueryMetricAsync(metricName, aggregationType, aggregationInterval, resourceUri);
var dimensionName = scrapeDefinition.AzureMetricConfiguration.Dimension?.Name;
var foundMetricValue = await AzureMonitorClient.QueryMetricAsync(metricName,dimensionName, aggregationType, aggregationInterval, resourceUri);

return new ScrapeResult(subscriptionId, scrapeDefinition.ResourceGroupName, resource.ServerName, resourceUri, foundMetricValue);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@ protected override async Task<ScrapeResult> ScrapeResourceAsync(string subscript
var resourceUri = string.Format(ResourceUriTemplate, subscriptionId, scrapeDefinition.ResourceGroupName, resource.CacheName);

var metricName = scrapeDefinition.AzureMetricConfiguration.MetricName;
var foundMetricValue = await AzureMonitorClient.QueryMetricAsync(metricName, aggregationType, aggregationInterval, resourceUri);
var dimensionName = scrapeDefinition.AzureMetricConfiguration.Dimension?.Name;
var foundMetricValue = await AzureMonitorClient.QueryMetricAsync(metricName,dimensionName, aggregationType, aggregationInterval, resourceUri);

return new ScrapeResult(subscriptionId, scrapeDefinition.ResourceGroupName, resource.CacheName, resourceUri, foundMetricValue);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,8 @@ protected override async Task<ScrapeResult> ScrapeResourceAsync(string subscript

var filter = $"EntityName eq '{resource.QueueName}'";
var metricName = scrapeDefinition.AzureMetricConfiguration.MetricName;
var foundMetricValue = await AzureMonitorClient.QueryMetricAsync(metricName, aggregationType, aggregationInterval, resourceUri, filter);
var dimensionName = scrapeDefinition.AzureMetricConfiguration.Dimension?.Name;
var foundMetricValue = await AzureMonitorClient.QueryMetricAsync(metricName,dimensionName, aggregationType, aggregationInterval, resourceUri, filter);

var labels = new Dictionary<string, string>
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,8 @@ protected override async Task<ScrapeResult> ScrapeResourceAsync(string subscript
resource.DatabaseName);

var metricName = scrapeDefinition.AzureMetricConfiguration.MetricName;
var foundMetricValue = await AzureMonitorClient.QueryMetricAsync(metricName, aggregationType, aggregationInterval, resourceUri);
var dimensionName = scrapeDefinition.AzureMetricConfiguration.Dimension?.Name;
var foundMetricValue = await AzureMonitorClient.QueryMetricAsync(metricName, dimensionName, aggregationType, aggregationInterval, resourceUri);

var labels = new Dictionary<string, string>
{
Expand Down
Loading

0 comments on commit 6c390c9

Please sign in to comment.