Add Micrometer integration

[mp911de]

See https://github.com/micrometer-metrics/micrometer

[madgnome]

I've implemented one MeterBinder for lettuce for our services, I'll be happy to contribute it back.

It exposes the following metrics:

• lettuce.command.count as counter
• lettuce.command.completion.PERCENTILE as gauge
• lettuce.command.firstresponse.PERCENTILE as gauge
  The command name is in a tag "command"

[dragontree101]

@madgnome how use meterBinder for lettuce? can you show the code?
except you exposed the three metrics, i also want to get active and idle num in lettuce's pool.
thanks very much.

[dragontree101]

mycode is

define RedisMetricsBinder and DefaultClientResources

change default eventbus time from 5 min to 30 second

@Bean
  public RedisMetricsBinder redisMetricsBinder(LettuceConnectionFactory redisConnectionFactory, MetricsHelper metricsHelper) {
    return new RedisMetricsBinder(redisConnectionFactory, metricsHelper);
  }

  @Bean(destroyMethod = "shutdown")
  public DefaultClientResources lettuceClientResources() {
    return DefaultClientResources.builder().commandLatencyPublisherOptions(
        DefaultEventPublisherOptions.builder().eventEmitInterval(Duration.ofSeconds(30)).build())
        .build();
  }

create MetricsHelper class to record metrics

public class MetricsHelper {

  private MeterRegistry meterRegistry;

  public MetricsHelper(MeterRegistry meterRegistry) {
    this.meterRegistry = meterRegistry;
  }

  public void counterIncrement(String name, String... tags) {
    this.meterRegistry.counter(name, tags).increment();
  }

  public void counterIncrement(String name, double increment, String... tags) {
    this.meterRegistry.counter(name, tags).increment(increment);
  }

  public <T extends Number> void gauge(String name, T number) {
    this.meterRegistry.gauge(name, number);
  }

  public <T extends Number> void gauge(String name, Iterable<Tag> tags, T number) {
    this.meterRegistry.gauge(name, tags, number);
  }

  public void timeRecord(String name, long amount, TimeUnit unit, String... tags) {
    Timer.builder(name).publishPercentiles(0.5, 0.75, 0.9, 0.95, 0.99).tags(tags)
        .register(this.meterRegistry).record(amount, unit);
  }

  public void summaryRecord(String name, long amount, String... tags) {
    DistributionSummary.builder(name).publishPercentiles(0.5, 0.75, 0.9, 0.95, 0.99).tags(tags)
        .register(this.meterRegistry).record(amount);
  }

}

and implement RedisMetricsBinder to receive event , and change event to metrics

public class RedisMetricsBinder implements MeterBinder {

  private final LettuceConnectionFactory redisConnectionFactory;
  private final MetricsHelper metricsHelper;
  private final Iterable<Tag> tags;

  public RedisMetricsBinder(LettuceConnectionFactory redisConnectionFactory,
      MetricsHelper metricsHelper) {
    this.tags = emptyList();
    this.redisConnectionFactory = redisConnectionFactory;
    this.metricsHelper = metricsHelper;
  }

  @Override
  public void bindTo(MeterRegistry registry) {
    redisConnectionFactory.getClientResources().eventBus().get()
        .filter(event -> event instanceof CommandLatencyEvent)
        .cast(CommandLatencyEvent.class)
        .subscribe(e ->
          e.getLatencies().forEach((k, v) -> {
            log.info(
                "key is {}, count is {}, completion max {}, completion min {}, completion percentiles {}",
                k.commandType().name(), v.getCount(), v.getCompletion().getMax(),
                v.getCompletion().getMin(), v.getCompletion().getPercentiles().toString());

            metricsHelper.counterIncrement("redis.command", v.getCount(), "count.name",
                k.commandType().name());
            metricsHelper.gauge("redis.completion.min",
                Lists.newArrayList(Tag.of("count.name", k.commandType().name())),
                v.getCompletion().getMin());
            metricsHelper.gauge("redis.completion.max",
                Lists.newArrayList(Tag.of("count.name", k.commandType().name())),
                v.getCompletion().getMax());
            metricsHelper.gauge("redis.completion.p50",
                Lists.newArrayList(Tag.of("count.name", k.commandType().name())),
                v.getCompletion().getPercentiles().get(50.0));
            metricsHelper.gauge("redis.completion.p90",
                Lists.newArrayList(Tag.of("count.name", k.commandType().name())),
                v.getCompletion().getPercentiles().get(90.0));
            metricsHelper.gauge("redis.completion.p95",
                Lists.newArrayList(Tag.of("count.name", k.commandType().name())),
                v.getCompletion().getPercentiles().get(95.0));
            metricsHelper.gauge("redis.completion.p99",
                Lists.newArrayList(Tag.of("count.name", k.commandType().name())),
                v.getCompletion().getPercentiles().get(99.0));
            metricsHelper.gauge("redis.completion.p999",
                Lists.newArrayList(Tag.of("count.name", k.commandType().name())),
                v.getCompletion().getPercentiles().get(99.9));
          }));
  }

}

i want to know this method to report redis metrics is ok?
@mp911de thanks!

[mp911de]

This approach uses Lettuce's latency collector and the publication mechanism to push metrics to Micrometer. I would rather suggest providing a CommandLatencyCollector implementation for Micrometer and using Timer directly. This is, to leave aggregation and sampling up to Micrometer/your monitoring system instead of using pre-aggregated results.

[dragontree101]

thanks! @mp911de
i see, i will implementation myself CommandLatencyCollector to record metrics.

[perlun]

@madgnome Late to the party, but would you have a chance to show us the MeterBinder you implemented for this? We are currently taking Redis & Lettuce into use, so being able to gather metrics for it would be incredibly useful.

[perlun]

FWIW, on a slightly related note, I looked into adding metrics for our connection-pool usage of Lettuce. This works, and produces metrics which can be visualized in e.g. VisualVM with the MBeans plugin.

diff --git a/src/foo/bar/managers/caching/RedisClientManager.java b/src/foo/bar/managers/caching/RedisClientManager.java
index 1ce97a7a89..eab511ac66 100644
--- a/src/foo/bar/managers/caching/RedisClientManager.java
+++ b/src/foo/bar/managers/caching/RedisClientManager.java
@@ -218,6 +218,9 @@ public class RedisClientManager {
         GenericObjectPoolConfig<Object> poolConfig = new GenericObjectPoolConfig<>();
         poolConfig.setMaxTotal( redisServer.getConnectionPoolSize() );
         poolConfig.setMinIdle( redisServer.getConnectionPoolSize() );
+        poolConfig.setJmxEnabled( true );
+        poolConfig.setJmxNameBase( "foo.bar.redis.pool:name=" );
+        poolConfig.setJmxNamePrefix( "RedisClientManager.pool" );
 
         return ConnectionPoolSupport
                 .createGenericObjectPool( () -> {

However, I'd much rather see that these metrics are propagated to our custom metric registry (which be Micrometer, Dropwizard, etc, based on how our software is configured). Right now, the code is quite tightly coupled to JMX which is unfortunate.

I looked a while yesterday into overriding this, by making a local copy of the ConnectionPoolSupport to be able to subclass the GenericObjectPool being returned (and override it to not publish metrics to JMX), but it seemed rather non-trivial because it uses a bunch of Lettuce-package-private types. Even after copying a bunch of these into my local project, I still couldn't make it work because some types are basically impossible to use without ugly reflection hacks... that about the time when I dropped the idea.

@mp911de Do you have any ideas on how to use ConnectionPoolSupport.createGenericObjectPool() with a custom metrics provider? If this is asking for too much, specifically Micrometer would also be quite fine since we are in the long run likely moving towards Micrometer in our architecture.

[mp911de]

What do you mean with custom metrics provider? ConnectionPoolSupport already overrides GenericObjectPool to customize it for proxying of connections.

There's a metrics integration via ClientResources and CommandLatencyCollector. You can subscribe to the EventBus to listen for CommandLatencyEvent.

I hadn't the chance yet to wrap my head around a Micrometer integration. Happy to support anyone who's working on a Lettuce/Micrometer integration.

[perlun]

What do you mean with custom metrics provider?

We have our own MetricRegistryWrapper interface with implementations for DropWizard, Micrometer etc. Ideally, I would like to be able to plug into this mechanism to basically subscribe to all metric events from the connection pool and publish it into the configured metric registry. commons-pool2 seemed quite tightly coupled to JMX when I looked at it, so that's why I thought about subclassing GenericObjectPool to override this - feel free to suggest other, better approaches. 😄

There's a metrics integration via ClientResources and CommandLatencyCollector. You can subscribe to the EventBus to listen for CommandLatencyEvent.

Thanks, that's a good suggestion which could be useful! For our specific use case, this might not be so bad since we want to publish it to our custom MetricRegistryWrapper anyway.

[rajki]

Is this available in a useable form at the moment? @madgnome is your code open sourced anywhere? Looking at integrating Lettuce with Micrometer in my application and exploring right now.

[perlun]

@rajki We ended up wrapping together something like this. (note: not directly usable as-is, needs to be tweaked a bit for reasons described after the snippet):

        ClientResources clientResources = DefaultClientResources.builder()
                .commandLatencyPublisherOptions( DefaultEventPublisherOptions.builder()
                        .eventEmitInterval( Duration.ofSeconds( 10 ) )
                        .build()
                )
                .commandLatencyCollector( new CustomCommandLatencyCollector() )
                .build();

        // Create the client something like this
        RedisClient client = RedisClient.create( clientResources, redisUri )

The CustomCommandLatencyCollector looks like this:

    private static class CustomCommandLatencyCollector implements CommandLatencyCollector {

        @Override
        public void recordCommandLatency( SocketAddress local, SocketAddress remote, ProtocolKeyword commandType, long firstResponseLatency, long completionLatency ) {
            // Note: replace with a Micrometer Timer call, calling 
            // record(duration, TimeUnit.NANOSECONDS) on a timer which is preferably cached
            // between calls. (Map.computeIfAbsent() is useful for this)
            metricRegistry.addTimerEvent( "method", commandType.name(), completionLatency, TimeUnit.NANOSECONDS );
        }

        @Override
        public void shutdown() {
        }

        @Override
        public Map<CommandLatencyId, CommandMetrics> retrieveMetrics() {
            return ImmutableMap.of();
        }

        @Override
        public boolean isEnabled() {
            return true;
        }
    }

Our particular use case does not use Micrometer directly, which is because we have a wrapper layer which redirects calls to the configured metric provider (Micrometer, Dropwizard, etc...) which is why the example is not exactly usable as-is. I am posting it anyway, in the hope that it will help others who, like me, find it frustrating to not know in which direction to start... 🙂

[rajki]

Thanks @perlun, this is very useful. I'll explore and see if I can come up with something that builds off of this suggestion.

[winster]

private final LettuceConnectionFactory redisConnectionFactory;
private final MetricsHelper metricsHelper;
private final Iterable tags;

public RedisMetricsBinder(LettuceConnectionFactory redisConnectionFactory,
MetricsHelper metricsHelper) {
this.tags = emptyList();
this.redisConnectionFactory = redisConnectionFactory;
this.metricsHelper = metricsHelper;
}

@OverRide
public void bindTo(MeterRegistry registry) {
redisConnectionFactory.getClientResources().eventBus().get()
.filter(event -> event instanceof CommandLatencyEvent)
.cast(CommandLatencyEvent.class)
.subscribe(e ->
e.getLatencies().forEach((k, v) -> {
log.info(
"key is {}, count is {}, completion max {}, completion min {}, completion percentiles {}",
k.commandType().name(), v.getCount(), v.getCompletion().getMax(),
v.getCompletion().getMin(), v.getCompletion().getPercentiles().toString());

        metricsHelper.counterIncrement("redis.command", v.getCount(), "count.name",
            k.commandType().name());
        metricsHelper.gauge("redis.completion.min",
            Lists.newArrayList(Tag.of("count.name", k.commandType().name())),
            v.getCompletion().getMin());
        metricsHelper.gauge("redis.completion.max",
            Lists.newArrayList(Tag.of("count.name", k.commandType().name())),
            v.getCompletion().getMax());
        metricsHelper.gauge("redis.completion.p50",
            Lists.newArrayList(Tag.of("count.name", k.commandType().name())),
            v.getCompletion().getPercentiles().get(50.0));
        metricsHelper.gauge("redis.completion.p90",
            Lists.newArrayList(Tag.of("count.name", k.commandType().name())),
            v.getCompletion().getPercentiles().get(90.0));
        metricsHelper.gauge("redis.completion.p95",
            Lists.newArrayList(Tag.of("count.name", k.commandType().name())),
            v.getCompletion().getPercentiles().get(95.0));
        metricsHelper.gauge("redis.completion.p99",
            Lists.newArrayList(Tag.of("count.name", k.commandType().name())),
            v.getCompletion().getPercentiles().get(99.0));
        metricsHelper.gauge("redis.completion.p999",
            Lists.newArrayList(Tag.of("count.name", k.commandType().name())),
            v.getCompletion().getPercentiles().get(99.9));
      }));

}
@dragontree101 Thanks for sharing code.

While using it, I get nullpointerexception for redisConnectionFactory at RedisMetricsBinder:bindTo.

Listing my whole redis configuration class below.

@Bean
public LettuceConnectionFactory redisConnectionFactory() {
    RedisStandaloneConfiguration config = new RedisStandaloneConfiguration();
    config.setHostName(redisConfigurationParameters.getHostname());
    config.setPort(redisConfigurationParameters.getPort());
    config.setDatabase(redisConfigurationParameters.getDatabase());
    return new LettuceConnectionFactory(config);
}

@Bean
public RedisTemplate<?, ?> redisTemplate() {
    RedisTemplate<byte[], byte[]> template = new RedisTemplate<>();
    template.setConnectionFactory(redisConnectionFactory());
    return template;
}

@Bean
public RedisMetricsBinder redisMetricsBinder(LettuceConnectionFactory redisConnectionFactory,
                                             MetricsHelper metricsHelper) {
    return new RedisMetricsBinder(redisConnectionFactory, metricsHelper);
}

@Bean(destroyMethod = "shutdown")
public DefaultClientResources lettuceClientResources() {
    return DefaultClientResources.builder().commandLatencyPublisherOptions(
            DefaultEventPublisherOptions.builder().eventEmitInterval(Duration.ofSeconds(30)).build())
            .build();
}

[asarkar]

Like me, if you came here looking for OOTB Micrometer support, and wondering what the conclusion is for this ticket, Micrometer support seems to be available in the 6.0.x release.
https://lettuce.io/core/release/reference/index.html#command.latency.metrics.micrometer

[steven-sheehy]

It's actually in 6.1 despite what the docs say.