Sentry Spring WebFlux integration creates a new hub and scope which is not used for OpenTelemetry transactions

[adinauer]

Discussed in #2715

Problem

SentrySpanProcessor creates a new transaction for a request which is bound to the current hub.
SentryWebFilter then creates a new hub cloned from the main hub. This causes a new and separate scope to be created. If a developer now manipulates the scope during the request, these changes are lost.

Possible Fixes

Creating a new hub in SentrySpanProcessor

• Will probably reintroduce OOM issues we had before as we can't ensure we'll find the same hub again during request handling which may lead to an ever growing stack of scopes eventually causing OOM.

Could store the hub in SentrySpanStorage and then look it up using OpenTelemetry span id

• Would probably require some generic mechanism in options that could be set by our OpenTelemetry integration where instead of creating a new hub and pushing a new scope we look up the hub in SentrySpanStorage.
• If not using a replacable mechanism we'd be tying the WebFlux integration to OpenTelemetry or would have to offer yet another package which is suboptimal

Workarounds

Won't affect Sentry integrations. Please give those a try and give us feedback.

Manually manipulation the Sentry transaction directly

import io.opentelemetry.api.trace.Span;
import io.sentry.ISpan;
import io.sentry.ITransaction;
import io.sentry.Sentry;
import io.sentry.SentrySpanStorage;
import io.sentry.SpanId;

  @GetMapping("{id}")
  Person person(@PathVariable Long id) {
    ISpan span = Sentry.getCurrentHub().getSpan().startChild("op1");
    try {
      @Nullable ITransaction transaction = TransactionFinder.findTransaction();
      transaction.setData("my-data-key", "my-data-value");
      System.out.println(transaction);
      LOGGER.info("Loading person with id={}", id);
      throw new IllegalArgumentException("Something went wrong [id=" + id + "]");
    } finally {
      span.finish();
    }
  }

  static class TransactionFinder {
    public static @Nullable ITransaction findTransaction() {
      String otelSpanId = Span.current().getSpanContext().getSpanId();
      return findTransaction(otelSpanId);
    }

    private static @Nullable ITransaction findTransaction(@Nullable String spanId) {
      if (spanId == null) {
        return null;
      }

      @Nullable ISpan span = SentrySpanStorage.getInstance().get(spanId);

      if (span instanceof ITransaction) {
        return (ITransaction) span;
      }

      if (span instanceof io.sentry.Span) {
        SpanId parentSpanId = ((io.sentry.Span) span).getParentSpanId();
        if (parentSpanId != null) {
          return findTransaction(parentSpanId.toString());
        }
      }

      return null;
    }
  }

Manually manipulating the OpenTelemetry span

Span.current().setAttribute("my-otel-attr", "otel-attr-value");

For manipulating things like request etc. this could also be done via EventProcessor or beforeSend / beforeSendTransaction:

  @Bean
  EventProcessor eventProcessor() {
    return new EventProcessor() {
      @Override
      public @Nullable SentryEvent process(@NotNull SentryEvent event, @NotNull Hint hint) {
        addOtelAttr(event, hint);
        return event;
      }

      @Override
      public @Nullable SentryTransaction process(@NotNull SentryTransaction transaction, @NotNull Hint hint) {
        addOtelAttr(transaction, hint);
        return transaction;
      }

      @SuppressWarnings("unchecked")
      private void addOtelAttr(final @NotNull SentryBaseEvent event, final @NotNull Hint hint) {
        Object otelContextObject = event.getContexts().get("otel");
        if (otelContextObject instanceof Map) {
          Map<String, Object> otelContext = (Map<String, Object>) otelContextObject;
          Object attributesObject = otelContext.get("attributes");
          if (attributesObject instanceof Map) {
            Map<String, Object> attributes = (Map<String, Object>) attributesObject;
            Object myValue = attributes.get("my-otel-attr");
            event.setExtra("my-extra-from-otel", myValue);
          }
        }
      }
    };
  }