A way to suppress redis.encode.start/end custom events

[paveljakov]

Our applications currently generates a lot of custom redis.encode.start and redis.encode.end events. Is there a way to suppress these custom events without disabling redis auto-collected telemetry?

We have tried to use new sampling override functionality, but it didn't work.

We have tried rules using db.statement and db.system attributes:

{
  "attributes": [
	{
	  "key": "db.statement",
	  "value": "INFO",
	  "matchType": "strict"
	},
	{
	  "key": "db.system",
	  "value": "redis",
	  "matchType": "strict"
	}
  ],
  "percentage": 0
}

Also we have tried using ai.operation.name attribute:

{
  "attributes": [
	{
	  "key": "ai.operation.name",
	  "value": "redis.encode.end",
	  "matchType": "strict"
	}
  ],
  "percentage": 0
}

Nothing helped, these events are still being sent to AI.

Also maybe there is a way to get complete list of Span attributes that can be used for sampling overrides (to filter out such dependencies like periodic clustered Quatz job selects, etc.)?

[MS-jgol]

@paveljakov
Just to make sure the syntax is correct (it probably is...), could you please copy the whole configuration/json part starting with "preveiw": {

Are you using the Java agent 3.0.3-BETA.2?
https://github.com/microsoft/ApplicationInsights-Java/releases/download/3.0.3-BETA.2/applicationinsights-agent-3.0.3-BETA.2.jar

[trask]

hey @paveljakov @MS-jgol, unfortunately the sampling overrides only apply to spans, and not to span events, so we will need to think of another way to address this

[paveljakov]

@trask, Understood, thank you for your response.

@MS-jgol, Just in case, here is our configuration json:

{
  "connectionString": "@CONNECTION_STRING@",
  "role": {
    "name": "document"
  },
  "customDimensions": {
    "owner": "${USERNAME}",
    "service.version": "${APP_VERSION}"
  },
  "selfDiagnostics": {
    "destination": "console",
    "level": "WARN"
  },
  "preview": {
    "sampling": {
      "overrides": [
        {
          "attributes": [
            {
              "key": "http.url",
              "value": "https?://[^/]+/actuator/health",
              "matchType": "regexp"
            }
          ],
          "percentage": 0
        },
        {
          "attributes": [
            {
              "key": "db.statement",
              "value": ".+\\sQRTZ_.+",
              "matchType": "regexp"
            }
          ],
          "percentage": 0
        },
        {
          "attributes": [
            {
              "key": "db.system",
              "value": "postgresql",
              "matchType": "strict"
            },
            {
              "key": "db.statement",
              "value": "",
              "matchType": "strict"
            }
          ],
          "percentage": 0
        },
        {
          "attributes": [
            {
              "key": "db.statement",
              "value": "INFO",
              "matchType": "strict"
            },
            {
              "key": "db.system",
              "value": "redis",
              "matchType": "strict"
            }
          ],
          "percentage": 0
        },
        {
          "attributes": [
            {
              "key": "ai.operation.name",
              "value": "redis\\.encode\\.\\w+",
              "matchType": "regexp"
            }
          ],
          "percentage": 0
        }
      ]
    },
    "processors": [
      {
        "type": "span",
        "include": {
          "matchType": "strict",
          "spanNames": [ "netty.request" ]
        },
        "name": {
          "fromAttributes": [ "http.method", "http.url" ],
          "separator": " "
        }
      },
      {
        "type": "span",
        "include": {
          "matchType": "regexp",
          "attributes": [
            { "key": "http.method", "value": "" }
          ],
          "spanNames": [ "^/" ]
        },
        "name": {
          "toAttributes": {
            "rules": [ "^(?<tempName>.*)$" ]
          }
        }
      },
      {
        "type": "span",
        "include": {
          "matchType": "strict",
          "attributes": [
            { "key": "tempName" }
          ]
        },
        "name": {
          "fromAttributes": [ "http.method", "tempName" ],
          "separator": " "
        }
      },
      {
        "type": "attribute",
        "include": {
          "matchType": "strict",
          "attributes": [
            { "key": "tempName" }
          ]
        },
        "actions": [
          { "key": "tempName", "action": "delete" }
        ]
      }
    ]
  }
}

We have tried some attributes listed in AI agent TRACE logs, i.e.:

2021-03-26 12:14:32.389+02 DEBUG c.m.a.agent.Exporter - exporting span: SpanData{spanContext=ImmutableSpanContext{traceId=613ac22122744600cf01d6817370c7e5, spanId=caea761f0e8e426f, traceFlags=01, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=true}, parentSpanContext=ImmutableSpanContext{traceId=613ac22122744600cf01d6817370c7e5, spanId=e1546d1b632362cb, traceFlags=01, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=true}, resource=Resource{attributes={os.description="Windows 10 10.0", os.type="WINDOWS", process.command_line="C:\Program Files\OpenJDK\openjdk-11.0.9_11;bin;java.exe -agentlib:jdwp=transport=dt_shmem,address=javadebug,suspend=y,server=n -Xmx256m -javaagent:C:/sources/ehealthfi/ehealthfi-document/build/applicationinsights.jar -XX:TieredStopAtLevel=1 -Xverify:none -Dspring.output.ansi.enabled=always -javaagent:C:\Users\pavelj\AppData\Local\JetBrains\Toolbox\apps\IDEA-U\ch-0\203.7717.56\plugins\Groovy\lib\agent\gragent.jar -javaagent:C:\Users\pavelj\AppData\Local\JetBrains\Toolbox\apps\IDEA-U\ch-0\203.7717.56\plugins\java\lib\rt\debugger-agent.jar -Dcom.sun.management.jmxremote -Dspring.jmx.enabled=true -Dspring.liveBeansView.mbeanDomain -Dspring.application.admin.enabled=true -Dfile.encoding=UTF-8", process.executable.path="C:\Program Files\OpenJDK\openjdk-11.0.9_11;bin;java.exe", process.pid=33208, process.runtime.description="Oracle Corporation OpenJDK 64-Bit Server VM 11.0.9+11", process.runtime.name="OpenJDK Runtime Environment", process.runtime.version="11.0.9+11", service.name="unknown_service:java", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.0.1"}}, instrumentationLibraryInfo=InstrumentationLibraryInfo{name=io.opentelemetry.javaagent.lettuce-5.1, version=null}, name=INFO, kind=CLIENT, startEpochNanos=1616753672386912900, endEpochNanos=1616753672389889200, attributes=AttributesMap{data={net.peer.ip=127.0.0.1, net.peer.port=6379, thread.name=lettuce-nioEventLoop-4-1, db.statement=INFO, db.system=redis, thread.id=241, net.transport=IP.TCP, net.peer.name=localhost, db.connection_string=redis://localhost:6379}, capacity=128, totalAddedValues=9}, totalAttributeCount=9, events=[ImmutableEventData{name=redis.encode.start, attributes={}, epochNanos=1616753672386975400, totalAttributeCount=0}, ImmutableEventData{name=redis.encode.end, attributes={}, epochNanos=1616753672386998500, totalAttributeCount=0}], totalRecordedEvents=2, links=[], totalRecordedLinks=0, status=ImmutableStatusData{statusCode=UNSET, description=}, hasEnded=true}

[trask]

These particular custom events were suppressed in 3.0.3

[jasmine-c]

@trask thank you for addressing this! I have the same issue myself.
Will this fix work for lettuce version 5.2 as well?

[trask]

Will this fix work for lettuce version 5.2 as well?

yes it should 👍