Open Research Service map not shown

[mothercow-dev]

HI,

I am using data-prepper and the service map is not showing up in Trace analytics in opensearch.

Is there a workaround?

The test application we used is gcp's microservice-demo.

Is there any additional data you need to verify?

[dashboard]

[service map]

[data-prepper]

[dlvenable]

@mothercow-dev ,

Thank you for reporting this. The first thing I'd like to know is if you have more than one service outputting traces. Data Prepper has a known limitation where the service-map does not generate when there is only one service - see #628.

[mothercow-dev]

@dlvenable

Thank you for your response.

The test environment consists of the following services
( adservice / cartservice / checkoutservice / currencyservice / emailservice / frontend / loadgenerator / paymentservice / productcatalogservice / recommendationservice / redis-cart / shippingservice )

#628 Content checked
I hope this is resolved quickly

[dlvenable]

@mothercow-dev ,

It looks like you have multiple services, so it might not be related to the bug I mentioned.

A few other data points that can help us resolve this, if you can provide them:

• Did you wait at least 5 minutes after sending data to check the service-map? It does often take about 5 minutes to populate.
• Are there any steps you can provide us so that we reproduce this issue?
• Can you provide us with a full trace with all the spans? We'd like to try to analyze this.

[mothercow-dev]

@dlvenable

Yes, I've been waiting for more than 5 minutes, but it's not showing up.

The setup looks like this.

GCP microservice-demo -> opentelemetry-collector -> data-prepper -> opensearch -> dashboard
(opentelemetry-java/python/dotnet)

The agents were automatically injected into the service via auto-instrumentation (CRD).

I don't understand what you mean by all spans, what more information do I need to give you?

Thanks.

[derek-ho]

@dlvenable there is an issue with this is seems like for golang apps, or maybe for isolated services

[dlvenable]

@mothercow-dev ,

We are looking into this issue. Can you provide some additional information to help us debug the problem?

• What version of Data Prepper are you using?
• What version of OpenSearch and OpenSearch Dashboards are you using?
• It sounds like you are using an open-source demo. Can you share a link to the particular tool?
• Can you share the data you have in the OpenSearch index for otel-v1-apm-span?
• And do you have sample data on the incoming spans into Data Prepper?

Thanks!

[dlvenable]

@mothercow-dev , Yes, this looks like #628.

[chenqi0805]

@mothercow-dev Thanks for the interest in trying-out trace ingestion in data-prepper. From the screenshot in the description it seems there is no service graph showing up in the service-map dashboard. Can you help us verify in your data:

1. what is the cardinality of the serviceName field values?
2. Is there any parent-child span pair that is crossing different serviceName? i.e. parent span serviceName=serviceA, child span serviceName=serviceB. That is essentially we want to look at in a full trace (traceId) with all the spans.

[chenqi0805]

@mothercow-dev Also, it might be worth checking the data-prepper logs to see if there is any error logs. We have a hot fix that is going to be released next week: #2363 which may impact the issue.

[mothercow-dev]

@dlvenable

Thank you for your response.

I'll share it with you.

data-prepper : 2.0.1
opensearch & dashboard : 2.5.0
demo app : https://github.com/GoogleCloudPlatform/microservices-demo
opentelemetry operrator : https://github.com/open-telemetry/opentelemetry-operator
OpenSearch index for otel-v1-apm-span:

{
    "policy": {
        "policy_id": "raw-span-policy",
        "description": "Managing raw spans for trace analytics",
        "last_updated_time": 1675327878346,
        "schema_version": 17,
        "error_notification": null,
        "default_state": "current_write_index",
        "states": [
            {
                "name": "current_write_index",
                "actions": [
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "rollover": {
                            "min_size": "50gb",
                            "min_index_age": "24h"
                        }
                    }
                ],
                "transitions": []
            }
        ],
        "ism_template": [
            {
                "index_patterns": [
                    "otel-v1-apm-span-*"
                ],
                "priority": 0,
                "last_updated_time": 1675327878345
            }
        ]
    }
}

data-prepper log:

instrumentation_library_spans {
    instrumentation_library {
      name: "io.opentelemetry.spring-webmvc-3.1"
      version: "1.23.0-alpha"
    }
    spans {
      trace_id: "\240Y\221\344M\024`\252\002;\t:\276\260n\016"
      span_id: "w\365a:\366\223M\323"
      parent_span_id: "@DJv\243J\252Q"
      name: "Controller.health"
      kind: SPAN_KIND_INTERNAL
      start_time_unix_nano: 1677920208289328769
      end_time_unix_nano: 1677920208289542751
      attributes {
        key: "thread.id"
        value {
          int_value: 50
        }
      }
      attributes {
        key: "thread.name"
        value {
          string_value: "http-nio-8080-exec-9"
        }
      }
      status {
      }
    }
  }
  instrumentation_library_spans {
    instrumentation_library {
      name: "io.opentelemetry.tomcat-7.0"
      version: "1.23.0-alpha"
    }
    spans {
      trace_id: "\240Y\221\344M\024`\252\002;\t:\276\260n\016"
      span_id: "@DJv\243J\252Q"
      name: "GET /health"
      kind: SPAN_KIND_SERVER
      start_time_unix_nano: 1677920208289000000
      end_time_unix_nano: 1677920208289626076
      attributes {
        key: "http.route"
        value {
          string_value: "/health"
        }
      }
      attributes {
        key: "http.user_agent"
        value {
          string_value: "kube-probe/1.23"
        }
      }
      attributes {

Thanks.

[mothercow-dev]

@chenqi0805

hi,

I'll share the values from my opensearch

Is there anything else you need???

thanks.

[chenqi0805]

@mothercow-dev Thanks for all the info! It might take us some time to reproduce your setup. But meanwhile we would like to inform you that we have a 2.1.1 pre-release image you can pull and try out in your setup to see if the issue persists: https://gallery.ecr.aws/t2m2d0w1/data-prepper

[mothercow-dev]

@chenqi0805
hi, all

I tried again with the latest version, but it didn't work
Same thing, only the service map doesn't come up.

Opensearch & Opensearch Dashboard - 2.6.0
Data-Prepper - 2.1.1
Opentelemetry-collector - contrib:0.71.0

Thank you.

[dlvenable]

@mothercow-dev , Do you have any composable index templates which might be in use?

Presently Data Prepper is only using the legacy index template style. So if a composable index template matches the indexes, OpenSearch will not use the legacy index template. We have #1275 open to support this. In the meantime, we have seen that some users with composable index templates are having issues. If this is the case for you the work-arounds are either:

• Remove the composable index template which matches on the trace indexes; or
• Create composable index template manually on OpenSearch