Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HTTP spans are duplicated if auto-instumentation in OpenTelemetry is enabled #19573

Closed
lmolkova opened this issue Jun 29, 2021 · 1 comment · Fixed by #29616
Closed

HTTP spans are duplicated if auto-instumentation in OpenTelemetry is enabled #19573

lmolkova opened this issue Jun 29, 2021 · 1 comment · Fixed by #29616
Assignees
Labels
bug This issue requires a change to an existing behavior in the product in order to be resolved. Client This issue points to a problem in the data-plane of the library. Monitor Monitor, Monitor Ingestion, Monitor Query OpenTelemetry OpenTelemetry instrumentation (not Monitor-specific)

Comments

@lmolkova
Copy link
Member

lmolkova commented Jun 29, 2021

  • Package Name: azure-core-tracing-opentelemetry
  • Package Version: any
  • Operating System: any
  • Python Version: any

Describe the bug
Http spans are tracked by Azure SDK and OTel auto-collector

To Reproduce

  1. configure opentelemtry to auto-trace http requests
pip install opentelemetry-instrumentation-requests
from opentelemetry.instrumentation.requests import RequestsInstrumentor
RequestsInstrumentor().instrument()
  1. export spans (anywhere, e.g. console)
  2. follow example, e.g. one in pypi
  3. http spans are duplicated
{
    "name": "HTTP PUT",
    "context": {
        "trace_id": "0x1d964fb974568d75d5a6b37c6fd5fd72",
        "span_id": "0x35d7dc26f4d2a3d9",
        "trace_state": "[]"
    },
    "kind": "SpanKind.CLIENT",
    "parent_id": "0x779d7555486ca027",
    "start_time": "2021-06-29T18:12:59.427323Z",
    "end_time": "2021-06-29T18:12:59.514322Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {
        "http.method": "PUT",
        "http.url": "https://azmondemo.blob.core.windows.net/mycontainer1?restype=container",
        "http.status_code": 201
    },
    "events": [],
    "links": [],
    "resource": {
        "telemetry.sdk.language": "python",
        "telemetry.sdk.name": "opentelemetry",
        "telemetry.sdk.version": "1.3.0",
        "service.name": "unknown_service"
    }
}
{
    "name": "/mycontainer1",
    "context": {
        "trace_id": "0x1d964fb974568d75d5a6b37c6fd5fd72",
        "span_id": "0x90f15d7360c527e3",
        "trace_state": "[]"
    },
    "kind": "SpanKind.CLIENT",
    "parent_id": "0x779d7555486ca027",
    "start_time": "2021-06-29T18:12:59.427323Z",
    "end_time": "2021-06-29T18:12:59.516327Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {
        "component": "http",
        "http.method": "PUT",
        "http.url": "https://azmondemo.blob.core.windows.net/mycontainer1?restype=container",
        "http.user_agent": "azsdk-python-storage-blob/12.8.1 Python/3.9.6 (Windows-10-10.0.19043-SP0)",
        "http.status_code": 201,
        "x-ms-client-request-id": "a2cbd4bf-d905-11eb-84d4-d03c1f05e55a",
        "x-ms-request-id": "1f663372-a01e-008d-1712-6d460f000000"
    },
    "events": [],
    "links": [],
    "resource": {
        "telemetry.sdk.language": "python",
        "telemetry.sdk.name": "opentelemetry",
        "telemetry.sdk.version": "1.3.0",
        "service.name": "unknown_service"
    }
}

Expected behavior
Only one HTTP span is tracked

Solution
[Update]: does not currently work: below solution requires taking a dependency on opentelemetry-instrumentation.

Please put the suppress_instrumentation key on the context to suppress auto-instrumentation when creating HTTP span here.
Value should be true. Please investigate if false needs to be populated after http span ends (i.e. if context leaks to other requests).

Additional context
Please try to repro and make sure it's fixed.
More context: open-telemetry/opentelemetry-specification#1767

Full example:

pip install opentelemetry-instrumentation-requests
# Declare OpenTelemetry as enabled tracing plugin for Azure SDKs
from azure.core.settings import settings
from azure.core.tracing.ext.opentelemetry_span import OpenTelemetrySpan

settings.tracing_implementation = OpenTelemetrySpan


# Regular open telemetry usage from here, see https://github.com/open-telemetry/opentelemetry-python
# for details
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import ConsoleSpanExporter
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
RequestsInstrumentor().instrument()


# Simple console exporter
exporter = ConsoleSpanExporter()

trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer(__name__)
trace.get_tracer_provider().add_span_processor(
    SimpleSpanProcessor(exporter)
)

# Example with Storage SDKs

from azure.storage.blob import BlobServiceClient

with tracer.start_as_current_span(name="MyApplication"):
    client = BlobServiceClient.from_connection_string('secret')
    client.create_container('mycontainer1')  # Call will be traced
@ghost ghost added the needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. label Jun 29, 2021
@lmolkova lmolkova added bug This issue requires a change to an existing behavior in the product in order to be resolved. Client This issue points to a problem in the data-plane of the library. Monitor Monitor, Monitor Ingestion, Monitor Query labels Jul 1, 2021
@ghost ghost removed the needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. label Jul 1, 2021
@lmolkova
Copy link
Member Author

lmolkova commented Jul 1, 2021

This is a cross-language problem, I'm looking for a generic solution, nothing can be done for now

@lmolkova lmolkova self-assigned this Jul 1, 2021
@lmolkova lmolkova added the OpenTelemetry OpenTelemetry instrumentation (not Monitor-specific) label Jun 16, 2022
azure-sdk pushed a commit to azure-sdk/azure-sdk-for-python that referenced this issue Jun 30, 2022
microsoft.automanage 2022-05-04 remove overrides (Azure#19573)

* Adds base for updating Microsoft.Automanage from version preview/2021-04-30-preview to version 2022-05-04

* Updates readme

* Updates API version in new specs and examples

* fix s360 swagger correctness issues

* change targetId to be readonly

* fix wrong commit

* remove patch from configurationProfileVersion

* fix api version

* remove overrides

* remove overrides and add managedBy
@github-actions github-actions bot locked and limited conversation to collaborators Jul 11, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug This issue requires a change to an existing behavior in the product in order to be resolved. Client This issue points to a problem in the data-plane of the library. Monitor Monitor, Monitor Ingestion, Monitor Query OpenTelemetry OpenTelemetry instrumentation (not Monitor-specific)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant