Support for custom exporter broke with versions V 3.1.1

[DarioBernardo]

Hello,

First, thank you for the excellent work on langtrace. It's been incredibly useful for our organization.

I’ve been using the SDK to export traces to our internal OpenTelemetry server (Signoz). Everything worked perfectly with langtrace-python-sdk==2.3.22. However, after upgrading to version 3.1.1 (and later), I stopped seeing traces in Signoz.

It seems like something might have changed in how custom exporters are set up. I noticed the documentation mentions that an API key is not required (despite the red warning), but I no longer see the familiar log message: "Exporting spans to custom remote exporter...".

Could you clarify if there are new setup requirements or changes affecting custom exporters in version 3.1.1? Any guidance would be greatly appreciated.

Thank you!

[karthikscale3]

Hi @DarioBernardo , thanks for the message and apologies for the issue you are facing. With 3.1.1 we added native support for OTEL collector setups. While this should not have ideally impacted your setup, could you make sure you have the following env vars setup correctly?

OTEL_EXPORTER_OTLP_ENDPOINT="<endpoint>"
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <secret>"

This will ensure langtrace automatically picks up OTEL_EXPORTER_OTLP_ENDPOINT as the endpoint to send traces to. You should see a log message during the initialization that says "exporting traces to custom endpoint ...".

Also could you please upgrade to the latest version 3.3.14 so we can rule out any other issues. Please let me know if you face any issues.

[karthikscale3]

To be precise, these are the env var settings I have which works well with signoz. Some users were facing issues when they had https:// in the OTEL_EXPORTER_OTLP_ENDPOINT and when it was removed, it started working.

OTEL_EXPORTER_OTLP_ENDPOINT="ingest.us.signoz.cloud:443"
OTEL_EXPORTER_OTLP_HEADERS="signoz-access-token=<token>"
OTEL_RESOURCE_ATTRIBUTES=service.name="rag chatbot"
OTEL_EXPORTER_OTLP_PROTOCOL="grpc"

[karthikscale3]

Also check out our guide here - https://www.langtrace.ai/blog/monitoring-a-langchain-rag-application-with-signoz-and-langtrace

[karthikscale3]

@DarioBernardo - please let us know if you were able to get this sorted. Thanks

[DarioBernardo]

Hi @karthikscale3, Thank you for your quick reply and apologies with the delay of mine.
No, unfortunately it didn't help. Here is my setup:

collector_host = os.environ.get("MY__OTEL_COLLECTOR_HOST", None)
collector_port = os.environ.get("My__OTEL_COLLECTOR_PORT", None)
otel_service_name = os.environ.get("MY__OTEL_SERVICE_NAME", None)

logging.info(f"OTEL Service name: {otel_service_name}     OTEL Collector Endpoint: {collector_host}:{collector_port}")

if collector_host:
    # Set OpenTelemetry
    resource = Resource(attributes={
        SERVICE_NAME: otel_service_name
    })

    collector_endpoint = f"{collector_host}:{collector_port}"

    ## Set export of traces
    provider = TracerProvider(resource=resource)
    processor = BatchSpanProcessor(OTLPSpanExporter(endpoint=collector_endpoint, insecure=True))
    provider.add_span_processor(processor)
    trace.set_tracer_provider(provider)

    langtrace.init(custom_remote_exporter=processor, batch=True)

    ## Set export of metrices
    reader = PeriodicExportingMetricReader(OTLPMetricExporter(endpoint=collector_endpoint, insecure=True))
    meterProvider = MeterProvider(resource=resource, metric_readers=[reader])
    metrics.set_meter_provider(meterProvider)

    ## Set export of logs
    log_exporter = OTLPLogExporter(endpoint=collector_endpoint, insecure=True)
    log_processor = BatchLogRecordProcessor(log_exporter)
    logger_provider = LoggerProvider(resource=resource)
    logger_provider.add_log_record_processor(log_processor)

    otel_logging_handler = LoggingHandler(level=logging.NOTSET, logger_provider=logger_provider)

The above set up works fine with langtrace 2.3.22. However in the latest version (3.3.14), if I don't set any env variable I get:

Initializing Langtrace SDK..
⭐ Leave our github a star to stay on top of our updates - https://github.com/Scale3-Labs/langtrace

Missing Langtrace API key, proceed to https://langtrace.ai to create one
Set the API key as an environment variable LANGTRACE_API_KEY

and i don't see any trace. However, If I set the following env variable:
"OTEL_EXPORTER_OTLP_ENDPOINT": "host.docker.internal:4317"
"OTEL_EXPORTER_OTLP_PROTOCOL": "tcp" (I also tried grpc but i think my collector runs on tcp)
( I am using a vscode with a dev container)
I see the following in the logs:

Initializing Langtrace SDK..
⭐ Leave our github a star to stay on top of our updates - https://github.com/Scale3-Labs/langtrace
[2024-12-16 12:19:13] opentelemetry.trace:WARNING:__init__.py:538 > Overriding of current TracerProvider is not allowed
Exporting spans to custom host: host.docker.internal:4317..

which is a good sign, however my application crashes with this error:

File "/opt/conda/lib/python3.11/site-packages/motor/core.py", line 1693, in _to_list
    result = get_more_result.result()
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/motor/core.py", line 1918, in _on_started
    pymongo_cursor = future.result()
                     ^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/langtrace_python_sdk/instrumentation/pymongo/patch.py", line 35, in traced_method
    span, deduce_args_and_kwargs(wrapped, *args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/langtrace_python_sdk/utils/__init__.py", line 39, in deduce_args_and_kwargs
    bound_args = sig.bind(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/inspect.py", line 3195, in bind
    return self._bind(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/inspect.py", line 3110, in _bind
    raise TypeError(msg) from None
TypeError: missing a required argument: 'pipeline'

Therefore I had to immediately revert.

Note: I didn't set the OTEL_EXPORTER_OTLP_HEADERS as I don't think i am requiring any access token, my application runs fine without.

Thank you for your help

[alizenhom]

Hey @DarioBernardo, apologies for the issue you're facing

from the code you just shared, i see that some logic is being duplicated for setting up open-telemetry which is actually abstracted away by langtrace it self.
can you try commenting this for now as this why it's triggering Overriding of current TracerProvider is not allowed

    processor = BatchSpanProcessor(OTLPSpanExporter(endpoint=collector_endpoint, insecure=True))
    provider.add_span_processor(processor)
    trace.set_tracer_provider(provider)

also can you expand how are you using signoz without access_token? as far as i know it's essential to connect to signoz and send traces

also i want to share an example with you sending traces to signoz maybe could point out some tips

"""Example of using the anthropic API to create a message."""

import anthropic
from dotenv import find_dotenv, load_dotenv

from langtrace_python_sdk import langtrace, with_langtrace_root_span

_ = load_dotenv(find_dotenv())

langtrace.init()


@with_langtrace_root_span("messages_create")
def messages_create():
    client = anthropic.Anthropic()

    message = client.messages.create(
        model="claude-3-opus-20240229",
        max_tokens=1000,
        temperature=0.0,
        system="Respond only in Yoda-speak.",
        messages=[{"role": "user", "content": "How are you today?"}],
        stream=True,
    )

    for response in message:
        pass

Enviroment variables

OTEL_EXPORTER_OTLP_ENDPOINT=ingest.eu.signoz.cloud:443
OTEL_EXPORTER_OTLP_HEADERS="signoz-access-token=<access-token>"
OTEL_RESOURCE_ATTRIBUTES=service.name=Otel Testing

The issue is still not 100% clear but will be more than happy to help you out with any assistance you need, we will be prioritising this issue anytime!

[DarioBernardo]

I am using signoz deployed on a private kubernetes cluster, so maybe for that reason the key is not necessary. I am not using signoz on cloud.
Now, in my code I initialize langtrace with the custom remote exported:

langtrace.init(custom_remote_exporter=processor, batch=True)

If I comment out the line you suggested I won't be able to initialize with the custom_reomote_exporter and in fact i will get:

Exporting spans to Langtrace cloud..

Later it doesn't work because it requires a key, and I don't get any traces.
Also my code in other parts uses opentelemetry trace.

[github-actions]

This issue has been automatically marked as stale due to inactivity. It will be closed in 3 days if no further activity occurs.

[alizenhom]

Hey @DarioBernardo, apologies for the long time taken on this issue
we have found a small issue that might have been causing this

can i kindly ask you to change langtrace initialization to just langtrace.init() -- this will result in

• using native oTel exporter (can be configured through env variables)

Env variables to be set

1. LANGTRACE_API_HOST or OTEL_EXPORTER_OTLP_ENDPOINT to https://ingest.eu.signoz.cloud:443
2. OTEL_RESOURCE_ATTRIBUTES=service.name=<Your service> optional
3. OTEL_EXPORTER_OTLP_HEADERS=signoz-ingestion-key=<key> you mentioned that this is not needed with your setup
4. OTEL_EXPORTER_OTLP_PROTOCOL to grpc (optional) it defaults to http

hopefully this should work with your setup, will be waiting for feedback and will be more than happy to help and apologies again