Latest python auto-instrumentation doesn't work - Requested component 'otlp_proto_grpc' not found in entry points for 'opentelemetry_metrics_exporter'

[dominthomas]

Describe your environment

• Python 3.8

• Container running in k8s

• simple fastAPI app

• OpenTelemetry Auto-Instrumentation Injection done via operator:
  https://github.com/open-telemetry/opentelemetry-operator#opentelemetry-auto-instrumentation-injection

• OpenTelemetry auto-instrumentation image version: 0.33b0

• OpenTelemetry Auto-Instrumentation Package Dependencies: requirements.txt
  

Additional Info:

• OpenTelemetry Operator Version: 0.60.0

Steps to reproduce
Following the official guide from here:
https://github.com/open-telemetry/opentelemetry-operator#opentelemetry-auto-instrumentation-injection

What is the expected behavior?
OpenTelemetry being initialised & traces befing forwarded

What is the actual behavior?
The following error is raised after the init container performs the auto-injection.

Configuration of configurator failed
Traceback (most recent call last):
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 105, in _load_configurators
    entry_point.load()().configure(auto_instrumentation_version=__version__)  # type: ignore
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 273, in configure
    self._configure(**kwargs)
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 289, in _configure
    _initialize_components(kwargs.get("auto_instrumentation_version"))
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 233, in _initialize_components
    trace_exporters, metric_exporters, log_exporters = _import_exporters(
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 201, in _import_exporters
    for (exporter_name, exporter_impl,) in _import_config_components(
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 170, in _import_config_components
    raise RuntimeError(
RuntimeError: Requested component 'otlp_proto_grpc' not found in entry points for 'opentelemetry_metrics_exporter'
Failed to auto initialize opentelemetry
Traceback (most recent call last):
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 121, in initialize
    _load_configurators()
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 109, in _load_configurators
    raise exc
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 105, in _load_configurators
    entry_point.load()().configure(auto_instrumentation_version=__version__)  # type: ignore
 File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 273, in configure
    self._configure(**kwargs)
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 289, in _configure
    _initialize_components(kwargs.get("auto_instrumentation_version"))
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 233, in _initialize_components
    trace_exporters, metric_exporters, log_exporters = _import_exporters(
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 201, in _import_exporters
    for (exporter_name, exporter_impl,) in _import_config_components(
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 170, in _import_config_components
    raise RuntimeError(
RuntimeError: Requested component 'otlp_proto_grpc' not found in entry points for 'opentelemetry_metrics_exporter'

Additional context
I've tried adding the missing otlp_proto_grpc and rebuilt the init container docker image, and tried using that image.
But it then throws this error:

Configuration of configurator failed
Traceback (most recent call last):
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 105, in _load_configurators
    entry_point.load()().configure(auto_instrumentation_version=__version__)  # type: ignore
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 344, in configure
    self._configure(**kwargs)
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 360, in _configure
    _initialize_components(kwargs.get("auto_instrumentation_version"))
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 304, in _initialize_components
    trace_exporters, metric_exporters, log_exporters = _import_exporters(
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 272, in _import_exporters
    for (exporter_name, exporter_impl,) in _import_config_components(
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 245, in _import_config_components
    component_impl = entry_point.load()
  File "/otel-auto-instrumentation/pkg_resources/__init__.py", line 2471, in load
    return self.resolve()
  File "/otel-auto-instrumentation/pkg_resources/__init__.py", line 2477, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/otel-auto-instrumentation/opentelemetry/exporter/otlp/proto/grpc/metric_exporter/__init__.py", line 18, in <module>
    from grpc import ChannelCredentials, Compression
  File "/otel-auto-instrumentation/grpc/__init__.py", line 22, in <module>
    from grpc import _compression
  File "/otel-auto-instrumentation/grpc/_compression.py", line 15, in <module>
    from grpc._cython import cygrpc
ImportError: cannot import name 'cygrpc' from 'grpc._cython' (/otel-auto-instrumentation/grpc/_cython/__init__.py)
Failed to auto initialize opentelemetry
Traceback (most recent call last):
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 121, in initialize
    _load_configurators()
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 109, in _load_configurators
    raise exc
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 105, in _load_configurators
    entry_point.load()().configure(auto_instrumentation_version=__version__)  # type: ignore
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 344, in configure
    self._configure(**kwargs)
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 360, in _configure
    _initialize_components(kwargs.get("auto_instrumentation_version"))
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 304, in _initialize_components
    trace_exporters, metric_exporters, log_exporters = _import_exporters(
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 272, in _import_exporters
    for (exporter_name, exporter_impl,) in _import_config_components(
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 245, in _import_config_components
    component_impl = entry_point.load()
  File "/otel-auto-instrumentation/pkg_resources/__init__.py", line 2471, in load
    return self.resolve()
  File "/otel-auto-instrumentation/pkg_resources/__init__.py", line 2477, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/otel-auto-instrumentation/opentelemetry/exporter/otlp/proto/grpc/metric_exporter/__init__.py", line 18, in <module>
    from grpc import ChannelCredentials, Compression
  File "/otel-auto-instrumentation/grpc/__init__.py", line 22, in <module>
    from grpc import _compression
  File "/otel-auto-instrumentation/grpc/_compression.py", line 15, in <module>
    from grpc._cython import cygrpc
ImportError: cannot import name 'cygrpc' from 'grpc._cython' (/otel-auto-instrumentation/grpc/_cython/__init__.py)

I've also tried updating all pip packages and dependencies to utilise opentelemetry-exporter-otlp-proto version 1.13.0 & opentelemetry-distro version 0.34b0 in this file: https://github.com/open-telemetry/opentelemetry-operator/blob/main/autoinstrumentation/python/requirements.txt
But I get the same error Requested component 'otlp_proto_grpc' not found.

To rule out other issue, I've tested dotnet auto-injeciton, and that works fine with no issues.

Regards

[dominthomas]

It looks like this issue was fixed as part of this PR: #2748
But that doesn't seem to be the case 🤔 @srikanthccv - any ideas?

[srikanthccv]

I don't think the operator supports the grpc for the reasons stated in the requirements.txt file. And there is no metrics HTTP library at the moment but that doesn't raise an issue.

[dominthomas]

Yes, you're right, I did see that comment - so not sure what's causing this.
Is it a case of dsiabling opentelemetry_metrics_exporter? Not sure if that's possible or if it's a req for that.

[srikanthccv]

If you are using the operator you would have to use the HTTP exporter for traces and logs and set the exporter env to otlp_proto_http instead of otlp_proto_grpc

[dominthomas]

Tried that, same error... looks like it's expecting otlp_proto_grpc for metrics regardless.

[aabmass]

Tried that, same error... looks like it's expecting otlp_proto_grpc for metrics regardless.

I was able to get it working with @srikanthccv's workaround, in the instrumentation manifest:

  python:
    env:
    - name: OTEL_METRICS_EXPORTER
      value: none

---

One issue is the opentelemetry-operator doesn't have a build of 0.34b0 and it is by default using very old 0.32b0. We need to bump the deps and have them build some new images, opened open-telemetry/opentelemetry-operator#1147

The other problem is the operator should probably set OTEL_METRICS_EXPORTER=none until open-telemetry/opentelemetry-python#2447 is fixed to prevent it from crashing. I don't the user should need to do this manual workaround. Opened open-telemetry/opentelemetry-operator#1149

[vasireddy99]

+1
I was able to get working by setting OTEL_METRICS_EXPORTER:NONE from the last couple of releases.

[aabmass]

This should be fixed in the latest release https://github.com/open-telemetry/opentelemetry-operator/releases/tag/v0.61.0. However you'll still have to change the OTLP endpoint for traces to use port 4318 as described in open-telemetry/opentelemetry-operator#924, but that is a separate issue.