Not able to auto-instrument sample python app using flask.

[biswajit-nanda]

Hello, Folks,

I have a very simple python app that I am trying to auto-instrument using the OpenTelemetry Operator. The injection is done and the pod is restarted after adding the annotations, but the python app does not send traces to the otel-collector running as sidecar.

Attached is a zip file PythonTestCase.zip that contains my python app, Dockerfile used to build the image and the yaml file used to deploy the app in K8s. It also contains the yaml file for the instrumentation resource, the yaml file that has the config for my sidecar otel collector, the sidecar otel collector logs and the kubectl describe output for my python application pod.

Steps Followed:
a. The python app was deployed using the attached yaml.
b. Cert Manager was installed.
c. Otel Operator was installed.
d. Deployed the attached SideCar Otel Collector configuration to the namespace where the app was deployed.
e. The instrumentation resource using the attached yaml was deployed to the same namespace where the app is deployed.
f. Required pod level annotations were added to the application deployment using the command below:
kubectl -n biswa-apps-python patch deployment python-downstream-eks -p '{"spec": {"template": {"metadata": {"annotations": {"sidecar.opentelemetry.io/inject":"true","instrumentation.opentelemetry.io/inject-python":"true"}}}}}'
g. Load was applied on the application pod from browser at: http://localhost:8080/hello after doing a K8s port forward.

Observations:
After applying the path, the application pod gets restarted as expected, the init container is created successfully and the copy task is performed successfully and the sidecar otel collector container is automatically added to the application pod. However, the traces from the application never make their way to the sidecar otel collector. I am verifying this by adding the debug level logging exported in the sidecar otel collector config.

I am not sure if I am missing something very obvious here or this is a bug or flask is not supported.

Environment:
K8s Version: 1.22.6 (Amazon EKS)
Otel Collector Version: v0.49.0 (automatically pulled by the operator)
Python Version: 3.6.15
OpenTelemetry Operator Version: latest (deployed from https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml)
Cert Manager Version: 1.7.1 (deployed from https://github.com/cert-manager/cert-manager/releases/download/v1.7.1/cert-manager.yaml)

Could someone kindly help me out here? I am running out of ideas now.

PythonTestCase.zip

[biswajit-nanda]

I see the following logs at my application startup:

Distribution distro configuration failed
Traceback (most recent call last):
File "/otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 37, in _load_distros
distro = entry_point.load()()
File "/otel-auto-instrumentation/pkg_resources/init.py", line 2470, in load
self.require(*args, **kwargs)
File "/otel-auto-instrumentation/pkg_resources/init.py", line 2493, in require
items = working_set.resolve(reqs, env, installer, extras=self.extras)
File "/otel-auto-instrumentation/pkg_resources/init.py", line 795, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'aiocontextvars; python_version < "3.7"' distribution was not found and is required by opentelemetry-api
Failed to auto initialize opentelemetry
Traceback (most recent call last):
File "/otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 114, in initialize
distro = _load_distros()
File "/otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 52, in _load_distros
raise exc
File "/otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 37, in _load_distros
distro = entry_point.load()()
File "/otel-auto-instrumentation/pkg_resources/init.py", line 2470, in load
self.require(*args, **kwargs)
File "/otel-auto-instrumentation/pkg_resources/init.py", line 2493, in require
items = working_set.resolve(reqs, env, installer, extras=self.extras)
File "/otel-auto-instrumentation/pkg_resources/init.py", line 795, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'aiocontextvars; python_version < "3.7"' distribution was not found and is required by opentelemetry-api

Looks like the instrumentation is not starting due to a python version issue.

[biswajit-nanda]

I now used a base image of 3.7 version of python and made progress. At least the application got instrumented and trying to emit the traces. However, the export fails with the following:

bisnanda@BISNANDA-M-W1CH:/Volumes/Data/AppDynamicsCloud/EKS$ kubectl -n biswa-apps-python logs -f python-downstream-eks-6f95f6f765-xc5gh -c python-downstream-eks

• Serving Flask app 'downstream-app' (lazy loading)
• Environment: production
  WARNING: This is a development server. Do not use it in a production deployment.
  Use a production WSGI server instead.
• Debug mode: off
• Running on all addresses (0.0.0.0)
  WARNING: This is a development server. Do not use it in a production deployment.
• Running on http://127.0.0.1:8080
• Running on http://172.25.21.201:8080 (Press CTRL+C to quit)
  172.25.20.104 - - [08/May/2022 01:46:12] "GET /hello HTTP/1.1" 200 -
  Exception while exporting Span batch.
  Traceback (most recent call last):
  File "/otel-auto-instrumentation/urllib3/connectionpool.py", line 710, in urlopen
  chunked=chunked,
  File "/otel-auto-instrumentation/urllib3/connectionpool.py", line 449, in _make_request
  six.raise_from(e, None)
  File "", line 3, in raise_from
  File "/otel-auto-instrumentation/urllib3/connectionpool.py", line 444, in _make_request
  httplib_response = conn.getresponse()
  File "/usr/local/lib/python3.7/http/client.py", line 1373, in getresponse
  response.begin()
  File "/usr/local/lib/python3.7/http/client.py", line 319, in begin
  version, status, reason = self._read_status()
  File "/usr/local/lib/python3.7/http/client.py", line 280, in _read_status
  line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/lib/python3.7/socket.py", line 589, in readinto
  return self._sock.recv_into(b)
  ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/otel-auto-instrumentation/requests/adapters.py", line 450, in send
timeout=timeout
File "/otel-auto-instrumentation/wrapt/wrappers.py", line 645, in call
args, kwargs)
File "/otel-auto-instrumentation/opentelemetry/instrumentation/urllib3/init.py", line 160, in instrumented_urlopen
return wrapped(*args, **kwargs)
File "/otel-auto-instrumentation/urllib3/connectionpool.py", line 786, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/otel-auto-instrumentation/urllib3/util/retry.py", line 550, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/otel-auto-instrumentation/urllib3/packages/six.py", line 769, in reraise
raise value.with_traceback(tb)
File "/otel-auto-instrumentation/urllib3/connectionpool.py", line 710, in urlopen
chunked=chunked,
File "/otel-auto-instrumentation/urllib3/connectionpool.py", line 449, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "/otel-auto-instrumentation/urllib3/connectionpool.py", line 444, in _make_request
httplib_response = conn.getresponse()
File "/usr/local/lib/python3.7/http/client.py", line 1373, in getresponse
response.begin()
File "/usr/local/lib/python3.7/http/client.py", line 319, in begin
version, status, reason = self._read_status()
File "/usr/local/lib/python3.7/http/client.py", line 280, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/local/lib/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/otel-auto-instrumentation/opentelemetry/sdk/trace/export/init.py", line 358, in _export_batch
self.span_exporter.export(self.spans_list[:idx]) # type: ignore
File "/otel-auto-instrumentation/opentelemetry/exporter/otlp/proto/http/trace_exporter/init.py", line 136, in export
resp = self._export(serialized_data)
File "/otel-auto-instrumentation/opentelemetry/exporter/otlp/proto/http/trace_exporter/init.py", line 111, in _export
timeout=self._timeout,
File "/otel-auto-instrumentation/requests/sessions.py", line 577, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/otel-auto-instrumentation/opentelemetry/instrumentation/requests/init.py", line 123, in instrumented_request
method, url, call_wrapped, get_or_create_headers
File "/otel-auto-instrumentation/opentelemetry/instrumentation/requests/init.py", line 152, in _instrumented_requests_call
return call_wrapped()
File "/otel-auto-instrumentation/opentelemetry/instrumentation/requests/init.py", line 120, in call_wrapped
return wrapped_request(self, method, url, *args, **kwargs)
File "/otel-auto-instrumentation/requests/sessions.py", line 529, in request
resp = self.send(prep, **send_kwargs)
File "/otel-auto-instrumentation/opentelemetry/instrumentation/requests/init.py", line 143, in instrumented_send
request.method, request.url, call_wrapped, get_or_create_headers
File "/otel-auto-instrumentation/opentelemetry/instrumentation/requests/init.py", line 152, in _instrumented_requests_call
return call_wrapped()
File "/otel-auto-instrumentation/opentelemetry/instrumentation/requests/init.py", line 140, in call_wrapped
return wrapped_send(self, request, **kwargs)
File "/otel-auto-instrumentation/requests/sessions.py", line 645, in send
r = adapter.send(request, **kwargs)
File "/otel-auto-instrumentation/requests/adapters.py", line 501, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

In the sidecar otel collector logs, I am seeing the following:
2022-05-08T01:46:16.275Z warn zapgrpc/zapgrpc.go:191 [core] grpc: Server.Serve failed to create ServerTransport: connection error: desc = "transport: http2Server.HandleStreams received bogus greeting from client: "POST / HTTP/1.1\r\nHost: l"" {"grpc_log": true}

[biswajit-nanda]

Workaround:

Found a workaround for this. In my Instrumentation Resource configuration, instead of using exporter endpoint as http://localhost:4317 (GRPC), I used http://localhost:4318/v1/traces (HTTP) and now spans from my python application are being exported successfully to my Otel Collector.

However, it still needs to be investigated why the traces fail to export while using GRPC endpoint in my instrumentation resource.

[mat-rumian]

@biswajit-nanda it doesn't have to be investigated :) Python auto-instrumentation is using OTLP HTTP protocol and only this exporter package is installed. You can find a list of all used packages here

[biswajit-nanda]

Thanks for the clarifications, @mat-rumian. I am closing this now.