Merge branch 'main' of github.com:open-telemetry/opentelemetry-python…

…-contrib into django-metrics-route

.pylintrc

@@ -45,6 +45,9 @@ suggestion-mode=yes
 # active Python interpreter and may run arbitrary code.
 unsafe-load-any-extension=no
 
+# Run python dependant checks considering the baseline version
+py-version=3.8
+
 
 [MESSAGES CONTROL]
 

CHANGELOG.md

@@ -7,10 +7,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## Unreleased
 
+- `opentelemetry-instrumentation-aws-lambda` Bugfix: AWS Lambda event source key incorrect for SNS in instrumentation library. 
+  ([#2612](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/2612))
+- `opentelemetry-instrumentation-system-metrics` Permit to use psutil 6.0+.
+  ([#2630](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/2630))
+
 ### Added
 
+- `opentelemetry-instrumentation-pyramid` Record exceptions raised when serving a request
+  ([#2622](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/2622))
 - `opentelemetry-sdk-extension-aws` Add AwsXrayLambdaPropagator
   ([#2573](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/2573))
+- `opentelemetry-instrumentation-confluent-kafka` Add support for version 2.4.0 of confluent_kafka
+  ([#2616](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/2616))
+- `opentelemetry-instrumentation-confluent-kafka` Add support for produce purge
+  ([#2638](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/2638))
 
 ### Breaking changes
 
@@ -37,13 +48,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   ([#2300](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/2300))
 - Rename AwsLambdaInstrumentor span attributes `faas.id` to `cloud.resource_id`, `faas.execution` to `faas.invocation_id`
   ([#2372](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/2372))
-- Drop support for instrumenting elasticsearch client < 6`
+- Drop support for instrumenting elasticsearch client < 6
   ([#2422](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/2422))
 - `opentelemetry-instrumentation-wsgi` Add `http.method` to `span.name`
   ([#2425](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/2425))
 - `opentelemetry-instrumentation-flask` Add `http.method` to `span.name`
   ([#2454](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/2454))
-- ASGI, FastAPI, Starlette: provide both send and receive hooks with `scope` and `message` for internal spans ([#2546](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/2546))
+- Record repeated HTTP headers in lists, rather than a comma separate strings for ASGI based web frameworks
+  ([#2361](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/2361))
+- ASGI, FastAPI, Starlette: provide both send and receive hooks with `scope` and `message` for internal spans
+- ([#2546](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/2546))
 
 ### Added
 

CONTRIBUTING.md

@@ -55,18 +55,18 @@ some aspects of development, including testing against multiple Python versions.
 To install `tox`, run:
 
 ```sh
-pip install tox
+$ pip install tox
 ```
 
 You can run `tox` with the following arguments:
 
-- `tox` to run all existing tox commands, including unit tests for all packages
+* `tox` to run all existing tox commands, including unit tests for all packages
   under multiple Python versions
-- `tox -e docs` to regenerate the API docs
-- `tox -e py312-test-instrumentation-aiopg` to e.g. run the aiopg instrumentation unit tests under a specific
+* `tox -e docs` to regenerate the API docs
+* `tox -e py312-test-instrumentation-aiopg` to e.g. run the aiopg instrumentation unit tests under a specific
   Python version
-- `tox -e spellcheck` to run a spellcheck on all the code
-- `tox -e lint-some-package` to run lint checks on `some-package`
+* `tox -e spellcheck` to run a spellcheck on all the code
+* `tox -e lint-some-package` to run lint checks on `some-package`
 
 `black` and `isort` are executed when `tox -e lint` is run. The reported errors can be tedious to fix manually.
 An easier way to do so is:
@@ -84,6 +84,7 @@ You can also configure it to run lint tools automatically before committing with
 
 ```console
 $ pre-commit install
+```
 
 See
 [`tox.ini`](https://github.com/open-telemetry/opentelemetry-python-contrib/blob/main/tox.ini)
@@ -128,13 +129,14 @@ pull requests (PRs).
 To create a new PR, fork the project in GitHub and clone the upstream repo:
 
 ```sh
-git clone https://github.com/open-telemetry/opentelemetry-python-contrib.git
+$ git clone https://github.com/open-telemetry/opentelemetry-python-contrib.git
+$ cd opentelemetry-python-contrib
 ```
 
 Add your fork as an origin:
 
 ```sh
-git remote add fork https://github.com/YOUR_GITHUB_USERNAME/opentelemetry-python-contrib.git
+$ git remote add fork https://github.com/YOUR_GITHUB_USERNAME/opentelemetry-python-contrib.git
 ```
 
 Run tests:
@@ -148,10 +150,10 @@ $ tox  # execute in the root of the repository
 Check out a new branch, make modifications and push the branch to your fork:
 
 ```sh
-git checkout -b feature
+$ git checkout -b feature
 # edit files
-git commit
-git push fork feature
+$ git commit
+$ git push fork feature
 ```
 
 Open a pull request against the main `opentelemetry-python-contrib` repo.
@@ -160,6 +162,7 @@ Open a pull request against the main `opentelemetry-python-contrib` repo.
 
 * If the PR is not ready for review, please put `[WIP]` in the title, tag it
   as `work-in-progress`, or mark it as [`draft`](https://github.blog/2019-02-14-introducing-draft-pull-requests/).
+* Make sure tests and lint are passing locally before requesting a review.
 * Make sure CLA is signed and CI is clear.
 
 ### How to Get PRs Reviewed
@@ -215,13 +218,26 @@ For a deeper discussion, see: https://github.com/open-telemetry/opentelemetry-sp
 2. Make sure you have `tox` installed. `pip install tox`.
 3. Run `tox` without any arguments to run tests for all the packages. Read more about [tox](https://tox.readthedocs.io/en/latest/).
 
+Some tests can be slow due to pre-steps that do dependencies installs. To help with that, you can run tox a first time, and after that run the tests using previous installed dependencies in toxdir as following:
+
+1. First time run (e.g., opentelemetry-instrumentation-aiopg)
+```console
+tox -e py312-test-instrumentation-aiopg
+```
+2. Run tests again without pre-steps:
+```console
+.tox/py312-test-instrumentation-aiopg/bin/pytest instrumentation/opentelemetry-instrumentation-aiopg
+```
+
 ### Testing against a different Core repo branch/commit
 
 Some of the tox targets install packages from the [OpenTelemetry Python Core Repository](https://github.com/open-telemetry/opentelemetry-python) via pip. The version of the packages installed defaults to the main branch in that repository when tox is run locally. It is possible to install packages tagged with a specific git commit hash by setting an environment variable before running tox as per the following example:
 
+```sh
 CORE_REPO_SHA=c49ad57bfe35cfc69bfa863d74058ca9bec55fc3 tox
+```
 
-The continuation integration overrides that environment variable with as per the configuration [here](https://github.com/open-telemetry/opentelemetry-python-contrib/blob/2518a4ac07cb62ad6587dd8f6cbb5f8663a7e179/.github/workflows/test.yml#L9).
+The continuous integration overrides that environment variable with as per the configuration [here](https://github.com/open-telemetry/opentelemetry-python-contrib/blob/main/.github/workflows/test.yml#L9).
 
 ## Style Guide
 
@@ -259,3 +275,24 @@ Below is a checklist of things to be mindful of when implementing a new instrume
 ## Expectations from contributors
 
 OpenTelemetry is an open source community, and as such, greatly encourages contributions from anyone interested in the project. With that being said, there is a certain level of expectation from contributors even after a pull request is merged, specifically pertaining to instrumentations. The OpenTelemetry Python community expects contributors to maintain a level of support and interest in the instrumentations they contribute. This is to ensure that the instrumentation does not become stale and still functions the way the original contributor intended. Some instrumentations also pertain to libraries that the current members of the community are not so familiar with, so it is necessary to rely on the expertise of the original contributing parties.
+
+## Updating supported Python versions
+
+### Bumping the Python baseline
+
+When updating the minimum supported Python version remember to:
+
+- Remove the version in `pyproject.toml` trove classifiers
+- Remove the version from `tox.ini`
+- Search for `sys.version_info` usage and remove code for unsupported versions
+- Bump `py-version` in `.pylintrc` for Python version dependent checks
+
+### Adding support for a new Python release
+
+When adding support for a new Python release remember to:
+
+- Add the version in `tox.ini`
+- Add the version in `pyproject.toml` trove classifiers
+- Update github workflows accordingly; lint and benchmarks use the latest supported version
+- Update `.pre-commit-config.yaml`
+- Update tox examples in the documentation

README.md

@@ -46,7 +46,6 @@ The Python auto-instrumentation libraries for [OpenTelemetry](https://openteleme
 * [Releasing](#releasing)
   * [Releasing a package as `1.0` stable](#releasing-a-package-as-10-stable)
 * [Contributing](#contributing)
-* [Running Tests Locally](#running-tests-locally)
 * [Thanks to all the people who already contributed](#thanks-to-all-the-people-who-already-contributed)
 
 ## Installation
@@ -143,14 +142,6 @@ Emeritus Maintainers:
 
 *Find more about the maintainer role in [community repository](https://github.com/open-telemetry/community/blob/main/community-membership.md#maintainer).*
 
-## Running Tests Locally
-
-1. Go to your Contrib repo directory. `cd ~/git/opentelemetry-python-contrib`.
-2. Create a virtual env in your Contrib repo directory. `python3 -m venv my_test_venv`.
-3. Activate your virtual env. `source my_test_venv/bin/activate`.
-4. Make sure you have `tox` installed. `pip install tox`.
-5. Run tests for a package. (e.g. `tox -e test-instrumentation-flask`.)
-
 ### Thanks to all the people who already contributed
 
 <a href="https://github.com/open-telemetry/opentelemetry-python-contrib/graphs/contributors">

dev-requirements.txt

@@ -14,7 +14,7 @@ bleach==4.1.0 # transient dependency for readme-renderer
 protobuf~=3.13
 markupsafe>=2.0.1
 codespell==2.1.0
-requests==2.31.0
+requests==2.32.3
 ruamel.yaml==0.17.21
 flaky==3.7.0
 pre-commit==3.7.0; python_version >= '3.9'

docs-requirements.txt

@@ -27,7 +27,7 @@ botocore~=1.0
 boto3~=1.0
 cassandra-driver~=3.25
 celery>=4.0
-confluent-kafka>= 1.8.2,<= 2.3.0
+confluent-kafka>= 1.8.2,<= 2.4.0
 elasticsearch>=6.0,<9.0
 flask~=2.0
 falcon~=2.0
@@ -38,8 +38,8 @@ mysqlclient~=2.1.1
 psutil>=5
 psycopg~=3.1.17
 pika>=0.12.0
-pymongo~=3.1
-PyMySQL~=0.9.3
+pymongo~=4.6.3
+PyMySQL~=1.1.1
 pyramid>=1.7
 redis>=2.6
 remoulade>=0.50

exporter/opentelemetry-exporter-prometheus-remote-write/test-requirements.txt

@@ -16,7 +16,7 @@ python-snappy==0.7.1
 requests==2.32.3
 tomli==2.0.1
 typing_extensions==4.10.0
-urllib3==2.2.1
+urllib3==2.2.2
 wrapt==1.16.0
 zipp==3.17.0
 -e exporter/opentelemetry-exporter-prometheus-remote-write

instrumentation/README.md

@@ -14,7 +14,7 @@
 | [opentelemetry-instrumentation-botocore](./opentelemetry-instrumentation-botocore) | botocore ~= 1.0 | No | experimental
 | [opentelemetry-instrumentation-cassandra](./opentelemetry-instrumentation-cassandra) | cassandra-driver ~= 3.25,scylla-driver ~= 3.25 | No | experimental
 | [opentelemetry-instrumentation-celery](./opentelemetry-instrumentation-celery) | celery >= 4.0, < 6.0 | No | experimental
-| [opentelemetry-instrumentation-confluent-kafka](./opentelemetry-instrumentation-confluent-kafka) | confluent-kafka >= 1.8.2, <= 2.3.0 | No | experimental
+| [opentelemetry-instrumentation-confluent-kafka](./opentelemetry-instrumentation-confluent-kafka) | confluent-kafka >= 1.8.2, <= 2.4.0 | No | experimental
 | [opentelemetry-instrumentation-dbapi](./opentelemetry-instrumentation-dbapi) | dbapi | No | experimental
 | [opentelemetry-instrumentation-django](./opentelemetry-instrumentation-django) | django >= 1.10 | Yes | experimental
 | [opentelemetry-instrumentation-elasticsearch](./opentelemetry-instrumentation-elasticsearch) | elasticsearch >= 6.0 | No | experimental

instrumentation/opentelemetry-instrumentation-aiohttp-client/test-requirements.txt

@@ -25,7 +25,7 @@ pytest-benchmark==4.0.0
 requests==2.32.3
 tomli==2.0.1
 typing_extensions==4.10.0
-urllib3==2.2.1
+urllib3==2.2.2
 Werkzeug==3.0.3
 wrapt==1.16.0
 yarl==1.9.4

instrumentation/opentelemetry-instrumentation-aiohttp-client/tests/test_aiohttp_client_integration.py

@@ -14,7 +14,6 @@
 
 import asyncio
 import contextlib
-import sys
 import typing
 import unittest
 import urllib.parse
@@ -117,10 +116,6 @@ def test_status_codes(self):
                 )
 
                 url = f"http://{host}:{port}/test-path?query=param#foobar"
-                # if python version is < 3.8, then the url will be
-                if sys.version_info[1] < 8:
-                    url = f"http://{host}:{port}/test-path#foobar"
-
                 self.assert_spans(
                     [
                         (

instrumentation/opentelemetry-instrumentation-asgi/src/opentelemetry/instrumentation/asgi/__init__.py

@@ -129,10 +129,10 @@ def client_response_hook(span: Span, scope: dict[str, Any], message: dict[str, A
 
 The name of the added span attribute will follow the format ``http.request.header.<header_name>`` where ``<header_name>``
 is the normalized HTTP header name (lowercase, with ``-`` replaced by ``_``). The value of the attribute will be a
-single item list containing all the header values.
+list containing the header values.
 
 For example:
-``http.request.header.custom_request_header = ["<value1>,<value2>"]``
+``http.request.header.custom_request_header = ["<value1>", "<value2>"]``
 
 Response headers
 ****************
@@ -163,10 +163,10 @@ def client_response_hook(span: Span, scope: dict[str, Any], message: dict[str, A
 
 The name of the added span attribute will follow the format ``http.response.header.<header_name>`` where ``<header_name>``
 is the normalized HTTP header name (lowercase, with ``-`` replaced by ``_``). The value of the attribute will be a
-single item list containing all the header values.
+list containing the header values.
 
 For example:
-``http.response.header.custom_response_header = ["<value1>,<value2>"]``
+``http.response.header.custom_response_header = ["<value1>", "<value2>"]``
 
 Sanitizing headers
 ******************
@@ -193,9 +193,10 @@ def client_response_hook(span: Span, scope: dict[str, Any], message: dict[str, A
 
 import typing
 import urllib
+from collections import defaultdict
 from functools import wraps
 from timeit import default_timer
-from typing import Any, Awaitable, Callable, Tuple
+from typing import Any, Awaitable, Callable, DefaultDict, Tuple
 
 from asgiref.compatibility import guarantee_single_callable
 
@@ -340,24 +341,19 @@ def collect_custom_headers_attributes(
     sanitize: SanitizeValue,
     header_regexes: list[str],
     normalize_names: Callable[[str], str],
-) -> dict[str, str]:
+) -> dict[str, list[str]]:
     """
     Returns custom HTTP request or response headers to be added into SERVER span as span attributes.
 
     Refer specifications:
      - https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/http.md#http-request-and-response-headers
     """
-    # Decode headers before processing.
-    headers: dict[str, str] = {}
+    headers: DefaultDict[str, list[str]] = defaultdict(list)
     raw_headers = scope_or_response_message.get("headers")
     if raw_headers:
-        for _key, _value in raw_headers:
-            key = _key.decode().lower()
-            value = _value.decode()
-            if key in headers:
-                headers[key] += f",{value}"
-            else:
-                headers[key] = value
+        for key, value in raw_headers:
+            # Decode headers before processing.
+            headers[key.decode()].append(value.decode())
 
     return sanitize.sanitize_header_values(
         headers,

instrumentation/opentelemetry-instrumentation-asgi/tests/test_asgi_custom_headers.py

@@ -152,7 +152,8 @@ def test_http_repeat_request_headers_in_span_attributes(self):
         span_list = self.exporter.get_finished_spans()
         expected = {
             "http.request.header.custom_test_header_1": (
-                "test-header-value-1,test-header-value-2",
+                "test-header-value-1",
+                "test-header-value-2",
             ),
         }
         span = next(span for span in span_list if span.kind == SpanKind.SERVER)
@@ -225,7 +226,8 @@ def test_http_repeat_response_headers_in_span_attributes(self):
         span_list = self.exporter.get_finished_spans()
         expected = {
             "http.response.header.custom_test_header_1": (
-                "test-header-value-1,test-header-value-2",
+                "test-header-value-1",
+                "test-header-value-2",
             ),
         }
         span = next(span for span in span_list if span.kind == SpanKind.SERVER)

instrumentation/opentelemetry-instrumentation-aws-lambda/src/opentelemetry/instrumentation/aws_lambda/__init__.py

@@ -306,9 +306,11 @@ def _instrumented_lambda_handler_call(  # noqa pylint: disable=too-many-branches
             disable_aws_context_propagation,
         )
 
-        span_kind = None
         try:
-            if lambda_event["Records"][0]["eventSource"] in {
+            event_source = lambda_event["Records"][0].get(
+                "eventSource"
+            ) or lambda_event["Records"][0].get("EventSource")
+            if event_source in {
                 "aws:sqs",
                 "aws:s3",
                 "aws:sns",