[Bug]: Instana interferes with exception raised when connecting to database

[ghost]

Problem Description

When connecting to a database server using sqlalchemy usually an OperationalError is raised. However, Instana somehow intercepts this and instead raises an AttributeError (see stacktrace below).

This is an issue, since python-testcontainers since version 3.5 checks for explicit Exceptions to decide whether to wait or not and AttributeError is not one of them: testcontainers/testcontainers-python#193

self = <instana.instrumentation.pep0249.ConnectionFactory object at 0x10a3e0f40>
args = ()
kwargs = {'database': 'test', 'host': 'localhost', 'password': 'test', 'port': 58725, ...}
connect_params = ((), {'database': 'test', 'host': 'localhost', 'password': 'test', 'port': 58725, ...})

    def __call__(self, *args, **kwargs):
        connect_params = (args, kwargs) if args or kwargs else None
    
        return self._wrapper_ctor(
>           connection=self._connect_func(*args, **kwargs),
            module_name=self._module_name,
            connect_params=connect_params)

../../.venv/lib/python3.9/site-packages/instana/instrumentation/pep0249.py:133: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

dsn = 'host=localhost user=test password=test port=58725 dbname=test'
connection_factory = None, cursor_factory = None
kwargs = {'database': 'test', 'host': 'localhost', 'password': 'test', 'port': 58725, ...}
kwasync = {}

    def connect(dsn=None, connection_factory=None, cursor_factory=None, **kwargs):
        """
        Create a new database connection.
    
        The connection parameters can be specified as a string:
    
            conn = psycopg2.connect("dbname=test user=postgres password=secret")
    
        or using a set of keyword arguments:
    
            conn = psycopg2.connect(database="test", user="postgres", password="secret")
    
        Or as a mix of both. The basic connection parameters are:
    
        - *dbname*: the database name
        - *database*: the database name (only as keyword argument)
        - *user*: user name used to authenticate
        - *password*: password used to authenticate
        - *host*: database host address (defaults to UNIX socket if not provided)
        - *port*: connection port number (defaults to 5432 if not provided)
    
        Using the *connection_factory* parameter a different class or connections
        factory can be specified. It should be a callable object taking a dsn
        argument.
    
        Using the *cursor_factory* parameter, a new default cursor factory will be
        used by cursor().
    
        Using *async*=True an asynchronous connection will be created. *async_* is
        a valid alias (for Python versions where ``async`` is a keyword).
    
        Any other keyword parameter will be passed to the underlying client
        library: the list of supported parameters depends on the library version.
    
        """
        kwasync = {}
        if 'async' in kwargs:
            kwasync['async'] = kwargs.pop('async')
        if 'async_' in kwargs:
            kwasync['async_'] = kwargs.pop('async_')
    
        dsn = _ext.make_dsn(dsn, **kwargs)
>       conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
E       psycopg2.OperationalError: connection to server at "localhost" (127.0.0.1), port 58725 failed: server closed the connection unexpectedly
E       	This probably means the server terminated abnormally
E       	before or while processing the request.

../../.venv/lib/python3.9/site-packages/psycopg2/__init__.py:122: OperationalError

The above exception was the direct cause of the following exception:

pytestconfig = <_pytest.config.Config object at 0x1075b5610>

    @pytest.fixture(scope="session")
    def _db_container(pytestconfig: Any) -> PostgresContainer:
>       with _create_container(
            "timescale/timescaledb:latest-pg11",
            PostgresContainer,
        ) as container:

../conftest.py:107: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../.venv/lib/python3.9/site-packages/testcontainers/core/container.py:69: in __enter__
    return self.start()
../../.venv/lib/python3.9/site-packages/testcontainers/core/generic.py:50: in start
    self._connect()
../../.venv/lib/python3.9/site-packages/testcontainers/core/waiting_utils.py:49: in wrapper
    return wrapped(*args, **kwargs)
../../.venv/lib/python3.9/site-packages/testcontainers/core/generic.py:28: in _connect
    engine.connect()
../../.venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py:2263: in connect
    return self._connection_cls(self, **kwargs)
../../.venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py:104: in __init__
    else engine.raw_connection()
../../.venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py:2369: in raw_connection
    return self._wrap_pool_connect(
../../.venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py:2339: in _wrap_pool_connect
    Connection._handle_dbapi_exception_noconnection(
../../.venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py:1581: in _handle_dbapi_exception_noconnection
    util.raise_(newraise, with_traceback=exc_info[2], from_=e)
../../.venv/lib/python3.9/site-packages/sqlalchemy/util/compat.py:182: in raise_
    raise exception
../../.venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py:2336: in _wrap_pool_connect
    return fn()
../../.venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py:304: in unique_connection
    return _ConnectionFairy._checkout(self)
../../.venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py:778: in _checkout
    fairy = _ConnectionRecord.checkout(pool)
../../.venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py:495: in checkout
    rec = pool._do_get()
../../.venv/lib/python3.9/site-packages/sqlalchemy/pool/impl.py:140: in _do_get
    self._dec_overflow()
../../.venv/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py:68: in __exit__
    compat.raise_(
../../.venv/lib/python3.9/site-packages/sqlalchemy/util/compat.py:182: in raise_
    raise exception
../../.venv/lib/python3.9/site-packages/sqlalchemy/pool/impl.py:137: in _do_get
    return self._create_connection()
../../.venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py:309: in _create_connection
    return _ConnectionRecord(self)
../../.venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py:440: in __init__
    self.__connect(first_connect_check=True)
../../.venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py:661: in __connect
    pool.logger.debug("Error on connect(): %s", e)
../../.venv/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py:68: in __exit__
    compat.raise_(
../../.venv/lib/python3.9/site-packages/sqlalchemy/util/compat.py:182: in raise_
    raise exception
../../.venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py:656: in __connect
    connection = pool._invoke_creator(self)
../../.venv/lib/python3.9/site-packages/sqlalchemy/engine/strategies.py:114: in connect
    return dialect.connect(*cargs, **cparams)
../../.venv/lib/python3.9/site-packages/sqlalchemy/engine/default.py:508: in connect
    return self.dbapi.connect(*cargs, **cparams)
../../.venv/lib/python3.9/site-packages/instana/instrumentation/pep0249.py:133: in __call__
    connection=self._connect_func(*args, **kwargs),
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

dsn = 'host=localhost user=test password=test port=58725 dbname=test'
connection_factory = None, cursor_factory = None
kwargs = {'database': 'test', 'host': 'localhost', 'password': 'test', 'port': 58725, ...}
kwasync = {}

    def connect(dsn=None, connection_factory=None, cursor_factory=None, **kwargs):
        """
        Create a new database connection.
    
        The connection parameters can be specified as a string:
    
            conn = psycopg2.connect("dbname=test user=postgres password=secret")
    
        or using a set of keyword arguments:
    
            conn = psycopg2.connect(database="test", user="postgres", password="secret")
    
        Or as a mix of both. The basic connection parameters are:
    
        - *dbname*: the database name
        - *database*: the database name (only as keyword argument)
        - *user*: user name used to authenticate
        - *password*: password used to authenticate
        - *host*: database host address (defaults to UNIX socket if not provided)
        - *port*: connection port number (defaults to 5432 if not provided)
    
        Using the *connection_factory* parameter a different class or connections
        factory can be specified. It should be a callable object taking a dsn
        argument.
    
        Using the *cursor_factory* parameter, a new default cursor factory will be
        used by cursor().
    
        Using *async*=True an asynchronous connection will be created. *async_* is
        a valid alias (for Python versions where ``async`` is a keyword).
    
        Any other keyword parameter will be passed to the underlying client
        library: the list of supported parameters depends on the library version.
    
        """
        kwasync = {}
        if 'async' in kwargs:
            kwasync['async'] = kwargs.pop('async')
        if 'async_' in kwargs:
            kwasync['async_'] = kwargs.pop('async_')
    
        dsn = _ext.make_dsn(dsn, **kwargs)
>       conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
E       AttributeError: 'NoneType' object has no attribute '_stan_scope'

../../.venv/lib/python3.9/site-packages/psycopg2/__init__.py:122: AttributeError

Minimal, Complete, Verifiable, Example

import sqlalchemy
from testcontainers.mysql import PostgresContainer

with PostgresContainer("dockette/postgres:latest") as container:
    engine = sqlalchemy.create_engine(container.get_connection_url())
    version, = engine.execute("select version()").fetchone()
    print(version)

Python Version

Python 3.9

Python Modules

requirements.txt


alembic==1.7.7; python_version >= "3.6"
argcomplete==2.0.0; python_full_version >= "3.6.1" and python_full_version < "4.0.0" and python_version >= "3.7" and python_version < "4.0"
ariadne==0.14.1
atomicwrites==1.4.0; python_version >= "3.7" and python_full_version < "3.0.0" and sys_platform == "win32" or sys_platform == "win32" and python_version >= "3.7" and python_full_version >= "3.4.0"
attrs==21.4.0; python_full_version >= "3.6.1" and python_version >= "3.7" and python_full_version < "4.0.0" and python_version < "4.0"
autowrapt==1.0
basictracer==3.2.0
black==22.3.0; python_full_version >= "3.6.2"
certifi==2021.10.8; python_version >= "3.7" and python_full_version < "3.0.0" and python_version < "4.0" or python_full_version >= "3.6.0" and python_version < "4" and python_version >= "3.7"
cffi==1.15.0; python_version >= "3.6"
chardet==4.0.0; python_full_version >= "3.6.1" and python_full_version < "4.0.0" and python_version >= "3.7" and python_version < "4.0"
charset-normalizer==2.0.7; python_full_version >= "3.5.0"
click==8.1.2; python_version >= "3.7" and python_full_version >= "3.6.2" and python_full_version < "4.0.0" and python_version < "4.0"
colorama==0.4.4; sys_platform == "win32" and python_version >= "3.7" and python_full_version >= "3.6.2" and platform_system == "Windows" and (python_version >= "3.7" and python_full_version < "3.0.0" and sys_platform == "win32" or sys_platform == "win32" and python_version >= "3.7" and python_full_version >= "3.5.0")
coverage==6.3.2; python_version >= "3.7"
cryptography==3.4.8; python_version >= "3.6"
datamodel-code-generator==0.11.20; python_full_version >= "3.6.1" and python_full_version < "4.0.0" and python_version >= "3.7" and python_version < "4.0"
deprecation==2.1.0; python_version >= "3.5"
dnspython==2.2.1; python_version >= "3.7" and python_version < "4.0" and python_full_version >= "3.6.1"
docker==5.0.3; python_version >= "3.6"
elasticsearch-dsl==7.4.0; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.4.0")
elasticsearch==7.17.2; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.4.0" and python_version < "4")
email-validator==1.1.3; python_full_version >= "3.6.1" and python_version >= "3.7" and python_version < "4.0"
fastapi==0.68.2; python_full_version >= "3.6.1"
flake8==4.0.1; python_version >= "3.6"
fysom==2.1.6
genson==1.2.2; python_full_version >= "3.6.1" and python_full_version < "4.0.0" and python_version >= "3.7" and python_version < "4.0"
graphql-core==3.1.7; python_version >= "3.6" and python_version < "4"
idna==3.3; python_full_version >= "3.6.1" and python_version >= "3.7" and python_version < "4.0"
importlib-metadata==4.2.0; python_version == "3.7" and python_full_version >= "3.6.2" and python_full_version < "4.0.0"
importlib-resources==5.6.0; python_version < "3.9" and python_version >= "3.7"
inflect==5.5.2; python_full_version >= "3.6.1" and python_full_version < "4.0.0" and python_version >= "3.7" and python_version < "4.0"
iniconfig==1.1.1; python_version >= "3.7"
instana==1.37.1
isodate==0.6.1; python_full_version >= "3.6.1" and python_full_version < "4.0.0" and python_version >= "3.7" and python_version < "4.0"
isort==5.10.1; python_full_version >= "3.6.1" and python_version < "4.0" and python_full_version < "4.0.0" and python_version >= "3.7"
jinja2==3.1.1; python_full_version >= "3.6.1" and python_full_version < "4.0.0" and python_version >= "3.7" and python_version < "4.0"
jsonschema==3.2.0; python_full_version >= "3.6.1" and python_full_version < "4.0.0" and python_version >= "3.7" and python_version < "4.0"
mako==1.2.0; python_version >= "3.7"
markupsafe==2.1.1; python_full_version >= "3.6.1" and python_full_version < "4.0.0" and python_version >= "3.7" and python_version < "4.0"
mccabe==0.6.1; python_version >= "3.6"
mockito==1.3.0
mypy-extensions==0.4.3; python_full_version >= "3.6.2" and python_version >= "3.7" and python_full_version < "4.0.0" and python_version < "4.0"
mypy==0.942; python_version >= "3.6"
openapi-schema-validator==0.1.6; python_full_version >= "3.6.1" and python_full_version < "4.0.0" and python_version >= "3.7" and python_version < "4.0"
openapi-spec-validator==0.3.3; python_full_version >= "3.6.1" and python_full_version < "4.0.0" and python_version >= "3.7" and python_version < "4.0"
opentracing==2.4.0
packaging==21.3; python_version >= "3.7"
pathspec==0.9.0; python_full_version >= "3.6.2" and python_full_version < "4.0.0" and python_version >= "3.7" and python_version < "4.0"
platformdirs==2.5.1; python_version >= "3.7" and python_full_version >= "3.6.2" and python_full_version < "4.0.0" and python_version < "4.0"
pluggy==1.0.0; python_version >= "3.7"
prance==0.21.8.0; python_full_version >= "3.6.1" and python_full_version < "4.0.0" and python_version >= "3.7" and python_version < "4.0"
protobuf==4.0.0rc2
psycopg2-binary==2.9.3; python_version >= "3.6"
py==1.11.0; python_version >= "3.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.7"
pycodestyle==2.8.0; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.6"
pycparser==2.21; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6"
pydantic==1.9.0; python_full_version >= "3.6.1"
pyflakes==2.4.0; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6"
pyjwt==1.7.1
pyparsing==3.0.7; python_version >= "3.7"
pyrsistent==0.16.1; python_full_version >= "3.6.1" and python_full_version < "4.0.0" and python_version >= "3.7" and python_version < "4.0"
pysnooper==1.1.1; python_full_version >= "3.6.1" and python_full_version < "4.0.0" and python_version >= "3.7" and python_version < "4.0"
pytest-asyncio==0.18.3; python_version >= "3.7"
pytest-cov==3.0.0; python_version >= "3.6"
pytest==7.1.1; python_version >= "3.7"
python-dateutil==2.8.2; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.4.0"
python-json-logger==2.0.2; python_version >= "3.5"
pywin32==227; sys_platform == "win32" and python_version >= "3.6"
pyyaml==6.0; python_version >= "3.6"
requests==2.27.1; python_version >= "3.7" and python_version < "4.0" and python_full_version >= "3.6.1" and python_full_version < "4.0.0"
ruamel.yaml.clib==0.2.6; platform_python_implementation == "CPython" and python_version < "3.11" and python_full_version >= "3.6.1" and python_full_version < "4.0.0" and python_version >= "3.7"
ruamel.yaml==0.17.21; python_full_version >= "3.6.1" and python_full_version < "4.0.0" and python_version >= "3.7" and python_version < "4.0"
semver==2.13.0; python_full_version >= "3.6.1" and python_full_version < "4.0.0" and python_version >= "3.7" and python_version < "4.0"
six==1.16.0; python_version >= "3.7" and python_version < "4.0" and python_full_version >= "3.6.1" and python_full_version < "4.0.0"
slacker==0.14.0
sqlalchemy==1.3.24; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.4.0")
starlette-context==0.3.3; python_version >= "3.7"
starlette==0.14.2; python_version >= "3.6"
testcontainers==3.5.3; python_version >= "3.6"
testfixtures==6.18.5
toml==0.10.2; python_full_version >= "3.6.1" and python_full_version < "4.0.0" and python_version >= "3.7" and python_version < "4.0"
tomli==2.0.1; python_version < "3.11" and python_full_version >= "3.6.2" and python_version >= "3.7" and python_full_version < "4.0.0"
typed-ast==1.5.2
typing-extensions==4.1.1; python_version >= "3.7" and python_version < "3.8" and python_full_version >= "3.6.2" and python_full_version < "4.0.0"
urllib3==1.26.9; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.5.0" and python_version < "4")
websocket-client==1.3.2; python_version >= "3.7"
wiremock==2.3.1; python_version >= "3.4"
wrapt==1.14.0; python_version >= "3.5" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.5"
zipp==3.8.0; python_version < "3.8" and python_version >= "3.7"

### Python Environment

```shell
PYTHONPATH=.

[Ferenc-]

Thanks for reporting.
Now the issue is on the internal board, it will be investigated.

[Ferenc-]

Hi @sebastian-hatala-leanix
Could you try out this initial workaround from this branch:
pip install -U git+https://github.com/instana/python-sensor@workaround/testcontainer-reconnect-sqlalchemy

[Ferenc-]

The fix has been merged.