airflow db migrate crashing with new connexion == 3.0.0 #35375

eselyavka · 2023-11-02T17:57:36Z

Apache Airflow version

2.7.2

What happened

Team, it looks like i just discovered bug in airflow 2.7.2(latest) which were introduced by this package https://pypi.org/project/connexion/3.0.0/, which were released several hours ago
I am running my airflow image in ubuntu 20.04 x86_64 docker image under the python 3.9.16

root@cbe8e0b21de3:/# python3.9 --version
Python 3.9.18

I am installing airflow via pip

pip install apache-airflow=2.7.2

I am seeing this error when performing airflow db migrate

root@cbe8e0b21de3:/# airflow db init
/usr/local/lib/python3.9/dist-packages/airflow/cli/commands/db_command.py:43 DeprecationWarning: `db init` is deprecated.  Use `db migrate` instead to migrate the db and/or airflow connections create-default-connections to create the default connections
DB: sqlite:////root/airflow/airflow.db
[2023-11-02T13:54:50.283-0400] {migration.py:213} INFO - Context impl SQLiteImpl.
[2023-11-02T13:54:50.285-0400] {migration.py:216} INFO - Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/dist-packages/airflow/__main__.py", line 59, in main
    args.func(args)
  File "/usr/local/lib/python3.9/dist-packages/airflow/cli/cli_config.py", line 49, in command
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/airflow/utils/providers_configuration_loader.py", line 55, in wrapped_function
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/airflow/cli/commands/db_command.py", line 49, in initdb
    db.initdb()
  File "/usr/local/lib/python3.9/dist-packages/airflow/utils/session.py", line 77, in wrapper
    return func(*args, session=session, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/airflow/utils/db.py", line 739, in initdb
    _create_db_from_orm(session=session)
  File "/usr/local/lib/python3.9/dist-packages/airflow/utils/db.py", line 727, in _create_db_from_orm
    command.stamp(config, "head")
  File "/usr/local/lib/python3.9/dist-packages/alembic/command.py", line 685, in stamp
    script.run_env()
  File "/usr/local/lib/python3.9/dist-packages/alembic/script/base.py", line 579, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/usr/local/lib/python3.9/dist-packages/alembic/util/pyfiles.py", line 93, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python3.9/dist-packages/alembic/util/pyfiles.py", line 109, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/lib/python3.9/dist-packages/airflow/migrations/env.py", line 117, in <module>
    run_migrations_online()
  File "/usr/local/lib/python3.9/dist-packages/airflow/migrations/env.py", line 111, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/usr/local/lib/python3.9/dist-packages/alembic/runtime/environment.py", line 938, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/usr/local/lib/python3.9/dist-packages/alembic/runtime/migration.py", line 612, in run_migrations
    for step in self._migrations_fn(heads, self):
  File "/usr/local/lib/python3.9/dist-packages/alembic/command.py", line 673, in do_stamp
    return script._stamp_revs(util.to_tuple(destination_revs), rev)
  File "/usr/local/lib/python3.9/dist-packages/alembic/script/base.py", line 494, in _stamp_revs
    self.revision_map.filter_for_lineage(
  File "/usr/local/lib/python3.9/dist-packages/alembic/script/revision.py", line 680, in filter_for_lineage
    id_, branch_label = self._resolve_revision_number(check_against)
  File "/usr/local/lib/python3.9/dist-packages/alembic/script/revision.py", line 754, in _resolve_revision_number
    self._revision_map
  File "/usr/local/lib/python3.9/dist-packages/sqlalchemy/util/langhelpers.py", line 1113, in __get__
    obj.__dict__[self.__name__] = result = self.fget(obj)
  File "/usr/local/lib/python3.9/dist-packages/alembic/script/revision.py", line 197, in _revision_map
    for revision in self._generator():
  File "/usr/local/lib/python3.9/dist-packages/alembic/script/base.py", line 149, in _load_revisions
    script = Script._from_filename(self, dir_name, filename)
  File "/usr/local/lib/python3.9/dist-packages/alembic/script/base.py", line 1035, in _from_filename
    module = util.load_python_file(dir_, filename)
  File "/usr/local/lib/python3.9/dist-packages/alembic/util/pyfiles.py", line 93, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python3.9/dist-packages/alembic/util/pyfiles.py", line 109, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/lib/python3.9/dist-packages/airflow/migrations/versions/0074_2_0_0_resource_based_permissions.py", line 30, in <module>
    from airflow.www.app import cached_app
  File "/usr/local/lib/python3.9/dist-packages/airflow/www/app.py", line 50, in <module>
    from airflow.www.extensions.init_views import (
  File "/usr/local/lib/python3.9/dist-packages/airflow/www/extensions/init_views.py", line 27, in <module>
    from connexion.decorators.validation import RequestBodyValidator
ModuleNotFoundError: No module named 'connexion.decorators.validation'

If i downgrade connexion to 2.14.2 db migration command succeed

root@cbe8e0b21de3:/# pip3.9 install connexion==2.14.2
Collecting connexion==2.14.2
  Downloading connexion-2.14.2-py2.py3-none-any.whl (95 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 95.1/95.1 kB 3.5 MB/s eta 0:00:00
Collecting clickclick<21,>=1.2 (from connexion==2.14.2)
  Downloading clickclick-20.10.2-py2.py3-none-any.whl (7.4 kB)
Requirement already satisfied: jsonschema<5,>=2.5.1 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (4.19.2)
Requirement already satisfied: PyYAML<7,>=5.1 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (6.0.1)
Requirement already satisfied: requests<3,>=2.9.1 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (2.31.0)
Requirement already satisfied: inflection<0.6,>=0.3.1 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (0.5.1)
Collecting werkzeug<2.3,>=1.0 (from connexion==2.14.2)
  Downloading Werkzeug-2.2.3-py3-none-any.whl (233 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 233.6/233.6 kB 8.1 MB/s eta 0:00:00
Requirement already satisfied: packaging>=20 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (23.2)
Requirement already satisfied: flask<2.3,>=1.0.4 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (2.2.5)
Requirement already satisfied: itsdangerous>=0.24 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (2.1.2)
Requirement already satisfied: click>=4.0 in /usr/local/lib/python3.9/dist-packages (from clickclick<21,>=1.2->connexion==2.14.2) (8.1.7)
Requirement already satisfied: Jinja2>=3.0 in /usr/local/lib/python3.9/dist-packages (from flask<2.3,>=1.0.4->connexion==2.14.2) (3.1.2)
Requirement already satisfied: importlib-metadata>=3.6.0 in /usr/local/lib/python3.9/dist-packages (from flask<2.3,>=1.0.4->connexion==2.14.2) (6.8.0)
Requirement already satisfied: attrs>=22.2.0 in /usr/local/lib/python3.9/dist-packages (from jsonschema<5,>=2.5.1->connexion==2.14.2) (23.1.0)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.9/dist-packages (from jsonschema<5,>=2.5.1->connexion==2.14.2) (2023.7.1)
Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.9/dist-packages (from jsonschema<5,>=2.5.1->connexion==2.14.2) (0.30.2)
Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.9/dist-packages (from jsonschema<5,>=2.5.1->connexion==2.14.2) (0.10.6)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.9/dist-packages (from requests<3,>=2.9.1->connexion==2.14.2) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests<3,>=2.9.1->connexion==2.14.2) (2.8)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/lib/python3/dist-packages (from requests<3,>=2.9.1->connexion==2.14.2) (1.25.8)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests<3,>=2.9.1->connexion==2.14.2) (2019.11.28)
Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.9/dist-packages (from werkzeug<2.3,>=1.0->connexion==2.14.2) (2.1.3)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.9/dist-packages (from importlib-metadata>=3.6.0->flask<2.3,>=1.0.4->connexion==2.14.2) (3.17.0)
Installing collected packages: werkzeug, clickclick, connexion
  Attempting uninstall: werkzeug
    Found existing installation: Werkzeug 3.0.1
    Uninstalling Werkzeug-3.0.1:
      Successfully uninstalled Werkzeug-3.0.1
  Attempting uninstall: connexion
    Found existing installation: connexion 3.0.0
    Uninstalling connexion-3.0.0:
      Successfully uninstalled connexion-3.0.0
Successfully installed clickclick-20.10.2 connexion-2.14.2 werkzeug-2.2.3
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
root@cbe8e0b21de3:/# airflow db init
/usr/local/lib/python3.9/dist-packages/airflow/cli/commands/db_command.py:43 DeprecationWarning: `db init` is deprecated.  Use `db migrate` instead to migrate the db and/or airflow connections create-default-connections to create the default connections
DB: sqlite:////root/airflow/airflow.db
[2023-11-02T13:56:12.691-0400] {migration.py:213} INFO - Context impl SQLiteImpl.
[2023-11-02T13:56:12.714-0400] {migration.py:216} INFO - Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running stamp_revision  -> 405de8318b3a
WARNI [airflow.models.crypto] empty cryptography key - values will not be stored encrypted.
Initialization done
root@cbe8e0b21de3:/#

What you think should happen instead

root@cbe8e0b21de3:/# pip3.9 install connexion==2.14.2
Collecting connexion==2.14.2
  Downloading connexion-2.14.2-py2.py3-none-any.whl (95 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 95.1/95.1 kB 3.5 MB/s eta 0:00:00
Collecting clickclick<21,>=1.2 (from connexion==2.14.2)
  Downloading clickclick-20.10.2-py2.py3-none-any.whl (7.4 kB)
Requirement already satisfied: jsonschema<5,>=2.5.1 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (4.19.2)
Requirement already satisfied: PyYAML<7,>=5.1 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (6.0.1)
Requirement already satisfied: requests<3,>=2.9.1 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (2.31.0)
Requirement already satisfied: inflection<0.6,>=0.3.1 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (0.5.1)
Collecting werkzeug<2.3,>=1.0 (from connexion==2.14.2)
  Downloading Werkzeug-2.2.3-py3-none-any.whl (233 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 233.6/233.6 kB 8.1 MB/s eta 0:00:00
Requirement already satisfied: packaging>=20 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (23.2)
Requirement already satisfied: flask<2.3,>=1.0.4 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (2.2.5)
Requirement already satisfied: itsdangerous>=0.24 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (2.1.2)
Requirement already satisfied: click>=4.0 in /usr/local/lib/python3.9/dist-packages (from clickclick<21,>=1.2->connexion==2.14.2) (8.1.7)
Requirement already satisfied: Jinja2>=3.0 in /usr/local/lib/python3.9/dist-packages (from flask<2.3,>=1.0.4->connexion==2.14.2) (3.1.2)
Requirement already satisfied: importlib-metadata>=3.6.0 in /usr/local/lib/python3.9/dist-packages (from flask<2.3,>=1.0.4->connexion==2.14.2) (6.8.0)
Requirement already satisfied: attrs>=22.2.0 in /usr/local/lib/python3.9/dist-packages (from jsonschema<5,>=2.5.1->connexion==2.14.2) (23.1.0)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.9/dist-packages (from jsonschema<5,>=2.5.1->connexion==2.14.2) (2023.7.1)
Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.9/dist-packages (from jsonschema<5,>=2.5.1->connexion==2.14.2) (0.30.2)
Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.9/dist-packages (from jsonschema<5,>=2.5.1->connexion==2.14.2) (0.10.6)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.9/dist-packages (from requests<3,>=2.9.1->connexion==2.14.2) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests<3,>=2.9.1->connexion==2.14.2) (2.8)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/lib/python3/dist-packages (from requests<3,>=2.9.1->connexion==2.14.2) (1.25.8)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests<3,>=2.9.1->connexion==2.14.2) (2019.11.28)
Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.9/dist-packages (from werkzeug<2.3,>=1.0->connexion==2.14.2) (2.1.3)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.9/dist-packages (from importlib-metadata>=3.6.0->flask<2.3,>=1.0.4->connexion==2.14.2) (3.17.0)
Installing collected packages: werkzeug, clickclick, connexion
  Attempting uninstall: werkzeug
    Found existing installation: Werkzeug 3.0.1
    Uninstalling Werkzeug-3.0.1:
      Successfully uninstalled Werkzeug-3.0.1
  Attempting uninstall: connexion
    Found existing installation: connexion 3.0.0
    Uninstalling connexion-3.0.0:
      Successfully uninstalled connexion-3.0.0
Successfully installed clickclick-20.10.2 connexion-2.14.2 werkzeug-2.2.3
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
root@cbe8e0b21de3:/# airflow db init
/usr/local/lib/python3.9/dist-packages/airflow/cli/commands/db_command.py:43 DeprecationWarning: `db init` is deprecated.  Use `db migrate` instead to migrate the db and/or airflow connections create-default-connections to create the default connections
DB: sqlite:////root/airflow/airflow.db
[2023-11-02T13:56:12.691-0400] {migration.py:213} INFO - Context impl SQLiteImpl.
[2023-11-02T13:56:12.714-0400] {migration.py:216} INFO - Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running stamp_revision  -> 405de8318b3a
WARNI [airflow.models.crypto] empty cryptography key - values will not be stored encrypted.
Initialization done
root@cbe8e0b21de3:/#

How to reproduce

docker run -it --rm --entrypoint /bin/bash ubuntu:focal
apt install software-properties-common
add-apt-repository ppa:deadsnakes/ppa
apt install python3.9 gcc python3.9-dev
apt install curl
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
apt install python3.9-distutils
python3.9 get-pip.py
pip install apache-airflow==2.7.2
airflow db init

Operating System

NAME="Ubuntu" VERSION="20.04.6 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.6 LTS" VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=focal UBUNTU_CODENAME=focal

Versions of Apache Airflow Providers

pip3.9 install apache-airflow==2.7.2

Deployment

Other

Deployment details

docker run -it --rm --entrypoint /bin/bash ubuntu:focal
apt update
apt install software-properties-common
add-apt-repository ppa:deadsnakes/ppa
apt update
apt install curl python3.9 gcc python3.9-dev python3.9-distutils
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python3.9 get-pip.py
pip install apache-airflow==2.7.2
airflow db init

Anything else

No response

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

The text was updated successfully, but these errors were encountered:

boring-cyborg · 2023-11-02T17:57:39Z

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

Taragolis · 2023-11-02T18:11:14Z

connexion already limited in:

Add connexion<3.0 upper bound #35218

Follow up task exists:

Connexion 3 breaks init-views #35234

For avoid further error on any new package updates which might potentially break airflow it is recommend way to use constraints file

potiuk · 2023-11-02T20:58:46Z

Also @eselyavka you should install airflow from the scratch (if you want reproducible install) following the only supported method - i.e. with constraints - https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html

Then you can upgrade any new dependencies as you see fit (and test) but the "base" installation is guaranteed to not cause problems when external dependencies are changing something in their behaviours

It is SPECIFICALLY designed to make sure that such breaking releases in "future" dependencies are not affecting user installations - for users who actually follow those instructions that is. You wil make yourself a favour to use the protections we implemented for our users.

Also you might want to watch my talk https://www.youtube.com/watch?v=zPjIQjjjyHI describing why and how it works if you are interested.

eselyavka · 2023-11-13T16:18:16Z

@potiuk thank you very much for detailed answer.

eselyavka added area:core kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet labels Nov 2, 2023

Taragolis closed this as completed Nov 2, 2023

potiuk mentioned this issue Nov 2, 2023

connexion 3.0.0 breaks standalone deployment #35390

Closed

2 tasks

RobbeSneyders mentioned this issue Nov 3, 2023

Getting error 'skip_error_handlers' in connexion==3.0.0 but not in connexion==2.14.2 spec-first/connexion#1783

Closed

Taragolis mentioned this issue Nov 4, 2023

Throwing issue on executing airflow db init code #35434

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

airflow db migrate crashing with new connexion == 3.0.0 #35375

airflow db migrate crashing with new connexion == 3.0.0 #35375

eselyavka commented Nov 2, 2023

boring-cyborg bot commented Nov 2, 2023

Taragolis commented Nov 2, 2023

potiuk commented Nov 2, 2023

eselyavka commented Nov 13, 2023

airflow db migrate crashing with new connexion == 3.0.0 #35375

airflow db migrate crashing with new connexion == 3.0.0 #35375

Comments

eselyavka commented Nov 2, 2023

Apache Airflow version

What happened

What you think should happen instead

How to reproduce

Operating System

Versions of Apache Airflow Providers

Deployment

Deployment details

Anything else

Are you willing to submit PR?

Code of Conduct

boring-cyborg bot commented Nov 2, 2023

Taragolis commented Nov 2, 2023

potiuk commented Nov 2, 2023

eselyavka commented Nov 13, 2023