Broken installation with --editable flag

[dimonchik-suvorov]

Apache Airflow version

2.5.3

What happened

Pip installation of Airflow in editable mode (--editable flag) installs separate apache-airflow-providers-common-sql module any way and it breaks further Airflow commands like airflow db check.

After discussion with @potiuk we figured out that this is something related to pip installation method for editable packages introduced in newer versions of pip.

What you think should happen instead

Airflow provider apache-airflow-providers-common-sql should not be installed separately since it should take everything from sources. airflow db check command should end successfully without errors.

How to reproduce

export INSTALL_PROVIDERS_FROM_SOURCES=true && pip --no-cache-dir install -e '.[common.sql,amazon,cncf.kubernetes,mysql,password,redis,datadog,slack,ssh,sentry,statsd,pagerduty,databricks]'

# pip list | grep airflow
apache-airflow                      2.5.3       /opt/airflow_install
apache-airflow-providers-common-sql 1.4.0

airflow db check
Error:

WARNING:root:Failed to log action module 'airflow.settings' has no attribute 'Session'
Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/opt/airflow_install/airflow/__main__.py", line 48, in main
    args.func(args)
  File "/opt/airflow_install/airflow/cli/cli_parser.py", line 52, in command
    return func(*args, **kwargs)
  File "/opt/airflow_install/airflow/utils/cli.py", line 108, in wrapper
    return f(*args, **kwargs)
  File "/opt/airflow_install/airflow/cli/commands/db_command.py", line 192, in check
    db.check()
  File "/opt/airflow_install/airflow/utils/session.py", line 74, in wrapper
    with create_session() as session:
  File "/usr/local/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/opt/airflow_install/airflow/utils/session.py", line 31, in create_session
    if not settings.Session:
AttributeError: module 'airflow.settings' has no attribute 'Session'. Did you mean: 'SASession'?

Operating System

Debian GNU/Linux 11 (bullseye)

Versions of Apache Airflow Providers

All providers taken from https://github.com/apache/airflow/tree/v2-5-stable

Deployment

Other Docker-based deployment

Deployment details

Installation was made in docker container based on python:3.10-slim-bullseye.
Docker version: 20.10.24
Inside container:
Python 3.10.10
pip 22.3.1
Airflow source code took from https://github.com/apache/airflow/tree/v2-5-stable

Anything else

Workarounds:

Uninstall provider

After installation with export INSTALL_PROVIDERS_FROM_SOURCES=true && pip --no-cache-dir install -e '.[common.sql,amazon,cncf.kubernetes,mysql,password,redis,datadog,slack,ssh,sentry,statsd,pagerduty,databricks]' is done do pip uninstall apache-airflow-providers-common-sql && rm -rf /usr/local/lib/python3.10/site-packages/airflow
This will remove unwanted package and Airflow will take classes only from sources.

Downgrade pip version

pip install -U pip==21.2.4 && pip install -U setuptools==57.4.0 - old versions of pip correctly install Airflow from sources. Questionable fix as for me because I can't predict how downgrading pip may or may not affect other installations... Suites for Python gurus who know what they're doing.

Add PYTHONPATH env variable

Let's say your Airflow source code lays under /opt/airflow_install dir. You can add this path to your PYTHONPATH env variable (or initialize this variable in case you don't have one).
export PYTHONPATH=/opt/airflow_install. This will tell your Python to look for sources in /opt/airflow_install and this will fix the error.

Related Airflow Slack threads:

https://apache-airflow.slack.com/archives/CCQ7EGB1P/p1681824350577539
https://apache-airflow.slack.com/archives/CCQ7EGB1P/p1681887721357829

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

Thanks! I will take a look!

[potiuk]

A huge - but one that fixes it properly fix for that one is coming in #36537 - moving airflow into the modern packaging tools world !