Remove pid arg from celery option to fix duplicate pid issue, Move celery command to provider package

[pavansharma36]

closes: #36541

---

[potiuk]

Tests failing

[pavansharma36]

@potiuk can you tell me why its failing to import module from airflow, core modules should be available to providers right

from airflow.cli.commands.daemon_utils import run_command_with_daemon_option
ModuleNotFoundError: No module named 'airflow.cli.commands.daemon_utils'

[potiuk]

@potiuk can you tell me why its failing to import module from airflow, core modules should be available to providers right

from airflow.cli.commands.daemon_utils import run_command_with_daemon_option ModuleNotFoundError: No module named 'airflow.cli.commands.daemon_utils'

It's failing in compatibility check. When you install the provider in Airflow 2.6 or 2.7, the module is not there because it has been added later. That's whe we have those compatibility checks - they install the provider in Airflow 2.6 and 2.7 and attempt to import all classes (and fail).

You need make sure to handle back-compatibility here. Likely you should in-line this method with a comment that it should be removed when min_airflow_version is 2.8.0 we have few other cases like that in various providers.

[pavansharma36]

@potiuk I tried copying daemon func but it goes way down where so many utils are not present.

Its better if someone with better codebase understanding do this. Let me know if I should create new PR with fix in core cli

[potiuk]

@potiuk I tried copying daemon func but it goes way down where so many utils are not present.

Its better if someone with better codebase understanding do this. Let me know if I should create new PR with fix in core cli

Actually I realized there is a simple way you can get back to what it was before. This command in Provider will be only useful since Airflow 2.9 so you can do something similar, when importing the daemon_utils to what is done in auth manager:

try:
    from airflow.auth.managers.base_auth_manager import BaseAuthManager, ResourceMethod
except ImportError:
    raise AirflowOptionalProviderFeatureException(
        "Failed to import BaseUser. This feature is only available in Airflow versions >= 2.8.0"
    )

[potiuk]

So simply go back to previous import and add the try/except/raise exception and all should be good.

[potiuk]

LGTM. @o-nikolas ?

[potiuk]

WDYT @o-nikolas ?

[o-nikolas]

WDYT @o-nikolas ?

So we went with deprecation messages instead of the approach of automating the action of keeping the two modules in sync via pre-commit?

I'm still worried there will be drift but I suppose adding the precommit stuff is a bit tricky so I'd be okay with this being merged. But I do agree with @potiuk's suggestion above.