Extend the same keyword arguments callable support in PythonOperator to some other sensors/operators

[yuqian90]

• This PR Standardises the callable signatures in PythonOperator, PythonSensor, ExternalTaskSensor, SimpleHttpOperator and HttpSensor.
• The callable facilities in PythonOperator have been refactored into airflow.utils.helper.make_kwargs_callable. And it's used in those other places to make them work the same way.

PythonOperator and PythonSensor already accept python_callable in this form, i.e. the callable can request any number of named arguments in context:

def python_callable(execution_date, ds_nodash):
   ...

However, ExternalTaskSensor and some other sensors/operator accepts only callables in this form:

def execution_date_fn(execution_date):
   ...

A more recent change #8702 added support for this form as well:

def execution_date_fn(execution_date, context):
   ...

Unfortunately, #8702 introduced an unpleasant side-effect. Because it enforces the number of arguments is either 1 or 2, it stops people from using common techniques such as loop variable capturing. E.g. this comment illustrate a problem: https://github.com/apache/airflow/pull/8702/files#r497897460

This PR addresses these problems by making execution_date_fn work the same way as python_callable. execution_date_fn is now much more flexible. It can accept any variable number of keyword arguments from context. The change is backward compatible. Existing usage continues to work.

E.g. the following callables are all legitimate execution_date_fn after this PR:

            def execution_date_fn(dt):
                ...

            def execution_date_fn(execution_date):
                ...

            def execution_date_fn(execution_date, context):
                ...

            def execution_date_fn(execution_date, ds_nodash):
                ...

            def execution_date_fn(execution_date, ds_nodash, dag):
                ...

[yuqian90]

@Fokko @ashb @BasPH @mik-laj hope you are interested since you worked on/reviewed #5990.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[mik-laj]

It is a backward compatible change?

[yuqian90]

It is a backward compatible change?

It's backward compatible with all existing usage except this one:

def execution_date_fn(dt, ctx)

I.e. if context from #8702 is needed, the argument must be named context and not ctx because it's now a keyword argument. This will continue to work:

def execution_date_fn(dt, context)

I'll add this to UPDATING.md.

[kaxil]

Can you please rebased your PR on latest Master since we have applied Black and PyUpgrade on Master.

It will help if your squash your commits into single commit first so that there are less conflicts.

[yuqian90]

Can you please rebased your PR on latest Master since we have applied Black and PyUpgrade on Master.

It will help if your squash your commits into single commit first so that there are less conflicts.

Thanks. I've rebased.

[github-actions]

The PR needs to run all tests because it modifies core of Airflow! Please rebase it to latest master or ask committer to re-run it!

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[ashb]

I'm tempted to merge this anyway -- most tests are passing.

[yuqian90]

Thanks @ashb and @kaxil !