Update base sensor operator to support XCOM return value

[mingshi-wang]

Inspired by the idea of #20547, this PR is to update the base sensor operator such that an optional XCom value can be returned and pushed when the operator is executed. Users only need to override the poke() method to return a PokeReturnValue object that encapsulates the XCOM value.

related: #20323

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[github-actions]

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

[josh-fell]

CC @dstandish since it's in the same vein as the draft PR #20547. Maybe you've already given the thumbs-up somewhere else.

[mingshi-wang]

CC @dstandish since it's in the same vein as the draft PR #20547. Maybe you've already given the thumbs-up somewhere else.
Hi @dstandish - can you please take a look at this PR? After this is merged, I will update #20530 to be based on this PR. Thanks!

[kurtqq]

that would be nice

[potiuk]

Nice one. Also it has the potential of being detected by our "min airflow vrsion" CI check (It would fail on import).

Anyone else? @dstandish ?

[dstandish]

Nice one. Also it has the potential of being detected by our "min airflow vrsion" CI check (It would fail on import).

Anyone else? @dstandish ?

It's good with me. I recall @potiuk that you were trying to find a way to make it possible to return a tuple. But I think this change should be compatible with that. What we could consider doing is, for 3.0, make it so a sensor must be true / false or PRV (i.e., no longer support "truthy" values) and then in 3.1 we could add support for tuple. WDYT?

[potiuk]

It's good with me. I recall @potiuk that you were trying to find a way to make it possible to return a tuple. But I think this change should be compatible with that. What we could consider doing is, for 3.0, make it so a sensor must be true / false or PRV (i.e., no longer support "truthy" values) and then in 3.1 we could add support for tuple. WDYT?

I am good for it actually as is. I think we do not need to do anything more :) .
Comparing to our discussion I think the main difference is that the The PokeReturnValue with __bool__ will actually return true/false properly even for older versions of Airlfow (but it will not store xcom value).

And you should be able to start using it in our community providers even now. The way it should be done is something like:

try:
   from ... import PokeReturnValue
except ImportError:
   PokeReturnValue = None

...

    if PokeReturnValue is not None:
         return PokeReturnValue(is_done, xcom_value)
    else:
        if is_done:
             xcom_push(xcom_value) # this is not proper way needs to be done properly.
        return is_done

I think we will figure out the details with the first sensor which will use it (and it will be detected in CI by failing import for 2.1) and we can - then add some pre-commit checks and better information for the developers of providers (but this can be done later).

UPDATE: Actually we can make future providers also 100% compatible until they have > 2.3.0.

[potiuk]

Which makes me think.. Well maybe actually we should add a documentation about it. @mingshi-wang - could you please add docs explaininig that this feature (pushing Xcom) is only available in 2.3.0 and add the example of how you can add provider compatible with both pre 2.3.0 and 2.1.0 ?

[potiuk]

Actually I realized that you CAN make it working in community providers INCLUDING XCom Push - simply the provider sensors will have to push the xcom manually until we are > 2.3.0 (which will be ~ December or so.

[potiuk]

Some doc update is needed as discussed.

[mingshi-wang]

Some doc update is needed as discussed.

Sure - I will put in some doc.

[mingshi-wang]

@potiuk - I have updated the doc. Could you take a look again? This will unblock me from finishing #20323. Thanks!

[potiuk]

❤️ it