Change Deferrable implementation for RedshiftResumeClusterOperator to follow standard #30864
Conversation
boring-cyborg
bot
added
area:providers
provider:amazon-aws
| self.redshift_hook = RedshiftHook(aws_conn_id=self.aws_conn_id) | ||
| async with self.redshift_hook.async_conn as client: | ||
| waiter = self.redshift_hook.get_waiter("cluster_resumed", deferrable=True, client=client) | ||
| await waiter.wait( |
There was a problem hiding this comment.
Should we also think about the case where we want to log some events in trigger periodically while waiting ? similar to discussion in #30853 (comment) ?
There was a problem hiding this comment.
Similarly here: #30244 - it almost seems that even if the waiter implementation in botocore does not provide any way to get callback while waiitng, we could have a wrapper around the waiter that could be used across many of the Deferrable Amazon operators (this would become super useful in 2.6.0 with the triggerer logs interleaved with task logs).
Add async custom waiter support in get_waiter, and base_waiter.py Add Deferrable mode to RedshiftCreateClusterOperator Add RedshiftCreateClusterTrigger and unit test Add README.md for writing Triggers for AMPP
Fix/add a few unit tests
Remove duplicate log statement Remove duplicate docstring
syedahsn
force-pushed
the
syedahsn/deferrable-redshift-resume-cluster
branch
from
85bf375
to
15c0a78
Compare
|
LGTM: @o-nikolas ? |
Yup, this is correct, and I believe Pause is good to go. There was some feedback, but I think Syed has addressed it (while he is on vacation at that!). P.S. I am on vacation starting today as well, responses will be delayed or non-existent 😁 |
…edshift-resume-cluster
Now that #30032 is merged, we can update the deferrable implementation for
RedshiftResumeClusterOperatorto use custom aiobotocore waiters , as well as theasync_connto simplify deferrable implmentation, and reduce code duplication. The method used here is described in the README.md included in the initial PR.@pankajastro @phanikumv
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.