-
Notifications
You must be signed in to change notification settings - Fork 14.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SSH task exit code added to XCOM as 'ssh_exit' key #27370
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In order to approve the PR a check was added to Static Checks requiring the use of double-quotes.
Co-authored-by: Dov Benyomin Sohacheski <[email protected]>
Co-authored-by: Dov Benyomin Sohacheski <[email protected]>
@bdsoha thanks, applied |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add unit test to cover this change
def raise_for_status(self, exit_status: int, stderr: bytes) -> None: | ||
def raise_for_status(self, exit_status: int, stderr: bytes, **kwargs) -> None: | ||
ti=kwargs["context"].get("task_instance") | ||
ti.xcom_push(key="ssh_exit", value=exit_status) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we should verify user wish to push to xcom.
users ask for this specifically with do_xcom_push
parameter
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@eladkal It is not optional on other operators:
airflow/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py
Lines 444 to 446 in b29ca4e
ti = context["ti"] | |
ti.xcom_push(key="pod_name", value=self.pod.metadata.name) | |
ti.xcom_push(key="pod_namespace", value=self.pod.metadata.namespace) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep. It should have unit tests added.
in progress. ut will be added. |
@@ -134,16 +134,18 @@ def exec_ssh_client_command(self, ssh_client: SSHClient, command: str): | |||
ssh_client, command, timeout=self.cmd_timeout, environment=self.environment, get_pty=self.get_pty | |||
) | |||
|
|||
def raise_for_status(self, exit_status: int, stderr: bytes) -> None: | |||
def raise_for_status(self, exit_status: int, stderr: bytes, **kwargs) -> None: | |||
ti=kwargs["context"].get("task_instance") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ti=kwargs["context"].get("task_instance") | |
ti = kwargs["context"].get("task_instance") |
Hi,
|
Also some notes about the test which I added. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks really cool now :). Very much needed. Let's see if our CI agrees.
@potiuk Static checks error in the #59009 run relates to |
Yep. Rebasing is alwasy a good idea. |
Awesome work, congrats on your first merged pull request! |
Based on #23788
Motivation. Provide the ability to analyze the SSH tasks exit code in order to build a dag logic based on these codes. Now, only
echo
output goes to XCOM's return_value key. And in case of failure nothing being passed to return_value. So in a dag we don't know what specific code is returned.What's done:
In SSHOperator, add XCOM key ssh_exit.
Task succeeded - ssh_exit set to 0.
echo
stuff still passed to return_value.Task failed - ssh_exit stores the code which was retrived from the ssh session.
P.S. I could imagine the alternative possible scenario - not to introduce a new XCOM key and just replace the return_value content, i.e. skip all other text output and just pass the ssh exit code there.
But my first approach does not reduce the functionality and the second one does. So I prefered the first.