Logs for EmrStepSensor

[virendhar-aws]

Description

Add feature to EmrStepSensor to bring back the spark task url & logs after task execution

Use case/motivation

After starting an EMR step task using EmrAddStepsOperator we generally have an EmrStepSensor to track the status of the step. The job ID is available for the sensor and is being poked at regular interval.

[2022-04-26, 22:07:43 UTC] {base_aws.py:100} INFO - Retrieving region_name from Connection.extra_config['region_name']
[2022-04-26, 22:07:44 UTC] {emr.py:316} INFO - Poking step s-123ABC123ABC on cluster j-123ABC123ABC
[2022-04-26, 22:07:44 UTC] {emr.py:74} INFO - Job flow currently PENDING
[2022-04-26, 22:08:44 UTC] {emr.py:316} INFO - Poking step s-123ABC123ABC on cluster j-123ABC123ABC
[2022-04-26, 22:08:44 UTC] {emr.py:74} INFO - Job flow currently PENDING
[2022-04-26, 22:09:44 UTC] {emr.py:316} INFO - Poking step s-123ABC123ABC on cluster j-123ABC123ABC
[2022-04-26, 22:09:44 UTC] {emr.py:74} INFO - Job flow currently COMPLETED
[2022-04-26, 22:09:44 UTC] {base.py:251} INFO - Success criteria met. Exiting.
[2022-04-26, 22:09:44 UTC] {taskinstance.py:1288} INFO - Marking task as SUCCESS. dag_id=datapipeline_sample, task_id=calculate_pi_watch_step, execution_date=20220426T220739, start_date=20220426T220743, end_date=20220426T220944

After the task is completed the status is displayed. If the user wants to review the logs of the task, it is a multistep process to get hold of the job logs from EMR cluster.

It will be a great addition to add the log url and possibly relay the logs to Airflow EmrStepSensor post completion of the task. This will be very handy when there are failures of many tasks and will make it a great user experience.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

It's more of a feature for ExtraLink https://airflow.apache.org/docs/apache-airflow/stable/howto/define_extra_link.html rather than putting the link to the log (or maybe both). Anyhow marked ait as good first issue for someone who would like to pick it up.

[phanikumv]

Hi @potiuk is someone working on this or can I pick this up for development ?
cc @kaxil

[kaxil]

Assigned it to you @phanikumv

[eladkal]

@phanikumv are you still working on this issue?

[o-nikolas]

Unassigning Phani so that someone else can pick this up.

[shubham22]

@syedahsn is going to pick this up. Can someone with access assign this to him?

[eladkal]

We can assign only to users who commented on the issue (github limitation)

[shubham22]

Oh, TIL! @syedahsn you know what to do : )

[syedahsn]

Can I be assigned this issue? Thanks @eladkal :)