Retrieve session logs when using Livy Operator

[collinmcnulty]

Description
The Airflow logs generated by the Livy operator currently only state the status of the submitted batch. To view the logs from the job itself, one must go separately to the session logs. I think that Airflow should have the option (possibly on by default) that retrieves the session logs after the batch reaches a terminal state if a polling_interval has been set.

Use case / motivation
When debugging a task submitted via Livy, the session logs are the first place to check. For most other tasks, including SparkSubmitOperator, viewing the first-check logs can be done in the Airflow UI, but for Livy you must go externally or write a separate task to retrieve them.

Are you willing to submit a PR?

I don't yet have a good sense of how challenging this will be to set up and test. I can try but if anyone else wants to go for it, don't let my attempt stop you.

Related Issues

None I could find

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[sreenath-kamath]

I can submit a PR for this, Will be a good addition to the LivyOpeartor

[potiuk]

Assigned you @sreenath-kamath !

[sreenath-kamath]

Have raised a PR #17393 for the same.