livy operator print sparkUiUrl in log output

[cocampbe]

Description

Adding the sparkUiUrl to the log output of the livy operator.

Use case / motivation

It would be useful for our end users to be able to go straight to the spark page from the log output that by tracking down the livy sessions. It would make troubleshooting spark jobs submitted by livy quicker. Our livy setup only stores sessions for about 3 hours. Our end users get frustrated trying to track down the spark job.

I think that checking when the state is in the starting phase, the url could be pulled and printed.

Are you willing to submit a PR?

Yes.

Related Issues

[potiuk]

That's a perfect case for Extra Operator link: https://airflow.apache.org/docs/apache-airflow/stable/howto/define_extra_link.html . Shall I assign you to that issue?

[cocampbe]

@potiuk I am not sure if that extra operator will work for this case. The livy api only stores data for about 3 hours in our case. The default is much shorter. Since the uri has to be pulled form the API and is not accessible long term, I am not sure the link would be valid after or even available later. I hope that makes sense.

[potiuk]

@potiuk I am not sure if that extra operator will work for this case. The livy api only stores data for about 3 hours in our case. The default is much shorter. Since the uri has to be pulled form the API and is not accessible long term, I am not sure the link would be valid after or even available later. I hope that makes sense.

You could store the link in the XCom right after submission, and retrieve it from the XCom to show it in the extra link.

[potiuk]

@potiuk I am not sure if that extra operator will work for this case. The livy api only stores data for about 3 hours in our case. The default is much shorter. Since the uri has to be pulled form the API and is not accessible long term, I am not sure the link would be valid after or even available later. I hope that makes sense.

But yeah. If the link is not functional after 3 hours, then probably you are right that exposing it via extra link is not a good idea. I Thought that you will still be able to access it after 3hrs (but not retrieve it), but if I read it correctly, this is not the case.

[cocampbe]

@potiuk This is what we needed. Fetching and logging livy session logs for LivyOperrator #17393. That is what we needed.