Job lifecycle notifications

[AdrianoKF]

The watcher Go application uses the Kubernetes client API to monitor jobs in the current namespace and send notifications for major lifecycle events.

See the README for additional details.

Original PR description follows:

---

Jobs need to be annotated in order to be eligible for notifications:

Use x-jobby.io/notify-channel: [slack|webhook] to determine the notification channel (HTTP webhook or Slack), and then control the destinations with x-jobby.io/slack-channel-ids (comma-separated) or x-jobby.io/webhook-urls (comma-separated) annotations.

If using the Slack notification channel, pass the Slack API token through the WATCHER_SLACK_API_TOKEN env variable.

Notification Channels

The code uses the notify package, which supports are variety of destinations.
So far, the following have been implemented:

• webhook: HTTP (POST) webhooks to a URL
• slack: Slack message to one or multiple channels, requires a Slack API token

Notification Types

The following notifications have been implemented (so far, only Kubernetes/Kueue Jobs are supported):

• Job goes from being suspended to running: implicated that the workload has been admitted by Kueue
  
• Job is completed: logs for all pods associated with the job are included in the message
  
• Job has any failed pods: logs the pods and their failed containers with additional information
  
• Workload preemption monitoring

Usage Example

The following job options set up Slack notifications for lifecycle events in a job:

@job(
    options=JobOptions(
        ...,
        labels={
           "x-jobby.io/notify-channel": "slack",
           "x-jobby.io/slack-channel-ids": "mlops-test",
        ),
    )
)
def my_job(): ...

Deployment

See watcher/README.md for details.

[codecov]

Codecov Report

Attention: Patch coverage is 2.07612% with 283 lines in your changes missing coverage. Please review.

Project coverage is 51.64%. Comparing base (4b383b3) to head (957c71f).

✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
watcher/pkg/util/kueue.go	0.00%	90 Missing ⚠️
watcher/pkg/compose/compose.go	0.00%	71 Missing ⚠️
watcher/main.go	0.00%	49 Missing ⚠️
watcher/pkg/util/k8s.go	0.00%	34 Missing ⚠️
watcher/pkg/notify/notify.go	0.00%	24 Missing ⚠️
watcher/pkg/util/util.go	37.50%	10 Missing ⚠️
watcher/pkg/util/slices.go	0.00%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #30      +/-   ##
==========================================
- Coverage   56.42%   51.64%   -4.78%     
==========================================
  Files          61       68       +7     
  Lines        2997     3286     +289     
==========================================
+ Hits         1691     1697       +6     
- Misses       1306     1589     +283     

Flag	Coverage Δ
backend	88.27% <ø> (ø)
client	47.92% <ø> (ø)
watcher	2.07% <2.07%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.