Release v0.5.0 requirements

[danielvegamyhre]

Targeting release around April 10th, 2024.

Must haves

• Introduce managedBy field #440
• JobSet TTL to clean up completed workloads #279
• Add production readiness status to readme #427
• Emit event when Job creation fails #447
• Emit event containing name of first failed Job which ultimately triggered JobSet failure #466

Nice to haves

• Feature: Configurable Failure Policy API #262
• Investigate ways to validate Job templates at JobSet validation time #422
• Comprehensive example task running training workload on GPUs using JobSet #429
• Comprehensive example task for running multislice TPU workloads with JobSet (and JobSet + Kueue) #428
• JobSet website + docs #401

[danielvegamyhre]

I think we're good to proceed with the release, I'll do it tomorrow.

[ahg-g]

I think we're good to proceed with the release, I'll do it tomorrow.

Can we include the status update PR?

[danielvegamyhre]

I think we're good to proceed with the release, I'll do it tomorrow.

Can we include the status update PR?

I'll try, it's almost done but after rebasing to include the TTL changes, there seems to be some bug / race condition causing the TTL feature to not work. The TTL integration test passes when I step through in debug mode, but not when run at full speed (it fails here, expected job deletion timestamps are not set).

These kind of issues sometimes take a while to determine the root cause (and I have other deadlines coming up this month I need to mindful of) and I don't want to delay the v0.5.0 too much if we can avoid it, since we are trying to coordinate the release with Kueue's next release on April 15th, so that our changes in #487 can be utilized in their Multi-Kueue work.

I'll see if I can figure it out tomorrow.

[dejanzele]

@danielvegamyhre can you describe the issue with TTL? I can work on fixing it.

I never managed to get a failure in the integration test neither locally nor on CI, I do have some experience with deflaking tests, I will try to run to replicate a failure.

[danielvegamyhre]

@dejanzele that would be super helpful! You can checkout the feature branch in #494 locally and run the integration tests. This line which checks that the expected number of jobs have been deleted fails. Adding print statements, it shows the jobs being listed do not have the deletion timestamp set when the test is run at full speed.

When I add breakpoints in the JobSet controller and step through the logic, the test passes.

To be clear, I don't think your PR implementing TTL support had a bug. The issue seems to be the interaction of my new changes (updating JobSet status only once per reconcile call, at the very end) with the existing TTL logic, causing jobs to not be deleted.

[dejanzele]

@danielvegamyhre this should fix it https://github.com/kubernetes-sigs/jobset/pull/494/files#r1560773603, tests pass locally on my side

[danielvegamyhre]

v0.5.0 was released in #515