test: add test that verifies job fails if attachment is accidentally deleted from bucket

[YutongLi291]

What was the problem/requirement? (What/Why)

If in the case that an user accidentally deletes a job attachment from the job attachments bucket (or clears the job attachments bucket completely), if a job is running that depends on any of the deleted job attachments, the job should fail at the syncInputJobAttachments stage since the job attachment is not able to be synced from the bucket to the worker.

We should test that this is indeed what happens, and that the worker is still able to process other jobs after this failure.

What was the solution? (How)

Add a test that

1. Submits a job with input job attachments in SUSPENDED status, this will ensure that the input job attachments are uploaded to the Job Attachments bucket but not synced to the worker yet, as the job has yet to start.
2. Using the manifest, find any of the hashed input job attachment files in the job attachments bucket, and delete it
3. Set the job status to READY, ensuring that the worker will start working on the job
4. Wait for the job to finish, in which case it should be in FAILED status since the syncInputJobAttachments step failed
5. Check that syncInputJobAttachments session action failed using ListSessionActions
6. Submit another test job that sleeps, ensuring it passes without fail to verify that the worker is still running as expected.

What is the impact of this change?

Better testing for worker agent code and handling failures

How was this change tested?

# Linux
source .e2e_linux_infra.sh
hatch run e2e-test

# Windows
source .e2e_windows_infra.sh
hatch run e2e-test

Was this change documented?

No

Is this a breaking change?

No

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[jericht]

Is this duplicated above in job_parameters variable?

[jericht]

HeadObject is a lighter API call if we just want to check for existence, but if the role doesn't have permission to call it already, don't worry about it.

[jericht]

Suggested change

	logging.info(f"Session actions: {session_actions}")
	LOG.info(f"Session actions: {session_actions}")

Should use same logger as above for consistency. I believe this method goes to root logger

[YutongLi291]

Will address in next PR