Create more detailed error responses when submitting batch jobs

[richscott]

NOTE: this PR is a continuation of the work/PR that our summer 2023 MLH fellow/intern Raaj Patel had created. Unfortunately, his internship ended and he has returned to university before that PR (#2623) was fully finished, approved, and merged. The only change to this code since he departed I have made is to fix a conflict in go.mod and go.sum.

Following is his description in the original PR:

What type of PR is this?

This PR adds functionality to JobSubmitResponse when an error has occured. Previously the only thing returned was the error. The changes made in this PR allow the code to do 2 things:

• Build a collection of errors, if mulitple jobs error for the same return, these changes will return them at the same time to the user. Rather than them having to reattempt submissions over and over again with 1 fix each time.
• Return more detailed errors to the user by utilizing the JobSubmitResponse Struct

Which issue(s) this PR fixes:

Fixed #1748

Original Description from issue 1748 in JIRA, opened by @d80tb7 :

When we reject a job (for example because it can't be scheduled on any node) the whole grpc call fails.
There is an error message that is returned, but users have said that this doesn't give enough infomation
about what was valid. Instead they ask:

is it possible for these errors to be returned on the Error field on the JobResponseItem that we're given?

[richscott]

Screenshots from Raaj Patel, showing the logging output before and after this change.

Submitted a bad sleep yaml job containing two jobs, both have invalid podSpecs as the requested and limit resources both contain "memory: 0"

[d80tb7]

This looks like a change in behaviour in that previously we used to return nil for the id when we rejected jobs but now we return a jobId. I think this is good in that we now have an id that the user can give us to search logs etc, but does have a couple of downsides:

• Users may get confused if they see jobIds that don't appear anywhere else in the system (e.g. lookout)
• It's a mildly breaking interface change- e.g. if clients previously had code that did if jobResponse.JobId == nil... this will now behave differently.

I think I'm mildly in favour of this change, but it might be good to get a second opinion- @severinson ?

[d80tb7]

Is there any limit to the size of the response we can generate here? E.g. if the user submits a batch of 100 jobs and they all fail with some massive error are we sure we're not going to generate a response that exceeds the max grpc response size? Likewise (although admittedly loess likely) could we end up generaling such a massive message that we would OOM.

If it were me I'd try and limit the size of the error message by saying:

• how may jobs failed validation (out of how may submitted)
• detailed error messages for the first n jobs that failed. (maybe n=5 or something)

[richscott]

@d80tb7 I'll replace that potentially-huge error with a concise/distilled one, using that logic as you suggest.

[richscott]

@d80tb7 An attempt to implement the logic described above is now in commit 894c265 - let me know what you think.