fix(job_attachments): distinguish large/small file uploads

[gahyusuh]

What was the problem/requirement? (What/Why)

Uploading multiple large files at the same time overall hurts the job bundles experience. Consider the difference:

• Parallel large files:
  • Job bundles starts 4 100MB file uploads in parallel. It works for a while, so each upload gets to 53%. There's a failure which stops everything.
  • Retry - it restarts everything from scratch, even though it uploaded 200MB of data last time
• Serial large files:
  • Job bundles starts 1 100MB file upload. It finishes it, starts a second 100MB upload. Starts the third, then fails (same 53% mark overall).
  • Retry - it restarts from the third upload.

What was the solution? (How)

We do serial large file uploads.

• Split the input files to upload into two separate queues based on their sizes, one for smaller files and another for larger ones.
  • Size threshold to draw a line on large vs. small file: 160 MB (= 20 * 8 MB. Picked multiple of the part size, 8 MB, which is a default value configured for multi-part uploads.)
• Process each queue differently.
  • First, process the whole "small file" queue with parallel object uploads.
  • Then, once the small files are done, process the whole "large file" queue with serial object uploads (but still parallel multi-part upload).
• For example: A job bundle has 15 1-MB files, and 5 1-GB files to upload. Those files are split into 'small file' queue (which includes 15 1-MB files,) and 'large file' queue (which includes 5 1-GB files.) First, 15 small files get uploaded in parallel object uploading, and then those 5 large files are uploaded in serial.

What is the impact of this change?

Better UX. Especially, when a job submission with multiple large files is canceled during upload, retrying on submitting (the same job) later can provide much faster submission speeds.

How was this change tested?

• Unit tests: ran hatch run lint && hatch run test and ensured all tests passed.
• Test script: ran scripted_tests/upload_cancel_test.py to verify that the files in 'large file' queue got uploaded serially.
• End-to-end test: made sure the job submission is working as normal.

Was this change documented?

No.

Is this a breaking change?

No.

[marofke]

Good stuff! We've been wanting to do this improvement for a while, so nice to see it go in. Just a few nit comments