post_live_metrics: Add _post_in_chunks.

[daavoo]

• Post metrics and parameters separately
• Try to post as many plots as possible and stop before reaching 30MB.
• Discard plots higher than 10MB.

Optimistically? expect metrics and parameters to not exceed 30MB.

[dberenbaum]

Any concerns about this spamming studio too much? Thinking of https://iterativeai.slack.com/archives/C01116VLT7D/p1692452930044229.

[daavoo]

Any concerns about this spamming studio too much? Thinking of https://iterativeai.slack.com/archives/C01116VLT7D/p1692452930044229.

As I understood it, the problem in the link was not about too many requests from the client to the server.
The problem was (still is) in the backend querying the database in a very inefficient way, so a single request in a project with a lot of steps per linear plot caused / can trigger this (https://github.com/iterative/studio/issues/7256).

We still need to fix that but the change here mainly affects images, which are handled differently from linear plots (the source of the linked problem).

The only change is when there the size of the images exceeds the server limit per single request:

Without this P.R, we just fail to send data on each step and try to send it in the next one. Failing forever and the request keeps getting bigger.
With this P.R:, we are able to send the data at each step bypassing the server limit.

[codecov-commenter]

Codecov Report

Patch coverage is 100.00% of modified lines.

Files Changed	Coverage
tests/test_post_live_metrics.py	100.00%

📢 Thoughts on this report? Let us know!.

[daavoo]

@amritghimire / @shcheklein any concerns with merging this or do you think it needs to wait on a solution for https://github.com/iterative/studio/pull/7310 ?

As I commented, this should mainly affect images not linear plots

[shcheklein]

I think it doesn't have to wait. The only thing that comes to my mind to check (besides the basic review, @amritghimire can you do that please?)- align the limits with Studio big_files check. See here for example - https://github.com/iterative/studio/pull/7299/files#diff-a67d283c73e8021405df0b1de3fb8dcf570563c21c70de5324cc2d8b661e561fR36

We skip 10Mb files on Studio (so they will disappear as we then parse an experiment).

Also, I now put a limit of 200 files per directory (not ideal, see the discussion and chime in the PR)- but as a safe guard. Here (or on the Studio side? or both?) we probably also want to limit the number of chunks, or number of files, etc.

[daavoo]

I think it doesn't have to wait. The only thing that comes to my mind to check (besides the basic review, @amritghimire can you do that please?)- align the limits with Studio big_files check. See here for example - https://github.com/iterative/studio/pull/7299/files#diff-a67d283c73e8021405df0b1de3fb8dcf570563c21c70de5324cc2d8b661e561fR36

We skip 10Mb files on Studio (so they will disappear as we then parse an experiment).

Also, I now put a limit of 200 files per directory (not ideal, see the discussion and chime in the PR)- but as a safe guard. Here (or on the Studio side? or both?) we probably also want to limit the number of chunks, or number of files, etc.

Added both checks

Outdated