Submissions randomly go missing

[j-mao]

For some reason, source code for submissions 316, 318, 325, 328 don't exist, but they are still in the database and cause trouble for the servers. Should investigate why these submissions disappeared.

[n8kim1]

is this not the race condition? (i'll keep digging)

[n8kim1]

thinking abt it, it might be good to separate the bucket part and pub/sub part of compilation code into two try/catch blocks

[j-mao]

I think our race condition had been "fixed" infrastructure-side when this happened. Even so I think the backend should be agnostic to the race condition and it affects the compile server more?

I hypothesise that this could be something to do with submitting twice in a row; potentially this is another race condition inside the backend/db. e.g. 484 is missing and submitted by the same team as 482-483.

[j-mao]

This is causing many problems with the compilation queue and needs to be fixed as soon as possible.

[j-mao]

Cause appears to be HTTP 500 on /api/0/submission; the row is added to the database, but no signed upload url is given to the browser and so the code never reaches the bucket.
Further testing indicates that the job also never reaches the compile server

[j-mao]

A sketchy but effective fix:

• While making the first submit, disable the submit button to prevent a second submit from starting before the first one finishes.
• If it 500s, send the request again. Can maybe even ask the backend to delete/reuse the previous request but this is probably less important.
• If it 201s, alert success, or do something that makes it physically impossible to submit again immediately. Also re-enable the submit button. This would also resolve Disable the submit button right after it's clicked #181

[n8kim1]

justify / display reason of "can't submit while prev submission processing" (as opposed to compiling, etc)

[n8kim1]

@zoemarschner @arvid220u let's make the sketchy fix for now; a more robust solution is better in long term and probably isn't difficult to carry out

[j-mao]

Further testing indicates that the job also never reaches the compile server

Looking at server logs from last night, sometimes it actually still does reach the pub/sub and compile server, so there may be more than one issue.

We will likely need a more robust solution to deal with this one, because the sketchy patch doesn't save the compile server from being loaded with non-existent submissions.

[j-mao]

Compile server is configured to retry things that are bad and ignore things that are really bad. Not a big problem right now anymore

[n8kim1]

@j-mao I saw some more submissions that go missing (unfortunately forget the IDs now); these were also a bunch in a row by the same team, w/ only the last one making it

this may be an issue worth looking into more

[n8kim1]

eg submissions 6250-6258 don't have data, w only 6259 having a bucket; these were all submissions by the same team, all within a couple seconds of each other

Is this behavior intended? and even if not, is it acceptable?