Bug Report: Successful codecovcli upload-process failing to process by codecov

[chriscerk]

Describe the bug
A clear and concise description of what the bug is.

Environment

• Browser: n/a
• System: codecov cli from CI (uploading coverage reports)

To Reproduce
Steps to reproduce the behavior:

1. Run codecovcli --verbose upload-process

Your upload is now processing.
When finished, results will be available at: OUR_ORG_LINK_TO_RESULTS
Process Upload complete

3. Wait awhile...
4. See "Currently no uploads" in the UI and "Missing Head Report"

Expected behavior
A clear and concise description of what you expected to happen.

Uploading code coverage to work.

⭐ If you have metrics on:

• queued
• total processed
  • successfully processed
  • failed to process

That would be interesting to review since we see "Currently no uploads" which according to the codecov cli should not be true.

Showing the status in the UI would be great, eg if it is queued vs. processing vs. completed

Screenshots

n/a

Additional context

This has been working fine for awhile, but since the past week this has been extremely flaky and blocking many engineering teams.

[drazisil-codecov]

Hi @chriscerk

Can you please share the full SHA?

[chriscerk]

@drazisil-codecov 0731ef732d1df7971ba12832cb27598c80434c0e is one example

[drazisil-codecov]

@chriscerk can you open a ticket and share the full CLI output from your CI? Preferably with verbose enabled? Something is not happening correctly (don't think it's anything you are doing wrong), I but I don't know what...yet.

You can mention this thread, if you want. I'll get it :)

[davidlj95]

Same here, I can provide some examples as they're public repositories

I use Codecov GitHub Action to upload the reports there. And then have some PR checks that are mandatory to merge to the main branch. Recently (last week) I had to skip those checks given they were stuck in pending state. Investigating a bit, seems despite the action logs to upload the coverage report succeeded, no uploads appeared for that commit in Codecov's app. Hence Codecov didn't notify the results to GitHub and the check was left as pending.

An example from today:

• GitHub Action code coverage upload logs
• Codecov app reports no uploads for that commit

An example from another repository, 2 days ago:

• GitHub Action code coverage upload logs
• Codecov app reports no uploads for that commit

In both cases I forced a CI/CD run again by ammending the commit to change the date of the commit + force-pushing and it worked.

[drazisil-codecov]

@adrian-codecov Here are a few examples of uploads failing because create-commit didn't complete in time. Happy to huddle, I don't have any ideas for solutions at this time. Should I open an investigation ticket, or just use this one?

[drazisil-codecov]

Thanks, @davidlj95 ! Very helpful to be able to confirm that the CLI is calling all the steps in order. Definitely a server-side race condition. :-/

[drazisil-codecov]

#485 is related

[davidlj95]

Today happened again. Here are the details:

An example from today:

• GitHub Action code coverage upload logs
• Codecov app reports no uploads for that commit

Let me know if I'm spamming and I won't add moar comments 😛 Just providing info to confirm the issue stil exists + provide details to diagnose the issue