Best practices for per-platform + cross-platform coverage ?

[HadrienG2]

So, I have a project with a lot of cross-platform code and a bit of platform-specific code. I'm testing coverage on both Linux and Windows, in two separate CI jobs, and I would like to have both a report of lines covered on each platform, and a report of all lines covered by at least one platform. What would be the recommended way to do this?

Right now, I'm doing one flag per platform, and it does eventually produce the result I'm interested in. But the GitHub action UX during a CI run is more than a little confusing because there is first a report of loss of coverage as the run for the first platform completes, then that report is later corrected as the run for the second platform completes. Also I think the codecov action status reports stays red even after this second run has completed, but that might be for unrelated reasons.

[jerrodcodecov]

But the GitHub action UX during a CI run is more than a little confusing because there is first a report of loss of coverage as the run for the first platform completes, then that report is later corrected as the run for the second platform completes.

Check out after_n_builds and set the field to "2" so that it always waits for both flags:
https://docs.codecov.com/docs/pull-request-comments#after_n_builds-for-flags

Also I think the codecov action status reports stays red even after this second run has completed, but that might be for unrelated reasons.

Can't comment on this one without seeing the repo, is it OSS?

[HadrienG2]

Thanks for the advice!

Check out after_n_builds and set the field to "2" so that it always waits for both flags:
https://docs.codecov.com/docs/pull-request-comments#after_n_builds-for-flags

Just to be clear about the semantics, if I want to wait for both flags to add a comment to the PR, then do I understand correctly that I should use the "global" variant...

codecov:
  notify:
    after_n_builds: 2

...and not the "per-flag" variant?

flag_management:
  individual_flags:
    - name: linux-latest
      after_n_builds: 1
    - name: windows-latest
      after_n_builds: 1

Also I think the codecov action status reports stays red even after this second run has completed, but that might be for unrelated reasons.

Can't comment on this one without seeing the repo, is it OSS?

Yes it is. Here is an example of a PR which has no code change on my side (just a dependency update), but still had a red codecov notification. HadrienG2/hwlocality#156 .

The relevant CI check is https://github.com/HadrienG2/hwlocality/runs/31159128413 .

However, looking at it further, I'm increasingly convinced that this is caused by something other than flags, maybe just code coverage measurement being a little flaky and varying by a line of two from one build to another for no good reason. I should really set up some thresholds for red notifications to avoid this at some point...