orchestrator validate step getting different code coverage from the same ci pool

[ypan0]

Hello everyone,

I'm running into an issue where on pipeline, same code, same type of scratch org in the same pool, keeps getting different code coverage, and diff a lot, also usually if I run validate locally, I can get 80%~85% in the same pool.
This is a unlocked package
@dxatscale/sfpowerscripts 10.2.6 (10.2.6)
sfdmu 4.12.7
sfpowerkit 4.0.3

node v12.22.3

any solution for this one?
Thank you

[ypan0]

looks like it's 79% on the third run

[dieffrei]

We have the same issue, even having specific unit tests for each class in a package, the code coverage is not calculated.
I also checked the code coverage result from salesforce, and it seems salesforce doesn't take into consideration or skip some test

[azlam-abdulsalam]

We are looking into this, the possible hypothesis is compilation is not finished due to delay at salesforce end, so tests is skipping those classes.

[dieffrei]

@azlam-abdulsalam I agree, since yesterday when I was investigating I saw that some classes didn't get code coverage due to "compilation in progress" status

[azlam-abdulsalam]

@ypan0 @dieffrei Initial analysis is pointing to the fact this happens only during parallel testing, as code coverage is getting skipped for classes as it being recompiled in parallel

We have disabled parallel testing in some orgs and it seems to be working better. Monitoring this issue

[azlam-abdulsalam]

This has been fixed in alpha and will be part of the next release. All these test classes that fail to contribute to coverage will be retriggered in serial mode and the test results will be converged for code coverage calculation