-
Notifications
You must be signed in to change notification settings - Fork 4.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Mono JIT/HardwareIntrinsics - Massive failures #75606
Comments
This is significantly blocking CI (19x impacted runs in ~1 day). We need to roll back the change that caused it or disable the test ASAP. @SamMonoRT @BrzVlad @fanyang-mono can you please help route it? |
BTW: I think I saw in one of the logs inner exception to be |
@tannergooding @fanyang-mono - can you confirm these failures are not related to #75470 |
@SamMonoRT, Int128 has no relation to the HardwareIntrinsics tests. They aren't used by eachother in any way. The Mono llvmaot failures look to be an issue with the I'd guess there isn't handling for the |
It might be related to #75438 or one of the other recent Mono SIMD changes |
It seems from above that these run on PR validation? So unless there's flakiness, it should be possible to figure which PR first merged with these failures? |
Right, I am working on it now. |
Here is the timeline of some issues related to simd intrinsics. There were some failures on android which I investigated, #74797 was a fix for it by disabling intrinsics. There were no llvm failures noticed on that PR, merged on main, roughly one week ago. Then tried to backport to .net 7. Noticed these same failures here on the backport PR. Tested locally my changed, saw that my change was introducing failures that were awkward to fix so gave up on the backport. Yesterday I disabled that original change via #75438, so behavior is as before. I tested again locally to see the impact of my revert. I notice again that the revert fixes the intrinsics issues for me so I don't understand why there are still failures today on CI for this. Since my PR didn't have failures but the backport PR did, I don't fully trust what is going on. Note that locally I didn't run the XUnit wrapper, I just ran runtime tests one at a time while aot-ing with llvm |
These are the PR's merged between a good rolling build 20220913.4 and the current bad rolling build 20220913.80 The failures are on x64 with Mono runtime. I suspect that #75464 might caused this. @lateralusX Any thoughts? |
The code in #75464 is guarded with MONO_ARCH_CODE_EXEC_ONLY/MONO_VALIDATE_PLT_ENTRY_INDEX defines. |
I verified that these test failures was exposed by #75438, but introduced by some other PR. Because #75438 enables intrinsics support back on non-full-aot mode Mono. The test failures were probably introduced by one of @matouskozak's AMD64 intrinsics PR's merged during the past week. I am currently working with @matouskozak to get a fix. The reason why it wasn't caught during PR validation is because a recent change (#74601) caused the effect that the problematic CI lane ( |
The failures was actually not introduced by @matouskozak's PR's. It was caused by #75055 for disabling LLVM for the JIT fall back. |
I have confirmed that these tests are not failing on CI any more. Here is the full story: |
63x failures in each run. Recent regression as of 9/13.
Platform: mono Linux x64 Release @ Ubuntu.1804.Amd64.Open
Last 30 days - JIT.HardwareIntrinsics.X86.Ssse3 in Runfo as of 9/14
There are more JIT.HardwareIntrinsics.* Work Items affected.
Example of error in Console log:
Report
Summary
The text was updated successfully, but these errors were encountered: