-
-
Notifications
You must be signed in to change notification settings - Fork 982
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
2X different results from identical copy-pasted code #2138
Comments
Hi @alex-jitbit Such differences can be caused by multiple reasons:
You need to use a profiler to find out what is causing the difference, get two identical trace files and compare them. If you are running on Intel hardware you can use VTune which is now free. We don't have the time to investigate such outliers for our users, sorry. |
@adamsitnik nope, the code I have is self-contained, it's a string parser, no external IO or async/task scheduling involved. Just simple array and Span manipulations. So, what you're actually saying is "whenever BDN produces any results, you cannot trust them until you trace the two code paths". OK, roger that. |
People open issues like this expecting that they have hit a BenchmarkDotNet bug. I've investigated multiple similar issues, it took days to do that and the reason behind such differences for "same code" was always either code alignment (#756) or memory alignment (#1587). #756 was mainly addressed by .NET 6 itself which started aligning the loops: https://www.youtube.com/watch?v=ibHOoIj8U8k We have not enabled memory randomization (#1587) by default because BDN would perform even more iterations (the results would not be stable so quickly) and it would work only for certain cases (it's not always possible to re-allocate the problematic memory inside global setup method). My main message here is "it's not a bug, it's alignment. If you don't believe me, use a different tool to verify that". |
@adamsitnik Yes, I understand. But this means, that when the code is ACTUALLY DIFFERENT we can not know for sure, was it the code changes that made the benchmark faster/slower, or just the "alignment". Feel free to close the issue tho, I understand the reasoning behind it. |
I have
MyClass
andMyClass2
which is just copy-pasted same identical code. Again - letter-by-letter same, as in "completely identical" just the names are different.Occasionally, the results I get (about every run out of 3) are drastically different
The version I use: 0.13.1
Maybe it's related to #2004
The text was updated successfully, but these errors were encountered: