Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

2X different results from identical copy-pasted code #2138

Closed
alex-jitbit opened this issue Oct 6, 2022 · 4 comments
Closed

2X different results from identical copy-pasted code #2138

alex-jitbit opened this issue Oct 6, 2022 · 4 comments

Comments

@alex-jitbit
Copy link

alex-jitbit commented Oct 6, 2022

I have MyClass and MyClass2 which is just copy-pasted same identical code. Again - letter-by-letter same, as in "completely identical" just the names are different.

Occasionally, the results I get (about every run out of 3) are drastically different

Method Mean Error StdDev Gen 0 Gen 1 Allocated
Test1 463.4 us 23.23 us 1.27 us 49.8047 10.7422 306 KB
Test2 295.2 us 47.14 us 2.58 us 49.8047 10.7422 306 KB

The version I use: 0.13.1

Maybe it's related to #2004

@alex-jitbit alex-jitbit changed the title 2X different results from identical class with different names (only names) 2X different results from identical copy-pasted code Oct 6, 2022
@adamsitnik
Copy link
Member

Hi @alex-jitbit

Such differences can be caused by multiple reasons:

  • your benchmarks doing something that is dependent on other things like IO, sys-calls, task scheduling etc
  • code alignment
  • memory alignment. PTAL at Memory Randomization #1587 (comment)

You need to use a profiler to find out what is causing the difference, get two identical trace files and compare them. If you are running on Intel hardware you can use VTune which is now free. We don't have the time to investigate such outliers for our users, sorry.

@adamsitnik adamsitnik closed this as not planned Won't fix, can't repro, duplicate, stale Oct 6, 2022
@alex-jitbit
Copy link
Author

@adamsitnik nope, the code I have is self-contained, it's a string parser, no external IO or async/task scheduling involved.

Just simple array and Span manipulations.

So, what you're actually saying is "whenever BDN produces any results, you cannot trust them until you trace the two code paths". OK, roger that.

@adamsitnik
Copy link
Member

People open issues like this expecting that they have hit a BenchmarkDotNet bug.

I've investigated multiple similar issues, it took days to do that and the reason behind such differences for "same code" was always either code alignment (#756) or memory alignment (#1587).

#756 was mainly addressed by .NET 6 itself which started aligning the loops: https://www.youtube.com/watch?v=ibHOoIj8U8k

We have not enabled memory randomization (#1587) by default because BDN would perform even more iterations (the results would not be stable so quickly) and it would work only for certain cases (it's not always possible to re-allocate the problematic memory inside global setup method).

My main message here is "it's not a bug, it's alignment. If you don't believe me, use a different tool to verify that".

@alex-jitbit
Copy link
Author

alex-jitbit commented Oct 6, 2022

@adamsitnik Yes, I understand. But this means, that when the code is ACTUALLY DIFFERENT we can not know for sure, was it the code changes that made the benchmark faster/slower, or just the "alignment".

Feel free to close the issue tho, I understand the reasoning behind it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants