Problems with [IterationSetup]

[Kosat]

Hi,

In one of my BenchmarkDotNet(0.11.5) benchmarks I have a need to reuse an instance of StreamReader which I initialize once at startup and then after each iteration I reset an underlying stream in it. Like below:

    [BenchmarkCategory(Categories.XYZ)]
    public class Perf_Benchmark1
    {
	private readonly StreamReader _reader;
		
	[GlobalSetup]
        public void Setup()
        {
            _reader = new StreamReader(...);
        }
		
	[IterationSetup]
        public void IterationSetup()
        {
            _reader.BaseStream.Seek(0, SeekOrigin.Begin);
        }
		
	[Benchmark]
        public object Benchmark()
        {
	    return SUT.DoSomething(_reader);
	}
    }

I know that the docs discourage developers from using [IterationSetup] but in my case its usage looks justified.
The problem is that once I've added the method [IterationSetup] public void IterationSetup() to my code
the BenchmarkDotNet has stopped inferring the InvocationCount correctly. Instead, it always defaults to InvocationCount=1 which is not appropriate for my benchmark and probably for any other real-life benchmarks.

Is this a bug? What'd be a better way of resetting the stream in between the iterations in the code above?

The minimum code to reproduce such behavior:

    [BenchmarkCategory(Categories.XYZ)]
    public class Perf_Benchmark1
    {	
        [IterationSetup]
        public void IterationSetup()
        {
            // can be just empty
        }
		
        [Benchmark]
        public List<int> Benchmark() => new List<int>();
    }

[Rizzen]

BenchmarkDotNet sets InvocationCount=1 and UnrollFactor=1 when IterationSetup or IterationCleanup is specified:

BenchmarkDotNet/src/BenchmarkDotNet/Jobs/JobExtensions.cs

Line 339 in 8aa6ade

return job.RunOncePerIteration();

[adamsitnik]

@Kosat as @Rizzen explained, this is the default behaviour since 0.11.0. You can read #730 for more details.

When I work with Streams in benchmarks I always set the position to 0 at the beginning of the benchmark.

https://github.com/dotnet/performance/blob/0c626c0de72e9e018bb02d0f92b97117098f1fee/src/benchmarks/micro/Serializers/Binary_ToStream.cs#L37-L41

So in your case, it would be:

[Benchmark]
public object Benchmark()
{
   _reader.BaseStream.Position = 0;

    return SUT.DoSomething(_reader);
}

You can use ETWProfiler to make sure that this operation does not dominate the reported time.

I am going to close this issue since it's by design. Please feel free to ask more questions.