Feature: Add CollectionDefinition.DisableParallelization

[NickCraver]

This spawned from a Twitter discussion here.

In every one of our libraries, especially with settings, event handlers, etc. some tests just aren't friendly with parallelization. We want parallel testing in the vast majority of cases (that can in itself reveal issues), but specific tests (or collections) we want to run non-parallel separately from everything else.

This change allows a CollectionDefinition to specify DisableParallelization = true to support this. The behavior is: all parallel collections (any not specifying this new constructor argument specifically) run first and in parallel (no behavior change from today). Then any collections specifying DisableParallelization = true are run, each is awaited in order. This means that one 1 DisableParallelization collection is running at a time.

Note the code is a bit more complicated to reduce allocations, but if we want to allocate and simplify I can take that route.

The alternatives to this sort of change is: override [Fact] everywhere and handle semaphores externally. Something that's pretty pervasive for a base level change. Or, split all tests into another assembly with 1 thread (disabling parallelization). This (from my experience) would result in a lot of code duplication and complication.

With this change, a user can specify:

[CollectionDefinition("Event Handler Tests", DisableParallelization = true)]
public class EventHandlerTestDefinition { }

[Collection("Event Handler Tests")]
public class EventHandlerTests
{
    [Fact]
    public void EventHandlerAddRemove() { }
}

...and those tests which have global impact (causing many other tests or themselves to fail) can run at the end, safely testing everything and avoiding many of the race conditions and inconsistencies otherwise present when they interact with all tests.

Questions: Is there a better property name for this behavior?

Side notes: it'd be handle to switch to nameof() in other places attribute constructors are fetched for seeing where they're used in VS 2017. If that's wanted, I'll open another PR specifically for that.

cc @bradwilson @onovotny

[dnfclas]

@NickCraver,
Thanks for having already signed the Contribution License Agreement. Your agreement was validated by .NET Foundation. We will now review your pull request.
Thanks,
.NET Foundation Pull Request Bot

[bradwilson]

Thanks!