MSTest v2.2.4 not working for some data tests: System.ArgumentException: Object of type 'System.Int32' cannot be converted to type 'System.UInt32

[juanmalm]

Description

After upgrading to MSTest 2.2.4 I'm getting System.ArgumentException for data tests with some data types

Steps to reproduce

• MSTest 2.2.4
• Create this test:

[TestMethod]
[DataRow(0U)]
[DataRow(1U)]
[DataRow(2U)]
public async Task TestWithUint(uint index)
{
    //ellided
}

Expected behavior

• Test to run without exceptions as prior to v2.2.4

Actual behavior

• Getting the exception:

    System.ArgumentException: Object of type 'System.Int32' cannot be converted to type 'System.UInt32'.
  Stack Trace: 
    RuntimeType.TryChangeType(Object value, Binder binder, CultureInfo culture, Boolean needsSpecialCast)
    RuntimeType.CheckValue(Object value, Binder binder, CultureInfo culture, BindingFlags invokeAttr)
    MethodBase.CheckArguments(Object[] parameters, Binder binder, BindingFlags invokeAttr, CultureInfo culture, Signature sig)
    RuntimeMethodInfo.InvokeArgumentsCheck(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
    RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
    MethodBase.Invoke(Object obj, Object[] parameters)
    ThreadOperations.ExecuteWithAbortSafety(Action action)

Environment

Windows 21H1 build 19043.1023
VS 16.9.6

[abatishchev]

Similar issue but different exception when my unit test accepts custom class:

An exception occurred while invoking executor 'executor://mstestadapter/v2': Type 'Service.Models.Resources.Account' with data contract name 'Account:http://schemas.datacontract.org/2004/07/Service.Models.Resources' is not expected. Add any types not known statically to the list of known types - for example, by using the KnownTypeAttribute attribute or by adding them to the list of known types passed to DataContractSerializer.
Stack trace:
   at System.Runtime.Serialization.XmlObjectSerializerWriteContext.SerializeAndVerifyType(DataContract dataContract, XmlWriterDelegator xmlWriter, Object obj, Boolean verifyKnownType, RuntimeTypeHandle declaredTypeHandle, Type declaredType)
   at System.Runtime.Serialization.Json.XmlObjectSerializerWriteContextComplexJson.SerializeWithXsiTypeAtTopLevel(DataContract dataContract, XmlWriterDelegator xmlWriter, Object obj, RuntimeTypeHandle originalDeclaredTypeHandle, Type graphType)
   at System.Runtime.Serialization.Json.DataContractJsonSerializerImpl.InternalWriteObjectContent(XmlWriterDelegator writer, Object graph)
   at System.Runtime.Serialization.Json.DataContractJsonSerializerImpl.InternalWriteObject(XmlWriterDelegator writer, Object graph)
   at System.Runtime.Serialization.XmlObjectSerializer.InternalWriteObject(XmlWriterDelegator writer, Object graph, DataContractResolver dataContractResolver)
   at System.Runtime.Serialization.XmlObjectSerializer.WriteObjectHandleExceptions(XmlWriterDelegator writer, Object graph, DataContractResolver dataContractResolver)
   at System.Runtime.Serialization.Json.DataContractJsonSerializerImpl.WriteObject(XmlDictionaryWriter writer, Object graph)
   at System.Runtime.Serialization.Json.DataContractJsonSerializerImpl.WriteObject(Stream stream, Object graph)
   at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Helpers.DataSerializationHelper.Serialize(Object[] data)
   at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.UnitTestElement.ToTestCase()
   at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.UnitTestDiscoverer.SendTestCases(String source, IEnumerable`1 testElements, ITestCaseDiscoverySink discoverySink, IDiscoveryContext discoveryContext, IMessageLogger logger)
   at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.UnitTestDiscoverer.DiscoverTestsInSource(String source, IMessageLogger logger, ITestCaseDiscoverySink discoverySink, IDiscoveryContext discoveryContext)
   at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestExecutionManager.RunTests(IEnumerable`1 sources, IRunContext runContext, IFrameworkHandle frameworkHandle, TestRunCancellationToken cancellationToken)
   at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestExecutor.RunTests(IEnumerable`1 sources, IRunContext runContext, IFrameworkHandle frameworkHandle)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.RunTestsWithSources.InvokeExecutor(LazyExtension`2 executor, Tuple`2 executorUriExtensionTuple, RunContext runContext, IFrameworkHandle frameworkHandle)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestInternalWithExecutors(IEnumerable`1 executorUriExtensionMap, Int64 totalTests)

NuGet packages when it works:

<PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.9.4" />
<PackageReference Include="MSTest.TestAdapter" Version="2.2.3" />
<PackageReference Include="MSTest.TestFramework" Version="2.2.3" />

and when it doesn't:

<PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.10.0" />
<PackageReference Include="MSTest.TestAdapter" Version="2.2.4" />
<PackageReference Include="MSTest.TestFramework" Version="2.2.4" />

Code:

[DynamicData(nameof(GetData), DynamicDataSourceType.Method)]
[DataTestMethod]
public void MyTest(Account account)
{
}

private static IEnumerable<object[]> GetData()
{
    yield return new object[]
    {
        new Account()
    };
}

[Sanan07]

Please use latest packages :
MSTest.TestAdapter v2.2.5-preview-20210605-01
MSTest.TestFramework v2.2.5-preview-20210605-01

[abatishchev]

@Sanan07 do you know when a stable version with the fix is planned to be release?

[Sanan07]

@Haplois can advice on it.

[Haplois]

15th of this month, @abatishchev.

[paul-michalik]

@Sanan07 and others: v2.2.5-preview fixes the data collection issues with DynamicData but it removes the ability to run tests cases in parallel. I went to v2.2.2 this is the last version which supports DynamicData properly AND executes tests in parallel (Scope = MethodLevel)

[Haplois]

@paul-michalik, we didn't discontinue support for running tests in parallel. You should still be able to run DynamicData tests in parallel in our latest preview. Do you have a repro for your issue?

[paul-michalik]

@Haplois Yes! Given a test suite with data driven test cases with a IEnumerable<object[]> data source, if I let the test cases run in parallel with method level parallelism then InvalidOperations exceptions are thrown because of parallel access to the Enumerables.

It looks as if the data source Enumerables were Move-d from different threads for the same test case. I realized that I do not know whether or not a single threaded access for pulling the data for each test case is actually guaranteed but it looks like, with v2.2.5, this assumption doesn't apply.

[Haplois]

After 2.2.4, as long as the values in data source are serializable, every row for the specific test case is considered a separate test case. If our assumptions break parallel test runs that's a regression. Is it possible for you to share a quick repro with us?

[paul-michalik]

After 2.2.4, as long as the values in data source are serializable, every row for the specific test case is considered a separate test case. If our assumptions break parallel test runs that's a regression. Is it possible for you to share a quick repro with us?

@Haplois I will share a complete project when I have more time, but in general, consider a test case like this:

[
  DataTestMethod,
  DynamicData(dynamicDataSourceName: nameof(GetData), DynamicDataSourceType.Method)
]
public async Task Create_Tasks_Requests_From_Invalid_Tasks_Definitions_Should_Return_BadRequest_Errors(Events.APIGatewayProxyRequest request, Requests.CreateTaskCommand.Arguments arguments)
{
    // test code
}

with GetData similar to this:

public static IEnumerable<object[]> GetData() =>
 new [] {
    new [] object {
         // initialize pair of objects
    },
    new [] object {
         // initialize pair of objects
    },
 }

If I run such test cases in parallel (method level parallelism) then I see all kinds of exception suggesting that the the test framework attempts to pull the enumerable on different threads. I have plenty of data sources structured like this:

public static IEnumerable<object[]> GetData()
{
    yield new InputData {
       // ...
    };

    yield new InputData {
       // ...
    };

    // etc.
}

These obviously can't be pulled in parallel either.

[Haplois]

@paul-michalik thanks for reporting this. We'll we working on fixing this. You can track it here #906.