Cryptic error message when DynamicData doesn't have any data

[jayaranigarg]

Pulling in from vstest

Description

Using DynamicDataAttribute without any data produces a cryptic error message:

Failed (test name)
Error Message:
Error in executing test. No result returned by extension. If using extension of
TestMethodAttribute then please contact vendor.

I'm less concerned with what the correct behavior should be here and more concerned with providing developers with meaningful errors. It was very confusing to figure out why my test was failing after my IEnumerable<object[]> was processed but before the first line of code was reached. Note: my DynamicDaa was a .Where() clause on different test data; the repro in this issue is a simplified version of my original test.

I don't think that a DynamicData test without any data should be considered a "passing test", but I hesitate to call it a "failing test."

Steps to reproduce

1. Run the command:
   dotnet new mstest
2. Update the nuget packages:

• MSTest.TestFramework 1.2.0 -> 1.3.0
• MSTest.TestAdapter 1.2.0 -> 1.3.0
• Microsoft.Net.TestSdk 15.6.0 -> 15.7.2

3. Replace the contents of UnitTest1.cs with:

using Microsoft.VisualStudio.TestTools.UnitTesting;
using System.Collections.Generic;

[TestClass]
public class UnitTest1
{
    public static IEnumerable<object[]> Empty => new object[][]{};
    [TestMethod]
    [DynamicData(nameof(Empty))]
    public void Object_IsNotNull(object o) => Assert.IsNotNull(o);
}

4. Run the command:
   dotnet test

Expected behavior

Something to the effect of:

Build started, please wait...
Build completed.

Test run for D:(masked)\bin\Debug\netcoreapp2.0\repro.dll(.NETCoreApp,Version
=v2.0)
Microsoft (R) Test Execution Command Line Tool Version 15.6.0
Copyright (c) Microsoft Corporation. All rights reserved.

Starting test execution, please wait...
Skipped Object_IsNotNull
Error Message:
DynamicDataAttribute specified an IEnumerable<object[]> to use for test parameters, but the enumerable was empty.

Total tests: 1. Passed: 0. Failed: 0. Skipped: 1.
Test Run Successful.
Test execution time: 1.7951 Seconds

Actual behavior

Build started, please wait...
Build completed.

Test run for D:(masked)\bin\Debug\netcoreapp2.0\repro.dll(.NETCoreApp,Version
=v2.0)
Microsoft (R) Test Execution Command Line Tool Version 15.6.0
Copyright (c) Microsoft Corporation. All rights reserved.

Starting test execution, please wait...
Failed Object_IsNotNull
Error Message:
Error in executing test. No result returned by extension. If using extension of
TestMethodAttribute then please contact vendor.

Total tests: 2. Passed: 1. Failed: 1. Skipped: 0.
Test Run Failed.
Test execution time: 1.8811 Seconds

Diagnostic logs

dotnet test -d bar.txt

TpTrace Information: 0 : 20280, 1, 2018/05/17, 19:14:59.862, 31094822327, vstest.console.dll, Using .Net Framework version:.NETCoreApp,Version=v2.0
TpTrace Verbose: 0 : 20280, 1, 2018/05/17, 19:14:59.888, 31094886291, vstest.console.dll, TestPluginCache: Discovering the extensions using extension path.
TpTrace Information: 0 : 20280, 1, 2018/05/17, 19:14:59.888, 31094888585, vstest.console.dll, AssemblyResolver: Microsoft.TestPlatform.Extensions.BlameDataCollector: Resolving assembly.
TpTrace Information: 0 : 20280, 1, 2018/05/17, 19:14:59.889, 31094891268, vstest.console.dll, AssemblyResolver: Microsoft.TestPlatform.Extensions.BlameDataCollector: Resolved assembly. 
TpTrace Information: 0 : 20280, 1, 2018/05/17, 19:14:59.891, 31094897652, vstest.console.dll, AssemblyResolver: Microsoft.TestPlatform.Extensions.EventLogCollector: Resolving assembly.
TpTrace Information: 0 : 20280, 1, 2018/05/17, 19:14:59.892, 31094900996, vstest.console.dll, AssemblyResolver: Microsoft.TestPlatform.Extensions.EventLogCollector: Resolved assembly. 
TpTrace Information: 0 : 20280, 1, 2018/05/17, 19:14:59.894, 31094907045, vstest.console.dll, AssemblyResolver: Microsoft.TestPlatform.TestHostRuntimeProvider: Resolving assembly.
TpTrace Information: 0 : 20280, 1, 2018/05/17, 19:14:59.895, 31094909858, vstest.console.dll, AssemblyResolver: Microsoft.TestPlatform.TestHostRuntimeProvider: Resolved assembly. 
TpTrace Information: 0 : 20280, 1, 2018/05/17, 19:14:59.898, 31094918490, vstest.console.dll, AssemblyResolver: Microsoft.VisualStudio.TestPlatform.Extensions.Trx.TestLogger: Resolving assembly.
TpTrace Information: 0 : 20280, 1, 2018/05/17, 19:14:59.899, 31094921559, vstest.console.dll, AssemblyResolver: Microsoft.VisualStudio.TestPlatform.Extensions.Trx.TestLogger: Resolved assembly. 
TpTrace Verbose: 0 : 20280, 1, 2018/05/17, 19:14:59.901, 31094927692, vstest.console.dll, TestPluginCache: Discovered the extensions using extension path ''.
TpTrace Verbose: 0 : 20280, 1, 2018/05/17, 19:14:59.901, 31094927983, vstest.console.dll, TestPluginCache: Discoverers are ''.
TpTrace Verbose: 0 : 20280, 1, 2018/05/17, 19:14:59.901, 31094928108, vstest.console.dll, TestPluginCache: Executors are ''.
TpTrace Verbose: 0 : 20280, 1, 2018/05/17, 19:14:59.901, 31094928208, vstest.console.dll, TestPluginCache: Setting providers are ''.
TpTrace Verbose: 0 : 20280, 1, 2018/05/17, 19:14:59.901, 31094928285, vstest.console.dll, TestPluginCache: Loggers are ''.
TpTrace Information: 0 : 20280, 1, 2018/05/17, 19:14:59.938, 31095044567, vstest.console.dll, RunTestsArgumentProcessor:Execute: Test run is starting.
TpTrace Verbose: 0 : 20280, 1, 2018/05/17, 19:14:59.938, 31095044904, vstest.console.dll, RunTestsArgumentProcessor:Execute: Queuing Test run.
TpTrace Information: 0 : 20280, 1, 2018/05/17, 19:14:59.938, 31095045348, vstest.console.dll, TestRequestManager.RunTests: run tests started.
TpTrace Information: 0 : 20280, 1, 2018/05/17, 19:14:59.951, 31095083609, vstest.console.dll, AssemblyMetadataProvider.GetFrameWork: Determined framework:'.NETCoreApp,Version=v2.0' for source: 'D:\(masked)\bin\Debug\netcoreapp2.0\repro.dll'
TpTrace Information: 0 : 20280, 1, 2018/05/17, 19:14:59.951, 31095083993, vstest.console.dll, Determined framework for all sources: .NETCoreApp,Version=v2.0
TpTrace Information: 0 : 20280, 1, 2018/05/17, 19:14:59.953, 31095090943, vstest.console.dll, AssemblyMetadataProvider.GetArchitecture: Determined architecture:AnyCPU info for assembly: D:\(masked)\bin\Debug\netcoreapp2.0\repro.dll
TpTrace Information: 0 : 20280, 1, 2018/05/17, 19:14:59.956, 31095099124, vstest.console.dll, Determined platform for all sources: X86
TpTrace Information: 0 : 20280, 1, 2018/05/17, 19:14:59.960, 31095112883, vstest.console.dll, Compatible sources list : 
TpTrace Information: 0 : 20280, 1, 2018/05/17, 19:14:59.960, 31095113143, vstest.console.dll, D:\(masked)\bin\Debug\netcoreapp2.0\repro.dll
TpTrace Information: 0 : 20280, 1, 2018/05/17, 19:14:59.962, 31095118462, vstest.console.dll, InferRunSettingsHelper.IsTestSettingsEnabled: Unable to navigate to RunSettings/MSTest. Current node: RunSettings
TpTrace Information: 0 : 20280, 1, 2018/05/17, 19:14:59.964, 31095124236, vstest.console.dll, TestPluginManager.CreateTestExtension: Attempting to load test extension: Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Hosting.DefaultTestHostManager
TpTrace Information: 0 : 20280, 1, 2018/05/17, 19:14:59.964, 31095125194, vstest.console.dll, TestPluginManager.CreateTestExtension: Attempting to load test extension: Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Hosting.DotnetTestHostManager
TpTrace Verbose: 0 : 20280, 1, 2018/05/17, 19:14:59.964, 31095126251, vstest.console.dll, TestEngine: Initializing Parallel Execution as MaxCpuCount is set to: 1
TpTrace Information: 0 : 20280, 1, 2018/05/17, 19:14:59.965, 31095127198, vstest.console.dll, InferRunSettingsHelper.IsTestSettingsEnabled: Unable to navigate to RunSettings/MSTest. Current node: RunSettings
TpTrace Verbose: 0 : 20280, 1, 2018/05/17, 19:14:59.979, 31095170867, vstest.console.dll, TestRequestSender is acting as server
TpTrace Verbose: 0 : 20280, 1, 2018/05/17, 19:14:59.980, 31095176328, vstest.console.dll, TestRunRequest.ExecuteAsync: Creating test run request.
TpTrace Verbose: 0 : 20280, 1, 2018/05/17, 19:14:59.981, 31095177431, vstest.console.dll, TestRunRequest.ExecuteAsync: Starting.
TpTrace Information: 0 : 20280, 1, 2018/05/17, 19:14:59.981, 31095177952, vstest.console.dll, TestRunRequest.ExecuteAsync: Starting run with settings:TestRunCriteria:
   KeepAlive=False,FrequencyOfRunStatsChangeEvent=10,RunStatsChangeEventTimeout=00:00:01.5000000,TestCaseFilter=,TestExecutorLauncher=
   Settingsxml=<RunSettings>
  <RunConfiguration>
    <ResultsDirectory>D:\(masked)\TestResults</ResultsDirectory>
    <TargetPlatform>X86</TargetPlatform>
    <TargetFrameworkVersion>.NETCoreApp,Version=v2.0</TargetFrameworkVersion>
    <DesignMode>False</DesignMode>
    <CollectSourceInformation>False</CollectSourceInformation>
  </RunConfiguration>
</RunSettings>

TpTrace Verbose: 0 : 20280, 1, 2018/05/17, 19:14:59.981, 31095178075, vstest.console.dll, TestRunRequest.ExecuteAsync: Wait for the first run request is over.
TpTrace Verbose: 0 : 20280, 1, 2018/05/17, 19:14:59.981, 31095179198, vstest.console.dll, ParallelProxyExecutionManager: Start execution. Total sources: 1
TpTrace Information: 0 : 20280, 1, 2018/05/17, 19:14:59.982, 31095180915, vstest.console.dll, ProxyParallelExecutionManager: Triggering test run for next source: D:\(masked)\bin\Debug\netcoreapp2.0\repro.dll
TpTrace Verbose: 0 : 20280, 1, 2018/05/17, 19:14:59.982, 31095181212, vstest.console.dll, ProxyParallelExecutionManager: No sources available for execution.
TpTrace Information: 0 : 20280, 1, 2018/05/17, 19:14:59.982, 31095181317, vstest.console.dll, TestRunRequest.ExecuteAsync: Started.
TpTrace Verbose: 0 : 20280, 4, 2018/05/17, 19:14:59.982, 31095181241, vstest.console.dll, ParallelProxyExecutionManager: Execution started. Started clients: 1
TpTrace Verbose: 0 : 20280, 1, 2018/05/17, 19:14:59.982, 31095181503, vstest.console.dll, TestRunRequest.WaitForCompletion: Waiting with timeout -1.
TpTrace Verbose: 0 : 20280, 4, 2018/05/17, 19:14:59.982, 31095182037, vstest.console.dll, ProxyExecutionManager: Test host is always Lazy initialize.
TpTrace Verbose: 0 : 20280, 4, 2018/05/17, 19:14:59.982, 31095182549, vstest.console.dll, TestRequestSender.InitializeCommunication: initialize communication. 
TpTrace Information: 0 : 20280, 4, 2018/05/17, 19:14:59.993, 31095215023, vstest.console.dll, SocketServer: Listening on end point : 127.0.0.1:6505
TpTrace Verbose: 0 : 20280, 4, 2018/05/17, 19:14:59.997, 31095227963, vstest.console.dll, DotnetTestHostmanager: Full path of dotnet.exe is C:\Program Files\dotnet\dotnet.exe
TpTrace Verbose: 0 : 20280, 4, 2018/05/17, 19:14:59.997, 31095228499, vstest.console.dll, DotnetTestHostmanager: Adding  --runtimeconfig "D:\(masked)\bin\Debug\netcoreapp2.0\repro.runtimeconfig.json" in args
TpTrace Verbose: 0 : 20280, 4, 2018/05/17, 19:14:59.997, 31095228691, vstest.console.dll, DotnetTestHostmanager: Adding  --depsfile "D:\(masked)\bin\Debug\netcoreapp2.0\repro.deps.json" in args
TpTrace Information: 0 : 20280, 4, 2018/05/17, 19:14:59.997, 31095229115, vstest.console.dll, AssemblyResolver: Microsoft.Extensions.DependencyModel: Resolving assembly.
TpTrace Information: 0 : 20280, 4, 2018/05/17, 19:14:59.998, 31095232105, vstest.console.dll, AssemblyResolver: Microsoft.Extensions.DependencyModel: Resolved assembly. 
TpTrace Verbose: 0 : 20280, 4, 2018/05/17, 19:14:59.999, 31095232909, vstest.console.dll, DotnetTestHostmanager: Reading file D:\(masked)\bin\Debug\netcoreapp2.0\repro.deps.json to get path of testhost.dll
TpTrace Verbose: 0 : 20280, 4, 2018/05/17, 19:15:00.056, 31095413396, vstest.console.dll, DotnetTestHostmanager: Relative path of testhost.dll with respect to package folder is microsoft.testplatform.testhost/15.7.2\lib/netstandard1.5/testhost.dll
TpTrace Verbose: 0 : 20280, 4, 2018/05/17, 19:15:00.076, 31095475081, vstest.console.dll, DotnetTestHostmanager: Looking for path microsoft.testplatform.testhost/15.7.2\lib/netstandard1.5/testhost.dll in folder C:\Users\(masked)\.dotnet\store\|arch|\|tfm|
TpTrace Verbose: 0 : 20280, 4, 2018/05/17, 19:15:00.077, 31095476860, vstest.console.dll, DotnetTestHostmanager: Looking for path microsoft.testplatform.testhost/15.7.2\lib/netstandard1.5/testhost.dll in folder C:\Users\(masked)\.nuget\packages
TpTrace Verbose: 0 : 20280, 4, 2018/05/17, 19:15:00.077, 31095478558, vstest.console.dll, DotnetTestHostmanager: Full path of testhost.dll is C:\Users\(masked)\.nuget\packages\microsoft.testplatform.testhost/15.7.2\lib/netstandard1.5/testhost.dll
TpTrace Verbose: 0 : 20280, 5, 2018/05/17, 19:15:00.079, 31095483694, vstest.console.dll, DotnetTestHostManager: Starting process 'C:\Program Files\dotnet\dotnet.exe' with command line 'exec --runtimeconfig "D:\(masked)\bin\Debug\netcoreapp2.0\repro.runtimeconfig.json" --depsfile "D:\(masked)\bin\Debug\netcoreapp2.0\repro.deps.json" "C:\Users\(masked)\.nuget\packages\microsoft.testplatform.testhost/15.7.2\lib/netstandard1.5/testhost.dll" --port 6505 --endpoint 127.0.0.1:06505 --role client --parentprocessid 20280 --diag "D:\(masked)\bar.host.18-05-17_19-14-59_99634_4.txt" --telemetryoptedin false'
TpTrace Verbose: 0 : 20280, 5, 2018/05/17, 19:15:00.086, 31095506360, vstest.console.dll, Test Runtime launched
TpTrace Verbose: 0 : 20280, 4, 2018/05/17, 19:15:00.088, 31095512555, vstest.console.dll, TestRequestSender.WaitForRequestHandlerConnection: waiting for connection with timeout: 60000
TpTrace Verbose: 0 : 20280, 9, 2018/05/17, 19:15:01.017, 31098416016, vstest.console.dll, Client connected, and starting MessageLoopAsync
TpTrace Verbose: 0 : 20280, 9, 2018/05/17, 19:15:01.331, 31099397946, vstest.console.dll, TestRequestSender.CheckVersionWithTestHost: onMessageReceived received message: (ProtocolVersion) -> 2
TpTrace Verbose: 0 : 20280, 4, 2018/05/17, 19:15:01.337, 31099415013, vstest.console.dll, TestRequestSender.InitializeExecution: Sending initializing execution with message: {"Version":2,"MessageType":"TestExecution.Initialize","Payload":["D:\\(masked)\\bin\\Debug\\netcoreapp2.0\\Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll"]}
TpTrace Verbose: 0 : 20280, 4, 2018/05/17, 19:15:01.352, 31099463788, vstest.console.dll, TestRequestSender.StartTestRun: Sending test run with message: {"Version":2,"MessageType":"TestExecution.StartWithSources","Payload":{"AdapterSourceMap":{"_none_":["D:\\(masked)\\bin\\Debug\\netcoreapp2.0\\repro.dll"]},"RunSettings":"<RunSettings>\r\n  <RunConfiguration>\r\n    <ResultsDirectory>D:\\(masked)\\TestResults</ResultsDirectory>\r\n    <TargetPlatform>X86</TargetPlatform>\r\n    <TargetFrameworkVersion>.NETCoreApp,Version=v2.0</TargetFrameworkVersion>\r\n    <DesignMode>False</DesignMode>\r\n    <CollectSourceInformation>False</CollectSourceInformation>\r\n  </RunConfiguration>\r\n</RunSettings>","TestExecutionContext":{"FrequencyOfRunStatsChangeEvent":10,"RunStatsChangeEventTimeout":"00:00:01.5000000","InIsolation":false,"KeepAlive":false,"AreTestCaseLevelEventsRequired":false,"IsDebug":false,"TestCaseFilter":null,"FilterOptions":null},"Package":null}}
TpTrace Verbose: 0 : 20280, 9, 2018/05/17, 19:15:01.346, 31099444808, vstest.console.dll, TestRequestSender.OnExecutionMessageReceived: Received message: {"Version":2,"MessageType":"TestSession.Message","Payload":{"MessageLevel":0,"Message":"Logging TestHost Diagnostics in file: D:\\(masked)\\bar.host.18-05-17_19-14-59_99634_4.txt"}}
TpTrace Verbose: 0 : 20280, 9, 2018/05/17, 19:15:01.357, 31099478075, vstest.console.dll, TestRunRequest:SendTestRunMessage: Starting.
TpTrace Information: 0 : 20280, 9, 2018/05/17, 19:15:01.357, 31099478588, vstest.console.dll, TestRunRequest:SendTestRunMessage: Completed.
TpTrace Verbose: 0 : 20280, 9, 2018/05/17, 19:15:01.672, 31100463248, vstest.console.dll, TestRequestSender.OnExecutionMessageReceived: Received message: {"Version":2,"MessageType":"TestExecution.Completed","Payload":{"TestRunCompleteArgs":{"TestRunStatistics":{"ExecutedTests":1,"Stats":{"Failed":1}},"IsCanceled":false,"IsAborted":false,"Error":null,"AttachmentSets":[],"ElapsedTimeInRunningTests":"00:00:00.1226074","Metrics":{}},"LastRunTests":{"NewTestResults":[{"TestCase":{"Id":"2d14f5e1-ff5f-49e3-db6c-d5d2f197a01a","FullyQualifiedName":"UnitTest1.Object_IsNotNull","DisplayName":"Object_IsNotNull","ExecutorUri":"executor://MSTestAdapter/v2","Source":"D:\\(masked)\\bin\\Debug\\netcoreapp2.0\\repro.dll","CodeFilePath":null,"LineNumber":-1,"Properties":[{"Key":{"Id":"MSTestDiscoverer.TestClassName","Label":"ClassName","Category":"","Description":"","Attributes":1,"ValueType":"System.String"},"Value":"UnitTest1"},{"Key":{"Id":"TestObject.Traits","Label":"Traits","Category":"","Description":"","Attributes":5,"ValueType":"System.Collections.Generic.KeyValuePair`2[[System.String],[System.String]][]"},"Value":[]}]},"Attachments":[],"Outcome":2,"ErrorMessage":"Error in executing test. No result returned by extension. If using extension of TestMethodAttribute then please contact vendor.","ErrorStackTrace":null,"DisplayName":null,"Messages":[],"ComputerName":null,"Duration":"00:00:00","StartTime":"2018-05-17T19:15:01.5444342-05:00","EndTime":"2018-05-17T19:15:01.5794362-05:00","Properties":[]}],"TestRunStatistics":{"ExecutedTests":1,"Stats":{"Failed":1}},"ActiveTests":[]},"RunAttachments":[],"ExecutorUris":["executor://mstestadapter/v2"]}}
TpTrace Verbose: 0 : 20280, 9, 2018/05/17, 19:15:01.673, 31100464713, vstest.console.dll, TestRequestSender.EndSession: Sending end session.
TpTrace Information: 0 : 20280, 8, 2018/05/17, 19:15:01.702, 31100557416, vstest.console.dll, TestRequestSender.OnClientProcessExit: Test host process exited. Standard error: 
TpTrace Information: 0 : 20280, 8, 2018/05/17, 19:15:01.702, 31100557789, vstest.console.dll, SocketServer: Stop: Cancellation requested. Stopping message loop.
TpTrace Warning: 0 : 20280, 9, 2018/05/17, 19:15:01.702, 31100557976, vstest.console.dll, ProxyOperationManager: Timed out waiting for test host to exit. Will terminate process.
TpTrace Verbose: 0 : 20280, 9, 2018/05/17, 19:15:01.796, 31100850636, vstest.console.dll, TestRunRequest:SendTestRunStatsChange: Starting.
TpTrace Information: 0 : 20280, 9, 2018/05/17, 19:15:01.796, 31100851257, vstest.console.dll, TestRunRequest:SendTestRunStatsChange: Completed.
TpTrace Verbose: 0 : 20280, 9, 2018/05/17, 19:15:01.796, 31100851638, vstest.console.dll, ParallelProxyExecutionManager: HandlePartialRunComplete: Total completed clients = 1, Run complete = True, Run canceled: False.
TpTrace Verbose: 0 : 20280, 9, 2018/05/17, 19:15:01.812, 31100899293, vstest.console.dll, TestRunRequest:TestRunComplete: Starting. IsAborted:False IsCanceled:False.
TpTrace Information: 0 : 20280, 9, 2018/05/17, 19:15:01.814, 31100908021, vstest.console.dll, TestRunRequest:TestRunComplete: Completed.
TpTrace Verbose: 0 : 20280, 9, 2018/05/17, 19:15:01.815, 31100908426, vstest.console.dll, TestRequestSender.SetOperationComplete: Setting operation complete.
TpTrace Verbose: 0 : 20280, 1, 2018/05/17, 19:15:01.814, 31100908138, vstest.console.dll, TestRunRequest.Dispose: Starting.
TpTrace Information: 0 : 20280, 1, 2018/05/17, 19:15:01.815, 31100909185, vstest.console.dll, TestRunRequest.Dispose: Completed.
TpTrace Information: 0 : 20280, 1, 2018/05/17, 19:15:01.815, 31100909321, vstest.console.dll, TestRequestManager.RunTests: run tests completed, sucessful: True.
TpTrace Information: 0 : 20280, 1, 2018/05/17, 19:15:01.815, 31100909570, vstest.console.dll, RunTestsArgumentProcessor:Execute: Test run is completed.
TpTrace Verbose: 0 : 20280, 1, 2018/05/17, 19:15:01.815, 31100909689, vstest.console.dll, Executor.Execute: Exiting with exit code of 1
TpTrace Verbose: 0 : 20280, 9, 2018/05/17, 19:15:01.815, 31100909681, vstest.console.dll, TestRequestSender: OnTestRunAbort: Operation is already complete. Skip error message.

Environment

dotnet --version

2.1.104

I am running Windows 7 SP1.

Finally, for good measure, a screenshot of the Test Explorer in VS:

[jayaranigarg]

@Jm-Fox : We indeed agree on your feedback. A better error message would make more sense.

We would be happy to accept a contribution for the same from you :)
Feel free to contact us if you need any help on that!