ConditionalFactAttribute on the tests are ignored when ran on VS Test Explorer

[Gnbrkm41]

It appears that ConditionalTheory Attributes on the tests are ignored during discovery / run and the tests are allowed to run regardless of the value returned from the predicate.

For example, Various tests under System.Net.Security.Tests were failing with TypeInitializationException, PlatformNotSupportedException and ObjectDisposedExceptions; it appears that most of them have ConditionalFact attributes, which when evaluated returns false.

Tests that had failed, though not all of them have the attribute:

Test	Duration	Traits	Error Message
System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_AllowTwoOnBothSidesWithSingleOverlapNonTls13_Success Failed	< 1 ms		System.TypeInitializationException : The type initializer for 'System.Net.Security.Tests.NegotiatedCipherSuiteTest' threw an exception. ---- System.ObjectDisposedException : Cannot access a disposed object. Object name: 'SslStream'.
System.Net.Security.Tests Failed (31)	33 sec		
System.Net.Security.Tests Failed (31)	33 sec		
ClientAsyncAuthenticateTest Failed (3)	448 ms		
System.Net.Security.Tests.ClientAsyncAuthenticateTest.ClientAsyncAuthenticate_AllServerVsIndividualClientSupportedProtocols_Success(clientProtocol: Ssl3) Failed	6 ms		System.ComponentModel.Win32Exception : 클라이언트 및 서버에 공용 알고리즘이 없기 때문에 통신할 수 없습니다.
System.Net.Security.Tests.ClientAsyncAuthenticateTest.ClientAsyncAuthenticate_EachSupportedProtocol_Success(protocol: Ssl3) Failed	10 ms		System.ComponentModel.Win32Exception : 클라이언트 및 서버에 공용 알고리즘이 없기 때문에 통신할 수 없습니다.
System.Net.Security.Tests.ClientAsyncAuthenticateTest.ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success(serverProtocol: Ssl3) Failed	432 ms		System.IO.IOException : Authentication failed because the remote party has closed the transport stream.
NegotiateStreamKerberosTest Failed (3)	14 ms		
System.Net.Security.Tests.NegotiateStreamKerberosTest.NegotiateStream_ClientAuthenticationRemote_Fails Failed	1 ms	category [outerloop]	System.InvalidOperationException : The test method expected 1 parameter value, but 0 parameter values were provided.
System.Net.Security.Tests.NegotiateStreamKerberosTest.NegotiateStream_ClientAuthenticationRemote_Success Failed	< 1 ms	category [outerloop]	System.InvalidOperationException : The test method expected 1 parameter value, but 0 parameter values were provided.
System.Net.Security.Tests.NegotiateStreamKerberosTest.NegotiateStream_ServerAuthenticationRemote_Success Failed	13 ms	category [outerloop]	System.NullReferenceException : Object reference not set to an instance of an object.
NegotiatedCipherSuiteTest Failed (18)	1 sec		
System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_AllowOneOnOneSideTls13_Success Failed	2 ms		System.PlatformNotSupportedException : CipherSuitesPolicy is not supported on this platform.
System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_AllowSameTwoOnBothSidesLessPreferredIsTls13_Success Failed	< 1 ms		System.TypeInitializationException : The type initializer for 'System.Net.Security.Tests.NegotiatedCipherSuiteTest' threw an exception. ---- System.ObjectDisposedException : Cannot access a disposed object. Object name: 'SslStream'.
System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_AllowSomeCipherSuitesWithNoEncryptionOption_Fails Failed	< 1 ms		System.TypeInitializationException : The type initializer for 'System.Net.Security.Tests.NegotiatedCipherSuiteTest' threw an exception. ---- System.ObjectDisposedException : Cannot access a disposed object. Object name: 'SslStream'.
System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_AllowTwoOnBothSidesWithNoOverlapNonTls13_Fails Failed	< 1 ms		System.TypeInitializationException : The type initializer for 'System.Net.Security.Tests.NegotiatedCipherSuiteTest' threw an exception. ---- System.ObjectDisposedException : Cannot access a disposed object. Object name: 'SslStream'.
System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_AllowedCipherSuitesIncludesSubsetOfInput_Success Failed	2 ms		System.PlatformNotSupportedException : CipherSuitesPolicy is not supported on this platform.
System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_NothingAllowed_Fails Failed	< 1 ms		System.PlatformNotSupportedException : CipherSuitesPolicy is not supported on this platform.
System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_OnlyNonTls13CipherSuiteAllowedButOtherSideDoesNotAllowIt_Fails Failed	< 1 ms		System.TypeInitializationException : The type initializer for 'System.Net.Security.Tests.NegotiatedCipherSuiteTest' threw an exception. ---- System.ObjectDisposedException : Cannot access a disposed object. Object name: 'SslStream'.
System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_OnlyNonTls13CipherSuitesAllowedButChosenProtocolDoesNotAllowIt_Fails Failed	484 ms		System.TypeInitializationException : The type initializer for 'System.Net.Security.Tests.NegotiatedCipherSuiteTest' threw an exception. ---- System.ObjectDisposedException : Cannot access a disposed object. Object name: 'SslStream'.
System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_OnlyTls13CipherSuiteAllowedButChosenProtocolsDoesNotAllowIt_Fails Failed	1 ms		System.PlatformNotSupportedException : CipherSuitesPolicy is not supported on this platform.
System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_OnlyTls13CipherSuiteAllowedOtherSideDoesNotAllowTls13_Fails Failed	< 1 ms		System.PlatformNotSupportedException : CipherSuitesPolicy is not supported on this platform.
System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_ThreeCipherSuitesWithTwoOverlapping_Success Failed	< 1 ms		System.TypeInitializationException : The type initializer for 'System.Net.Security.Tests.NegotiatedCipherSuiteTest' threw an exception. ---- System.ObjectDisposedException : Cannot access a disposed object. Object name: 'SslStream'.
System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_TwoCipherSuitesWithAllOverlapping_Success Failed	< 1 ms		System.TypeInitializationException : The type initializer for 'System.Net.Security.Tests.NegotiatedCipherSuiteTest' threw an exception. ---- System.ObjectDisposedException : Cannot access a disposed object. Object name: 'SslStream'.
System.Net.Security.Tests.NegotiatedCipherSuiteTest.NegotiatedCipherSuite_SslProtocolIsLowerThanTls13_ShouldMatchTheProtocol Failed (3)	599 ms		
System.Net.Security.Tests.NegotiatedCipherSuiteTest.NegotiatedCipherSuite_SslProtocolIsLowerThanTls13_ShouldMatchTheProtocol(protocol: Tls) Failed	555 ms		System.TypeInitializationException : The type initializer for 'System.Net.Security.Tests.NegotiatedCipherSuiteTest' threw an exception. ---- System.ObjectDisposedException : Cannot access a disposed object. Object name: 'SslStream'.
System.Net.Security.Tests.NegotiatedCipherSuiteTest.NegotiatedCipherSuite_SslProtocolIsLowerThanTls13_ShouldMatchTheProtocol(protocol: Tls11) Failed	21 ms		System.TypeInitializationException : The type initializer for 'System.Net.Security.Tests.NegotiatedCipherSuiteTest' threw an exception. ---- System.ObjectDisposedException : Cannot access a disposed object. Object name: 'SslStream'.
System.Net.Security.Tests.NegotiatedCipherSuiteTest.NegotiatedCipherSuite_SslProtocolIsLowerThanTls13_ShouldMatchTheProtocol(protocol: Tls12) Failed	23 ms		System.TypeInitializationException : The type initializer for 'System.Net.Security.Tests.NegotiatedCipherSuiteTest' threw an exception. ---- System.ObjectDisposedException : Cannot access a disposed object. Object name: 'SslStream'.
System.Net.Security.Tests.NegotiatedCipherSuiteTest.NegotiatedCipherSuite_SslProtocolIsTls13_ShouldBeTls13 Failed	464 ms		System.ObjectDisposedException : Cannot access a disposed object. Object name: 'SslStream'.
System.Net.Security.Tests.NegotiatedCipherSuiteTest.Tls13IsSupported_GetValue_ReturnsTrue Failed	< 1 ms		System.TypeInitializationException : The type initializer for 'System.Net.Security.Tests.NegotiatedCipherSuiteTest' threw an exception. ---- System.ObjectDisposedException : Cannot access a disposed object. Object name: 'SslStream'.
ServerAsyncAuthenticateTest Failed (2)	435 ms		
System.Net.Security.Tests.ServerAsyncAuthenticateTest.ServerAsyncAuthenticate_AllClientVsIndividualServerSupportedProtocols_Success(serverProtocol: Ssl3) Failed	6 ms		System.ComponentModel.Win32Exception : 클라이언트 및 서버에 공용 알고리즘이 없기 때문에 통신할 수 없습니다.
System.Net.Security.Tests.ServerAsyncAuthenticateTest.ServerAsyncAuthenticate_EachSupportedProtocol_Success(protocol: Ssl3) Failed	429 ms		System.ComponentModel.Win32Exception : 클라이언트 및 서버에 공용 알고리즘이 없기 때문에 통신할 수 없습니다.
SslStreamEKUTest Failed (4)	30 sec		
System.Net.Security.Tests.SslStreamEKUTest.SslStream_NoEKUClientAuth_Ok Failed	34 ms		System.Security.Authentication.AuthenticationException : The remote certificate is invalid according to the validation procedure.
System.Net.Security.Tests.SslStreamEKUTest.SslStream_NoEKUServerAuth_Ok Failed	15 sec		System.TimeoutException : Task timed out after 00:00:15
System.Net.Security.Tests.SslStreamEKUTest.SslStream_SelfSignedClientEKUClientAuth_Ok Failed	15 sec		System.TimeoutException : Task timed out after 00:00:15
System.Net.Security.Tests.SslStreamEKUTest.SslStream_ServerEKUClientAuth_Fails Failed	31 ms		Assert.Throws() Failure Expected: typeof(System.IO.IOException) Actual:   typeof(System.Security.Authentication.AuthenticationException): The remote certificate is invalid according to the validation procedure. ---- System.Security.Authentication.AuthenticationException : The remote certificate is invalid according to the validation procedure.
SslStreamNetworkStreamTest Failed (1)	436 ms		
System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_SendReceiveOverNetworkStream_AuthenticationException Failed	436 ms	category [nonfreebsdtests], category [nonnetbsdtests], category [nonosxtests], category [nonwindowstests]	Assert.Contains() Failure Not found: SSL_ERROR_SSL In value:  클라이언트 및 서버에 공용 알고리즘이 없기 때문에 통신할 수 없습니다.

I'm running VS2019 16.5 P1. It's possible that there's other repositories this might be suitable for but I am not entirely sure which one it is, or if this is specific to something on this repo.

[wfurt]

any idea @safern or @ViktorHofer ???

[safern]

@Gnbrkm41 does this apply for ConditionalFact and ConditionalTheory or only for one of them?

[Gnbrkm41]

Both it seems. It also happens for other attributes like PlatformSpecific.

[ViktorHofer]

Yes, that's the expected behavior as VS Test Explorer by default doesn't filter on traits. I started a discussion with vstest some months ago to allow specifying default included/excluded traits via the runsettings file.

[safern]

@ViktorHofer do you want to close this issue then, or open is fine to have it under your radar to track the status of the discussion you're having with VS Test Explorer

[ViktorHofer]

Tracking vstest issue: microsoft/vstest#2273

[ViktorHofer]

Let's keep this open on our side for tracking.

[safern]

Thanks @ViktorHofer -- I assigned to you since you're tracking that work.

[Gnbrkm41]

@ViktorHofer, Has this been addressed? Looks like the tracking issue on the vstest side has been fixed.

[ViktorHofer]

Unfortunately not as we require an updated xunit.runner.visualstudio package: xunit/visualstudio.xunit#205 (comment)

cc @clairernovotny

[clairernovotny]

Sorry about that, just released it:

https://www.nuget.org/packages/xunit.runner.visualstudio/2.4.2

not sure what templates need to be updated to use 2.4.2 instead of 2.4.1