Clean Enlistment Test Failures #23532
Comments
|
The console log should include the error messages from the tests that failed. Is these any more information higher up in the log? |
|
@mellinoe Thanks for taking a look. I posted a gist of the log here. Here are some excerpts:
|
|
There are a few spurious System.Drawing failures in there -- I will try to submit a PR that fixes them. @tarekgh Could you look at the Globalization failure and see if we know about it? @danmosemsft I think he is hitting the issue you described here: dotnet/corefx#21518. Did we file anything about that, or did it get fixed at some point? Out of curiosity, what version of Windows are you running on? |
|
@kingces95 what Windows version you are running on? |
|
Win 10 Enterprise, fully patched. Running inside parallels with a mac host. |
|
@kingces95 if you can give me the exact version? in the command window type "systeminfo" and then give me the line like the following: I am just wondering why this is failing on your machine and not failing in ours? |
|
@tarekgh The console tests are failing on some of my machines too. I think it has something to do with high-dpi and/or having physical monitor attached. If somebody wants to stop by my office, I will be happy to provide repro (I do not think it repros over TS). |
|
@mellinoe I don't see that info (about the console failure) but if it is that, I believe it was a bug in RS2. Not sure about RS1. No, I did not complete the PR dotnet/corefx#21518 - It's just setting console title to a huge string, so IMO it's not worth bothering to fix, the right thing is to adorn this test so it doesn't run on affected Windows versions. |
|
It may even have been particular builds of RS2 - I know it reproed consistently on my home laptop, but never at work. Again, that assumes it was failing for that reason and not some other random console problem. |
|
@jkotas the globalization test failure has nothing to do with the machine configuration in general. do you see the globalization test failing on your machine? |
|
@danmosemsft is right. Windows had a regression in RS2 which they fixed on RS3. The regression caused the framework to fail when setting the console title with long text. I agree it is not worth adding work around in the framework for this one as the problem occurred on RS2 only and happen when setting the Window title to a long string which is not a common scenario at all. |
|
Globalization tests are fine for me. The tests I see failing intermittently or on some machines are |
|
@kingces95 one more ask. could you run "intl.cpl" and then click on "Additional Settings" button and tell me the value of "No. of digits after decimal"? I assume you are running with en-US locale. thanks for your help. |
|
@jkotas if you have RS2 then "WindowAndCursorProps.Title_Set_Windows(lengthOfTitle: 1024) [FAIL]" expected to fail |
|
perhaps @tarekgh you could slap a check for Windows version on that one so it doesn't come up again? I should have done this myself. |
|
@danmosemsft I'll submit a PR for that. |
|
Sure. I'm running It should be easy to reproduce my config: I'm running in a parallels VM on a MacBookPro. Start with a MBP, install latest version of Parallels, the follow the prompts. Parallels will automatically download windows and away you go.. patch, install VS, clone, build, test, bam. Reproduces every time.. |
|
|
|
What tool is everyone using to render the testResults.xml files and\or generate a dashboard of the test results? |
|
@kingces95 I don't think anybody is doing anything like that. |
|
Hm. So how are folks investigating test failures? They're just looking at the raw console output and raw xml xunit output? |
|
Thanks, @kingces95 for sending the OS info. It looks in your case you customized the globalization settings as it looks different than the exact version I am running on. But it is a good thing to fix the test anyway to handle such cases. I'll try to get the globalization tests fixed soon. |
Basically, yeah. Hopefully when we sort out your initial problems and get to a clean state, it is more manageable. Usually the tests are expected to be 100% clean, unless you have changed something (and then the failure should be obvious). Also, it is possible to run individual test projects, which cuts down on some of the verbosity and makes things more manageable, especially when you are only working on one library at a time. |
|
@mellinoe Is there any more information you need about the |
|
@kingces95 Could you print out the failure message from the latest version of the tests in the master branch? I added some logging that should make it clear what the problem is. |
|
And unfortunately, I'm not familiar with the |
I don't know anything about this particular failure. What platform is this on? What version of Windows? |
|
@davidsh Sure. I'm running OS Version: 10.0.15063 N/A Build 15063. It should be easy to reproduce my environment: I'm running in a parallels VM on a MacBookPro. Start with a MBP, install latest version of Parallels, the follow the prompts. Parallels will automatically download windows and away you go.. patch, install VS, clone, build, test, bam. Reproduces every time.. The XML and Console logs from my test run are above. @mellinoe copied what looks to be the relevant portion of the console log above. If you're unable to reproduce it, then I could give you TeamViewer creds to a VM running on my laptop. |
|
cc: @Priya91 @mellinoe
|
|
@Priya91 It's actually running in Windows 10, in Parallels, on a Mac. But, yeah, if you have a Mac then that'll make it much easier to reconstruct the environment. Just install Parallels, let it download windows, install VS, sync, build, run tests. Could very well be that the virtual networking behaves in an unexpected fashion. It'd be great if we could add a mac/parallels run to the CI. |
|
@kingces95 Since you already have the repro environment, can you report what is the output of the following code on the windows vm with .NET framework and .NET Core, using System;
using System.Net;
public class Program
{
public static void Main(string[] args)
{
Console.WriteLine(Dns.GetHostEntryAsync(IPAddress.None).GetAwaiter().GetResult().AddressList[0]);
}
} |
|
@Priya91, they both print out the same thing: VS2015 / Desktop 4.6.1 -> |
|
@kingces95 Could you let me know what the error message is from the System.Drawing.Common tests is, if you are still hitting issues there? I added some logging that will help me diagnose the issue. |
|
Thanks @kingces95 ! @davidsh The same issue repros on .NET Framework as well. Can you please take a look? |
Are you saying the test is failing in CI against .NET Framework Desktop (on a real Windows machine)? Or are you saying that that the test is failing when run against .NET Framework in a Windows container on top of Parallels on a Mac? |
|
@mellinoe I updated the gist links above to the System.Drawing.Common XML as well as the console log during the test run. Hope it helps! |
Yes, the same code, when run against .NET Framework should throw SocketException, but it does not on a windows vm on top of parallels on mac. Below is the repro code used, same as the failing test. using System;
using System.Net;
public class Program
{
public static void Main(string[] args)
{
Console.WriteLine(Dns.GetHostEntryAsync(IPAddress.None).GetAwaiter().GetResult().AddressList[0]);
}
} |
Ok, so the variable here is that the "hardware" is not a real machine but rather the Parallels hypervisor architecture on a Mac. cc: @karelz You'll need a Mac person to troubleshoot this since this is a Mac specific problem causing different behaviors to leak thru into the Windows container. |
Naively I would expect Windows under Parallels on Mac to behave in similar way as in any other virtualization technology (with Windows/Linux host). Ideally it should behave the same way as on physical HW (I assume that's how Windows VM on Windows/Linux host behaves). Given that we do not have unlimited Mac HW, adding new CI legs to run Windows in Mac/Parallels is very costly for us. It also adds complexity to the infrastructure - and we have much higher priority gaps/bugs in infrastructure, so this would be very low pri from that perspective. Overall, it feels like being reactive to discovered differences seems the right thing to do from cost/benefit point of view at this moment. If there is a way to detect we are running under Mac/Parallels virtualization, we should be able to disable/modify the tests in such environment. Is it possible to detect? |
|
Parallels is injecting extra DNS mappings. They maybe unusual, but I would not consider them to be a bug. You can replicate the parallels behavior on regular Windows installation by going to
|
|
Also, adding the |
Interesting. We'll need to study this test and replace "255.255.255.255" with something else that is appropriate for this particular test. |
|
Thanks @jkotas! We should probably split off the Networking test discussion and harden the test more against not-so-common environment settings. |
Based on this discussion, it sounds like these particular tests are invalid. The test was originally created because it was assumed that doing a DNS lookup against IPAddress.None (which is the broadcast address, 255.255.255.255) would always be invalid. Then it was tweaked to ignore the test on MacOS. public static IEnumerable<object[]> GetNoneAddresses()
{
yield return new object[] { IPAddress.None };
}
[PlatformSpecific(~TestPlatforms.OSX)] // macOS will resolve IPAddress.None to broadcasthost and produce a valid listing
[Theory]
[MemberData(nameof(GetNoneAddresses))]
public async Task Dns_GetHostEntryAsync_NoneIPAddress_Fail(IPAddress address)
{
string addressString = address.ToString();
await Assert.ThrowsAnyAsync<SocketException>(() => Dns.GetHostEntryAsync(address));
await Assert.ThrowsAnyAsync<SocketException>(() => Dns.GetHostEntryAsync(addressString));
}
[PlatformSpecific(TestPlatforms.OSX)] // macOS will resolve IPAddress.None to broadcasthost and produce a valid listing
[Theory]
[MemberData(nameof(GetNoneAddresses))]
public async Task Dns_GetHostEntryAsync_NoneIPAddress_Success(IPAddress address)
{
IPHostEntry result = await Dns.GetHostEntryAsync(address);
Assert.NotNull(result);
Assert.NotNull(result.AddressList);
Assert.Equal(1, result.AddressList.Length);
Assert.Equal(address, result.AddressList[0]);
}But in reality, since this IP address could have a DNS mapping on any platform via hard-coding it in the hosts file, this test should be removed. I will submit a PR to take care of it. |
The Dns_GetHostEntryAsync_* are fundamentially invalid because it's possible for the broadcast address, 255.255.255.255, to have an DNS mapping via manually modifying the hosts file. This was actually happening on Mac systems as well as an virtual environment running on top of that (i.e. Windows on Parallels). Ref: https://github.com/dotnet/corefx/issues/23992#issuecomment-330250642 Contributes to #23992
|
Woot! Everything is fixed!
@mellinoe I'd close this issue out except I'm not sure which change fixed the drawing issue so I'm not sure if it's actually fixed or just a sporadic error and I happened to get lucky this time. |
|
@kingces95 Glad to hear it! Drawing was fixed by dotnet/corefx#24009 and dotnet/corefx#24097. The problem in both is that the tests were making assumptions about what kind of printer you had installed, basically, and would fail if something was different than expected. |
* Microsoft.ServiceModel.Syndication skeleton project * Adding the existing classes of SyndicationFeed from .net fx * Added the needed references to get the code to compile * Changed some namespaces * Fixed errors when reading feeds and replaced some buffers * Cleaning code for PR * Deleted some unused files * Added the posibility that the user creates his own date parser when reading a feed from outside, also as part of our default date parser, if it can't parse the date it will just assign a default date to avoid the crash * Added correct testnames and Copyright * Initial changes to add custom parsers * Added more delegates as parsers * Added custom parser delegates * Save changes * Initial SyndicationFeed * Removed the dependence of the old SR class * Cleaned the code, deleted Diagnostics, fixed throwing resources * Cleaned code from most of the unnecesary comments * Formatted the code with CodeFormatter Tool * Moved the call of itemParser to be called with each item * Test using custom parsers * Fixed image issue where image tag was using original title and url, for RSS formatting Image issue fixed and added some tests * Test clase jorge * Save changes with Jorge cass * Initial Jorge * saving changes * Save changes * Fixed image and items issue * Fixed disjoint items in atom * Run codeFormatter * Adding parsing to optional elements int the spec added unittesting * Added Icon parsing to Atom10 feedFormatter. Added unit test * Adding Async Writing RssFormater writes new optional spec elements * Added Icon writing on Arom Writer * Fixed some warnings * Improved custom parsing for RSS * Added custom parsing for Atom feed formatter, added test * added nameof() to all exceptions when needed. * Adding Extension Methods for XmlReader * Fixing code for PR * Fixing code for PR * Added check for skip days to allow only accepted days. * Improved flexibility for Default dateparser * Add wrong dates example * Fixing code review * Fixed warnings on some unawaited methods. * Added async extension methods for XmlReader * Add XmlWriterWrapper method extensions * Changed ReadCategoryFromAsync to return a SyndicationCategory * Fixed sync method exposed GetReader. * Edited XmlReaderWrapper, moved methods to extension methods. * Removed unnecesary variables from Wrappers. * Fixed bug from ServiceModel.Syndication * Make BigInteger ctor opt path for == int.MinValue reachable. The BigInteger(ReadOnlySpan<byte> value) ctor contains an optimised path for value being larger than 4 bytes and the result being equal to int.MinValue. However this path is inside a test that precludes that value, so can never be reached. Restructure so that the path is reachable. * Fix ServiceModel.SyndicationFeed project dependencies - Change M.SM.SyndicationFeed to be a .NET Standard 2.0 library - Change tests to use official .NET Core 2.0 release from preview * Add btrfs and other missing file system types (dotnet#24102) * Uppercase * Add more filesystems from stat.c. Change to stat names where possible * Add some more local friendly names * Add some more remote file types from stat.c * Add entry to switch present in enum * Build errors * Remove Fixed entries that should be RAM * comment * Change case of 0X to 0x * Move cramfs to Fixed * ConcurrentQueue 128-byte cache line (dotnet#22724) ConcurrentQueue 128-byte cache line * Add warning by default in SGEN (dotnet#24054) * Output warning by default if run the tool directly without /quiet parameter. * add quiet parameter in the command. * fix parameter error. * Update the warning. * Add the target to copy the serializer to publish folder. (dotnet#24096) * Wrap cert callback leaked exceptions in WinHttpHandler (dotnet#24107) Similar to the fix done for CurlHandler (dotnet#21938), fix WinHttpHandler so that leaked exceptions from the user-provided certificate callback are properly wrapped. The ManagedHandler still needs to be fixed since it is not wrapping properly. Contributes to #21904 * Ensure ProcessModule for main executable is first in modules list (dotnet#24106) * Disable NegotiateStreamTest fixture for distros without working Kerberos (dotnet#24098) Disable NegotiateStreamTest fixture entirely because setup-kdc.sh is broken on some distros * Expose and add tests for Guid.ctor(Span)/TryWriteBytes/TryFormat * Address remaining PR feedback * #24112 Replaced documentation summary of TryPop with text similar to TryDequeue. (dotnet#24113) * Wrap exceptions from ManagedHandler's server validation callback (dotnet#24111) To match other handlers' behaviors. * Fix libgdiplus function loading on OSX. * Prevent ServiceControllerTests from dispose when already disposed (dotnet#24042) * Disable ServiceProcessTest that has been failing on CI and official builds * Prevent dispose from been called when already disposed * Add logging to repro if RemoveService is been called twice on the same ServiceController * Data-annotations length fix (dotnet#24101) * Updated in MinLengthAttribute and MaxLengthAttribute to support ICollection<T> * Added tests * Fixed typo * Trying to address two failing checks: - Linux x64 Release Build - UWP CoreCLR x64 Debug Build * Implemented changes requested in review - Extracted Count checking to an external helper to obey DRY - Removed dependency of ICollection<T> and changed to simple reflection Count property lookup * Added requested tests * Added catch for MissingMetadataException. * Extracted code from try-catch. * Added comment as requested. * Typo correction * Remove System.Drawing.Common from the netfx compat package (temporary). * Updating corefx to use new roslyn compiler (dotnet#24076) * Updating corefx to use new roslyn compiler * Updating to new version of the compiler and use the switch so tests pass on desktop * Fix System.Reflection.Metadata tests * Stop running Math.Clamp tests on UAP as API is not there (dotnet#24125) * Stop running Math.Clamp tests on UAP as API is not there * Disable only for AOT * Fix instantiating PrincipleContext with Domain type (dotnet#24122) * Fix instantiating PrincipleContext with Domain type * Enhancement * Disable Test_Write_Metric_eventListener on UWP. (dotnet#24127) * Revert "Remove System.Drawing.Common from the netfx compat package (temporary)." This reverts commit f6b0fbd. * Update BuildTools, CoreClr, CoreFx, CoreSetup, Standard to prerelease-02019-01, preview1-25718-02, preview1-25718-03, preview1-25717-02, preview1-25718-01, respectively (dotnet#24075) * Fixed compile warning/error on FreeBSD (dotnet#24141) * Marking {ReadOnly}Span as readonly structs (dotnet#23908) * Marking {ReadOnly}Span as readonly structs, fixing issue #23809 * Adding readonly attributes to reference assemblies. * Using "readonly ref" keyword instead of attributes. * Adding a LangVersion 7.2 property * System.Drawing: Throw ArgumentNullException on Unix as well (dotnet#24140) * Throw ArgumentNullException when stream is null * Throw ArgumentNullException when stream is null * Remove invalid NameResolution tests (dotnet#24147) The Dns_GetHostEntryAsync_* are fundamentially invalid because it's possible for the broadcast address, 255.255.255.255, to have an DNS mapping via manually modifying the hosts file. This was actually happening on Mac systems as well as an virtual environment running on top of that (i.e. Windows on Parallels). Ref: https://github.com/dotnet/corefx/issues/23992#issuecomment-330250642 Contributes to #23992 * Bump system.drawing.common.testdata to 1.0.6 * Update BuildTools to prerelease-02019-02 (dotnet#24146) * Fix path to test data * Fix whitespace * Add GraphicsTests based on Mono's test suite. * Consolidate more code in the "System.Drawing" namespace. * Remove all remaining Win32 codepaths from the mono codebase. All of this code now implicitly assumes that it will be run on a Unix platform. * Consolidate the rest of the gdipFunctions.cs file into Gdip.cs and GdipNative.Unix.cs * Consolidate the GraphicsUnit and ImageType enumerations -- they were duplicated. * Remove the mono Status enum and use the Windows constants instead in all Unix code. * Move all files into the regular directory structure. Suffix them with ".Unix" and ".Windows" when there are collisions. * Tiny bit of code cleanup * Add conditionals for recent versions of mono * Remove duplicate tests. * Fix multiplying TextureBrush with a disposed matrix on Unix (dotnet#24109) * Fix multiplying TextureBrush with a disposed matrix on Unix * Enable another passing test * Fix accidentally committed file * Add an error code fixup to Bitmap.Unix.cs to match Windows. * Bump System.Drawing.Common.TestData to 1.0.6 (dotnet#24149) * Bump system.drawing.common.testdata to 1.0.6 * Fix path to test data * Fix whitespace * Cleanup - Add/simplify using statements of disposable resources (Graphics, Bitmap) * Validate HatchStyle passed to HatchBrush ctor * Delete accidentally duplicated HatchBrush tests in LinearGradientBrushTests * Remove references to historical Mono bug IDs * Renable some already passing tests * Remove Thread.Sleep workaround * Update BuildTools to prerelease-02020-01 (dotnet#24172) * Add thread-local based switch to opt-in to ManagedHandler * Address PR feedback * PR feedback * Fix memory map imports (dotnet#24176) * Fix memory map imports Imports lost the last error attribute. Add it back and change the results to be the more correct "bool". Tweak the usage based on the new return type. #24159 * Move spin wait * Remove unused FEATURE_RANDOMIZED_STRING_HASHING (dotnet#24178) * Adding System.Data.Odbc package and including in metapackage * Fix MultiplyTransform with a disposed brush * Fix for 2nd Connection hangs in Mirroring Environment (dotnet#24181) * Switch tests to use thread-local switch for ManagedHandler And as a result re-enable parallelism of the test suite, which on my machine reduces the running time of the outerloop tests from 150s to 45s. * Update CoreClr, CoreSetup, ProjectNTfs, ProjectNTfsTestILC, Standard to preview1-25720-03, preview1-25719-04, beta-25721-01, beta-25721-01, preview1-25721-01, respectively * CoreFx #22406 Span based APIs - Text Reader Writer (dotnet#23786) * [Drawing] Move remaining "Unix" files into regular directory structure * As part of this, the "Windows" version of ImageFormat.cs is now being used everywhere. This file just contains a bunch of GUID's that are not platform-specific in any way. * Change AsSpan to property Span and rename AsMemory to Memory * Add Metafile and MetaHeader tests based on Mono's System.Drawing unit tests * Update project file * MetafileTests: Remove duplicates, and re-enable tests on Unix which are now working. Delete duplicate MetaHeadertests * Rationalize using statements * Update project file * Fix Metafile exception behavior * Simplify tests, remove duplicate test * Don't catch an ArgumentException just to get a NullReferenceException instead. * Assert.False(Object.ReferenceEquals => Assert.NotSame * PR feedback * Remove duplicate tests, fix typo * Remove X11 dependency for tests which are enabled on Unix. * Workaround libgdiplus glitches Don't test metafileHeader.Bounds when using libgdiplus. Don't test metafile.GetBounds on Unix, force MetafileHeader.Bounds to return an empty rectangle when MetafileSize == 0 Don't validate graphicsUnit on Unix. * Move Syndication to root/src * Disable Build of S.SM.Syndication. * Increase file descriptor limit in S.N.Http tests on macOS * Expose/test String.Create span-based method (dotnet#23872) * Expose/test String.Create span-based method * Address PR feedback * Update build to clang/llvm 3.9 (dotnet#24177) Update scripts, docs and build pipeline docker images to clang/llvm/lldb 3.9 * Update ProjectNTfs, ProjectNTfsTestILC, Standard to beta-25722-00, beta-25722-00, preview1-25722-01, respectively (dotnet#24207) * Remove the line that will copy the generated serializer to the pack. (dotnet#24199) * Revert "Update build to clang/llvm 3.9 (dotnet#24177)" This reverts commit 21e008a. * Remove stale SetStateMachine call in test * Ssl Stream Async Write (dotnet#23715) * Change from APM to Async/Await for write side * Added nameof * Reacting to review * Reacting to review * SSLStream : Fixed spelling mistake in file name (extra a) (dotnet#24221) * Fixed spelling mistake in file name * Fix csproj * Update ProjectNTfs, ProjectNTfsTestILC to beta-25723-00, beta-25723-00, respectively (dotnet#24222)
ghost
locked as resolved and limited conversation to collaborators
After installing Windows 10 on a Parallels VM (on a MacPro), installing VS 2015, cloning, then running
build, andbuild-testsfrom aVS 2015 Developer Command PromptI consistently get the following errors:Verified fixed by dotnet/corefx#24052
Verified fixed by dotnet/corefx#24016
Verified fixed by dotnet/corefx#24147
Verified fixed by dotnet/corefx#24009 and dotnet/corefx#24097
The console log.
The text was updated successfully, but these errors were encountered: