Visual Studio Bug: Clicking on test result sometimes kills VS2019

[bwandersen]

Hi

We are using ::testing::RegisterTest to dynamically register test cases. The cases show up in the Test Explorer and can be run from here without any issue. However, if you click on the test result, then Visual Studio gets stuck and has to be killed from the task manager. This works fine in Visual Studio 16.9, but no longer works in 16.10.

Now the strange part is that on machines with Resharper enabled, this issue goes away. Disabling Resharper makes the problem appear again. Our guess is the recent changes to the test explorer has introduced some minor discrepancy between google test adapter and the vs test explorer.

Does anyone know what the problem might be?

Here is an example that reproduces the problem:

`#include "stdafx.h"
#include <gtest/gtest.h>

#ifdef _MSC_VER
// Windows service specifics
#define WIN32_LEAN_AND_MEAN
#define NOMINMAX 1
#include <windows.h>
#endif

#include

using namespace std::string_literals;
using ::testing::ValuesIn;
using ::testing::Values;
using ::testing::Return;
using ::testing::An;
using ::testing::_;

class klXlTestRunner : public testing::Test
{
protected:

static void SetUpTestSuite() 
{
}

static void TearDownTestSuite() 
{
}

void SetUp() override
{
}

void TearDown() override
{

}

};

class XlTest : public klXlTestRunner {
public:
explicit XlTest(int ) : data(_) {}
void TestBody() override
{
}

private:
int data_;
};

bool RegisterXlTests()
{

std::string suite("suite");
std::string name("name");

::testing::RegisterTest(suite.c_str(),
	name.c_str(),
	nullptr,
	nullptr,
	__FILE__,
	__LINE__,
	[=]() -> klXlTestRunner* { return new XlTest(0); });

return true;

}

GTEST_API_ int main(int argc, char** argv) {
printf("Running main() from %s\n", FILE);
if (RegisterXlTests())
{
testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}
return 0;
}`

[csoltenborn]

Sorry for the late answer, and thanks for the code to reproduce the issue! I assume that with "click on the test result" you mean selecting the test in the test explorer to see the test result below!?

I am on VS 16.10.3 right now. Unfortunately, your code does not compile out of the box. However, the SampleTests solution also contains dynamically registered tests. With those, I can not reproduce the issue, neither with ReSharper enabled nor disabled.

Could you try on your machine? And if the issue does not occur (my tests are registered differently), could you try to integrate your code into the Test project of that solution, check if the problem remains, and then attach the resulting code?

[bwandersen]

Thanks for taking the time to reply. Much appreciated

I assume that with "click on the test result" you mean selecting the test in the test explorer to see the test result below!?

Yes, correct

I am on VS 16.10.3 right now. Unfortunately, your code does not compile out of the box. However, the SampleTests solution also contains dynamically registered tests. With those, I can not reproduce the issue, neither with ReSharper enabled nor disabled. Could you try on your machine?

I build SampleTests solution and looked the Api.Created.Test you refer to. For some reason it will only show the last of the two testcases in the test explorer. You can switch them around, and then you will get the other one. In any case, I took the code above and modified it to run against 1.8.1 (we use 1.10). That worked nicely. The result can be "clicked" and the test case can run (regardless of resharper is on or off). I will try to use the style in the test case in our solution and see how it goes.

If interested, here is the h-file. It needs to be included in Tests project.
DynamicallyRegisteredTest.txt

The function to register the test cases needs to be added to main.cpp
Main.txt

I had change the extension .txt.

Thank you, Bjarke

[LPhil]

I have the same Problem with VS 16.10.3. If you try to access the source over the test explorer, VS stops and only the Taskmanger you help by kill the process away. I first noticed this behavior in 16.10.3.

[csoltenborn]

@bwandersen

I build SampleTests solution and looked the Api.Created.Test you refer to. For some reason it will only show the last of the two testcases in the test explorer. You can switch them around, and then you will get the other one.

I'm not sure what you mean by that. I must admit :-)

In any case, I took the code above and modified it to run against 1.8.1 (we use 1.10). That worked nicely. The result can be "clicked" and the test case can run (regardless of resharper is on or off). I will try to use the style in the test case in our solution and see how it goes.

Note that I'm not sure at all whether this is an appropriate way to create and register tests - I guess I picked up code somewhere just to have something to write tests against. At first sight, your approach seems cleaner (since mine e.g. uses internal API)...

If interested, here is the h-file. It needs to be included in Tests project.
DynamicallyRegisteredTest.txt

The function to register the test cases needs to be added to main.cpp
Main.txt

Yes, I tried the code, and it appears to work just fine - thanks for attaching! That leaves me with a case I cannot reproduce locally, though :-)

[csoltenborn]

@LPhil Is this also true for dynamically created tests in your case, or also other tests? What gtest version are you on? Can you maybe provide some code to reproduce the example, preferably as an extension of the SampleTests solution?

[bwandersen]

I build SampleTests solution and looked the Api.Created.Test you refer to. For some reason it will only show the last of the two testcases in the test explorer. You can switch them around, and then you will get the other one.

I'm not sure what you mean by that. I must admit :-)

There are two test cases in Api.Created.Test, and only one of them shows up in the test explorer. You can change which one shows up by making it the last case in the Api.Created.Test cpp file .. but not it is not important. Sry for the confusion.

In any case, I took the code above and modified it to run against 1.8.1 (we use 1.10). That worked nicely. The result can be "clicked" and the test case can run (regardless of resharper is on or off). I will try to use the style in the test case in our solution and see how it goes.

Note that I'm not sure at all whether this is an appropriate way to create and register tests - I guess I picked up code somewhere just to have something to write tests against. At first sight, your approach seems cleaner (since mine e.g. uses internal API)

I think RegsiterTest is an addition they did for 1.10+. It does almost the same the code in the test case. My code expanded on the example from their documentation Registering tests programmatically. It tried copying the example, and that still has the bug. Resharper still removes the bug when enabled.

If interested, here is the h-file. It needs to be included in Tests project.
DynamicallyRegisteredTest.txt
The function to register the test cases needs to be added to main.cpp
Main.txt

Yes, I tried the code, and it appears to work just fine - thanks for attaching! That leaves me with a case I cannot reproduce locally, though :-)

Yes, sry, but at least it shows that 1.8.1 works. It points to problem between 1.10 and VS 16.10.1-3. I will keep trying to get it working for 1.10, and let you know how it turns out.

[daveabramson]

I have the same issue where selecting the test in test explorer causes VS to hang forever, but it seems to be triggered differently than the dynamically registered test scenario above.

1. Tests defined using TEST do not cause VS to hang when selected in the test explorer
2. Tests defined using TEST_F cause VS to hang when selected in the test explorer

Posted this issue with devcommunity (https://developercommunity.visualstudio.com/t/visual-studio-freezes-when-selecting-test-in-test/1499235?from=email). They asked me to update to 16.11.0, which I did. On first startup after update I got the follow error, that I was not getting on my previous version (16.10.2):

And this message in the tests output pane:

The next time I started after the update, I only got the message in the test output pane.

Also, the same behavior regarding selection of tests in test explorer still exists.

I hope this is useful information.

[csoltenborn]

Thanks! This is indeed useful information, I would say, but probably more for the VS guys then for me - this is almost certainly a VS regression since there have been no changes to GTA for quite some time, and since it also appears to happen for TAfGT, the MS maintained fork of GTA.

Basically, GTA just reports the tests found (and then the tests' results) to the VS test framework - GTA has nothing to do with the Test Explorer. The only thing I could think of is that for a particular kind of tests, some fields of the "report objects" are filled with values which make VS crash recently. This would still be a VS regression, but I might have some influence on that. However, since I was not yet able to reproduce the issue at all, it's probably rather hard for me to figure that one out.

My suggestion would be to also report this at TAfGT's repository or the repository of the test framework itself, and to point to this issue from there...

[luke727]

I also experienced VS2019 locking up while clicking on an individual test in the text explorer. I was able to work around it by changing the Linker -> Debugging -> Generate Debug Info value from /DEBUG to /DEBUGFULL. Hope this helps anyone else running into the same issue.

[csoltenborn]

@luke727 That's interesting, thanks for sharing!

@LPhil @bwandersen @daveabramson Can you confirm this by any chance? I would then add this to the trouble shooting section of the docs (and I would also check whether I can see anything in the code which might cause this).

[bwandersen]

I just checked and the project already had /DEBUG:FULL on. So this sadly does not help on VS 16.10.2. I am going to upgrade to VS 16.11.4 tomorrow. One of my colleagues upgraded to this version and doesn't have the problem. I will let you know how it turns out.

[luke727]

I'm on VS 16.11.3 if that makes any difference.

[csoltenborn]

@bwandersen Any news from your side? Just being curious...

[daveabramson]

I don't have the debug FULL option in my projects

I'm doing a cross compilation.

A colleague of mine found that if we removed out tests from namespaces the freezing issue went away. So that is what we did. It wasn't TEST_F as we initially thought, it just so happened that the TEST_F tests where in namespaces and the TEST tests were not.

[bwandersen]

@csoltenborn Sorry for the late answer. We had a big project going live, and it left little time for other things. So, I tried v. 16.11.4 and v16.11.6. The problem still persists, and ReSharper still "fixes" the issue.

@daveabramson Our test cases are not enclosed in a namespace, and I think that is what you are referring to. Yes?

[petebannister]

I'm getting a similar problem of VS freezing when clicking on a test. Which means you can't right-click to debug a test. However I am only getting this problem in some projects, not all.

If the project is a 'classical' sln/vcxproj then it appears to be working fine.

If the project was built with cmake then it is not. I'm trying to figure out what is different here - it could be a lot of things. The googletest version might be different since in the cmake case it has come from vcpkg but in the sln case it came from a forked conan recipe.

[petebannister]

In the cmake case, it is using gtest_discover_tests in the CMakeLists.txt and looking at the output in DebugView is like this:

[9008] GEMS TEST: C:/work/unit-test-example/build.vs/x64-Debug/bin/Debug/UnitTestExampleTest.exe
[33708] TableControl: QueueUpdate 18/01/2022 10:48:24:159 
[33708] TableControl: Updating entries 18/01/2022 10:48:24:268 
[33708] TableControl: Updating entries (UI thread) 18/01/2022 10:48:24:455 
[33708] Informational: ========== Starting test discovery ========== 
[33708] Informational: ========== Test discovery finished: 3 Tests found in 976.4 ms ========== 
[33708] TableControl: QueueUpdate 18/01/2022 10:48:29:325 
[33708] TableControl: Updating entries 18/01/2022 10:48:29:435 
[33708] TableControl: Refreshing UI 18/01/2022 10:48:29:442 

For the sln case the output is quite different:

[5896] Informational: ========== Starting test discovery ========== 
[52048] DBGHELP: Symbol Search Path: . 
[52048] DBGHELP: C:\work\MyTest\_build\VC\Win32\Debug\MyTest.exe - OK 
[52048] DBGHELP: .\MyTest.pdb - file not found 
[52048] DBGHELP: .\symbols\exe\MyTest.pdb - file not found 
[52048] DBGHELP: MyTest - private symbols & lines  
[52048]         C:\work\MyTest\_build\VC\Win32\Debug\MyTest.pdb 
[5896] Warning: Could not locate debug symbols for 'C:\work\MyTest\_build\VC\Win32\Debug\MyTest.exe'. To make use of '--list_content' discovery, ensure that debug symbols are available or make use of '<ForceListContent>' via a .runsettings file. 
[23756] TEST: C:\work\MyTest\_build\VC\Win32\Debug\MyTest.exe
[5896] Informational: Found 283 tests in executable C:\work\MyTest\_build\VC\Win32\Debug\MyTest.exe 
[5896] Informational: ========== Test discovery finished: 283 Tests found in 21.2 sec ========== 

[petebannister]

VS also freezes clicking on a test that using TEST instead of TEST_F (from the cmake based project).

[mychmcg]

I'm experiencing this issue when clicking on individual parameterized tests, but only when they are given a name generator with INSTANTIATE_TEST_CASE_P. It doesn't seem like INSTANTIATE_TEST_CASE_P is supposed to support a name generator arg, but the test names are correctly being generated in the explorer window and the tests run properly. Still, when clicking an individual param test it hangs before it can update the Test Detail Summary window.

Also apparently Google Test projects in VS 16.11.8 don't recognize INSTANTIATE_TEST_SUITE_P.... so using the correct macro is not an option.

[Knitschi]

I have the same Problem. We use typed test suites and instantiate them with INSTANTIATE_TYPED_TEST_SUITE_P(). The test cases that are created in this way freeze Visual Studio when clicking on them. But the Problem does not happen all the time. Yesterday I it worked for a few ours, than I must have made some change that broke it, because all of a sudden it did no longer work.

I am on VS 16.10.4

[csoltenborn]

I have pinned this issue now because the problem appears to not go away, but I'm almost certain that this not a GTA problem (explanation in the discussion). I strongly suggest that you talk to the VS guys about this.

[Knitschi]

I will try to create a minimalistic reproducer this weekend.

[csoltenborn]

Feel free to do so, but note that a) there has been an attempt before, but I still couldn't reproduce the issue (see discussion), b) I'm rather sure that this is a VS issue, so even if I'm able to, it's unclear whether I can fix it at all, c) I will not be able to spend a lot of time on this at the moment, so even if I can fix it in theory, it must be pretty easy to do so.

Not trying to stop you, but I think that there's a rather high probability that your attempts will not help after all...

[Knitschi]

I created a little CMake based projects. The instructions how to build it and reproduce the bug are in the README.

https://github.com/Knitschi/GoogleTestAdapterIssue328

The problem does not appear when you generate and open the solution for Visual Studio 17 2022 and use the version of google test adapter that is shipped with Visual Studio 2022.

[csoltenborn]

Thanks for your great code example - took me like 3mins to get his up and running :-)

And indeed - I have seen my first freeze. However, it's not that simple:

• I was on VS 16.11.9 first - could click all tests, no freezing (before and after running the tests)
• updated VS to 16.11.11, rebuilt the solution - saw the freeze, but only after clicking the third test
• restarted VS, rebuilt the solution - no freeze after selecting the tests for several times

This makes me even more sure that this is a VS issue, and that I can't do anything about it (since the tests appear to be rather similar, so it's probably not going to help to change some of the test cases' fields).

To quickly repeat why I'm so convinced, let me reiterate how GTA (or the VS Test framework) works:

• if the tests are built, my code is called and passed the changed executables.
• GTA then scans the executables, identifies the tests contained in them, and reports these to VS
• Now, things happen with these tests that are beyond my knowledge (e.g., they are displayed in the Test Explorer), but GTA is not at all involved in this (no GTA code is executed after GTA reported the tests)
• This changes as soon as tests are run from the test explorer. Depending on the run option (basically "Run all" vs. "Run selection", one of two GTA methods is called (executables, test cases)
• GTA runs the according tests, parses the test output, and reports the test results to VS
• Now, things happen with the results (e.g., they are displayed in the Test Explorer), but once more, GTA is not all involved in this

TL;DR: GTA is not involved in the Test Explorer at all, despite reporting the tests to VS (which are then displayed in the Test Explorer). Selecting tests in the Test Explorer does not execute GTA code. Since selection of tests does not always make VS crash, I think it's very unlikely that the way GTA reports tests to VS is broken in any way (and could be fixed to report the "VS freezes" issue).

I will now rename the issue slightly and not spend time on it in the future (unless I see VERY convincing arguments to do so)...