Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support VS 2019 #17

Open
twenzel opened this issue Mar 14, 2019 · 48 comments
Open

Support VS 2019 #17

twenzel opened this issue Mar 14, 2019 · 48 comments
Labels
enhancement New feature or request

Comments

@twenzel
Copy link

twenzel commented Mar 14, 2019

Please add support for Visual Studio 2019.
@p10tyr
Copy link
Owner

p10tyr commented Mar 14, 2019 •
edited
Loading

It should work to be honest. Did you try?

-EDIT
You can install it but it wont work. Sorry. when I get some free time I will try and get to it.

@p10tyr
Copy link
Owner

p10tyr commented Mar 14, 2019 •
edited
Loading

unless the installer is locked to vs2017.. otherwise its just a config thing in the installer.
ill check

@twenzel
Copy link
Author

twenzel commented Mar 14, 2019

That's the problem, the installer is locked to VS2017 and the Extension gallery within VS2019 does not list your extension.

@p10tyr
Copy link
Owner

p10tyr commented Mar 26, 2019 •
edited
Loading

I installed VS2019 and tested it. There was an issue on installer with dependancy
Please use 1.16.4 as there was some dependency that I did not need and is not installed in VS2019 anyway.

@p10tyr p10tyr added the enhancement New feature or request label Mar 26, 2019
@bobted
Copy link

bobted commented Apr 9, 2019

I have 1.16.4 installed in 2019 Pro, but get the below error when running MSTest unit tests.

System.TypeAccessException: Attempt by method 'PrestoCoverage.PrestoCoverageCore.OnTestExecutionStarting(System.Object, Microsoft.VisualStudio.TestWindow.Extensibility.OperationStateChangedEventArgs)' to access type 'Microsoft.VisualStudio.TestWindow.Controller.TestRunConfiguration' failed.
   at PrestoCoverage.PrestoCoverageCore.OnTestExecutionStarting(Object sender, OperationStateChangedEventArgs stateArgs) in C:\Projects\PrestoCoverage\PrestoCoverage\PrestoCoverage\PrestoCoverageCore.cs:line 66
   at PrestoCoverage.PrestoCoverageContainerDiscoverer.OperationState_StateChanged(Object sender, OperationStateChangedEventArgs e) in C:\Projects\PrestoCoverage\PrestoCoverage\PrestoCoverage\PrestoCoverageContainerDiscoverer.cs:line 73
   at System.EventHandler`1.Invoke(Object sender, TEventArgs e)
   at Microsoft.VisualStudio.TestWindow.Controller.OperationData.OnStateChanged(OperationStateChangedEventArgs e)
   at Microsoft.VisualStudio.TestWindow.Controller.OperationData.Microsoft.VisualStudio.TestWindow.Controller.IOperationData.OnRequestStateChanged(Object sender, RequestStateChangedEventArgs e)
   at Microsoft.VisualStudio.TestWindow.Controller.Request.SetState(RequestStates newState

Hopefully this helps.

@p10tyr
Copy link
Owner

p10tyr commented Apr 10, 2019

I am looking into this. I noticed this after installing but didnt get round to it yet.

@replaysMike
Copy link

replaysMike commented Apr 30, 2019 •
edited
Loading

I too get this exact issue. Using 1.16.4 on VS 2019 16.0.3. Unfortunate as I was hoping to use this.

@replaysMike
Copy link

I should note for others as well, you actually need to uninstall the extension to be able to run tests again as disabling it didn't affect the error.

@p10tyr
Copy link
Owner

p10tyr commented May 1, 2019

Sorry guys... Something has changed in VS2019 and I am not sure when I will have time to sort it out.

@p10tyr
Copy link
Owner

p10tyr commented May 9, 2019 •
edited
Loading

Well I found the problem.. They changed the TestWindows dll in VS2019 so when you try and run a test I try to use reflection to get some information about what is going on with the tests.

Why they did is beyond me but I am too stupid to work out how to get this information using the "Extensions way" because there is no docs anywhere about this. I got this working in 2017 thanks to hours of debugging Visual Studio in Visual Studio and trying all sorts of magic out.

The thing I was using in VS2017, the DLL I had to manually reference had a public sealed class and the 2019 version is internal sealed which prevents me from casting to this type. I can probably go deeper into reflection and get this... but for the love of god.. why?!

#region Assembly Microsoft.VisualStudio.TestWindow.Core, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
// C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Common7\IDE\CommonExtensions\Microsoft\TestWindow\Microsoft.VisualStudio.TestWindow.Core.dll
#endregion

namespace Microsoft.VisualStudio.TestWindow.Controller
{
    **public** sealed class TestRunConfiguration : RequestConfiguration, ITestRunRequestConfiguration, TestPlatform.ObjectModel.Client.ITestRunConfiguration, Controller.ITestRunConfigurationProvider
    {

#region Assembly Microsoft.VisualStudio.TestWindow.Core, Version=16.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
// C:\Program Files (x86)\Microsoft Visual Studio\2019\Preview\Common7\IDE\CommonExtensions\Microsoft\TestWindow\Microsoft.VisualStudio.TestWindow.Core.dll
#endregion

namespace Microsoft.VisualStudio.TestWindow.Controller
{
    **internal** sealed class TestRunConfiguration :

So this is why we are getting the error message ... access type 'Microsoft.VisualStudio.TestWindow.Controller.TestRunConfiguration' failed.

@p10tyr
Copy link
Owner

p10tyr commented May 9, 2019 •
edited
Loading

Well digging in a bit deeper everything got changed to internal.

If I was conspiracy theorists I would say that one of the Microsoft guys from the Coverlet git saw this and the only reason to close this up to keep this kind of feature for Enterprise users only. But I have no proof of that, I am tired and pissed about this and just speculating. So sorry guys....

No VS2019 support!

@replaysMike
Copy link

Ah man don’t you hate when they do stuff like this? I wonder if it was to prevent not having Enterprise to get these stats. I wonder how they do it in DotCover?

@p10tyr
Copy link
Owner

p10tyr commented May 10, 2019 •
edited
Loading

Well if I play the conspiracy card I have to be fair and play the good guy theory card too.

I believe VS2019 is going to force all extensions to run async and they are probably locking down sync access like this down. The thing is the plugin and coverlet run under the VS Thread, luckily its not a massive issue because it doesn't slow down usage while writing code.. and when we run tests we usually go make a brew so nobody notices a minor speed slowdown.

So really.. I have to redo everything to be async and it would be nice if dependency injection worked from VS into extensions (which it doesnt) And all the shit I have seen is still COM hooks, weird export attributes and then my attempt with meta programming.

I have actually stared at the DotCover (and others) source code for hours. I can hardly make head and tails of it but I have literally nicked their VisualStudioExtensions file to help with a few things.

But I just cannot work out how they hook into the Test Window. Usually when I struggle with these things its because I am doing it wrong. But I cannot find any good useful information about Extension developing (like not the shitty examples that avoid using anything useful from VS) 😢

So why do I need the test window?
A private object "Configuration" contains the following information about the test runner.

  1. Is the debugger attached?
    1. If yes then do not attach Coverlet because coverlet modifes part of the binary and causes debugging to fail
  2. Which Test projects are being run?
    1. and iterate over them to give me the exact path of the DLL so I can tell Coverlet what to use

I have spent lots of time trying to get that information from the IDE Interfaces but its just not available. The only reason I found the Configuration object was because I installed Rosyln Analysers that expose private and polymorphic things while debugging... and I spent a lot of time in that shitty little inspector window clicking into layers of properties trying to find something useful. And now they took it away.

@p10tyr
Copy link
Owner

p10tyr commented May 10, 2019

I wonder how they do it in DotCover?

I replied before I was staring at the DotCover source code... there is no source code.. I got confused with another open source project AxoCover I think.

The way DotCover does it is by running its own TestRunner out of process. They gather the data then from their own internal gubbins.

@dozer75
Copy link

dozer75 commented May 11, 2019

Sad thing it didn't work in 2019 since it looks to be a simple and handy extension, but could you remove 2019 from supported versions on the marketplace until this is solved? So people doesn't get confused?

@guillaume86
Copy link

Do you need TestWindow.dll for the basic watch coverage.json files and color gutters functionality?
I don't mind running dotnet watch with the appropriate settings myself.

@p10tyr
Copy link
Owner

p10tyr commented Jun 3, 2019

Ahh yes. I completely forgot about that aspect I am so frustrated with this automatic thing.
Will try and sort that asap to get it working with 2019 in a limited state at least, until I find some other way to fix the testwindow thing

@iainnicol
Copy link

iainnicol commented Jun 6, 2019 •
edited
Loading

@ppumkin, you say

So why do I need the test window?
A private object "Configuration" contains the following information about the test runner.

Given the trouble we've had with reflection, I have been thinking a bit about possible alternative approaches.

  1. Is the debugger attached?
    i. If yes then do not attach Coverlet because coverlet modifes part of the binary and causes debugging to fail

I eventually realised the question to ask is: How does coverlet itself handle this? Coverlet has a package for direct VSTest DLL integration. Another package is offered for MSBuild integration, but that doesn't help us because the VS test runner uses... VSTest directly.

Now, the coverlet VSTest integration is essentially a DataCollector. The coverlet docs suggest running this collector from the dotnet test command line. But collectors can also be used from .runsettings files, including from Visual Studio.

So IMHO we don't really need any code here. Instead we just need documentation, mainly an example .runsettings file.

  1. Which Test projects are being run?

And hopefully the answer here is similar: we need a DataCollector, though this time we write our own, to simply add to our list of recently run test projects. This might require some IPC between the data collector and the main PrestoCoverage instance.

What do you think? Is this worth me or you prototyping?

@p10tyr
Copy link
Owner

p10tyr commented Jun 6, 2019

I know what I am trying to do is probably incorrect. I have learnt about this years ago in development. If it seems too difficult it’s probably because it’s the wrong way 😂

I am a newb to extensions and what you have proposed is a different way. Probably the correct way. So yes please try and make a branch if you will.

All we need to know is to attach coverlet before the test runs... ? And when they finish to aggregate the results.

@p10tyr
Copy link
Owner

p10tyr commented Jun 6, 2019

https://github.com/Microsoft/vstest/tree/master/test/TestAssets/OutOfProcDataCollector

This looks promising. I am sure I have come across before... not sure why I decided to go down reflection route. Probably couldn’t get enough information out of that data collector.

@iainnicol
Copy link

Ah. Well hopefully I have better luck then!

I am amused by your comments about difficult things being the wrong way. Yes. But sometimes wrong code is the only way!

@p10tyr
Copy link
Owner

p10tyr commented Jun 6, 2019

Please try it. I hope you can find a solution.

@iainnicol
Copy link

Good news. This approach works. The code needs tidied up, but I hope to contribute it next week.

Turns out this is bleeding edge stuff. The coverlet DataCollector was only released a week ago, the day I started looking at this! It just wasn't available when you took a look at data collectors.

@p10tyr
Copy link
Owner

p10tyr commented Jun 17, 2019 •
edited
Loading

Ahhh right. Very interesting. I have a feeling your findings will make it a lot easier understand now. But is this stuff is compatible with VS2017 ? Or that can still stay with the fugly reflection garbage?

Cannot wait to see how you fixed it 👍

@LoriBru
Copy link

LoriBru commented Jul 17, 2019

Ehi guys! Any update on the VS2019 compatibility?

@Hviid
Copy link

Hviid commented Aug 14, 2019

@iainnicol Any news on this issue?

@iainnicol
Copy link

iainnicol commented Aug 14, 2019 •
edited
Loading

Ah, sorry, mainly I got distracted by other things. I will work on this this weekend.

I had a minor but required pull request rejected for Presto [edit: coverlet] itself, which is unfortunate. I may need a hack. Worst case, I suppose, isn't that bad because Presto [edit: coverlet] has already been forked.

@p10tyr
Copy link
Owner

p10tyr commented Aug 17, 2019 via email

@iainnicol
Copy link

Oops. I indeed meant coverlet.

We need coverlet to target netstandard (coverlet-coverage/coverlet#470) so that the coverlet VS Test collector works within Visual Studio. Otherwise their VS Test collector only works from the console.

So what to to? Hopefully VS has changed and now works with netcoreapp. If not, maybe we can still consume the coverlet nuget package, but hack the metadata to look like netstandard. Else we have to fork. I know you've had to fork coverlet before, but I don't want to add a second reason for the fork!

@p10tyr
Copy link
Owner

p10tyr commented Aug 18, 2019 via email

@cmenzi
Copy link

cmenzi commented Aug 27, 2019

Any updates on this?

@p10tyr
Copy link
Owner

p10tyr commented Aug 27, 2019 via email

This was referenced Sep 3, 2019
Closed
Closed
@p10tyr
Copy link
Owner

p10tyr commented Sep 3, 2019

Good news. This approach works. The code needs tidied up, but I hope to contribute it next week.

Turns out this is bleeding edge stuff. The coverlet DataCollector was only released a week ago, the day I started looking at this! It just wasn't available when you took a look at data collectors.

@iainnicol I have a fork running here https://github.com/ppumkin/coverlet we can change whatever we need to get this going. I am also building the Experimental Coverlet package using that fork, which on on the VS Store.

Problem is that coverlet do not want to "open" the coverlet.core because they afraid to change API's that may cause backlash from the community... but also they not offering any solutions so i just forked it and told them im going to build the core one on my own responsibility. They said OK.

So if you got a coverlet PR, do it against my fork, i will rebuild the package that this project uses and lets get VS2019 working... ??! 😄

This was referenced Sep 3, 2019
Closed
Closed
@p10tyr
Copy link
Owner

p10tyr commented Sep 3, 2019 •
edited
Loading

I released 1.16.5 that will not cause that dumb ass exception to show any more..

but the automagic coverage DOES NOT WORK IN 2019 YET

You can only use the watch folder function in VS2019 for NOW.. atleast something for the time being.

Sorry for taking so long to get to it.

@jotatoledo
Copy link

jotatoledo commented Sep 13, 2019 •
edited
Loading

Just found out your extension after almost giving up hope on coverage-tools for VS.

Sorry for taking so long to get to it.

Thanks for the amazing work. This is OSS, so dont feel sorry for progress taking its time!

@adrianhara
Copy link

I'm using VS2019 Professional and get no gutters. The json report is generated under c:\coverlet\xunittestcoverage.json , looks good, the extension is installed, but no gutters. Any idea what I could try?

@brianhill-bfs
Copy link

I have the same issue as @adrianhara with VS2019 Professional. I have coverage output, but nothing loaded in VS.

@siuarima9
Copy link

I'm using vs2019 but this extension isn't working..

@eduardocp
Copy link

@ppumkin

I think this resolve the problem of internal classes/methods:

dynamic testRequestConfiguration = stateArgs.Operation.GetType()
	.GetProperty("Configuration")
	.GetValue(stateArgs.Operation);

var sources = testRequestConfiguration.GetType()
	.GetProperty("TestSources")
	.GetValue(testRequestConfiguration);

//if (testRequestConfiguration.Debug)
//	return;

foreach (string testDll in sources)
{
	...
}

I also saw that the Debug property has been removed...

@dozer75
Copy link

dozer75 commented Nov 28, 2019

Tried 1.16.6 today with Visual Studio 16.3.10 and still not working. Getting:

[2019-11-28 9:31:24.778 pm Error] System.MissingMethodException: Method not found: 'Boolean Microsoft.VisualStudio.TestWindow.Controller.TestRunConfiguration.get_Debug()'.
   at PrestoCoverage.PrestoCoverageCore.OnTestExecutionStarting(Object sender, OperationStateChangedEventArgs stateArgs)
   at PrestoCoverage.PrestoCoverageContainerDiscoverer.OperationState_StateChanged(Object sender, OperationStateChangedEventArgs e) in C:\Projects\PrestoCoverage\PrestoCoverage\PrestoCoverage\PrestoCoverageContainerDiscoverer.cs:line 73
   at System.EventHandler`1.Invoke(Object sender, TEventArgs e)
   at Microsoft.VisualStudio.TestWindow.Controller.OperationData.<>c__DisplayClass44_0.<OnStateChanged>b__0()
   at Microsoft.VisualStudio.TestWindow.Extensibility.ILoggerExtensions.CallInternalWithLogging(ILogger2 log, Action action, Boolean shouldThrow)
--- End of stack trace from previous location where exception was thrown ---

@pcbl
Copy link

pcbl commented Nov 29, 2019

same issue as @dozer75 , when running the tests on Vs 2019 16.3.10.

@eduardocp
Copy link

@ppumkin had already commented above:

I released 1.16.5 that will not cause that dumb ass exception to show any more..

but the automagic coverage DOES NOT WORK IN 2019 YET

You can only use the watch folder function in VS2019 for NOW.. atleast something for the time being.

Sorry for taking so long to get to it.

@dozer75
Copy link

dozer75 commented Nov 30, 2019

@eduardocp Well, he stated that the "dumb ass exception" shouldn't occur anymore after 1.6.15, but it does in 1.6.16, that's why I added the message... :)

To clarify to anyone else what's this means:
If you run tests in Visual Studio 2019 it will not do auto coverage when you run the test using test explorer in visual studio (you'll get that "dumb ass exception")...

However; if you run manually using dotnet test /p:CollectCoverage=true /p:CoverletOutput=<Path> where <Path> is the folder configured in Tools -> Options -> PrestoCoverage -> General Settings -> Path the coverage will be displayed.

@AXMIM
Copy link

AXMIM commented Dec 20, 2019

I'm unable to install it on VS Professionnal 2019 Version 16.2.5
I get the following message.
I rebooted a few times already to no avail.
It is sad to see this extension get screwed up by VS 2019.

2019-12-20 11:41:35 - Microsoft VSIX Installer
2019-12-20 11:41:35 - -------------------------------------------
2019-12-20 11:41:35 - vsixinstaller.exe version:
2019-12-20 11:41:35 - 16.4.1057
2019-12-20 11:41:35 - -------------------------------------------
2019-12-20 11:41:35 - Command line parameters:
2019-12-20 11:41:35 - C:\Program Files (x86)\Microsoft Visual Studio\Installer\resources\app\ServiceHub\Services\Microsoft.VisualStudio.Setup.Service\vsixinstaller.exe,/appidinstallpath:C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\Common7\IDE\devenv.exe,/skuName:Pro,/skuVersion:16.2.29306.81,/appidname:Microsoft Visual Studio Professional 2019,/culture:en-US,/noep,C:\Users\maxiveil\AppData\Local\Temp\VSIXjfohq50j.vsix
2019-12-20 11:41:35 - -------------------------------------------
2019-12-20 11:41:35 - Microsoft VSIX Installer
2019-12-20 11:41:35 - -------------------------------------------
2019-12-20 11:41:36 - Setup instance 8d590194 requires a reboot. Reboot the machine and try again.
2019-12-20 11:41:36 - System.InvalidOperationException: Setup instance 8d590194 requires a reboot. Reboot the machine and try again.
à VSIXInstaller.SetupExtensions.GetLaunchableInstances(IQuery query)
à VSIXInstaller.SupportedSKUs.EnumerateIsolatedInstalls(Action1 callback) à VSIXInstaller.SupportedSKUs.AddInstalledLocationBasedSKUs(IntPtr userToken) à VSIXInstaller.SupportedSKUs.InitializeSupportedSKUs(IntPtr userToken) à VSIXInstaller.ExtensionService.InitializeSupportedSKUs(ICommandLineData cmdLineData, IntPtr duplicatedUserToken) à VSIXInstaller.App.Initialize(Boolean isRepairSupported) à VSIXInstaller.App.Initialize() à System.Threading.Tasks.Task1.InnerInvoke()
à System.Threading.Tasks.Task.Execute()
--- Fin de la trace de la pile à partir de l'emplacement précédent au niveau duquel l'exception a été levée ---
à Microsoft.VisualStudio.Telemetry.WindowsErrorReporting.WatsonReport.GetClrWatsonExceptionInfo(Exception exceptionObject)

@BDomzalski
Copy link

It's nearly half-year since @iainnicol checked that the approach with DataCollectors works and a fork of Coverlet has been created. Any progress? I know from own experience that finding time might be almost impossible. I would like to see this working so maybe I could help?

@p10tyr
Copy link
Owner

p10tyr commented Feb 10, 2020 via email

@BDomzalski
Copy link

I did a quick look around the forks but did not spot where it could be that those collectors were used. I only infer that base on the following two comments: description of possible solution and approach works.

@p10tyr
Copy link
Owner

p10tyr commented Feb 10, 2020 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests