Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix codelens action for opening a dropdown on parametrized tests in windows 10 #8647

Merged

Conversation

phloose
Copy link

@phloose phloose commented Nov 18, 2019

Fixes #8627. When the path of an actual test file (given by vscode.Uri.fsPath) is later compared against the fullPath of a single item of tests.TestFiles the drive letter mismatches and aborts further actions despite the rest of the path is the same.

  • Pull request represents a single change (i.e. not fixing disparate/unrelated things in a single PR)
  • Title summarizes what is changing
  • Has a news entry file (remember to thank yourself!)
  • Appropriate comments and documentation strings in the code
  • Has sufficient logging.
  • Has telemetry for enhancements.
  • Unit tests & system/integration tests are added/updated
  • Test plan is updated as appropriate
  • package-lock.json has been regenerated by running npm install (if dependencies have changed)
  • The wiki is updated with any design decisions/details.

@codecov-io
Copy link

codecov-io commented Nov 18, 2019

Codecov Report

❗ No coverage uploaded for pull request base (master@c9d6887). Click here to learn what that means.
The diff coverage is 100%.

Impacted file tree graph

@@           Coverage Diff            @@
##             master   #8647   +/-   ##
========================================
  Coverage          ?   57.1%           
========================================
  Files             ?     550           
  Lines             ?   28841           
  Branches          ?    4358           
========================================
  Hits              ?   16469           
  Misses            ?   11471           
  Partials          ?     901
Impacted Files Coverage Δ
src/client/testing/display/picker.ts 46.3% <100%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c9d6887...9acd9b0. Read the comment docs.

Copy link
Member

@ericsnowcurrently ericsnowcurrently left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just missing a test. Otherwise LGTM. Thanks for working on this!

@phloose
Copy link
Author

phloose commented Nov 28, 2019

This is just missing a test.

@ericsnowcurrently I added a test. I am not really happy with all that mocking, though. But it ensures that respective parts of the method TestDisplay.displayFunctionTestPickerUI, that are responsible for showing the picker, can be reached when the drive letters for fileName and item.fullPath are different.

Should i squash the two commits into one when everything is ok from your side?

@ericsnowcurrently
Copy link
Member

Don't worry about squashing. We always squash-and-merge for PRs. :)

Copy link
Member

@ericsnowcurrently ericsnowcurrently left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First of all, I apologize for the state of the existing tests (in that file). They are too dense and not particularly careful. So they aren't a great example, unfortunately.

Furthermore, there is basically zero coverage of the TestDisplay class's methods. Basically, there should have been a bunch of tests there already covering all the behavior and all you would have to do is add one test (based on an existing one) that exercises the relevant behavior on Windows. Obviously no such tests exist.

I spent some time yesterday looking into the possibility of (me) writing those missing tests, but doing so is going to take a lot of work. This relates in part to your earlier statement (assuming I understood you correctly):

I am not really happy with all that mocking, though.

Yeah, all the mocks aren't ideal and we are working on a better approach. Also, mocks like these are mostly the province of unit tests, as opposed to other kinds of tests. So perhaps you mean you'd rather have functional tests (running against actual system resources)?

For unit tests we want:

  • low-level focus
  • determinism (even on different host OS)
  • explicit control of all inputs (direct & indirect)
  • verification of all outputs (direct & indirect), at least relative to the suite of tests in aggregate (and usually on a per-test basis)

Consequent to all the above, I've made a number suggestions that will help align your tests with the what should have been there already. :) There are a few more comments I considered leaving, but let's start with what I have done already.

Thanks!

src/test/testing/display/picker.unit.test.ts Outdated Show resolved Hide resolved
src/test/testing/display/picker.unit.test.ts Outdated Show resolved Hide resolved
src/test/testing/display/picker.unit.test.ts Outdated Show resolved Hide resolved
src/test/testing/display/picker.unit.test.ts Outdated Show resolved Hide resolved
src/test/testing/display/picker.unit.test.ts Outdated Show resolved Hide resolved
src/test/testing/display/picker.unit.test.ts Outdated Show resolved Hide resolved
src/test/testing/display/picker.unit.test.ts Outdated Show resolved Hide resolved
src/test/testing/display/picker.unit.test.ts Outdated Show resolved Hide resolved
src/test/testing/display/picker.unit.test.ts Outdated Show resolved Hide resolved
src/test/testing/display/picker.unit.test.ts Outdated Show resolved Hide resolved
@phloose
Copy link
Author

phloose commented Dec 4, 2019

@ericsnowcurrently Thanks a lot for your very detailed review! I really appreciate that! I'll go through your suggestions and advices and change the respective parts.

Regarding the coverage i was really puzzled when i saw that there were no tests at all for the TestDisplay class... Thats the reason i setup the respective suite in general but also wasn't completly sure how to structure it because i didn't now if you would adapt to it or what else should be in there. So i am happy to lay a foundation for future tests of that class.

So perhaps you mean you'd rather have functional tests

I always prefer to have some kind of real/unmocked functionality in tests i write (be it unit or functional tests...). Since i am developing most of the time in python where mocking is really easy and flexible i am always careful when i need to mock. Sometimes i have the feeling that in the end you test that you configured your mocks right, which possibly masks other errors that bubble up later. Of course i know that especially in unit tests you need to mock when the surrounding depeneds a lot on the state of other classes/objects (like in this case with TestDisplay).

Copy link
Member

@ericsnowcurrently ericsnowcurrently left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good. Thanks for making those changes. I only have a few other comments.

src/test/testing/display/picker.unit.test.ts Outdated Show resolved Hide resolved
src/test/testing/display/picker.unit.test.ts Outdated Show resolved Hide resolved
src/test/testing/display/picker.functional.test.ts Outdated Show resolved Hide resolved
src/test/testing/display/picker.functional.test.ts Outdated Show resolved Hide resolved
src/test/testing/display/picker.functional.test.ts Outdated Show resolved Hide resolved
@phloose
Copy link
Author

phloose commented Dec 11, 2019

@ericsnowcurrently I changed the tests according to your suggestions. In order to keep the git history clean i would prefer to squash the testing related commits into one as there are five commits that have a lot of changes in between.

Fixes microsoft#8627. When the path of an actual test file (given by
vscode.Uri.fsPath) is later compared against the fullPath of a single
item of tests.TestFiles the drive letter mismatches on Windows and
aborts further actions despite the rest of the path is the same. As a
result code lenses on parametrized tests do not open a dropdown for
selecting single tests.
@phloose
Copy link
Author

phloose commented Dec 11, 2019

i just saw that something changed with the FileSystem class because compiling fails?

EDIT:
Did a rebase on master and saw that FileSystem needs now PlatformService() as argument when making an instance.

@phloose phloose force-pushed the codelens-on-parametrized-tests-8627 branch from 9b2d5ef to 1be096d Compare December 11, 2019 12:16
@ericsnowcurrently
Copy link
Member

Don't worry about squashing. We only squash-and-merge when landing PRs, so that will be done automatically.

Copy link
Member

@ericsnowcurrently ericsnowcurrently left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great. Thanks for all those changes. I'm happy with where this PR is at.

I've made a number of comments about formatting and one about maybe changing a declaration from ITestDisplay to TestDisplay. None of them are critical and I'm approving the PR.

I'll give you time to make any of those recommended changes that you want and then merge it. FYI, I'm also going to run the PR through our full CI test suite just to verify the change more broadly. However, there's nothing you need to do relative to that.

src/test/testing/display/picker.unit.test.ts Outdated Show resolved Hide resolved
src/test/testing/display/picker.unit.test.ts Outdated Show resolved Hide resolved
src/test/testing/display/picker.unit.test.ts Outdated Show resolved Hide resolved
src/test/testing/display/picker.unit.test.ts Outdated Show resolved Hide resolved
src/test/testing/display/picker.unit.test.ts Outdated Show resolved Hide resolved
src/test/testing/display/picker.functional.test.ts Outdated Show resolved Hide resolved
src/test/testing/display/picker.functional.test.ts Outdated Show resolved Hide resolved
src/test/testing/display/picker.functional.test.ts Outdated Show resolved Hide resolved
src/test/testing/display/picker.functional.test.ts Outdated Show resolved Hide resolved
@ericsnowcurrently
Copy link
Member

Again, thanks for all your work on this. I really appreciate your willingness to accommodate my various suggestions. 😄

@phloose phloose force-pushed the codelens-on-parametrized-tests-8627 branch from 1be096d to 3e22980 Compare December 11, 2019 19:18
@phloose
Copy link
Author

phloose commented Dec 11, 2019

Again, thanks for all your work on this. I really appreciate your willingness to accommodate my various suggestions. 😄

Thank you for your help and suggestions! I found that super helpful and really appreciate that! You explained everything and it made sense to me. This has definitely taken me further in terms of writing tests in general and in learning typescript!

@phloose phloose force-pushed the codelens-on-parametrized-tests-8627 branch from 3e22980 to 9acd9b0 Compare December 11, 2019 20:53
@ericsnowcurrently ericsnowcurrently merged commit 2302553 into microsoft:master Dec 11, 2019
@lock lock bot locked as resolved and limited conversation to collaborators Dec 19, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Python Testing: Codelens above parametrized tests doesn't open dropdown for selecting tests
4 participants