Skip to content

Commit

Permalink
Fixes to UI Tests related to LS (#7659)
Browse files Browse the repository at this point in the history 
* Fix LS UI Tests
* Warn instead of throwing exception
* Create file
* Increase delays
* Added comments
  • Loading branch information
@DonJayamanne
DonJayamanne authored Sep 29, 2019
1 parent b7ad1d1 commit 9315191
Show file tree
Hide file tree
Showing 10 changed files with 144 additions and 140 deletions.
5 changes: 4 additions & 1 deletion uitests/bootstrap/extension/extension.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ const vscode = require('vscode');
const fs = require('fs');
const path = require('path');
const util = require('util');

const lsOutputOpenedFile = path.join(__dirname, '..', 'lsoutputdisplayed.log');
let activated = false;
async function sleep(timeout) {
return new Promise(resolve => setTimeout(resolve, timeout));
Expand Down Expand Up @@ -91,6 +91,9 @@ function activate(context) {
continue;
}
}
// Creating this file indicates the fact that the command was executed successfully.
// The existence of this file is checked by the ui tests.
await new Promise(resolve => fs.writeFile(lsOutputOpenedFile, '', resolve));
});
vscode.commands.registerCommand('smoketest.runInTerminal', async () => {
const filePath = path.join(__dirname, '..', 'commands.txt');
Expand Down
8 changes: 8 additions & 0 deletions uitests/features/languageServer/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# Notes:

* Why do have delays of >= 120 seconds when using LS.
* 1. This is to provide LS sufficient time to analyze the entire workspace.
* E.g. when using a `conda` environment, LS takes a long time to complete analysis.
* This results in delays in responsiveness of LS, i.e. features such as `intellisense`, `code navigation`, etc might not be ready or won't respond in time.
* To determine whether the LS has completed analysis, we ensure the statusbar item `Analyzing in background` is no longer visible.
* 2. Also, it takes time to download & extract the Language Server
62 changes: 13 additions & 49 deletions uitests/features/languageServer/basic.feature
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,20 @@
@code:code/languageServer/basic
Feature: Language Server
@smoke
Scenario Outline: Check output of 'Python' output panel when starting VS Code with Jedi <jedi_enable>d
When I <jedi_enable> the workspace setting "python.jediEnabled"
Scenario: Check output of 'Python' output panel when starting VS Code with Jedi enabled
When I enable the workspace setting "python.jediEnabled"
And I wait for the Python extension to activate
And I select the command "Python: Show Output"
Then the text "<first_text_in_ooutput_panel>" will be displayed in the output panel within <time_to_activate> seconds
Then the text "Jedi Python language engine" will be displayed in the output panel within 10 seconds

Examples:
| jedi_enable | time_to_activate | first_text_in_ooutput_panel |
| enable | 10 | Jedi Python language engine |
| disable | 120 | Microsoft Python language server |
@smoke
Scenario: Check output of 'Python' and 'Python Language Server' output panel when starting VS Code with Language Server enabled
When I disable the workspace setting "python.jediEnabled"
And I wait for the Python extension to activate
And I select the command "Python: Show Output"
Then the text "Microsoft Python language server" will be displayed in the output panel within 120 seconds
When I select the command "Python: Show Language Server Output"
Then the text "Initializing for" will be displayed in the output panel within 120 seconds

Scenario Outline: Language Server is downloaded with http.proxyStrictSSL setting <enabled_disabled>
When I open VS Code for the first time
Expand All @@ -26,45 +30,5 @@ Feature: Language Server

Examples:
| enabled_disabled | protocol_to_look_for |
| enabled | https:// |
| disabled | http:// |

@autoretry
Scenario Outline: Navigate to definition of a variable when extension has already been activated with Jedi <jedi_enable>d
When I reload VS Code
And I <jedi_enable> the workspace setting "python.jediEnabled"
And I wait for the Python extension to activate
And I select the command "Python: Show Output"
Then the text "<first_text_in_ooutput_panel>" will be displayed in the output panel within <time_to_activate> seconds
# Because LS is slow.
And wait for <time_to_activate> seconds
When I open the file "my_sample.py"
And I go to line 3, column 10
# Wait for intellisense to kick in (sometimes slow in jedi & ls)
And I wait for 10 seconds
When I select the command "Go to Definition"
Then the cursor is on line 1

Examples:
| jedi_enable | time_to_activate | first_text_in_ooutput_panel |
| enable | 10 | Jedi Python language engine |
| disable | 120 | Microsoft Python language server |

# @autoretry @wip
# Scenario Outline: Navigate to definition of a variable after opening a file with Jedi <jedi_enabled>
# Given the workspace setting "python.jediEnabled" is <jedi_enabled>
# When I open the file "my_sample.py"
# And I wait for the Python extension to activate
# And I select the command "Python: Show Output"
# Then the text "<first_text_in_ooutput_panel>" will be displayed in the output panel within <time_to_activate> seconds
# And the text "<second_text_in_output_panel>" will be displayed in the output panel within <time_to_activate> seconds
# When I go to line 3, column 10
# # Wait for intellisense to kick in (sometimes slow in jedi & ls)
# And I wait for 10 seconds
# And I select the command "Go to Definition"
# Then the cursor is on line 1

# Examples:
# | jedi_enabled | time_to_activate | first_text_in_ooutput_panel | second_text_in_output_panel |
# | enabled | 10 | Jedi Python language engine | Jedi Python language engine |
# | disabled | 120 | Microsoft Python language server | Initializing for |
| enabled | Downloading https:// |
| disabled | Downloading http:// |
73 changes: 28 additions & 45 deletions uitests/features/languageServer/goToDefinition.feature
View file
Original file line number Diff line number Diff line change
@@ -1,64 +1,47 @@
@ls
@code:code/languageServer/basic
Feature: Language Server
Scenario Outline: When <reload_or_start_vs_for_first_time> with Jedi <jedi_enable>d then output contains <text_in_output_panel>
When <reload_or_start_vs_for_first_time>
And I <jedi_enable> the workspace setting "python.jediEnabled"
And I wait for the Python extension to activate
And I select the command "<output_panel_command>"
Then the text "<text_in_output_panel>" will be displayed in the output panel within <time_to_activate> seconds

Examples:
| jedi_enable | reload_or_start_vs_for_first_time | time_to_activate | text_in_output_panel | output_panel_command |
| enable | I open VS Code for the first time | 5 | Jedi Python language engine | Python: Show Output |
| enable | I reload VS Code | 5 | Jedi Python language engine | Python: Show Output |
| disable | I open VS Code for the first time | 120 | Microsoft Python language server | Python: Show Output |
| disable | I open VS Code for the first time | 120 | Downloading | Python: Show Language Server Output |
| disable | I reload VS Code | 120 | Microsoft Python language server | Python: Show Output |

@autoretry
Scenario Outline: When <reload_or_start_vs_for_first_time> with Jedi <jedi_enable>d then navigate to definition of a variable
Scenario Outline: When <reload_or_start_vs_for_first_time> then navigate to the definition of a variable after opening file with Jedi enabled
When <reload_or_start_vs_for_first_time>
And I <jedi_enable> the workspace setting "python.jediEnabled"
And I enable the workspace setting "python.jediEnabled"
And I wait for the Python extension to activate
And I select the command "<output_panel_command>"
Then the text "<text_in_output_panel>" will be displayed in the output panel within <time_to_activate> seconds
# Because LS is slow.
And wait for <time_to_activate> seconds
And I select the command "Python: Show Output"
Then the text "Jedi Python language engine" will be displayed in the output panel within 10 seconds
# Sometimes LS & Jedi are slow.
And wait for 10 seconds
When I open the file "my_sample.py"
And I go to line 3, column 10
# Wait for intellisense to kick in (sometimes slow in jedi & ls)
And I wait for 5 seconds
And I select the command "Go to Definition"
And I wait for 10 seconds
When I select the command "Go to Definition"
Then the cursor is on line 1

Examples:
| jedi_enable | reload_or_start_vs_for_first_time | time_to_activate | text_in_output_panel | output_panel_command |
| enable | I open VS Code for the first time | 5 | Jedi Python language engine | Python: Show Output |
| enable | I reload VS Code | 5 | Jedi Python language engine | Python: Show Output |
| disable | I open VS Code for the first time | 120 | Microsoft Python language server | Python: Show Output |
| disable | I open VS Code for the first time | 120 | Downloading | Python: Show Language Server Output |
| disable | I reload VS Code | 120 | Microsoft Python language server | Python: Show Output |
| reload_or_start_vs_for_first_time |
| I open VS Code for the first time |
| I reload VS Code |

@autoretry
Scenario Outline: When I open VS Code for the first time with Jedi <jedi_enable>d, open a file then navigate to definition of a variable
When I open VS Code for the first time
And I <jedi_enable> the workspace setting "python.jediEnabled"
And I select the command "Python: Show Output"
Scenario Outline: When <reload_or_start_vs_for_first_time> then navigate to the definition of a variable after opening file with Language Server enabled
When <reload_or_start_vs_for_first_time>
And I disable the workspace setting "python.jediEnabled"
And I wait for the Python extension to activate
And I open the file "my_sample.py"
And I select the command "<output_panel_command>"
Then the text "<text_in_output_panel>" will be displayed in the output panel within <time_to_activate> seconds
# Because LS is slow.
And wait for <time_to_activate> seconds
When I go to line 3, column 10
And I select the command "Python: Show Output"
Then the text "Microsoft Python language server" will be displayed in the output panel within 120 seconds
When I select the command "Python: Show Language Server Output"
Then the text "Initializing for" will be displayed in the output panel within 120 seconds
# Sometimes LS & Jedi are slow.
When I wait for 10 seconds
Then the status bar item containing the text "Analyzing in background" will be hidden within 120 seconds
When I open the file "my_sample.py"
And I go to line 3, column 10
# Wait for intellisense to kick in (sometimes slow in jedi & ls)
And I wait for 5 seconds
And I select the command "Go to Definition"
And I wait for 10 seconds
When I select the command "Go to Definition"
Then the cursor is on line 1

Examples:
| jedi_enable | time_to_activate | text_in_output_panel | output_panel_command |
| enable | 5 | Jedi Python language engine | Python: Show Output |
| disable | 120 | Microsoft Python language server | Python: Show Output |
| disable | 120 | Downloading | Python: Show Language Server Output |
| reload_or_start_vs_for_first_time |
| I open VS Code for the first time |
| I reload VS Code |
44 changes: 28 additions & 16 deletions uitests/features/languageServer/intellisense.feature
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,19 @@ Feature: Language Server
And I <jedi_enable> the workspace setting "python.jediEnabled"
And I wait for the Python extension to activate
And I select the command "Python: Show Output"
Then the text "<first_text_in_ooutput_panel>" will be displayed in the output panel within <time_to_activate> seconds
And wait for <time_to_activate> seconds
Then the text "<first_text_in_output_panel>" will be displayed in the output panel within <time_to_activate> seconds
And I select the command "<second_output_panel_command>"
Then the text "<last_text_in_second_output_panel>" will be displayed in the output panel within <time_to_activate> seconds
# Sometimes LS & Jedi are slow.
And wait for 10 seconds
# Though this only applies to Language Server, there's no harm in testing this in Jedi (it won't exist).
# One less column in the example section.
And the status bar item containing the text "Analyzing in background" will be hidden within 120 seconds
# Get more realestate on UI (hide what we don't need).
And select the command "View: Close Panel"
When I open the file "intelli_sample.py"
# Wait for intellisense to kick in (sometimes slow in jedi & ls)
And I wait for <wait_time> seconds
And wait for 10 seconds
And I go to line 3, column 13
And I press ctrl+space
Then auto completion list contains the item "excepthook"
Expand All @@ -40,20 +46,26 @@ Feature: Language Server
And auto completion list contains the item "count"

Examples:
| jedi_enable | reload_or_start_vs_for_first_time | time_to_activate | first_text_in_ooutput_panel | wait_time |
| enable | I open VS Code for the first time | 5 | Jedi Python language engine | 5 |
| enable | I reload VS Code | 5 | Jedi Python language engine | 5 |
| disable | I open VS Code for the first time | 120 | Microsoft Python language server | 5 |
| disable | I reload VS Code | 120 | Microsoft Python language server | 5 |
| jedi_enable | reload_or_start_vs_for_first_time | time_to_activate | first_text_in_output_panel | last_text_in_second_output_panel | wait_time | second_output_panel_command |
| enable | I open VS Code for the first time | 10 | Jedi Python language engine | Jedi Python language engine | 10 | Python: Show Output |
| enable | I reload VS Code | 10 | Jedi Python language engine | Jedi Python language engine | 10 | Python: Show Output |
| disable | I open VS Code for the first time | 120 | Microsoft Python language server | Microsoft Python language server | 10 | Python: Show Language Server Output |
| disable | I reload VS Code | 120 | Microsoft Python language server | Microsoft Python language server | 10 | Python: Show Language Server Output |

@autoretry
Scenario Outline: When <reload_or_start_vs_for_first_time> with Jedi <jedi_enable>d then intellisense works for untitled files
When <reload_or_start_vs_for_first_time>
And I <jedi_enable> the workspace setting "python.jediEnabled"
And I wait for the Python extension to activate
And I select the command "Python: Show Output"
Then the text "<first_text_in_ooutput_panel>" will be displayed in the output panel within <time_to_activate> seconds
And wait for <time_to_activate> seconds
Then the text "<first_text_in_output_panel>" will be displayed in the output panel within <time_to_activate> seconds
And I select the command "<second_output_panel_command>"
Then the text "<last_text_in_second_output_panel>" will be displayed in the output panel within <time_to_activate> seconds
# Sometimes LS & Jedi are slow.
And wait for 10 seconds
# Though this only applies to Language Server, there's no harm in testing this in Jedi (it won't exist).
# One less column in the example section.
And the status bar item containing the text "Analyzing in background" will be hidden within 120 seconds
# Get more realestate on UI (hide what we don't need).
And select the command "View: Close Panel"
When I create a new file with the following content
Expand All @@ -64,16 +76,16 @@ Feature: Language Server
"""
And I change the language of the file to "Python"
# Wait for intellisense to kick in (sometimes slow in jedi & ls)
And I wait for <wait_time> seconds
And wait for 10 seconds
And I go to line 3, column 13
And I press ctrl+space
Then auto completion list contains the item "excepthook"
And auto completion list contains the item "exec_prefix"
And auto completion list contains the item "executable"

Examples:
| jedi_enable | reload_or_start_vs_for_first_time | time_to_activate | first_text_in_ooutput_panel | wait_time |
| enable | I open VS Code for the first time | 5 | Jedi Python language engine | 5 |
| enable | I reload VS Code | 5 | Jedi Python language engine | 5 |
| disable | I open VS Code for the first time | 120 | Microsoft Python language server | 5 |
| disable | I reload VS Code | 120 | Microsoft Python language server | 5 |
| jedi_enable | reload_or_start_vs_for_first_time | time_to_activate | first_text_in_output_panel | last_text_in_second_output_panel | wait_time | second_output_panel_command |
| enable | I open VS Code for the first time | 10 | Jedi Python language engine | Jedi Python language engine | 10 | Python: Show Output |
| enable | I reload VS Code | 10 | Jedi Python language engine | Jedi Python language engine | 10 | Python: Show Output |
| disable | I open VS Code for the first time | 120 | Microsoft Python language server | Microsoft Python language server | 10 | Python: Show Language Server Output |
| disable | I reload VS Code | 120 | Microsoft Python language server | Microsoft Python language server | 10 | Python: Show Language Server Output |
Loading

0 comments on commit 9315191

Please sign in to comment.