Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add telemetry for download, extract, and analyze. #2597

Merged
merged 11 commits into from
Sep 18, 2018
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ npm-debug.log
!yarn.lock
coverage/
.vscode-test/**
.venv
.venv*/
pythonFiles/experimental/ptvsd/**
debug_coverage*/**
languageServer/**
Expand Down
1 change: 1 addition & 0 deletions news/1 Enhancements/2461.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add telemetry to download, extract, and analyze, phases of the Python Language Server
1 change: 1 addition & 0 deletions news/2 Fixes/2297.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Stop duplicate initializations of Python Language Server progress reporter.
23 changes: 23 additions & 0 deletions src/client/activation/downloader.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,12 @@ import { ProgressLocation, window } from 'vscode';
import { createDeferred } from '../../utils/async';
import { IFileSystem } from '../common/platform/types';
import { IExtensionContext, IOutputChannel } from '../common/types';
import { captureTelemetry } from '../telemetry';
import {
PYTHON_LANGUAGE_SERVER_DOWNLOADED,
PYTHON_LANGUAGE_SERVER_EXTRACTED
} from '../telemetry/constants';
import { LanguageServerTelemetry } from '../telemetry/types';
import { PlatformData, PlatformName } from './platformData';
import { IDownloadFileService } from './types';

Expand All @@ -28,6 +34,11 @@ export const DownloadLinks = {
};

export class LanguageServerDownloader {
//tslint:disable-next-line:no-unused-variable
private lsDownloadTelemetry: LanguageServerTelemetry = {};

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove

//tslint:disable-next-line:no-unused-variable
private lsExtractTelemetry: LanguageServerTelemetry = {};

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove


constructor(
private readonly output: IOutputChannel,
private readonly fs: IFileSystem,
Expand Down Expand Up @@ -59,6 +70,12 @@ export class LanguageServerDownloader {
}
}

@captureTelemetry(
PYTHON_LANGUAGE_SERVER_DOWNLOADED,
{},
true,
(props?: LanguageServerTelemetry) => props ? props.success = true : undefined

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change second argument to {success:true}
And remove this last argument , no need of callback

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I can do that - but let's discuss a bit first.

What if the TelemetryProperties instance handed to the decorator doesn't have a success field within?

What if I want to do other things than just set properties on the telemetry? Or set a field called something other than success accordingly?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, now that you have me thinking, what about instead of a callback that gets a reference to the properties only, the beforeFailedEmit gets both the properties and the error?

Copy link

@DonJayamanne DonJayamanne Sep 15, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding additional properties is easy. Today you just need to add a type deffinition. You have already done that. I have added type deffinition to the twenty to ensure we don't add anything, this way we know exactly what's sent in the telemetry, else we have to go around looking at the code base to find the telemetry properties.

What if I want to do other things than just set properties on the telemetry? Or set a field called something other than success accordingly?

Like what? We haven't had the need for such things. All we need with telemetry is send data.
Let's not plan for tomorrow and overload the purpose of the decoratotor. Just send telemetry. If you want pre and post method execution, then you need another decorator, not this.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, I buy that, I'll make the change.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to pile on here, let's not build things we don't need. Even things we do need we shouldn't build until we need them unless they have major architectural implications. Less is more.

)
private async downloadFile(uri: string, title: string): Promise<string> {
this.output.append(`Downloading ${uri}... `);
const tempFile = await this.fs.createTemporaryFile(downloadFileExtension);
Expand Down Expand Up @@ -101,6 +118,12 @@ export class LanguageServerDownloader {
return tempFile.filePath;
}

@captureTelemetry(
PYTHON_LANGUAGE_SERVER_EXTRACTED,
this.languageServerStartupTelemetry,
true,
(props?: LanguageServerTelemetry) => props ? props.success = true : undefined

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change second argument to {success:true}
And remove this last argument , no need of callback

)
private async unpackArchive(extensionPath: string, tempFilePath: string): Promise<void> {
this.output.append('Unpacking archive... ');

Expand Down
15 changes: 14 additions & 1 deletion src/client/activation/languageServer.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.

'use strict';

import { inject, injectable } from 'inversify';
import * as path from 'path';
import {
Expand Down Expand Up @@ -28,12 +30,15 @@ import {
import { IEnvironmentVariablesProvider } from '../common/variables/types';
import { IServiceContainer } from '../ioc/types';
import { LanguageServerSymbolProvider } from '../providers/symbolProvider';
import { captureTelemetry } from '../telemetry';
import {
PYTHON_LANGUAGE_SERVER_DOWNLOADED,
PYTHON_LANGUAGE_SERVER_ENABLED,
PYTHON_LANGUAGE_SERVER_ERROR
PYTHON_LANGUAGE_SERVER_ERROR,
PYTHON_LANGUAGE_SERVER_STARTUP
} from '../telemetry/constants';
import { getTelemetryReporter } from '../telemetry/telemetry';
import { LanguageServerTelemetry } from '../telemetry/types';
import { IUnitTestManagementService } from '../unittests/types';
import { LanguageServerDownloader } from './downloader';
import { InterpreterData, InterpreterDataService } from './interpreterDataService';
Expand Down Expand Up @@ -76,6 +81,8 @@ export class LanguageServerExtensionActivator implements IExtensionActivator {
private surveyBanner: IPythonExtensionBanner;
// tslint:disable-next-line:no-unused-variable
private progressReporting: ProgressReporting | undefined;
//tslint:disable-next-line:no-unused-variable
private languageServerStartupTelemetry: LanguageServerTelemetry = {};

constructor(@inject(IServiceContainer) private readonly services: IServiceContainer) {
this.context = this.services.get<IExtensionContext>(IExtensionContext);
Expand Down Expand Up @@ -141,6 +148,12 @@ export class LanguageServerExtensionActivator implements IExtensionActivator {
(this.configuration.getSettings() as PythonSettings).removeListener('change', this.onSettingsChanged.bind(this));
}

@captureTelemetry(
PYTHON_LANGUAGE_SERVER_STARTUP,
this.languageServerStartupTelemetry,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please do not access this inside the decoratotor.
Just change second parameter to {success: true}

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this not valid in a decorator? Perhaps a static variable then (gross...) or I'll just do your suggestion. Loose typing like this can confuse me at times...

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need this, all we need is a constant object.

Loose typing like this can confuse me at times...

It's not loose typing at all, you can't set the second argument to anything, you'll get to compilation errors, meaning it's strongly typed.

Once again it is not at all loosely typed.

See the definition for the function, check the second argument:

	export function captureTelemetry(
    eventName: string,
    properties?: TelemetryProperties,

true,
(props?: LanguageServerTelemetry) => props ? props.success = true : undefined

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove this last two argument, not necessary.

)
private async startLanguageServer(clientOptions: LanguageClientOptions): Promise<boolean> {
// Determine if we are running MSIL/Universal via dotnet or self-contained app.

Expand Down
39 changes: 39 additions & 0 deletions src/client/activation/progress.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,18 @@
import { Progress, ProgressLocation, window } from 'vscode';
import { Disposable, LanguageClient } from 'vscode-languageclient';
import { createDeferred, Deferred } from '../../utils/async';
import { StopWatch } from '../../utils/stopWatch';
import { sendTelemetryEvent } from '../telemetry';
import { PYTHON_LANGUAGE_SERVER_ANALYSISTIME } from '../telemetry/constants';
import { LanguageServerTelemetry } from '../telemetry/types';

export class ProgressReporting {
private statusBarMessage: Disposable | undefined;
private progress: Progress<{ message?: string; increment?: number }> | undefined;
private progressDeferred: Deferred<void> | undefined;
private progressTimer: StopWatch | undefined;
private progressTimeout: NodeJS.Timer | undefined;
private ANALYSIS_TIMEOUT_MS: number = 60000;

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to be a constant, not a member.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. I will make the change. Thanks for calling this out!


constructor(private readonly languageClient: LanguageClient) {
this.languageClient.onNotification('python/setStatusBarMessage', (m: string) => {
Expand All @@ -19,7 +26,14 @@ export class ProgressReporting {
});

this.languageClient.onNotification('python/beginProgress', async _ => {
if (this.progressDeferred) { // if we restarted, no worries as reporting will still funnel to the same place.
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the fix for #2297 and related...

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For #2297 (while most of the problem is not here, restart is rare) you need to track 'stateChange' on the language client. When it gets to 'stopped' LS has terminated.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool! Thanks for the tip. I'll move this code to another PR that we can review separately and cover all the appropriate cases.

Copy link
Author

@d3r3kk d3r3kk Sep 17, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MikhailArkhipov please see #2606

return;
}

this.progressDeferred = createDeferred<void>();
this.progressTimer = new StopWatch();
this.progressTimeout = setTimeout(this.handleTimeout, this.ANALYSIS_TIMEOUT_MS);

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change to setTimeout(this.handleTimeout.bind(this), .....
You need to preserve callback scope.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool - that callback scope still isn't always clear to me, thanks for catching this.

window.withProgress({
location: ProgressLocation.Window,
title: ''
Expand All @@ -40,7 +54,32 @@ export class ProgressReporting {
if (this.progressDeferred) {
this.progressDeferred.resolve();
this.progressDeferred = undefined;
this.completeAnalysisTracking(true);
this.progress = undefined;
}
});
}

private completeAnalysisTracking(isSuccess: boolean): void {
if (this.progressTimer) {
const lsAnalysisTelemetry: LanguageServerTelemetry = {
success: isSuccess
};
sendTelemetryEvent(
PYTHON_LANGUAGE_SERVER_ANALYSISTIME,
this.progressTimer.elapsedTime,
lsAnalysisTelemetry
);
this.progressTimer = undefined;
}

if (this.progressTimeout) {
this.progressTimeout = undefined;
}
}

// tslint:disable-next-line:no-any
private handleTimeout(_args: any[]): void {
this.completeAnalysisTracking(false);
}
}
2 changes: 2 additions & 0 deletions src/client/common/constants.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ export const PYTHON = [
{ scheme: 'untitled', language: PYTHON_LANGUAGE }
];

export const PVSC_EXTENSION_ID = 'ms-python.python';

export namespace Commands {
export const Set_Interpreter = 'python.setInterpreter';
export const Set_ShebangInterpreter = 'python.setShebangInterpreter';
Expand Down
3 changes: 3 additions & 0 deletions src/client/telemetry/constants.ts
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,12 @@ export const UNITTEST_STOP = 'UNITTEST.STOP';
export const UNITTEST_RUN = 'UNITTEST.RUN';
export const UNITTEST_DISCOVER = 'UNITTEST.DISCOVER';
export const UNITTEST_VIEW_OUTPUT = 'UNITTEST.VIEW_OUTPUT';
export const PYTHON_LANGUAGE_SERVER_ANALYSISTIME = 'PYTHON_LANGUAGE_SERVER.ANALYSIS_TIME';
export const PYTHON_LANGUAGE_SERVER_ENABLED = 'PYTHON_LANGUAGE_SERVER.ENABLED';
export const PYTHON_LANGUAGE_SERVER_EXTRACTED = 'PYTHON_LANGUAGE_SERVER.EXTRACTED';
export const PYTHON_LANGUAGE_SERVER_DOWNLOADED = 'PYTHON_LANGUAGE_SERVER.DOWNLOADED';
export const PYTHON_LANGUAGE_SERVER_ERROR = 'PYTHON_LANGUAGE_SERVER.ERROR';
export const PYTHON_LANGUAGE_SERVER_STARTUP = 'PYTHON_LANGUAGE_SERVER.STARTUP';
export const PYTHON_LANGUAGE_SERVER_PLATFORM_NOT_SUPPORTED = 'PYTHON_LANGUAGE_SERVER.PLATFORM_NOT_SUPPORTED';

export const TERMINAL_CREATE = 'TERMINAL.CREATE';
16 changes: 15 additions & 1 deletion src/client/telemetry/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,15 @@ export function sendTelemetryEvent(eventName: string, durationMs?: number, prope
}

// tslint:disable-next-line:no-any function-name
export function captureTelemetry(eventName: string, properties?: TelemetryProperties, captureDuration: boolean = true) {
export function captureTelemetry(
eventName: string,
properties?: TelemetryProperties,
captureDuration: boolean = true,
// tslint:disable-next-line:no-any
beforeSuccessEmit?: (props?: any) => void,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need these two new parameters. Just remove the last two parameters.
Remove the 3rd and 4th arguments.

// tslint:disable-next-line:no-any
beforeFailEmit?: (props?: any) => void
) {
// tslint:disable-next-line:no-function-expression no-any
return function (target: Object, propertyKey: string, descriptor: TypedPropertyDescriptor<any>) {
const originalMethod = descriptor.value;
Expand All @@ -48,11 +56,17 @@ export function captureTelemetry(eventName: string, properties?: TelemetryProper
// tslint:disable-next-line:prefer-type-cast
(result as Promise<void>)
.then(data => {
if (beforeSuccessEmit) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove this if condition and this new code.

beforeSuccessEmit(properties);
}
sendTelemetryEvent(eventName, stopWatch.elapsedTime, properties);
return data;
})
// tslint:disable-next-line:promise-function-async
.catch(ex => {
if (beforeFailEmit) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove this if condition and this code. Change as follows:

properties = properties ? properties : {};
properties.success = false;

Copy link
Author

@d3r3kk d3r3kk Sep 17, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I cannot quite understand this exactly.

What happens when properties is not null, and was not given with a success member?

For instance, what happens when this comes from the decorator on CodeExecutionManager::executeFileInterTerminal?

That use of TelemetryProperties is using the union-ed type CodeExecutionTelemetry and has no success field.

properties = properties ? properties : {};
if ((<LanguageServerTelemetry>properties).success) {
    (<LanguageServerTelemetry>properties).success = false;
}

Reference: Type Guards and Differentiating Types

Copy link
Author

@d3r3kk d3r3kk Sep 17, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, I will have to do the type-check/type-cast it would seem:

image

Copy link

@DonJayamanne DonJayamanne Sep 17, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just change as follows:

properties = (properties ? properties : {}) as any;
if (properties.success !== undefined) {
    properties.success = false;
}

Check for undefined

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I don't think we want to do this. Assuming that success is always false when an exception is thrown (or Promise rejected) in this one case isn't necessarily going to be the right path for the next TelemetryProperties type we add with a success field.

I want to go back to doing something more explicit in the downloader/languageServer themselves instead.

beforeFailEmit(properties);
}
sendTelemetryEvent(eventName, stopWatch.elapsedTime, properties);
return Promise.reject(ex);
});
Expand Down
3 changes: 2 additions & 1 deletion src/client/telemetry/telemetry.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,15 @@
import { extensions } from 'vscode';
// tslint:disable-next-line:import-name
import TelemetryReporter from 'vscode-extension-telemetry';
import { PVSC_EXTENSION_ID } from '../common/constants';

// tslint:disable-next-line:no-any
let telemetryReporter: TelemetryReporter;
export function getTelemetryReporter() {
if (telemetryReporter) {
return telemetryReporter;
}
const extensionId = 'ms-python.python';
const extensionId = PVSC_EXTENSION_ID;
// tslint:disable-next-line:no-non-null-assertion
const extension = extensions.getExtension(extensionId)!;
// tslint:disable-next-line:no-unsafe-any
Expand Down
17 changes: 16 additions & 1 deletion src/client/telemetry/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,10 @@ export type FormatTelemetry = {
formatSelection: boolean;
};

export type LanguageServerTelemetry = {
success?: boolean;
};

export type LinterTrigger = 'auto' | 'save';

export type LintingTelemetry = {
Expand Down Expand Up @@ -81,4 +85,15 @@ export type TerminalTelemetry = {
pythonVersion?: string;
interpreterType?: InterpreterType;
};
export type TelemetryProperties = FormatTelemetry | LintingTelemetry | EditorLoadTelemetry | PythonInterpreterTelemetry | CodeExecutionTelemetry | TestRunTelemetry | TestDiscoverytTelemetry | FeedbackTelemetry | TerminalTelemetry | DebuggerTelemetryV2 | SettingsTelemetry;
export type TelemetryProperties = FormatTelemetry
| LanguageServerTelemetry
| LintingTelemetry
| EditorLoadTelemetry
| PythonInterpreterTelemetry
| CodeExecutionTelemetry
| TestRunTelemetry
| TestDiscoverytTelemetry
| FeedbackTelemetry
| TerminalTelemetry
| DebuggerTelemetryV2
| SettingsTelemetry;
3 changes: 2 additions & 1 deletion src/test/aaFirstTest/aaFirstTest.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@

import { expect } from 'chai';
import { extensions } from 'vscode';
import { PVSC_EXTENSION_ID } from '../../client/common/constants';
import { initialize } from '../initialize';

// NOTE:
Expand All @@ -17,6 +18,6 @@ suite('Activate Extension', () => {
await initialize();
});
test('Python extension has activated', async () => {
expect(extensions.getExtension('ms-python.python')!.isActive).to.equal(true, 'Extension has not been activated');
expect(extensions.getExtension(PVSC_EXTENSION_ID)!.isActive).to.equal(true, 'Extension has not been activated');
});
});
3 changes: 2 additions & 1 deletion src/test/initialize.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import * as path from 'path';
import * as vscode from 'vscode';
import { IExtensionApi } from '../client/api';
import { PythonSettings } from '../client/common/configSettings';
import { PVSC_EXTENSION_ID } from '../client/common/constants';
import { clearPythonPathInWorkspaceFolder, PYTHON_PATH, resetGlobalPythonPathSetting, setPythonPathInWorkspaceRoot } from './common';

export * from './constants';
Expand Down Expand Up @@ -32,7 +33,7 @@ export async function initialize(): Promise<any> {
PythonSettings.dispose();
}
export async function activateExtension() {
const extension = vscode.extensions.getExtension<IExtensionApi>('ms-python.python')!;
const extension = vscode.extensions.getExtension<IExtensionApi>(PVSC_EXTENSION_ID)!;
if (extension.isActive) {
return;
}
Expand Down
3 changes: 2 additions & 1 deletion src/test/performance/load.perf.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import * as fs from 'fs-extra';
import { EOL } from 'os';
import * as path from 'path';
import { commands, extensions } from 'vscode';
import { PVSC_EXTENSION_ID } from '../../client/common/constants';
import { StopWatch } from '../../utils/stopWatch';

const AllowedIncreaseInActivationDelayInMS = 500;
Expand All @@ -22,7 +23,7 @@ suite('Activation Times', () => {
return;
}
test(`Capture Extension Activation Times (Version: ${process.env.ACTIVATION_TIMES_EXT_VERSION}, sample: ${sampleCounter})`, async () => {
const pythonExtension = extensions.getExtension('ms-python.python');
const pythonExtension = extensions.getExtension(PVSC_EXTENSION_ID);
if (!pythonExtension) {
throw new Error('Python Extension not found');
}
Expand Down
4 changes: 2 additions & 2 deletions src/test/performanceTest.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ import * as download from 'download';
import * as fs from 'fs-extra';
import * as path from 'path';
import * as request from 'request';
import { EXTENSION_ROOT_DIR } from '../client/common/constants';
import { EXTENSION_ROOT_DIR, PVSC_EXTENSION_ID } from '../client/common/constants';

const NamedRegexp = require('named-js-regexp');
const StreamZip = require('node-stream-zip');
Expand Down Expand Up @@ -123,7 +123,7 @@ class TestRunner {
}

private async getReleaseVersion(): Promise<string> {
const url = 'https://marketplace.visualstudio.com/items?itemName=ms-python.python';
const url = `https://marketplace.visualstudio.com/items?itemName=${PVSC_EXTENSION_ID}`;
const content = await new Promise<string>((resolve, reject) => {
request(url, (error, response, body) => {
if (error) {
Expand Down