Language service fails when using Python 2.7 interpreter

[DonJayamanne]

• Open a workspace with Python code
• Select Python 2.7 interpreter
• Error message is displayed, see below

@MikhailArkhipov

[MikhailArkhipov]

Specific Python? Works on Windows & Ubuntu. Might be threading issue - is it consistent when you flip between interpreters? If you can, attach to dotnet.exe from VSC (.NET Core Attach in C# extension), as in https://github.com/Microsoft/vscode-python/blob/master/CONTRIBUTING%20-%20PYTHON_ANALYSIS.md#debugging-c-code-in-python-analysis-engine.

[patrys]

I have a project (under an NDA unfortunately) where I can trigger this with 100% success (or, should I say, 100% failure). I'll try to isolate a package that triggers it or trace the ptvs code if time permits.

[DonJayamanne]

If you can, attach to dotnet.exe from VSC (.NET Core Attach in C# extension), as in /CONTRIBUTING%20-%20PYTHON_ANALYSIS.md@master#debugging-c-code-in-python-analysis-engine.

todo @DonJayamanne

[MilkyHearts]

Works prefect in Python3 but not 2.7 >.<

[patrys]

JS stack trace

Object reference not set to an instance of an object.: Error: Object reference not set to an instance of an object.
    at new ResponseError (/Users/patrys/.vscode-insiders/extensions/ms-python.python-2018.6.0/node_modules/vscode-jsonrpc/lib/messages.js:46:28)
    at handleResponse (/Users/patrys/.vscode-insiders/extensions/ms-python.python-2018.6.0/node_modules/vscode-jsonrpc/lib/main.js:430:48)
    at processMessageQueue (/Users/patrys/.vscode-insiders/extensions/ms-python.python-2018.6.0/node_modules/vscode-jsonrpc/lib/main.js:258:17)
    at Immediate.<anonymous> (/Users/patrys/.vscode-insiders/extensions/ms-python.python-2018.6.0/node_modules/vscode-jsonrpc/lib/main.js:242:13)
    at runCallback (timers.js:672:20)
    at tryOnImmediate (timers.js:645:5)
    at processImmediate [as _immediateCallback] (timers.js:617:5)

Value cannot be null.
Parameter name: type: Error: Value cannot be null.
Parameter name: type
    at new ResponseError (/Users/patrys/.vscode-insiders/extensions/ms-python.python-2018.6.0/node_modules/vscode-jsonrpc/lib/messages.js:46:28)
    at handleResponse (/Users/patrys/.vscode-insiders/extensions/ms-python.python-2018.6.0/node_modules/vscode-jsonrpc/lib/main.js:430:48)
    at processMessageQueue (/Users/patrys/.vscode-insiders/extensions/ms-python.python-2018.6.0/node_modules/vscode-jsonrpc/lib/main.js:258:17)
    at Immediate.<anonymous> (/Users/patrys/.vscode-insiders/extensions/ms-python.python-2018.6.0/node_modules/vscode-jsonrpc/lib/main.js:242:13)
    at runCallback (timers.js:672:20)
    at tryOnImmediate (timers.js:645:5)
    at processImmediate [as _immediateCallback] (timers.js:617:5)

[DonJayamanne]

@MikhailArkhipov Does this work on Python 2.7 for you?

[patrys]

C# stack trace

[Error - 12:45:42] System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.PythonTools.Analysis.Values.BuiltinNamespace`1.Equals(Object obj)
   at System.Linq.Enumerable.All[TSource](IEnumerable`1 source, Func`2 predicate)
   at Microsoft.PythonTools.Analysis.AnalysisValue.Microsoft.PythonTools.Analysis.IAnalysisSet.Union(IEnumerable`1 items, Boolean canMutate)
   at Microsoft.PythonTools.Analysis.AnalysisValueSetExtensions.GetMember(IAnalysisSet self, Node node, AnalysisUnit unit, String name)
   at Microsoft.PythonTools.Analysis.Values.LazyValueInfo.ResolveOnce(AnalysisUnit unit, ResolutionContext context, Boolean& cacheResult)
   at Microsoft.PythonTools.Analysis.Values.LazyValueInfo.Resolve(AnalysisUnit unit, ResolutionContext context)
   at Microsoft.PythonTools.Analysis.AnalysisValueSetExtensions.Resolve(IAnalysisSet self, AnalysisUnit unit, ResolutionContext context, Boolean& changed)
   at Microsoft.PythonTools.Analysis.Values.LazyValueInfo.Resolve(AnalysisUnit unit, ResolutionContext context)
   at Microsoft.PythonTools.Analysis.AnalysisValueSetExtensions.Resolve(IAnalysisSet self, AnalysisUnit unit, ResolutionContext context, Boolean& changed)
   at Microsoft.PythonTools.Analysis.Analyzer.ExpressionEvaluator.AssignTo(Node assignStmt, Expression left, IAnalysisSet values)
   at Microsoft.PythonTools.Analysis.Analyzer.DDG.Walk(AssignmentStatement node)
   at Microsoft.PythonTools.Parsing.Ast.AssignmentStatement.Walk(PythonWalker walker)
   at Microsoft.PythonTools.Analysis.Analyzer.DDG.Walk(SuiteStatement node)
   at Microsoft.PythonTools.Parsing.Ast.SuiteStatement.Walk(PythonWalker walker)
   at Microsoft.PythonTools.Analysis.Analyzer.DDG.Walk(TryStatement node)
   at Microsoft.PythonTools.Parsing.Ast.TryStatement.Walk(PythonWalker walker)
   at Microsoft.PythonTools.Analysis.Analyzer.DDG.Walk(SuiteStatement node)
   at Microsoft.PythonTools.Parsing.Ast.SuiteStatement.Walk(PythonWalker walker)
   at Microsoft.PythonTools.Analysis.Analyzer.FunctionAnalysisUnit.AnalyzeWorker(DDG ddg, CancellationToken cancel)
   at Microsoft.PythonTools.Analysis.Analyzer.DDG.Analyze(Deque`1 queue, CancellationToken cancel, Action`1 reportQueueSize, Int32 reportQueueInterval)
   at Microsoft.PythonTools.Analysis.PythonAnalyzer.AnalyzeQueuedEntries(CancellationToken cancel)
   at Microsoft.PythonTools.Intellisense.AnalysisQueue.GroupAnalysis.Analyze(CancellationToken cancel)
   at Microsoft.PythonTools.Intellisense.AnalysisQueue.Worker(Object threadStarted)

[MikhailArkhipov]

Yes, but this probably code specific scenario. I will add null check.

[zooba]

Guessing it's in here:

        public override bool Equals(object obj) {
            if (obj is BuiltinNamespace<MemberContainerType> bn && GetType() == bn.GetType()) {
                return _type.Equals(bn._type);
            }
            return false;
        }

IIRC, _type can be null in some circumstances (typically not-quite-correct code). We should check whether they're both null before doing the .Equals (and we need the .Equals, because == may not call the right methods)

[MikhailArkhipov]

OK

[albireox]

I'm having the same problem with Python 2.7 (I think it affect all my codes) but also with 3.6.5/3.7.0 in some cases. For at least a project I get

[Error - 13:11:25] System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.PythonTools.Analysis.Values.BuiltinNamespace`1.GetHashCode()
   at Microsoft.PythonTools.Analysis.AnalysisSetDetails.AnalysisHashSet.Contains(AnalysisValue key)
   at Microsoft.PythonTools.Analysis.AnalysisSetDetails.AnalysisHashSet.Add(AnalysisValue item, Boolean canMutate)
   at Microsoft.PythonTools.Analysis.AnalysisSetDetails.AnalysisSetTwoObject.Union(IEnumerable`1 items, Boolean canMutate)
   at Microsoft.PythonTools.Analysis.AnalysisValueSetExtensions.GetMember(IAnalysisSet self, Node node, AnalysisUnit unit, String name)
   at Microsoft.PythonTools.Analysis.Analyzer.ExpressionEvaluator.EvaluateWorker(Node node)
   at Microsoft.PythonTools.Analysis.Analyzer.ExpressionEvaluator.Evaluate(IList`1 nodes)
   at Microsoft.PythonTools.Analysis.Analyzer.ExpressionEvaluator.EvaluateCall(ExpressionEvaluator ee, Node node)
   at Microsoft.PythonTools.Analysis.Analyzer.ExpressionEvaluator.EvaluateWorker(Node node)
   at Microsoft.PythonTools.Analysis.Analyzer.DDG.Walk(AssignmentStatement node)
   at Microsoft.PythonTools.Parsing.Ast.AssignmentStatement.Walk(PythonWalker walker)
   at Microsoft.PythonTools.Analysis.Analyzer.DDG.Walk(SuiteStatement node)
   at Microsoft.PythonTools.Parsing.Ast.SuiteStatement.Walk(PythonWalker walker)
   at Microsoft.PythonTools.Analysis.Analyzer.FunctionAnalysisUnit.AnalyzeWorker(DDG ddg, CancellationToken cancel)
   at Microsoft.PythonTools.Analysis.Analyzer.DDG.Analyze(Deque`1 queue, CancellationToken cancel, Action`1 reportQueueSize, Int32 reportQueueInterval)
   at Microsoft.PythonTools.Analysis.PythonAnalyzer.AnalyzeQueuedEntries(CancellationToken cancel)
   at Microsoft.PythonTools.Intellisense.AnalysisQueue.GroupAnalysis.Analyze(CancellationToken cancel)
   at Microsoft.PythonTools.Intellisense.AnalysisQueue.Worker(Object threadStarted)

with Python 3 (from pyenv on macOS).

The Analyzing workspace, XX items remaining ... message in the status bar gets stuck.

[DonJayamanne]

I'm no longer able to replicate this issue.

[albireox]

Still showing up for me.

[ventilooo]

$ code --version
>> 1.25.1
>> 1dfc5e557209371715f655691b1235b6b26a06be
>> x64

I'm with ubuntu 16.04
I'm having the same issue as @DonJayamanne.

[patrys]

I've spent the entire evening debugging the language server in Visual Studio 2017 (btw, the docs are out of date). I was not able to attach to the process quick enough as it crashed during initialization. In the end I've resorted to adding Debugger.Launch() in Python/Product/Analysis/LanguageServer/Server.cs.

This allowed me to discover which part of the code causes the crash. It turns out that to reproduce it's enough to pip install future.

Now to the problem itself:

I'm not yet sure why but the language server visits future.builtins during LoadKnownTypesAsync which seems like it should only load the builtin types:

https://github.com/Microsoft/PTVS/blob/ed2c0ff379e521f30aa8add9cb511d273058b305/Python/Product/Analysis/PythonAnalyzer.cs#L183

It then breaks when trying to import the future.types.newbytes module because this._analyzer is null:

https://github.com/Microsoft/PTVS/blob/ed2c0ff379e521f30aa8add9cb511d273058b305/Python/Product/Analysis/Interpreter/Ast/AstPythonInterpreter.cs#L207-L214

The analyzer is not set until after known type loading is complete:

https://github.com/Microsoft/PTVS/blob/ed2c0ff379e521f30aa8add9cb511d273058b305/Python/Product/Analysis/PythonAnalyzer.cs#L185

I can prevent it from crashing (but can't say if/what else become broken by this workaround) by calling _interpreter.Initialize(this) before LoadKnownTypesAsync(token).

[patrys]

I know why it ends up parsing future.builtins while trying to load known types. futures installs a module named builtins and it's loaded by site.py. That module in turn contains:

if sys.version_info[0] < 3:
    from __builtin__ import *
    # Overwrite any old definitions with the equivalent future.builtins ones:
    from future.builtins import *

[patrys]

Paging @brettcannon, @MikhailArkhipov and @zooba as I don't think I can fix it on my own.

[woodvillage]

I have the same error.

[brettcannon]

@patrys thanks for digging into this! The folks who can help with this are currently out on vacation so it might take them a little while to get around to this.

[Gocy015]

Experiencing the same issue.. all my python 2.7 project is not able to run ..

[Prof-pengyin]

Same problem here.

[wwwzyb2002]

Same error, expect fixed soon.

[patrys]

Please vote on the issue instead of writing "me too".

[MikhailArkhipov]

@patrys - if you uncomment

// #define WAIT_FOR_DEBUGGER

in AnalysisVsc/Program.cs the process will wait until you attach the debugger.

[hyunghunny]

The same issue happened to me. I've tried to fix it and finally I know that how can it be resolved:

• Closes the python project and deletes /.vscode under the project folder
• Restarts VS Code and re-configure your python project.

I thought that the previous setting for Python is not compatible with the latest one.

Hope above is helpful to someone who suffers from the same issue.

[patrys]

@hyunghunny Your workaround works by effectively removing the python.jediEnabled setting. That's all you need to do, no need to delete entire folders. Unfortunately it does not fix anything, it just no longer uses the new language service (where the bug used to live).

[PeterSorrell]

Based on your last message regarding "python.jediEnabled", I have just inserted the following statement into User Settings and Workspace Settings and I still get the error message:
{
"python.pythonPath": "C:\Program Files\IronPython 2.7\ipy.exe",
"python.jediEnabled": false,
"python.linting.pylintEnabled": false
}
Any more news on a fix for this problem?
Thanks.

[PeterSorrell]

Update:
After the changes to User Settings and Workspace Settings I found that the debugger would not start, so I removed "python.jediEnabled": false, which made the debugger work again. Very strange!

[albireox]

This still fails for me even if I have removed my languageServer directory and forced the language server to be redownloaded. Is that expected? Maybe the version of the LS vscode-python is downloading doesn't incorporate those changes yet.

[brettcannon]

@albireox it doesn't have any fixes yet.

@PeterSorrell do realize we don't have official support for IronPython.

[MikhailArkhipov]

@brettcannon - fix is in LS, we can rev up insiders build.

[norheim]

Don't know if this will work for others, but I turned the jediEnabled setting off and then on(so first to false and then back to true) and now it is working. Good luck!

[MikhailArkhipov]

@norheim - the bug was a thread race condition so it may work sometimes. If it was consistent, we'd seen it before :-)

[brettcannon]

For people bumping up against this issue, if you install a development build of the extension then it will download a newer version of the language server which should have a fix in it.

(Even if you already have a development build installed, please uninstall it and install a newer copy as we updated the URL that the language server is downloaded from.)

[qyloxe]

confirm: after installation of development build (vsix) the problem disappeared. GJ!