ms-python.python vscode extension fails with "... Couldn't find a valid ICU package ..."

[JamesReynolds]

Describe the bug
After installing ms-python.python vscode extension the extension fails to start:

Process terminated. Couldn't find a valid ICU package installed on the system. Set the configuration flag System.Globalization.Invariant to true if you want to run with no globalization support.
   at System.Environment.FailFast(System.String)
   at System.Globalization.GlobalizationMode.GetGlobalizationInvariantMode()
   at System.Globalization.GlobalizationMode..cctor()
   at System.Globalization.CultureData.CreateCultureWithInvariantData()
   at System.Globalization.CultureData.get_Invariant()
   at System.Globalization.CultureInfo..cctor()
   at System.FormattableString.Invariant(System.FormattableString)
   at Microsoft.Python.Core.Services.ServiceManager.AddService(System.Object, System.Type)
   at Microsoft.Python.LanguageServer.Services.CoreShell..ctor()
   at Microsoft.Python.LanguageServer.Services.CoreShell.Create()
   at Microsoft.Python.LanguageServer.Server.Program.Main(System.String[])
Aborted (core dumped)

To Reproduce
Steps to reproduce the behavior:

1. Clone nkgpgs at 5ccbcf2
2. Install using:

let 
  unstable = import nixpkgs/default.nix {};
in
  unstable.vscode-with-extensions.override {
        vscodeExtensions = with unstable.vscode-extensions; [
          ms-python.python
        ];
      }

Expected behavior
The extension loads correctly when a python file is loaded in vscode.
Also, when directly running the language server (Microsoft.Python.LanguageServer) there is no output.

Additional context
The binary can be fixed using:

patchelf --set-rpath /nix/store/5d57jfb6wk4m5hl7dfv46hc0wlnzkjlq-icu4c-64.2/lib System.Globalization.Native.so
patchelf --set-rpath /nix/store/z26dmlk813907qs472b1caj7pjdmv9bs-openssl-1.1.1d/lib System.Security.Cryptography.Native.OpenSsl.so

Unfortunately, the postFixup hook in python/default.nix appears to run before the autoPatchelfHook stage - so any changes added there are undone.

Metadata

• system: "x86_64-linux"
• host os: Linux 4.19.84, NixOS, 19.09.1292.e1843646b04 (Loris)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.3
• channels(root): "nixos-19.09.1292.e1843646b04"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

[JamesReynolds]

@jonringer thank you for the fast response! I can't seem to get this to work using my install method - applying the patch adds to the rpath of Microsoft.Python.LanguageServer - but doesn't add to the path of System.Globalization.Native.so or System.Security.Cryptography.Native.OpenSsl.so.

When I directly run autopatchElf on both those files in one of the hooks it doesn't seem to modify the rpaths either.

This fixes my problem:

--- a/pkgs/build-support/setup-hooks/auto-patchelf.sh
+++ b/pkgs/build-support/setup-hooks/auto-patchelf.sh
@@ -104,11 +104,11 @@ autoPatchelfFile() {
     local interpreter="$(< "$NIX_CC/nix-support/dynamic-linker")"
     if isExecutable "$toPatch"; then
         patchelf --set-interpreter "$interpreter" "$toPatch"
-        if [ -n "$runtimeDependencies" ]; then
-            for dep in $runtimeDependencies; do
-                rpath="$rpath${rpath:+:}$dep/lib"
-            done
-        fi
+    fi
+    if [ -n "$runtimeDependencies" ]; then
+        for dep in $runtimeDependencies; do
+            rpath="$rpath${rpath:+:}$dep/lib"
+        done
     fi

...but I don't know the repercussions that might have elsewhere.

[JamesReynolds]

OK, this is even stranger... it can't find SSL with everything in the rpath - but can find it when the rpaths are very specific. So this works:

$ find ~/.vscode -name System*.so -exec basename {} \; -exec patchelf --print-rpath {} \;
System.Globalization.Native.so
/nix/store/5d57jfb6wk4m5hl7dfv46hc0wlnzkjlq-icu4c-64.2/lib
System.Net.Security.Native.so
/nix/store/2fhgqdw9i45va31ls1fjsdywa0zqgppx-libkrb5-1.17/lib
System.Native.so

System.Net.Http.Native.so
/nix/store/h9vdy2fkz5i82ng5j15bsv0nrxpn9jlf-curl-7.66.0/lib
System.IO.Compression.Native.so
/nix/store/cg9l4lrvfc9azjsdzgfaxkcbfsmyyzmg-zlib-1.2.11/lib
System.Security.Cryptography.Native.OpenSsl.so
/nix/store/z26dmlk813907qs472b1caj7pjdmv9bs-openssl-1.1.1d/lib

but this does not:

$ find /nix/store/xgbxb46ff4nch7rlsv58ll1f8505jzha-vscode-extension-ms-python-python-2019.10.44104/share/vscode/extensions/ms-python.python/languageServer.0.4.24/ -name System*.so -exec basename {} \; -exec patchelf --print-rpath {} \;
System.Globalization.Native.so
/nix/store/bz63fld1a1hwc1c6cma7zyjbq20fkfln-icu4c-64.2/lib:/nix/store/3imfnws6p22wv5xhxdnr0rwbglackf8c-openssl-1.1.1d-bin/lib
System.Net.Security.Native.so
/nix/store/bz63fld1a1hwc1c6cma7zyjbq20fkfln-icu4c-64.2/lib:/nix/store/3imfnws6p22wv5xhxdnr0rwbglackf8c-openssl-1.1.1d-bin/lib:/nix/store/fvj8qzbcjsr4b2zmnm3v4hsa3hv0z21a-libkrb5-1.17/lib
System.Native.so
/nix/store/bz63fld1a1hwc1c6cma7zyjbq20fkfln-icu4c-64.2/lib:/nix/store/3imfnws6p22wv5xhxdnr0rwbglackf8c-openssl-1.1.1d-bin/lib
System.Net.Http.Native.so
/nix/store/bz63fld1a1hwc1c6cma7zyjbq20fkfln-icu4c-64.2/lib:/nix/store/3imfnws6p22wv5xhxdnr0rwbglackf8c-openssl-1.1.1d-bin/lib:/nix/store/bf8mji0pvnmi2kh1xk87rs7xivn30kq3-curl-7.67.0/lib
System.IO.Compression.Native.so
/nix/store/bz63fld1a1hwc1c6cma7zyjbq20fkfln-icu4c-64.2/lib:/nix/store/3imfnws6p22wv5xhxdnr0rwbglackf8c-openssl-1.1.1d-bin/lib:/nix/store/8wanm00dzn5vlajz1rznbf6dbj8931d6-zlib-1.2.11/lib
System.Security.Cryptography.Native.OpenSsl.so
/nix/store/bz63fld1a1hwc1c6cma7zyjbq20fkfln-icu4c-64.2/lib:/nix/store/3imfnws6p22wv5xhxdnr0rwbglackf8c-openssl-1.1.1d-bin/lib

I can do a bit more investigation to see if I can pull a stack trace from vscode running the server - I've got some other bits to do this morning (UK) first though.

Edit - It also fails if openssl / icu are flipped around so openssl occurs first.

[jonringer]

I think the underlying failure, is that isExecutable() returns false on the shared libraries, thus skips editing their rpaths

[JamesReynolds]

I've just checked without the additional runtimeDependencies and unfortunately neither of System.Globalization.Native.so or System.Security.Cryptography.Native.OpenSsl.so parse to find icu or openssl as a dependency. If you add both rpaths to System.Security.Cryptography.Native.OpenSsl.so then it breaks it - it must have only SSL in its rpath - but System.Globalization.Native.so must have icu... I get the feeling this can only be fixed by manually calling patchelf.

Is there a way I can add in a hook to do this? postfixUp seems to get called before autopatchElf is run - so the paths get destroyed by the later run.

[jonringer]

This seems to work on unstable (with no edits), not sure why stable is broken

[JamesReynolds]

Shall I upgrade dependencies from stable to locate the fix? Is that a necessary step to completing or can the fix be confirmed in unstable?

[jul1u5]

It doesn't work for me on unstable NixOS 20.03pre204199.3140fa89c51 (Markhor).
I get the same error.

[jul1u5]

Still does not work for me. Are there any other workarounds for using Python extension on VS Code?
Perhaps this is related: dotnet/core#2186 (comment)

[jonringer]

if you look at the strace, it's looking for a libicuuc.so which can be found in the icu package

How to expose this? I'm not sure

[jonringer]

ran into a similar issue with dotnet-sdk. There's likely an executable or shared library that doesn't have it's rpath set correctly.

[mjlbach]

Also interested in this as nvim-lsp uses the dotnet version of microsoft's python language server, and if I should separately package the server

[jonringer]

I believe the python language server is already packaged

[mjlbach]

@jonringer I wasn't able to find it, it's a subset of the vscode extension package, but for use with nvim/vim it should be on PATH

[jonringer]

I was thinking of python3Packages.python-language-server but that's a different upstream. nevermind

[marcus233]

Also interested in this as nvim-lsp uses the dotnet version of microsoft's python language server, and if I should separately package the server

I'd be interested in a separate server package. Emacs can also use mspyls (and is my preference because it's faster and supports 3.8. palantir's pyls does not.)

[pedrovelho]

I decided to write this the hope that someone with a problem like mine arrive here. I am not sure it is related but I really did not find any similar issue somewhere else. I am experiencing a similar problem, however my extension is installed differently. I am on 20.03. I just added the package vscode-with-extensions, and my python extension is failing. The file ~/.vscode/extensions/ms-python.python-2020.4.74986/languageServer.0.5.45/Microsoft.Python.LanguageServer exists and permissions are correct. However when trying to run it from my default zsh shell I have:

> ~/.vscode/extensions/ms-python.python-2020.4.74986/languageServer.0.5.45/Microsoft.Python.LanguageServer
zsh: no such file or directory: /home/pvelho/.vscode/extensions/ms-python.python-2020.4.74986/languageServer.0.5.45/Microsoft.Python.LanguageServer

Using bash I get the cannot execute binary file as when the ELF is not meant to run on the architecture:

/usr/bin/env bash ./Microsoft.Python.LanguageServer
./Microsoft.Python.LanguageServer: ./Microsoft.Python.LanguageServer: cannot execute binary file

I used ldd and discovered a missing library dependency:

> ldd Microsoft.Python.LanguageServer
	linux-vdso.so.1 (0x00007ffc3dfdc000)
	libpthread.so.0 => /nix/store/6m2k8kx8h216jlx9dg3lp4m90bz05yck-glibc-2.30/lib/libpthread.so.0 (0x00007f4b9b4cd000)
	libdl.so.2 => /nix/store/6m2k8kx8h216jlx9dg3lp4m90bz05yck-glibc-2.30/lib/libdl.so.2 (0x00007f4b9b4c8000)
	libstdc++.so.6 => not found
	libm.so.6 => /nix/store/6m2k8kx8h216jlx9dg3lp4m90bz05yck-glibc-2.30/lib/libm.so.6 (0x00007f4b9b388000)
	libgcc_s.so.1 => /nix/store/6m2k8kx8h216jlx9dg3lp4m90bz05yck-glibc-2.30/lib/libgcc_s.so.1 (0x00007f4b9b36e000)
	libc.so.6 => /nix/store/6m2k8kx8h216jlx9dg3lp4m90bz05yck-glibc-2.30/lib/libc.so.6 (0x00007f4b9b1af000)
	/lib64/ld-linux-x86-64.so.2 => /nix/store/6m2k8kx8h216jlx9dg3lp4m90bz05yck-glibc-2.30/lib64/ld-linux-x86-64.so.2 (0x00007f4b9b4f0000)

Sorry if I post this in the wrong place. Please let me know if this is related or not, and if you have any idea how to fix it i would appreciate.

[thomasjm]

Hi all -- I just created a PR that builds the Microsoft Python Language Server independently from VS Code. It should make it easier to use this language server with other editors and might help with the issue mentioned in this thread. Any testing or review would be appreciated. PR is here: #85930

[thomasjm]

@pedrovelho you shouldn't run bash ./Microsoft.Python.LanguageServer, this is a binary file and not a Bash script. Maybe you can poke around in ~/.vscode/extensions/ms-python.python-2020.4.74986/languageServer.0.5.45/ and check that the file you're trying to run has the executable bit set and isn't a broken symlink or something? If it ends up being a linker problem it might be relevant to this thread, otherwise I think it's something else.

[jraygauthier]

It is possible to patch this kind of thing as follow;

    patchelf_add_icu_as_needed() {
      declare elf="''${1?}"
      declare icu_major_v="${
        with builtins; head (splitVersion (parseDrvName icu.name).version)}"

      for icu_lib in icui18n icuuc icudata; do
        patchelf --add-needed "lib''${icu_lib}.so.$icu_major_v" "$elf"
      done
    }

    patchelf_common() {
      declare elf="''${1?}"

      patchelf_add_icu_as_needed "$elf"
      patchelf --add-needed "libssl.so" "$elf"
      patchelf --set-interpreter "$(cat $NIX_CC/nix-support/dynamic-linker)" \
        --set-rpath "${lib.makeLibraryPath [ stdenv.cc.cc openssl.out icu.out ]}:\$ORIGIN" \
        "$elf"
    }
patchelf_common my-elf-requiring-icu-and-ssl

[jonringer]

I was thinking of just making vscode into a buildFHSUserEnv and just add dotnet dependencies and a few other runtimes to the closure

This would solve a lot of other issues with extensions having pre-built binaries

[stale]

I marked this as stale due to inactivity. → More info