Compatibility with Windows Store Python

[rpanderson]

Trying to reproduce #1 on another Windows 10 host, I came across another issue related to Windows Store distribution. Specifically, I'm using Python 3.8 from Windows Store:

Python 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 23:03:10) [MSC v.1916 64 bit (AMD64)] on win32

In shell.py > _ModuleConfig > _get_appid the following fails on store-bought Python:

interpreter_dir = Path(sys.executable).resolve().parent

with:

OSError: [WinError 1920] The file cannot be accessed by the system: 'C:\\Users\\rpanderson\\AppData\\Local\\Microsoft\\WindowsApps\\python.exe'

A minimal example is:

>>> from pathlib import Path
>>> import sys
>>> Path(sys.executable).resolve()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.752.0_x64__qbz5n2kfra8p0\lib\pathlib.py", line 1172, in resolve
    s = self._flavour.resolve(self, strict=strict)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.752.0_x64__qbz5n2kfra8p0\lib\pathlib.py", line 205, in resolve
    s = self._ext_to_normal(_getfinalpathname(s))
OSError: [WinError 1920] The file cannot be accessed by the system: 'C:\\Users\\rpanderson\\AppData\\Local\\Microsoft\\WindowsApps\\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\\python.exe'

It looks like virtualenv had to contend with this, see:

• [20.0.10] windows store support pypa/virtualenv#1709
• Fix Windows Store support pypa/virtualenv#1725

There's this relevant open issue for pathlib.WindowsPath.resolve as well, in which this message states:

For example, resolve() fails for a path in another user's profile, which grants access only to the user, system, and administrators.

In what I observed here, I think it tries to resolve the executable to C:\Program Files\WindowsApps which requires elevation to access.

[rpanderson]

I wonder if desktop-app can/should leverage a lot of the infrastructure in virtualenv, e.g. virtualenv.discovery.py_info.PythonInfo.

[chrisjbillington]

Ah, that's annoying.

I'm only resolving it to ensure the case is right, since venv makes all the paths in sys lower case which gives different hashes than the correct case - but I can os.path.normcase instead of resolve() so that'll be fine.

Will look into whether there's much we can leverage from virtualenv. I just want a string unique to the Python installation for use in the appids, I don't actually need to access the directory..

[chrisjbillington]

Should be fixed by f64a318. However, this will change all the appids, so appid stuff will be expected to break until uninstalling and reinstalling a shortcut.

[rpanderson]

Installing from scratch:

C:\Users\rpanderson>pip install -e git+https://github.com/chrisjbillington/desktop-app
C:\Users\rpanderson>pip install -e src\desktop-app\example
C:\Users\rpanderson>desktop-app install oink
 -> created C:\Users\rpanderson\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Old McDonald's Farm\Oink App.lnk
C:\Users\rpanderson>desktop-app install oink --path C:\Users\rpanderson\Desktop
 -> created C:\Users\rpanderson\Desktop\Oink App.lnk

The additional shortcut creation was required as the start menu on the second host I'm testing this on isn't populated by %APPDATA\Microsoft\Windows\Start Menu\Programs but instead by %PROGRAMDATA\Microsoft\Windows\Start Menu\Programs.

The resulting shortcut is broken, i.e. does not point to oink-gui correctly. The shortcut target is:

C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.752.0_x64__qbz5n2kfra8p0\Scripts\oink-gui

but for this store-bought Python the Scripts folder is actually:

C:\Users\rpanderson\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\Scripts

If I amend the shortcut target, I see #1, as before.

[chrisjbillington]

Great.

Will look into the difference between %APPDATA% and %PROGRAMDATA%. I was not aware that different Windows installations differed in how they populate the start menu.

The scripts path being wrong I should be able to fix:

It looks like the Scripts directory being wrong is due to to the Windows store doing kind of the equivalent of a pip install --user, or of modifying the install prefix to ensure user packages are separate (from what? What's in site-packages if it's a clean Python installation...?). desktop-app is knowingly incompatible with that at the moment, here's what I had about it in a draft of the README.

Limitations
===========

`desktop-app` is compatible with packages installed system-wide and within virtual
environments. However, it is not compatible with packages installed either with `pip
install --user`, or with a modified `--prefix` such as how Debian-based distributions
install packages to `/usr/local/` when `pip` is run with `sudo`.

This is because `desktop-app` can't find the `entry_points` scripts in these cases in
order to point start menu shortcuts and .desktop files at them. It just looks in
`sysconfig.get_path('scripts')`, and turns out that looking in all other possible places
is non-trivial. Obviously it's possible since `pip uninstall` manages to remove these
files (evidence that it can find them!), but I haven't figured it out yet and it's not
really worth it when `sudo pip install` and `pip install --user` are usually bad ideas
anyway.

However, it is possible, just means I have to hook into the same logic as pip in order to determine the path to local packages and local scripts, and determine which prefix a module was installed to (its import path does not suffice since editable installs are located elsewhere and linked to by a .egg-link file for metadata and easy-install.pth for import able modules).

So I can do that! I had thought prior to now it was out of scope enough to not be worth it, but being compatible with windows store Python is a good reason.

[chrisjbillington]

It occurs to me that this might be the source of the issues we're seeing, and if so there might not be much we can do about it within the current model of setting AppUserModel_ids:

3.2.1. Known Issues
Because of restrictions on Microsoft Store apps, Python scripts may not have full write access to shared locations such as TEMP and the registry. Instead, it will write to a private copy. If your scripts must modify the shared locations, you will need to install the full installer.

[chrisjbillington]

the start menu on the second host I'm testing this on isn't populated by %APPDATA\Microsoft\Windows\Start Menu\Programs but instead by %PROGRAMDATA\Microsoft\Windows\Start Menu\Programs.

From my cursory googling, it sounds like in normal circumstances, both are used.

How sure are you that this is a difference in configuration on your system, as opposed to say, Windows failing to update its start menu database in a timely manner when a new shortcut is created (this SO comment hints at that possibility)? If there's a possibility of the latter, I wonder if 4322167 might have helped - with that change I see a visual flicker of the taskbar as Windows reloads things, and the shortcut is immediately in the start menu (whereas otherwise I have to wait some seconds for it to appear).

Nonetheless I'll experiment with installing to %PROGRAMDATA%

[chrisjbillington]

The broken shortcuts issue ought to be fixed by 0ea163e (verified by cursory testing).

desktop-app now checks which of the (possibly many) installation locations the module is in and determines the appropriate scripts directory as one of these three:

• sysconfig.get_path('scripts') for packages installed directly to the Python environment (i.e. any path in the list site.getsitepackages()),
• sysconfig.get_path('scripts', scheme='nt_user' if WINDOWS else 'posix_user') for packages installed to site.getusersitepackages() (this is the case with microsoft-store-bought Python - also the default for pip-install on Ubuntu)
• Or a corner case, /usr/local/bin if the package is installed anywhere in /usr/local (even if this is a path in site.getsitepackages()). This a Debian quirk - it's where things go when installed with sudo pip install in Debian/Ubuntu/etc.

One remaining use case is "the user ran python setup.py install --root=<some_random_dir> with a completely custom install location, not even in the import path". This occurs for example when building packages for Linux distros - you're 'installing' everything to a subdirectory to be zipped up and shipped, not installing on your system. For that case I'll add a --root argument to desktop-app install that will just assume <root>/{sysconfig.get_path('scripts')} is the scripts dir (if this seems like an improbable use case, it's how I plan to install stable labscript packages to my own system - via a proper distro package).

[chrisjbillington]

I'm seeing the shortcuts not even being created. Python things its creating them, but they are not visible in the file browser. Python can see them however. Here you can see dir shows no such folder Old Macdonald's Farm, but python os.listdir does:

This must be because of this (already speculated about above with regard to appid setting):

3.2.1. Known Issues
Because of restrictions on Microsoft Store apps, Python scripts may not have full write access to shared locations such as TEMP and the registry. Instead, it will write to a private copy. If your scripts must modify the shared locations, you will need to install the full installer.

[chrisjbillington]

Found the "private copy" of the start menu that store-bought Python sees!