Detect whether Win10 has linux distro installed

[AenBleidd]

Looks like GPUGrid started to test new application that can be run on linux installed on Win10: https://www.gpugrid.net/forum_thread.php?id=4716&nowrap=true#48980
As I understand BOINC can't detect the presence of linux on Win10.
I think it is a powerful option so boinc client should have an option to make such detection and send it to project.

[JuhaSointusalo]

Some thoughts about WSL support.

• Platform vs something else
  Declaring support for Linux apps via alt platform would be nice but WSL can't run all Linux apps yet. GPU apps for example won't work. Linux apps also need to be run with WSL launcher. The same way VirtualBox is supported should work. Some extra info in scheduler request, plan class on server and a wrapper with the app.
• Multiple distros
  Since Fall Creators Update there is now several distros available. Some projects may be picky about which Linux distros they support. BOINC should check which distros are installed. BOINC server should somehow (e.g. command line parameter) inform client side which distro was accepted.
• Distro versions
  Users may upgrade a distro in place or swap the filesystem for an entirely different distro. As such the name of the distro doesn't necessarily match what's actually running. BOINC should check the distro version.
• Might not be safe to use
  WSL has a very simple init system which doesn't support things like starting services or anything else really. If I have understood correctly right now people put stuff in profile to have it run automatically when starting WSL. If WSL ever gets a fancier init system then using WSL might start some services in it. So running anything in WSL, even just for querying the distro version, might not be side effect free. BOINC should have an option to not use WSL.
• Some distros may be ok to use
  Users may have installed distros for non-BOINC use but may have also installed distros for BOINC to use. BOINC should have an option to list which distros it can use. The previous bullet point, option to not use WSL, could be combined with this.
• App IPC
  On Windows BOINC uses Windows shared memory segments for IPC and on Unixes memory mapped files. That means client and apps can't talk to each other and things like suspending a task won't work. Unix versions have also older code that uses System V shared memory segments. Windows and System V name the memory segment differently and it seems unlikely they could be made compatible. Memory mapped files ought to be compatible but needs to be tested. If memory mapped files work then BOINC needs to switch to using them on Windows. This would also help running Windows apps on Unixes with Wine.
• Per-user distro installs
  Distros are installed on a per-user basis. Every user has their own copy of the distro filesystem. On multi-user hosts one user might start a task with Linux app but not finish it and the next user to log in might not have distros installed. Or if the next user has the same distro installed the app might have created some files on the first user's filesystem which is now not available. Running Linux apps needs a specialised wrapper that records which user started the task and only allows the same user to continue it.
• Default distro can change
  Users can change which distro WSL runs by default. Any file written to the filesystem of one distro won't be available on other distros. An app's checkpoint file format might depend on the version of some system library of a distro. Wrapper needs to record which distro it used to launch an app the first time and then use the same distro when continuing the task. Or alternatively, the wrapper needs to always use some specific distro instead of the default.
• Client can't monitor or control Linux apps
  Normally client expects that it knows the process ID of the app it started and if the app starts child processes they form a process tree. That doesn't work with WSL. A process tree is kept only up to WSL launcher but the actual Linux app is not a child process of the WSL launcher. That means client can't get CPU usage of the app or kill it if necessary. Client can be notified of additional processes it needs to monitor but that can't be used. When looking at a Linux process it has one PID on Win32 side and another on WSL side and there is no way to map the PIDs. Additional complication is that when looking from WSL side Linux apps form a process tree but when looking from Win32 side Linux apps have no parent or child processes. The only way to get around that that I can think of is to have a sort of proxy on WSL side that monitors and controls the task. So the specialised wrapper would be two part, Win32 side launcher and WSL side monitor.

[AenBleidd]

#2491 is a starting point for this ticket.
Also I'd like to be the responsible for for this feature because I'm interested in it

[JuhaSointusalo]

I figured I'd comment on the actual implementation in #2491 and have the higher level observations somewhere else, like here.

Also I'd like to be the responsible for for this feature

No problem. I've just been looking at WSL on and off since I can't remember when, year ago or earlier so I might have some thoughts about it. I would have liked to run LHC's Atlas native app with it but can't do that because the app needs CVMFS which needs FUSE but WSL doesn't support FUSE yet. I do have GPUGRID app running on WSL though. I wanted to see how well WSL works on FCU and Windows kept nagging about upgrading to 1803 so I had to fix the app right now.

[AenBleidd]

I have a very interesting question.

We can get a list of linux distros available via wsl on windows but this list will give us only pretty names.
And there is no way to run particular distro application (e.g. ubuntu.exe) just looking at this list.
There are two ways to do this.
1 - Hard-coded names of distro applications
2 - Create an xml (or any other format, this can be decided later) file within github repo (or on berkeley site as it done for project_list.xml file but I'd prefer to have it on github repo for fast, clear and visible changes) with the list of available distros on Windows Store and their application names. This will require some extra work but from my POV this way is more flexible because it doesn't require to provide a new list of linux distros without building new release. Also I can keep an eye in future on the list of linux distros available on Windows Store to keep this list up-to-date.

I need your opinion, guys.

CC: @JuhaSointusalo, @TheAspens, @davidpanderson, @RichardHaselgrove, @ChristianBeer

[AenBleidd]

Here's the current list of linux distros:

Distro	Command	Comments
Kali Linux	kali
Debian GNU/Linux	debian
SUSE Linux Enterprise Server 12	sles-12
openSUSE Leap 42	opensuse-42
Ubuntu 18.04	ubuntu1804
Ubuntu	ubuntu	Always current stable, now it's Ubuntu 16.04
Ubuntu 16.04	ubuntu

I didn't check them all yet, but I guess it is better not to guess application name using next command
wslconfig /list
but to have predefined list instead.
I'll update this table with the pretty names shown with the command above.

[RichardHaselgrove]

The reference to project_list.xml (actually all_projects_list.xml) is apposite. Keeping the WSL option list dynamically maintained in data seems appropriate, and we have a precedent. But I would urge transparency - such as direct GitHub visibility - for the updating process.

all_projects_list.xml probably underwent more revisions than any other file during the v7.10.2/3 release process, and at one stage we had four different and incompatible versions in circulation - partly because of late news and requests arriving from projects, but also because of additional data required upstream. Some of the changes were applied silently, with no record on GitHub.

We might be down to three versions now, but I've got a ferry to catch in the morning and I'll leave the cross-checking until I get back.

[JuhaSointusalo]

Based on the very limited sample size I had assumed the list from wslconfig /list would work by just appending .exe to the listed distros. Pretty lame if that doesn't work :(

Maintaining a list has a drawback that it'll only work for distros you know of. It wouldn't work for unofficial sideloaded distros.

Quick googling brought up this comment. HKCU:\Software\Microsoft\Windows\CurrentVersion\Lxss itself isn't enough but maybe combined with WSL API would work.

[AenBleidd]

Quick googling brought up this comment. HKCU:\Software\Microsoft\Windows\CurrentVersion\Lxss itself isn't enough but maybe combined with WSL API would work.

I think this is a better solution. This will require a lot of code change but it's really better.
Thank you very much, @JuhaSointusalo

[truboxl]

I know this is just an initial work to provide WSL detection in BOINC, besides the points listed above:

1. WSL does not have GPU support Cannot find GPU devices on Bash microsoft/WSL#829
2. WSL does not fully support process prioritisation Set process priority microsoft/WSL#894
3. WSL does not provide /dev/input that BOINC Linux Client idle detection depends on

Probably highly inefficient right now and may change in the future...

[AenBleidd]

Point 3 can be fully implenented via wrapper. And I plan to do this too. Maybe Point 2 also can be covered by wrapper I guess but I'm not sure. Point 1 can be left as a limitation for now. In any case, I think that having this support even if it is limited is better than doesn't have it at all. Best regards, Vitalii Sent via Android сб, 16 июн. 2018 г., 14:57 Jia Yuan Lo <[email protected]>:

…

I know this is just an initial work to provide WSL detection in BOINC, besides the points listed above: 1. WSL does not have GPU support microsoft/WSL#829 <microsoft/WSL#829> 2. WSL does not fully support process prioritisation microsoft/WSL#894 <microsoft/WSL#894> 3. WSL does not provide /dev/input that BOINC Linux Client idle detection depends on Probably highly inefficient right now and may change in the future... — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#2370 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADFZodS4dKmfo9btPFx8eYtVtojvWo5jks5t9PK1gaJpZM4SNvSJ> .

[AenBleidd]

Part 2 in #2568 is ready for review

[truboxl]

With 0e16d02 merged, I think we can safely cross out Point 3 now...

I just happen to know that if there are 32bit tasks offered to WSL, it will not execute and just fails them. Probably need to force the client to only accept 64bit tasks through cc_config.xml way or resort to hack of enabling 32bit support in WSL especially Ubuntu

There's also the potential of wrong file permissions and you have to do chmod every reboot or else the client just abort the tasks prematurely (for me, caused by dual boot Linux tinkering)

[AenBleidd]

@truboxl, I think the way we will use WSL as another 'platform'. BOINC will not accept linux tasks on Windows. So in this case developers, I guess, will be aware about the bitness. So it would be easier for client 'cause there will be no need to make such kind of task check

[truboxl]

I think this can be closed for clean up...