Make it possible to build on WSL

[tobiasdiez]

This ticket is to inform you about how to build sage with the Windows Subsystem for Linux and to provide the necessary changes to make this work. Since this is my first contribution to sage, please let me know if I've overseen something obvious and please be lenient in case I broke some written or unwritten rules.

Steps to build sage on WSL

including set-up git infrastructure since this is partly different on Windows as well.

Prerequisites

• Install WSL (Ubuntu 18.04 LTS)

Git (run everything in git bash)

• Clone repository

• Correct line endings (necessary if global core.autocrlf is set to true):

  git config --local core.autocrlf false 
  git rm --cached -r .
  git reset --hard
  

* Setup remote trac (nessary if you want to contribute)
  
  Follow the instructions to set-up the ssh key https://help.github.com/en/articles/working-with-ssh-key-passphrases and then follow http://doc.sagemath.org/html/en/developer/trac.html#trac-authentication-through-ssh and http://doc.sagemath.org/html/en/developer/manual_git.html#chapter-manual-git

git remote add trac git://trac.sagemath.org/sage.git -t master
git remote set-url --push trac [email protected]:sage.git

* Switch to development branch

git checkout develop

## Build sage (run everything from Ubuntu bash)
* Install prerequisites 
  This is a slightly extended version of the [corresponding script for Linux](http://doc.sagemath.org/html/en/installation/source.html#linux-recommended-installation).

sudo apt-get update

sudo apt-get install binutils pixz gcc g++ gfortran make m4 perl tar git patch openssl libssl-dev libz-dev bc libbz2-dev liblzma-dev libgmp-dev libffi-dev libgf2x-dev libcurl4-openssl-dev curl yasm
sudo apt-get install gettext libgettextpo-dev
sudo apt-get install automake pkgconf

* Change into sage folder
* Set how many cores you want to use for building (important, otherwise it will take ages)

export MAKE="make -j4"

* Build (there will be errors on the way, so ignore them using the `-k` switch)

make -k

* Test sage

./sage
2+2

# Changes necessary to make this work
In order to make the above procedure work, I had to change a few things on the way. I have pushed these changes to the branch `public/wls`.
* Linux symlinks do not work under Windows. Thus, if you clone the repository the symlinks are replaced by ordinary text files containing the target path of the symlink. To fix this, I replaced the symlinks in `build/pgks/gfortran/` by the respective files in `build/pgks/gcc` and similarily in `build/pgks/python2/` by the respective files in `build/pgks/python3/`. (There is probably a more elegant way to do this)
* Error

spgk-install should not be marked executable in the build/pkgs directory

   was fixed by changing `build/bin/sage-spkg`. This is a quick hack and probably needs to be fixed properly. The trick used for cygwin does not work since `uname = Linux` when run from WSL.
* Error 

OSError: [Errno 22] Invalid argument: './src/doc/pari/Programming_in_GP:_other_specific_functions.html'
open sage/upstream/giac-X.tar.bz2

   was fixed by changes in `tar_file.py`. This is also a quick hack and needs a better fix, e.g. rename the file to a valid name?
* Error

The headers or library files could not be found for zlib,
a required dependency when compiling Pillow from source.

Please see the install instructions at:
https://pillow.readthedocs.io/en/latest/installation.html

   was fixed by deactivating `zlib` in `pillow/spkg-install`. Note sure if this has any negative side effects.

# Remaining issues
With the above changes, the `make` build works in principle. There are a few small issues that needs further improvement.
* Creating the documentation during `make` or by `./sage --docbuild all html` fails with the errors of the form

OSError: WARNING: Unable to fetch /mnt/c/sage/local/share/doc/sage/doctrees/en/reference/algebras/environment.pickle
There is a syntax error in your configuration file: invalid syntax (conf.py, line 1)
Build finished. The built documents can be found in /mnt/c/sage/local/share/doc/sage/inventory/en/reference/algebras

   I guess, the problem is that `conf.py` in `src\doc\en\reference\algebras` is a symbolic link.
* At startup of sage, you get the warning

/mnt/c/sage/local/lib/python2.7/site-packages/psutil/_pslinux.py:519: RuntimeWarning: 'sin' and 'sout' swap memory stats couldn't be determined and were set to 0 ([Errno 2] No such file or directory: '/proc/vmstat')
warnings.warn(msg, RuntimeWarning)

  Probably, there is nothing we can do, since this is a known limitation of WSL: https://github.com/microsoft/WSL/issues/1071

CC:  @embray @slel @dimpase @soehms @pfasante

Component: **build**

Keywords: **windows, wsl, make**

Branch/Commit: **[public/wls](https://github.com/sagemath/sagetrac-mirror/tree/public/wls) @ [`cd8116e`](https://github.com/sagemath/sagetrac-mirror/commit/cd8116ead3b66e949c414bce76e154eddf69e47c)**

Reviewer: **Tobias Diez**

_Issue created by migration from https://trac.sagemath.org/ticket/28549_

[sagetrac-git]

Changed commit from f0ae571 to cd8116e

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

dcb1e1b	Ignore errors that scripts are marked executable
16b3269	Replace symbolic links with their target
e13d5c0	Disable zlib for pillow
ac782b4	Improve error message in case checksum.ini couldn't be found
cd8116e	Do not extract files containing a colon in the name

[tobiasdiez]

comment:2

Is there anything I need/can do? What is the procedure here (this is my first contribution to sage...)?

[embray]

comment:4

Is this for WSL 1 or WSL 2? Also, my understanding was that symlinks work better now in WSL 2 but I haven't tried it or looked into it in detail.

[tobiasdiez]

comment:5

It's for WSL 1. As far as I know, WSL 2 is still in preview (or at least requires a insiders build of Windows).

The problem with the symbolic links could have been caused by the fact that I cloned the repository under Windows, which replaced the symbolic links with text files. It might work if you clone under WSL.

[embray]

comment:6

Replying to @tobiasdiez:

The problem with the symbolic links could have been caused by the fact that I cloned the repository under Windows, which replaced the symbolic links with text files. It might work if you clone under WSL.

Indeed, I think any build process for Sage on WSL should be 100% under WSL. This is likely also the cause of some of the permission problems you noted.

[embray]

comment:7

Ticket retargeted after milestone closed

[mkoeppe]

comment:8

Using git in Windows with git config --global core.symlinks true may help. See #29295

[dimpase]

comment:11

I cannot view the branch on trac.
https://github.com/sagemath/sagetrac-mirror/commits/public/wls

and it looks quite outdated.
Rebase?

[tobiasdiez]

comment:12

It might outdated because the original commits are now already 7 months old. I would appreciate if I could get advice concerning the open points raised above. I don't want to rebase and then have to wait another year to get feedback. (Sorry for the slightly aggressive tone but honestly it wasn't a great experience so far contributing to sage.)

[mkoeppe]

comment:13

I think guidance regarding symlinks and lineendings when checking out on Windows should be added to the Installation Manual / Developer Manual.

We now have #29513 sage-spkg: Remove check for executable permission on spkg-install.in script templates

[mkoeppe]

comment:14

The commit "Do not extract files containing a colon in the name" needs a better solution. Upstream bug report to avoid non-portable file names

[mkoeppe]

comment:15

Replying to @tobiasdiez:

It might outdated because the original commits are now already 7 months old. I would appreciate if I could get advice concerning the open points raised above.

Trac notifications by email are not very reliable, unfortunately, so you may have missed comments 6 and 8 above.

[mkoeppe]

comment:16

Replying to @mkoeppe:

The commit "Do not extract files containing a colon in the name" needs a better solution. Upstream bug report to avoid non-portable file names

See #29521

[mkoeppe]

comment:18

The zlib problem is likely not specific to WSL at all. Since 9.0 we have implemented lots of portability fixes for various distributions, including Ubuntu.

[dimpase]

comment:19

Replying to @tobiasdiez:

(Sorry for the slightly aggressive tone but honestly it wasn't a great experience so far contributing to sage.)

apologies, this reflects the number of active contibutors to "core" Sage who use WSL (or Windows in general), unfortunately.

At least now we have started to test on Windows using GitHub Actions, so if we managed this with WSL too we would be in a good shape here.

[mkoeppe]

comment:20

So here's what we need:

• Modify giac upstream tarball to remove non-portable filenames #29521 - to fix the colons
• guidance regarding symlinks and lineendings when checking out on Windows should be added to the Installation Manual / Developer Manual (where git operation is discussed)
• perhaps a check in configure to reject source trees that have been checked out with a wrong symlinks/lineendings mode. This would also be useful for Cygwin.

I don't think automatic testing on WSL is necessary. WSL1 is really just a stock Linux distribution running on a slightly weird filesystem.

[mkoeppe]

comment:32

Replying to @soehms:

But at least those tests which lead to mathematical wrong answers should be fixed or marked with a note that they are not validated under WSL.

Definitely. The failure with number_of_partitions(241, algorithm='bober') should become a separate ticket. Is this already a doctest?

[soehms]

comment:33

Replying to @mkoeppe:

Is this already a doctest?

Yes, see the first line of the list shown in comment 24. I've just simplified that to it's first occurence which is reproducible 241 on both 9.1.rc0 and on stable 9.0 installed from binaries.

But, I have to correct my statement from comment 24, that the bug would not occur on version 8.1. That is not true, but here, 241 it not in the list of failures (see ticket #29600 I've created for that issue).

[soehms]

comment:34

Friedrich, I hope you had a good start at Bosch this month and time enough left to contribute your experience on doing Sage under WSL.

[pfasante]

comment:35

Thanks, Sebastian!

I think I also used WSL2 when building Sage in it. At least my experience was as yours: not more problems than under a normal Ubuntu. (However I also did everything within the WSL and tried to leave out Windows as much as possible.) So not sure how much I can help here..

Just for reference: my google-groups post re my WSL try is here https://groups.google.com/forum/#!searchin/sage-devel/wsl|sort:date/sage-devel/WoEWxYmwQnI/aeqy6YxoDQAJ

[mkoeppe]

comment:36

Let's close this ticket in favor of #25206 (Update README, installation manual and developer's guide on Windows (WSL and Cygwin))

[fchapoton]

Changed keywords from windows, wls, make to windows, wsl, make

[mkoeppe]

Changed author from Tobias Diez to none

[mkoeppe]

Reviewer: Tobias Diez

[embray]

comment:40

I'm -1 on removing symlinks from the repository just to support git checkout on Windows (which does support symlinks in WSL). If you want to develop Sage on Windows I think you should also use git through WSL and not using the native Windows git. Same goes when using Cygwin (where this works fine).

[embray]

comment:41

diff --git a/build/sage_bootstrap/package.py b/build/sage_bootstrap/package.py
index aff9212..3ee6c6a 100644
--- a/build/sage_bootstrap/package.py
+++ b/build/sage_bootstrap/package.py
@@ -229,9 +229,11 @@ class Package(object):
         self.__md5 = result.get('md5', None)
         self.__sha1 = result.get('sha1', None)
         self.__cksum = result.get('cksum', None)
-        self.__tarball_pattern = result['tarball']
+        self.__tarball_pattern = result.get('tarball', "")
         # Name of the directory containing the checksums.ini file
         self.__tarball_package_name = os.path.realpath(checksums_ini).split(os.sep)[-2]
+        if not result.get('tarball'):
+            log.warn("Could not find checksum in %s" % checksums_ini)

This can be written just result.get('tarball'), without specifying a default. In this case the default is already None. Then below you can write if not self.__tarball_pattern:.

Out of curiosity though why would this happen? Shouldn't this always be here? And why an explicit check for 'tarball' and not for the other required fields?

[embray]

comment:42

diff --git a/build/sage_bootstrap/uncompress/tar_file.py b/build/sage_bootstrap/uncompress/tar_file.py
index cf707e7..b6d8dc9 100644
--- a/build/sage_bootstrap/uncompress/tar_file.py
+++ b/build/sage_bootstrap/uncompress/tar_file.py
@@ -116,6 +116,9 @@ class SageBaseTarFile(tarfile.TarFile):
             The additional ``**kwargs`` are for Python 2/3 compatibility, since
             different versions of this method accept additional arguments.
         """
+        if ":" in targetpath:
+            return
+
         old_umask = os.umask(self.umask)
         try:
             super(SageBaseTarFile, self)._extract_member(tarinfo, targetpath,

This check is overly broad since this will work on any POSIX system (and should work on WSL too but I understand it's presently a limitation with the NTFS filesystem).

Instead perhaps it should try extracting the file anyways but properly handle whatever exception occurs when trying to extract it on WSL, and produce a warning as well.

[embray]

comment:43

This is a laudable effort and I want to support it, but I think as-is it's making too many changes to support this one one build environment, to the detriment of all other build environments (especially the removal of symlinks). I think some of the git-related problems also stem from mixing native Windows and WSL gits, but please correct me if I'm wrong on that. I haven't tried WSL since WSL2 came out some I'm not completely versed in what limitations exist or not (but I'm quite sure I recall it supporting symlinks just fine).

I don't have access to a fast Windows machine right now, but when I get back to my office I will, and then I can try to help with this.

[mkoeppe]

comment:44

Note the milestone on this ticket...

[embray]

comment:45

D'oh. Sorry for the noise. Regardless, it looks like there might have been some useful things in here? Or is the colon-in-filename issue resolved now?

[mkoeppe]

comment:46

Yes, it's all done, see for example comment 21.

[tobiasdiez]

comment:48

Thanks embray for the review. This issue is indeed somewhat old, I've described how to set up a working build envirnmont using WSL here: #25206 comment:63