Building Cosmopolitan on Windows

[alisonatwork]

After having a look at the cross9 gcc from https://justine.lol/cosmopolitan/windows-compiling.html it occurred to me that we should be able to compile Cosmopolitan libc itself on Windows. The main missing piece is GNU make, which there is a small static version of inside Chocolatey at https://chocolatey.org/packages/make

I just had a quick experiment and had to add -Wno-attributes, but it actually succeeded in building a couple of the bootstrap binaries, and they Just Work. (Update Alas, the bootstrap binaries were vendored.) First bug happens here...

build/bootstrap/package.com -o o//libc/stubs/stubs.a.pkg o//libc/stubs/abort.o o//libc/stubs/assertfail.o o//libc/stubs/cxapurevirtual.o o//libc/stubs/debugbreak.o o//libc/stubs/gcov.o o//libc/stubs/instrumentation.o o//libc/stubs/ld.o o//libc/stubs/panic.o o//libc/stubs/retpoline.o o//libc/stubs/stackchkguard.o o//libc/stubs/stackguard.o o//libc/stubs/typeinfo.o o//libc/stubs/ubsan.o o//libc/stubs/xnu.o
error:build/bootstrap/package.com: check failed: 0x1300 == 0x198 (87)
package.com EXITED WITH 77: build/bootstrap/package.com -o o//libc/stubs/stubs.a.pkg o//libc/stubs/abort.o o//libc/stubs/assertfail.o o//libc/stubs/cxapurevirtual.o o//libc/stubs/debugbreak.o o//libc/stubs/gcov.o o//libc/stubs/instrumentation.o o//libc/stubs/ld.o o//libc/stubs/panic.o o//libc/stubs/retpoline.o o//libc/stubs/stackchkguard.o o//libc/stubs/stackguard.o o//libc/stubs/typeinfo.o o//libc/stubs/ubsan.o o//libc/stubs/xnu.o

make: *** [build/rules.mk:77: o//libc/stubs/stubs.a.pkg] Error 77

This seems promising to me. I would really love to be able to build end-to-end all on Windows without spinning up a WSL VM. Is this something worth pursuing? It will need a few Makefile tweaks, which we could probably do same way as LLVM mode.

[alisonatwork]

Tried again with a debug version of package.com:

$ build/bootstrap/package.com -o o//libc/stubs/stubs.a.pkg o//libc/stubs/abort.o o//libc/stubs/assertfail.o o//libc/stubs/cxapurevirtual.o o//libc/stubs/debugbreak.o o//libc/stubs/gcov.o o//libc/stubs/instrumentation.o o//libc/stubs/ld.o o//libc/stubs/panic.o o//libc/stubs/retpoline.o o//libc/stubs/stackchkguard.o o//libc/stubs/stackguard.o o//libc/stubs/typeinfo.o o//libc/stubs/ubsan.o o//libc/stubs/xnu.o
error:tool/build/lib/persist.c:119:package.com: check failed on XXXXXXXX pid XXXXX
        CHECK_EQ(filesize, writev(fd, iov, iovlen));
                 → 0x1300 (filesize)
                == 0x198 (writev(fd, iov, iovlen))
        EINVAL/err=87/errno:87/GetLastError:203 The parameter is incorrect.
        C:/Users/XXXXXXXX/git/cosmopolitan/build/bootstrap/package.com \
                -o \
                o//libc/stubs/stubs.a.pkg \
                o//libc/stubs/abort.o \
                o//libc/stubs/assertfail.o \
                o//libc/stubs/cxapurevirtual.o \
                o//libc/stubs/debugbreak.o \
                o//libc/stubs/gcov.o \
                o//libc/stubs/instrumentation.o \
                o//libc/stubs/ld.o \
                o//libc/stubs/panic.o \
                o//libc/stubs/retpoline.o \
                o//libc/stubs/stackchkguard.o \
                o//libc/stubs/stackguard.o \
                o//libc/stubs/typeinfo.o \
                o//libc/stubs/ubsan.o \
                o//libc/stubs/xnu.o
7770000ffc10 0000004080a6 __die+0x34
7770000ffc20 000000408018 __check_fail+0x23d
7770000ffd70 0000004079e9 PersistObject+0x655
7770000ffe30 0000004057dc WritePackage+0xd2
7770000ffee0 0000004071ae Package+0x4c
7770000fff10 0000004072e8 main+0x57
7770000fffe0 0000004017f4 cosmo+0x40
7770000ffff0 000000418650 _jmpstack+0x17

[jart]

Earlier on this project's history I took certain liberties with functions like writev being defined as allowing partial completion. But I've been working to improve that since I'm reasonably certain it can't happen unless the program has signal handlers that return. It looks like I need to add more tests to make sure the polyfills are working as intended.

[jart]

I've now addressed the issue you encountered. Thanks for the report. It's amazing how easily code compiles on Windows when you have the canonical GNU/Linux compiler. It'd do much to attract contributors to this project if Windows and Mac users didn't need to SSH into a Linux box and write code in a terminal like me. I posted further thoughts on this subject in the associated PR here: #116 (comment)

[alisonatwork]

Thanks for the new package.com, that got the build a lot further for me. I don't have time right now to get the build fully done, but I'm happy to be the guinea pig and try fix some of the issues in the coming week(s).

Some short notes of problems I have hit or worked around so far.

1. gcc 9 EXITED WITH 5 error happens fairly often. I think this is some kind of filesystem race condition when using POSIX-y I/O, like it isn't opening or flushing quickly enough. Dirty workaround for now is to while ! make; do echo "retry"; done.
2. ...but keep on eye on it, because you will also hit make (e=206): The filename or extension is too long when it tries to link libc. This can probably be solved by linking in batches or using an environment variable or something.
3. If you didn't get there, it's because __attribute__((__visibility__("hidden"))) doesn't work, but that can be easily fixed in c.inc.
4. Makefile - comment out SANITY line to skip Linux checks.
5. build/definitions.mk - override the gcc paths (should move to build/config.mk with a MODE setting)

Regarding VS Code-related comments on the PR:

Should we put that in the .vscode/settings.json file?

I would be happy to include a settings.json with debug log level, although it also outputs some spurious logging like focus gained/lost. The other thing I'm not sure about is whether including settings.json will annoy VS Code developers who want to set up their own workspace overrides in that file.

a shell script wrapper that runs third_party/gcc/unbundle.sh to extract the compiler if it's not there

I think we can definitely do something like this to bootstrap VS Code IntelliSense. It might even work right now on Linux.

Regarding cross9:

I'd support a change that downloads cross9 automatically for vscode users on windows, provided it verifies a hardcoded sha256 sum.

Good idea. If we can get the build working as a proof of concept, the next step will be to make it as simple as possible. I think it'd be nice to provide an experience that's as easy for Windows-based developers as it currently is for Linux-based developers. That is: install Git, clone the repo, install make, type make, and off it goes.

Question on cross9.zip - where is this sourced from? Is it manually compiled for Cosmopolitan, or is it a vendored gcc from elsewhere?

I really like the idea of using a "small" static binary for cross-compiling. If we can't get there with chibi yet then gcc is a decent substitute. I suspect that on Windows you could install the whole MSYS2 stack to get further, but that feels rather heavyweight to me. What drew me to this project is the idea of stripping C back to the bare essentials, especially on Windows where it quickly gets messy after going beyond the bundled csc.exe. I prefer the DOS/BBS era feeling, where you can just unzip one thing and go.

[alisonatwork]

Another data point on the gcc 9 EXITED WITH 5 error... I gave Windows-based compile a shot with LLVM as well (just needed to add -target x86_64-pc-linux-gnu to TARGET_ARCH definition), and started to get the occasional clang 11 EXITED WITH 5. I now see this is coming out of compile.com, so I wonder if perhaps one of the things we are doing in there is a bit too aggressive for Windows to cope with? That might be something that can be worked around.

[jart]

Cross9 is the result of trial and error last year using artifacts sourced from gnu / msys2 / mingw64 / etc. That's documented by the accompanying zipped work directory which contains sources and patches. I don't want to be in the business of distributing GCC long term. I posted the experiment upon request because no one else distributes an elf compiler for windows and chibicc doesn't have a linker yet. As soon as that happens we'll have a ~350kb C11 toolchain that'll satisfy your retro nostalgia without requiring you to sacrifice modern features. It may not be as good as GCC and Clang at multiplying matrices, but a tiny compiler is better than no compiler. GNU Make is less of an issue since it's stable, easy to compile, and the prebuilt binaries distributed by other channels work fine, although it might be nice to vendor someday simply for the sake of total hermeticity.

It's likely the vfork() + execve() polyfills used by compile.com need to be improved. For example, we've needed to use exponential backoff in our rmdir() polyfill thanks to the limitless peculiarities of the new technology executive. Argument limits can be worked around using tricks like ar.com rcsD foo.a @ARGFILE.txt. Hidden doesn't concern me since we're barely using it. Everything else sounds about right. I'm most interested in learning more about why the system interface is misbehaving since that's foundational to everything we do here.

[alisonatwork]

Okay, I have tried to create a load test which will recreate these EXITED WITH 5 errors.

I started with a tiny fake C compiler, compiled with the csc.exe that comes bundled with Windows in the .NET Framework. This is to try avoid any usage of MINGW64 or anything weird and UNIX-y that could be causing problems.

using System;
class FakeCc {
  static void Main(string[] args) {
    if (Array.Exists(args, arg => arg == "--version")) {
      Console.WriteLine("fakecc.exe (.net) 0.0.1");
      return;
    }
    string c = Array.Find(args, arg => arg.EndsWith(".c") || arg.EndsWith(".S"));
    string o = Array.Find(args, arg => arg.EndsWith(".o"));
    System.IO.File.Copy(c, o, true);
  }
}

Then, in Git Bash:

$ pwd
/c/Users/alicl/git/cosmopolitan
$ mkdir tmp
$ cd tmp
$ cp ../fakecc.exe ../build/bootstrap/compile.com .
$ dd if=/dev/urandom bs=1024 count=5 of=blob.c
5+0 records in
5+0 records out
5120 bytes (5.1 kB, 5.0 KiB) copied, 0.0102543 s, 499 kB/s
$ for i in `seq 1 1000`; do cp blob.c blob-$i.c; done
$ for i in *.c; do ./compile.com ./fakecc.exe $i $i.o; done
./fakecc.exe blob.c blob.c.o
./fakecc.exe blob-1.c blob-1.c.o
./fakecc.exe blob-10.c blob-10.c.o
# ... eventually the EXITED WITH 5 error occurs

I tried it with empty files (0 bytes) and the error did not occur.

I also tried on Powershell

PS C:\Users\alicl\git\cosmopolitan\tmp> foreach ($f in gi ".\*.c") {
>> .\compile.com .\fakecc.exe "$f" "$f.o"
>> }
.\fakecc.exe C:\Users\alicl\git\cosmopolitan\tmp\blob-1.c C:\Users\alicl\git\cosmopolitan\tmp\blob-1.c.o
.\fakecc.exe C:\Users\alicl\git\cosmopolitan\tmp\blob-10.c C:\Users\alicl\git\cosmopolitan\tmp\blob-10.c.o
.\fakecc.exe C:\Users\alicl\git\cosmopolitan\tmp\blob-100.c C:\Users\alicl\git\cosmopolitan\tmp\blob-100.c.o
# etc ...

No error! Two possibilities:

1. Powershell is slower and allowing the processes time to finish whatever they were doing that would've caused the "5" error.
2. The additional MINGW64 layer of Git Bash is causing problems somehow.

The problem is that to use GNU Make on Windows, we need to go through MINGW64 anyway. So I made a Makefile...

PS C:\Users\alicl\git\cosmopolitan\tmp> cat .\Makefile
SRCS = $(wildcard *.c)
OBJS = $(patsubst %.c,%.c.o,$(SRCS))

all: $(OBJS)

%.c.o: %.c
        ./compile.com ./fakecc.exe -o $@ $<
PS C:\Users\alicl\git\cosmopolitan\tmp> make
./compile.com ./fakecc.exe -o blob-971.c.o blob-971.c
./fakecc.exe -o blob-971.c.o blob-971.c
./compile.com ./fakecc.exe -o blob-972.c.o blob-972.c
./fakecc.exe -o blob-972.c.o blob-972.c
./compile.com ./fakecc.exe -o blob-973.c.o blob-973.c
./fakecc.exe -o blob-973.c.o blob-973.c
# etc

Works. Both inside Powershell and also inside Git Bash. It seems this is really some kind of laggy race condition type thing. Does any of this help @jart ? I'm not really sure where to start looking for the problem with the NT fork/exec.

[jart]

I know that MinGW and Cygwin I'm pretty sure set up special infrastructure for handling things like posix signals, so it could be possible that some code on their end is upset with compiler.com for not interacting with that. It sounds like a tough nut to crack. It would be nice to have this fixed. But one thing worth noting is that in my personal experience, the fastest possible native build on Windows is still going to go slower than than a build that runs inside a Linux VM on Windows, due to the multiprocessing model tools like make assume. One avenue that might be worth exploring is getting GNU Make to build with Cosmopolitan. Then we could rule out the influence of msys.

[alisonatwork]

Just to log my status on this, I have been noodling around with various different things on Windows in my spare time. Obviously WSL is the easy solution, but I'd still like to get it built once all the way through as a proof of concept. It's a good learning experience to dig through all the problems.

I've switched to experimenting with LLVM to try to remove another MingW/MSYS layer from the stack.

The make problem outside of Git Bash is tricky, because GNU make on Windows is also tied to the existence of sh.exe existing somewhere in the path. It seems like you can't reassign SHELL inside the makefile to point somewhere else, either. Win32 make source is here, FYI: https://sourceforge.net/projects/ezwinports/files/

Anyway, my latest tack has been seeing what kind of minimal shell I could get to work on Windows. It doesn't need to be interactive or have job control to be usable by make. I got oksh (https://github.com/ibara/oksh) compiling under Cosmopolitan, which was cool. Crashes hard on Windows, though. Then I switched to Unbourne shell in the examples directory. I've made some small changes in my local to get it not completely and utterly failing on Windows due to different PATH handling, but nothing worth sharing yet. Something weird is happening in interactive mode where everything comes out as an error, even things that don't when passed with -c. I haven't isolated it yet.

TLDR, it's a work in progress. If I find anything that is easy to isolate and might also improve other platforms, I'll send a PR.

[jart]

Yeah the git debian almquist shell in examples/unbourne.c is the shell of shells, so getting that to work better on Windows would be a good area of focus. The first thing Ken Thompson built when he created the UNIX operating system was its shell, since that's what made everything else possible. Would you permit me to order you a W. Richard Stevens book that can help you accomplish this task? If so, please email me your address.

[pkulchenko]

Obviously WSL is the easy solution, but I'd still like to get it built once all the way through as a proof of concept. It's a good learning experience to dig through all the problems.

I'm interested in this as well, as I'm running windows8 and have attempted to compile redbean using cross9; I ran into several issues that I haven't been able to resolve, so ended up going the WSL route on a different machine, but it would still be great to be able to compile using cross9 binaries (I have mingw/msys and clang installed as well).

My main issue was that the make scripts have -musl- binaries hardcoded, so it was difficult to reorganize them. There are also dependencies on ln, which should be replaced with cp in the pre-win10 configuration or something similar. I just ran out of patience working through those issues... ;)

[jart]

Issue #180 also offers new hope for LLVM builds on Windows if we can git bisect a regression since the 0.2 release.

As for compiling the cosmo repo itself on Windows, I've gotten it to come pretty close in the distant past. However in my experience, even if that works, it still goes faster to build cosmo inside a Linux VM or container on Windows than it does to build on Windows itself, due to the way Windows is. The same is true of MacOS.

I'm open to patches that help us improve. It'd be nice to be build-anywhere run-anywhere if it can be done elegantly and particularly if there's a way to set up GitHub workflows so it's tested. It would however be sad if the Windows workflow ended up being ten minutes when the Travis Linux workflow currently takes 2 minutes.

[alisonatwork]

Unfortunately the past few months I haven't really had the time to look at this project, but I did find the LLVM approach more promising than cross9 gcc back in March. I still feel like the main sticking point is the Makefile and associated build scripts and binaries. We could probably get it all building if we wrote a custom build just for Windows, but it'd be nice if it was a bit more one size fits all, in the spirit of the project. Hopefully I will have a chance to get back to this at some point.

[Kreijstal]

couldnt you do it in msys2 enviroment, yeah it is heavy but, it should be possible?

[jart]

It's now possible to compile the mono repo on Windows. It's a little rough around the edges until we fix #1010 so you can't use the latest release. But you should be able to use https://github.com/jart/cosmopolitan/releases/tag/3.2. Here's what you do. You grab the cosmocc.zip. Extract that into o/third_party/gcc/ of the mono repo. Next you'll want the programs from cosmos.zip on the release page to go into your /c/bin/ folder and put that on your $PATH. It comes with a make program you can. You should do that using cosmos' bash shell. Then when you run make on cosmos it should work fine. Sooo, some assembly required. It'll get better soon. But it's definitely possible. I've seen it with mine own eyes.