[WIP] Windows

[lbguilherme]

Warning: This is very very initial.

Fixes #26

Roadmap:

• Make it compile code
• Add {% if flag?(:windows) %} all over the core library and make the prelude portable
  • Basic IO
  • Fibers
  • Concurrent IO
  • Exceptions (backtraces)
  • Threads
  • Mutex
  • Process
  • Regex (should work out of the box, but needs to compile libpcre)
  • GC
  • Encodings
  • ...
• Port rest of std library
  • DNS
  • Sockets
  • FileUtils and Dir
  • ...
• Remove dependency on wine clang.exe hack
• Compile crystal.exe and self-host it
• Pass all specs
• Figure out how to do CI testing for Windows
• Add support for 32-bit Windows as well
• Release Crystal for Windows!

How to use it: (from Linux)

1. Get the branch win from this fork:

   git clone https://github.com/lbguilherme/crystal.git
   git checkout win
   

2. Install Wine: https://www.winehq.org/

3. Download Clang 3.8.1 for Windows x64. Install it somewhere with Wine. Make sure this works:

   $ wine bin/clang.exe --version
   clang version 3.8.1 (branches/release_38)
   Target: x86_64-pc-windows-msvc
   

4. Install MXE from my fork (I'll get this patch upstream at some point Update gc to 7.6.0 (and add libatomic_ops because of it) mxe/mxe#1539)

   git clone https://github.com/lbguilherme/mxe.git
   git checkout updated-gc
   # Check http://mxe.cc/#requirements for a list of dependencies
   # (or just run the comand below and get an error listing what is missing)
   make MXE_TARGETS=x86_64-w64-mingw32.static gcc gc libevent libiconv # This will take some considerable time
   

5. Build the crystal compiler having at least LLVM 3.8 in your system. Or maybe download some older clang.exe, compactible with the LLVM in your Crystal. If needed, build it: make

6. Write some simple Crystal:

   # foo.cr
   ch = Channel(String).new
   
   spawn do
     3.times { puts ch.receive }
   end
   
   ch.send "Hello Crystal"
   ch.send "  running on Windows!"
   sleep 1
   ch.send "\nYEAH!!"
   

7. And finally: Build code!

./bin/crystal build foo.cr --ll --single-module --cross-compile --target x86_64-windows-gnu --prelude=windows-prelude
wine /clang-path/bin/clang.exe -g3 foo.ll -c -o foo.o
/mxe-path/usr/bin/x86_64-w64-mingw32.static-gcc -g3 foo.o -o foo.exe -liconv -lkernel32 -lgc -levent -lws2_32

8. And run:

   $ wine foo.exe
   Hello Windows!

FAQ

• When will it be ready? I'm doing it in my free time, so I really have no idea.

• Why not use CYGWIN? I really believe Windows should be a first-class target and that this is fundamental for Crystal's future as a language. The more native the better. I would ditch even glibc if it were possible (at some point it will be).

• How can I help? Pick a very small Crystal code. Try to compile it. See a lot of errors and crashes. Do whatever it takes to make them go away. Send a patch with your changes.

[asterite]

Nice!

I think we should try, for one release, to push this forward until we have at least a bootstrappable compiler to work with. Then things should be much easier.

I noticed some files are completely hidden behind a {% if flag?(:windows) %}. Maybe we can split these files into three. For example: termios.cr, termios_posix.cr, termios_windows.cr and then termios.cr would do the switch over the flag.

Another thing probably worth doing (on my side) is to mark a whole lib with a CallConvention, so all funs inside it will inherit that call convention. It looks like all windows functions use the X86_StdCall convention, right?

[lbguilherme]

I noticed some files are completely hidden behind a {% if flag?(:windows) %}. Maybe we can split these files into three. For example: termios.cr, termios_posix.cr, termios_windows.cr and then termios.cr would do the switch over the flag.

This is similar to what Go does, right? Some files will probably need a full rewrite, like scheduler.cr. Splitting into multiple files is the way to go.

Some compiler support would help here, as it wouldn't be possible anymore to require *. For example: splitting io/file_descriptor.cr into io/file_descriptor.posix.cr and io/file_descriptor.windows.cr fails because io.cr has require "./io/*" in it.

Another thing probably worth doing (on my side) is to mark a whole lib with a CallConvention, so all funs inside it will inherit that call convention. It looks like all windows functions use the X86_StdCall convention, right?

That would be awesome! (That's correct, all WinAPI follows this convention)

---

[asterite]

Oh, the thing about require is true. I guess for now we'll have to break those into individual requires. I think there aren't many cases like that, maybe io or socket are affected, but there aren't many files under those directories. We could try to let the compiler know how to handle this, but I don't know how would that work... maybe check if other filenames exist and see if, splitting by dots, choose the one for which all of the flags meet.

[RX14]

How about being able to mark a file as ignored or not at the top of a file, similar to surrounding the whole file in a macro if flags, then simply require every platform's files? Maybe make it so any macro expression works.

[asterite]

@RX14 That's a good idea. When I read the {% if flag? %} at the top I thought maybe we could have return inside a macro, something like:

# file_windows.cr
{% return unless flag?(:windows) %}

# ...

So code below the {% return ... %} will only be analyzed if the condition is met. The "problem" is that this would be a special rule in the macro language, but maybe it's not that bad.

[RX14]

@asterite Honestly I wouldn't use return, instead something like ignore_file. And only allow it as the first statement in a file, otherwise it'll get messy.

[Sija]

stray else

[lbguilherme]

Oops, messed up selecting which lines to commit. Fixed.

[ysbaddaden]

Eventually, maybe not fully duplicate implementations, but extract the OS specific calls, and limit full extracts for cases where POSIX and Windows have strictly nothing in common.

Some methods may require a specific implementation, but IO:: Buffered#buffered_close, for example, only differs on a single platform specific call. We could keep a single implementation that would call a #close_os_impl method for example.

Same for Scheduler: the overall implementation remains identical, only some helpers are POSIX specific, and some others will be Windows specific.

[shayneoAtNorwood]

I know this isn't a useful comment, but I just want to chip in and say this is wonderful stuff guys. Windows support=closer to world domination. mwahaha

[jsaak]

You may be familiar with libuv. I do not know why, but crystal uses different library. Maybe you can use libuv in windows port?

[lbguilherme]

What I'm doing for the Windows port is writing all the primitives from scratch using WinAPI. It's quite simple to use and I think not using abstractions here good. libevent (the one crystal uses) has poor support for IOCP, that's why I have chosen not to use it here. One surprising fact is that Windows' async IOs are much more powerful than posix's. The first big difference is that IO on regular disk files can be asynchronous, while on Linux this is always blocking (looks like glibc does some work under the cover to simulate nonblockingness using a thread pool). Maybe we should implement a specialised solution for each platform here, to take the best each one can do. Another technical difference: On Posix you try to do an IO operation and it fails with EWOULDBLOCK. You then wait for the IO to be readable/writeable and repeat the original operation. On Windows you start an async IO operation and then wait for the result in a event queue. The event will contain the operation results. Any library that tries to abstract all this difference in a portable interface has to lose something.

…

---- On Sun, 04 Dec 2016 15:16:33 -0300 [email protected] wrote ---- You may be familiar with libuv. I do not know why, but crystal uses different library. Maybe you can use libuv in windows port? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[canyonblue77]

Novice Coder and wanted some feedback. Since this thread is about Windows and there is a 1% chance I could benefit the effort I wanted to share my attempt at porting code for starting a new process from kernel32/Windows library.(linux fork?) No idea what I'm actually doing but hey, fake it till ya make it... Is this close too accurate? Would additional ports of other functions be beneficial? Or just annoying background noise, ya won't hurt my feelings. :)

@[CallConvention("X86_StdCall")]
fun winfork = CreateProcessA(
        lpApplicationName : Char*,
        lpCommandLine : Char*,
        lpProcessAttributes : SECURITY_ATTRIBUTES*,
        lpThreadAttributes : SECURITY_ATTRIBUTES*,
        bInheritHandles : Bool,
        dwCreationFlags : Int,
        lpEnvironment : DWord*,
        lpCurrentDirectory : Char*,
        lpStartupInfo : STARTUPINFO,
        lpProcessInformation : PROCESS_INFORMATION
) : Bool`

struct SECURITY_ATTRIBUTES
        length : Dword
        security_descriptors : Void*
        inherit_handle : Bool
end

struct STARTUPINFO
        cb : Dword
        Reserved : Char*
        Desktop : Char*
        Title : Char*
        dwX : Dword
        dwY : Dword
        dwXSize : Dword
        dwYSize : Dword
        dwXCountChars : Dword
        dwYCountChars : Dword
        dwFillAttribute : Dword
        dwFlags : Dword
        ShowWindow : UInt16
        cbReserved2 : UInt16
        Reserved2 : Void*
        StdInput : Handle
        StdOutput : Handle
        StdError : Handle
end

struct PROCESS_INFORMATION
        Process : Handle
        Thread : Handle
        ProcessId : Dword
        ThreadId : Dword
end

[bcardiff]

@lbguilherme which version of wine are you using here?

Sorry about that push I messed with the remote name (I didn't expect github will allow me to push on you fork :-S). I wanted to bring the to the latest crystal version and see if there is anything I can contribute with.

[txe]

The branch doesn't contain the latest changes from crystal repository, so this is why there is no failures

[bararchy]

@txe How hard would it be to rebase it ?

[txe]

Even if it's hard, I'm ready to rebase it and add Crystal::System. I just want to know if it will be enough to merge it finally,

[mverzilli]

Yes. If it doesn't break tests for already supported platforms we can make it a priority to merge this, aiming at having it in 0.24.

[txe]

Good, then I will try to prepare the branch this week.

[RX14]

My concern is that if nobody is willing to port this work over to Crystal::System now why would they be willing to do so after this PR is merged. If that work doesn't happen we're left with an ugly duality of files which have to keep the exact same interface over 2 platforms with nothing enforcing that. Imagine PRing a fix to the posix version of the stdlib which tweaks a method signature a little, what is there to enforce that that change also takes place on windows? That's part of the reason why we have Crystal::System: to create a clean, minimal interface to platform-specifics that changes at a rate much slower than the rest of the standard library.

It also invalidates a lot of the open PRs by renaming the posix-supporting files. Even if this PR doesn't break anything visibly outside the standard library, it does break the flow for everyone already working on crystal by renaming everything platform specific.

Please, let's be patient and do windows support properly.

[akzhan]

Will CI allowed for Windows builds?

[bararchy]

@RX14 It seems though that @txe is willing to do the System change.

[mverzilli]

@RX14, sure. Making this a priority means we should prioritize everything that stands on the way to it as well :). Maybe I rushed a bit to say it was for 0.24, but I don't see it that far.

@akzhan we can start with experimental support (which is to say no-CI). I wouldn't tie this to having a CI on Windows. Otherwise we'll deadlock progress on both.

[RX14]

@bararchy I was more responding to @oprypin's suggestion to merge this essentially as-is.

Rebasing this to use Crystal::System is a lot of hard work and design decisions which affect linux as well (but I think it's well worth it). Plus rebasing to use Crystal::System naturally splits up this PR into separately mergeable sections, so at that point you might as well PR each bit seperately to speed up the process.

AFAIK we still have the exception handling issue on windows as well, is that still an issue? That's a compiler change that can be PRed seperately right?

[txe]

As I see, @RX14 is going to create a Crystal::System version of IO::FileDescriptor, so I can wait for it and then use it as an example of Crystal::System to do PR for: process.cr, scheduler.cr, file.cr, file\stat.cr, thread.cr, dir.cr.
What do you think?

[hanyuone]

Any news on this?

[RX14]

@Qwerp-Derp The next step is to get #4707 merged.

[bararchy]

What's holding #4707 back ?

[sdogruyol]

Let's rock now that #4707 is merged 👍

[sdogruyol]

Update #4832 is also merged 👍

[piedoom]

Is there any news on this? I've been checking here every few weeks to see progress.

[ghost]

There is AppVeyor for CI testing on Windows

[ylluminate]

Just a heads up, interesting discussion from Nanobox guys with some interest in using Crystal instead of Go (see #crystal) if there was native Windows support. While Windows doesn't personally mean much to me, as a platform and for growing the exposure, Crystal could have some really great wins publicly if this were gnawed on to completion.

[ilog2000]

Congratulations on one year anniversary of this pull request!

[RX14]

If we get #5333 merged, there's a relatively tiny patch (that already works) to getting hello world working on windows. I think that hello world is working as long as it isn't ifdef hell. And after that we can finally get to merging in @txe and @lbguilherme's work. It's been a year but I think we can do it this time.

[RX14]

That patch has now become #5339.

I'll probably close this issue once it's merged. Although full or even basic windows support won't have been added by that PR, it'll be a slow slog of dissecting this current PR and putting it's bits in Crystal::System, which will take a while. So the most logical time to close this PR will be then, since it's obvious this isn't going to get merged as-is.

[ylluminate]

Thanks @RX14. I know the Nanobox guys were thrilled to hear the progress and many are excited to be able to have this opportunity to start using Crystal broadly.