Improve robustness of stack overflow detection

[jclark]

Follow on from #91.

Currently implemented solution is as follows:

• in the runtime in main, we set up a stack_guard global variable which is 1Mb beyond the frame pointer of main
• after doing an alloca for each BIR register, but before saving the parameters into registers, we do an extra alloca and compare that against stack_guard; we then panic, if it is the result of the alloca is past the stack_guard

LLVM optimizations have the effect of

• turning the alloca into the frame pointer of the function (roughly)
• optimizing out the check for inlined functions

Net result is that we can exceed the stack_guard by up to the size of one frame. This is not a problem in practice because there's plenty of space between the stack_guard and the actual stack limit.

Potential improvements:

• identify how much stack space we actually have, using rlimit and then use that to set stack_guard (long term we will be implementing strands, which will each have their own stack, so this isn't very useful)
• estimate our frame size and panic if the frame_size would take the stack pointer beyond stack guard
• figure out a way to deal with inlining e.g.
  • disable inlining for functions which we estimate to have a large frame size
  • twiddle LLVM's inlining controls (inline-threshold, inlinehint-threshold)
  • make stack_guard volatile (maybe won't help)

I suspect a 100% solution will require doing something within LLVM (e.g. add a pass to insert stack overflow checks).

[manuranga]

Did some research on existing implementations, here is a summary

1) Guard page

1.1) Plain Guard pages -

• Used by pthreads with pthread_attr_setguardsize

1.2) Guard pages with probing - Same as above but prevents "Stack Clash" exploit by checking each page when allocating chunks bigger than a page.

• Used on Windows, by default
• Used by GCC when enabled by -fstack-check
• Used by Rust, Julia, WAVM via LLVM's probe-stack attribute (contributed by Rust).

2) Per function checking

2.1) Manual frame size counting

• Used by Go

2.2) Abusing LLVM segmented stacks
Same as above but with LLVM's help. AFAIU we can use this regardless of the stack growth strategy we are actually using, ie: no growth, copy over, linked stacks. Following steps are needed to make it work.

a. Pass "split-stack" as an attribute each function
b. Write stack size to thread control block (%gs:0x330 - on mac, %fs:112 - on linux and so on)
c. Provide a _morestack implementation, we can print and abort for now.

• Used by Rust previously.

Additional info

• In approach 2, we need to leave space for red zone https://en.wikipedia.org/wiki/Red_zone_(computing)
• When using guard pages, main thread and ulimit -s unlimited may need special treatment rust/#43052
• For guard page probing, windows has built in _chkstk function

History of Rust stack overflow
Rust was using green threads + segmented stack
They decided to move away from segmented stack but still kept above 2.2 approach just to print stack overflow
They moved away from green threads
Later they moved to 1.2 (rust/#16012)
Futures are implemented in a completely different way

[jclark]

Useful info: 2.2 looks promising. But this can wait for quite some time.