Separate userspace TLS from lthreads

[davidchisnall]

As part of the relayering work, we should ensure that nothing in src/sched depends on libc.

• Extend OE to use %gs to store per-ethread state.
• Extend OE to provide a per-ethread pointer for library users to consume.
• Move the lthread scheduler to use that pointer. Currently, it is stored inside TLS, which produces some interesting circular dependencies. The lthread scheduler is accessed via a static function in the musl codebase, which calls another musl function.
• Revert musl changes that rely on lthreads
• Return TLS region ownership entirely to musl.
• Once LKL does not support clone #155 is completed, revert to using clone to create threads

[mikbras]

You probably already know this, but when I implemented a libos POC (based on MUSL) I had to handle the following syscalls generated by the pthread layer.

• SYS_set_tid_address
• SYS_tkill
• SYS_futex
• SYS_set_thread_area
• SYS_membarrier
• SYS_get_robust_list
• SYS_set_robust_list

[davidchisnall]

Thanks @mikbras. A few comments for each one:

• SYS_set_tid_address is equivalent to the issue in Intercept clone to handle CLONE_CHILD_CLEARTID #154, we will need to intercept this and integrate it with our futex / lthread implementation. I believe that the correct way of doing this is to store the clear_child_tid value in the lthread structure and make the lthread cleanup code clear the value and wake the associated futex.
• SYS_tkill should already be handled by Linux.
• SYS_futex we are already replacing.
• SYS_set_thread_area we don't handle this yet, but it should be a simple wrfsbase instruction.
• SYS_membarrier I'm curious as to how you implemented this one. I was under the impression that it was impossible to do efficiently in SGX. I believe that we could implement it by setting a flag that caused every other ethread's lthread scheduler to perform an ocall. It may be sufficient in practice to just deschedule the current lthread until every ethread has switched to a different lthread.
• SYS_get_robust_list / SYS_set_robust_list, I believe that these should be handled by Linux for us already.

[davidchisnall]

@vtikoo, please can you update us on how @jhand2 is getting on with the %gs things in OE? This is now a blocker for other work.

[davidchisnall]

@mikbras, I'm curious that you needed to implement set_thread_area: this is a system call only on 32-bit x86 Linux. On 64-bit, it uses arch_prctl to set the %fs register. I think we can get away without that and just use wrfsbase from musl directly from musl's __set_thread_area and leave the C code unmodified. We may have to implement arch_prctl for things that want to use %fs for external threading libraries, but that's probably very low priority.