Low-level C interop + static linking

[nomeata]

I believe we want the compiler to use existing code written in other languages (C, maybe rust), the main use cases I can think of being

• a mature bigint library like libtommath
• maybe a more advanced memory manager and GC that we would like to write in a better language
• other parts of the RTS
• eventually other stuff (crypto comes to mind).

For this, we need some form of linking. The two options are

1. we link the same way we eventually want to link ActorScript libraries (see https://github.com/dfinity-lab/discussion/issues/65)
2. we link statically, following the convention at https://github.com/WebAssembly/tool-conventions/blob/master/Linking.md

Option 1. has the issue that it is a custom format, tailored for a first-class view of modules, so we’d have to somehow modify existing C libraries to expose their functionality using this interface, for example by writing a C shim. The advantage is on-chain code re-use.

With 2. I hope we can link against the output of a clang-compiles C library directly. And it is more self-contained: we could make asc invoke the linker at the end and get a fully self-contained wasm module out, as we do now.

Does anyone disagree with the need for including such foreign code, or that static linking in an LLVM-compatible way is a good way to go?

If not, then this implies the following:

• We have to start using a custom Wasm AST that expressive enough to generate the relocation section. This requires (at least)
  • A variant of i32.const that indicates that the argument is a table entry.
  • A variant of i32.const that indicates that the argument is a static memory address
• This allows us to extend our custom Wasm serialization code to produce the relocation section
• For, say, libtommath, we need to export alloc and free functions that integrate with our memory manager. As long as all data structures in libtommath are just blobs (no internal pointers), I believe this is possible without significant changes to the memory manager.
• We put the object file for the included libraries somewhere where asc can find them, and asc invokes lld at the end.

This is a big and somewhat daunting task, but it seems we need to tackle that eventually, so we might as well do it now. At least it will enable us to do implement runtime features in a better language, useful for whoever gets to implement a better memory manager.

Nice side effect: ldd removes unused functions and data sections, so we get a bit of dead code elimination for free.

I am not 100% that ldd makes no assumptions about the code that are not true for us (it has a flag

Note: This issue is not about any source level FFI or linking.

BTW, I just saw that the emscripten people have a plan for dynamic linking (https://github.com/WebAssembly/tool-conventions/blob/master/DynamicLinking.md). This requires less relocation work, so it would be easier to implement on the asc side (but it would require dedicated support on the host side).

[rossberg]

Something like this might make sense, but I'm not sure we would want to use it for things like the AS language runtime. Statically linking the runtime would

• induce a lot of code duplication and code size bloat,
• prevent upgrading the language runtime without devs rebuilding their app.

As we discussed in Zürich, the latter is rather problematic, because e.g. we would not be able to distribute critical fixes or other upgrades to the AS runtime.

[nomeata]

prevent upgrading the language runtime without devs rebuilding their app.

I think that is a feature, not a bug :-). (And if it is a feature, why do we care about this for the run-time, but not for, say, the generated code? And if we care about that, maybe we simply want people to upload AS source code, and compile server-side? Would have the advantage that one can review the actual source code before trusting a dapp).

[rossberg]

It's a feature for cementing bugs. ;)

You are right that there is an analogous problem with code generation. But should the existence of a problem justify making it a larger problem?

[nomeata]

I’d put it this way: the ability to swap out the runtime in an running process is too obscure that I would use it as a design guideline (unless we seriously get into the business of creating a hotswappable programming language, where this is a more natural requirement).

[nomeata]

Closing this. Will not create a JIRA ticket right away, but only when it is both clear that we need this (in the near term), and have an idea how this should look like (in particular with regard to how to do the linking.)