Structure and documentation

[Mec-iS]

I am taking some time to organize the file-system structure and document some tool-chain practices in the codebase. I am considering different naming conventions, as the different "cwa" and "internals" and respective synonyms can be confusing. Maybe it is time to define what is "core", what is "utils" and /or "binding" and draw a diagram of the stack (network, gateaway, binaries and respective interfaces).
Please @Xe drop some notes here about what you are expecting from the different directories/layers in the codebase; in general, you are using Go for the "infrastructure/operational layer" and as a "testing container" for arbitrary WASM modules (with examples of function crosscompiled from Rust). Am I correct?

[Xe]

I am sorry this is not as documented as I would have liked. I was planning on doing a full documentation push soon, but I guess it will actually end up sooner rather than later. I have a draft for the design of Archway in progress [HEAVY WIP], but it is currently incomplete as I do research and understand the problem space while I write up the spec. Sorry it's taking a while, designing this stuff prescriptively is hard.

I am taking some time to organize the file-system structure and document some tool-chain practices in the codebase. I am considering different naming conventions, as the different "cwa" and "internals" and respective synonyms can be confusing.

It is standard Go best practice to put command binaries in cmd/, implementation-specific code in internal/ and for other folders to contain code as makes sense. I admit this is slightly confusing given that this is a combination of a Go and a Rust project, and I follow Go package design a bit more to how the language authors intended than most people do.
If Olin were an OS, the "kernel" would be internal/abi/* and the Rust code would be the "userland" in cwa/*.

As far as diagrams go: here's an attempt to explain the request flow for cmd/cwa-cgi:

digraph G {
  subgraph cluster_0 {
    style=filled;
    color=lightgrey;
    node [style=filled,color=white];
    cwa_cgi [label="cmd/cwa-cgi"]
    internal_cwagi [label="internal/cwagi"]
    internal_abi_cwa [label="internal/abi/cwa"]
    
    cwa_cgi -> internal_cwagi [label="HTTP request\nfrom user"];
    internal_cwagi -> internal_abi_cwa [label="Prepare as\nCWAGI request\nto pooled VM"];
    label = "Go";
  }
  
  start -> cwa_cgi;
  cwa_main -> internal_cwagi [label="i32 return"];
  internal_cwagi -> cwa_cgi [label="HTTP response\nfrom stdout"];
  internal_abi_cwa -> cwa_main [label="export cwa_main"];
  cwa_cgi -> end [label="Success"];
  inner_workings -> internal_abi_cwa [label="syscalls"];
  internal_abi_cwa -> inner_workings;

  subgraph cluster_1 {
    node [style=filled];
    cwa_main;
    inner_workings [label="friendly_main"];
    cwa_main -> inner_workings [label="(normally\ninvisible)"];
    inner_workings -> cwa_main[label="i32 return"];
    label="WebAssembly (Rust)";
  }
  
  start [shape=Mdiamond,label="request\ncomes in"];
  end [shape=Msquare,label="End"];
}

Which looks like this:

The cwa/* folder contains rust code that is useful when running Olin in CommonWA mode (currently this is the only mode implemented past examples in raw WebAssembly meant to help teach the concept of syscalls).

in general, you are using Go for the "infrastructure/operational layer" and as a "testing container" for arbitrary WASM modules (with examples of function crosscompiled from Rust). Am I correct?

In general, Olin is currently implemented in Go. The fact that it uses Go is, ultimately, nothing more than a choice of technical implementation detail. The underlying design for this doesn't actually depend on Rust being inside the WebAssembly modules, but the current implementation (as an imperfect reflection of the abstract idea) does so. As far as the underlying design cares, the VM's could be Quake 3 byte code for all it matters as long as there's some way to fulfill the contract given in syscalls.

[Mec-iS]

Wow. Will go through this and try to have an idea about scopes/layers/aspects/domains/tools/whatever (:

[Xe]

For tracking things, here's a bunch of conversations I've had about WebAssembly, system calls, Olin and other inspirations for Olin: https://gist.github.com/Xe/9721c3df0fb0a8267448f821527242b7 (best read if you clone the gist and use a local text editor, sorry). The results of these need to be incorporated into future blogposts and potentially design documentation in the future.

[Xe]

@bb010g by the way, do you know how to make Rust documentation for things and emit that as HTML? (if not let's find out)

[bb010g]

There's Rustdoc, but I'm not sure if you want raw HTML to be put into other stuff or actual Rust documentation. Related to docs, I need to go through and fully document how all this is built and tested currently.

[Mec-iS]

usually there should be a dedicated repository for that. I will take time to document the Rust code for now.
Basic doc here

[UPDATE]
Talking about the diagram, one of the first objective should be IMO to define a functional representation. The current diagram is a request-response workflow one, with relevance on the languages used; next step is to describe, by layer or by scope, competences of the different components. Main components of the system at the moment, in a layered perspective from the highest layer to the lowest:

1. Commands (external interface)
2. Internal Gateway (Interface)
3. Internal Binary (Interface)
4. System-level (execution layer)

Also, as far as I understood, there is an "inter-layer" between 2 and 3 in charge of spawning VMs (wow).

The same system should be described in an alternative representation from a functional point of view (aka tooling, servicing, compiling, executing, etc.). This could be a nice starting material.