Integration of Lingo and the Language and Diagram Server

[cmnrd]

We are currently in the process of designing the new LF package manager lingo. One of the biggest design questions, is how our language and diagram server will interact with lingo. I hope this discussion will serve as a place to exchange and discuss some concepts. I'll add my thoughts on some aspects in a separate post below.

[cmnrd]

Currently, the VS-Code extension directly includes the jar for our language and diagram server (LDS). It spawns the server in a separate process and then interacts with it. Since we want to move most (if not all) of the build logic that is currently implemented in lfc out and into lingo, the LDS will also need to interact with lingo. In its simplest form, the LDS could run lingo and tell it to build the project (or a specific file). However, there is a fundamental design flaw. Lingo will need to ship its own version of lfc, in order to be able to build programs. Hence we would have essentially two instances of our compiler, one shipped by lfc and one shipped by the Code extension. While the former would be used for building, the latter would be used for validation and diagram synthesis. This design however is far from ideal and poses many risks as we need to keep the two instances in sync.

I believe there is a simple way out of this problem and all it takes is to go all-in with using lingo as the single front-end tool. Instead of bundling the LDS with the Code extension, we would bundle it with lingo. We could provide a simple command (e.g. lingo lds) that spawns an instance of the server. This way the Code extension can simply ask lingo to get a server, and does not need to provide one itself (we probably still want to ship lingo with the extension though).

I sketched the whole toolflow below.

I think this design has several positive aspects. By bundling both lfc and the language server into lingo we can control the version, and in fact they could be packed into the same jar. Further, we introduce a layer of indirection between the Code extension and our tooling. We might want to create a rust based LDS implementation in the future. This change however can be fully transparent, as we have lingo lds as a front-end command and other tools do not need to spawn a specific version. We can also think about merging the lfc and LDS boxes in one way or another. And finally, we can use this front-end to spawn configured instances of the server. By putting lingo in charge of spawning the server, we have all the package information available and can pass the relevant information to the server. So in principle, we could have per package language servers.

[lhstrh]

As we're making progress with lingo, we should revisit this discussion and see where we stand... Good topic for next week's meeting?

[petervdonovan]

Update: see lf-lang/lingo#50. I hope that that PR, which is small and uncomplicated, will relieve the pressure to settle on a complete, long-term solution. I think that the PR indicates that for our near-term problems, WASM is likely to suffice as a quick workaround that will not accrue a lot of tech debt.

I also think that WASM is a high-quality solution that might even be the best long-term solution. However, a long-term solution is not necessarily what we currently need, and given how little work I think it will be to address this with WASM, I don't think that we need to agree on WASM as a long-term solution as a prerequisite for using WASM in the near term.

Besides the small scale of the proposed changes, the other reason why I do not think this solution will accrue a lot of tech debt is that I am under the impression that librarifying and giving callers control over side-effectful code (like file system access and running subprocesses) are good, modular design practices anyway, regardless of any WASM compatibility constraint. In particular, requiring a procedure to take a file system accessor as an argument makes it explicit in the signature of a function what side-effectful capabilities it has. This is kind of analogous to capability-based security in the sense that as the caller, you can know for sure, no matter how lingo was written, that if one of its procedures works in your WASM module and it doesn't require a certain capability as an argument, then it can't sneakily do some wild thing in the file system that you don't know about. Parameterizing code by file system access capabilities also can enable things like virtual file systems, tracing or restricting which directories the code accesses, and easily switching between making code work locally and making it operate on the file system of a docker container or a remote system in the cloud (kind of like what we did recently for the Monte Carlo Pi demo).

[petervdonovan]

Related: this repo from the Bytecode Alliance

[lhstrh]

This is very interesting idea, @petervdonovan. So rather than build some three-way communication between language server, extension, and lingo, we just call the WASM version of the lingo code directly from the extension, meaning that the extension can essentially play the role of lingo?

[petervdonovan]

Yes.

And the WASM version of the lingo code literally is the lingo code, except with some functionality made public in the form of a (Rust) library. All the WASM-specific stuff, including the compilation options that make it actually be WASM, can be placed in a thin wrapper library that is completely separate. A prototype of this already works (which is why I'm reasonably comfortable to present this as a "quick" workaround)