Spritely Goblins and Agoric CapTP interop discussion #1827
Comments
|
To pile onto this issue a bit, I'd like to also talk about possible compatibility with capnproto rpc. It seems likely that it may be hard to just shoehorn Goblins and Agoric's stuff into capnproto, and given that we already have a deployed protocol with several implementations that limits our ability to make big protocol changes on our end, but I suspect if we're careful we can design things so that it is at least possible to write a proxy that bridges the two protocols. There is a bit more flexibility on the capnproto end with features not-yet implemented. If folks are open to using capnp at least at the serialization level, that could simplify things a bit. I think probably the biggest divergence in design is that capnproto is all built around static types. But I think from an API perspective we could solve that by, when goblins/agoric objects are viewed from capnproto, they see an interface like: interface Goblin {
callFn @0 (args :List(Value)) -> (results :LIst(Value));
callMethod @1 (name :Text, args :List(Value)) -> (results :List(Value));
# ...
}...with the exact set of methods signatures matching whatever you guys decide to support; see @cwebber's comment about lambdas only vs. separate method/property access. the struct Value {
union {
string @0 :Text;
float @1 :Float64;
list @2 :List(Value);
# ...
}
}When viewing capnproto objects from the agoric/goblin side, we could have the bridge use introspection (which is not implemented, but has been brought up on the mailing list recently) to map the untyped method calls to whatever methods that the object actually supports. |
Excellent! Distributed acyclic gc needs local WeakRefs, which JS now has! Our on-chain implementation also needs determinism, and it will be a long time until we have good support for deterministic WeakRefs for on-chain use (long story). Thus, we do not expect on-chain computation to emit gc info for now. However, our solo vats do not need determinism, and so should be able to do distributed acyclic GC. Any vat, on chain or off, should be able to react to gc messages it receives since there's nothing non-deterministic about that. If you're following the way E/CapTP of old did it, I don't think this will be a barrier to interoperation. Doing our side of the distributed acyclic gc protocol is not top of our priority list; but if you send us a PR I'll happily review it!
You're correct that we should both converge on something non-JSON-based. But in the meantime, JSON is rather universal. Of course, we have another level of encoding into JSON (
Exactly. We're already making use of this transparency when a vats in one SwingSet talks to vats in another SwingSet. To each normal vat, the SwingSet kernel seems to be the rest of the world. There's a special (normally) per-swingset vat we call the "comms vat" that speaks captp to the rest of the world. To the swingset kernel, the comms vat seems to be the rest of the world. To the comms vat, the swingset kernel seems to be all local vats. Each remote swingset appears the same as each remote solo vat. So there should not be an issue here.
Scalars:
I don't understand what this is but you have me curious. Set us a PR and I'll happily review it!
We should be careful about whether we mean the same thing by the term "symbol". In any case, we can both provide the equivalent optimization for string method names. Let's not worry much about optimization until we consider transitioning from JSON.
I like that point. In any case, we need to support partitioned connections anyway, as that is fundamental to the meaning of some transports, such as
;)
Yes. Although we've reluctantly come to the conclusion that we cannot always shorten across arbitrary mixtures of transports. Nevertheless, we should shorten across enough of the cases that matter, and currently do not. When we cannot shorted, we need a better understanding of how to cope with the resulting identity confusions.
We should converge on the wormhole-op-like chain of unacknowledged A-to-C messages as extra payload in the A-to-B message where Alice introduces Bob to Carol. Unlike all other certificate chains, this is perpetually self shortening, and has zero length chains in quiescence.
yes, "properties". Currently, we are not transmitting remote property gets. Only
This is a form of property get, which we do not yet support remotely. But we will, so be prepared.
Looking forward!
Since it seems we're both agreed that neither side can tell the other is not a vat, and since the least the other side could be is a vat, I think we're already agreed on the semantics and "VatTP" is already a good name.
YES!!!! |
Good! I very much want to see this work. I think that building a bridge / gateway that converts between the two protocols is a good approach, at least at first, as that let's the protocols remain different but practically interoperable. If more convergence is possible, a working bridge is a good place to stand to gradually learn how.
Unfortunately we have a political-ism closest-binary-attractor problem. @kentonv did capnp after he did proto2. Unsurprisingly, I prefer capnp to proto2. However, Cosmos / ICF / IBC, the foundation we layer VatTP and CapTP on top of, has converged on proto2. Thus, when we move from JSON to something reasonable, if it is proto-like, there are strong reasons pulling us towards proto2. I wish it were not so. This probably means we cannot actually get rid of the bridge / gateway between the protocols.
Yes. This is the technique used for implementing dynamic languages in static languages --- all types of the dynamic language fit into branches of one type of the static language. This encoding of the dynamic language's types prevent it from talking to any pre-existing statically typed objects, except again via a bridge / gateway.
Yes, perfect! Any idea when to expect it in capnp? A possibly bigger issue separating capnp from CapTP is the issue of storage management. CapTP starts with the dynamic language perspective of a dynamic graph of objects that get gc'ed when unreachable. IIUC, capnp starts with a much more C++-inspired RAII perspective, probably even better suited to Rust, where the programmer must be aware of the transfer of the obligation to delete. The obligation to delete also implies to power to delete, which leads to different ocap design patterns. I'm not sure to what degree even a bridge / gateway can paper of the rift between these approaches. But I am hopeful! In any case, I think Goblins and CapTP are much closer than either are to capnp, and so we should try reconciling Goblins and CapTP first, and without a gateway, before we proceed to capnp. |
Quoth ECMA-404: "whether a processor of JSON texts interprets such a surrogate pair as a single code point or as an |
|
Ok, what does EcmaScript standard |
|
I believe the relevant link is https://www.ecma-international.org/ecma-262/#sec-utf16encoding |
Hard to say; per the conversation it's been on @kentonv's TODO list for a while, and I don't know how likely Ryan is to plow forward and build this thing -- the perils of open source. It's also something where each implemenation would have to support it; it would not be sufficient for this to exist merely in the bridge.
This is ultimately an issue for implementations of capnp in less RAII-oriented languages too. Note that at the protocol level I don't think anything is dfferent, and it's easy enough to wire things up the GC and making things work (as I do with in the Haskell implementation), but there do indeed exist APIs that treat dropping a capability as a semantically meaningful event. iirc the node implementation allows you to call close() on a capability to explicitly finalize it. I've been meaning to add something similar to the Haskell implementation. But for capbility languages that's even maybe a bit dicey as a solution, since it means if you pass the capability to another local object that you don't trust, naively, that other object can revoke your access, so more thought is perhaps needed.
This sounds reasonable; I will follow along and perhaps note any significant implications that occur to me. |
That's not quite right. Cap'n Proto uses reference counting, much like CapTP. You can very well drop references based on GC finalization (if your language supports that), and some implementations do. That said, in my experience, I have found that deterministic destruction and RAII is powerful for making many kinds of capability interactions feel more reliable and solid. Take the revoker pattern, for instance. In a RAII model, I can reasonably expect "fail-closed" behavior -- if anything goes wrong, the capability gets revoked, rather than being left exposed. CapTP has So Cap'n-Proto-based protocols -- at least the ones I design -- are more likely to rely on deterministic destruction. On another note, realistically, I do not believe distributed garbage collection is a tractable problem. In order to perform well, GC algorithms are deeply dependent on heuristic amortization. Most GCs are dependent on "memory pressure" signals to activate them, but what is memory pressure in a distributed system with many different vats that each have their own memory? If one vat in the network signals it is under memory pressure, are all other vats obliged to spin up their GCs to try to release capabilities that might point into the pressure'd vat? That seems problematic for a huge number of reasons. My understanding is that CapTP never really did solve distributed GC, but made the assumption that it could be solved later. My take is that it can't be solved and we should instead embrace explicit reference counting. |
|
I think the right approach at an API level probably depends on:
The Go implementation has shifted to a more explicit refcounting approach as @kentonv suggests, since in Go a well tuned program can go a very long time between GC cycles, and the runtime does not have good visibility into the true cost of remote capabilities. For something like Haskell on the other hand, GC cycles run very frequently anyway; programs tend to allocate like crazy, and the major GC is not incremental, so it's not obvious to me that this is going to be a problem in practice, for most protocols. For particular APIs it may make sense to handle resources in a way that is more manual however, if the underlying objects are likely to be significantly more expensive than the memory footprint of a client would expect. This mirrors the reality of managing lifetimes of e.g. files in languages that are otherwise GC oriented, and I think it probably makes sense to have a similar dual manual/automatic design for APIs like this. But I've found it's common for the memory usage of a server object to not be that dramatically different from the client stub. It seems like a shame to impose the burden of manual resource management on all APIs just for the benefit of what I suspect are a minority of cases. |
|
One non JSON, non capnproto binary format you might want to look at. It is inspired by and was meant as a replacement of JOSS in Captp of E. |
|
Any macro space benchmarks on what the size ratio is compared to JSON? Also, given that both this binary format or JSON would then be compressed by some generic fast compression algorithm, say zip, what's the size ratio post compression? |
|
With msgpack without use of the extension types listed in that EBNF file the size is slightly less than JSON. And due to higher entropy per bit, it does not compress well with deflate. However if the serializer uses the letrec and ibid extension type agressively then quite a lot of repeated items in a captp message flow will only require, 2, 3, 4, or 6 bytes per ibid. Main focus of mine when I composed this format was more on ease of implementation on various platforms such as mid range MCUs, self-descriptivity, and some support for streaming encoding and decoding. |
|
I figure that I should chime in; in the Thanksgiving spirit, one big issue is better than two little issues, and after this, I'll go and have a Taco Tuesday that cannot be beat and not get up until the next morning. The short version is that Monte has now a very basic, bumbling CapTP based on JSON encoding and AMP framing. The protocol is one-directional, can't build cycles, doesn't do GC, and has serious speed issues. I have a raytracer which I use as a speed test; the distributed version is much slower than the single-core version. Part of this speed loss is due to encoding overhead, and we'll probably use Capn Proto RPC eventually simply because it promises to be less slow; pure-Monte decoding is often dominated by copy overhead, but we can do zero-copy slicing of buffers just like the Capn intended. (Our JSON decoding is also pure-Monte and not terribly fast. Many things could be optimized.) From an interoperability standpoint, Monte is very much like Goblins. Objects have only one single invokable behavior, the call, which has only one signature: a triple of a string, a list of refs, and a map of strings to refs. Incidentally, this signature encapsulates multimethods, E-style calls, Capn-style calls, and even verbless function calls, but we probably can't put this to use other than via Postel's Law. I should also point out that, once everything snaps together and into place, we expect to start doing fun high-level things. Already, the distributed raytracer works by compiling Monte code for raytracing maths into a so-called "muffin" module which has no dependencies, packing the muffin into a compact bytecode, and shipping the bytecode across AMP. This suggests that we could start to host Monte objects as una. Right now we have to deal with some details in our Capn implementation, and I need to implement weakmaps, but we should continue plodding along. |
I think this is The Right Thing and it could be great if Agoric would also adopt the "one true invocation path" approach... just make method dispatch on objects a special case. When handling an incoming message, if a function, pass in all arguments. When passing a message to an object, take the first argument as a string (or symbol or whatever, sounds like string here though) and apply it to that method. As for the property-access for arrays/hashmaps which @erights told me is also used for promise pipelining, this too can be done. Lambda the Ultimate. There is one potential problem with the above for Agoric, I'd suspect, which is when you want to do method invocation or property access on an object... I anticipate (but am not sure) that doing property access on an object effectively constructs a facet with only that property on it. Is this correct @erights? If so there are two workarounds: 1) make a magic method for property access, like I'm arguing this very strongly for a good reason, and not just that it makes my life (and @MostAwesomeDude's) easier: I tried adding an approach where function invocation, property access, and method invocation were all handled as three separate thing. This made a mess of become, and my programs started breaking and I lost all the lovely easy-to-program-with-time-travel stuff I made. I can expand on why this is but maybe it doesn't matter here. Anyway, we need to decide on a unified thing: I'd argue the simplest is best. The fact that @MostAwesomeDude has found that a lambda-the-ultimate approach worked with something that's probably much closer to python/javascript than scheme indicates that it could probably work for Agoric as well. But @erights, I suspect you have opinions. Care to share them? |
|
BTW another thing to work out: keywords! Given that I just argued for lambda-the-ultimate, the "easiest" approach would be to say "there is only one argument list, and that is the positional argument list". Which means that you just layer keywords on top of that. Various languages do this, in different ways, but translating between them is probably no problem from an ocap perspective. Just to list them (and this is off the top of my head, I might get something wrong):
Aside from the question of "do we need to add keywords as a type in CapTP" (let's pretend we didn't hear anyone ask that) the nice thing here is that Lambda is the Ultimate and all of these can call each other just fine. However, some languages do things "Differently" (and Differently is the name of a dragon that eats standardization efforts): Python and Racket actually have a separate syntax/argument list for applying keyword arguments. Thus Goblins, being first built on Racket, follows this convention: messages have a slot for both positional arguments (a list) and keyword arguments (a hashmap) when encoded across the wire. Racket has made very good CS hygiene arguments as to why this is a good idea but that's kind of irrelevant because we're living in a world where other languages have no idea what this is. But I'm planning a port of Goblins to Guile soon anyway, aiming for my first CapTP across implementations interop test (since it should be mostly easy, aside from this), so this is a problem I'll need to consider anyway. Here's an observation though: all of these languages support passing arguments via an argument list, even if they also support passing arguments via a separate keyword map. Probably the easiest thing to do (even if it may feel less nice when doing captp across eg two python or two racket implementations) is to say "Sorry, you only get positional arguments." I could look it up, but @MostAwesomeDude, what does AMP do btw? |
|
@washort and I had a series of discussions about this, culminating in a realization years ago in a Portland bar: The correct way to do it is what Racket and Python (and Monte) do, with both a list of arguments and also a (string-indexed) map of named arguments. The main reasoning in favor is indeed some abstract CS principles; it's about how to upgrade objects in the presence of optional behaviors and arguments, and it's akin to using row polymorphism for the indices of the calling convention. TBH if we had had a faster implementation plan and better data structures (some sort of hybrid ordered HAMT?), then Monte probably would have stopped having anonymous positional arguments. The readability argument alone is incredibly compelling, and when mixed with the upgrade argument, it's almost sublime. Indeed, our Capn Proto libraries use keyword arguments to build message buffers, precisely because it's going to be less brittle overall and matches Capn's own preferred semantics for upgrade. AMP only supports string-indexed maps of strings. This makes decoding conversations easy. Paraphrasing our AMP implementation, we send the JSON-armored encoding of a map like this: It would be possible to push the one map into the other map, flattening them by one level. We'll probably do that for Capn Proto RPC, turning the argument list into a special nameless keyword argument. (Capn Proto RPC has a position for selectors, which could be used to store verbs.) |
Here is why the Lambda Ultimation on the protocol level is immaterial in my opinion: If the protocol has three diffrent kinds of deliverOps & deliverOnlyOps then the protocol decoder can just funnel them together into one kind of invocation, following a convention, on the vat interrior side. The local remote ref proxy presences can defunnel invocations, that follow the same convention, into the apropos protocol deliverOps & deliverOnlyOps. If there is only one kind of deliverOps & deliverOnlyOps then, if the convention is known, the invocations, property accesses, and function/closure can be defunneled on the vat interrior side. Conversely the local remote proxy presence can do the funneling for those into the one deliverOps & deliverOnlyOps protocol messages. |
|
Looks like @zarutian is surprising us all by already exploring implementing Spritely's version of CapTP on Javascript / Agoric's stuff...!
Whoa, I did not expect we'd be starting to have an interop conversation quite this fast...! Amazing work @zarutian, please keep it up! |
|
Oh, also I started this OcapN writeup which might be the right place to start coordinating on "this generation" of CapTP? |
|
A bit of a warning though, the code is somewhat messy and quite a few of the comments and commit messages are in Icelandic, at the time of this post. |
|
See https://github.com/Agoric/agoric-sdk/pull/2909/files for a draft more detailed semantics of the Agoric CapTP's Although somewhat biased towards JS, the abstract semantics above the divider is intended to be language independent enough to serve as a basis of inter-language interoperability, e.g., with Goblins and perhaps with Cap'n Proto. We'll see. |
|
What's the status of the interop effort? Silvermint is likely to want CapTP in Go, and it would be nice if it worked with at least one of the implementations. Also, it looks like maybe endojs/endo now hosts the canonical CapTP implementation rather than Agoric. Should I open an interop issue there referring to this one? |
|
This has mostly been coordinated on the repo @cwebber linked above: https://github.com/ocapn/ocapn ...though as you can see there's been a bit of a lull; iiuc @cwebber has been very busy getting the spritely institute up and running, so that's sapped some momentum, but hopefully this will get picked back up again. If you're looking for CapTP in Go, you could use the capnproto implementation: https://github.com/capnproto/go-capnproto2. Part of the interop effort is making sure that whatever we settle on is at least bridgeable to capnproto. (Note that I recommend using the latest alpha rather than the v2 branch; despite not being finalized it fixes some systemic problems with v2 and is therefore going to be more reliable). |
|
let's call this subsumed by the 19 Oct Spritely funding announcement and the work that's now underway: |
Spritely is getting to a more serious state, and the Spritely Goblins layer is now implementing CapTP (still a few things to be done, but I think it's maybe at feature-parity with Agoric's implementation... a little bit ahead actually because we now have distributed acyclic garbage collection). Of course, that couldn't have happened without the wonderful help of folks at Agoric and on the cap-talk list who answered many questions for me.
Now the question is, can we get our CapTP implementations to interop? Obviously I'm not the only one that thinks it's a good idea.
So let's talk about what's different so far, of substance and non-substance.
Non-substantial things to eventually address:
Of medium substance:
Of major substance:
The biggest one:
<-, and Lambda is the Ultimate. In Agoric's world, there are multiple kinds afaict: one for invoking methods, one for invoking procedures, one for fetching attributes (or are they called properties? I forget what term is used). Additionally, Agoric's implementation has a very nice feature where some kinds of scalars can also be indexed against to retrieve by numerical or string keys in a promise pipelining system, and Goblins doesn't provide this yet.I have a suggestion on how to solve this last one taking good ideas for both but it's worth making a separate post on this issue for that since this is already long. :)
Thankfully, our worlds are made easier by both of us keeping the CapTP layer separate from what's traditionally called the VatTP layer (confusing name imo, but not a big deal... I like MachineTP better for multiple reasons but we can talk about that later.)
Okay... so I think that lays out the scope of stuff to think about for interoperability. Let's plan on getting our stuff talking!
The text was updated successfully, but these errors were encountered: