Allow a module to declare that it produces network messages

[jclark]

Ballerina's service abstraction allows a module to declare how it responds to incoming messages: a client sends a message and gets a response.

In the async world a module produces messages, and the messages are, from the client's perspective, produced spontaneously, rather than in response to a request from the client. A client interacts with such a service by subscription: after subscription, the client will end up getting called for each message produced by the service. The mechanism by which the message produced by the module ends up being delivered to the client is determined by the network protocol in use.

When a Ballerina module needs to produce a message, it should be able to do so my making a call on a suitable object (perhaps a client object). The module should be able to create this object in a declarative way, so as to

• abstract out the networking details into library-provided classes (as Listeners do for services)
• enable the generation of artifacts that allow non-Ballerina network clients to subscribe to and consume the events in a type-safe way (e.g. using a GraphQL subscription)

This should probably be done as part of #440 and in particular #747.

[jclark]

@sanjiva This is my take on the requirement we discussed today.

[jclark]

We could have a publisher declaration and Publisher object type analogous to our current listener declaration and Listener object type.

A publisher declaration would specify

• identifier for the publisher being declared
• identifier of a class that implements Publisher
• arguments to the constructor of that class
• a client object type (or perhaps we just put the body inline like with service declarations)

The Publisher class uses the arguments plus a typedesc for the client object type to create a client object belonging to the client object type, which is then bound to the identifier. The application can then produce network messages by making calls to the client object. (This is analogous to how a service object uses a client object to send replies to the client.)

All the handling of subscriptions is dealt with by the Publisher class.

[jclark]

Actually we probably need to split this into two declarations, just as we do with listener and service. We need separate publisher and topic/channel declarations. For each topic/channel declaration, the Publisher object (created by the publisher declaration) creates a client object that can be used to publish to that topic/channel.

[sameerajayasoma]

Here is a well-written article that explains this use case. https://www.infoq.com/articles/microservices-inside-out/

[sameerajayasoma]

I am unsure whether we should handle subscriptions at the Ballerina level. Typically, publishers publish messages to a broker, which are then dispatched to subscribers.

[jclark]

Key requirements.

• it must be possible to implement an event producer in Ballerina in such a way that it is possible to generate an AsyncAPI description from the implementation
• it should be possible to incorporate this into our existing graphical view, which shows network interactions
• a program must be able to produce events even if it does not provide any services
• it must be able to work with a broker (i.e. the Ballerina program would simply send events that it produces to a broker, which would take care of managing subscriptions from event consumers)

[jclark]

This is an attempt at an approach with no new syntax. One point to note is that our sequence diagram view for functions recognizes client endpoints just from where the declaration occurs and the fact that the type is a client object type.

We introduce a Publisher object type, which is the dual of Listener: a Listener takes network messages and turns them into remote method calls on a service object; a Publisher takes remote method calls on a client object and turns them into network messages. There would be a Publisher class for each messaging protocol.

We can define a type for Publisher by using @sanjiva's favourite language feature:

type Publisher distinct object {
   public function newClientWithType(typedesc<client object {}> t = <>) returns t;
};

Then a module would do:

AmqpPublisher p = new("broker.example.com");
client object {
   // Define our event type
   remote function log(LogLevel level, string detail);
} c = p.newClientWithType();

public function main() {
   c->log("info", "starting up");
  foo();
   c->log("info", "shutting");
}

If we have a module-level variable with object type T initialized by a method call to newClientWithType on an object belong to the Publisher distinct object type, then we know that the remote method members of T describe events produced by the module.

Note that (just as with Listener) Ballerina does not yet have the language features needed to allow a Publisher to be implemented in Ballerina. Apart from the issue with implementing functions with dependent types, we would need something like dynamicNew.

Note also that the client object that emits events could potentially be declared at function level: not sure if this is good or bad.

[jclark]

If you are creating events from a service, then allowing events to be generated from multiple threads would get a little bit painful, since you would need to say:

final isolated client object {
   // Define our event type
   isolated remote function log(LogLevel level, string detail);
} c = p.newClientWithType();

[jclark]

Note that this now looks independent of #747. Whereas #747 is about using events to communicate between strands, this is about defining the network interactions of a program.

[jclark]

This needs the clarification in #1182 to ensure that the Publisher can participate in graceful shutdown.