Support "background calls" style of invocations

[igalshilman]

We are missing a simple way to trigger a function without waiting it to actually complete.

In a recently learned use case, a user would like to trigger a long running workflow.

The way we currently do it, is by adding a service, that will do a background call to the real workflow service (like in the factoring example, verification test, etc') and indicate to the original caller that the request was durably accepted.

What can be useful for restate:

• a way to indicate to the ingress that the caller is not interested in a response.
• with the semantics of - the call has been accepted, and will be proceed later.
• some correlation information would be very nice to have via the headers or response body. Like the sid that can be useful for introspection, deletion etc'.

Here is an example of a possible interaction:

curl -H "Content-Type: application/json" -H "x-restate-background-call: true" -d '{ "foo" : "bar" }' restate:9090/invoke/foo.bar.workflow/start` 

a response example can be

202 Accepted
{
   sid: "....."
}

Not sure if headers are the correct approach in the long run, but it seems like less moving parts to manage at the moment.
Later I can see more requirements that would justify a brand new ingress endpoint, but not right now.

[slinkydeveloper]

The way I've been thinking about this over the past months is to implement this at the ingress level, by implementing a well defined dev.restate.Ingress built-in service or similar. This service will have a contract like the following (names and details to figure out):

service Ingress {
  rpc BackgroundInvoke(InvokeRequest) returns (BackgroundInvokeResponse);
}

message InvokeRequest {
  string service = 1;
  string method = 2;
  bytes payload = 3;
}

message BackgroundInvokeResponse {
  string sid = 1;
}

With this contract in place, we need a bit of glue code and we just reuse and profit everything we have in the ingress, including gRPC, grpc web and connect support. Notably:

• Keep this feature within the same paradigms that we already defined for the ingress, reducing the mental overload of "special cases" and "different entrypoints"
• This approach works with gRPC and grpc-web. I believe the header approach won't work well with gRPC, as the result must be the same as defined in the contract, so even if sending an empty response with some response headers it's valid in terms of parsing, it doesn't make much sense.
• Straightforward to implement and fit in the current codebase (the header solution probably requires a good amount of code for a different codepath).
• Retain some flexibility for future features. That InvokeRequest can host new flags as soon as we introduce new features/semantics. The Ingress service itself will probably grow with more methods as well.
• In general, I like having a well defined contract for this operation, rather than an header, as it more explicitly sets the expectations.

At the end of the day the user experience will look close to what you posted:

curl -H "Content-Type: application/json" -d '{ "service": "MySvc", "method": "MyMethod", "payload" : {"foo" : "bar"} }' restate:9090/dev.restate.Ingress/BackgroundInvoke` 

WDYT?

[slinkydeveloper]

Relevant discussion about returning invocation metadata: #288

[igalshilman]

I like it as well @slinkydeveloper, I'd be happy if we can have that 👍