services/horizon: support async transaction submission

[leighmcculloch]

What problem does your feature solve?

Horizon accepts transactions for submission to the network. It does this by:

1. Accepting the transaction.
2. Adding the transaction to a queue of transaction for the account.
3. Submitting the transaction to stellar-core when it is that transactions turn.
4. Waiting for the transaction result to be ingested.
5. Finally responding to the client with the transaction result.

It can take Horizon 5-30 seconds to perform the above steps.

A client's process/thread is held and blocked for the 5-30 seconds, consuming resources. During this time the client has no visibility into if the Horizon server has received the transaction or what is happening, and so the client must wait until it gets the final response 5-30 seconds later to know if any progress is being made.

Applications do not always want to keep a http connection open for ~5-30 seconds to find out the final state of the submitted transaction. In some cases applications would prefer to submit and lookup the transaction later to see if it was successful, failed, or just not included. In cases of scale and depending on the technologies the client is using, holding a http connection open for ~5-30 seconds may be prohibitive. Applications of scale usually avoid holding up processes for long periods of time. Applications that wish to submit multiple transactions must start multiple threads and handle error cases across each which is challenging to code and get right.

Additionally, all applications need to monitor the state of the transaction in failure or timeout cases anyway, since

@tomerweller and I found that when writing example code that submitted many transactions that Horizon's synchronous submission API made it difficult to adequately use resources on the client.

I also found that when writing code for github.com/stellar/starlight that Horizon's submission logic was inconvenient since I had to write concurrency code to be able to do other things while a transaction was sent to Horizon, or block the application while waiting unnecessarily.

In both cases it would have been much more convenient if immediate validation failure errors were available immediately and transaction results from being included in a ledger were available later.

What would you like to see?

I'd like an option for applications to submit transactions to Horizon asynchronously.

In other words, Horizon won't wait for the transaction result to be seen on the ledger before providing a response to the client.

An immediate failure from stellar-core's validation that runs during submission would still be returned immediately, but if stellar-core indicates that the transaction is "PENDING", horizon would return immediately in that case so that the calling application can move on and check back on transaction result later.

I think the existing transaction queue to be utilized, so if multiple transactions are submitted horizon still orders their submission to stellar-core in order of their sequence number, as it does today. Preserving this behavior for async transaction submission means that async and sync submission would have many of the same properties and behaviors, all that would be different is how the caller experiences a response. The async endpoint isn't intended to introduce new behaviors like skipping the transaction queue, the intent is for the only difference in behavior to be whether the client has to wait for the transaction result or not.

What alternatives are there?

Keep things the way they are and require that clients block their own processes.

Experimenting

I experimented with an incomplete implementation of this idea, although the transaction queue functionality was not preserved: #3564. I came to the conclusion during this experiment that preserving the transaction queue functionality is pretty important, otherwise the semantics between async and sync submission would be subtly different which would be difficult to understand as a user.

cc @tomerweller @ire-and-curses @stellar/horizon-committers

[ire-and-curses]

Prior discussion with a related idea from @bartekn: stellar/stellar-core#2920 (comment)

[sreuland]

May also be worthwhile to consider in the proposal to offer a couple async callback patterns for Horizon clients to choose from in their implementation, since, going async requires some additional 'lifting' on the client side and depending on the client context(server, mobile, web) different callbacks may be easier/advantageous for them, a few options are:

• Horizon Tx async endpoint returns SSE response that Horizon fills with finalized tx status:
• Horizon Tx async endpoint returns a unique url that returns finalized status of newly submitted tx, client sets up it's own long polling against it.
• Horizon Tx async endpoint accepts a 'webhook_url' request parameter, Horizon invokes the 'webhook_url' with finalized tx status.

[leighmcculloch]

Prior discussion with a related idea from @bartekn: stellar/stellar-core#2920 (comment)

@ire-and-curses That discussion ☝🏻 is related, but is mostly independent to this proposal. This issue (#4082) is focused on giving the user the option of not waiting for the transaction result but otherwise behaves the same as a sync submission. If the ideas in that other discussion are implemented they would probably apply to both sync and async submission I imagine, but regardless async could be implemented before or after those ideas.

[leighmcculloch]

Great ideas, thanks @sreuland.

going async requires some additional 'lifting' on the client side and depending on the client context(server, mobile, web)

Clients already have burdens of an async integration because even if a client waits for Horizon's response today, the client still needs to query the status of the transaction themselves in failure cases because even if a transaction fails it is always possible for it to succeed shortly after. That is to say, I agree, we need to make it easier for the different client contexts to collect the final result, but they also need to do this today regardless of this proposal.

I think the SSE capability you mention would be particularly useful:

• Horizon Tx async endpoint returns SSE response that Horizon fills with finalized tx status:

This sounds like a really useful feature, sort of similar to the existing transaction stream endpoints we can use on accounts, and it would provide a similar experience to the existing sync endpoint if you posted then called the SSE endpoint immediately following. It would allow the client to do something in between briefly only if Horizon didn't return a validation error and then return to waiting.

• Horizon Tx async endpoint returns a unique url that returns finalized status of newly submitted tx, client sets up it's own long polling against it.

FYI clients can do this today. The URL to poll is known before you submit, https://horizon.stellar.org/transactions/.

[leighmcculloch]

I think the existing transaction queue to be utilized, so if multiple transactions are submitted horizon still orders their submission to stellar-core in order of their sequence number, as it does today.

One problem with preserving ordering behavior is that it would cause the client to have to wait again while a transaction is in the queue, so it's possible that the idea of preserving that similarity between async and sync is not practical.

[sreuland]

on the SSE response from tx async request, one caveat there is that it needs the web server to be on HTTP/2 protocol to enable the browser/client to multiplex many SSE sessions over a singe connection, otherwise if server is HTTP 1.1, a couple SSE will occupy all the available http connections(limited to like 6 or fewer) per http host on client browsers quickly.

I checked the horizon web server responses, it looks like they are currently on HTTP 1.1, so would likely need to change server over to HTTP/2 which should mostly be transparent as any http client will handle negotiating to either 1.1 or 2. Going to HTTP/2 may provide a slightly better throughput in general as it requires less spin-up/tear down of sockets on the client in general.

This sse-per-tx-request would also require profiling server resource impact since each SSE is longer lived, consuming a tcp connection and occasional thread, if we start spawning these SSE for every tx request, they may be fairly short-lived but still combined with how many tx's are submitted could potentially surpass server thresholds for max tcp connections, etc.

An alternative pattern would be to suggest for clients to set up a listener(e.g. js EventSource) first on one of the current SSE endpoints such as /accounts or /transactions and then submit async tx's, and inform their listener to check for related tx attribs in the stream.

[Shaptic]

Unfortunately, HTTP/2 isn't an option without a much bigger investment into finding a compatible HTTP client library for the JavaScript SDK. The current solution, axios, doesn't support HTTP/2 (see stellar/js-stellar-sdk#612). Though I think it might finally be time to overcome that hurdle.

[sreuland]

Thanks for correction, ok, can't assume the switch to HTTP/2 would be transparent from SDK's then, each SDK would need to be evaluated for ability to support HTTP/2 connections based on the http client libs it's using. Perhaps in non-breaking steps, one consideration would be to add HTTP/2 support as config option in SDKs first, and then do server HTTP/2 changes after, so SDK's can negotiate under the hood to a Horizon server that is on either HTTP version for compatibility.