Horizon txsub endpoint should return 503 Service Unavailable when underlying Stellar-Core not synced

[bartekn]

What version are you using?

2.1.0

What did you do?

During the SDF outage on April, 6th 2021 Horizon POST /transactions was accepting transactions and forwarding them to Stellar-Core. While Stellar-Core (maybe?) broadcasted these transactions to the network Horizon was not able to ingest ledgers (because Stellar-Core was not synced) thus it couldn't check the status of submitted transactions.

What did you expect to see?

Horizon should detect if Stellar-Core it's using for submitting transactions is synced. If not, it should return 503 Service Unavailable and do not attempt to submit transactions.

The status can be probably checked using stellarcore.Client.Info.

What did you see instead?

All valid transactions submitted during the outage were forwarded to Stellar-Core and were getting 504 Gateway Timeout.