Rate limiting for streaming endpoints is behaving incorrectly

[bartekn]

What version are you using?

1.6.0

What did you do?

@marcelosalloum was streaming historical data using /transactions endpoint. Request rate was under allowed quota.

What did you expect to see?

200 OK responses only.

What did you see instead?

Some 429 Too Many Requests responses.

---

After debugging the issue it seems that rate limiting is broken when streaming. Users pay 1 point per DB access while streaming:

go/services/horizon/internal/handler.go

Lines 78 to 100 in 3284133

	func (we *web) streamHandler(jfn interface{}, sfn streamFunc, params interface{}) http.HandlerFunc {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
	ctx := r.Context()
	stream := sse.NewStream(ctx, w)
	var oldHash [32]byte
	for {
	lastLedgerState := ledger.CurrentState()
	// Rate limit the request if it's a call to stream since it queries the DB every second. See
	// https://github.com/stellar/go/issues/715 for more details.
	rateLimiter := we.rateLimiter
	if rateLimiter != nil {
	limited, _, err := rateLimiter.RateLimiter.RateLimit(rateLimiter.VaryBy.Key(r), 1)
	if err != nil {
	stream.Err(errors.Wrap(err, "RateLimiter error"))
	return
	}
	if limited {
	stream.Err(sse.ErrRateLimited)
	return
	}
	}

but also 1 point for HTTP request (via middleware):

go/services/horizon/internal/middleware.go

Lines 202 to 207 in 3284133

	func (w *web) RateLimitMiddleware(next http.Handler) http.Handler {
	if w.rateLimiter == nil {
	return next
	}
	return w.rateLimiter.RateLimit(next)
	}

This means that when streaming historical data the allowed request quota is actually 2x smaller.