Skip to content

Commit

Permalink
Prevent rapid reset http2 DOS on API server
Browse files Browse the repository at this point in the history
This change fully addresses CVE-2023-44487 and CVE-2023-39325 for
the API server when the client is unauthenticated.

The changes to util/runtime are required because otherwise a large
number of requests can get blocked on the time.Sleep calls.

For unauthenticated clients (either via 401 or the anonymous user),
we simply no longer allow such clients to hold open http2
connections.  They can use http2, but with the performance of http1
(with keep-alive disabled).

Since this change has the potential to cause issues, the
UnauthenticatedHTTP2DOSMitigation feature gate can be disabled to
remove this protection (it is enabled by default).  For example,
when the API server is fronted by an L7 load balancer that is set up
to mitigate http2 attacks, unauthenticated clients could force
disable connection reuse between the load balancer and the API
server (many incoming connections could share the same backend
connection).  An API server that is on a private network may opt to
disable this protection to prevent performance regressions for
unauthenticated clients.

For all other clients, we rely on the golang.org/x/net fix in
golang/net@b225e7c
That change is not sufficient to adequately protect against a
motivated client - future changes to Kube and/or golang.org/x/net
will be explored to address this gap.

The Kube API server now uses a max stream of 100 instead of 250
(this matches the Go http2 client default).  This lowers the abuse
limit from 1000 to 400.

Signed-off-by: Monis Khan <[email protected]>

Kubernetes-commit: 238d89c9a068dcd7ab994be1b3e646ce8d296ef8
  • Loading branch information
enj authored and k8s-publishing-bot committed Oct 8, 2023
1 parent 1f3a244 commit 68b0397
Show file tree
Hide file tree
Showing 2 changed files with 35 additions and 6 deletions.
15 changes: 9 additions & 6 deletions pkg/util/runtime/runtime.go
Original file line number Diff line number Diff line change
Expand Up @@ -126,14 +126,17 @@ type rudimentaryErrorBackoff struct {
// OnError will block if it is called more often than the embedded period time.
// This will prevent overly tight hot error loops.
func (r *rudimentaryErrorBackoff) OnError(error) {
now := time.Now() // start the timer before acquiring the lock
r.lastErrorTimeLock.Lock()
defer r.lastErrorTimeLock.Unlock()
d := time.Since(r.lastErrorTime)
if d < r.minPeriod {
// If the time moves backwards for any reason, do nothing
time.Sleep(r.minPeriod - d)
}
d := now.Sub(r.lastErrorTime)
r.lastErrorTime = time.Now()
r.lastErrorTimeLock.Unlock()

// Do not sleep with the lock held because that causes all callers of HandleError to block.
// We only want the current goroutine to block.
// A negative or zero duration causes time.Sleep to return immediately.
// If the time moves backwards for any reason, do nothing.
time.Sleep(r.minPeriod - d)
}

// GetCaller returns the caller of the function that calls it.
Expand Down
26 changes: 26 additions & 0 deletions pkg/util/runtime/runtime_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,9 @@ import (
"os"
"regexp"
"strings"
"sync"
"testing"
"time"
)

func TestHandleCrash(t *testing.T) {
Expand Down Expand Up @@ -156,3 +158,27 @@ func captureStderr(f func()) (string, error) {

return <-resultCh, nil
}

func Test_rudimentaryErrorBackoff_OnError_ParallelSleep(t *testing.T) {
r := &rudimentaryErrorBackoff{
minPeriod: time.Second,
}

start := make(chan struct{})
var wg sync.WaitGroup
for i := 0; i < 30; i++ {
wg.Add(1)
go func() {
<-start
r.OnError(nil) // input error is ignored
wg.Done()
}()
}
st := time.Now()
close(start)
wg.Wait()

if since := time.Since(st); since > 5*time.Second {
t.Errorf("OnError slept for too long: %s", since)
}
}

0 comments on commit 68b0397

Please sign in to comment.