Working with Ephemeral Clusters

[mickeypash]

Hey team,

This is a bit of an open ended question. I have a Teleport set up with multiple clusters.
These clusters are ephemeral i.e. they are torn down all the time. DynamoDB is littered with old cluster definitions.

My first step was to write a naive script that would purge all the cluster states from DynamoDB.
However I'm still seeing errors in the logs.

• Is there a way I can clear the proxy's cache?
• How should I remove clusters from Teleport?
• Are this questions sensible or am I missing something?

Auth server logs for my main admin cluster:

ERRO [AUTH:1]    "Failed to retrieve client pool. Client cluster ip-10-xx-xxx-xxx, target cluster graviton, error:  
ERROR REPORT:
Original Error: *trace.NotFoundError key \"/authorities/host/ip-10-xx-xxx-xxx\" is not found
Stack Trace:
     /go/src/github.com/gravitational/teleport/lib/backend/memory/memory.go:186 
github.com/gravitational/teleport/lib/backend/memory.
(*Memory).Get
    /go/src/github.com/gravitational/teleport/lib/backend/report.go:161 
github.com/gravitational/teleport/lib/backend.
(*Reporter).Get
    /go/src/github.com/gravitational/teleport/lib/backend/wrap.go:89 
github.com/gravitational/teleport/lib/backend.
(*Wrapper).Get
    /go/src/github.com/gravitational/teleport/lib/services/local/trust.go:207 
github.com/gravitational/teleport/lib/services/local.
(*CA).GetCertAuthority
    /go/src/github.com/gravitational/teleport/lib/cache/cache.go:913 
github.com/gravitational/teleport/lib/cache.
(*Cache).GetCertAuthority
   go/src/github.com/gravitational/teleport/lib/auth/middleware.go:546 
github.com/gravitational/teleport/lib/auth.ClientCertPool
    /go/src/github.com/gravitational/teleport/lib/auth/middleware.go:253 github.com/gravitational/teleport/lib/auth.
(*TLSServer).GetConfigForClient
    /opt/go/src/crypto/tls/handshake_server.go:141 crypto/tls.
(*Conn).readClientHello
    /opt/go/src/crypto/tls/handshake_server.go:40 crypto/tls.
(*Conn).serverHandshake
    /opt/go/src/crypto/tls/conn.go:1362 crypto/tls.
(*Conn).Handshake
    /go/src/github.com/gravitational/teleport/lib/multiplexer/tls.go:141 
github.com/gravitational/teleport/lib/multiplexer.(*TLSListener).detectAndForward
    /opt/go/src/runtime/asm_amd64.s:1375
 runtime.goexit
   User Message: key \"/authorities/host/ip-10-xx-xxx-xxx\" is not found\n." auth/middleware.go:261

Proxy (I see loads of errors for this)
The web console still has a list of clusters that a are long gone.

WARN [REVERSE:S] Re-init the cache on error: connection error: desc = "transport: Error while dialing cluster MyTempCluster is offline". cache/cache.go:646

[klizhentas]

You can use tctl rm rc/<cluster-name from the root cluster to remove leaf cluster. Also would be great if we supported auto-expiring clusters in the future, but this feature is not here yet.

[mickeypash]

Hey thanks Alex! 🙌
Is there a way I can do this via the web api? I am trying to figure out how I could automate this step.
Also can you explain why I still see errors even after deleting entries from DynamoDB?
There seems to be some caching going on but what is the expiry? The logs suggest that the entries are in the cache long after they've been deleted from the Dynamo table.

[mickeypash]

Oh to answer my own question. I found the API docs here: https://goteleport.com/teleport/docs/api-reference/
Now the hurdle is to convince my team to upgrade and to start using Golang :)