TestIntegrations/Disconnection/concurrent_connection_limits_exceeded_node_recording flakiness

[zmb3]

Failure

Link(s) to logs

• https://github.com/gravitational/teleport/actions/runs/6148508988/job/16682365820

Relevant snippet

        	            				/__w/teleport/teleport/integration/integration_test.go:2058
        	Error:      	Async error
        	Test:       	TestIntegrations/Disconnection/concurrent_connection_limits_exceeded_node_recording
        	Messages:   	Expected error to contain "administratively prohibited", got: failed connecting to host localhost:37531: failed to receive cluster details response
        	            		failed to dial target host
        	            		direct dialing to nodes not found in inventory is not supported

[zmb3]

FYI @lxea looks like more flakiness due to disabling dialing non-inventory nodes

[zmb3]

Got a similar hit in TestIntegrations/Disconnection/verify_that_lost_connections_to_auth_server_terminate_controlled_connections

https://github.com/gravitational/teleport/actions/runs/6227499962/job/16902262197

[zmb3]

https://github.com/gravitational/teleport/actions/runs/6409770371/job/17401690974

{"caller":"auth/middleware.go:338","component":"auth:29","level":"debug","message":"Server certificate cert(00000000-0000-0000-0000-000000000000.local-site issued by local-site:254193836761677461292139071673744226409).","timestamp":"2023-10-04T18:09:47Z"}
    integration_test.go:2164: 
        	Error Trace:	/__w/teleport/teleport/integration/integration_test.go:2164
        	            				/__w/teleport/teleport/integration/integration_test.go:2057
        	Error:      	Async error
        	Test:       	TestIntegrations/Disconnection/concurrent_connection_limits_exceeded_proxy_recording
        	Messages:   	Expected error to contain "administratively prohibited", got: failed connecting to host localhost:35533: failed to receive cluster details response
        	            		failed to dial target host
        	            		direct dialing to nodes not found in inventory is not supported

[nklaassen]

I got the same failure in https://github.com/gravitational/teleport/actions/runs/7066054434/job/19237240519?pr=35283

[GavinFrazar]

passes locally, failed in https://github.com/gravitational/teleport/actions/runs/7280185796/job/19838095046?pr=35949

=== FAIL: integration TestIntegrations/Disconnection/verify_that_lost_connections_to_auth_server_terminate_controlled_connections (3.85s)
    integration_test.go:2176: 
        	Error Trace:	/__w/teleport/teleport/integration/integration_test.go:2180
        	            				/opt/go/src/runtime/asm_amd64.s:1650
        	Error:      	Received unexpected error:
        	            	rpc error: code = Canceled desc = context canceled
    integration_test.go:2176: 
        	Error Trace:	/__w/teleport/teleport/integration/integration_test.go:2181
        	            				/opt/go/src/runtime/asm_amd64.s:1650
        	Error:      	"[]" should have 1 item(s), but has 0
    integration_test.go:2176: 
        	Error Trace:	/__w/teleport/teleport/integration/integration_test.go:2176
        	            				/__w/teleport/teleport/integration/integration_test.go:2297
        	            				/__w/teleport/teleport/integration/integration_test.go:2197
        	Error:      	Condition never satisfied
        	Test:       	TestIntegrations/Disconnection/verify_that_lost_connections_to_auth_server_terminate_controlled_connections
        --- FAIL: TestIntegrations/Disconnection/verify_that_lost_connections_to_auth_server_terminate_controlled_connections (3.85s)