Node cannot connect over reverse_tunnel port

[Joerger]

Expected behavior:

The following proxy+auth and node configs should result in the node connecting successfully.

Auth+Proxy:

teleport:
  nodename: server01
  log:
    output: stderr
    severity: DEBUG

auth_service:
  enabled: yes
  cluster_name: "example.com"
  tokens:
    - node:foo
  authentication:
    type: local
    second_factor: off
    
proxy_service:
  enabled: yes
  public_addr: ["proxy.example.com"]
  listen_addr: 0.0.0.0:3023
  web_listen_addr: 0.0.0.0:3080
  tunnel_public_addr: proxy.example.com:3024

ssh_service:
  enabled: no

Node:

teleport:
  nodename: server03
  auth_token: "foo"
  auth_servers:
    - proxy.example.com:3024
  log:
    output: stderr
    severity: DEBUG

auth_service:
  enabled: no

proxy_service:
  enabled: no

ssh_service:
  enabled: yes
  listen_addr: 0.0.0.0:3022

Current behavior

The node is failing to connect with the following logs:

2022-05-04T17:18:28-07:00 DEBU [PROC:1]    Attempting to connect to Auth Server directly. auth-addrs:[proxy.example.com:3024] service/connect.go:867
2022-05-04T17:18:28-07:00 DEBU [PROC:1]    Failed to connect to Auth Server directly. auth-addrs:[proxy.example.com:3024] service/connect.go:873
2022-05-04T17:18:28-07:00 DEBU [PROC:1]    Attempting to discover reverse tunnel address. auth-addrs:[proxy.example.com:3024] service/connect.go:882
2022-05-04T17:18:28-07:00 DEBU [PROC:1]    Attempting to connect to Auth Server through tunnel. auth-addrs:[proxy.example.com:3024] service/connect.go:884
2022-05-04T17:18:28-07:00 DEBU             Attempting GET proxy.example.com:3024/webapi/find webclient/webclient.go:113
2022-05-04T17:18:28-07:00 WARN             Request for GET proxy.example.com:3024/webapi/find falling back to PLAIN HTTP webclient/webclient.go:131
2022-05-04T17:18:28-07:00 ERRO [PROC:1]    Failed to resolve tunnel address Get "http://proxy.example.com:3024/webapi/find": malformed HTTP response "SSH-2.0-Teleport" reversetunnel/transport.go:90
2022-05-04T17:18:28-07:00 DEBU [PROC:1]    Failed to connect to Auth Server directly. auth-addrs:[proxy.example.com:3024] error:[
ERROR REPORT:
Original Error: *trace.ConnectionProblemError Get "https://teleport.cluster.local/v2/domain": tls: first record does not look like a TLS handshake
Stack Trace:
	/home/bjoerger/gravitational/teleport/lib/httplib/httplib.go:145 github.com/gravitational/teleport/lib/httplib.ConvertResponse
	/home/bjoerger/gravitational/teleport/lib/auth/clt.go:287 github.com/gravitational/teleport/lib/auth.(*Client).Get
	/home/bjoerger/gravitational/teleport/lib/auth/clt.go:378 github.com/gravitational/teleport/lib/auth.(*Client).GetDomainName
	/home/bjoerger/gravitational/teleport/lib/auth/clt.go:1467 github.com/gravitational/teleport/lib/auth.(*Client).GetLocalClusterName
	/home/bjoerger/gravitational/teleport/lib/service/connect.go:971 github.com/gravitational/teleport/lib/service.(*TeleportProcess).newClientDirect
	/home/bjoerger/gravitational/teleport/lib/service/connect.go:868 github.com/gravitational/teleport/lib/service.(*TeleportProcess).newClient
	/home/bjoerger/gravitational/teleport/lib/service/connect.go:165 github.com/gravitational/teleport/lib/service.(*TeleportProcess).connect
	/home/bjoerger/gravitational/teleport/lib/service/connect.go:125 github.com/gravitational/teleport/lib/service.(*TeleportProcess).connectToAuthService
	/home/bjoerger/gravitational/teleport/lib/service/connect.go:64 github.com/gravitational/teleport/lib/service.(*TeleportProcess).reconnectToAuthService
	/home/bjoerger/gravitational/teleport/lib/service/service.go:2062 github.com/gravitational/teleport/lib/service.(*TeleportProcess).registerWithAuthServer.func1
	/home/bjoerger/gravitational/teleport/lib/service/supervisor.go:494 github.com/gravitational/teleport/lib/service.(*LocalService).Serve
	/home/bjoerger/gravitational/teleport/lib/service/supervisor.go:263 github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1
	/home/bjoerger/.tools/go/src/runtime/asm_amd64.s:1571 runtime.goexit
User Message: Get "https://teleport.cluster.local/v2/domain": tls: first record does not look like a TLS handshake] service/connect.go:891
2022-05-04T17:18:28-07:00 DEBU [PROC:1]    Failed to connect to Auth Server through tunnel. auth-addrs:[proxy.example.com:3024] error:[Get "https://teleport.cluster.local/v2/domain": Get "http://proxy.example.com:3024/webapi/find": malformed HTTP response "SSH-2.0-Teleport"] service/connect.go:892
2022-05-04T17:18:28-07:00 ERRO [PROC:1]    "Node failed to establish connection to cluster: Failed to connect to Auth Server directly or over tunnel, no methods remaining.\n\tGet \"https://teleport.cluster.local/v2/domain\": tls: first record does not look like a TLS handshake, Get \"https://teleport.cluster.local/v2/domain\": Get \"http://proxy.example.com:3024/webapi/find\": malformed HTTP response \"SSH-2.0-Teleport\"." service/connect.go:86
2022-05-04T17:18:28-07:00 ERRO [PROC:1]    Failed to resolve tunnel address Get "http://proxy.example.com:3024/webapi/find": malformed HTTP response "SSH-2.0-Teleport" reversetunnel/transport.go:90

Bug details:

• Teleport version: master + v9.1.2

[Joerger]

@rosstimothy It looks like this is related to #9958

The node connects successfully when I replace

teleport/lib/service/connect.go

Line 903 in cceac2d

resolver := reversetunnel.WebClientResolver(process.ExitContext(), authServers, lib.IsInsecureDevMode())

with

resolver := reversetunnel.StaticResolver(authServers[0].Addr)

[espadolini]

Was that ever supposed to work? All the (current, admittedly) docs state that auth_servers should point to either the auth servers' listener or the proxy servers' web listener.

@Joerger does registration work with that change? And does the node start in listener mode, or reverse tunnel mode?

[webvictim]

I'm with @espadolini - I don't think we've ever supported this.

Perhaps #11471 makes it sound like it should work?

[espadolini]

The funny part is that (with that change) it would totally work - except for first time connections, which would have no way to ever connect anonymously to the cluster to register, because all they have is the address of a SSH listener.

[Joerger]

I thought that this was possible in the past, but you're right that it only works for 2nd+ time connections, as the first connection must register with proxy/auth. I was probably only using tunnel address on pre-registered nodes. Thanks for explaining!