[Platform] YSQL backups with node-to-node TLS encryption enabled hang forever

[tvesely]

A backup of a YSQL namespace with the Yugaware platform will hang indefinitely. The yb_backup.py script calls ysql_dump on a node:

$ ps -ef wwf | grep -C4 23947
root   23928 1995  0 Jan19 ? Ss 0:00 \_ sshd: ybuser [priv]
ybuser 23932 23928 0 Jan19 ? S  0:00 | \_ sshd: ybuser@notty
ybuser 23933 23932 0 Jan19 ? Ss 0:00 | \_ bash -c cd / && sudo -u yugabyte bash -c '/home/yugabyte/master/postgres/bin/ysql_dump --host=10.35.101.43 --masters=10.35.101.41:7100,10.35.101.42:7100,10.35.101.43:7100 --include-yb-metadata --serializable-deferrable --create --schema-only --dbname=yugabyte --file=/tmp/yb_backup_fwjnxcllppncgfkg/YSQLDump'
root   23945 23933 0 Jan19 ? S  0:00 | \_ sudo -u yugabyte bash -c /home/yugabyte/master/postgres/bin/ysql_dump --host=10.35.101.43 --masters=10.35.101.41:7100,10.35.101.42:7100,10.35.101.43:7100 --include-yb-metadata --serializable-deferrable --create --schema-only --dbname=yugabyte --file=/tmp/yb_backup_fwjnxcllppncgfkg/YSQLDump
yugabyte 23947 23945 0 Jan19 ? Sl 9:00 | \_ /home/yugabyte/master/postgres/bin/ysql_dump --host=10.35.101.43 --masters=10.35.101.41:7100,10.35.101.42:7100,10.35.101.43:7100 --include-yb-metadata --serializable-deferrable --create --schema-only --dbname=yugabyte --file=/tmp/yb_backup_fwjnxcllppncgfkg/YSQLDump

Because node_to_node_encryption is enabled, the ysql_dump utility is not able to connect to the master server. The ysql_dump command attempts to connect to the master nodes without TLS encryption, and the master server drops the connection. Because ysql_dump never gets a response from the master nodes, it retries the connection indefinitely...

[yugabyte@yb-demo-tvesely-yugabyte-4-n1 ~]$ /home/yugabyte/master/postgres/bin/ysql_dump --host=10.35.101.43 --masters=10.35.101.41:7100,10.35.101.42:7100,10.35.101.43:7100 --include-yb-metadata --serializable-deferrable --create --schema-only --dbname=yugabyte
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0120 21:47:39.557555 8802 ybc_pggate_tool.cc:58] Setting custom master addresses: 10.35.101.41:7100,10.35.101.42:7100,10.35.101.43:7100
I0120 21:47:39.558488 8802 pggate.cc:107] Reset YSQL bind address to 0.0.0.0:5432
I0120 21:47:39.558554 8802 server_base_options.cc:124] Updating master addrs to {10.35.101.41:7100},{10.35.101.42:7100},{10.35.101.43:7100}
I0120 21:47:39.558671 8802 mem_tracker.cc:249] MemTracker: hard memory limit is 53.480759 GB
I0120 21:47:39.558678 8802 mem_tracker.cc:251] MemTracker: soft memory limit is 45.458645 GB
I0120 21:47:39.559253 8802 thread_pool.cc:166] Starting thread pool { name: pggate_ybclient queue_limit: 10000 max_workers: 1024 }
I0120 21:47:39.559726 8809 async_initializer.cc:80] Starting to init ybclient
I0120 21:47:39.559849 8809 client-internal.cc:1976] New master addresses: [10.35.101.41:7100,10.35.101.42:7100,10.35.101.43:7100]
E0120 21:48:39.856043 8809 async_initializer.cc:93] Failed to initialize client: Timed out (yb/rpc/rpc.cc:213): Could not locate the leader master: GetLeaderMasterRpc(addrs: [10.35.101.41:7100, 10.35.101.42:7100, 10.35.101.43:7100], num_attempts: 344) passed its deadline 6672347.447s (passed: 60.294s): Network error (yb/util/net/socket.cc:535): recvmsg got EOF from remote (system error 108)

ysql_dump can be forced to use TLS by setting the following environment variables:

FLAGS_certs_dir=</path/to/certs/dir>
FLAGS_use_node_to_node_encryption=true

I believe that the Yugaware platform either needs to set these environment variables when calling ysql_dump, or the client itself needs to be made aware of the master's TLS settings.

[chirag-yb]

@streddy-yb This should be targeted for 2.4.1

[VijiYB]

Validated on 2.4.2
created universe with node-node TLS enabled.
Ran some workloads for ysql
Took backup & Restored