[YSQL] YSQL process is failed after rapid load of tables creation and load

[pilshchikov]

Jira Link: DB-11268

Description

Case:

1. 3 nodes RF=3 cluster c6g.large instances
2. Start cycle:
   2.1. Start SqlInserts workload, wait till load will be started
   2.2. Start SqlUpdates workload, wait till load will be started
   2.3. Start SqlDataLoad workload, wait till load will be started
   2.4. Start SqlSecondaryIndex workload, wait till load will be started
   2.5. Start SqlSnapshotTxns workload, wait till load will be started
   2.6. Start SqlForeignKeysAndJoins workload, wait till load will be started

After each 4 iterations will drop all tables created
On #8 iteration SqlForeignKeysAndJoins was failed to start on table creation step

2024-05-11 10:13:26,084 [main] ERROR SqlForeignKeysAndJoins - Failed to create tables
com.yugabyte.util.PSQLException: ERROR: Rpc timeout, passed: 16.479s, timeout: 15.000s, now: 5148.289s, last_read_time_: 5131.810s
	at com.yugabyte.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2675) ~[yb-stress-sample-apps-1.1.50.jar:?]
	at com.yugabyte.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2365) ~[yb-stress-sample-apps-1.1.50.jar:?]
	at com.yugabyte.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:355) ~[yb-stress-sample-apps-1.1.50.jar:?]
	at com.yugabyte.jdbc.PgStatement.executeInternal(PgStatement.java:490) ~[yb-stress-sample-apps-1.1.50.jar:?]
	at com.yugabyte.jdbc.PgStatement.execute(PgStatement.java:408) ~[yb-stress-sample-apps-1.1.50.jar:?]
	at com.yugabyte.jdbc.PgStatement.executeWithFlags(PgStatement.java:329) ~[yb-stress-sample-apps-1.1.50.jar:?]
	at com.yugabyte.jdbc.PgStatement.executeCachedSql(PgStatement.java:315) ~[yb-stress-sample-apps-1.1.50.jar:?]
	at com.yugabyte.jdbc.PgStatement.executeWithFlags(PgStatement.java:291) ~[yb-stress-sample-apps-1.1.50.jar:?]
	at com.yugabyte.jdbc.PgStatement.executeUpdate(PgStatement.java:265) ~[yb-stress-sample-apps-1.1.50.jar:?]
	at com.yugabyte.sample.apps.SqlForeignKeysAndJoins.createTablesIfNeeded(SqlForeignKeysAndJoins.java:101) [yb-stress-sample-apps-1.1.50.jar:?]
	at com.yugabyte.sample.Main.run(Main.java:174) [yb-stress-sample-apps-1.1.50.jar:?]
	at com.yugabyte.sample.Main.main(Main.java:89) [yb-stress-sample-apps-1.1.50.jar:?]

Happen on: 2.20.3.1-b2
But also reproduces on:

In logs im seeing that Postgres process was restarted exactly in 10:13 timeframe. There is no FATAL or cores found
Logs and link on report result in jira task

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

• I confirm this issue does not contain any sensitive information.

[pilshchikov]

not reproducable anymore