[CDCSDK] "Snapshot too old" error when dynamic table addition is disabled

[yugabyte-ci]

Jira Link: DB-13877

It is linked with the time at which we try to read the publication details i.e. last_pub_refresh_time. In the YugabyteDB cluster, if the flag for dynamic table support cdcsdk_enable_dynamic_table_support is disabled and hence last_pub_refresh_time value will not be changed and this will stay at the initial value assigned to it at the time of slot creation i.e. the consistent snapshot time.

Upon every task restart, we try to read the sys catalog as of the last_pub_refresh_time to get publication details and in the above case, we will try to read it as some timestamp equivalent to consistent snapshot time (set initially at the time of publication creation) which raised the “Snapshot too old” error because the history was cleared.

Workaround:
The current slot is not useful now, it needs to be dropped. Then the identified and tested workaround for the above is to enable dynamic table addition preview flag and increasing system catalog history retention. To enable the workaround, these steps need to be followed:

1. Since we are guaranteeing that the system can recover within x hours by the use of other retention flags, we should also ensure that the system catalog keeps the history for the same period of time.
   a. Set the tserver GFlag timestamp_syscatalog_history_retention_interval_sec to match x hours
2. Drop the existing slot.
3. Add the tserver GFlag cdcsdk_enable_dynamic_table_support to allowed_preview_flags_csv
   a. Set the flag true - this is required to keep updating the last_pub_refresh_time