[BACKPORT pg15-cherrypicks] all: Bulk port from master - 60+61

Summary:
5a76f6a [#23179] CDCSDK: Support data types with dynamically alloted oids in CDC
5820ccd [PLAT-14710] Do not return apiToken in response to getSessionInfo
dcfa9cd [docs] updates to CVE table status column (#23225)
c0b1ee8 [docs] Fix load balance keyword in drivers page (#23253)
2becb46 [docs] Add basic troubleshooting steps for read restart errors. (#23228)
1b9be2e [PLAT-12733] Kubernetes overrides in v2 api
50422f8 [#23011] YSQL: Enable ALTER TABLE IF EXISTS t RENAME c1 TO c2
7c55b95 [PLAT-14073] DB scoped failover+repair
5ac65eb [#22449] YSQL: import wal2json wal2json_2_6
9e046fb [#23163] YSQL: pg_partman: make 'inherit_template_properties' idempotent
2248dcd [#23163] YSQL: pg_partman: make 'apply_publications' idempotent
7b32d05 [#23163] YSQL: pg_partman: Make partition creation idempotent
b0349fe [PLAT-14710] update api doc for /session_info
fe37ffd [#23240] CDCSDK: Make test replicationConnectionConsumptionAllDataTypes more resilient

Test Plan: Jenkins: rebase: pg15-cherrypicks

Reviewers: tfoucher

Tags: #jenkins-ready

Differential Revision: https://phorge.dev.yugabyte.com/D36789

docs/content/preview/architecture/transactions/read-restart-error.md

@@ -32,6 +32,35 @@ A detailed scenario that explains how clock skew can result in the above mention
 
   * It should return the data even if the client issued the read from a different node after writing the data, because the following guarantee needs to be maintained: the database should always return data that was committed in the past (past refers to the user-perceived past, and not based on machine clocks).
 
-  * It should not return the data if the write is performed in the future (that is, after the read point had been chosen) **and** had a write timestamp later than the read point. Because if that is allowed, everything written in future would be trivially visible to the read. Note that the latter condition is important because it is okay to return data that was written after the read point was picked, if the write timestamp was earlier than the read point (this doesn't break and consistency or isolation guarantees).
+  * It should not return the data if the write is performed in the future (that is, after the read point had been chosen) **and** had a write timestamp later than the read point. Because if that is allowed, everything written in future would be trivially visible to the read. Note that the latter condition is important because it is okay to return data that was written after the read point was picked, if the write timestamp was earlier than the read point (this doesn't break any consistency or isolation guarantees).
 
 * If node `N2` finds writes in the range `(T1, T1+max_clock_skew]`, to avoid breaking the strong guarantee that a reader should always be able to read what was committed earlier, and to avoid reading data with a later write timestamp which was also actually written after the read had been issued, node `N2` raises a `Read restart` error.
+
+## Troubleshooting
+
+You can handle and mitigate read restart errors using the following techniques:
+
+- Implement retry logic in the application. Application retries can help mitigate read restart errors. Moreover, a statement or a transaction may fail in other ways such as transaction conflicts or infrastructure failures. Therefore, a retry mechanism is strongly recommended for a cloud-native, distributed database such as YugabyteDB.
+- Use SERIALIZABLE READ ONLY DEFERRABLE mode whenever possible. Read restart errors usually occur when the query is a SELECT statement with a large output footprint and there are concurrent writes that satisfy the SELECT statement.
+
+  Using DEFERRABLE will avoid a read restart error altogether. However, the tradeoff is that the statement waits out the maximum permissible clock skew before reading the data (which is max_clock_skew_usec that has a default of 500ms). This is not an issue for large SELECT statements running in the background because latency is not a priority.
+
+  Examples:
+
+  Set transaction properties at the session level.
+  ```sql
+  SET SESSION CHARACTERISTICS AS TRANSACTION ISOLATION LEVEL SERIALIZABLE READ ONLY DEFERRABLE;
+  SELECT * FROM large_table;
+  ```
+
+  Enclose the offending query within a transaction block.
+  ```sql
+  BEGIN TRANSACTION ISOLATION LEVEL SERIALIZABLE READ ONLY DEFERRABLE;
+  SELECT * FROM large_table;
+  COMMIT;
+  ```
+- Using read only, deferrable transactions is not always feasible, either because the query is not read only, or the query is part of a read-write transaction, or because an additional 500ms of latency is not acceptable. In these cases, try increasing the value of `ysql_output_buffer_size`.
+
+  This will enable YugabyteDB to retry the query internally on behalf of the user. As long as the output of a statement hasn't crossed ysql_output_buffer_size to result in flushing partial data to the external client, the YSQL query layer retries read restart errors for all statements in a Read Committed transaction block, for the first statement in a Repeatable Read transaction block, and for any standalone statement outside a transaction block. As a tradeoff, increasing the buffer size also increases the memory consumed by the YSQL backend processes, resulting in a higher risk of out-of-memory errors.
+
+  Be aware that increasing `ysql_output_buffer_size` is not a silver bullet. For example, the COPY command can still raise a read restart error even though the command has a one line output. Increasing `ysql_output_buffer_size` is not useful in this scenario. The application must retry the COPY command instead. Another example is DMLs such as INSERT/UPDATE/DELETE. These do not have enough output to overflow the buffer size. However, when these statements are executed in the middle of a REPEATABLE READ transaction (e.g. BEGIN ISOLATION LEVEL REPEATABLE READ; ... INSERT ... COMMIT;), a read restart error cannot be retried internally by YugabyteDB. The onus is on the application to ROLLBACK and retry the transaction.

docs/content/preview/drivers-orms/java/yugabyte-jdbc.md

@@ -126,7 +126,7 @@ The following table describes the connection parameters required to connect, inc
 | user | User connecting to the database | yugabyte
 | password | User password | yugabyte
 | `load-balance` | [Uniform load balancing](../../smart-drivers/#cluster-aware-connection-load-balancing) | Defaults to upstream driver behavior unless set to 'true'
-| `yb-servers-refresh-interval` | If `load_balance` is true, the interval in seconds to refresh the servers list | 300
+| `yb-servers-refresh-interval` | If `load-balance` is true, the interval in seconds to refresh the servers list | 300
 | `topology-keys` | [Topology-aware load balancing](../../smart-drivers/#topology-aware-connection-load-balancing) | If `load-balance` is true, uses uniform load balancing unless set to comma-separated geo-locations in the form `cloud.region.zone`.
 
 The following is an example JDBC URL for connecting to YugabyteDB:

docs/content/preview/drivers-orms/nodejs/yugabyte-node-driver.md

@@ -76,7 +76,7 @@ The following table describes the connection parameters required to connect, inc
 | user | Database user | yugabyte |
 | password | User password | yugabyte |
 | `loadBalance` | [Uniform load balancing](../../smart-drivers/#cluster-aware-connection-load-balancing) | Defaults to upstream driver behavior unless set to 'true' |
-| `ybServersRefreshInterval` | If `load_balance` is true, the interval in seconds to refresh the node list | 300
+| `ybServersRefreshInterval` | If `loadBalance` is true, the interval in seconds to refresh the node list | 300
 | `topologyKeys` | [Topology-aware load balancing](../../smart-drivers/#topology-aware-connection-load-balancing) | If `loadBalance` is true, uses uniform load balancing unless set to comma-separated geo-locations in the form `cloud.region.zone`. |
 
 Create a client to connect to the cluster using a connection string. The following is an example connection string for connecting to a YugabyteDB cluster with uniform and topology load balancing:

docs/content/preview/secure/vulnerability-disclosure-policy.md

@@ -14,7 +14,7 @@ type: docs
 
 An important part of Yugabyte's strategy for building a secure platform for our users is vulnerability reporting. We value working with the broader security research community and understand that fostering that relationship will help Yugabyte improve its own security posture. We take vulnerabilities very seriously regardless of source, and strongly encourage people to report security vulnerabilities **privately to our security team** before disclosing them in a public forum. Our goal is to surface vulnerabilities and resolve them privately before they can be exploited.
 
-## Our Commitment
+## Our commitment
 
 1. **In scope** We commit to investigate and address any reported issues, and request that you use the following process for the reporting of security vulnerabilities in the following products:
 
@@ -30,7 +30,7 @@ An important part of Yugabyte's strategy for building a secure platform for our
 
 1. We assure you that we will not initiate legal action against researchers who are acting in good faith and adhering to this process.
 
-## The Process
+## The process
 
 1. **Report the Concern.** If you have any security concerns or would like to report undisclosed security vulnerabilities in our products or services, please email us at [[email protected]](mailto:[email protected]). Note that we do not accept bug reports at this address.
 
@@ -49,7 +49,7 @@ An important part of Yugabyte's strategy for building a secure platform for our
 
 1. **Use Common Sense.** Please use common sense when looking for security issues with our products. Attacking or compromising Yugabyte users' installations, or attacks on our infrastructure are not permitted.
 
-## Next Steps
+## Next steps
 
 1. We will promptly investigate any reported issue. In certain cases, we may work privately with you to resolve the vulnerability. We may choose not to disclose information publicly while we investigate and mitigate any risk.
 
@@ -81,18 +81,18 @@ Note that this policy covers only vulnerabilities in the query layer of PostgreS
 
 | Product | Name | Fixed in YugabyteDB version | Status |
 | :------ | :--- | :--------------------- | :----- |
-| PostgreSQL&nbsp;(YSQL) | {{<cve "CVE-2019-10127">}} | | Not applicable |
-| PostgreSQL (YSQL) | {{<cve "CVE-2019-10128">}} | | Not applicable |
+| PostgreSQL&nbsp;(YSQL) | {{<cve "CVE-2019-10127">}} | | Not applicable: YugabyteDB only runs on Linux, this vulnerability is Windows-specific. |
+| PostgreSQL (YSQL) | {{<cve "CVE-2019-10128">}} | | Not applicable: YugabyteDB only runs on Linux, this vulnerability is Windows-specific. |
 | PostgreSQL (YSQL) | {{<cve "CVE-2019-10129">}} | [v2.7.1](/preview/releases/ybdb-releases/end-of-life/v2.7/#v2-7-1-1-may-25-2021)| Resolved |
 | PostgreSQL (YSQL) | {{<cve "CVE-2019-10130">}} | [v2.12.11.0](/preview/releases/ybdb-releases/end-of-life/v2.12/#v2.12.11.0), {{<release "2.14.3.0">}}, {{<release "2.15.4.0">}} | Resolved |
 | PostgreSQL (YSQL) | {{<cve "CVE-2019-10164">}} | [v2.12.11.0](/preview/releases/ybdb-releases/end-of-life/v2.12/#v2.12.11.0), {{<release "2.14.3.0">}}, {{<release "2.15.3.0">}} | Resolved |
 | PostgreSQL (YSQL) | {{<cve "CVE-2019-10208">}} | [v2.12.11.0](/preview/releases/ybdb-releases/end-of-life/v2.12/#v2.12.11.0), {{<release "2.14.3.0">}}, {{<release "2.15.4.0">}} | Resolved |
 | PostgreSQL (YSQL) | {{<cve "CVE-2019-10209">}} | [v2.12.11.0](/preview/releases/ybdb-releases/end-of-life/v2.12/#v2.12.11.0), {{<release "2.14.3.0">}}, {{<release "2.15.4.0">}} | Resolved |
-| PostgreSQL (YSQL) | {{<cve "CVE-2019-10210">}} | | Not applicable |
-| PostgreSQL (YSQL) | {{<cve "CVE-2019-10211">}} | | Not applicable |
-| PostgreSQL (YSQL) | {{<cve "CVE-2019-3466">}} | | Not applicable |
-| PostgreSQL (YSQL) | {{<cve "CVE-2020-10733">}} | | Not applicable |
-| PostgreSQL (YSQL) | {{<cve "CVE-2020-14349">}} | | Not applicable |
+| PostgreSQL (YSQL) | {{<cve "CVE-2019-10210">}} | | Not applicable: YugabyteDB only runs on Linux, this vulnerability is Windows-specific. |
+| PostgreSQL (YSQL) | {{<cve "CVE-2019-10211">}} | | Not applicable: YugabyteDB only runs on Linux, this vulnerability is Windows-specific. |
+| PostgreSQL (YSQL) | {{<cve "CVE-2019-3466">}} | | Not applicable: pg_ctlcluster is not included in installation. |
+| PostgreSQL (YSQL) | {{<cve "CVE-2020-10733">}} | | Not applicable: YugabyteDB only runs on Linux, this vulnerability is Windows-specific. |
+| PostgreSQL (YSQL) | {{<cve "CVE-2020-14349">}} | | Not applicable: YugabyteDB does not use logical replication. |
 | PostgreSQL (YSQL) | {{<cve "CVE-2020-14350">}} | [v2.12.11.0](/preview/releases/ybdb-releases/end-of-life/v2.12/#v2.12.11.0), {{<release "2.14.5.0">}}, {{<release "2.16.0.0">}}, {{<release "2.17.1.0">}}| Resolved |
 | PostgreSQL (YSQL) | {{<cve "CVE-2020-1720">}} | | Resolved |
 | PostgreSQL (YSQL) | {{<cve "CVE-2020-25694">}} | [v2.7.1](/preview/releases/ybdb-releases/end-of-life/v2.7/#v2-7-1-1-may-25-2021) or later | Resolved |
@@ -111,5 +111,5 @@ Note that this policy covers only vulnerabilities in the query layer of PostgreS
 | PostgreSQL (YSQL) | {{<cve "CVE-2022-2625">}} | [v2.12.10.0](/preview/releases/ybdb-releases/end-of-life/v2.12/#v2.12.10.0), {{<release "2.14.2.0">}}, {{<release "2.15.3.0">}}| Resolved |
 | PostgreSQL (YSQL) | {{<cve "CVE-2023-2454">}} | {{<release "2.18.1.0">}}| Resolved |
 | PostgreSQL (YSQL) | {{<cve "CVE-2023-2455">}} | {{<release "2.14.10.2">}}, {{<release "2.16.5.0">}}, {{<release "2.18.0.0">}}, {{<release "2.20.0.0">}}| Resolved |
-| PostgreSQL (YSQL) | {{<cve "CVE-2023-32305">}} |  | Not applicable |
+| PostgreSQL (YSQL) | {{<cve "CVE-2023-32305">}} |  | Not applicable: [aiven-extras](https://github.com/aiven/aiven-extras) is not included in installation. |
 | PostgreSQL (YSQL) | {{<cve "CVE-2023-39417">}} | {{<release "2.20.1.0">}}, {{<release "2.14.15.0">}}, {{<release "2.16.9.0">}}, {{<release "2.18.5.0">}}| Resolved |

docs/content/stable/architecture/transactions/read-restart-error.md

@@ -32,6 +32,35 @@ A detailed scenario that explains how clock skew can result in the above mention
 
   * It should return the data even if the client issued the read from a different node after writing the data, because the following guarantee needs to be maintained: the database should always return data that was committed in the past (past refers to the user-perceived past, and not based on machine clocks).
 
-  * It should not return the data if the write is performed in the future (that is, after the read point had been chosen) **and** had a write timestamp later than the read point. Because if that is allowed, everything written in future would be trivially visible to the read. Note that the latter condition is important because it is okay to return data that was written after the read point was picked, if the write timestamp was earlier than the read point (this doesn't break and consistency or isolation guarantees).
+  * It should not return the data if the write is performed in the future (that is, after the read point had been chosen) **and** had a write timestamp later than the read point. Because if that is allowed, everything written in future would be trivially visible to the read. Note that the latter condition is important because it is okay to return data that was written after the read point was picked, if the write timestamp was earlier than the read point (this doesn't break any consistency or isolation guarantees).
 
 * If node `N2` finds writes in the range `(T1, T1+max_clock_skew]`, to avoid breaking the strong guarantee that a reader should always be able to read what was committed earlier, and to avoid reading data with a later write timestamp which was also actually written after the read had been issued, node `N2` raises a `Read restart` error.
+
+## Troubleshooting
+
+You can handle and mitigate read restart errors using the following techniques:
+
+- Implement retry logic in the application. Application retries can help mitigate read restart errors. Moreover, a statement or a transaction may fail in other ways such as transaction conflicts or infrastructure failures. Therefore, a retry mechanism is strongly recommended for a cloud-native, distributed database such as YugabyteDB.
+- Use SERIALIZABLE READ ONLY DEFERRABLE mode whenever possible. Read restart errors usually occur when the query is a SELECT statement with a large output footprint and there are concurrent writes that satisfy the SELECT statement.
+
+  Using DEFERRABLE will avoid a read restart error altogether. However, the tradeoff is that the statement waits out the maximum permissible clock skew before reading the data (which is max_clock_skew_usec that has a default of 500ms). This is not an issue for large SELECT statements running in the background because latency is not a priority.
+
+  Examples:
+
+  Set transaction properties at the session level.
+  ```sql
+  SET SESSION CHARACTERISTICS AS TRANSACTION ISOLATION LEVEL SERIALIZABLE READ ONLY DEFERRABLE;
+  SELECT * FROM large_table;
+  ```
+
+  Enclose the offending query within a transaction block.
+  ```sql
+  BEGIN TRANSACTION ISOLATION LEVEL SERIALIZABLE READ ONLY DEFERRABLE;
+  SELECT * FROM large_table;
+  COMMIT;
+  ```
+- Using read only, deferrable transactions is not always feasible, either because the query is not read only, or the query is part of a read-write transaction, or because an additional 500ms of latency is not acceptable. In these cases, try increasing the value of `ysql_output_buffer_size`.
+
+  This will enable YugabyteDB to retry the query internally on behalf of the user. As long as the output of a statement hasn't crossed ysql_output_buffer_size to result in flushing partial data to the external client, the YSQL query layer retries read restart errors for all statements in a Read Committed transaction block, for the first statement in a Repeatable Read transaction block, and for any standalone statement outside a transaction block. As a tradeoff, increasing the buffer size also increases the memory consumed by the YSQL backend processes, resulting in a higher risk of out-of-memory errors.
+
+  Be aware that increasing `ysql_output_buffer_size` is not a silver bullet. For example, the COPY command can still raise a read restart error even though the command has a one line output. Increasing `ysql_output_buffer_size` is not useful in this scenario. The application must retry the COPY command instead. Another example is DMLs such as INSERT/UPDATE/DELETE. These do not have enough output to overflow the buffer size. However, when these statements are executed in the middle of a REPEATABLE READ transaction (e.g. BEGIN ISOLATION LEVEL REPEATABLE READ; ... INSERT ... COMMIT;), a read restart error cannot be retried internally by YugabyteDB. The onus is on the application to ROLLBACK and retry the transaction.

docs/content/stable/drivers-orms/java/yugabyte-jdbc.md

@@ -121,7 +121,7 @@ The following table describes the connection parameters required to connect, inc
 | user | User connecting to the database | yugabyte
 | password | User password | yugabyte
 | `load-balance` | [Uniform load balancing](../../smart-drivers/#cluster-aware-connection-load-balancing) | Defaults to upstream driver behavior unless set to 'true'
-| `yb-servers-refresh-interval` | If `load_balance` is true, the interval in seconds to refresh the servers list | 300
+| `yb-servers-refresh-interval` | If `load-balance` is true, the interval in seconds to refresh the servers list | 300
 | `topology-keys` | [Topology-aware load balancing](../../smart-drivers/#topology-aware-connection-load-balancing) | If `load-balance` is true, uses uniform load balancing unless set to comma-separated geo-locations in the form `cloud.region.zone`.
 
 The following is an example JDBC URL for connecting to YugabyteDB:

docs/content/stable/drivers-orms/nodejs/yugabyte-node-driver.md

@@ -76,7 +76,7 @@ The following table describes the connection parameters required to connect, inc
 | user | Database user | yugabyte |
 | password | User password | yugabyte |
 | `loadBalance` | [Uniform load balancing](../../smart-drivers/#cluster-aware-connection-load-balancing) | Defaults to upstream driver behavior unless set to 'true' |
-| `ybServersRefreshInterval` | If `load_balance` is true, the interval in seconds to refresh the node list | 300
+| `ybServersRefreshInterval` | If `loadBalance` is true, the interval in seconds to refresh the node list | 300
 | `topologyKeys` | [Topology-aware load balancing](../../smart-drivers/#topology-aware-connection-load-balancing) | If `loadBalance` is true, uses uniform load balancing unless set to comma-separated geo-locations in the form `cloud.region.zone`. |
 
 Create a client to connect to the cluster using a connection string. The following is an example connection string for connecting to a YugabyteDB cluster with uniform and topology load balancing: