[BACKPORT pg15-cherrypicks] all: Bulk port from master - 45

Summary:
 25d427a [PLAT-13975] Support for non-restart gflags upgrade in Kubernetes universes
 26cd461 [PLAT-14165] Health check to report on time drift.
 9b83b80 [#23036] docdb: Switch CloneStateInfo object from scoped_refptr to std::shared_ptr
 e5399d9 [PLAT-14511] Improved yba installer migrations to allow some versions to not exist
 0db5d16 [14279] YSQL: Add basic flag validation for ysql_pg_conf_csv, ysql_hba_conf_csv and ysql_ident_conf_csv
 841fbdb [#21829] docdb: Fix scan that does not honor timeouts in certain scenarios, to not run into runaway scan scenarios.
 f20048b [#22840] DocDB: Fix DFATAL in WaitQueue::Impl::HandleWaiterStatusRpcResponse
 da6cbd5 [DOC-396] Set up SSO using Jumpcloud (#22861)
 d0e93c9 [PLAT-14104] Network configuration module for YNP
 63cceb3 [#22837] YSQL: Fix fast path insert into tables with identity columns
 5c7715f [#23057] YSQL: Fix missing newlines and pfree
 b3bc338 [doc][yba] DR clarifications (#23050)
 f897398 [#21500] YSQL: Skip schema version mismatch tests with Connection Manager enabled
 e77b95b [PLAT-14504] - feat : db audit log ui improvements
 7e289b5 [#21859] ASH: Make ASH uuid data human readable when viewed through /rpcz
 23acae3 [#22957] YSQL: Make connection sticky for GUC variables that cannot be SET in an explicit transaction with YSQL Connection Manager
 cfa6300 [#22556] docdb: Speed up backward scans for flat doc reader for packed row V2
 a446f27 [PLAT-14530] - fix : Edit Azure Provider is failing

"I worked around it by removing the two PNG files from the revision." - Steve Varnau

Test Plan: Jenkins: rebase: pg15-cherrypicks

Reviewers: jason, tfoucher

Tags: #jenkins-ready

Differential Revision: https://phorge.dev.yugabyte.com/D36285

docs/content/preview/yugabyte-cloud/managed-security/managed-authentication.md

@@ -38,7 +38,12 @@ Using federated authentication, you can use an enterprise IdP to manage access t
 
 Note that after federated authentication is enabled, only Admin users can sign in using email-based login.
 
-Currently, YugabyteDB Aeon supports Microsoft Entra ID, PingOne, and Okta enterprise IdPs, exclusively using the OIDC (OpenID Connect) protocol.
+Currently, YugabyteDB Aeon supports the following enterprise IdPs, exclusively using the OIDC (OpenID Connect) protocol:
+
+- Microsoft Entra ID
+- PingOne
+- Okta
+- JumpCloud
 
 ### Prerequisites
 
@@ -178,6 +183,64 @@ To configure federated authentication in YugabyteDB Aeon, do the following:
 1. Enter the Okta domain for your application.
 1. Click **Enable**.
 
+You are redirected to sign in to your IdP to test the connection. After the test connection is successful, federated authentication is enabled.
+
+  {{% /tab %}}
+
+  {{% tab header="JumpCloud" lang="jumpcloud" %}}
+
+**Create an application in JumpCloud**
+
+To use JumpCloud for your IdP, do the following:
+
+1. Sign in to JumpCloud using an administrator account.
+
+1. Create an application.
+
+    - Under **SSO Applications**, click **Add New Application**.
+    - Select **Custom Application**, and make sure the integration supports "SSO with OIDC" on the next page.
+    - Under **Manage Single Sign-On (SSO)**, select **Configure SSO with OIDC**, and click **Next**.
+    - Under **Enter General Info**, add the application name (for **Display Label**), **Description**, and logo (for **User Portal Image**), and select **Show this application in User Portal**.
+
+      This information is displayed as a tile when users sign in to YugabyteDB Aeon.
+
+    - Click **Configure Application**.
+
+1. Configure your application.
+
+    Under **SSO > Endpoint Configuration**, configure the following:
+
+    - **Redirect URIs** - enter `https://yugabyte-cloud.okta.com/oauth2/v1/authorize/callback`.
+    - **Client Authentication Type** - select **Client Secret Post**.
+    - **Login URL** - enter `https://cloud.yugabyte.com/login`.
+
+    Under **Attribute Mapping**, for **Standard Scopes**, select **Email** and **Profile**.
+
+    Click **Activate** when you are done.
+
+    You will be prompted in a pop up to save the **Client ID** and **Client Secret**. Save these in a secure location, you will need to provide these credentials in YugabyteDB Aeon.
+
+1. Configure Attributes and Identity Management as required.
+
+1. Integrate the user in JumpCloud.
+
+    - Navigate to **User Groups**, select the user groups you want to access YugabyteDB Aeon, and click **Save** when you are done.
+
+To configure JumpCloud federated authentication in YugabyteDB Aeon, you need the following application properties:
+
+- **Client ID** and **Client Secret** of the application you created. These are the credentials you saved when you activated your application. The **Client ID** is also displayed on the **SSO** tab.
+
+For more information, refer to the [JumpCloud](https://jumpcloud.com/support/sso-with-oidc) documentation.
+
+**Configure**
+
+To configure federated authentication in YugabyteDB Aeon, do the following:
+
+1. Navigate to **Security > Access Control > Authentication** and click **Enable Federated Authentication** to display the **Enable Federated Authentication** dialog.
+1. Choose JumpCloud identity provider.
+1. Enter the client ID and secret of the JumpCloud application you created.
+1. Click **Enable**.
+
 You are redirected to sign in to your IdP to test the connection. After the test connection is successful, federated authentication is enabled.
 
   {{% /tab %}}

docs/content/preview/yugabyte-platform/back-up-restore-universes/disaster-recovery/disaster-recovery-failover.md

@@ -41,6 +41,8 @@ If the DR primary is terminated for some reason, do the following:
 
 At this point, the DR configuration is halted and needs to be repaired.
 
+![Disaster recovery failed](/images/yb-platform/disaster-recovery/disaster-recovery-failed.png)
+
 ## Repair DR after failover
 
 There are two options to repair a DR that has failed over:
@@ -54,12 +56,18 @@ To repair DR, do the following:
 
 1. Navigate to your (new) DR primary universe and select **xCluster Disaster Recovery**.
 
-1. Click **Repair DR**.
+1. Click **Repair DR** to display the **Repair DR** dialog.
+
+    ![Repair DR](/images/yb-platform/disaster-recovery/disaster-recovery-repair.png)
 
-1. If the original DR primary has recovered and is fully functional with no active alerts, choose **Reuse the current DR replica**.
+1. If the current DR replica (formerly the DR primary) has recovered and is fully functional with no active alerts, choose **Reuse the current DR replica**.
 
     To use a new universe as the DR replica, choose **Select a new universe as DR replica** and select the universe.
 
 1. Click **Initiate Repair**.
 
-If your eventual desired configuration is for the other universe (that is, the one you have added to DR to act as DR replica) to be the DR primary, follow the steps for [Planned switchover](../disaster-recovery-switchover/).
+After the repair is complete, if your eventual desired configuration is for the replica (that is, the former primary if you chose Reuse, or the new one you added to DR to act as DR replica) to be the DR primary, follow the steps for [Planned switchover](../disaster-recovery-switchover/).
+
+{{< warning title="Important" >}}
+Do not attempt a switchover if you have not first repaired DR.
+{{< /warning >}}

docs/content/preview/yugabyte-platform/back-up-restore-universes/disaster-recovery/disaster-recovery-setup.md

@@ -23,10 +23,21 @@ Create two universes, the DR primary universe which will serve reads and writes,
 Ensure the universes have the following characteristics:
 
 - Both universes are running the same version of YugabyteDB (v2.18.0.0 or later).
-- Both universes have the same encryption in transit settings. Encryption in transit is recommended, and you should create the DR primary and DR replica universes with TLS enabled.
+- Both universes have the same [encryption in transit](../../../security/enable-encryption-in-transit/) settings. Encryption in transit is recommended, and you should create the DR primary and DR replica universes with TLS enabled.
 - They can be backed up and restored using the same backup configuration.
 - They have enough disk space to support storage of write-ahead logs (WALs) in case of a network partition or a temporary outage of the DR replica universe. During these cases, WALs will continue to write until replication is restored. Consider sizing your disk according to your ability to respond and recover from network or other infrastructure outages.
-- They have enough disk space. DR requires more disk space to store write ahead logs (WAL) in case of a network partition, or a temporary outage of the DR replica universe.
+- DR enables [Point-in-time-recovery](../../pitr/) (PITR) on the DR replica, requiring additional disk space for the replica.
+
+    PITR is used by DR during failover to restore the database to a consistent state. Note that if the DR replica universe already has PITR configured, that configuration is replaced by the DR configuration.
+
+    You can change the retention period for PITR used for DR by changing the following [runtime configuration](../../../administer-yugabyte-platform/manage-runtime-config/):
+
+    ```sh
+    yb.xcluster.transactional.pitr.default_retention_period
+    ```
+
+    The default value is 3 days.
+
 - Neither universe is already being used for xCluster replication.
 
 Prepare your database and tables on the DR primary. The DR primary can be empty or have data. If the DR primary has a lot of data, the DR setup will take longer because the data must be copied in full to the DR replica before on-going asynchronous replication starts.

docs/content/stable/yugabyte-platform/back-up-restore-universes/disaster-recovery/disaster-recovery-failover.md

@@ -41,6 +41,8 @@ If the DR primary is terminated for some reason, do the following:
 
 At this point, the DR configuration is halted and needs to be repaired.
 
+![Disaster recovery failed](/images/yb-platform/disaster-recovery/disaster-recovery-failed.png)
+
 ## Repair DR after failover
 
 There are two options to repair a DR that has failed over:
@@ -54,12 +56,18 @@ To repair DR, do the following:
 
 1. Navigate to your (new) DR primary universe and select **xCluster Disaster Recovery**.
 
-1. Click **Repair DR**.
+1. Click **Repair DR** to display the **Repair DR** dialog.
+
+    ![Repair DR](/images/yb-platform/disaster-recovery/disaster-recovery-repair.png)
 
-1. If the original DR primary has recovered and is fully functional with no active alerts, choose **Reuse the current DR replica**.
+1. If the current DR replica (formerly the DR primary) has recovered and is fully functional with no active alerts, choose **Reuse the current DR replica**.
 
     To use a new universe as the DR replica, choose **Select a new universe as DR replica** and select the universe.
 
 1. Click **Initiate Repair**.
 
-If your eventual desired configuration is for the other universe (that is, the one you have added to DR to act as DR replica) to be the DR primary, follow the steps for [Planned switchover](../disaster-recovery-switchover/).
+After the repair is complete, if your eventual desired configuration is for the replica (that is, the former primary if you chose Reuse, or the new one you added to DR to act as DR replica) to be the DR primary, follow the steps for [Planned switchover](../disaster-recovery-switchover/).
+
+{{< warning title="Important" >}}
+Do not attempt a switchover if you have not first repaired DR.
+{{< /warning >}}

docs/content/stable/yugabyte-platform/back-up-restore-universes/disaster-recovery/disaster-recovery-setup.md

@@ -23,10 +23,21 @@ Create two universes, the DR primary universe which will serve reads and writes,
 Ensure the universes have the following characteristics:
 
 - Both universes are running the same version of YugabyteDB (v2.18.0.0 or later).
-- Both universes have the same encryption in transit settings. Encryption in transit is recommended, and you should create the DR primary and DR replica universes with TLS enabled.
+- Both universes have the same [encryption in transit](../../../security/enable-encryption-in-transit/) settings. Encryption in transit is recommended, and you should create the DR primary and DR replica universes with TLS enabled.
 - They can be backed up and restored using the same backup configuration.
 - They have enough disk space to support storage of write-ahead logs (WALs) in case of a network partition or a temporary outage of the DR replica universe. During these cases, WALs will continue to write until replication is restored. Consider sizing your disk according to your ability to respond and recover from network or other infrastructure outages.
-- They have enough disk space. DR requires more disk space to store write ahead logs (WAL) in case of a network partition, or a temporary outage of the DR replica universe.
+- DR enables [Point-in-time-recovery](../../pitr/) (PITR) on the DR replica, requiring additional disk space for the replica.
+
+    PITR is used by DR during failover to restore the database to a consistent state. Note that if the DR replica universe already has PITR configured, that configuration is replaced by the DR configuration.
+
+    You can change the retention period for PITR used for DR by changing the following [runtime configuration](../../../administer-yugabyte-platform/manage-runtime-config/):
+
+    ```sh
+    yb.xcluster.transactional.pitr.default_retention_period
+    ```
+
+    The default value is 3 days.
+
 - Neither universe is already being used for xCluster replication.
 
 Prepare your database and tables on the DR primary. The DR primary can be empty or have data. If the DR primary has a lot of data, the DR setup will take longer because the data must be copied in full to the DR replica before on-going asynchronous replication starts.

docs/content/v2.20/yugabyte-platform/back-up-restore-universes/disaster-recovery/disaster-recovery-failover.md

@@ -41,6 +41,8 @@ If the DR primary is terminated for some reason, do the following:
 
 At this point, the DR configuration is halted and needs to be repaired.
 
+![Disaster recovery failed](/images/yb-platform/disaster-recovery/disaster-recovery-failed.png)
+
 ## Repair DR after failover
 
 There are two options to repair a DR that has failed over:
@@ -54,12 +56,18 @@ To repair DR, do the following:
 
 1. Navigate to your (new) DR primary universe and select **xCluster Disaster Recovery**.
 
-1. Click **Repair DR**.
+1. Click **Repair DR** to display the **Repair DR** dialog.
+
+    ![Repair DR](/images/yb-platform/disaster-recovery/disaster-recovery-repair.png)
 
-1. If the original DR primary has recovered and is fully functional with no active alerts, choose **Reuse the current DR replica**.
+1. If the current DR replica (formerly the DR primary) has recovered and is fully functional with no active alerts, choose **Reuse the current DR replica**.
 
     To use a new universe as the DR replica, choose **Select a new universe as DR replica** and select the universe.
 
 1. Click **Initiate Repair**.
 
-If your eventual desired configuration is for the other universe (that is, the one you have added to DR to act as DR replica) to be the DR primary, follow the steps for [Planned switchover](../disaster-recovery-switchover/).
+After the repair is complete, if your eventual desired configuration is for the replica (that is, the former primary if you chose Reuse, or the new one you added to DR to act as DR replica) to be the DR primary, follow the steps for [Planned switchover](../disaster-recovery-switchover/).
+
+{{< warning title="Important" >}}
+Do not attempt a switchover if you have not first repaired DR.
+{{< /warning >}}

docs/content/v2.20/yugabyte-platform/back-up-restore-universes/disaster-recovery/disaster-recovery-setup.md

@@ -23,10 +23,21 @@ Create two universes, the DR primary universe which will serve reads and writes,
 Ensure the universes have the following characteristics:
 
 - Both universes are running the same version of YugabyteDB (v2.18.0.0 or later).
-- Both universes have the same encryption in transit settings. Encryption in transit is recommended, and you should create the DR primary and DR replica universes with TLS enabled.
+- Both universes have the same [encryption in transit](../../../security/enable-encryption-in-transit/) settings. Encryption in transit is recommended, and you should create the DR primary and DR replica universes with TLS enabled.
 - They can be backed up and restored using the same backup configuration.
 - They have enough disk space to support storage of write-ahead logs (WALs) in case of a network partition or a temporary outage of the DR replica universe. During these cases, WALs will continue to write until replication is restored. Consider sizing your disk according to your ability to respond and recover from network or other infrastructure outages.
-- They have enough disk space. DR requires more disk space to store write ahead logs (WAL) in case of a network partition, or a temporary outage of the DR replica universe.
+- DR enables [Point-in-time-recovery](../../pitr/) (PITR) on the DR replica, requiring additional disk space for the replica.
+
+    PITR is used by DR during failover to restore the database to a consistent state. Note that if the DR replica universe already has PITR configured, that configuration is replaced by the DR configuration.
+
+    You can change the retention period for PITR used for DR by changing the following [runtime configuration](../../../administer-yugabyte-platform/manage-runtime-config/):
+
+    ```sh
+    yb.xcluster.transactional.pitr.default_retention_period
+    ```
+
+    The default value is 3 days.
+
 - Neither universe is already being used for xCluster replication.
 
 Prepare your database and tables on the DR primary. The DR primary can be empty or have data. If the DR primary has a lot of data, the DR setup will take longer because the data must be copied in full to the DR replica before on-going asynchronous replication starts.

java/yb-pgsql/src/test/java/org/yb/pgsql/TestAlterTableWithConcurrentTxn.java

@@ -13,6 +13,7 @@
 package org.yb.pgsql;
 
 import static org.yb.AssertionWrappers.*;
+import static org.junit.Assume.*;
 
 import java.sql.Connection;
 import java.sql.Statement;
@@ -780,6 +781,13 @@ public void testDmlTransactionAfterAlterOnCurrentResourceWithoutCachedMetadata()
     // b) For non PG metadata cached table:
     //    Transaction should not conflict because new schema is used for
     //    DML operation and the original schema is not already cached.
+
+    // The test fails with Connection Manager as it is expected that a new
+    // session would latch onto a new physical connection. Instead, two logical
+    // connections use the same physical connection, leading to unexpected
+    // results as per the expectations of the test.
+    assumeFalse(BasePgSQLTest.UNIQUE_PHYSICAL_CONNS_NEEDED, isTestRunningWithConnectionManager());
+
     for (AlterCommand alterType : AlterCommand.values()) {
       LOG.info("Run INSERT txn after ALTER " + alterType + " cache set " + !cacheMetadataSetTrue);
       runDmlTxnWithAlterOnCurrentResource(Dml.INSERT, alterType, !cacheMetadataSetTrue,
@@ -792,6 +800,13 @@ public void testDmlTransactionAfterAlterOnCurrentResourceWithoutCachedMetadata()
 
   @Test
   public void testMultipleDmlTransactionWithAlterOnCurrentResource() throws Exception {
+
+    // The test fails with Connection Manager as it is expected that each new
+    // session would latch onto a new physical connection. Instead, any two
+    // logical connections share the same physical connection, leading to
+    // unexpected results as per the expectations of the test.
+    assumeFalse(BasePgSQLTest.UNIQUE_PHYSICAL_CONNS_NEEDED, isTestRunningWithConnectionManager());
+
     LOG.info("Run multiple transactions before altering the resource");
     runMultipleTxnsBeforeAlterTable();
     LOG.info("Run multiple transactions before and after altering the resource");

java/yb-pgsql/src/test/java/org/yb/pgsql/TestPgTransactions.java

@@ -1087,4 +1087,30 @@ public void testMiscellaneous() throws Exception {
       s1.execute("DROP TABLE test");
     }
   }
+
+  /**
+   * GHI 22837
+   * Verify that inserts into a table with identity column are executed without transaction
+   *
+   * @throws Exception
+   */
+  @Test
+  public void testIdentityNoTransaction() throws Exception {
+    Statement statement = connection.createStatement();
+    // Identity column is a primary key
+    statement.execute("CREATE TABLE test_id_pk (" +
+                      "id int generated always as identity PRIMARY KEY, v int)");
+    verifyStatementTxnMetric(statement, "INSERT INTO test_id_pk(v) VALUES (1)", 1);
+    verifyStatementTxnMetric(statement, "INSERT INTO test_id_pk(v) VALUES (2)", 1);
+    verifyStatementTxnMetric(statement, "INSERT INTO test_id_pk(v) VALUES (3)", 1);
+    statement.execute("DROP TABLE test_id_pk");
+
+    // Identity column is not a primary key
+    statement.execute("CREATE TABLE test_id_non_pk (" +
+                      "id int generated always as identity, k int PRIMARY KEY, v int)");
+    verifyStatementTxnMetric(statement, "INSERT INTO test_id_non_pk(k, v) VALUES (1, 1)", 1);
+    verifyStatementTxnMetric(statement, "INSERT INTO test_id_non_pk(k, v) VALUES (2, 2)", 1);
+    verifyStatementTxnMetric(statement, "INSERT INTO test_id_non_pk(k, v) VALUES (3, 3)", 1);
+    statement.execute("DROP TABLE test_id_non_pk");
+  }
 }