[BACKPORT pg15-cherrypicks] all: Bulk port from master - 91

Summary: Manually excluded: f77dd6a [#23251] YCQL, ASH: Fix CQL test failures with ASH 1ebc289 [PLAT-13833]Upgrade ion java library 1302a2b [PLAT-15087] Connection pooling fails when ysqlAuth is enabled 1050ec4 [docs] Add a limitation for CDC in docs (#23586) Excluded: b57e3c6 [#23490] YSQL: Tighten notion of equality for update optimizations 9b4c4b5 [PLAT-10264] collect audit logs, ha config and xcluster data in support bundle Excluded: 9e0c569 Revert "[PLAT-14786] Add support to node_agent install to use bind ip and node_external_fqdn" 920989b [PLAT-14788]mask SAS token in backup config response and logs Excluded: 0ea4f54 [#23367] CDCSDK: Cleanup expired and not of interest tables from CDC stream Excluded: 0dc3a4a [#23737] YSQL: Change ysql conn mgr tests to fix them with warmup random mode eba9b49 [PLAT-13845] Upgrade aws sdk to 1.52+ f44c92e [PLAT-12933] [k8s] Ability to roll N nodes at a time during upgrades for multi-AZ(region) universes b8f0308 [PLAT-14008] Avoid rolling YBA managed n2n certificates when not needed (e.g. during Gflag Upgrades) Test Plan: Jenkins: rebase: pg15-cherrypicks Reviewers: jason, tfoucher Differential Revision: https://phorge.dev.yugabyte.com/D37738
yugabyte · Sep 4, 2024 · 6237668 · 6237668
1 parent b315db7
commit 6237668
Show file tree

Hide file tree

Showing 44 changed files with 912 additions and 237 deletions.
diff --git a/...content/preview/explore/change-data-capture/using-logical-replication/_index.md b/...content/preview/explore/change-data-capture/using-logical-replication/_index.md
@@ -105,7 +105,7 @@ To understand how the various features and configuration of the connector, see [
 
 - CDC is not supported on a target table for xCluster replication [11829](https://github.com/yugabyte/yugabyte-db/issues/11829).
 
-- Currently we don't support schema evolution for changes that require table rewrites (ex: ALTER TYPE).
+- Currently, CDC doesn't support schema evolution for changes that require table rewrites (for example, [ALTER TYPE](../../../api/ysql/the-sql-language/statements/ddl_alter_table/#alter-type-with-table-rewrite)), or DROP TABLE and TRUNCATE TABLE operations.
 
 - YCQL tables aren't currently supported. Issue [11320](https://github.com/yugabyte/yugabyte-db/issues/11320).
 

diff --git a/...preview/explore/change-data-capture/using-yugabytedb-grpc-replication/_index.md b/...preview/explore/change-data-capture/using-yugabytedb-grpc-replication/_index.md
@@ -58,7 +58,7 @@ To understand how the various features and configuration of the connector, see [
 * A single stream can only be used to stream data from one namespace only.
 * There should be a primary key on the table you want to stream the changes from.
 * CDC is not supported on a target table for xCluster replication [11829](https://github.com/yugabyte/yugabyte-db/issues/11829).
-* Currently we don't support schema evolution for changes that require table rewrites (ex: ALTER TYPE).
+* Currently, CDC doesn't support schema evolution for changes that require table rewrites (for example, [ALTER TYPE](../../../api/ysql/the-sql-language/statements/ddl_alter_table/#alter-type-with-table-rewrite)), or DROP TABLE and TRUNCATE TABLE operations.
 * YCQL tables aren't currently supported. Issue [11320](https://github.com/yugabyte/yugabyte-db/issues/11320).
 
 In addition, CDC support for the following features will be added in upcoming releases:

diff --git a/.../content/stable/explore/change-data-capture/using-logical-replication/_index.md b/.../content/stable/explore/change-data-capture/using-logical-replication/_index.md
@@ -105,7 +105,7 @@ To understand how the various features and configuration of the connector, see [
 
 - CDC is not supported on a target table for xCluster replication [11829](https://github.com/yugabyte/yugabyte-db/issues/11829).
 
-- Currently we don't support schema evolution for changes that require table rewrites (ex: ALTER TYPE).
+- Currently, CDC doesn't support schema evolution for changes that require table rewrites (for example, [ALTER TYPE](../../../api/ysql/the-sql-language/statements/ddl_alter_table/#alter-type-with-table-rewrite)), or DROP TABLE and TRUNCATE TABLE operations.
 
 - YCQL tables aren't currently supported. Issue [11320](https://github.com/yugabyte/yugabyte-db/issues/11320).
 

diff --git a/.../stable/explore/change-data-capture/using-yugabytedb-grpc-replication/_index.md b/.../stable/explore/change-data-capture/using-yugabytedb-grpc-replication/_index.md
@@ -56,7 +56,7 @@ To understand how the various features and configuration of the connector, see [
 * A single stream can only be used to stream data from one namespace only.
 * There should be a primary key on the table you want to stream the changes from.
 * CDC is not supported on a target table for xCluster replication [11829](https://github.com/yugabyte/yugabyte-db/issues/11829).
-* Currently we don't support schema evolution for changes that require table rewrites (ex: ALTER TYPE).
+* Currently, CDC doesn't support schema evolution for changes that require table rewrites (for example, [ALTER TYPE](../../../api/ysql/the-sql-language/statements/ddl_alter_table/#alter-type-with-table-rewrite)), or DROP TABLE and TRUNCATE TABLE operations.
 * YCQL tables aren't currently supported. Issue [11320](https://github.com/yugabyte/yugabyte-db/issues/11320).
 
 In addition, CDC support for the following features will be added in upcoming releases:

diff --git a/docs/content/v2.14/explore/change-data-capture/_index.md b/docs/content/v2.14/explore/change-data-capture/_index.md
@@ -153,6 +153,7 @@ If a new table is added to a namespace on which there is an active stream ID, a
 * YCQL tables aren't currently supported. Issue [11320](https://github.com/yugabyte/yugabyte-db/issues/11320).
 * CDC behaviour is undefined on downgrading from a CDC supported version (2.13 and newer) to an unsupported version (2.12 and older) and upgrading it back. Issue [12800](https://github.com/yugabyte/yugabyte-db/issues/12800)
 * CDC is not supported on a target table for xCluster replication [11829](https://github.com/yugabyte/yugabyte-db/issues/11829).
+* Currently, CDC doesn't support schema evolution for changes that require table rewrites (for example, [ALTER TYPE](../../api/ysql/the-sql-language/statements/ddl_alter_table/)), or DROP TABLE and TRUNCATE TABLE operations.
 * A single stream can only be used to stream data from one namespace only.
 * There should be a primary key on the table you want to stream the changes from.
 

diff --git a/docs/content/v2.18/explore/change-data-capture/cdc-overview.md b/docs/content/v2.18/explore/change-data-capture/cdc-overview.md
@@ -47,6 +47,7 @@ The core primitive of CDC is the _stream_. Streams can be enabled and disabled o
 * YCQL tables aren't currently supported. Issue [11320](https://github.com/yugabyte/yugabyte-db/issues/11320).
 * CDC behaviour is undefined on downgrading from a CDC supported version (2.13 and newer) to an unsupported version (2.12 and older) and upgrading it back. Issue [12800](https://github.com/yugabyte/yugabyte-db/issues/12800).
 * CDC is not supported on a target table for xCluster replication [11829](https://github.com/yugabyte/yugabyte-db/issues/11829).
+* Currently, CDC doesn't support schema evolution for changes that require table rewrites (for example, [ALTER TYPE](../../../api/ysql/the-sql-language/statements/ddl_alter_table/)), or DROP TABLE and TRUNCATE TABLE operations.
 * A single stream can only be used to stream data from one namespace only.
 * There should be a primary key on the table you want to stream the changes from.
 

diff --git a/docs/content/v2.20/explore/change-data-capture/cdc-overview.md b/docs/content/v2.20/explore/change-data-capture/cdc-overview.md
@@ -47,7 +47,7 @@ The core primitive of CDC is the _stream_. Streams can be enabled and disabled o
 * A single stream can only be used to stream data from one namespace only.
 * There should be a primary key on the table you want to stream the changes from.
 * CDC is not supported on a target table for xCluster replication [11829](https://github.com/yugabyte/yugabyte-db/issues/11829).
-* Currently we don't support schema evolution for changes that require table rewrites (ex: ALTER TYPE).
+* Currently, CDC doesn't support schema evolution for changes that require table rewrites (for example, [ALTER TYPE](../../../api/ysql/the-sql-language/statements/ddl_alter_table/#alter-type-with-table-rewrite)), or DROP TABLE and TRUNCATE TABLE operations.
 * YCQL tables aren't currently supported. Issue [11320](https://github.com/yugabyte/yugabyte-db/issues/11320).
 
 In addition, CDC support for the following features will be added in upcoming releases:

diff --git a/managed/build.sbt b/managed/build.sbt
@@ -174,11 +174,11 @@ libraryDependencies ++= Seq(
   "org.yaml" % "snakeyaml" % "2.1",
   "org.bouncycastle" % "bcpkix-jdk15on" % "1.61",
   "org.springframework.security" % "spring-security-core" % "5.8.11",
-  "com.amazonaws" % "aws-java-sdk-ec2" % "1.12.599",
-  "com.amazonaws" % "aws-java-sdk-kms" % "1.12.599",
-  "com.amazonaws" % "aws-java-sdk-iam" % "1.12.599",
-  "com.amazonaws" % "aws-java-sdk-sts" % "1.12.599",
-  "com.amazonaws" % "aws-java-sdk-s3" % "1.12.599",
+  "com.amazonaws" % "aws-java-sdk-ec2" % "1.12.768",
+  "com.amazonaws" % "aws-java-sdk-kms" % "1.12.768",
+  "com.amazonaws" % "aws-java-sdk-iam" % "1.12.768",
+  "com.amazonaws" % "aws-java-sdk-sts" % "1.12.768",
+  "com.amazonaws" % "aws-java-sdk-s3" % "1.12.768",
   "com.amazonaws" % "aws-java-sdk-elasticloadbalancingv2" % "1.12.327",
   "com.amazonaws" % "aws-java-sdk-route53" % "1.12.400",
   "com.amazonaws" % "aws-java-sdk-cloudtrail" % "1.12.498",

diff --git a/managed/src/main/java/com/yugabyte/yw/commissioner/KubernetesUpgradeTaskBase.java b/managed/src/main/java/com/yugabyte/yw/commissioner/KubernetesUpgradeTaskBase.java
@@ -13,6 +13,7 @@
 import com.yugabyte.yw.common.operator.OperatorStatusUpdater;
 import com.yugabyte.yw.common.operator.OperatorStatusUpdater.UniverseState;
 import com.yugabyte.yw.common.operator.OperatorStatusUpdaterFactory;
+import com.yugabyte.yw.forms.RollMaxBatchSize;
 import com.yugabyte.yw.forms.UniverseDefinitionTaskParams;
 import com.yugabyte.yw.forms.UniverseDefinitionTaskParams.Cluster;
 import com.yugabyte.yw.forms.UpgradeTaskParams;
@@ -72,8 +73,17 @@ protected void createPrecheckTasks(Universe universe) {
               .filter(n -> n.state != NodeDetails.NodeState.Live)
               .findFirst();
       if (nonLive.isEmpty()) {
-        List<MastersAndTservers> split = nodesToBeRestarted.splitToSingle();
-        createCheckNodesAreSafeToTakeDownTask(split, getTargetSoftwareVersion(), false);
+        RollMaxBatchSize rollMaxBatchSize = getCurrentRollBatchSize(universe);
+        // Use only primary nodes
+        MastersAndTservers forCluster =
+            nodesToBeRestarted.getForCluster(
+                universe.getUniverseDetails().getPrimaryCluster().uuid);
+
+        if (!forCluster.isEmpty()) {
+          List<MastersAndTservers> split =
+              UpgradeTaskBase.split(universe, forCluster, rollMaxBatchSize);
+          createCheckNodesAreSafeToTakeDownTask(split, getTargetSoftwareVersion(), true);
+        }
       }
     }
   }
@@ -128,6 +138,10 @@ private List<NodeDetails> fetchTServerNodes(Universe universe, UpgradeOption upg
 
   public abstract SubTaskGroupType getTaskSubGroupType();
 
+  private RollMaxBatchSize getCurrentRollBatchSize(Universe universe) {
+    return getCurrentRollBatchSize(universe, taskParams().rollMaxBatchSize);
+  }
+
   // Wrapper that takes care of common pre and post upgrade tasks and user has
   // flexibility to manipulate subTaskGroupQueue through the lambda passed in parameter
   public void runUpgrade(Runnable upgradeLambda) {
@@ -323,7 +337,6 @@ public void createUpgradeTask(
           null,
           ServerType.MASTER,
           softwareVersion,
-          taskParams().sleepAfterMasterRestartMillis,
           universeOverrides,
           azOverrides,
           isMasterChanged,
@@ -333,7 +346,9 @@ public void createUpgradeTask(
           commandType,
           enableYbc,
           ybcSoftwareVersion,
-          /* addDelayAfterStartup */ true);
+          PodUpgradeParams.builder()
+              .delayAfterStartup(taskParams().sleepAfterMasterRestartMillis)
+              .build());
     }
 
     if (isTServerChanged) {
@@ -348,7 +363,6 @@ public void createUpgradeTask(
           null,
           ServerType.TSERVER,
           softwareVersion,
-          taskParams().sleepAfterTServerRestartMillis,
           universeOverrides,
           azOverrides,
           false, // master change is false since it has already been upgraded.
@@ -358,7 +372,10 @@ public void createUpgradeTask(
           commandType,
           enableYbc,
           ybcSoftwareVersion,
-          /* addDelayAfterStartup */ true);
+          PodUpgradeParams.builder()
+              .delayAfterStartup(taskParams().sleepAfterTServerRestartMillis)
+              .rollMaxBatchSize(getCurrentRollBatchSize(universe))
+              .build());
 
       if (enableYbc) {
         Set<NodeDetails> primaryTservers = new HashSet<>(universe.getTServersInPrimaryCluster());
@@ -392,7 +409,6 @@ public void createUpgradeTask(
             null,
             ServerType.TSERVER,
             softwareVersion,
-            taskParams().sleepAfterTServerRestartMillis,
             universeOverrides,
             azOverrides,
             false, // master change is false since it has already been upgraded.
@@ -402,7 +418,10 @@ public void createUpgradeTask(
             commandType,
             enableYbc,
             ybcSoftwareVersion,
-            /* addDelayAfterStartup */ true);
+            PodUpgradeParams.builder()
+                .delayAfterStartup(taskParams().sleepAfterTServerRestartMillis)
+                .rollMaxBatchSize(getCurrentRollBatchSize(universe))
+                .build());
 
         if (enableYbc) {
           Set<NodeDetails> replicaTservers =
@@ -428,7 +447,6 @@ public void createUpgradeTask(
           null,
           ServerType.MASTER,
           softwareVersion,
-          taskParams().sleepAfterMasterRestartMillis,
           universeOverrides,
           azOverrides,
           isMasterChanged,
@@ -438,7 +456,9 @@ public void createUpgradeTask(
           commandType,
           enableYbc,
           ybcSoftwareVersion,
-          true);
+          PodUpgradeParams.builder()
+              .delayAfterStartup(taskParams().sleepAfterMasterRestartMillis)
+              .build());
     }
   }
 

diff --git a/managed/src/main/java/com/yugabyte/yw/commissioner/UpgradeTaskBase.java b/managed/src/main/java/com/yugabyte/yw/commissioner/UpgradeTaskBase.java
@@ -15,7 +15,6 @@
 import com.yugabyte.yw.commissioner.tasks.upgrade.SoftwareUpgradeYB;
 import com.yugabyte.yw.common.PlacementInfoUtil;
 import com.yugabyte.yw.common.PlatformServiceException;
-import com.yugabyte.yw.common.config.UniverseConfKeys;
 import com.yugabyte.yw.common.gflags.GFlagsUtil;
 import com.yugabyte.yw.common.kms.util.EncryptionAtRestUtil;
 import com.yugabyte.yw.forms.RollMaxBatchSize;
@@ -194,36 +193,10 @@ protected void createPrecheckTasks(Universe universe) {
   }
 
   private RollMaxBatchSize getCurrentRollBatchSize(Universe universe) {
-    RollMaxBatchSize rollMaxBatchSize = new RollMaxBatchSize();
-    if (taskParams().rollMaxBatchSize != null
-        && confGetter.getConfForScope(universe, UniverseConfKeys.upgradeBatchRollEnabled)) {
-      rollMaxBatchSize = taskParams().rollMaxBatchSize;
-    } else {
-      RollMaxBatchSize max = getMaxNodesToRoll(universe);
-      int percent =
-          confGetter.getConfForScope(universe, UniverseConfKeys.upgradeBatchRollAutoPercent);
-      int number =
-          confGetter.getConfForScope(universe, UniverseConfKeys.upgradeBatchRollAutoNumber);
-      int numberToSet = 0;
-      if (percent > 0) {
-        numberToSet = max.getPrimaryBatchSize() * percent / 100;
-      } else if (number > 1) {
-        numberToSet = Math.min(number, max.getPrimaryBatchSize());
-      }
-      if (numberToSet > 1) {
-        rollMaxBatchSize.setPrimaryBatchSize(numberToSet);
-        rollMaxBatchSize.setReadReplicaBatchSize(numberToSet);
-      }
-    }
-    if (getTaskCache() != null && getTaskCache().get(SPLIT_FALLBACK) != null) {
-      RollMaxBatchSize fallback = getTaskCache().get(SPLIT_FALLBACK, RollMaxBatchSize.class);
-      // Setting this only for primary cluster, RR still can be rolled with any speed.
-      rollMaxBatchSize.setPrimaryBatchSize(fallback.getPrimaryBatchSize());
-    }
-    return rollMaxBatchSize;
+    return getCurrentRollBatchSize(universe, taskParams().rollMaxBatchSize);
   }
 
-  private List<MastersAndTservers> split(
+  public static List<MastersAndTservers> split(
       Universe universe, MastersAndTservers forCluster, RollMaxBatchSize rollMaxBatchSize) {
     List<MastersAndTservers> result = new ArrayList<>();
     forCluster.mastersList.forEach(

diff --git a/managed/src/main/java/com/yugabyte/yw/commissioner/tasks/EditKubernetesUniverse.java b/managed/src/main/java/com/yugabyte/yw/commissioner/tasks/EditKubernetesUniverse.java
@@ -495,7 +495,6 @@ private boolean editCluster(
           curPlacement,
           ServerType.MASTER,
           newIntent.ybSoftwareVersion,
-          DEFAULT_WAIT_TIME_MS,
           universeOverrides,
           azOverrides,
           true,
@@ -505,7 +504,7 @@ private boolean editCluster(
           KubernetesCommandExecutor.CommandType.HELM_UPGRADE,
           universe.isYbcEnabled(),
           universe.getUniverseDetails().getYbcSoftwareVersion(),
-          /* addDelayAfterStartup */ false);
+          PodUpgradeParams.DEFAULT);
 
       upgradePodsTask(
           universe.getName(),
@@ -514,7 +513,6 @@ private boolean editCluster(
           curPlacement,
           ServerType.TSERVER,
           newIntent.ybSoftwareVersion,
-          DEFAULT_WAIT_TIME_MS,
           universeOverrides,
           azOverrides,
           false,
@@ -524,7 +522,7 @@ private boolean editCluster(
           KubernetesCommandExecutor.CommandType.HELM_UPGRADE,
           universe.isYbcEnabled(),
           universe.getUniverseDetails().getYbcSoftwareVersion(),
-          /* addDelayAfterStartup */ false);
+          PodUpgradeParams.DEFAULT);
     } else if (instanceTypeChanged) {
       upgradePodsTask(
           universe.getName(),
@@ -533,7 +531,6 @@ private boolean editCluster(
           curPlacement,
           ServerType.TSERVER,
           newIntent.ybSoftwareVersion,
-          DEFAULT_WAIT_TIME_MS,
           universeOverrides,
           azOverrides,
           false,
@@ -543,7 +540,7 @@ private boolean editCluster(
           KubernetesCommandExecutor.CommandType.HELM_UPGRADE,
           universe.isYbcEnabled(),
           universe.getUniverseDetails().getYbcSoftwareVersion(),
-          /* addDelayAfterStartup */ false);
+          PodUpgradeParams.DEFAULT);
     } else if (masterAddressesChanged) {
       // Update master_addresses flag on Master
       // and tserver_master_addrs flag on tserver without restart.