[BACKPORT pg15-cherrypicks] all: Bulk port from master - 59

Summary: a64fbfc [#23120] YSQL: Do not wait for a safe snapshot in serializable read only deferrable mode. 9f82c01 [#23188] DocDB: Persist new colocated_id mapping discovered as part of processing CHANGE_METADATA_OP in xCluster ClusterConfig. 1522522 Update third-party dependencies to remove CentOS 7 cd7410c 2.18.1.1 release notes (#23241) 550458d [#23047] docdb: Fix cotable ids in flushed frontier at restore 225ddfe [PLAT-14700] Make node-agent error message on installation to be more precise 9c9a059 [PLAT-14563] Fixing disk mounting logic to use mount points defined in the config 84eefbd [DOC-368] Azure workload identity docs changes (#22881) 781af0d [#23183] xCluster: Move Setup, Bootstrap, Alter and Delete Target Replication functions out of CatalogManager 1d646b1 [docs] Add ysql_output_buffer_size to yb-tserver config reference. (#23233) 2d95dd2 Fix the issue yaml parsing dd9e85b [PLAT-14534]Add regex match for GCP Instance template d3bba18 update diagram (#23245) 78b317c [/PLAT-14708] Fix JSON field name in TaskInfo query ac9164b [#23173] DocDB: Allow large bytes to be passed to RateLimiter Test Plan: Jenkins: rebase: pg15-cherrypicks Reviewers: jason, tfoucher Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D36723
yugabyte · Jul 19, 2024 · b0de783 · b0de783
1 parent a062db4
commit b0de783
Show file tree

Hide file tree

Showing 71 changed files with 3,833 additions and 3,149 deletions.
diff --git a/build-support/thirdparty_archives.yml b/build-support/thirdparty_archives.yml
@@ -1,90 +1,101 @@
-sha: 31776c4936a67fe6f1fc218fb64a6a9909c77311
+sha: b6b07342fdfd4a65ee2608d75dd31e4b0ecc0737
 archives:
 
   - os_type: almalinux8
     architecture: x86_64
     compiler_type: clang17
-    tag: v20240620163555-31776c4936-almalinux8-x86_64-clang17
+    tag: v20240713003527-b6b07342fd-almalinux8-x86_64-clang17
+
+  - os_type: almalinux8
+    architecture: x86_64
+    compiler_type: clang18
+    tag: v20240713003521-b6b07342fd-almalinux8-x86_64-clang18
 
   - os_type: almalinux8
     architecture: x86_64
     compiler_type: gcc11
-    tag: v20240620163541-31776c4936-almalinux8-x86_64-gcc11
+    tag: v20240713003520-b6b07342fd-almalinux8-x86_64-gcc11
 
   - os_type: almalinux9
     architecture: x86_64
     compiler_type: clang17
-    tag: v20240620163555-31776c4936-almalinux9-x86_64-clang17
+    tag: v20240713003540-b6b07342fd-almalinux9-x86_64-clang17
 
   - os_type: almalinux9
     architecture: x86_64
     compiler_type: gcc12
-    tag: v20240620163623-31776c4936-almalinux9-x86_64-gcc12
+    tag: v20240713003537-b6b07342fd-almalinux9-x86_64-gcc12
 
-  - os_type: centos7
+  - os_type: amzn2
     architecture: aarch64
-    compiler_type: clang16
-    tag: v20240620163739-31776c4936-centos7-aarch64-clang16
+    compiler_type: clang17
+    tag: v20240713003725-b6b07342fd-amzn2-aarch64-clang17
 
-  - os_type: centos7
+  - os_type: amzn2
     architecture: aarch64
-    compiler_type: clang16
+    compiler_type: clang17
     lto_type: full
-    tag: v20240620163735-31776c4936-centos7-aarch64-clang16-full-lto
+    tag: v20240713003827-b6b07342fd-amzn2-aarch64-clang17-full-lto
 
-  - os_type: centos7
+  - os_type: amzn2
     architecture: aarch64
-    compiler_type: clang17
-    tag: v20240620163738-31776c4936-centos7-aarch64-clang17
+    compiler_type: clang18
+    tag: v20240713003831-b6b07342fd-amzn2-aarch64-clang18
 
-  - os_type: centos7
+  - os_type: amzn2
     architecture: aarch64
-    compiler_type: clang17
+    compiler_type: clang18
     lto_type: full
-    tag: v20240620163737-31776c4936-centos7-aarch64-clang17-full-lto
+    tag: v20240713003853-b6b07342fd-amzn2-aarch64-clang18-full-lto
 
-  - os_type: centos7
+  - os_type: amzn2
     architecture: x86_64
     compiler_type: clang17
-    tag: v20240620163551-31776c4936-centos7-x86_64-clang17
+    tag: v20240713003538-b6b07342fd-amzn2-x86_64-clang17
 
-  - os_type: centos7
+  - os_type: amzn2
     architecture: x86_64
     compiler_type: clang17
     lto_type: full
-    tag: v20240620163542-31776c4936-centos7-x86_64-clang17-full-lto
+    tag: v20240713003542-b6b07342fd-amzn2-x86_64-clang17-full-lto
 
-  - os_type: centos7
+  - os_type: amzn2
     architecture: x86_64
-    compiler_type: gcc11
-    tag: v20240620163544-31776c4936-centos7-x86_64-gcc11
+    compiler_type: clang18
+    tag: v20240713003544-b6b07342fd-amzn2-x86_64-clang18
+
+  - os_type: amzn2
+    architecture: x86_64
+    compiler_type: clang18
+    lto_type: full
+    tag: v20240713003540-b6b07342fd-amzn2-x86_64-clang18-full-lto
 
   - os_type: macos
     architecture: arm64
     compiler_type: clang
-    tag: v20240620173126-31776c4936-macos-arm64
+    tag: v20240713011052-b6b07342fd-macos-arm64
 
   - os_type: macos
     architecture: x86_64
     compiler_type: clang
-    tag: v20240620163640-31776c4936-macos-x86_64
+    tag: v20240713003540-b6b07342fd-macos-x86_64
 
   - os_type: ubuntu20.04
     architecture: x86_64
     compiler_type: clang16
-    tag: v20240620163547-31776c4936-ubuntu2004-x86_64-clang16
+    tag: v20240713003520-b6b07342fd-ubuntu2004-x86_64-clang16
 
   - os_type: ubuntu22.04
     architecture: x86_64
     compiler_type: clang17
-    tag: v20240620163538-31776c4936-ubuntu2204-x86_64-clang17
+    tag: v20240713003517-b6b07342fd-ubuntu2204-x86_64-clang17
 
   - os_type: ubuntu22.04
     architecture: x86_64
     compiler_type: gcc11
-    tag: v20240620163535-31776c4936-ubuntu2204-x86_64-gcc11
+    tag: v20240713003527-b6b07342fd-ubuntu2204-x86_64-gcc11
 
   - os_type: ubuntu23.04
     architecture: x86_64
     compiler_type: gcc13
-    tag: v20240620163558-31776c4936-ubuntu2304-x86_64-gcc13
+    tag: v20240713003516-b6b07342fd-ubuntu2304-x86_64-gcc13
diff --git a/docs/content/preview/architecture/docdb-replication/cdc-logical-replication.md b/docs/content/preview/architecture/docdb-replication/cdc-logical-replication.md
@@ -19,9 +19,9 @@ CDC in YugabyteDB is based on the PostgreSQL Logical Replication model. The fund
 
 ## Architecture
 
-![Logical-Replication-Architecture](/images/architecture/logical_replication_architecture.png)
+![Logical replication architecture](/images/architecture/cdc-logical-replication-architecture.png)
 
-The following are the main components of the Yugabyte CDC solution -
+The following are the main components of the Yugabyte CDC solution:
 
 1. Walsender - A special purpose PG backend responsible for streaming changes to the client and handling acknowledgments.
 
@@ -41,7 +41,7 @@ The initial snapshot data for each table is consumed by executing a correspondin
 
 First, a `SET LOCAL yb_read_time TO '<consistent_point commit time> ht'` command should be executed on the connection (session). The SELECT statement corresponding to the snapshot query should then be executed as part of the same transaction.
 
-The HybridTime value to use in the `SET LOCAL yb_read_time `command is the value of the `snapshot_name` field that is returned by the `CREATE_REPLICATION_SLOT` command. Alternatively, it can be obtained by querying the `pg_replication_slots` view.
+The HybridTime value to use in the `SET LOCAL yb_read_time` command is the value of the `snapshot_name` field that is returned by the `CREATE_REPLICATION_SLOT` command. Alternatively, it can be obtained by querying the `pg_replication_slots` view.
 
 During Snapshot consumption, the snapshot data from all tables will be from the same consistent state (`consistent_point`). At the end of Snapshot consumption, the state of the target system is at/based on the `consistent_point`. History of the tables as of the `consistent_point` is retained on the source until the snapshot is consumed.
 
@@ -67,14 +67,16 @@ VWAL collects changes across multiple tablets, assembles the transactions, assig
 
 Walsender sends changes to the output plugin, which filters them according to the slot's publication and converts them into the client's desired format. These changes are then streamed to the client using the appropriate streaming replication protocols determined by the output plugin. Yugabyte follows the same streaming replication protocols as defined in PostgreSQL.
 
+<!--TODO (Siddharth): Fix the Links to the protocol section.
+
 {{< note title="Note" >}}
-<!--TODO (Siddharth): Fix the Link to the protocol section. -->
-Refer to [Replication Protocol](../../../explore/logical-replication/#Streaming-Protocol) for more details.
+Refer to [Replication Protocol](../../../explore/change-data-capture/using-logical-replication/#streaming-protocol) for more details.
 
 {{< /note >}}
 
 {{< tip title="Explore" >}}
-<!--TODO (Siddharth): Fix the Link to the getting started section. -->
-See [Getting Started with Logical Replication](../../../explore/logical-replication/getting-started) to set up Logical Replication in YugabyteDB.
+
+See [Getting Started with Logical Replication](../../../explore/change-data-capture/using-logical-replication/getting-started/) to set up Logical Replication in YugabyteDB.
 
 {{< /tip >}}
+-->
diff --git a/docs/content/preview/reference/configuration/yb-tserver.md b/docs/content/preview/reference/configuration/yb-tserver.md
@@ -855,6 +855,14 @@ Default: `-1` (disables logging statement durations)
 
 Specifies the lowest YSQL message level to log.
 
+##### --ysql_output_buffer_size
+
+Size of YSQL layer output buffer, in bytes. YSQL buffers query responses in this output buffer until either a buffer flush is requested by the client or the buffer overflows.
+
+As long as no data has been flushed from the buffer, the database can retry queries on retryable errors. For example, you can increase the size of the buffer so that YSQL can retry [read restart errors](../../../architecture/transactions/read-restart-error).
+
+Default: `262144` (256kB, type: int32)
+
 ### YCQL
 
 The following flags support the use of the [YCQL API](../../../api/ycql/):

diff --git a/docs/content/preview/releases/yba-releases/v2.18.md b/docs/content/preview/releases/yba-releases/v2.18.md
@@ -33,6 +33,31 @@ What follows are the release notes for all releases in the **YugabyteDB Anywhere
 
 For an RSS feed of all release series to track the latest product updates, point your feed reader to the [RSS feed for releases](../index.xml).
 
+## v2.18.8.1 - July 18, 2024 {#v2.18.8.1}
+
+**Build:** `2.18.8.1-b3`
+
+**Third-party licenses:** [YugabyteDB](https://downloads.yugabyte.com/releases/2.18.8.1/yugabytedb-2.18.8.1-b3-third-party-licenses.html), [YugabyteDB Anywhere](https://downloads.yugabyte.com/releases/2.18.8.1/yugabytedb-anywhere-2.18.8.1-b3-third-party-licenses.html)
+
+### Download
+
+<ul class="nav yb-pills">
+ <li>
+   <a href="https://downloads.yugabyte.com/releases/2.18.8.1/yba_installer_full-2.18.8.1-b3-linux-x86_64.tar.gz">
+     <i class="fa-brands fa-linux"></i>
+     <span>Linux x86</span>
+   </a>
+ </li>
+</ul>
+
+For instructions on installing YugabyteDB Anywhere, refer to [Install YugabyteDB Anywhere](../../../yugabyte-platform/install-yugabyte-platform/).
+
+### Bug fixes
+
+#### Other
+
+* Repairs build failure in CentOS 7 pex/yugabundle builder Docker image. PLAT-14543
+
 ## v2.18.8.0 - June 21, 2024 {#v2.18.8.0}
 
 **Build:** `2.18.8.0-b42`

diff --git a/docs/content/preview/releases/ybdb-releases/v2.18.md b/docs/content/preview/releases/ybdb-releases/v2.18.md
@@ -33,6 +33,47 @@ What follows are the release notes for the YugabyteDB v2.18 release series. Cont
 
 For an RSS feed of all release series to track the latest product updates, point your feed reader to the [RSS feed for releases](../index.xml).
 
+## v2.18.8.1 - July 18, 2024 {#v2.18.8.1}
+
+**Build:** `2.18.8.1-b3`
+
+**Third-party licenses:** [YugabyteDB](https://downloads.yugabyte.com/releases/2.18.8.1/yugabytedb-2.18.8.1-b3-third-party-licenses.html), [YugabyteDB Anywhere](https://downloads.yugabyte.com/releases/2.18.8.1/yugabytedb-anywhere-2.18.8.1-b3-third-party-licenses.html)
+
+### Downloads
+
+<ul class="nav yb-pills">
+ <li>
+   <a href="https://downloads.yugabyte.com/releases/2.18.8.1/yugabyte-2.18.8.1-b3-darwin-x86_64.tar.gz">
+     <i class="fa-brands fa-apple"></i>
+     <span>macOS</span>
+   </a>
+ </li>
+ <li>
+   <a href="https://downloads.yugabyte.com/releases/2.18.8.1/yugabyte-2.18.8.1-b3-linux-x86_64.tar.gz">
+     <i class="fa-brands fa-linux"></i>
+     <span>Linux x86</span>
+   </a>
+ </li>
+ <li>
+   <a href="https://downloads.yugabyte.com/releases/2.18.8.1/yugabyte-2.18.8.1-b3-el8-aarch64.tar.gz">
+     <i class="fa-brands fa-linux"></i>
+     <span>Linux ARM</span>
+   </a>
+ </li>
+</ul>
+
+### Docker
+
+```sh
+docker pull yugabytedb/yugabyte:2.18.8.1-b3
+```
+
+### Improvements
+
+#### DocDB
+
+* Allows asynchronous DNS cache updating and resolution retry upon failure to reduce RPC call delays and prevent unexpected leadership changes. {{<issue 22930>}},{{<issue 22311>}}
+
 ## v2.18.8.0 - June 21, 2024 {#v2.18.8.0}
 
 **Build:** `2.18.8.0-b42`

diff --git a/docs/content/preview/yugabyte-platform/configure-yugabyte-platform/aws.md b/docs/content/preview/yugabyte-platform/configure-yugabyte-platform/aws.md
@@ -101,7 +101,7 @@ Enter a Provider name. The Provider name is an internal tag used for organizing
 
 **Credential Type**. YBA requires the ability to create VMs in AWS. To do this, you can do one of the following:
 
-- **Specify Access ID and Secret Key** - Create an AWS Service Account with the required permissions (refer to [Cloud permissions](../../prepare/cloud-permissions/cloud-permissions-nodes/)), and provide your AWS Access Key ID and Secret Access Key.
+- **Specify Access ID and Secret Key** - Create an AWS Service Account with the required permissions (refer to [Cloud permissions](../../prepare/cloud-permissions/cloud-permissions-nodes-aws/)), and provide your AWS Access Key ID and Secret Access Key.
 - **Use IAM Role from this YBA host's instance** - Provision the YBA VM instance with an IAM role that has sufficient permissions by attaching an [IAM role](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html) to the YBA VM in the **EC2** tab. This option is only available if YBA is installed on AWS.
 
 **Use AWS Route 53 DNS Server**. Choose whether to use the cloud DNS Server / load balancer for universes deployed using this provider. Generally, SQL clients should prefer to use [smart client drivers](../../../drivers-orms/smart-drivers/) to connect to cluster nodes, rather than load balancers. However, in some cases (for example, if no smart driver is available in the language), you may use a DNS Server or load-balancer. The DNS Server acts as a load-balancer that routes clients to various nodes in the database universe. YBA integrates with [Amazon Route53](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/Welcome.html) to provide managed Canonical Name (CNAME) entries for your YugabyteDB universes, and automatically updates the DNS entry as nodes get created, removed, or undergo maintenance.

diff --git a/docs/content/preview/yugabyte-platform/configure-yugabyte-platform/azure.md b/docs/content/preview/yugabyte-platform/configure-yugabyte-platform/azure.md
@@ -51,13 +51,13 @@ When deploying a universe, YBA uses the provider configuration settings to do th
 
 ## Prerequisites
 
-You need to add the following Azure cloud provider credentials via YBA:
+You need to add the following Azure cloud provider credentials:
 
+- Application client ID and (if using credentials) client secret
+- Resource group name
 - Subscription ID
 - Tenant ID
 - SSH port and user
-- Application client ID and secret
-- Resource group
 
 YBA uses the credentials to automatically provision and deprovision YugabyteDB instances.
 
@@ -107,11 +107,29 @@ Enter a Provider name. The Provider name is an internal tag used for organizing
 
 ### Cloud Info
 
-- **Client ID** represents the [ID of an application](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal#option-2-create-a-new-application-secret) registered in your Azure Active Directory.
-- **Client Secret** represents the secret of an application registered in your Azure Active Directory. You need to enter the `Value` of the secret (not the `Secret ID`).
-- **Resource Group** represents the group in which YugabyteDB nodes compute and network resources are created. Your Azure Active Directory application (client ID and client secret) needs to have `Network Contributor` and `Virtual Machine Contributor` roles assigned for this resource group.
-- **Subscription ID** is required for cost management. The virtual machine resources managed by YBA are tagged with this subscription.
-- **Tenant ID** represents the Azure Active Directory tenant ID which belongs to an active subscription. To find your tenant ID, follow instructions provided in [How to find your Azure Active Directory tenant ID](https://learn.microsoft.com/en-us/azure/active-directory/fundamentals/how-to-find-tenant).
+Enter the following details of your Azure cloud account, as described in [Azure cloud permissions](../../prepare/cloud-permissions/cloud-permissions-nodes-azure/).
+
+#### Client ID
+
+Provide the ID of the application you registered with the Microsoft Identity Platform.
+
+#### Credential type
+
+If you [added credentials](https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app?tabs=client-secret#add-credentials) in the form of a client secret to your registered application:
+
+1. Select **Specify Client Secret**.
+1. Enter the Client Secret of the application associated with the Client ID you provided. You need to enter the `Value` of the secret (not the `Secret ID`).
+
+If you are using the [managed identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/qs-configure-portal-windows-vm) of the Azure VM hosting YugabyteDB Anywhere to authenticate:
+
+- Select **Use Managed Identity from this YBA host's instance**.
+
+#### Additional fields
+
+- **Resource Group** is the name of the resource group you created for your application, and in which YugabyteDB node compute and network resources will be created.
+- **Subscription ID** is required for cost management. The virtual machine resources managed by YBA are tagged with this subscription. To get the subscription ID, open Subscriptions in Azure portal and find your subscription. Then, copy the Subscription ID.
+- Optionally, if you created a different resource group for your network interfaces, provide the **Network Resource Group** name and the associated **Network Subscription ID**. If you do not provide a Network Resource Group or Subscription ID, network resources will be created in the default resource group.
+- **Tenant ID** represents the tenant ID which belongs to an active subscription. To find your tenant ID, follow instructions provided in [How to find your Microsoft Entra tenant ID](https://learn.microsoft.com/en-us/entra/fundamentals/how-to-find-tenant).
 - **Private DNS zone** lets you use a custom domain name for the nodes in your universe. For details and instructions, see [Define a private DNS zone](#define-a-private-dns-zone).
 
 ### Regions

diff --git a/docs/content/preview/yugabyte-platform/configure-yugabyte-platform/gcp.md b/docs/content/preview/yugabyte-platform/configure-yugabyte-platform/gcp.md
@@ -110,7 +110,7 @@ Enter a Provider name. The Provider name is an internal tag used for organizing
 
 ### Cloud Info
 
-If your YBA instance is not running inside GCP, you need to supply YBA with credentials to the desired GCP project by uploading a configuration file. To do this, set **Credential Type** to **Upload Service Account config** and proceed to upload the JSON file that you obtained when you created your service account, as described in [Cloud permissions](../../prepare/cloud-permissions/cloud-permissions-nodes/).
+If your YBA instance is not running inside GCP, you need to supply YBA with credentials to the desired GCP project by uploading a configuration file. To do this, set **Credential Type** to **Upload Service Account config** and proceed to upload the JSON file that you obtained when you created your service account, as described in [Cloud permissions](../../prepare/cloud-permissions/cloud-permissions-nodes-gcp/).
 
 If your YBA instance is running inside GCP, the preferred method for authentication to the GCP APIs is to add a service account role to the GCP instance running YBA and then configure YBA to use the instance's service account. To do this, set **Credential Type** to **Use service account from this YBA host's instance**.
 

diff --git a/...view/yugabyte-platform/prepare/cloud-permissions/cloud-permissions-nodes-aws.md b/...view/yugabyte-platform/prepare/cloud-permissions/cloud-permissions-nodes-aws.md
@@ -123,8 +123,8 @@ If using a service account, record the following two pieces of information about
 
 | Save for later | To configure |
 | :--- | :--- |
-| Access key ID | [AWS cloud provider](../../../configure-yugabyte-platform/aws/) |
-| Secret Access Key | [AWS cloud provider](../../../configure-yugabyte-platform/aws/) |
+| Access key ID | [AWS provider configuration](../../../configure-yugabyte-platform/aws/) |
+| Secret Access Key | |
 
 ### IAM role
 
@@ -178,4 +178,4 @@ If you will be using your own custom SSH keys, then ensure that you have them wh
 
 | Save for later | To configure |
 | :--- | :--- |
-| Custom SSH keys | [AWS provider](../../../configure-yugabyte-platform/kubernetes/) |
+| Custom SSH keys | [AWS provider configuration](../../../configure-yugabyte-platform/kubernetes/) |