[BACKPORT pg15-cherrypicks] all: Bulk port from master - 107

Summary: 5d3e83e [PLAT-15199] Change TP API URLs according to latest refactoring a50a730 [doc][yba] YBDB compatibility (#23984) 0c84dbe [#24029] Update the callhome diagnostics not to send gflags details. b53ed3a [PLAT-15379][Fix PLAT-12510] Option to use UTC when dealing with cron exp. in backup schedule f0eab8f [PLAT-15278]: Fix DB Scoped XCluster replication restart 344bc76 Revert "[PLAT-15379][Fix PLAT-12510] Option to use UTC when dealing with cron exp. in backup schedule" 3628ba7 [PLAT-14459] Swagger fix bb93ebe [#24021] YSQL: Add --TEST_check_catalog_version_overflow 9ab7806 [#23927] docdb: Add gflag for minimum thread stack size Excluded: 8c8adc0 [#18822] YSQL: Gate update optimizations behind preview flag 5e86515 [#23768] YSQL: Fix table rewrite DDL before slot creation 123d496 [PLAT-14682] Universe task should only unlock itself and make unlock aware of the lock config de9d4ad [doc][yba] CIS hardened OS support (#23789) e131b20 [#23998] DocDB: Update usearch and other header-only third-party dependencies 1665662 Automatic commit by thirdparty_tool: update usearch to commit 240fe9c298100f9e37a2d7377b1595be6ba1f412. 3adbdae Automatic commit by thirdparty_tool: update fp16 to commit 98b0a46bce017382a6351a19577ec43a715b6835. 9a819f7 Automatic commit by thirdparty_tool: update hnswlib to commit 2142dc6f4dd08e64ab727a7bbd93be7f732e80b0. 2dc58f4 Automatic commit by thirdparty_tool: update simsimd to tag v5.1.0. 9a03432 [doc][ybm] Azure private link host (#24086) 039c9a2 [#17378] YSQL: Testing for histogram_bounds in pg_stats 09f7a0f [#24085] DocDB: Refactor HNSW wrappers 555af7d [#24000] DocDB: Shutting down shared exchange could cause TServer to hang 5743a03 [PLAT-15317]Alert emails are not in the correct format. 8642555 [PLAT-15379][Fix PLAT-12510] Option to use UTC when dealing with cron exp. in backup schedule 253ab07 [PLAT-15400][PLAT-15401][PLAT-13051] - Connection pooling ui issues and other ui issues 57576ae [#16487] YSQL: Fix flakey TestPostgresPid test bc8ae45 Update ports for CIS hardened (#24098) 6fa33e6 [#18152, #18729] Docdb: Fix test TestPgIndexSelectiveUpdate cc6d2d1 [docs] added and updated cves (#24046) Excluded: ed153dc [#24055] YSQL: fix pg_hint_plan regression with executing prepared statement Test Plan: Jenkins: rebase: pg15-cherrypicks Reviewers: jason, jenkins-bot Differential Revision: https://phorge.dev.yugabyte.com/D38322
yugabyte · Sep 24, 2024 · 218324e · 218324e
1 parent d6078ca
commit 218324e
Show file tree

Hide file tree

Showing 124 changed files with 10,854 additions and 1,240 deletions.
diff --git a/CMakeLists.txt b/CMakeLists.txt
@@ -608,6 +608,7 @@ include_directories(src)
 include_directories("src/inline-thirdparty/usearch")
 include_directories("src/inline-thirdparty/fp16")
 include_directories("src/inline-thirdparty/hnswlib")
+include_directories("src/inline-thirdparty/simsimd")
 
 
 enable_testing()

diff --git a/bin/yugabyted b/bin/yugabyted
@@ -8283,8 +8283,8 @@ class Diagnostics(object):
         payload = {
             "data_dir_size": self.get_dir_size(self.configs.saved_data.get("data_dir")),
             "num_cpus": multiprocessing.cpu_count(),
-            "master_flags": self.configs.saved_data.get("master_flags"),
-            "tserver_flags": self.configs.saved_data.get("tserver_flags"),
+            # "master_flags": self.configs.saved_data.get("master_flags"),
+            # "tserver_flags": self.configs.saved_data.get("tserver_flags"),
             "is_docker" : str(os.path.exists("/.dockerenv"))
         }
         if Diagnostics.first_install is not None:

diff --git a/build-support/inline_thirdparty.yml b/build-support/inline_thirdparty.yml
@@ -16,7 +16,7 @@
 dependencies:
   - name: usearch
     git_url: https://github.com/unum-cloud/usearch
-    commit: 4fbb56e02aa928a011abdedb66adfef128123e5f
+    commit: 240fe9c298100f9e37a2d7377b1595be6ba1f412
     src_dir: include
     dest_dir: usearch
 

diff --git a/docs/config/_default/menus.toml b/docs/config/_default/menus.toml
@@ -848,15 +848,6 @@
     showSection = true
     hideLink = true
 
-[[preview_yugabyte-platform]]
-  name = "Servers for nodes"
-  parent = "prepare"
-  weight = 40
-  identifier = "server-nodes"
-  [preview_yugabyte-platform.params]
-    showSection = true
-    hideLink = true
-
 ########## Menus (in preview) for YB Managed section
 
 [[preview_yugabyte-cloud]]
@@ -1078,15 +1069,6 @@
     showSection = true
     hideLink = true
 
-[[stable_yugabyte-platform]]
-  name = "Servers for nodes"
-  parent = "prepare"
-  weight = 40
-  identifier = "server-nodes"
-  [stable_yugabyte-platform.params]
-    showSection = true
-    hideLink = true
-
 ########## Menus (in stable) for APIs
 
 # [[stable_api]]

diff --git a/docs/content/preview/deploy/checklist.md b/docs/content/preview/deploy/checklist.md
@@ -123,7 +123,7 @@ The following is a list of default ports along with the network access required
 
   - 7000 for viewing the YB-Master Admin UI.
 
-- To use the database from the app, the following ports need to be accessible from the app or CLI:
+- To access the database from applications or clients, the following ports need to be accessible from the applications or CLI:
 
   - 5433 for YSQL
   - 9042 for YCQL

diff --git a/docs/content/preview/reference/configuration/default-ports.md b/docs/content/preview/reference/configuration/default-ports.md
@@ -68,9 +68,30 @@ The following common ports are required for firewall rules:
 | HTTP for YugabyteDB Anywhere (alternate) | 8080 |
 | HTTPS for YugabyteDB Anywhere  | 443 |
 | HTTP for Replicated | 8800 |
-| SSH  **   | 54422 |
+| Custom SSH port for universe nodes | 54422 |
 
-** 54422 is a custom SSH port for universe nodes.
+### Firewall changes for CIS hardened images
+
+Running YugabyteDB on CIS hardened RHEL 8 or 9 requires the following changes to the firewall:
+
+```sh
+#!/bin/bash
+
+sudo dnf repolist
+sudo dnf config-manager --set-enabled extras
+sudo dnf install -y firewalld
+sudo systemctl start firewalld
+
+ports=(5433 9042 7100 9100 18018 9070 7000 9000 12000 13000 15433)
+
+for port in "${ports[@]}"; do
+   sudo firewall-cmd --zone=public --add-port=${port}/tcp --permanent
+done
+
+sudo firewall-cmd --reload
+```
+
+If you have customized any port settings, be sure to replace the port numbers as appropriate.
 
 ## Prometheus monitoring
 

diff --git a/docs/content/preview/reference/configuration/operating-systems.md b/docs/content/preview/reference/configuration/operating-systems.md
@@ -23,7 +23,9 @@ Unless otherwise noted, operating systems are supported by all supported version
 | AlmaLinux 9      | {{<icon/yes>}} | {{<icon/yes>}} |       |
 | Oracle Linux 8   | {{<icon/yes>}} |                | |
 | Red Hat Enterprise Linux 8 | {{<icon/yes>}} |      | Recommended for production |
-| Red Hat Enterprise Linux&nbsp;9.3 and later| {{<icon/yes>}} |  | Supported in v2.20.3 and later.  {{<badge/ea>}} |
+| Red Hat Enterprise Linux 8 CIS Hardened | {{<icon/yes>}} |      | |
+| Red Hat Enterprise Linux&nbsp;9.3 and later | {{<icon/yes>}} |  | Supported in v2.20.3 and later.  {{<badge/ea>}} |
+| Red Hat Enterprise Linux&nbsp;9 CIS Hardened | {{<icon/yes>}} |  | Supported in v2.20.3 and later.  {{<badge/ea>}} |
 | SUSE&nbsp;Linux&nbsp;Enterprise&nbsp;Server&nbsp;15&nbsp;SP5 | {{<icon/yes>}} |     | {{<badge/ea>}} |
 | Ubuntu 20        | {{<icon/yes>}} | {{<icon/yes>}} |       |
 | Ubuntu 22        | {{<icon/yes>}} | {{<icon/yes>}} | Supported in v2.18.5, v2.20.1 |
@@ -32,8 +34,50 @@ The following table describes operating systems and architectures that are no lo
 
 | Operating system | x86            | ARM            | Notes |
 | :--------------- | :------------- | :------------- | :---- |
-| Amazon Linux 2   | {{<icon/no>}} | {{<icon/no>}} | Supported in v2.18.0 and later<br>Deprecated in v2.20<br> Removed support in v2.21. |
-| CentOS 7          | {{<icon/no>}}  |                | Deprecated in v2.20<br> Removed support in v2.21. |
+| Amazon Linux 2   | {{<icon/no>}}  | {{<icon/no>}}  | Supported in v2.18.0 and later<br>Deprecated in v2.20<br> Removed support in v2.21. |
+| CentOS 7         | {{<icon/no>}}  |                | Deprecated in v2.20<br> Removed support in v2.21. |
 | Oracle Linux 7   | {{<icon/no>}}  |                | Deprecated in v2.20<br> Removed support in v2.21. |
 | Red Hat Enterprise Linux 7 | {{<icon/no>}} |       | Deprecated in v2.20<br> Removed support in v2.21. |
 | Ubuntu 18        | {{<icon/no>}}  | {{<icon/no>}}  | Deprecated in v2.20<br> Removed support in v2.21. |
+
+## Using CIS hardened operating systems
+
+YugabyteDB supports RHEL CIS hardened operating systems based on the following images:
+
+- [CIS Red Hat Enterprise Linux 8 Benchmark-Level 1](https://aws.amazon.com/marketplace/pp/prodview-kg7ijztdpvfaw?sr=0-7&?ref=_ptnr_cis_website)
+
+- [CIS Red Hat Enterprise Linux 9 Benchmark-Level 1](https://aws.amazon.com/marketplace/server/procurement?productId=fa2dc596-6685-4c0b-b258-3c415342c908)
+
+To use these images for YugabyteDB or YugabyteDB Anywhere, you need to make the following modifications.
+
+### YugabyteDB clusters
+
+To use a CIS hardened image for cluster nodes:
+
+1. Install the image on the nodes.
+1. [Change the firewall rules](../default-ports/#firewall-changes-for-cis-hardened-images).
+
+### YugabyteDB Anywhere
+
+To use a CIS hardened image for installing YugabyteDB Anywhere:
+
+1. Install the image on the node.
+1. [Change the firewall rules](../../../yugabyte-platform/prepare/networking/#firewall-changes-for-cis-hardened-images) for YugabyteDB Anywhere installation.
+
+### YugabyteDB Anywhere universe
+
+To use a CIS hardened image for on-premises universe nodes:
+
+1. Install the image on the nodes.
+1. [Change the firewall rules](../default-ports/#firewall-changes-for-cis-hardened-images).
+1. If you want SSH access to database nodes, during [provisioning](../../../yugabyte-platform/prepare/server-nodes-software/software-on-prem-manual/#pre-provision-nodes-manually), add the `yugabyte` user to `sshd_config`.
+1. Create a custom `tmp` directory during [provisioning](../../../yugabyte-platform/prepare/server-nodes-software/software-on-prem-manual/#custom-tmp-directory-for-cis-hardened-rhel-8-or-9).
+1. After you create the [on-premises provider](../../../yugabyte-platform/configure-yugabyte-platform/on-premises-provider/), set the [provider runtime configuration](../../../yugabyte-platform/administer-yugabyte-platform/manage-runtime-config/) flag `yb.filepaths.remoteTmpDirectory` to the custom tmp directory.
+1. When creating universes using the provider, set YB-Master and YB-TServer [configuration flag](../../../yugabyte-platform/manage-deployments/edit-config-flags/) `tmp_dir` to the custom tmp directory.
+
+To use a CIS hardened image for universes on cloud providers:
+
+1. Download the image.
+1. [Change the firewall rules](../default-ports/#firewall-changes-for-cis-hardened-images).
+1. Upload the modified image to your cloud provider.
+1. [Add the modified image](../../../yugabyte-platform/configure-yugabyte-platform/aws/) to the Linux version catalog.
diff --git a/docs/content/preview/releases/yba-releases/_index.md b/docs/content/preview/releases/yba-releases/_index.md
@@ -34,43 +34,43 @@ For information on release versioning, see [Versioning](../versioning/).
 
 YugabyteDB Anywhere is a control plane for deploying and managing YugabyteDB universes. You can use YugabyteDB Anywhere to deploy universes with an equivalent or earlier version of YugabyteDB.
 
+Qualification tests for each new version of YugabyteDB Anywhere are run on the latest version of YugabyteDB in each release series.
+
 ### Supported versions
 
 Every version of YugabyteDB Anywhere supports the same version and prior releases of YugabyteDB, down to and including the two preceding LTS release series and any intervening STS releases. This provides a span of support of approximately 2 years.
 
-YugabyteDB Anywhere v2.20.x supports the following YugabyteDB release series:
+For example, YugabyteDB Anywhere v2.20.x supports the following YugabyteDB release series:
 
 - [v2.20.x](../ybdb-releases/v2.20/) (LTS)
 - [v2.18.x](../ybdb-releases/v2.18/) (STS)
 - [v2.16.x](../ybdb-releases/end-of-life/v2.16/) (STS)
 - [v2.14.x](../ybdb-releases/v2.14/) (LTS)
 
-Qualification tests for each new version of YugabyteDB Anywhere are run on the latest version of YugabyteDB in each release series.
+For information on YugabyteDB release support timelines, refer to [YugabyteDB releases](../ybdb-releases).
 
 {{< warning title="YugabyteDB v2.14 and v2.18 End of Maintenance" >}}
 v2.14 and v2.18 will reach end of maintenance in mid-2024. If you are running universes on these release series, you should consider upgrading those universes to the next LTS release series (v2.20).
 {{< /warning >}}
 
 For information on managing YugabyteDB releases and upgrading universes using YugabyteDB Anywhere, refer to [Upgrade the YugabyteDB software](../../yugabyte-platform/manage-deployments/upgrade-software/).
 
-For information on YugabyteDB release support timelines, refer to [YugabyteDB releases](../ybdb-releases).
-
-### Upgrading YugabyteDB Anywhere
+## Upgrading YugabyteDB Anywhere
 
 Keep YugabyteDB Anywhere up-to-date with the latest stable version to get the latest fixes and improvements, as well as to be able to deploy the latest releases of YugabyteDB.
 
+{{< warning title="Replicated end of life" >}}
+YugabyteDB Anywhere will end support for Replicated installation at the end of 2024. You can migrate existing Replicated YugabyteDB Anywhere installations using YBA Installer. To perform the migration, you must first upgrade to YugabyteDB Anywhere v2.20.1 or later using Replicated.
+{{< /warning >}}
+
 You can't use YugabyteDB Anywhere to deploy versions of YugabyteDB that are newer than your YugabyteDB Anywhere instance. To upgrade a universe to a more recent version of YugabyteDB, you may first have to upgrade YugabyteDB Anywhere.
 
 - For YugabyteDB upgrades in YugabyteDB Anywhere, you can only upgrade from a _stable_ version to another _stable_ version, or from a _preview_ version to another _preview_ version. Optionally, you can [skip tests](#skip-tests) during upgrades.
 
 - For YugabyteDB Anywhere upgrades, you can only upgrade from a _stable_ version to another _stable_ version, or from a _preview_ version to another _preview_ version. Optionally, you can [skip tests](#skip-tests) during upgrades.
 
-{{< warning title="Replicated end of life" >}}
-YugabyteDB Anywhere will end support for Replicated installation at the end of 2024. You can migrate existing Replicated YugabyteDB Anywhere installations using YBA Installer. To perform the migration, you must first upgrade to YugabyteDB Anywhere v2.20.1 or later using Replicated.
-{{< /warning >}}
-
-For information on upgrading YugabyteDB Anywhere, refer to [Upgrade YugabyteDB Anywhere](../../yugabyte-platform/upgrade/).
+For instructions on upgrading YugabyteDB Anywhere, refer to [Upgrade YugabyteDB Anywhere](../../yugabyte-platform/upgrade/).
 
-#### Skip tests
+### Skip tests
 
-Optionally, you can set a runtime flag `yb.skip_version_checks`, to skip all YugabyteDB and YugabyteDB Anywhere version checks during upgrades. For more information, contact {{% support-platform %}}.
+Optionally, you can set a runtime flag `yb.skip_version_checks` to skip all YugabyteDB and YugabyteDB Anywhere version checks during upgrades. For more information, contact {{% support-platform %}}.
diff --git a/docs/content/preview/secure/encryption-at-rest.md b/docs/content/preview/secure/encryption-at-rest.md
@@ -21,6 +21,10 @@ Note that encryption can be applied at the following levels:
 - At the database layer, in which case the encryption process and its associated capabilities, such as key rotation, are cluster-wide.
 - At the file system level, in which case it is the responsibility of the operations teams to manage the process manually on every node. It is important to note that the degree to which file systems or external encryption mechanisms support online operations can vary (for example, when the database processes are still running).
 
+    If you are using third party disk encryption software, such as Vormetric or CipherTrust, the disk encryption service must be up and running on the node before starting any YugabyteDB services. If YugabyteDB processes start _before_ the encryption service, restarting an already encrypted node can result in data corruption.
+
+    To avoid issues, stop YugabyteDB services on the node _before_ enabling or disabling the disk encryption service.
+
 ## Enable encryption
 
 You enable encryption as follows:

diff --git a/docs/content/preview/secure/vulnerability-disclosure-policy.md b/docs/content/preview/secure/vulnerability-disclosure-policy.md
@@ -105,7 +105,7 @@ Note that this policy covers only vulnerabilities in the query layer of PostgreS
 | PostgreSQL (YSQL) | {{<cve "CVE-2021-23222">}} | [v2.8.1.0](/preview/releases/ybdb-releases/end-of-life/v2.8/#v2.8.1.0), [v2.6.7.0](/preview/releases/ybdb-releases/end-of-life/v2.6/#v2.6.7.0), [v2.11.1.0](/preview/releases/ybdb-releases/end-of-life/v2.11/#v2.11.1.0) | Resolved |
 | PostgreSQL (YSQL) | {{<cve "CVE-2021-32027">}} | [v2.7.0.0](/preview/releases/ybdb-releases/end-of-life/v2.7/#v2.7.0.0) | Resolved |
 | PostgreSQL (YSQL) | {{<cve "CVE-2021-32028">}} | [v2.7.2.0](/preview/releases/ybdb-releases/end-of-life/v2.7/#v2.7.2.0) | Resolved |
-| PostgreSQL (YSQL) | {{<cve "CVE-2021-32029">}} | {{<release "2.21.1.0">}} | Resolved |
+| PostgreSQL (YSQL) | {{<cve "CVE-2021-32029">}} | {{<release "2.14.15.0">}}, {{<release "2.18.6.0">}}, {{<release "2.20.2.0">}}, {{<release "2.21.1.0">}} | Resolved |
 | PostgreSQL (YSQL) | {{<cve "CVE-2021-3393">}} | {{<release "2.17.1.0">}} | Resolved |
 | PostgreSQL (YSQL) | {{<cve "CVE-2021-3677">}} |  | Resolved |
 | PostgreSQL (YSQL) | {{<cve "CVE-2021-43766">}} |  [v2.12.0.0](/preview/releases/ybdb-releases/end-of-life/v2.12/#v2.12.0.0), {{<release "2.14.0.0">}} | Resolved |
@@ -116,3 +116,6 @@ Note that this policy covers only vulnerabilities in the query layer of PostgreS
 | PostgreSQL (YSQL) | {{<cve "CVE-2023-2455">}} | {{<release "2.14.10.2">}}, [v2.16.5.0](/preview/releases/ybdb-releases/end-of-life/v2.16/#v2.16.5.0), {{<release "2.18.0.0">}}, {{<release "2.20.0.0">}}| Resolved |
 | PostgreSQL (YSQL) | {{<cve "CVE-2023-32305">}} |  | Not applicable: [aiven-extras](https://github.com/aiven/aiven-extras) is not included in installation. |
 | PostgreSQL (YSQL) | {{<cve "CVE-2023-39417">}} | {{<release "2.20.1.0">}}, {{<release "2.14.15.0">}}, [v2.16.9.0](/preview/releases/ybdb-releases/end-of-life/v2.16/#v2.16.9.0), {{<release "2.18.5.0">}}| Resolved |
+| PostgreSQL (YSQL) | {{<cve "CVE-2023-5868">}} | {{<release "2.23.0.0">}} | Resolved |
+| PostgreSQL (YSQL) | {{<cve "CVE-2023-5869">}} | {{<release "2.23.0.0">}} | Resolved |
+| PostgreSQL (YSQL) | {{<cve "CVE-2023-5870">}} | {{<release "2.23.0.0">}} | Resolved |