Merge commit '12d6abeb9cbb3c893bd1a0a964da5ac120155eb9' into pg15-mas…

…ter-merge Merge YB master commit 12d6abe titled [#21103] DocDB: Hold Session shared pointer in the CDC State Table callback and committed 2024-03-26T20:41:12-07:00 into YB pg15. This is right before several compilation breakages on master. YB pg15 initial merge refers to 55782d5. - index.c: - reindex_relation: major conflict between YB pg15 merge 5627af5 and YB master fad94f7. A lot of context is present in the commit message of YB pg15 merge 5627af5. This time, the PG lines that were once put under (!(IsYBRelation(iRel) && iRel->rd_index->indisprimary)) were moved back out, so place them like in the order they were before (particularly, the indexNamespaceId declaration/definition goes higher up to the top of the scope). - cluster.c: - make_new_heap: YB pg15 merge 5627af5 makes trivial adjustments to code that YB master fad94f7 moves into helper YbRelationSetNewRelfileNode and calls just the helper here. - tablecmds.c: - ExecuteTruncateGuts: YB pg15 merge 5627af5 adjusts reindex_relation call. YB master fad94f7 switches is_yb_table_rewrite from false to true. Apply the same transformation in the adjusted call. - nodeYbBitmapTablescan.c: - CreateYbBitmapTableScanDesc: this function created by YB master f689455 returns what is assigned to ss_currentScanDesc, but such assignment was already done by YB pg15 merge ad2fedc in at least YbBitmapTableNext. Move that better code into YbBitmapTableNext. This also requires a signature change, and since the more future YB master 773869c is more suitable, get inspiration from that. - YbBitmapTableNext: YB pg15 merge ad2fedc adds extra lines for setting up ss_currentScanDesc, and YB master a28acc7 adds extra line node->skipped_tuples = 0 in the same location. Add the one line from master. As for the setting up of ss_currentScanDesc, see above on CreateYbBitmapTableScanDesc. - ExecReScanYbBitmapTableScan: YB master f689455 adds two things to this function: - free old node->ss.ss_currentScanDesc if exists - set new node->ss.ss_currentScanDesc The first is already handled (in a better pg15 way) by YB pg15 merge f689455. The second is also not needed because ss_currentScanDesc is already set in YbBitmapTableNext (and you can also see this part is removed by future YB master 773869c). - ExecEndYbBitmapTableScan: YB master f689455 adds "if (scanDesc)" which is likewise already done like "if (tsdesc != NULL)" by YB pg15 merge f689455. Discard the master change. - costsize.c: - includes: YB master 8fe3336 adds new include optimizer/ybcplan.h. Already, there is an effort to move such includes into the already-existing YB includes section below, such as in YB pg15 merge 5fe2012, so do the same in this case. - createplan.c: - includes: YB master 8fe3336 adds new include optimizer/ybcplan.h even though it already exists in the YB includes below. Drop this change. - function declarations: YB master 8fe3336 moves extract_pushdown_clauses (despite the lack of YB prefix, this is a YB-owned function) out to ybcplan.h. This has adjacent line conflicts, and the only changes by YB pg15 to this are indentation (which appears to be already fixed in the new location). Trivial resolution. - extract_pushdown_clauses: YB master 8fe3336 moves this function out to ybcplan.c. There have been no modifications to this function by YB pg15. Conflict is from adjacent lines. Trivial resolution. - syscache.c: - YbLoadPinnedObjectsCache: YB master 8487e59 deletes this function and moves some contents to YbInitPinnedCacheIfNeeded while YB pg15 bea1ffb deletes the whole thing. Delete the whole thing. - pg_yb_utils.c: - YBTxnDdlProcessUtility: YB master 8487e59 changes the signature of YbInitPinnedCacheIfNeeded, and YB pg15 merge 5627af5 adds #endif in the same area. Adjacent line conflict. - YbRelationSetNewRelfileNode: YB master fad94f7 moves code from make_new_heap to here, so apply the YB pg15 adjustments (which is only heap_open/heap_close to table_open/table_close). - logical.h: - LogicalDecodingContext: YB master 5c023f3 adds yb_start_decoding_at field in same area that upstream PG f95d53eded55ecbf037f6416ced6af29a2c3caca adds end_xact. Trivial merge. - syscache.h: - function declarations: (same as syscache.cc) - explain.c: - YbExplainDistinctPrefixLen: YB master e75e20d adds set_deparse_context_planstate call. Upstream PG 6ef77cf46e81f45716ec981cb08781d426181378 changes that to set_deparse_context_plan with slightly different middle argument as well. Simple resolution. - pg_yb_utils.h: - YbDdlRollbackEnabled: fix error: ‘inline’ is not at beginning of declaration [-Werror=old-style-declaration] by swapping "inline" and "bool". This was introduced by YB master 6091cc8. Not sure why it isn't an issue on master. - pg15_tests/passing_tests.tsv: - PggateTestSelect.GetTableKeyRanges: YB master cb302e0 renames PggateTestSelect.GetTableKeyRanges to PggateTestSelect.GetRangeShardedTableKeyRanges and disables the test, so remove it from this list. - PgSingleServerRestartTest.GetSafeTimeBeforeConsensusStarted: this test hits TRAP: FailedAssertion("!IsTransactionOrTransactionBlock()", File: "../../../../../../../src/postgres/src/backend/utils/activity/pgstat.c", Line: 580, PID: 1541456) but that was also the case before this merge. The issue now is that it crash loops on starting postmaster (this test kills and restarts tserver): FATAL: pre-existing shared memory block (key 33732133, ID 24018961) is still in use Bisecting commits would point to which commit introduced this issue and may help understand it, but in the interest of time, punt this to another day and just remove it from the passing list. - TestPgReplicationSlot.java: - testStartLsnValues: YB master 5c023f3 adds this test, but it fails in pg15: com.yugabyte.util.PSQLException: Database connection failed when reading from copy Ignore the test for now, just like some other tests in the same file.
yugabyte · May 9, 2024 · b728fbd · b728fbd
2 parents bea1ffb + 12d6abe
commit b728fbd
Show file tree

Hide file tree

Showing 834 changed files with 27,549 additions and 9,578 deletions.
diff --git a/.arclint b/.arclint
@@ -33,6 +33,7 @@
      "(^managed/RUNTIME-FLAGS[.]md$)",
      "(^managed/yba-cli/docs/.*)",
      "(^managed/yba-cli/.*/LICENSE$)",
+     "(^managed/yba-cli/NOTICE$)",
      "(^managed/yba-cli/internal/formatter/tabwriter/*)",
      "(^managed/yba-cli/internal/formatter/templates/*)",
      "(^[.]clang-tidy)",

diff --git a/.github/workflows/create-repository-dispatch.yml b/.github/workflows/create-repository-dispatch.yml
@@ -7,25 +7,6 @@ jobs:
   createRepositoryDispatch:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v4
-      - name: "Determine release train"
-        id: yb_train
-        run: |
-          my_ver=$(cat version.txt)
-          git fetch --depth=1 origin master
-          main_ver=$(git show origin/master:managed/src/main/resources/version.txt)
-          if [[ "$main_ver" == "$my_ver" ]]; then
-            train="master"
-          else
-            if [[ $my_ver =~ ^([0-9]+\.[0-9]+)\.* ]]; then
-              train="${BASH_REMATCH[1]}"
-            else
-              echo "Could not find release version from version.txt"
-              exit 1
-            fi
-          fi
-          echo "Release train determined: $train"
-          echo "yb_train=${train}" >> "$GITHUB_ENV"
 
       - name: "Trigger Repository Dispatch - yugabyte/terraform-gcp-yugabyte"
         run: |
@@ -34,8 +15,7 @@ jobs:
           -H "Content-Type: application/json" https://api.github.com/repos/yugabyte/terraform-gcp-yugabyte/dispatches \
           --data '{"event_type": "build-on-release", "client_payload":
                    {"prerelease": "${{ github.event.release.prerelease }}",
-                    "release": "${{github.event.release.tag_name}}",
-                    "yb_release_train": "'${yb_train}'" } }'
+                    "release": "${{github.event.release.tag_name}}" } }'
 
       - name: "Trigger Repository Dispatch - yugabyte/terraform-aws-yugabyte"
         run: |
@@ -44,8 +24,7 @@ jobs:
           -H "Content-Type: application/json" https://api.github.com/repos/yugabyte/terraform-aws-yugabyte/dispatches \
           --data '{"event_type": "build-on-release", "client_payload":
                    {"prerelease": "${{ github.event.release.prerelease }}",
-                    "release": "${{github.event.release.tag_name}}",
-                    "yb_release_train": "'${yb_train}'" } }'
+                    "release": "${{github.event.release.tag_name}}" } }'
 
       - name: "Trigger Repository Dispatch - yugabyte/terraform-azure-yugabyte"
         run: |
@@ -54,8 +33,7 @@ jobs:
           -H "Content-Type: application/json" https://api.github.com/repos/yugabyte/terraform-azure-yugabyte/dispatches \
           --data '{"event_type": "build-on-release", "client_payload":
                    {"prerelease": "${{ github.event.release.prerelease }}",
-                    "release": "${{github.event.release.tag_name}}",
-                    "yb_release_train": "'${yb_train}'" } }'
+                    "release": "${{github.event.release.tag_name}}" } }'
 
       - name: "Trigger Repository Dispatch - yugabyte/azure-resource-manager"
         run: |
@@ -64,8 +42,7 @@ jobs:
           -H "Content-Type: application/json" https://api.github.com/repos/yugabyte/azure-resource-manager/dispatches \
           --data '{"event_type": "build-on-release", "client_payload":
                    {"prerelease": "${{ github.event.release.prerelease }}",
-                    "release": "${{github.event.release.tag_name}}",
-                    "yb_release_train": "'${yb_train}'" } }'
+                    "release": "${{github.event.release.tag_name}}" } }'
 
       - name: "Trigger Repository Dispatch - yugabyte/gcp-deployment-manager"
         run: |
@@ -74,8 +51,7 @@ jobs:
           -H "Content-Type: application/json" https://api.github.com/repos/yugabyte/gcp-deployment-manager/dispatches \
           --data '{"event_type": "build-on-release", "client_payload":
                    {"prerelease": "${{ github.event.release.prerelease }}",
-                    "release": "${{github.event.release.tag_name}}",
-                    "yb_release_train": "'${yb_train}'" } }'
+                    "release": "${{github.event.release.tag_name}}" } }'
 
       - name: "Trigger Repository Dispatch - yugabyte/aws-cloudformation"
         run: |
@@ -84,8 +60,7 @@ jobs:
           -H "Content-Type: application/json" https://api.github.com/repos/yugabyte/aws-cloudformation/dispatches \
           --data '{"event_type": "build-on-release", "client_payload":
                    {"prerelease": "${{ github.event.release.prerelease }}",
-                    "release": "${{github.event.release.tag_name}}",
-                    "yb_release_train": "'${yb_train}'" } }'
+                    "release": "${{github.event.release.tag_name}}" } }'
 
       - name: "Trigger Repository Dispatch - yugabyte/homebrew-tap"
         run: |
@@ -94,22 +69,13 @@ jobs:
           -H "Content-Type: application/json" https://api.github.com/repos/yugabyte/homebrew-tap/dispatches \
           --data '{"event_type": "update-on-release", "client_payload":
                    {"prerelease": "${{ github.event.release.prerelease }}",
-                    "release": "${{github.event.release.tag_name}}",
-                    "yb_release_train": "'${yb_train}'" } }'
-
-      - name: "Trigger Repository Dispatch - yugabyte/charts"
-        run: |
-          curl -XPOST -u "${{ secrets.PAT_USERNAME}}:${{secrets.PAT_TOKEN}}" \
-          -H "Accept: application/vnd.github.everest-preview+json" \
-          -H "Content-Type: application/json" https://api.github.com/repos/yugabyte/charts/dispatches \
-          --data '{"event_type": "update-on-release", "client_payload":
-                   {"prerelease": "${{ github.event.release.prerelease }}",
-                    "release": "${{github.event.release.tag_name}}",
-                    "yb_release_train": "'${yb_train}'" } }'
+                    "release": "${{github.event.release.tag_name}}" } }'
 
       - name: "Trigger Repository Dispatch - yugabyte/yugabyte-operator"
         run: |
           curl -XPOST -u "${{ secrets.PAT_USERNAME}}:${{secrets.PAT_TOKEN}}" \
           -H "Accept: application/vnd.github.everest-preview+json" \
           -H "Content-Type: application/json" https://api.github.com/repos/yugabyte/yugabyte-operator/dispatches \
-          --data '{"event_type": "update-on-release",  "client_payload": {"prerelease": "${{ github.event.release.prerelease }}", "release": "${{github.event.release.tag_name}}"  }}'
+          --data '{"event_type": "update-on-release",  "client_payload":
+                   {"prerelease": "${{ github.event.release.prerelease }}",
+                    "release": "${{github.event.release.tag_name}}"  }}'
diff --git a/bin/yugabyted b/bin/yugabyted
@@ -2364,6 +2364,8 @@ class ControlScript(object):
                                                             get("master_webserver_port")),
                                 "-tserver_ui_port={}".format(self.configs.saved_data.
                                                             get("tserver_webserver_port")),
+                                "-data_dir={}".format(self.configs.saved_data.
+                                                            get("data_dir")),
                                 "-warnings={}".format("|".join(warnings_for_ui))
                             ]
                         if self.configs.saved_data.get("secure"):

diff --git a/build-support/tsan-suppressions.txt b/build-support/tsan-suppressions.txt
@@ -89,6 +89,9 @@ race:yb::ExternalDaemon::LogTailerThread::LogTailerThread
 # Known well implemented lock-free containers
 race:boost::lockfree
 
+# Suppress TSAN data race in boost::regex_match (potential false positive)
+race:boost::regex_detail_500::saved_state::saved_state
+
 # -------------------------------------------------------------------------------------------------
 # Supressions from libcds:
 

diff --git a/docs/.npmrc b/docs/.npmrc
@@ -0,0 +1,2 @@
+@fortawesome:registry=https://npm.fontawesome.com/
+//npm.fontawesome.com/:_authToken=624632FA-A763-410F-9ABA-C486C9538C86
diff --git a/docs/assets/scss/_code_block.scss b/docs/assets/scss/_code_block.scss
@@ -199,7 +199,7 @@ main {
           position: absolute;
           top: 0;
           left: 0;
-          font: 15px/1 "Font awesome 6 free";
+          font: 15px/1 $font-awesome-font-name;
           font-weight: 700;
           content: "\f00c";
           color: #48ab52 !important;

diff --git a/docs/assets/scss/_dropdown_list.scss b/docs/assets/scss/_dropdown_list.scss
@@ -74,7 +74,7 @@
       vertical-align: 0.255em;
       content: "\f107";
       color: #97a5b0;
-      font-family: "Font Awesome 6 Free";
+      font-family: $font-awesome-font-name;
       border: 0;
     }
   }

diff --git a/docs/assets/scss/_nav.scss b/docs/assets/scss/_nav.scss
@@ -5,6 +5,10 @@
   display: none;
 }
 
+.mobile-search.desktop-hide {
+  display: none !important;
+}
+
 .hidden-scroll {
   overflow: hidden;
 
@@ -1086,7 +1090,7 @@ body {
 
 @media (max-width: 767px) {
   .mobile-search.desktop-hide {
-    display: block;
+    display: block !important;
   }
 
   aside.td-sidebar {

diff --git a/docs/assets/scss/_sidebar-tree.scss b/docs/assets/scss/_sidebar-tree.scss
@@ -73,7 +73,7 @@
         width: 10px;
         color: #6d7c88;
         height: 10px;
-        font-family: "Font Awesome 6 Free";
+        font-family: $font-awesome-font-name;
         transition: margin-right 0.2s linear;
         font-size: 11px;
         line-height: 10px;
@@ -206,7 +206,7 @@
 
             &::before {
               content: "\f0da";
-              font-family: "Font Awesome 6 Free";
+              font-family: $font-awesome-font-name;
               color: #97a5b0;
             }
           }

diff --git a/docs/assets/scss/_styles_project.scss b/docs/assets/scss/_styles_project.scss
@@ -1,3 +1,17 @@
+// Font Awesome Pro
+@import "../vendor/Font-Awesome-Pro/scss/fontawesome.scss";
+@import "../vendor/Font-Awesome-Pro/scss/brands.scss";
+@import "../vendor/Font-Awesome-Pro/scss/duotone.scss";
+@import "../vendor/Font-Awesome-Pro/scss/light.scss";
+@import "../vendor/Font-Awesome-Pro/scss/regular.scss";
+@import "../vendor/Font-Awesome-Pro/scss/sharp-light.scss";
+@import "../vendor/Font-Awesome-Pro/scss/sharp-regular.scss";
+@import "../vendor/Font-Awesome-Pro/scss/sharp-solid.scss";
+@import "../vendor/Font-Awesome-Pro/scss/sharp-thin.scss";
+@import "../vendor/Font-Awesome-Pro/scss/solid.scss";
+@import "../vendor/Font-Awesome-Pro/scss/thin.scss";
+@import "../vendor/Font-Awesome-Pro/scss/v4-shims.scss";
+
 @import "./_yb_headings.scss";
 @import "./_yb_bullets-and-number-list.scss";
 @import "./_dropdown_list.scss";
@@ -134,6 +148,15 @@ html {
   img {
     max-width: 66px;
     height: auto;
+  }
+
+	> i[class^="fa-"] {
+    font-size: 75px;
+		color: $YB_ORANGE;
+  }
+
+  > i.fa-brands {
+    font-size: 86px;
   }
 }
 

diff --git a/docs/assets/scss/_variables_project.scss b/docs/assets/scss/_variables_project.scss
@@ -1,5 +1,7 @@
 $primary: #fe6f42;
 $secondary: #312965;
+$fa-font-path: "../webfonts/Font-Awesome-Pro";
+$font-awesome-font-name: "Font Awesome 6 Pro";
 $google_font_name: "Inter";
 $google_font_family: "Inter:100,200,300,400,500,600,700,800,900";
 

diff --git a/docs/config/_default/config.toml b/docs/config/_default/config.toml
@@ -28,6 +28,15 @@ enableGitInfo = true
   [[module.imports]]
     path = "github.com/trunkcode/hugo-seo"
     disable = false
+  [[module.imports]]
+    path = "../node_modules/@fortawesome/fontawesome-pro"
+    disable = false
+    [[module.imports.mounts]]
+      source = "scss"
+      target = "assets/vendor/Font-Awesome-Pro/scss"
+    [[module.imports.mounts]]
+      source = "webfonts"
+      target = "static/webfonts/Font-Awesome-Pro"
 
 [markup.goldmark.parser.attribute]
     block = true
diff --git a/docs/config/_default/menus.toml b/docs/config/_default/menus.toml
@@ -340,15 +340,15 @@
   name = "YugabyteDB"
   weight = 4
   identifier = "yugabytedb"
-  url = "/stable/"
+  url = "/preview/"
   [home.params]
     showSection = true
 
 [[home]]
   name = "YugabyteDB Anywhere"
   weight = 5
   identifier = "yugabytedb-anywhere"
-  url = "/stable/yugabyte-platform/"
+  url = "/preview/yugabyte-platform/"
   [home.params]
     showSection = true
 

diff --git a/docs/config/_default/params.toml b/docs/config/_default/params.toml
@@ -49,12 +49,12 @@ version_menu_pagelinks = true
   sidebar_search_disable = true
   ul_show = 1
 
+[[versions]]
+  url = "/preview"
+  version = "v2.21 (Preview)"
 [[versions]]
   url = "/stable"
   version = "v2.20 (LTS)"
-[[versions]]
-  url = "/preview"
-  version = "v2.19 (Preview)"
 [[versions]]
   url = "/v2.18"
   version = "v2.18 (STS)"
@@ -72,8 +72,10 @@ version_menu_pagelinks = true
 # Custom params, regardless of theme
 [yb]
   terms_of_service = "https://www.yugabyte.com/terms-of-service/"
-  preview_version = "v2.19"
-  preview_version_slug = "stable"
+  preview_version = "v2.21"
+  preview_version_slug = "preview"
+  # To disable heading/title icons for particular page, define `hideHeadingIcon: true` on that page params.
+  heading_icons = true
 
 # Header logo link
 [yb.navbar_logo]

diff --git a/docs/content/preview/admin/yb-admin.md b/docs/content/preview/admin/yb-admin.md
@@ -1661,7 +1661,7 @@ Encryption status: ENABLED with key id <key_id_2>
 
 #### create_change_data_stream
 
-Creates a change data capture (CDC) DB stream for the specified table.
+Create a change data capture (CDC) DB stream for the specified namespace using the following command.
 
 **Syntax**
 
@@ -1682,7 +1682,32 @@ For example:
     create_change_data_stream ysql.yugabyte
 ```
 
+##### Creating a stream for Transactional CDC
 
+Create a change data capture (CDC) DB stream for the specified namespace that can be used for Transactional CDC using the following command.
+This feature is {{<badge/tp>}}. Use the [yb_enable_cdc_consistent_snapshot_streams](../../reference/configuration/yb-tserver/#yb-enable-cdc-consistent-snapshot-streams) flag to enable the feature.
+
+**Syntax**
+
+```sh
+yb-admin \
+    -master_addresses <master-addresses> \
+    create_change_data_stream ysql.<namespace_name> EXPLICIT CHANGE USE_SNAPSHOT
+```
+
+* *master-addresses*: Comma-separated list of YB-Master hosts and ports. Default value is `localhost:7100`.
+* *namespace_name*: The namespace on which the DB stream ID is to be created.
+* `EXPLICIT`: Checkpointing type on the server.
+* `CHANGE`: Record type indicating to the server that the stream should send only the new values of the changed columns.
+* `USE_SNAPSHOT`: Snapshot option indicating intention of client to consume the snapshot. If you don't want the client to consume the snapshot, use the `NOEXPORT_SNAPSHOT` option.
+
+For example:
+
+```sh
+./bin/yb-admin \
+    -master_addresses 127.0.0.1:7100 \
+    create_change_data_stream ysql.yugabyte EXPLICIT CHANGE USE_SNAPSHOT
+```
 
 ##### Enabling before image
 
@@ -1699,7 +1724,7 @@ yb-admin \
 * *master-addresses*: Comma-separated list of YB-Master hosts and ports. Default value is `localhost:7100`.
 * *namespace_name*: The namespace on which the DB stream ID is to be created.
 * `IMPLICIT`: Checkpointing type on the server.
-* `ALL`: Record type indicating the server that the stream should send the before image too.
+* `ALL`: Record type indicating to the server that the stream should send the before image too.
 
 A successful operation of the above command returns a message with a DB stream ID: