Pgtap subtree merge #4
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Resolves yugabyte#112.
Just like any other table. Required for Postgres 10.
Also isnt_partitioned().
Not even tested yet, but this should be a good start to it.
Add a patch to remove most of the partition-testing functions on 9.6 and ealier.
Otherwise it gets ignored on 9.4 and earlier. Resolves yugabyte#143.
bar record should be bare record
Architecture of CDC via logical replication
…lObjectMetadata. Summary: Addding a new field ObjectName to the model SqlObjectMetadata. Jira: DB-12159 Test Plan: Manual Tests Reviewers: djiang Reviewed By: djiang Subscribers: yugabyted-dev Differential Revision: https://phorge.dev.yugabyte.com/D36633
Summary: Created this new diff , because I'm not able to land the original [[ https://phorge.dev.yugabyte.com/D35615 | one ]] Implemented task details view. Instead of using react query , we use the data from the redux store. Old task components use redux store. We want to make sure we display the same progress across the ui. [[ https://www.figma.com/design/rv2p1rVjhxfUTTjcRgsirF/Long-Running-Progress-Report?node-id=201-4409&m=dev | FIGMA ]] For now , this diff integrates the task details only to the backup. For other Components, we will be doing this in the following tickets. This diff also integrate the before and after task details Test Plan: Tested manually {F265990} {F265991} {F265992} {F265993} {F265994} {F265995} {F265996} Reviewers: lsangappa Reviewed By: lsangappa Subscribers: yugaware Differential Revision: https://phorge.dev.yugabyte.com/D36645
Summary:
Add the following v2 Group Mapping CRUD APIs :
```
GET /customers/:cUUID/auth/group_mappings
Response body:
[
{
"group_identifier": "test-group1",
"uuid": "b0968020-5626-4ae5-b516-d4d77e5302c8",
"type": "OIDC",
"role_resource_definitions": [
{
"role_uuid": "f3671ea7-6579-4fc0-ba33-f8da45ff19da"
},
{
"role_uuid": "9ce94a1f-3ed7-4a2f-8a6a-bda354d2dd1b",
"resource_group": {
"resource_definition_set": [
{
"resource_type": "UNIVERSE",
"allow_all": true,
"resource_uuid_set": []
},
{
"resource_type": "OTHER",
"allow_all": false,
"resource_uuid_set": [
"f33e3c9b-75ab-4c30-80ad-cba85646ea39"
]
}
]
}
}
]
}
]
PUT v2/customers/:cUUID/auth/group_mappings
Request body same as above.
DELETE /customers/:custUUID/auth/group_mappings/:groupUUID
```
The diff also has migrations and other rbac changes required to add role bindings for Groups.
Please refer to the design doc for complete context - https://docs.google.com/document/d/1qGYu6swY8xWFzqhNWZ6uKHHtKSO9847lm2I0sDB5okU
Test Plan:
Manually tested all changes.
UTs pending.
Reviewers: #yba-api-review, sneelakantan, svarshney, skurapati
Reviewed By: #yba-api-review, sneelakantan
Subscribers: sneelakantan, sanketh, yugaware
Differential Revision: https://phorge.dev.yugabyte.com/D36269
…dit universe op
Summary:
In case of both FULL MOVE and UPDATE operations, we show modal dialog regarding confirmation of what changes has been done to configuration between
old userIntent and new userIntent
However, in both scenarios, we never showed the diff in instance tags when it changed, hence adding support for this
Test Plan:
Please refer to screenshots
{F266135}
{F266136}
Reviewers: jmak
Reviewed By: jmak
Subscribers: yugaware
Differential Revision: https://phorge.dev.yugabyte.com/D36590
* URI example * DOC-418 * Update docs/content/preview/yugabyte-platform/administer-yugabyte-platform/oidc-authentication.md * copy to other versions * image update
* logical replication skeleton * move docs in subsection * Fix sections * Fix subsection * Fix file name * Fix name * fix links * Fix links * review comments * page titles * minor edits --------- Co-authored-by: Dwight Hodge <[email protected]>
* Restructure YugabyteDB gRPC Connector docs
* EIT reorg * Encryption in transit section update * links * minor edits * links * misc edits * edits * misc edits * minor edits * typo * DOC-358 * review comments * misc edits * minor edits * Apply suggestions from code review Co-authored-by: Sanketh I <[email protected]> * review comments * format * review comments * format * minor edits * review comment * copy to stable * copy to stable * fix pages in stable --------- Co-authored-by: Sanketh I <[email protected]>
Summary:
Added support for Premium SSD V2 disks. This requires updating API to at least 2022.03.
Smart resize is forbidden for that type of disks (so only full move could be used to modify device)
As long this type is only available in select regions, we show a warning to user that creation could fail.
Test Plan:
Create azure universe with premiumV2 disks and customized iops/throughput (verify iops and throughput are passed in request).
Edit universe and expand disk - verify that this is performed by full move
{F263004}
Reviewers: muthu, sanketh, rmadhavan
Reviewed By: muthu
Subscribers: yugaware
Differential Revision: https://phorge.dev.yugabyte.com/D36246
Summary: Fixed one special case in placing replicas Test Plan: sbt test 1) Create universe with rf-3 and 6 nodes in single az 2) Increase RF to 7 3) Verify ok (previously was failing) Reviewers: svarshney Reviewed By: svarshney Subscribers: yugaware Differential Revision: https://phorge.dev.yugabyte.com/D36626
…to ENHANCED_POSTGRES_COMPATIBILITY group
Summary:
Do not allow gflags value change if ENHANCED POSTGRES COMPATIBILITY is turned on.
- Handled in Create Universe with and without RR flows.
- Handled in Edit Flags flow
Test Plan:
Tested manually
{F266264}
Reviewers: kkannan
Reviewed By: kkannan
Subscribers: ui, yugaware
Differential Revision: https://phorge.dev.yugabyte.com/D36678
…e read only deferrable mode. Summary: PostgreSQL uses SSI to implement serializable isolation level. Read only serializable isolation transactions may abort other serializable transactions by creating a cycle in the transaction dependency graph. This can be avoided by waiting for all concurrent transactions to finish by using the READ ONLY DEFERRABLE mode. On the other hand, YugabyteDB does not use SSI for serializable transactions and uses 2-phase locking instead. And YB assigns a different meaning to the DEFERRABLE mode for READ ONLY serializable transactions as compared to Pg. In YB, DEFERRABLE READ ONLY transactions wait out the maximum possible clock skew to avoid read restart errors (see https://docs.yugabyte.com/preview/architecture/transactions/read-restart-error/). So, in YB's use of DEFERRABLE, there is no need to wait for concurrent transactions to complete, apart from waiting out the max_clock_skew_usec. Jira: DB-12053 Test Plan: Jenkins ``` ./yb_build.sh --java-test TestPgRegressWaitQueues ``` There are tests already in modify-transaction-characteristics that use a read only deferrable transaction for a select statement. Modified the test output to remove waiting. Reviewers: pjain, bkolagani, smishra Reviewed By: pjain Subscribers: yql Differential Revision: https://phorge.dev.yugabyte.com/D36671
…s part of processing CHANGE_METADATA_OP in xCluster ClusterConfig. Summary: Problem: When a new table is added to a colocated database, the table needs to be created on both xCluster source and target with the same colocation_id. As part of processing the ChangeMetadataOp for the AddTable operation received from the source, the target creates a mapping of source->target schema versions of the newly created table. However, this is not getting persisted in ClusterConfig if the source/target schema versions are default values. As a result while replication may work immediately after setup, upon a restart of the T-server, the mapping of source-target schema versions may be lost and replication may stall until another schema change happens on the source. Fix: Fix is to detect the non-existent colocation_id correctly and persisting the ClusterConfig. Test Plan: ybt xcluster_ysql_colocated-test XClusterYsqlColocatedTest.DatabaseReplication The test was failing without the fix and is passing with the fix. Reviewers: hsunder, xCluster Reviewed By: hsunder Subscribers: ybase Differential Revision: https://phorge.dev.yugabyte.com/D36552
Summary: CentOS 7 has reached its EOL and has been removed from the third-party archive build matrix. Use Amazon Linux 2 instead of CentOS 7 as the "default" OS for third-party dependencies during a YugabyteDB build. With the removal of CentOS 7, Amazon Linux 2 currently has the oldest supported glibc version (2.26) of any Linux distribution that we use in our build matrix, so third-party pre-built archives built with Amazon Linux 2 can be used for a non-sanitizer YugabyteDB build with any supported Linux distribution. Also the new third-party version enables Clang 18 (not yet supported in the YugabyteDB build itself). Test Plan: Jenkins Reviewers: steve.varnau Reviewed By: steve.varnau Subscribers: ybase Differential Revision: https://phorge.dev.yugabyte.com/D36673
Summary: When restoring a snopshot of a colocated tablet to a new database/table group, all tables are re-created in the new database so that the cotable ids are different from those in the snapshot. At restore, the cotables in flushed frontiers should be updated to the ids of newly created tables, otherwise we will probably hit the following issue after restore: ``` 1. we have 3 sst files after restore 1.sst (smallest:old_id=0, largest:old_id=0) 2.sst (smallest:old_id=0, largest:old_id=0) 3.sst (smallest:old_id=0, largest:old_id=0) 2. compact 1.sst and 2.sst and generate 4.sst 3.sst (smallest:old_id=0, largest:old_id=0) 4.sst (smallest:new_id=1, largest:new_id=1) After compaction, schema packing with version 0 for new_id can be dropped because from frontier we can only find new_id=1 3. When compact 3.sst and 4.sst there are still rows with version 0 for old_id but schema version 0 has been GCed in step 2 ``` Jira: DB-11979 Test Plan: PackedRows/YBBackupTestWithPackedRowsAndColocation.*/1 CrossColocationTests/YBBackupCrossColocation.TestYSQLRestoreWithInvalidIndex/1 TableRewriteTests/YBBackupTestWithTableRewrite.TestYSQLBackupAndRestoreAfterRewrite/1 TableRewriteTests/YBBackupTestWithTableRewrite.TestYSQLBackupAndRestoreAfterFailedRewrite/1 Reviewers: sergei, zdrudi, mhaddad Reviewed By: zdrudi Subscribers: ybase, qhu Differential Revision: https://phorge.dev.yugabyte.com/D36041
… precise Summary: This error happens on verification with existing certs after the existence of node-agent on YBA is verified with the api token first. Test Plan: Verified by manually running it. ``` * Select your Zone. 1. Zone ID: f5d02b20-aed4-469d-823a-4c8e5f9fe2c8, Zone Code: us-west-2a 2. Zone ID: b7ff22b9-2278-475d-971d-607587a589ed, Zone Code: us-west-2b 3. Zone ID: 02267e7e-59f5-4bbb-8308-3d4b53c8cc7e, Zone Code: us-west-2c * The current value is Zone ID: 02267e7e-59f5-4bbb-8308-3d4b53c8cc7e, Zone Code: us-west-2c. Enter new option number or enter to skip: • Completed Node Agent Configuration • Checking for existing Node Agent with IP 10.9.101.164 ⨯ Node agent already exists on YBA but local credentials may be invalid. It may need to be unregistered first - open /home/ec2-user/node-agent/cert/node_agent.key: no such file or directory ``` Reviewers: cwang, sanketh, nbhatia Reviewed By: cwang Subscribers: yugaware Differential Revision: https://phorge.dev.yugabyte.com/D36664
…n the config Summary: [PLAT-14563] Fixing disk mounting logic to use mount points defined in the config Test Plan: checked bash script, will need to run on test node Reviewers: svarshney Reviewed By: svarshney Subscribers: yugaware Differential Revision: https://phorge.dev.yugabyte.com/D36351
* added info * minor edit * change from review * changes from review and copy to stable * review comment fix * refined the order * edits * edits * minor edits * review comments * copy to stable --------- Co-authored-by: Dwight Hodge <[email protected]>
…rget Replication functions out of CatalogManager Summary: Moving all functions related to Setup, Bootstrap, Alter and Delete Target Replication functions out of CatalogManager. Setup functions have been moved to `SetupUniverseReplicationHelper` class of `xcluster_universe_replication_setup_helper.cc` Bootstrap functions have been moved to `SetupUniverseReplicationWithBootstrapHelper` class of `xcluster_bootstrap_helper.cc` Alter functions have been moved to `AlterUniverseReplicationHelper` class of `xcluster_universe_replication_alter_helper.cc` Delete function has been moved to `xcluster_replication_group.cc` `SetupUniverseReplication`, `IsSetupUniverseReplicationDone`, `SetupNamespaceReplicationWithBootstrap`, `IsSetupNamespaceReplicationWithBootstrapDone`, `AlterUniverseReplication`, `DeleteUniverseReplication` RPC functions have been moved to `XClusterManager` and `XClusterTargetManager`. Moved `GetAutoFlagConfigVersionIfCompatible` from `xcluster_replication_group.cc` to `xcluster_universe_replication_setup_helper.cc` Created new helpers `CatalogManager::GetTableGroupAndColocationInfo`, `CatalogManager::InsertNewUniverseReplication`, `CatalogManager::GetUniverseReplicationBootstrap`, `CatalogManager::InsertNewUniverseReplicationBootstrap`, `CatalogManager::InsertReplicationBootstrapToClear`, `CatalogManager::ReplaceUniverseReplication` and `CatalogManager::RemoveUniverseReplicationFromMap`. These will later be moved to XClusterManager as part of yugabyte#23187 Removed some gFlags that are no longer used. `ns_replication_sync_retry_secs` `ns_replication_sync_backoff_secs` `ns_replication_sync_error_backoff_secs` `xcluster_fail_create_consumer_snapshot` These are all either test flags or never used in production so they are safe to remove without deprecating them. Fixes yugabyte#23183 Jira: DB-12126 Test Plan: Jenkins Reviewers: jhe, xCluster, slingam, mlillibridge Reviewed By: mlillibridge Subscribers: ybase Differential Revision: https://phorge.dev.yugabyte.com/D36361
…ugabyte#23233) @netlify preview/reference/configuration/yb-tserver/#ysql-output-buffer-size
Summary: Fixes the issue yaml parsing. We changed the formatting for yaml list. This diff fixes the usage for the same. Test Plan: Prepared alma9 node using ynp. Verified universe creation. Reviewers: vbansal, asharma Reviewed By: asharma Subscribers: yugaware Differential Revision: https://phorge.dev.yugabyte.com/D36711
Summary: Added regex match for gcp instance template. Regex taken from gcp documentation [[https://cloud.google.com/compute/docs/reference/rest/v1/instanceTemplates | here]]. Test Plan: Tested manually that validation fails with invalid characters. Reviewers: #yba-api-review!, svarshney Reviewed By: svarshney Subscribers: yugaware Differential Revision: https://phorge.dev.yugabyte.com/D36543
Summary: This was missed when task params were moved out from details field. Test Plan: Trivial - existing tests should succeed. Reviewers: vbansal, cwang Reviewed By: vbansal Subscribers: yugaware Differential Revision: https://phorge.dev.yugabyte.com/D36705
Summary: RateLimiter has a debug assert that you cannot `Request` more than `GetSingleBurstBytes`. In release mode we do not perform this check and any call gets stuck forever. This change allows large bytes to be requested on RateLimiter. It does so by breaking requests larger than `GetSingleBurstBytes` into multiple smaller requests. This change is a temporary fix to allow xCluster to operate without any issues. RocksDB RateLimiter has multiple enhancements over the years that would help avoid this and more starvation issues. Ex: facebook/rocksdb@cb2476a. We should consider pulling in those changes. Fixes yugabyte#23173 Jira: DB-12112 Test Plan: RateLimiterTest.LargeRequests Reviewers: slingam Reviewed By: slingam Subscribers: ybase Differential Revision: https://phorge.dev.yugabyte.com/D36703
…oids in CDC Summary: This diff adds support for data types with dynamically alloted oids in CDC (for ex: hstore, enum array, etc). Such types contain invalid pg_type_oid for the corresponding columns in docdb schema. In the current implemtation, in `ybc_pggate`, while decoding the cdc records we look at the `type_map_` to obtain YBCPgTypeEntity, which is then used for decoding. However the `type_map_` does not contain any entries for the data types with dynamically alloted oids. As a result, this causes segmentation fault. To prevent such crashes, CDC prevents addition of tables with such columns to the stream. This diff removes the filtering logic and adds the tables to the stream even if it has such a type column. A function pointer will now be passed to `YBCPgGetCDCConsistentChanges`, which takes attribute number and the table_oid and returns the appropriate type entity by querying the `pg_type` catalog table. While decoding if a column is encountered with invalid pg_type_oid then, the passed function is invoked and type entity is obtained for decoding. **Upgrade/Rollback safety:** This diff adds a field `optional int32 attr_num` to DatumMessagePB. These changes are protected by the autoflag `ysql_yb_enable_replication_slot_consumption` which already exists but has not yet been released. Jira: DB-12118 Test Plan: Jenkins: urgent All the existing cdc tests ./yb_build.sh --java-test 'org.yb.pgsql.TestPgReplicationSlot#replicationConnectionConsumptionAllDataTypesWithYbOutput' Reviewers: skumar, stiwary, asrinivasan, dmitry Reviewed By: stiwary, dmitry Subscribers: steve.varnau, skarri, yql, ybase, ycdcxcluster Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D36689
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.