diff --git a/docs/content/preview/explore/ysql-language-features/postgresql-compatibility.md b/docs/content/preview/explore/ysql-language-features/postgresql-compatibility.md index 4a5e2f1d609a..6c02d1cc9329 100644 --- a/docs/content/preview/explore/ysql-language-features/postgresql-compatibility.md +++ b/docs/content/preview/explore/ysql-language-features/postgresql-compatibility.md @@ -4,7 +4,6 @@ linkTitle: PostgreSQL compatibility description: Summary of YugabyteDB's PostgreSQL compatibility aliases: - /preview/ysql/postgresql-compatibility/ - - /preview/explore/ysql-language-features/postgresql-compatibility/ menu: preview: identifier: explore-ysql-postgresql-compatibility @@ -148,7 +147,7 @@ Note: When enabling the cost models, ensure that packed row for colocated tables #### YugabyteDB Anywhere -To enable EPCM in YugabyteDB Anywhere v2024.1, see the [Release notes](/preview/releases/yba-releases/v2024.1/#highlights). +To enable EPCM in YugabyteDB Anywhere v2024.1, see the [Release notes](/preview/releases/yba-releases/v2024.1/#v2024.1.0.0). To enable EPCM in YugabyteDB Anywhere v2024.2 or later: diff --git a/docs/content/preview/yugabyte-voyager/migrate/assess-migration.md b/docs/content/preview/yugabyte-voyager/migrate/assess-migration.md index 840d750db7fc..dd06aebe5f68 100644 --- a/docs/content/preview/yugabyte-voyager/migrate/assess-migration.md +++ b/docs/content/preview/yugabyte-voyager/migrate/assess-migration.md @@ -1,8 +1,8 @@ --- title: YB Voyager Migration Assessment headerTitle: Migration assessment -linkTitle: Assess migration -headcontent: Create a migration assessment report +linkTitle: Migration assessment +headcontent: Assess the migration complexity and get schema changes, data distribution and cluster sizing description: Steps to create a migration assessment report to ensure successful migration using YugabyteDB Voyager. menu: preview_yugabyte-voyager: @@ -12,29 +12,26 @@ menu: badges: tp type: docs --- - -The Voyager Migration Assessment feature is specifically designed to optimize the database migration process from various source databases, currently supporting PostgreSQL and Oracle to YugabyteDB. Voyager conducts a detailed analysis of the source database by capturing essential metadata and metrics. It generates a comprehensive assessment report that recommends effective migration strategies, and provides key insights on ideal cluster configurations for optimal performance with YugabyteDB. +The Voyager Migration Assessment feature streamlines database migration from PostgreSQL and Oracle to YugabyteDB. It analyzes the source database, captures essential metadata, and generates a report with recommended migration strategies and cluster configurations for optimal performance with YugabyteDB. ## Overview -When you run an assessment, Voyager collects metadata or metrics from the source database. This includes table columns metadata, sizes of tables and indexes, read and write IOPS for tables and indexes, and so on. With this data, Voyager generates an assessment report with the following key details: - -- **Database compatibility**. An assessment of the compatibility of the source database with YugabyteDB, identifying unsupported features and data types. +When you run an assessment, Voyager gathers key metadata and metrics from the source database, such as table column details, sizes of tables and indexes, and read/write IOPS. -- **Cluster size evaluation**. Estimated resource requirements for the target environment, to help with planning and scaling your infrastructure. The sizing logic depends on various factors such as the size and number of tables in the source database, as well as the throughput requirements (read/write IOPS). +![Migration Assessment Overview](/images/migrate/voyager-migration-assessment-overview.png) -- **Schema evaluation**. Reviews the database schema to suggest effective sharding strategies for tables and indexes. +Voyager then generates a report that includes: -- **Performance metrics**. Voyager analyzes performance metrics to understand workload characteristics and provide recommendations for optimization in YugabyteDB. +- **Recommended schema changes:** Analyzes compatibility with YugabyteDB, highlighting unsupported features and data types. +- **Recommended cluster sizing:** Estimates the resources needed for the target environment based on table sizes, number of tables, and throughput requirements. +- **Recommended data distribution:** Suggests effective sharding strategies for tables and indexes. +- **Performance metrics:** Analyzes workload characteristics to recommend optimizations in YugabyteDB. +- **Migration time estimate:** Provides an estimated time for data import into YugabyteDB based on experimental data. -- **Migration time estimate**. An estimate of the time needed to import data into YugabyteDB after export from the source database. These estimates are calculated based on various experiments during data import to YugabyteDB. - -{{< warning title="Caveat" >}} +{{< warning title="Note" >}} The recommendations are based on testing using a [RF3](../../../architecture/docdb-replication/replication/#replication-factor) YugabyteDB cluster on instance types with 4GiB memory per core and running v2024.1. {{< /warning >}} -Note that if providing database access to the client machine running Voyager is not possible, you can gather metadata from the source database using the provided bash scripts and then use Voyager to assess the migration. - The following table describes the type of data that is collected during a migration assessment. | Data | Collected | Details | @@ -46,151 +43,85 @@ The following table describes the type of data that is collected during a migrat | Performance metrics | Optional | Voyager captures performance metrics from the database (IOPS) for rightsizing the target environment. | | Server or database credentials | No | No server or database credentials are collected. | -### PostgreSQL sample Migration Assessment Report +## Get started with migration assessment -You can view a sample Migration Assessment report for PostgreSQL [here](/files/sample-report.html). - -## Generate a Migration Assessment Report +To get started with migration assessment, do the following: 1. [Install yb-voyager](../../install-yb-voyager/). -1. Prepare the source database. +1. Install YugabyteDB to view migration assessment report in the [yugabyted](/preview/reference/configuration/yugabyted/) UI. Using the yugabyted UI, you can visualize and review the database migration workflow performed by YugabyteDB Voyager. + 1. Start a local YugabyteDB cluster. Refer to the steps described in [Use a local cluster](/preview/quick-start/). + 1. To see the Voyager migration workflow details in the UI, set the following environment variables before starting the migration: - {{< tabpane text=true >}} + ```sh + export CONTROL_PLANE_TYPE=yugabyted + export YUGABYTED_DB_CONN_STRING= + ``` - {{% tab header="PostgreSQL" %}} + Provide the standard PostgreSQL connection parameters, including database, user name, host name, and port. For example, `postgresql://yugabyte:yugabyte@127.0.0.1:5433` -1. Create a new user, `ybvoyager` as follows: +1. Assess migration - Voyager supports two primary modes for conducting migration assessments, depending on your access to the source database as follows:

- ```sql - CREATE USER ybvoyager PASSWORD 'password'; - ``` + {{< tabpane text=true >}} -1. Grant necessary permissions to the `ybvoyager` user. + {{% tab header="With source database connectivity" %}} - ```sql - /* Switch to the database that you want to migrate.*/ - \c +This mode requires direct connectivity to the source database from the client machine where voyager is installed. You initiate the assessment by executing the `assess-migration` command of `yb-voyager`. This command facilitates a live analysis by interacting directly with the source database, to gather metadata required for assessment. A sample command is as follows: + +```sh +yb-voyager assess-migration --source-db-type postgresql \ + --source-db-host hostname --source-db-user ybvoyager \ + --source-db-password password --source-db-name dbname \ + --source-db-schema schema1,schema2 --export-dir /path/to/export/dir +``` - /* Grant the USAGE permission to the ybvoyager user on all schemas of the database.*/ + {{% /tab %}} - SELECT 'GRANT USAGE ON SCHEMA ' || schema_name || ' TO ybvoyager;' FROM information_schema.schemata; \gexec + {{% tab header="Without source database connectivity" %}} - /* The above SELECT statement generates a list of GRANT USAGE statements which are then executed by psql because of the \gexec switch. The \gexec switch works for PostgreSQL v9.6 and later. For older versions, you'll have to manually execute the GRANT USAGE ON SCHEMA schema_name TO ybvoyager statement, for each schema in the source PostgreSQL database. */ +PostgreSQL only. In situations where direct access to the source database is restricted, there is an alternative approach. Voyager includes packages with scripts for PostgreSQL at `/etc/yb-voyager/gather-assessment-metadata`. - /* Grant SELECT permission on all the tables. */ +You can perform the following steps with these scripts: - SELECT 'GRANT SELECT ON ALL TABLES IN SCHEMA ' || schema_name || ' TO ybvoyager;' FROM information_schema.schemata; \gexec - ``` +1. On a machine which has access to the source database, copy the scripts and install dependencies psql and pg_dump version 14 or later. Alternatively, you can install yb-voyager on the machine to automatically get the dependencies. -1. Execute the following command to refresh statistics for all tables in your database or schema: +1. Run the `yb-voyager-pg-gather-assessment-metadata.sh` script by providing the source connection string, the schema names, path to a directory where metadata will be saved, and an optional argument of an interval to capture the IOPS metadata of the source (in seconds with a default value of 120). For example, - ```sql - ANALYZE; + ```sh + /path/to/yb-voyager-pg-gather-assessment-metadata.sh 'postgresql://ybvoyager@host:port/dbname' 'schema1|schema2' '/path/to/assessment_metadata_dir' '60' ``` - {{% /tab %}} +1. Copy the metadata directory to the client machine on which voyager is installed, and run the `assess-migration` command by specifying the path to the metadata directory as follows: - {{% tab header="Oracle" %}} - -1. Create a role that has the privileges as listed in the following table: - - | Permission | Object type in the source schema | - | :--------- | :---------------------------------- | - | `SELECT` | VIEW, SEQUENCE, TABLE PARTITION, TABLE, SYNONYM, MATERIALIZED VIEW | - | `EXECUTE` | TYPE | - - Change the `` appropriately in the following snippets, and run the following steps as a privileged user. - - ```sql - CREATE ROLE _reader_role; - - BEGIN - FOR R IN (SELECT owner, object_name FROM all_objects WHERE owner=UPPER('') and object_type in ('VIEW','SEQUENCE','TABLE PARTITION','SYNONYM','MATERIALIZED VIEW')) - LOOP - EXECUTE IMMEDIATE 'grant select on '||R.owner||'."'||R.object_name||'" to _reader_role'; - END LOOP; - END; - / - - BEGIN - FOR R IN (SELECT owner, object_name FROM all_objects WHERE owner=UPPER('') and object_type ='TABLE' MINUS SELECT owner, table_name from all_nested_tables where owner = UPPER('')) - LOOP - EXECUTE IMMEDIATE 'grant select on '||R.owner||'."'||R.object_name||'" to _reader_role'; - END LOOP; - END; - / - - BEGIN - FOR R IN (SELECT owner, object_name FROM all_objects WHERE owner=UPPER('') and object_type = 'TYPE') - LOOP - EXECUTE IMMEDIATE 'grant execute on '||R.owner||'."'||R.object_name||'" to _reader_role'; - END LOOP; - END; - / - - GRANT SELECT_CATALOG_ROLE TO _reader_role; - GRANT SELECT ANY DICTIONARY TO _reader_role; - GRANT SELECT ON SYS.ARGUMENT$ TO _reader_role; - - ``` - -1. Create a user `ybvoyager` and grant `CONNECT` and `_reader_role` to the user: - - ```sql - CREATE USER ybvoyager IDENTIFIED BY password; - GRANT CONNECT TO ybvoyager; - GRANT _reader_role TO ybvoyager; - ``` - -1. Execute the following PL/SQL block to gather up-to-date statistics for your schema: - -```sql -BEGIN - DBMS_STATS.GATHER_SCHEMA_STATS('YOUR_SCHEMA_NAME'); -END; -/ -``` + ```sh + yb-voyager assess-migration --source-db-type postgresql \ + --assessment-metadata-dir /path/to/assessment_metadata_dir --export-dir /path/to/export/dir + ``` {{% /tab %}} -{{< /tabpane >}} - -1. Assess migration - Voyager supports two primary modes for conducting migration assessments, depending on your access to the source database as follows: - - 1. **With source database connectivity**: This mode requires direct connectivity to the source database from the client machine where voyager is installed. You initiate the assessment by executing the `assess-migration` command of `yb-voyager`. This command facilitates a live analysis by interacting directly with the source database, to gather metadata required for assessment. A sample command is as follows: + {{< /tabpane >}} - ```sh - yb-voyager assess-migration --source-db-type postgresql \ - --source-db-host hostname --source-db-user ybvoyager \ - --source-db-password password --source-db-name dbname \ - --source-db-schema schema1,schema2 --export-dir /path/to/export/dir - ``` + The output of both the methods is a migration assessment report, and its path is printed on the console. - 1. **Without source database connectivity** (only PostgreSQL): In situations where direct access to the source database is restricted, there is an alternative approach. Voyager includes packages with scripts for PostgreSQL at `/etc/yb-voyager/gather-assessment-metadata`. You can perform the following steps with these scripts: + {{< warning title="Important" >}} +For the most accurate migration assessment, the source database must be actively handling its typical workloads at the time the metadata is gathered. This ensures that the recommendations for sharding strategies and cluster sizing are well-aligned with the database's real-world performance and operational needs. + {{< /warning >}} - 1. On a machine which has access to the source database, copy the scripts and install dependencies psql, and pg_dump version 14 or later. Alternatively, you can install yb-voyager on the machine to automatically get the dependencies. - 1. Run the `yb-voyager-pg-gather-assessment-metadata.sh` script by providing the source connection string, the schema names, path to a directory where metadata will be saved, and an optional argument of an interval to capture the IOPS metadata of the source (in seconds with a default value of 120). For example, +1. View the assessment report. - ```sh - /path/to/yb-voyager-pg-gather-assessment-metadata.sh 'postgresql://ybvoyager@host:port/dbname' 'schema1|schema2' '/path/to/assessment_metadata_dir' '60' - ``` + Use the YugabyteDB UI to review the assessment report, which includes migration strategies, complexity, and effort estimates. - 1. Copy the metadata directory to the client machine on which voyager is installed, and run the `assess-migration` command by specifying the path to the metadata directory as follows: + After generating the report, navigate to the **Migrations** tab in the YugabyteDB UI at to see the available migrations. - ```sh - yb-voyager assess-migration --source-db-type postgresql \ - --assessment-metadata-dir /path/to/assessment_metadata_dir --export-dir /path/to/export/dir - ``` + ![Migration Landing Page](/images/migrate/ybd-landing-page.png) + ![Migration Assessment Page](/images/migrate/ybd-assessment-page.png) - The output of both the methods is a migration assessment report, and its path is printed on the console. +1. Create a target YugabyteDB cluster as follows: - {{< warning title="Important" >}} -For the most accurate migration assessment, the source database must be actively handling its typical workloads at the time the metadata is gathered. This ensures that the recommendations for sharding strategies and cluster sizing are well-aligned with the database's real-world performance and operational needs. - {{< /warning >}} + 1. Create a cluster in [Enhanced PostgreSQL Compatibility Mode](../../../explore/ysql-language-features/postgresql-compatibility/#enhanced-postgresql-compatibility-mode), based on the sizing recommendations in the assessment report. -1. Create a target YugabyteDB cluster as follows: + For a universe in YugabyteDB Anywhere, [enable compatibility mode](../../../explore/ysql-language-features/postgresql-compatibility/#yugabytedb-anywhere) by setting flags on the universe. - 1. Create a cluster in [Enhanced Postgres Compatibility Mode](/preview/releases/ybdb-releases/v2024.1/#highlights) based on the sizing recommendations in the assessment report. For a universe in YugabyteDB Anywhere, [enable the compatibility mode](/preview/releases/yba-releases/v2024.1/#highlights) by setting some flags on the universe. 1. Create a database with colocation set to TRUE. ```sql @@ -204,26 +135,33 @@ For the most accurate migration assessment, the source database must be actively - [Live migration with fall-forward](../../migrate/live-fall-forward/) - [Live migration with fall-back](../../migrate/live-fall-back/) -## Bulk assessment +## Assess a fleet of databases (Oracle only) -The Bulk Assessment command (`assess-migration-bulk`) allows you assess multiple schemas in one or more database instances simultaneously. Bulk assessment enables the following: +Use the Bulk Assessment command ([assess-migration-bulk](../reference/assess-migration/#assess-migration-bulk)) to assess multiple schemas across one or more database instances simultaneously. It offers: -- Multi-schema assessment - Assess multiple schemas across one or more database instances with a single command, streamlining the migration planning process. -- Centralized reporting - All assessment reports are generated and stored in a single organized directory, so you can review the migration assessments for all schemas in one place. +- Multi-Schema Assessment: Assess multiple schemas in different database instances with a single command, simplifying migration planning. +- Centralized Reporting: All assessment reports are generated and stored in one organized directory, making -![Oracle bulk assessment](/images/migrate/assess-migration-bulk.png) +### Command -### Prerequisites +To perform a bulk assessment, use the following command syntax: -Prepare source databases for each schema as described in [Generate a migration assessment](#generate-a-migration-assessment-report). +```sh +yb-voyager assess-migration-bulk \ + --fleet-config-file /path/to/fleet_config_file.csv \ + --bulk-assessment-dir /path/to/bulk-assessment-dir \ + [--continue-on-error true|false] \ + [--start-clean true|false] +``` ### Fleet configuration file -Bulk assessment is managed using a fleet configuration file, which specifies the details of schemas to be assessed. The file is in CSV format. +Bulk assessment is managed using a fleet configuration file, which specifies the schemas to be assessed. The file is in CSV format. -The first row of the file includes the headers that describe the fields to be included in each subsequent row. Each row after the header represents a different schema to be assessed. +- Header Row: The first row contains headers that define the fields for each schema. +- Schema Rows: Each subsequent row corresponds to a different schema to be assessed. -The following table describes the fields that can be included in the fleet configuration file. +The following table outlines the fields that can be included in the fleet configuration file. | Field | Description | | :--- | :--- | @@ -246,18 +184,6 @@ oracle,example-host2,1521,,ORCL_SID,,admin,password,schema2 oracle,,,,,tns_alias,oracle_user,password,schema3 ``` -### Command - -To perform a bulk assessment, use the following command syntax: - -```sh -yb-voyager assess-migration-bulk \ - --fleet-config-file /path/to/fleet_config_file.csv \ - --bulk-assessment-dir /path/to/bulk-assessment-dir \ - [--continue-on-error true|false] \ - [--start-clean true|false] -``` - ### Directory structure After the bulk assessment is completed, the top-level directory specified using the `--bulk-assessment-dir` flag includes subdirectories for each assessed schema. Additionally, a top-level report is generated that provides links to the individual assessment reports for each schema. @@ -280,37 +206,6 @@ After the bulk assessment is completed, the top-level directory specified using └── yb-voyager-assess-migration-bulk.log ``` -## Visualize the Migration Assessment report - -[yugabyted](/preview/reference/configuration/yugabyted/) UI allows you to visualize the database migrations performed by YugabyteDB Voyager. The UI provides details of migration complexity, SQL objects details from the source database, YugabyteDB sharding strategy, conversion issues (if any), and also allows you to track the percentage completion of data export from the source database and data import to the target YugabyteDB cluster. - -### Prerequisite - -Before you begin the Voyager migration, start a local YugabyteDB cluster. Refer to the steps described in [Use a local cluster](/preview/quick-start/). - -### Send Voyager details to a local YugabyteDB cluster - -Set the following environment variables before starting the migration: - -```sh -export CONTROL_PLANE_TYPE=yugabyted -export YUGABYTED_DB_CONN_STRING= -``` - -For example, `postgresql://yugabyte:yugabyte@127.0.0.1:5433` - -### Assess Migration - -Voyager Migration Assessment conducts a detailed analysis of the source database by capturing essential metadata and metrics. Yugabyted UI allows you to go over the assessment report which includes recommendations of effective migration strategies, migration complexity, and provides an overview on effort involved in migrating from the source database. - -After [generating a Migration Assessment Report](#generate-a-migration-assessment-report), from yugabyted UI, navigate to **Migrations** tab, available at [http://127.0.0.1:15433](http://127.0.0.1:15433) to see a list of the available migrations. - -![Migration Landing Page](/images/migrate/ybd-landing-page.png) - -#### Migration Assessment UI - -![Migration Assessment Page](/images/migrate/ybd-assessment-page.png) - ## Learn more -- [Assess migration CLI](../../reference/assess-migration/) \ No newline at end of file +- [Assess migration CLI](../../reference/assess-migration/) diff --git a/docs/static/images/migrate/voyager-migration-assessment-overview.png b/docs/static/images/migrate/voyager-migration-assessment-overview.png new file mode 100644 index 000000000000..58b685698ec1 Binary files /dev/null and b/docs/static/images/migrate/voyager-migration-assessment-overview.png differ