Add support for PostgreSQL 17

[vitabaks]

Added support for Postgres 17. It is now used by default.

[vitabaks]

Test: Upgrade

PLAY [pg_upgrade.yml | Upgrade PostgreSQL 16 to the new version 17] ***

TASK [Gathering Facts] *********************************************************
ok: [10.172.2.22]
ok: [10.172.2.20]
ok: [10.172.2.21]

TASK [Include main variables] **************************************************
ok: [10.172.2.20]
ok: [10.172.2.21]
ok: [10.172.2.22]

TASK [Include upgrade variables] ***********************************************
ok: [10.172.2.20]
ok: [10.172.2.21]
ok: [10.172.2.22]
[WARNING]: Module remote_tmp /var/lib/postgresql/.ansible/tmp did not exist and
was created with a mode of 0700, this may cause issues when running as another
user. To avoid this, create the remote_tmp dir with the correct permissions
manually

TASK [[Prepare] Get Patroni Cluster Leader Node] *******************************
ok: [10.172.2.21]
ok: [10.172.2.22]
ok: [10.172.2.20]

TASK [[Prepare] Add host to group "primary" (in-memory inventory)] *************
ok: [10.172.2.20] => (item=10.172.2.20)

TASK [[Prepare] Add hosts to group "secondary" (in-memory inventory)] **********
ok: [10.172.2.20] => (item=10.172.2.21)
ok: [10.172.2.20] => (item=10.172.2.22)

TASK [Print Patroni Cluster info] **********************************************
ok: [10.172.2.20] => {
    "msg": [
        "Cluster Name: postgres-cluster",
        "Cluster Leader: pgnode01"
    ]
}

PLAY [(1/6) PRE-UPGRADE: Perform Pre-Checks] ***********************************

TASK [Include main variables] **************************************************
ok: [10.172.2.20]
ok: [10.172.2.21]
ok: [10.172.2.22]

TASK [Include OS-specific variables] *******************************************
ok: [10.172.2.20]
ok: [10.172.2.21]
ok: [10.172.2.22]

TASK [Include upgrade variables] ***********************************************
ok: [10.172.2.20]
ok: [10.172.2.21]
ok: [10.172.2.22]

TASK [Running Pre-Checks] ******************************************************

TASK [upgrade : [Pre-Check] Make sure the ansible required Python library is installed] ***
changed: [10.172.2.21] => (item=pexpect)
changed: [10.172.2.20] => (item=pexpect)
changed: [10.172.2.22] => (item=pexpect)

TASK [upgrade : [Pre-Check] Test PostgreSQL database access using a unix socket] ***
ok: [10.172.2.20]
ok: [10.172.2.21]
ok: [10.172.2.22]

TASK [upgrade : [Pre-Check] Check the current version of PostgreSQL] ***********
ok: [10.172.2.20]

TASK [upgrade : Set variable 'current_pg_version'] *****************************
ok: [10.172.2.20]

TASK [upgrade : [Pre-Check] Ensure new data directory is different from the current one] ***
ok: [10.172.2.20]

TASK [upgrade : [Pre-Check] Make sure that physical replication is active] *****
ok: [10.172.2.20]

TASK [upgrade : [Pre-Check] Make sure there is no high replication lag (more than 10.00 MB)] ***
ok: [10.172.2.20]

TASK [upgrade : [Pre-Check] Make sure there are no long-running transactions (more than 15 seconds)] ***
ok: [10.172.2.21]
ok: [10.172.2.20]
ok: [10.172.2.22]

TASK [upgrade : [Pre-Check] Make sure that SSH key-based authentication is configured between cluster nodes] ***
included: /workspace/autobase/automation/roles/upgrade/tasks/ssh-keys.yml for 10.172.2.20, 10.172.2.21, 10.172.2.22

TASK [upgrade : Make sure that the openssh-client package is installed] ********
ok: [10.172.2.21]
ok: [10.172.2.22]
ok: [10.172.2.20]

TASK [upgrade : Make sure the SSH key for user "postgres" exists] **************
changed: [10.172.2.20]
changed: [10.172.2.21]
changed: [10.172.2.22]

TASK [upgrade : Fetch public SSH keys from database servers] *******************
ok: [10.172.2.20]
ok: [10.172.2.22]
ok: [10.172.2.21]

TASK [upgrade : Add public SSH keys to authorized_keys] ************************
changed: [10.172.2.20]
changed: [10.172.2.22]
changed: [10.172.2.21]

TASK [upgrade : Remove public SSH keys from localhost] *************************
ok: [10.172.2.20 -> localhost] => (item=10.172.2.20)
ok: [10.172.2.20 -> localhost] => (item=10.172.2.21)
ok: [10.172.2.20 -> localhost] => (item=10.172.2.22)

TASK [upgrade : known_hosts | for each host, scan for its ssh public key] ******
ok: [10.172.2.21] => (item=10.172.2.20)
ok: [10.172.2.20] => (item=10.172.2.20)
ok: [10.172.2.22] => (item=10.172.2.20)
ok: [10.172.2.20] => (item=10.172.2.21)
ok: [10.172.2.22] => (item=10.172.2.21)
ok: [10.172.2.21] => (item=10.172.2.21)
ok: [10.172.2.22] => (item=10.172.2.22)
ok: [10.172.2.21] => (item=10.172.2.22)
ok: [10.172.2.20] => (item=10.172.2.22)

TASK [upgrade : known_hosts | for each host, add/update the public key in the "~postgres/.ssh/known_hosts"] ***
changed: [10.172.2.20] => (item=None)
changed: [10.172.2.22] => (item=None)
changed: [10.172.2.21] => (item=None)
changed: [10.172.2.20] => (item=None)
changed: [10.172.2.21] => (item=None)
changed: [10.172.2.22] => (item=None)
changed: [10.172.2.21] => (item=None)
changed: [10.172.2.20] => (item=None)
changed: [10.172.2.22] => (item=None)
changed: [10.172.2.21]
changed: [10.172.2.22]
changed: [10.172.2.20]

TASK [upgrade : [Pre-Check] Make sure that the sshpass package are installed] ***
ok: [10.172.2.22]
ok: [10.172.2.20]
ok: [10.172.2.21]

TASK [upgrade : [Pre-Check] Make sure that the rsync package are installed] ****
ok: [10.172.2.20]
ok: [10.172.2.21]
ok: [10.172.2.22]

TASK [upgrade : [Pre-Check] Rsync Checks: create testrsync file on Primary] ****
changed: [10.172.2.20]

TASK [upgrade : [Pre-Check] Rsync Checks: test rsync and ssh key access] *******
changed: [10.172.2.20] => (item=10.172.2.21)
changed: [10.172.2.20] => (item=10.172.2.22)

TASK [upgrade : [Pre-Check] Cleanup testrsync file] ****************************
changed: [10.172.2.22]
changed: [10.172.2.20]
changed: [10.172.2.21]

TASK [upgrade : [Pre-Check] Check if PostgreSQL tablespaces exist] *************
ok: [10.172.2.20]

TASK [upgrade : Make sure that the 'recovery.signal' file is absent in the data directory] ***
ok: [10.172.2.21]
ok: [10.172.2.22]
ok: [10.172.2.20]

TASK [upgrade : Ensure correct permissions for PgBouncer unix socket directory] ***
ok: [10.172.2.21] => (item=pgbouncer)
ok: [10.172.2.20] => (item=pgbouncer)
ok: [10.172.2.22] => (item=pgbouncer)
ok: [10.172.2.20] => (item=pgbouncer-2)
ok: [10.172.2.22] => (item=pgbouncer-2)
ok: [10.172.2.21] => (item=pgbouncer-2)
ok: [10.172.2.20] => (item=pgbouncer-3)
ok: [10.172.2.22] => (item=pgbouncer-3)
ok: [10.172.2.21] => (item=pgbouncer-3)
ok: [10.172.2.21] => (item=pgbouncer-4)
ok: [10.172.2.22] => (item=pgbouncer-4)
ok: [10.172.2.20] => (item=pgbouncer-4)

TASK [upgrade : [Pre-Check] Test PgBouncer access via unix socket] *************
ok: [10.172.2.21] => (item=pgbouncer)
ok: [10.172.2.20] => (item=pgbouncer)
ok: [10.172.2.22] => (item=pgbouncer)
ok: [10.172.2.22] => (item=pgbouncer-2)
ok: [10.172.2.20] => (item=pgbouncer-2)
ok: [10.172.2.21] => (item=pgbouncer-2)
ok: [10.172.2.21] => (item=pgbouncer-3)
ok: [10.172.2.20] => (item=pgbouncer-3)
ok: [10.172.2.22] => (item=pgbouncer-3)
ok: [10.172.2.22] => (item=pgbouncer-4)
ok: [10.172.2.21] => (item=pgbouncer-4)
ok: [10.172.2.20] => (item=pgbouncer-4)

PLAY [(2/6) PRE-UPGRADE: Install new PostgreSQL packages] **********************

TASK [Include main variables] **************************************************
ok: [10.172.2.20]
ok: [10.172.2.21]
ok: [10.172.2.22]

TASK [Include OS-specific variables] *******************************************
ok: [10.172.2.20]
ok: [10.172.2.21]
ok: [10.172.2.22]

TASK [Include upgrade variables] ***********************************************
ok: [10.172.2.20]
ok: [10.172.2.21]
ok: [10.172.2.22]

TASK [Install packages] ********************************************************

TASK [upgrade : Update apt cache] **********************************************
ok: [10.172.2.21]
ok: [10.172.2.22]
ok: [10.172.2.20]

TASK [upgrade : Install PostgreSQL 17 packages] ********************************
changed: [10.172.2.22] => (item=postgresql-17)
changed: [10.172.2.20] => (item=postgresql-17)
changed: [10.172.2.21] => (item=postgresql-17)
ok: [10.172.2.22] => (item=postgresql-client-17)
ok: [10.172.2.20] => (item=postgresql-client-17)
ok: [10.172.2.21] => (item=postgresql-client-17)
ok: [10.172.2.22] => (item=postgresql-contrib-17)
ok: [10.172.2.20] => (item=postgresql-contrib-17)
ok: [10.172.2.21] => (item=postgresql-contrib-17)
changed: [10.172.2.22] => (item=postgresql-server-dev-17)
changed: [10.172.2.20] => (item=postgresql-server-dev-17)
changed: [10.172.2.21] => (item=postgresql-server-dev-17)
changed: [10.172.2.22] => (item=postgresql-17-dbgsym)
changed: [10.172.2.20] => (item=postgresql-17-dbgsym)
changed: [10.172.2.21] => (item=postgresql-17-dbgsym)

TASK [upgrade : Install TimescaleDB package] ***********************************
changed: [10.172.2.22] => (item=timescaledb-2-postgresql-17)
changed: [10.172.2.21] => (item=timescaledb-2-postgresql-17)
changed: [10.172.2.20] => (item=timescaledb-2-postgresql-17)

PLAY [(3/6) PRE-UPGRADE: Initialize new db, schema compatibility check, and pg_upgrade --check] ***

TASK [Include main variables] **************************************************
ok: [10.172.2.20]
ok: [10.172.2.21]
ok: [10.172.2.22]

TASK [Include OS-specific variables] *******************************************
ok: [10.172.2.20]
ok: [10.172.2.21]
ok: [10.172.2.22]

TASK [Include upgrade variables] ***********************************************
ok: [10.172.2.20]
ok: [10.172.2.21]
ok: [10.172.2.22]

TASK [Create Data directory and initdb] ****************************************

TASK [upgrade : Make sure new PostgreSQL data directory "/pgdata/17/main" exists] ***
changed: [10.172.2.22]
changed: [10.172.2.20]
changed: [10.172.2.21]

TASK [upgrade : Make sure new PostgreSQL data directory "/pgdata/17/main" is not initialized] ***
ok: [10.172.2.20]
ok: [10.172.2.21]
ok: [10.172.2.22]

TASK [upgrade : Get the current install user] **********************************
ok: [10.172.2.20]

TASK [upgrade : Get the current encodig and data_checksums settings] ***********
ok: [10.172.2.20]

TASK [upgrade : Initialize new PostgreSQL data directory with default config files] ***
changed: [10.172.2.22]
changed: [10.172.2.21]
changed: [10.172.2.20]

TASK [Copy files] **************************************************************

TASK [upgrade : Get the current shared_preload_libraries settings] *************
ok: [10.172.2.20]

TASK [upgrade : Check if PostgreSQL is running] ********************************
ok: [10.172.2.20]

TASK [upgrade : Start new PostgreSQL on port 5433 to check the schema compatibility] ***
changed: [10.172.2.20]

TASK [upgrade : Wait for PostgreSQL to start] **********************************
changed: [10.172.2.20]

TASK [upgrade : Check the compatibility of the database schema with the PostgreSQL 17] ***
changed: [10.172.2.20]

TASK [upgrade : Wait for the schema compatibility check to complete.] **********
changed: [10.172.2.20]

TASK [upgrade : Checking the result of the schema compatibility] ***************
ok: [10.172.2.20]

TASK [upgrade : Result of checking the compatibility of the scheme - success] ***
ok: [10.172.2.20] => {
    "msg": "The database schema are compatible with PostgreSQL 17"
}

TASK [upgrade : Drop new PostgreSQL to re-initdb (perform pg_dropcluster)] *****
changed: [10.172.2.20]

TASK [upgrade : Reinitialize the database after checking schema compatibility] ***
included: /workspace/autobase/automation/roles/upgrade/tasks/initdb.yml for 10.172.2.20

TASK [upgrade : Make sure new PostgreSQL data directory "/pgdata/17/main" exists] ***
changed: [10.172.2.20]

TASK [upgrade : Make sure new PostgreSQL data directory "/pgdata/17/main" is not initialized] ***
ok: [10.172.2.20]

TASK [upgrade : Get the current install user] **********************************
ok: [10.172.2.20]

TASK [upgrade : Get the current encodig and data_checksums settings] ***********
ok: [10.172.2.20]

TASK [upgrade : Initialize new PostgreSQL data directory with default config files] ***
changed: [10.172.2.20]

TASK [upgrade : Get the current shared_preload_libraries settings] *************
ok: [10.172.2.20]

TASK [upgrade : Set the variable: pg_shared_preload_libraries_value] ***********
ok: [10.172.2.20]

TASK [upgrade : Verify the two clusters are compatible (pg_upgrade --check)] ***
changed: [10.172.2.20]

TASK [upgrade : Print the result of the pg_upgrade check] **********************
ok: [10.172.2.20] => {
    "failed_when_result": false,
    "pg_upgrade_check_result.stdout_lines": [
        "Performing Consistency Checks on Old Live Server",
        "------------------------------------------------",
        "Checking cluster versions                                     ok",
        "Checking database user is the install user                    ok",
        "Checking database connection settings                         ok",
        "Checking for prepared transactions                            ok",
        "Checking for contrib/isn with bigint-passing mismatch         ok",
        "Checking data type usage                                      ok",
        "Checking for presence of required libraries                   ok",
        "Checking database user is the install user                    ok",
        "Checking for prepared transactions                            ok",
        "Checking for new cluster tablespace directories               ok",
        "",
        "*Clusters are compatible*"
    ]
}

PLAY [(4/6) PRE-UPGRADE: Prepare the Patroni configuration] ********************

TASK [Include main variables] **************************************************
ok: [10.172.2.20]
ok: [10.172.2.21]
ok: [10.172.2.22]

TASK [Include OS-specific variables] *******************************************
ok: [10.172.2.20]
ok: [10.172.2.21]
ok: [10.172.2.22]

TASK [Include upgrade variables] ***********************************************
ok: [10.172.2.20]
ok: [10.172.2.21]
ok: [10.172.2.22]

TASK [Patroni config] **********************************************************

TASK [upgrade : Backup the patroni.yml configuration file] *********************
changed: [10.172.2.22]
changed: [10.172.2.20]
changed: [10.172.2.21]

TASK [upgrade : Remove patroni.dynamic.json file] ******************************
changed: [10.172.2.20]
changed: [10.172.2.22]
changed: [10.172.2.21]

TASK [upgrade : Edit patroni.yml | update parameters: data_dir, bin_dir, config_dir] ***
changed: [10.172.2.21] => (item=data_dir: /pgdata/17/main)
changed: [10.172.2.22] => (item=data_dir: /pgdata/17/main)
changed: [10.172.2.20] => (item=data_dir: /pgdata/17/main)
changed: [10.172.2.22] => (item=bin_dir: /usr/lib/postgresql/17/bin)
changed: [10.172.2.21] => (item=bin_dir: /usr/lib/postgresql/17/bin)
changed: [10.172.2.20] => (item=bin_dir: /usr/lib/postgresql/17/bin)
changed: [10.172.2.20] => (item=config_dir: /etc/postgresql/17/main)
changed: [10.172.2.21] => (item=config_dir: /etc/postgresql/17/main)
changed: [10.172.2.22] => (item=config_dir: /etc/postgresql/17/main)

TASK [upgrade : Edit patroni.yml | check if the 'db_user_namespace' parameter is specified] ***
ok: [10.172.2.22]
ok: [10.172.2.21]
ok: [10.172.2.20]

TASK [upgrade : Edit patroni.yml | check if the 'old_snapshot_threshold' parameter is specified] ***
ok: [10.172.2.20]
ok: [10.172.2.21]
ok: [10.172.2.22]

TASK [upgrade : Edit patroni.yml | check if the 'trace_recovery_messages' parameter is specified] ***
ok: [10.172.2.21]
ok: [10.172.2.22]
ok: [10.172.2.20]

TASK [upgrade : Copy pg_hba.conf to /etc/postgresql/17/main] *******************
changed: [10.172.2.21]
changed: [10.172.2.22]
changed: [10.172.2.20]

PLAY [(5/6) UPGRADE: Upgrade PostgreSQL] ***************************************

TASK [Include main variables] **************************************************
ok: [10.172.2.20]
ok: [10.172.2.21]
ok: [10.172.2.22]

TASK [Include OS-specific variables] *******************************************
ok: [10.172.2.20]
ok: [10.172.2.21]
ok: [10.172.2.22]

TASK [Include upgrade variables] ***********************************************
ok: [10.172.2.20]
ok: [10.172.2.21]
ok: [10.172.2.22]

TASK [Enable maintenance mode] *************************************************

TASK [upgrade : Pause Patroni cluster (enable maintenance mode)] ***************
changed: [10.172.2.20]

TASK [upgrade : Gather facts from balancers] ***********************************
ok: [10.172.2.20] => (item=10.172.2.20)
ok: [10.172.2.20 -> 10.172.2.21] => (item=10.172.2.21)
ok: [10.172.2.20 -> 10.172.2.22] => (item=10.172.2.22)

TASK [upgrade : Stop confd service] ********************************************
changed: [10.172.2.20] => (item=10.172.2.20)
changed: [10.172.2.20 -> 10.172.2.21] => (item=10.172.2.21)
changed: [10.172.2.20 -> 10.172.2.22] => (item=10.172.2.22)

TASK [upgrade : Update haproxy conf file (disable http-checks)] ****************
changed: [10.172.2.20] => (item=10.172.2.20)
changed: [10.172.2.20 -> 10.172.2.21] => (item=10.172.2.21)
changed: [10.172.2.20 -> 10.172.2.22] => (item=10.172.2.22)

TASK [upgrade : Reload haproxy service] ****************************************
changed: [10.172.2.20] => (item=10.172.2.20)
changed: [10.172.2.20 -> 10.172.2.21] => (item=10.172.2.21)
changed: [10.172.2.20 -> 10.172.2.22] => (item=10.172.2.22)

TASK [upgrade : Stop Patroni service] ******************************************
changed: [10.172.2.20]
changed: [10.172.2.21]
changed: [10.172.2.22]
FAILED - RETRYING: [10.172.2.20]: Wait until the Patroni cluster is stopped (30 retries left).
FAILED - RETRYING: [10.172.2.20]: Wait until the Patroni cluster is stopped (29 retries left).

TASK [upgrade : Wait until the Patroni cluster is stopped] *********************
ok: [10.172.2.20]

TASK [Stop Services] ***********************************************************

TASK [upgrade : Execute CHECKPOINT before stopping PostgreSQL] *****************
changed: [10.172.2.21]
changed: [10.172.2.22]
changed: [10.172.2.20]

TASK [upgrade : Wait for the CHECKPOINT to complete] ***************************
changed: [10.172.2.22]
changed: [10.172.2.21]
changed: [10.172.2.20]

TASK [upgrade : Wait until replication lag is less than 10.00 MB] **************
ok: [10.172.2.20]

TASK [upgrade : Perform PAUSE on all pgbouncers servers] ***********************
included: /workspace/autobase/automation/roles/upgrade/tasks/pgbouncer_pause.yml for 10.172.2.20, 10.172.2.21, 10.172.2.22

TASK [upgrade : Ensure correct permissions for PgBouncer unix socket directory] ***
ok: [10.172.2.22] => (item=pgbouncer)
ok: [10.172.2.21] => (item=pgbouncer)
ok: [10.172.2.20] => (item=pgbouncer)
ok: [10.172.2.22] => (item=pgbouncer-2)
ok: [10.172.2.21] => (item=pgbouncer-2)
ok: [10.172.2.20] => (item=pgbouncer-2)
ok: [10.172.2.20] => (item=pgbouncer-3)
ok: [10.172.2.21] => (item=pgbouncer-3)
ok: [10.172.2.22] => (item=pgbouncer-3)
ok: [10.172.2.22] => (item=pgbouncer-4)
ok: [10.172.2.21] => (item=pgbouncer-4)
ok: [10.172.2.20] => (item=pgbouncer-4)

TASK [upgrade : PAUSE PgBouncer pools] *****************************************
changed: [10.172.2.20]

TASK [upgrade : Stop PostgreSQL on the Leader] *********************************
changed: [10.172.2.20]

TASK [upgrade : Stop PostgreSQL on the Replica] ********************************
changed: [10.172.2.21]
changed: [10.172.2.22]

TASK [upgrade : Check if PostgreSQL 16 is stopped] *****************************
ok: [10.172.2.20]
ok: [10.172.2.22]
ok: [10.172.2.21]

TASK [upgrade : Check if PostgreSQL 17 is stopped] *****************************
ok: [10.172.2.22]
ok: [10.172.2.20]
ok: [10.172.2.21]

TASK [Check 'Latest checkpoint location'] **************************************

TASK [upgrade : Get 'Latest checkpoint location' on the Leader] ****************
ok: [10.172.2.20]

TASK [upgrade : Get 'Latest checkpoint location' on the Replicas] **************
ok: [10.172.2.22]
ok: [10.172.2.21]

TASK [upgrade : Print 'Latest checkpoint location' for the Leader] *************
ok: [10.172.2.20] => {
    "msg": "Leader's latest checkpoint location: 0/6000028"
}

TASK [upgrade : Print 'Latest checkpoint location' for the Replica] ************
ok: [10.172.2.21] => {
    "msg": "Replica: 10.172.2.21, latest checkpoint location: 0/6000028"
}
ok: [10.172.2.22] => {
    "msg": "Replica: 10.172.2.22, latest checkpoint location: 0/6000028"
}

TASK [upgrade : Check if all 'Latest checkpoint location' values match] ********
ok: [10.172.2.21]
ok: [10.172.2.22]

TASK [upgrade : SUCCESS: 'Latest checkpoint location' values match on all cluster nodes] ***
ok: [10.172.2.20] => {
    "msg": "'Latest checkpoint location' is the same on the leader and its standbys"
}

TASK [Upgrade Primary] *********************************************************

TASK [upgrade : Upgrade the PostgreSQL to version 17 on the Primary (using pg_upgrade --link)] ***
changed: [10.172.2.20]

TASK [upgrade : Print the result of the pg_upgrade] ****************************
ok: [10.172.2.20] => {
    "msg": [
        [
            "Performing Consistency Checks",
            "-----------------------------",
            "Checking cluster versions                                     ok",
            "Checking database user is the install user                    ok",
            "Checking database connection settings                         ok",
            "Checking for prepared transactions                            ok",
            "Checking for contrib/isn with bigint-passing mismatch         ok",
            "Checking data type usage                                      ok",
            "Creating dump of global objects                               ok",
            "Creating dump of database schemas                             ok",
            "Checking for presence of required libraries                   ok",
            "Checking database user is the install user                    ok",
            "Checking for prepared transactions                            ok",
            "Checking for new cluster tablespace directories               ok",
            "",
            "If pg_upgrade fails after this point, you must re-initdb the",
            "new cluster before continuing.",
            "",
            "Performing Upgrade",
            "------------------",
            "Setting locale and encoding for new cluster                   ok",
            "Analyzing all rows in the new cluster                         ok",
            "Freezing all rows in the new cluster                          ok",
            "Deleting files from new pg_xact                               ok",
            "Copying old pg_xact to new server                             ok",
            "Setting oldest XID for new cluster                            ok",
            "Setting next transaction ID and epoch for new cluster         ok",
            "Deleting files from new pg_multixact/offsets                  ok",
            "Copying old pg_multixact/offsets to new server                ok",
            "Deleting files from new pg_multixact/members                  ok",
            "Copying old pg_multixact/members to new server                ok",
            "Setting next multixact ID and offset for new cluster          ok",
            "Resetting WAL archives                                        ok",
            "Setting frozenxid and minmxid counters in new cluster         ok",
            "Restoring global objects in the new cluster                   ok",
            "Restoring database schemas in the new cluster                 ok",
            "Adding \".old\" suffix to old global/pg_control                 ok",
            "",
            "If you want to start the old cluster, you will need to remove",
            "the \".old\" suffix from /pgdata/16/main/global/pg_control.old.",
            "Because \"link\" mode was used, the old cluster cannot be safely",
            "started once the new cluster has been started.",
            "Linking user relation files                                   ok",
            "Setting next OID for new cluster                              ok",
            "Sync data directory to disk                                   ok",
            "Creating script to delete old cluster                         ok",
            "Checking for extension updates                                ok",
            "",
            "Upgrade Complete",
            "----------------",
            "Optimizer statistics are not transferred by pg_upgrade.",
            "Once you start the new server, consider running:",
            "    /usr/lib/postgresql/17/bin/vacuumdb -U postgres --all --analyze-in-stages",
            "Running this script will delete the old cluster's data files:",
            "    ./delete_old_cluster.sh"
        ]
    ]
}

TASK [Upgrade Secondary] *******************************************************

TASK [upgrade : Make sure that the new data directory "/pgdata/17/main" are empty on the Replica] ***
changed: [10.172.2.21] => (item=absent)
changed: [10.172.2.22] => (item=absent)
changed: [10.172.2.22] => (item=directory)
changed: [10.172.2.21] => (item=directory)

TASK [upgrade : Upgrade the PostgreSQL on the Replica (using rsync --hard-links)] ***
changed: [10.172.2.20]
FAILED - RETRYING: [10.172.2.20]: Wait for the rsync to complete. (1800 retries left).

TASK [upgrade : Wait for the rsync to complete.] *******************************
changed: [10.172.2.20]

TASK [Create WAL dir symlink] **************************************************

TASK [upgrade : Make sure /pgdata/17/main/pg_wal is not symlink] ***************
ok: [10.172.2.22]
ok: [10.172.2.20]
ok: [10.172.2.21]

TASK [upgrade : Make sure the custom WAL directory "/pgwal/17/pg_wal" exists and is empty] ***
ok: [10.172.2.21] => (item=absent)
ok: [10.172.2.20] => (item=absent)
ok: [10.172.2.22] => (item=absent)
changed: [10.172.2.22] => (item=directory)
changed: [10.172.2.21] => (item=directory)
changed: [10.172.2.20] => (item=directory)

TASK [upgrade : Synchronize /pgdata/17/main/pg_wal to /pgwal/17/pg_wal] ********
changed: [10.172.2.22]
changed: [10.172.2.20]
changed: [10.172.2.21]

TASK [upgrade : Rename pg_wal to pg_wal_old] ***********************************
changed: [10.172.2.22]
changed: [10.172.2.20]
changed: [10.172.2.21]

TASK [upgrade : Create symlink /pgdata/17/main/pg_wal -> /pgwal/17/pg_wal] *****
changed: [10.172.2.20]
changed: [10.172.2.22]
changed: [10.172.2.21]

TASK [upgrade : Remove pg_wal_old directory] ***********************************
changed: [10.172.2.21]
changed: [10.172.2.20]
changed: [10.172.2.22]

TASK [Remove old cluster from DCS] *********************************************

TASK [upgrade : Remove existing cluster "postgres-cluster" from DCS] ***********
changed: [10.172.2.20]

TASK [Start Services] **********************************************************

TASK [upgrade : Start Patroni service on the Cluster Leader] *******************
changed: [10.172.2.20]

TASK [upgrade : Wait for Patroni port "8008" to become open on the host] *******
ok: [10.172.2.20]

TASK [upgrade : Check Patroni is healthy on the Leader] ************************
ok: [10.172.2.20]

TASK [upgrade : Perform RESUME PgBouncer pools on the Leader] ******************
included: /workspace/autobase/automation/roles/upgrade/tasks/pgbouncer_resume.yml for 10.172.2.20

TASK [upgrade : RESUME PgBouncer pools] ****************************************
changed: [10.172.2.20]

TASK [upgrade : Start Patroni service on the Cluster Replica] ******************
changed: [10.172.2.22]
changed: [10.172.2.21]

TASK [upgrade : Wait for Patroni port "8008" to become open on the host] *******
ok: [10.172.2.21]
ok: [10.172.2.22]

TASK [upgrade : Check Patroni is healthy on the Replica] ***********************
ok: [10.172.2.21]
ok: [10.172.2.22]

TASK [upgrade : Perform RESUME PgBouncer pools on the Replica] *****************
included: /workspace/autobase/automation/roles/upgrade/tasks/pgbouncer_resume.yml for 10.172.2.21, 10.172.2.22

TASK [upgrade : RESUME PgBouncer pools] ****************************************
changed: [10.172.2.22]
changed: [10.172.2.21]

TASK [upgrade : Check PostgreSQL is started and accepting connections] *********
ok: [10.172.2.21]
ok: [10.172.2.22]
ok: [10.172.2.20]

TASK [Disable maintenance mode] ************************************************

TASK [upgrade : Update haproxy conf file (enable http-checks)] *****************
changed: [10.172.2.20] => (item=10.172.2.20)
changed: [10.172.2.20 -> 10.172.2.21] => (item=10.172.2.21)
changed: [10.172.2.20 -> 10.172.2.22] => (item=10.172.2.22)

TASK [upgrade : Reload haproxy service] ****************************************
changed: [10.172.2.20] => (item=10.172.2.20)
changed: [10.172.2.20 -> 10.172.2.21] => (item=10.172.2.21)
changed: [10.172.2.20 -> 10.172.2.22] => (item=10.172.2.22)

TASK [upgrade : Start confd service] *******************************************
changed: [10.172.2.20] => (item=10.172.2.20)
changed: [10.172.2.20 -> 10.172.2.21] => (item=10.172.2.21)
changed: [10.172.2.20 -> 10.172.2.22] => (item=10.172.2.22)

PLAY [(6/6) POST-UPGRADE: Analyze a PostgreSQL database (update optimizer statistics) and Post-Upgrade tasks] ***

TASK [Include main variables] **************************************************
ok: [10.172.2.20]
ok: [10.172.2.21]
ok: [10.172.2.22]

TASK [Include OS-specific variables] *******************************************
ok: [10.172.2.20]
ok: [10.172.2.21]
ok: [10.172.2.22]

TASK [Include upgrade variables] ***********************************************
ok: [10.172.2.20]
ok: [10.172.2.21]
ok: [10.172.2.22]

TASK [Analyze database] ********************************************************

TASK [upgrade : pg_terminator: Monitor locks and terminate the 'ANALYZE' blockers] ***
changed: [10.172.2.20]

TASK [upgrade : Run vacuumdb to analyze the PostgreSQL databases] **************
changed: [10.172.2.20]

TASK [Update extensions] *******************************************************

TASK [upgrade : Get a list of databases] ***************************************
ok: [10.172.2.20]

TASK [upgrade : Update extensions in each database] ****************************
included: /workspace/autobase/automation/roles/upgrade/tasks/update_extensions.yml for 10.172.2.20 => (item=template1)
included: /workspace/autobase/automation/roles/upgrade/tasks/update_extensions.yml for 10.172.2.20 => (item=postgres)

TASK [upgrade : Get list of installed PostgreSQL extensions (database: template1)] ***
ok: [10.172.2.20]

TASK [upgrade : Get list of old PostgreSQL extensions (database: template1)] ***
ok: [10.172.2.20]

TASK [upgrade : The extensions are up-to-date (database: template1)] ***********
ok: [10.172.2.20] => {
    "msg": [
        "The extension versions are up-to-date for the database template1",
        "No update is required."
    ]
}

TASK [upgrade : Get list of installed PostgreSQL extensions (database: postgres)] ***
ok: [10.172.2.20]

TASK [upgrade : Get list of old PostgreSQL extensions (database: postgres)] ****
ok: [10.172.2.20]

TASK [upgrade : The extensions are up-to-date (database: postgres)] ************
ok: [10.172.2.20] => {
    "msg": [
        "The extension versions are up-to-date for the database postgres",
        "No update is required."
    ]
}

TASK [Running Post-Checks] *****************************************************

TASK [upgrade : Make sure that physical replication is active] *****************
ok: [10.172.2.20]

TASK [upgrade : Create a table "test_replication" with 10000 rows on the Primary] ***
changed: [10.172.2.20]

TASK [upgrade : Wait until the PostgreSQL replica is synchronized] *************
ok: [10.172.2.21]
ok: [10.172.2.22]

TASK [upgrade : Drop a table "test_replication"] *******************************
changed: [10.172.2.20]

TASK [upgrade : Print the result of checking the number of records] ************
ok: [10.172.2.21] => {
    "msg": [
        "The PostgreSQL Replication is OK for replica pgnode02",
        "The number of records in the test_replication table the same as the Primary (10000 rows)"
    ]
}
ok: [10.172.2.22] => {
    "msg": [
        "The PostgreSQL Replication is OK for replica pgnode03",
        "The number of records in the test_replication table the same as the Primary (10000 rows)"
    ]
}

TASK [Running Post-Upgrade tasks] **********************************************

TASK [upgrade : Ensure the current data directory is the new data directory] ***
ok: [10.172.2.21]
ok: [10.172.2.20]
ok: [10.172.2.22]

TASK [upgrade : Delete the old PostgreSQL data directory (perform pg_dropcluster)] ***
changed: [10.172.2.20]
changed: [10.172.2.21]
changed: [10.172.2.22]

TASK [upgrade : Delete the old PostgreSQL WAL directory] ***********************
ok: [10.172.2.22]
ok: [10.172.2.21]
ok: [10.172.2.20]

TASK [upgrade : Remove old PostgreSQL packages] ********************************
changed: [10.172.2.20] => (item=postgresql-16)
changed: [10.172.2.21] => (item=postgresql-16)
changed: [10.172.2.22] => (item=postgresql-16)
changed: [10.172.2.22] => (item=postgresql-client-16)
changed: [10.172.2.20] => (item=postgresql-client-16)
changed: [10.172.2.21] => (item=postgresql-client-16)
ok: [10.172.2.21] => (item=postgresql-contrib-16)
ok: [10.172.2.22] => (item=postgresql-contrib-16)
ok: [10.172.2.20] => (item=postgresql-contrib-16)
ok: [10.172.2.22] => (item=postgresql-server-dev-16)
ok: [10.172.2.21] => (item=postgresql-server-dev-16)
ok: [10.172.2.20] => (item=postgresql-server-dev-16)
ok: [10.172.2.21] => (item=postgresql-16-dbgsym)
ok: [10.172.2.22] => (item=postgresql-16-dbgsym)
ok: [10.172.2.20] => (item=postgresql-16-dbgsym)

TASK [upgrade : Collecting statistics in progress. Wait for the analyze to complete.] ***
changed: [10.172.2.20]

TASK [upgrade : Stop pg_terminator script] *************************************
changed: [10.172.2.20]

TASK [upgrade : Check the Patroni cluster state] *******************************
ok: [10.172.2.20]

TASK [upgrade : Check the current PostgreSQL version] **************************
ok: [10.172.2.20]

TASK [upgrade : List the Patroni cluster members] ******************************
ok: [10.172.2.20] => {
    "msg": [
        "+ Cluster: postgres-cluster (7444924166131954559) --+-----------+",
        "| Member   | Host        | Role    | State     | TL | Lag in MB |",
        "+----------+-------------+---------+-----------+----+-----------+",
        "| pgnode01 | 10.172.2.20 | Leader  | running   |  2 |           |",
        "| pgnode02 | 10.172.2.21 | Replica | streaming |  2 |         0 |",
        "| pgnode03 | 10.172.2.22 | Replica | streaming |  2 |         0 |",
        "+----------+-------------+---------+-----------+----+-----------+"
    ]
}

TASK [upgrade : Upgrade completed] *********************************************
ok: [10.172.2.20] => {
    "msg": [
        "PostgreSQL upgrade completed.",
        "Current version: 17.2 (Ubuntu 17.2-1.pgdg22.04+1)"
    ]
}

PLAY RECAP *********************************************************************
10.172.2.20                : ok=313  changed=135  unreachable=0    failed=0    skipped=728  rescued=0    ignored=0
10.172.2.21                : ok=225  changed=105  unreachable=0    failed=0    skipped=683  rescued=0    ignored=0
10.172.2.22                : ok=225  changed=105  unreachable=0    failed=0    skipped=683  rescued=0    ignored=0
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=223  rescued=0    ignored=0

passed