Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[UPGRADE] Upgrade repmgr to 5.2.1 #2074

Closed
11 tasks done
erzetpe opened this issue Feb 19, 2021 · 9 comments
Closed
11 tasks done

[UPGRADE] Upgrade repmgr to 5.2.1 #2074

erzetpe opened this issue Feb 19, 2021 · 9 comments
Assignees
@przemyslavic @to-bar
Labels
area/postgres type/improvement

Comments

@erzetpe
Copy link
Contributor

erzetpe commented Feb 19, 2021 •
edited
Loading

Is your feature request related to a problem? Please describe.
We need to upgrade Replication Manager if possible and needed to the newer version.

Describe the solution you'd like
We want to upgrade if possible and needed to the newer version of Replication Manager.

Describe alternatives you've considered
None.

Additional context
This task can be done before upgrading PostgreSQL.

DoD checklist

  • Changelog updated
  • COMPONENTS.md updated / doesn't need to be updated
  • Feature has automated tests
  • Automated tests passed (QA pipelines)
    • apply
    • upgrade
  • Idempotency tested
  • Documentation added / updated / doesn't need to be updated
  • All conversations in PR resolved
  • Solution meets requirements and is done according to design doc
  • Usage compliant with license
@mkyc
Copy link
Contributor

mkyc commented Feb 26, 2021

isn't it duplicate off #1081 ?

@mkyc mkyc added this to the S20210520 milestone May 7, 2021
@rpudlowski93 rpudlowski93 mentioned this issue May 11, 2021
Closed
@rpudlowski93
Copy link
Contributor

Yes it is, I closed the task #1081

@rpudlowski93 rpudlowski93 changed the title [UPGRADE] Upgrade of Rep Mgr [UPGRADE] Upgrade of Replication Manager May 11, 2021
@atsikham
Copy link
Contributor

https://repmgr.org/docs/current/upgrading-repmgr-extension.html

@plirglo plirglo self-assigned this May 18, 2021
@plirglo
Copy link
Contributor

plirglo commented May 18, 2021

Starting with PostgreSQL 13 there is a name and value change of one of the parameters used for the streaming replication of a replica server. Until Postgres 12 it was called “wal_keep_segments” but with PostgreSQL 13 it will become “wal_keep_size”.

@plirglo plirglo removed their assignment May 20, 2021
@plirglo plirglo modified the milestones: S20210520, S20210603 May 20, 2021
@mkyc mkyc modified the milestones: S20210603, S20210617 Jun 7, 2021
@to-bar to-bar self-assigned this Jun 8, 2021
@mkyc mkyc modified the milestones: S20210617, S20210701 Jun 18, 2021
@to-bar
Copy link
Contributor

to-bar commented Jun 25, 2021

It was decided to upgrade repmgr for PostgreSQL 10 first, then integrate it with PostgreSQL 13. This means that this task may be done independently from PostgresSQL upgrade.

@mkyc mkyc modified the milestones: S20210701, S20210715 Jul 5, 2021
@mkyc mkyc mentioned this issue Jul 16, 2021
Closed
@to-bar to-bar changed the title [UPGRADE] Upgrade of Replication Manager [UPGRADE] Upgrade repmgr to 5.2.1 Jul 21, 2021
@przemyslavic przemyslavic self-assigned this Jul 21, 2021
@to-bar
Copy link
Contributor

to-bar commented Jul 21, 2021 •
edited
Loading

Moved to Needs review because of PR #2451

@przemyslavic
Copy link
Collaborator

przemyslavic commented Jul 26, 2021 •
edited
Loading

Repmgr fails to upgrade on Azure CentOS environments. Reproduced while trying to upgrade from epicli v1.0 and v1.1.

2021-07-26T09:56:21.0414645Z[38;21m09:56:21 INFO cli.engine.ansible.AnsibleCommand - TASK [postgresql : repmgr for PG 10 | Start repmgr service] ********************
2021-07-26T09:56:21.5055916Z[38;21m09:56:21 INFO cli.engine.ansible.AnsibleCommand - changed: [ci-10todevazceflannel-postgresql-vm-1]
2021-07-26T09:56:21.5245245Z[31;21m09:56:21 ERROR cli.engine.ansible.AnsibleCommand - fatal: [ci-10todevazceflannel-postgresql-vm-0]: FAILED! => {"changed": false, "msg": "Unable to start service repmgr10: Job for repmgr10.service failed because the control process exited with error code. See \"systemctl status repmgr10.service\" and \"journalctl -xe\" for details.\n"}


[root@ci-11todevazceflannel-postgresql-vm-0 ~]# systemctl status repmgr10.service
● repmgr10.service - A replication manager, and failover management tool for PostgreSQL
   Loaded: loaded (/usr/lib/systemd/system/repmgr10.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Mon 2021-07-26 09:52:16 UTC; 16min ago
  Process: 31220 ExecStart=/usr/pgsql-10/bin/repmgrd -f ${REPMGRDCONF} -p ${PIDFILE} -d --verbose (code=exited, status=1/FAILURE)

Jul 26 09:52:16 ci-11todevazceflannel-postgresql-vm-0 systemd[1]: Starting A replication manager, and failover management tool for PostgreSQL...
Jul 26 09:52:16 ci-11todevazceflannel-postgresql-vm-0 repmgrd[31220]: [2021-07-26 09:52:16] [NOTICE] using provided configuration file "/etc/repmgr....conf"
Jul 26 09:52:16 ci-11todevazceflannel-postgresql-vm-0 repmgrd[31220]: [2021-07-26 09:52:16] [NOTICE] redirecting logging output to "/var/log/postgr...r.log"
Jul 26 09:52:16 ci-11todevazceflannel-postgresql-vm-0 systemd[1]: repmgr10.service: control process exited, code=exited status=1
Jul 26 09:52:16 ci-11todevazceflannel-postgresql-vm-0 systemd[1]: Failed to start A replication manager, and failover management tool for PostgreSQL.
Jul 26 09:52:16 ci-11todevazceflannel-postgresql-vm-0 systemd[1]: Unit repmgr10.service entered failed state.
Jul 26 09:52:16 ci-11todevazceflannel-postgresql-vm-0 systemd[1]: repmgr10.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

[postgres@ci-11todevazceflannel-postgresql-vm-0 ~]$ repmgr service status
 ID | Name                                  | Role    | Status    | Upstream                              | repmgrd     | PID   | Paused? | Upstream last seen
----+---------------------------------------+---------+-----------+---------------------------------------+-------------+-------+---------+--------------------
 1  | ci-11todevazceflannel-postgresql-vm-1 | primary | * running |                                       | running     | 32193 | no      | n/a

 2  | ci-11todevazceflannel-postgresql-vm-0 | standby |   running | ci-11todevazceflannel-postgresql-vm-1 | not running | n/a   | n/a     | n/a

[root@ci-11todevazceflannel-postgresql-vm-0 ~]# journalctl -t repmgrd
-- Logs begin at Mon 2021-07-26 08:14:34 UTC, end at Mon 2021-07-26 10:10:19 UTC. --
Jul 26 09:17:22 ci-11todevazceflannel-postgresql-vm-0 repmgrd[19299]: [2021-07-26 09:17:22] [NOTICE] using provided configuration file "/etc/repmgr/10/repmg
Jul 26 09:17:22 ci-11todevazceflannel-postgresql-vm-0 repmgrd[19299]: [2021-07-26 09:17:22] [NOTICE] repmgrd (repmgr 4.0.6) starting up
Jul 26 09:17:22 ci-11todevazceflannel-postgresql-vm-0 repmgrd[19299]: [2021-07-26 09:17:22] [NOTICE] redirecting logging output to "/var/log/postgresql/repm
Jul 26 09:17:23 ci-11todevazceflannel-postgresql-vm-0 repmgrd[19570]: [2021-07-26 09:17:23] [NOTICE] using provided configuration file "/etc/repmgr/10/repmg
Jul 26 09:17:23 ci-11todevazceflannel-postgresql-vm-0 repmgrd[19570]: [2021-07-26 09:17:23] [NOTICE] repmgrd (repmgr 4.0.6) starting up
Jul 26 09:17:23 ci-11todevazceflannel-postgresql-vm-0 repmgrd[19570]: [2021-07-26 09:17:23] [NOTICE] redirecting logging output to "/var/log/postgresql/repm
Jul 26 09:52:16 ci-11todevazceflannel-postgresql-vm-0 repmgrd[31220]: [2021-07-26 09:52:16] [NOTICE] using provided configuration file "/etc/repmgr/10/repmg
Jul 26 09:52:16 ci-11todevazceflannel-postgresql-vm-0 repmgrd[31220]: [2021-07-26 09:52:16] [NOTICE] redirecting logging output to "/var/log/postgresql/repm

[root@ci-11todevazceflannel-postgresql-vm-0 ~]# cat /var/log/postgresql/repmgr.log
[2021-07-26 09:17:22] [NOTICE] starting monitoring of node "ci-11todevazceflannel-postgresql-vm-0" (ID: 2)
[2021-07-26 09:17:23] [NOTICE] starting monitoring of node "ci-11todevazceflannel-postgresql-vm-0" (ID: 2)
[2021-07-26 09:52:16] [NOTICE] repmgrd (repmgrd 5.2.1) starting up
[2021-07-26 09:52:16] [INFO] connecting to database "host=10.1.6.4 user=epi_repmgr dbname=epi_repmgr connect_timeout=2"
[2021-07-26 09:52:16] [ERROR] an older version of the "repmgr" extension is installed
[2021-07-26 09:52:16] [DETAIL] extension version 4.0 is installed but newer version 5.2 is available
[2021-07-26 09:52:16] [HINT] verify the repmgr installation is updated properly before continuing

Possible cause:
[2021-07-26 09:52:16] [ERROR] an older version of the "repmgr" extension is installed

It started working after executing systemctl start repmgr10.service command.

[root@ci-11todevazceflannel-postgresql-vm-0 ~]# systemctl status repmgr10.service
● repmgr10.service - A replication manager, and failover management tool for PostgreSQL
   Loaded: loaded (/usr/lib/systemd/system/repmgr10.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2021-07-26 10:13:30 UTC; 3s ago
  Process: 31592 ExecStart=/usr/pgsql-10/bin/repmgrd -f ${REPMGRDCONF} -p ${PIDFILE} -d --verbose (code=exited, status=0/SUCCESS)
 Main PID: 31595 (repmgrd)
   CGroup: /system.slice/repmgr10.service
           └─31595 /usr/pgsql-10/bin/repmgrd -f /etc/repmgr/10/repmgr.conf -p /var/run/repmgr/repmgrd-10.pid -d --verbose

Jul 26 10:13:30 ci-11todevazceflannel-postgresql-vm-0 systemd[1]: Starting A replication manager, and failover management tool for PostgreSQL...
Jul 26 10:13:30 ci-11todevazceflannel-postgresql-vm-0 repmgrd[31592]: [2021-07-26 10:13:30] [NOTICE] using provided configuration file "/etc/repmgr....conf"
Jul 26 10:13:30 ci-11todevazceflannel-postgresql-vm-0 repmgrd[31592]: [2021-07-26 10:13:30] [NOTICE] redirecting logging output to "/var/log/postgr...r.log"
Jul 26 10:13:30 ci-11todevazceflannel-postgresql-vm-0 systemd[1]: Can't open PID file /var/run/repmgr/repmgrd-10.pid (yet?) after start: No such fi...ectory
Jul 26 10:13:30 ci-11todevazceflannel-postgresql-vm-0 systemd[1]: Started A replication manager, and failover management tool for PostgreSQL.
Hint: Some lines were ellipsized, use -l to show in full.

[root@ci-11todevazceflannel-postgresql-vm-0 ~]# cat /var/log/postgresql/repmgr.log
[2021-07-26 09:17:22] [NOTICE] starting monitoring of node "ci-11todevazceflannel-postgresql-vm-0" (ID: 2)
[2021-07-26 09:17:23] [NOTICE] starting monitoring of node "ci-11todevazceflannel-postgresql-vm-0" (ID: 2)
[2021-07-26 09:52:16] [NOTICE] repmgrd (repmgrd 5.2.1) starting up
[2021-07-26 09:52:16] [INFO] connecting to database "host=10.1.6.4 user=epi_repmgr dbname=epi_repmgr connect_timeout=2"
[2021-07-26 09:52:16] [ERROR] an older version of the "repmgr" extension is installed
[2021-07-26 09:52:16] [DETAIL] extension version 4.0 is installed but newer version 5.2 is available
[2021-07-26 09:52:16] [HINT] verify the repmgr installation is updated properly before continuing
[2021-07-26 10:13:30] [NOTICE] repmgrd (repmgrd 5.2.1) starting up
[2021-07-26 10:13:30] [INFO] connecting to database "host=10.1.6.4 user=epi_repmgr dbname=epi_repmgr connect_timeout=2"
INFO:  set_repmgrd_pid(): provided pidfile is /var/run/repmgr/repmgrd-10.pid
[2021-07-26 10:13:30] [NOTICE] starting monitoring of node "ci-11todevazceflannel-postgresql-vm-0" (ID: 2)
[2021-07-26 10:13:30] [INFO] "connection_check_type" set to "ping"
[2021-07-26 10:13:30] [INFO] monitoring connection to upstream node "ci-11todevazceflannel-postgresql-vm-1" (ID: 1)

[root@ci-11todevazceflannel-postgresql-vm-0 ~]# journalctl -t repmgrd
-- Logs begin at Mon 2021-07-26 08:14:34 UTC, end at Mon 2021-07-26 10:13:43 UTC. --
Jul 26 09:17:22 ci-11todevazceflannel-postgresql-vm-0 repmgrd[19299]: [2021-07-26 09:17:22] [NOTICE] using provided configuration file "/etc/repmgr/10/repmg
Jul 26 09:17:22 ci-11todevazceflannel-postgresql-vm-0 repmgrd[19299]: [2021-07-26 09:17:22] [NOTICE] repmgrd (repmgr 4.0.6) starting up
Jul 26 09:17:22 ci-11todevazceflannel-postgresql-vm-0 repmgrd[19299]: [2021-07-26 09:17:22] [NOTICE] redirecting logging output to "/var/log/postgresql/repm
Jul 26 09:17:23 ci-11todevazceflannel-postgresql-vm-0 repmgrd[19570]: [2021-07-26 09:17:23] [NOTICE] using provided configuration file "/etc/repmgr/10/repmg
Jul 26 09:17:23 ci-11todevazceflannel-postgresql-vm-0 repmgrd[19570]: [2021-07-26 09:17:23] [NOTICE] repmgrd (repmgr 4.0.6) starting up
Jul 26 09:17:23 ci-11todevazceflannel-postgresql-vm-0 repmgrd[19570]: [2021-07-26 09:17:23] [NOTICE] redirecting logging output to "/var/log/postgresql/repm
Jul 26 09:52:16 ci-11todevazceflannel-postgresql-vm-0 repmgrd[31220]: [2021-07-26 09:52:16] [NOTICE] using provided configuration file "/etc/repmgr/10/repmg
Jul 26 09:52:16 ci-11todevazceflannel-postgresql-vm-0 repmgrd[31220]: [2021-07-26 09:52:16] [NOTICE] redirecting logging output to "/var/log/postgresql/repm
Jul 26 10:13:30 ci-11todevazceflannel-postgresql-vm-0 repmgrd[31592]: [2021-07-26 10:13:30] [NOTICE] using provided configuration file "/etc/repmgr/10/repmg
Jul 26 10:13:30 ci-11todevazceflannel-postgresql-vm-0 repmgrd[31592]: [2021-07-26 10:13:30] [NOTICE] redirecting logging output to "/var/log/postgresql/repm

[postgres@ci-11todevazceflannel-postgresql-vm-0 ~]$ repmgr service status
 ID | Name                                  | Role    | Status    | Upstream                              | repmgrd | PID   | Paused? | Upstream last seen
----+---------------------------------------+---------+-----------+---------------------------------------+---------+-------+---------+--------------------
 1  | ci-11todevazceflannel-postgresql-vm-1 | primary | * running |                                       | running | 32193 | no      | n/a
 2  | ci-11todevazceflannel-postgresql-vm-0 | standby |   running | ci-11todevazceflannel-postgresql-vm-1 | running | 31595 | no      | 0 second(s) ago

@przemyslavic przemyslavic mentioned this issue Jul 27, 2021
Merged
@to-bar
Copy link
Contributor

to-bar commented Jul 27, 2021

PR #2459 updated.

@przemyslavic
Copy link
Collaborator

Fixed.
✔️ epicli apply dev
✔️ epicli upgrade from v0.9.2
✔️ epicli upgrade from v1.0.1
✔️ epicli upgrade from v1.0.0

Tested (only) on AWS Ubuntu, Azure Ubuntu, Azure CentOS (because of issues with Grafana repository #2449).

@erzetpe erzetpe closed this as completed Jul 29, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@przemyslavic przemyslavic

@to-bar to-bar

Labels
area/postgres type/improvement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@erzetpe @atsikham @mkyc @rpudlowski93 @przemyslavic @to-bar @plirglo