Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Consul: Add patroni_maximum_lag_on_replica #570

Merged
merged 1 commit into from
Feb 3, 2024
Merged

Consul: Add patroni_maximum_lag_on_replica #570

merged 1 commit into from
Feb 3, 2024

Conversation

vitabaks
Copy link
Owner

@vitabaks vitabaks commented Feb 3, 2024

Related PR: #569

@vitabaks vitabaks added the enhancement Improvement of the current functionality label Feb 3, 2024
@vitabaks vitabaks self-assigned this Feb 3, 2024
@vitabaks
Copy link
Owner Author

vitabaks commented Feb 3, 2024

Test (dcs_type: "consul") Consul DNS

Consul service config:

root@pgnode02:/# cat /etc/consul/conf.d/service_postgres-cluster-replica.json 
{
  "service": {
    "name": "postgres-cluster",
    "id": "postgres-cluster-replica",
    "port": 6432,
    "checks": [{"http": "http://10.172.0.21:8008/replica?lag=100MB", "interval": "2s"}, {"args": ["systemctl", "status", "pgbouncer"], "interval": "5s"}],
    "tags": ["replica"]
  }
}

Test commands:

# Set recovery_min_apply_delay on replica
psql -h 10.172.0.21 -p 5432 -U postgres -c "alter system set recovery_min_apply_delay='2min'"
psql -h 10.172.0.21 -p 5432 -U postgres -c "select pg_reload_conf()"
# Observe the replication lag
for i in {1..600}; do psql -h 10.172.0.20 -U postgres -p 5432 -c " select now(),client_addr,pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(),replay_lsn)) as total_lag from pg_stat_replication"; sleep 2; done
# Run check Patroni REST API for replica
for i in {1..600}; do echo $(date); curl -I http://10.172.0.21:8008/replica?lag=100MB; sleep 2; done
# Connect to replicas and check listen_addresses
for i in {1..600}; do echo $(date); psql -h replica.postgres-cluster.service.consul -p 6432 -U postgres -c "show listen_addresses"; sleep 2; done 
# Generate data to create a lag
pgbench -h 10.172.0.20 -p 5432 -U postgres -i -s 10 postgres

Result:



              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:06.374191+00 | 10.172.0.21 | 0 bytes
 2024-02-03 17:04:06.374191+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:08.425093+00 | 10.172.0.21 | 0 bytes
 2024-02-03 17:04:08.425093+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:10.480205+00 | 10.172.0.21 | 0 bytes
 2024-02-03 17:04:10.480205+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:12.536182+00 | 10.172.0.21 | 30 MB
 2024-02-03 17:04:12.536182+00 | 10.172.0.22 | 14 MB
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:14.596481+00 | 10.172.0.21 | 103 MB
 2024-02-03 17:04:14.596481+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:16.659394+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:16.659394+00 | 10.172.0.22 | 1529 kB
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:18.709996+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:18.709996+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:20.762715+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:20.762715+00 | 10.172.0.22 | 0 bytes
(2 rows)

             now             | client_addr | total_lag 
-----------------------------+-------------+-----------
 2024-02-03 17:04:22.8177+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:22.8177+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:24.871828+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:24.871828+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:26.933359+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:26.933359+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:28.985552+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:28.985552+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:31.037892+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:31.037892+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:33.092482+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:33.092482+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:35.142875+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:35.142875+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:37.195859+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:37.195859+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:39.248178+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:39.248178+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:41.303553+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:41.303553+00 | 10.172.0.22 | 0 bytes
(2 rows)

             now              | client_addr | total_lag 
------------------------------+-------------+-----------
 2024-02-03 17:04:43.35183+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:43.35183+00 | 10.172.0.22 | 0 bytes
(2 rows)

             now              | client_addr | total_lag 
------------------------------+-------------+-----------
 2024-02-03 17:04:45.40089+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:45.40089+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:47.450368+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:47.450368+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:49.499742+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:49.499742+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:51.559821+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:51.559821+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:53.614633+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:53.614633+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:55.660729+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:55.660729+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:57.705132+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:57.705132+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:59.753692+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:59.753692+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:01.809445+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:01.809445+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:03.859244+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:03.859244+00 | 10.172.0.22 | 0 bytes
(2 rows)

             now              | client_addr | total_lag 
------------------------------+-------------+-----------
 2024-02-03 17:05:05.90904+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:05.90904+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:07.958272+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:07.958272+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:10.007084+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:10.007084+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:12.057643+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:12.057643+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:14.111944+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:14.111944+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:16.173645+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:16.173645+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:18.220997+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:18.220997+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:20.270979+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:20.270979+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:22.316597+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:22.316597+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:24.366947+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:24.366947+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:26.412905+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:26.412905+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:28.468592+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:28.468592+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:30.512527+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:30.512527+00 | 10.172.0.22 | 0 bytes
(2 rows)

             now              | client_addr | total_lag 
------------------------------+-------------+-----------
 2024-02-03 17:05:32.56451+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:32.56451+00 | 10.172.0.22 | 0 bytes
(2 rows)

             now              | client_addr | total_lag 
------------------------------+-------------+-----------
 2024-02-03 17:05:34.61194+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:34.61194+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:36.667695+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:36.667695+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:38.713613+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:38.713613+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:40.770731+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:40.770731+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:42.827773+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:42.827773+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:44.892396+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:44.892396+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:46.959894+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:46.959894+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:49.012017+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:49.012017+00 | 10.172.0.22 | 0 bytes
(2 rows)

             now              | client_addr | total_lag 
------------------------------+-------------+-----------
 2024-02-03 17:05:51.07243+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:51.07243+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:53.126158+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:53.126158+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:55.181139+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:55.181139+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:57.238373+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:57.238373+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:59.302401+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:59.302401+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:06:01.360461+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:06:01.360461+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:06:03.413082+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:06:03.413082+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:06:05.479486+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:06:05.479486+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:06:07.545312+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:06:07.545312+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:06:09.665645+00 | 10.172.0.21 | 0 bytes
 2024-02-03 17:06:09.665645+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:06:11.734666+00 | 10.172.0.21 | 0 bytes
 2024-02-03 17:06:11.734666+00 | 10.172.0.22 | 0 bytes
(2 rows)


Sat Feb 3 17:04:04 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 17:04:06 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 17:04:08 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 17:04:10 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 17:04:12 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 17:04:14 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 17:04:16 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 17:04:18 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:20 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:22 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:24 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:26 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:28 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:30 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:32 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:34 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:36 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:38 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:40 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:42 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:44 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:46 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:48 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:50 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:52 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:54 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:56 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:58 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:00 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:02 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:04 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:06 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:08 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:10 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:12 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:14 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:16 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:18 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:20 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:22 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:24 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:26 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:28 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:30 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:32 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:34 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:36 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:38 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:40 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:42 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:44 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:46 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:49 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:51 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:53 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:55 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:57 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:59 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:06:01 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:06:03 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:06:05 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:06:07 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:06:09 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 17:06:11 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 17:06:13 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 17:06:15 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 17:06:17 UTC 2024
HTTP/1.0 200 OK


Sat Feb 3 17:04:04 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.21,127.0.0.1
(1 row)

Sat Feb 3 17:04:06 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.21,127.0.0.1
(1 row)

Sat Feb 3 17:04:08 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.21,127.0.0.1
(1 row)

Sat Feb 3 17:04:11 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.21,127.0.0.1
(1 row)

Sat Feb 3 17:04:13 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.21,127.0.0.1
(1 row)

Sat Feb 3 17:04:15 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.21,127.0.0.1
(1 row)

Sat Feb 3 17:04:17 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.21,127.0.0.1
(1 row)

Sat Feb 3 17:04:19 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:21 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:23 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:25 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:27 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:29 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:31 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:33 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:35 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:37 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:39 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:41 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:43 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:45 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:47 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:49 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:52 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:54 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:56 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:58 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:00 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:02 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:04 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:06 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:08 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:10 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:12 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:14 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:16 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:18 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:20 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:22 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:24 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:26 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:28 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:30 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:32 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:34 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:37 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:39 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:41 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:43 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:45 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:47 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:49 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:51 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:53 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:55 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:57 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:59 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:06:01 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:06:03 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:06:05 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:06:07 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:06:09 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.21,127.0.0.1
(1 row)

Sat Feb 3 17:06:11 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.21,127.0.0.1
(1 row)

Sat Feb 3 17:06:13 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.21,127.0.0.1
(1 row)

Sat Feb 3 17:06:16 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.21,127.0.0.1
(1 row)

Sat Feb 3 17:06:18 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.21,127.0.0.1
(1 row)

We observe that during a lag period of more than 100MB (definitely in patroni_maximum_lag_on_replica), replica 10.172.0.21 is removed from read traffic balancing (port 5001) and connections are routed only to the replica (10.172.0.22) without a high replication lag. Until the lag is below the threshold, and then replica 10.172.0.21 is available again to balance the read-only traffic.

passed

@vitabaks vitabaks merged commit e99e5a2 into master Feb 3, 2024
17 checks passed
@vitabaks vitabaks deleted the consul-max-lag branch February 3, 2024 18:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@vitabaks vitabaks

Labels
enhancement Improvement of the current functionality
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@vitabaks