Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[HAproxy] Fixup number of connections #4650

Closed
wants to merge 2 commits into from

Conversation

Markuze
Copy link
Contributor

@Markuze Markuze commented Sep 29, 2022

  • [haproxy] config fixup
  • Allowing more TCP connections to the backend
  1. I'm decreasing all non-validator backends to no more than 16 connections, as each is taking up a connection slot and resources.
  2. I'm increasing the total number of connections to 8K = we have 30-sec timeout for a NOISE connection, and we allow no more than 256 new TCP connections per second ~ 30 * 256 < 8K, this is needed otherwise, even whitelisted connections won't go trough.

Description

Test Plan


This change is Reviewable

@Markuze Markuze requested a review from a team as a code owner September 29, 2022 19:24
@Markuze Markuze added the CICD:run-e2e-tests when this label is present github actions will run all land-blocking e2e tests from the PR label Sep 29, 2022
@Markuze Markuze requested review from sitalkedia, a team and zekun000 and removed request for a team September 29, 2022 19:38
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@Markuze Markuze force-pushed the markuze/haproxy_fixup_nr_connections branch from 33efaac to 8f6f2bc Compare September 30, 2022 19:10
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@Markuze Markuze force-pushed the markuze/haproxy_fixup_nr_connections branch from 8f6f2bc to 5220051 Compare September 30, 2022 19:53
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions
Copy link
Contributor

❌ Forge suite land_blocking failure on 5220051aab7fa746f715c8cf90e1d88c11a4607d

Forge test runner terminated:
Trailing Log Lines:
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:120"},"thread_name":"main","hostname":"forge-e2e-pr-4650-1664569582-5220051aab7fa746f715c8cf90e1d88c11","timestamp":"2022-09-30T20:48:47.976721Z","message":"Deployment aptos-node-0-haproxy has no status"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:111"},"thread_name":"main","hostname":"forge-e2e-pr-4650-1664569582-5220051aab7fa746f715c8cf90e1d88c11","timestamp":"2022-09-30T20:48:57.982985Z","message":"Deployment aptos-node-0-haproxy has 0 ready_replicas"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:120"},"thread_name":"main","hostname":"forge-e2e-pr-4650-1664569582-5220051aab7fa746f715c8cf90e1d88c11","timestamp":"2022-09-30T20:48:57.983006Z","message":"Deployment aptos-node-0-haproxy has no status"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:111"},"thread_name":"main","hostname":"forge-e2e-pr-4650-1664569582-5220051aab7fa746f715c8cf90e1d88c11","timestamp":"2022-09-30T20:49:07.987832Z","message":"Deployment aptos-node-0-haproxy has 0 ready_replicas"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:120"},"thread_name":"main","hostname":"forge-e2e-pr-4650-1664569582-5220051aab7fa746f715c8cf90e1d88c11","timestamp":"2022-09-30T20:49:07.987852Z","message":"Deployment aptos-node-0-haproxy has no status"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:280"},"thread_name":"main","hostname":"forge-e2e-pr-4650-1664569582-5220051aab7fa746f715c8cf90e1d88c11","timestamp":"2022-09-30T20:49:08.013363Z","message":"Deleting namespace forge-e2e-pr-4650: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:388"},"thread_name":"main","hostname":"forge-e2e-pr-4650-1664569582-5220051aab7fa746f715c8cf90e1d88c11","timestamp":"2022-09-30T20:49:08.013388Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-4650"}
Failed to run tests:
Deployment not ready
Error: Deployment not ready
Debugging output:
NAME                                     READY   STATUS             RESTARTS       AGE
aptos-node-0-haproxy-7f7dc6d99f-r4h9v    0/1     Terminating        8              17m
aptos-node-1-haproxy-7668db8745-bflgb    0/1     Terminating        8              17m
aptos-node-10-haproxy-5c8db4df8b-4lvpf   0/1     Terminating        8              17m
aptos-node-11-haproxy-575fd79658-5dzrn   0/1     Terminating        8              17m
aptos-node-11-validator-0                1/1     Terminating        0              17m
aptos-node-12-haproxy-b88f574df-j774g    0/1     CrashLoopBackOff   8 (78s ago)    17m
aptos-node-12-validator-0                1/1     Running            0              17m
aptos-node-13-haproxy-864dbf8f6-b8r2p    0/1     CrashLoopBackOff   8 (79s ago)    17m
aptos-node-13-validator-0                1/1     Running            0              17m
aptos-node-14-haproxy-76bc7b5db8-gd7g9   0/1     CrashLoopBackOff   8 (87s ago)    17m
aptos-node-14-validator-0                1/1     Running            0              17m
aptos-node-15-haproxy-6d5d8d7d76-fnd24   0/1     CrashLoopBackOff   8 (101s ago)   17m
aptos-node-15-validator-0                1/1     Running            0              17m
aptos-node-16-haproxy-78958798f5-knz7g   0/1     CrashLoopBackOff   8 (91s ago)    17m
aptos-node-16-validator-0                1/1     Running            0              17m
aptos-node-17-haproxy-7d8b7bc7dc-cxhzk   0/1     CrashLoopBackOff   8 (75s ago)    17m
aptos-node-17-validator-0                1/1     Running            0              17m
aptos-node-18-haproxy-8fd5ffd4f-n2bqg    0/1     CrashLoopBackOff   8 (93s ago)    17m
aptos-node-18-validator-0                1/1     Running            0              17m
aptos-node-19-haproxy-845f89bbbb-ttkwb   0/1     CrashLoopBackOff   8 (100s ago)   17m
aptos-node-19-validator-0                1/1     Running            0              17m
aptos-node-2-fullnode-eforge87-0         1/1     Running            0              17m
aptos-node-2-haproxy-85df977b45-xvkrg    0/1     CrashLoopBackOff   8 (67s ago)    17m
aptos-node-2-validator-0                 1/1     Running            0              17m
aptos-node-20-haproxy-6577c55fc7-whjvt   0/1     CrashLoopBackOff   8 (59s ago)    17m
aptos-node-20-validator-0                1/1     Running            0              17m
aptos-node-21-haproxy-d458cd77-jkjs4     0/1     CrashLoopBackOff   8 (72s ago)    17m
aptos-node-21-validator-0                1/1     Running            0              17m
aptos-node-22-haproxy-86775b4bdd-fcpxs   0/1     CrashLoopBackOff   8 (81s ago)    17m
aptos-node-22-validator-0                1/1     Running            0              17m
aptos-node-23-haproxy-764d64848f-98ttc   0/1     CrashLoopBackOff   8 (96s ago)    17m
aptos-node-23-validator-0                1/1     Running            0              17m
aptos-node-24-haproxy-767d66c458-wtr45   0/1     CrashLoopBackOff   8 (78s ago)    17m
aptos-node-24-validator-0                1/1     Running            0              17m
aptos-node-25-haproxy-75f766c56d-l9fdj   0/1     CrashLoopBackOff   8 (85s ago)    17m
aptos-node-25-validator-0                1/1     Running            0              17m
aptos-node-26-haproxy-84f8cd9ff5-fxtxn   0/1     CrashLoopBackOff   8 (70s ago)    17m
aptos-node-26-validator-0                1/1     Running            0              17m
aptos-node-27-haproxy-6b45d66765-2n4dt   0/1     CrashLoopBackOff   8 (95s ago)    17m
aptos-node-27-validator-0                1/1     Running            0              17m
aptos-node-28-haproxy-75897fc898-pchqr   0/1     CrashLoopBackOff   8 (86s ago)    17m
aptos-node-28-validator-0                1/1     Running            0              17m
aptos-node-29-haproxy-bc946b6b6-wbrvm    0/1     CrashLoopBackOff   8 (102s ago)   17m
aptos-node-29-validator-0                1/1     Running            0              17m
aptos-node-3-fullnode-eforge87-0         1/1     Running            0              17m
aptos-node-3-haproxy-7df6c65484-fr6zq    0/1     CrashLoopBackOff   8 (86s ago)    17m
aptos-node-3-validator-0                 1/1     Running            0              17m
aptos-node-30-haproxy-d6f94f477-stqpq    0/1     CrashLoopBackOff   8 (57s ago)    17m
aptos-node-30-validator-0                1/1     Running            0              17m
aptos-node-31-haproxy-76d5797d96-lxzhc   0/1     CrashLoopBackOff   8 (83s ago)    17m
aptos-node-31-validator-0                1/1     Running            0              17m
aptos-node-32-haproxy-5f75c75f5-ffj9g    0/1     CrashLoopBackOff   8 (100s ago)   17m
aptos-node-32-validator-0                1/1     Running            0              17m
aptos-node-33-haproxy-7cd59fb799-j7hst   0/1     CrashLoopBackOff   8 (87s ago)    17m
aptos-node-33-validator-0                1/1     Running            0              17m
aptos-node-34-haproxy-5777d7658-dvww6    0/1     CrashLoopBackOff   8 (97s ago)    17m
aptos-node-34-validator-0                1/1     Running            0              17m
aptos-node-35-haproxy-6d8f7d7cd6-gjhpv   0/1     CrashLoopBackOff   8 (83s ago)    17m
aptos-node-35-validator-0                1/1     Running            0              17m
aptos-node-36-haproxy-7548d84d94-gh79d   0/1     CrashLoopBackOff   8 (62s ago)    17m
aptos-node-36-validator-0                1/1     Running            0              17m
aptos-node-37-haproxy-8565b646f-hwcjt    0/1     CrashLoopBackOff   8 (75s ago)    17m
aptos-node-37-validator-0                1/1     Running            0              17m
aptos-node-38-haproxy-7b746c9697-6jwl9   0/1     CrashLoopBackOff   8 (71s ago)    17m
aptos-node-38-validator-0                1/1     Running            0              17m
aptos-node-39-haproxy-b5448b89f-fqdwn    0/1     CrashLoopBackOff   8 (89s ago)    17m
aptos-node-39-validator-0                1/1     Running            0              17m
aptos-node-4-fullnode-eforge87-0         1/1     Running            0              17m
aptos-node-4-haproxy-66bb6bdbbb-grczm    0/1     CrashLoopBackOff   8 (77s ago)    17m
aptos-node-4-validator-0                 1/1     Running            0              17m
aptos-node-40-haproxy-668786c7cd-kcwxk   0/1     CrashLoopBackOff   8 (82s ago)    17m
aptos-node-40-validator-0                1/1     Running            0              17m
aptos-node-41-haproxy-65bf857d4d-q9c89   0/1     CrashLoopBackOff   8 (84s ago)    17m
aptos-node-41-validator-0                1/1     Running            0              17m
aptos-node-42-haproxy-5b7c75c764-xlcpm   0/1     CrashLoopBackOff   8 (106s ago)   17m
aptos-node-42-validator-0                1/1     Running            0              17m
aptos-node-43-haproxy-58897b8b47-64ncx   0/1     CrashLoopBackOff   8 (77s ago)    17m
aptos-node-43-validator-0                1/1     Running            0              17m
aptos-node-44-haproxy-6ccd878b76-6sqf2   0/1     CrashLoopBackOff   8 (61s ago)    17m
aptos-node-44-validator-0                1/1     Running            0              17m
aptos-node-45-haproxy-76bf8b4b8-swtf6    0/1     CrashLoopBackOff   8 (71s ago)    17m
aptos-node-45-validator-0                1/1     Running            0              17m
aptos-node-46-haproxy-5cfd5dc9f4-mxb79   0/1     CrashLoopBackOff   8 (86s ago)    17m
aptos-node-46-validator-0                1/1     Running            0              17m
aptos-node-47-haproxy-6dc8cfd8d6-jntgf   0/1     CrashLoopBackOff   8 (94s ago)    17m
aptos-node-47-validator-0                1/1     Running            0              17m
aptos-node-48-haproxy-5857c674f9-tps7s   0/1     CrashLoopBackOff   8 (92s ago)    17m
aptos-node-48-validator-0                1/1     Running            0              17m
aptos-node-49-haproxy-5bdd86bb9d-2f8nd   0/1     CrashLoopBackOff   8 (83s ago)    17m
aptos-node-49-validator-0                1/1     Running            0              17m
aptos-node-5-fullnode-eforge87-0         1/1     Running            0              17m
aptos-node-5-haproxy-7b68f854c9-tgnjh    0/1     CrashLoopBackOff   8 (98s ago)    17m
aptos-node-5-validator-0                 1/1     Running            0              17m
aptos-node-50-haproxy-7bbb68bffb-tkxj7   0/1     CrashLoopBackOff   8 (95s ago)    17m
aptos-node-50-validator-0                1/1     Running            0              17m
aptos-node-51-haproxy-584fb985c7-hr8sn   0/1     CrashLoopBackOff   8 (97s ago)    17m
aptos-node-51-validator-0                1/1     Running            0              17m
aptos-node-52-haproxy-59576b594c-xbcrf   0/1     CrashLoopBackOff   8 (82s ago)    17m
aptos-node-52-validator-0                1/1     Running            0              17m
aptos-node-53-haproxy-5d5784444b-tng7r   0/1     CrashLoopBackOff   8 (74s ago)    17m
aptos-node-53-validator-0                1/1     Running            0              17m
aptos-node-54-haproxy-5496dbb4b6-29nwr   0/1     CrashLoopBackOff   8 (95s ago)    17m
aptos-node-54-validator-0                1/1     Running            0              17m
aptos-node-55-haproxy-79bfb9d664-ssz75   0/1     CrashLoopBackOff   8 (62s ago)    17m
aptos-node-55-validator-0                1/1     Running            0              17m
aptos-node-56-haproxy-7dc7f8f48b-l9hjk   0/1     CrashLoopBackOff   8 (85s ago)    17m
aptos-node-56-validator-0                1/1     Running            0              17m
aptos-node-57-haproxy-59f48c5b86-w76v6   0/1     CrashLoopBackOff   8 (79s ago)    17m
aptos-node-57-validator-0                1/1     Running            0              17m
aptos-node-58-haproxy-77474f5bc5-z4bjz   0/1     CrashLoopBackOff   8 (94s ago)    17m
aptos-node-58-validator-0                1/1     Running            0              17m
aptos-node-59-haproxy-5589b74795-6t5n2   0/1     CrashLoopBackOff   8 (89s ago)    17m
aptos-node-59-validator-0                1/1     Running            0              17m
aptos-node-6-fullnode-eforge87-0         1/1     Running            0              17m
aptos-node-6-haproxy-68dcb85974-chl92    0/1     CrashLoopBackOff   8 (81s ago)    17m
aptos-node-6-validator-0                 1/1     Running            0              17m
aptos-node-60-haproxy-5557bb86cb-25cs7   0/1     CrashLoopBackOff   8 (77s ago)    17m
aptos-node-60-validator-0                1/1     Running            0              17m
aptos-node-61-haproxy-5646f7dd44-5wz5v   0/1     CrashLoopBackOff   8 (91s ago)    17m
aptos-node-61-validator-0                1/1     Running            0              17m
aptos-node-62-haproxy-845c645689-f4tg5   0/1     CrashLoopBackOff   8 (78s ago)    17m
aptos-node-62-validator-0                1/1     Running            0              17m
aptos-node-63-haproxy-6476fff958-qbxsz   0/1     CrashLoopBackOff   8 (72s ago)    17m
aptos-node-63-validator-0                1/1     Running            0              17m
aptos-node-64-haproxy-6966dd986c-n8hq2   0/1     CrashLoopBackOff   8 (76s ago)    17m
aptos-node-64-validator-0                1/1     Running            0              17m
aptos-node-65-haproxy-7685dff689-rpdnl   0/1     CrashLoopBackOff   8 (84s ago)    17m
aptos-node-65-validator-0                1/1     Running            0              17m
aptos-node-66-haproxy-85b59bb5d5-zf4vm   0/1     CrashLoopBackOff   8 (88s ago)    17m
aptos-node-66-validator-0                1/1     Running            0              17m
aptos-node-67-haproxy-5b6df59bcb-98fzm   0/1     CrashLoopBackOff   8 (82s ago)    17m
aptos-node-67-validator-0                1/1     Running            0              17m
aptos-node-68-haproxy-5858fc684f-mhb7x   0/1     CrashLoopBackOff   8 (88s ago)    17m
aptos-node-68-validator-0                1/1     Running            0              17m
aptos-node-69-haproxy-c8795974d-cnhjl    0/1     CrashLoopBackOff   8 (97s ago)    17m
aptos-node-69-validator-0                1/1     Running            0              17m
aptos-node-7-fullnode-eforge87-0         1/1     Running            0              17m
aptos-node-7-haproxy-74c97f98bc-htdk5    0/1     CrashLoopBackOff   8 (74s ago)    17m
aptos-node-7-validator-0                 1/1     Running            0              17m
aptos-node-70-haproxy-5f456c4685-s6kw5   0/1     CrashLoopBackOff   8 (66s ago)    17m
aptos-node-70-validator-0                1/1     Running            0              17m
aptos-node-71-haproxy-dc76bb46b-ml2xh    0/1     CrashLoopBackOff   8 (90s ago)    17m
aptos-node-71-validator-0                1/1     Running            0              17m
aptos-node-72-haproxy-865446c587-mxp2z   0/1     CrashLoopBackOff   8 (81s ago)    17m
aptos-node-72-validator-0                1/1     Running            0              17m
aptos-node-73-haproxy-5b7ff647c5-2kfpd   0/1     CrashLoopBackOff   8 (90s ago)    17m
aptos-node-73-validator-0                1/1     Running            0              17m
aptos-node-74-haproxy-5d4c9b7444-vd6wk   0/1     CrashLoopBackOff   8 (95s ago)    17m
aptos-node-74-validator-0                1/1     Running            0              17m
aptos-node-75-haproxy-69c6644497-cnbcj   0/1     CrashLoopBackOff   8 (91s ago)    17m
aptos-node-75-validator-0                1/1     Running            0              17m
aptos-node-76-haproxy-84dbcff8bc-6ql9m   0/1     CrashLoopBackOff   8 (84s ago)    17m
aptos-node-76-validator-0                1/1     Running            0              17m
aptos-node-77-haproxy-7b98cb457b-6flp9   0/1     CrashLoopBackOff   8 (68s ago)    17m
aptos-node-77-validator-0                1/1     Running            0              17m
aptos-node-78-haproxy-5947f86657-whtmb   0/1     CrashLoopBackOff   8 (82s ago)    17m
aptos-node-78-validator-0                1/1     Running            0              17m
aptos-node-79-haproxy-6f4974b48-b4n5t    0/1     CrashLoopBackOff   8 (95s ago)    17m
aptos-node-79-validator-0                1/1     Running            0              17m
aptos-node-8-fullnode-eforge87-0         1/1     Running            0              17m
aptos-node-8-haproxy-7979485858-f6zjn    0/1     CrashLoopBackOff   8 (94s ago)    17m
aptos-node-8-validator-0                 1/1     Running            0              17m
aptos-node-80-haproxy-68bc9b6977-ffsnd   0/1     CrashLoopBackOff   8 (74s ago)    17m
aptos-node-80-validator-0                1/1     Running            0              17m
aptos-node-81-haproxy-65b645656f-g7pnb   0/1     CrashLoopBackOff   8 (61s ago)    17m
aptos-node-81-validator-0                1/1     Running            0              17m
aptos-node-82-haproxy-6f6846b5bd-ks26k   0/1     CrashLoopBackOff   8 (89s ago)    17m
aptos-node-82-validator-0                1/1     Running            0              17m
aptos-node-83-haproxy-67846c7db5-v6k5v   0/1     CrashLoopBackOff   8 (76s ago)    17m
aptos-node-83-validator-0                1/1     Running            0              17m
aptos-node-84-haproxy-cff7cbb66-mrn6j    0/1     CrashLoopBackOff   8 (71s ago)    17m
aptos-node-84-validator-0                1/1     Running            0              17m
aptos-node-85-haproxy-6487649946-q7vmw   0/1     CrashLoopBackOff   8 (80s ago)    17m
aptos-node-85-validator-0                1/1     Running            0              17m
aptos-node-86-haproxy-5756574b4c-cv5jj   0/1     CrashLoopBackOff   8 (70s ago)    17m
aptos-node-86-validator-0                1/1     Running            0              17m
aptos-node-87-haproxy-75cf5f8dfc-jhfs9   0/1     CrashLoopBackOff   8 (76s ago)    17m
aptos-node-87-validator-0                1/1     Running            0              17m
aptos-node-88-haproxy-689569fb6f-jkkd2   0/1     CrashLoopBackOff   8 (86s ago)    17m
aptos-node-88-validator-0                1/1     Running            0              17m
aptos-node-89-haproxy-6c7b56dff5-5rx5k   0/1     CrashLoopBackOff   8 (64s ago)    17m
aptos-node-89-validator-0                1/1     Running            0              17m
aptos-node-9-fullnode-eforge87-0         1/1     Running            0              17m
aptos-node-9-haproxy-76799fc4f-jnm59     0/1     CrashLoopBackOff   8 (78s ago)    17m
aptos-node-9-validator-0                 1/1     Running            0              17m
aptos-node-90-haproxy-97bf45d7f-tnrjk    0/1     CrashLoopBackOff   8 (95s ago)    17m
aptos-node-90-validator-0                1/1     Running            0              17m
aptos-node-91-haproxy-776857954f-lxgm9   0/1     CrashLoopBackOff   8 (86s ago)    17m
aptos-node-91-validator-0                1/1     Running            0              17m
aptos-node-92-haproxy-64956db954-dgxxj   0/1     CrashLoopBackOff   8 (78s ago)    17m
aptos-node-92-validator-0                1/1     Running            0              17m
aptos-node-93-haproxy-69b8bb8745-mbhff   0/1     CrashLoopBackOff   8 (98s ago)    17m
aptos-node-93-validator-0                1/1     Running            0              17m
aptos-node-94-haproxy-86bbbd87df-z5hbp   0/1     CrashLoopBackOff   8 (97s ago)    17m
aptos-node-94-validator-0                1/1     Running            0              17m
aptos-node-95-haproxy-75b4965cb7-x5lxc   0/1     CrashLoopBackOff   8 (90s ago)    17m
aptos-node-95-validator-0                1/1     Running            0              17m
aptos-node-96-haproxy-6c46dbcf78-svvs5   0/1     CrashLoopBackOff   8 (68s ago)    17m
aptos-node-96-validator-0                1/1     Running            0              17m
aptos-node-97-haproxy-697d66dfb6-xvkmr   0/1     CrashLoopBackOff   8 (81s ago)    17m
aptos-node-97-validator-0                1/1     Running            0              17m
aptos-node-98-haproxy-fd6ddb569-jzxjl    0/1     CrashLoopBackOff   8 (88s ago)    17m
aptos-node-98-validator-0                1/1     Running            0              17m
aptos-node-99-haproxy-7ccdcc8fbb-g5jcg   0/1     CrashLoopBackOff   8 (86s ago)    17m
aptos-node-99-validator-0                1/1     Running            0              17m
genesis-aptos-genesis-eforge87-7pqlj     0/1     Completed          0              21m

@github-actions
Copy link
Contributor

❌ Forge suite compat failure on 843b204dce971d98449b82624f4f684c7a18b991 ==> 5220051aab7fa746f715c8cf90e1d88c11a4607d

Forge test runner terminated:
Trailing Log Lines:
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:120"},"thread_name":"main","hostname":"forge-compat-pr-4650-1664569589-843b204dce971d98449b82624f4f684","timestamp":"2022-09-30T20:49:54.731835Z","message":"Deployment aptos-node-0-haproxy has no status"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:111"},"thread_name":"main","hostname":"forge-compat-pr-4650-1664569589-843b204dce971d98449b82624f4f684","timestamp":"2022-09-30T20:50:04.739792Z","message":"Deployment aptos-node-0-haproxy has 0 ready_replicas"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:120"},"thread_name":"main","hostname":"forge-compat-pr-4650-1664569589-843b204dce971d98449b82624f4f684","timestamp":"2022-09-30T20:50:04.739811Z","message":"Deployment aptos-node-0-haproxy has no status"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:111"},"thread_name":"main","hostname":"forge-compat-pr-4650-1664569589-843b204dce971d98449b82624f4f684","timestamp":"2022-09-30T20:50:14.748983Z","message":"Deployment aptos-node-0-haproxy has 0 ready_replicas"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:120"},"thread_name":"main","hostname":"forge-compat-pr-4650-1664569589-843b204dce971d98449b82624f4f684","timestamp":"2022-09-30T20:50:14.749007Z","message":"Deployment aptos-node-0-haproxy has no status"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:280"},"thread_name":"main","hostname":"forge-compat-pr-4650-1664569589-843b204dce971d98449b82624f4f684","timestamp":"2022-09-30T20:50:14.768688Z","message":"Deleting namespace forge-compat-pr-4650: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:388"},"thread_name":"main","hostname":"forge-compat-pr-4650-1664569589-843b204dce971d98449b82624f4f684","timestamp":"2022-09-30T20:50:14.768717Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-4650"}
Failed to run tests:
Deployment not ready
Error: Deployment not ready
Debugging output:
NAME                                     READY   STATUS             RESTARTS      AGE
aptos-node-0-haproxy-7f548fc4c7-85xj8    0/1     Pending            0             20m
aptos-node-0-validator-0                 1/1     Running            0             20m
aptos-node-1-haproxy-76779b4bbb-7cfzx    0/1     Pending            0             20m
aptos-node-1-validator-0                 1/1     Running            0             20m
aptos-node-10-haproxy-56cd759f6-gcft5    0/1     Pending            0             20m
aptos-node-10-validator-0                1/1     Running            0             20m
aptos-node-11-haproxy-5497f8b8f8-88gh4   0/1     Pending            0             20m
aptos-node-11-validator-0                1/1     Running            0             20m
aptos-node-12-haproxy-c6c55d7ff-gn7rl    0/1     Pending            0             20m
aptos-node-12-validator-0                1/1     Running            0             20m
aptos-node-13-haproxy-bdf58d6b4-gqlzr    0/1     Pending            0             20m
aptos-node-13-validator-0                1/1     Running            0             20m
aptos-node-14-haproxy-5b74675b5d-l6f5b   0/1     Pending            0             20m
aptos-node-14-validator-0                1/1     Running            0             20m
aptos-node-15-haproxy-5889bffbb8-gghzf   0/1     Pending            0             20m
aptos-node-15-validator-0                1/1     Running            0             20m
aptos-node-16-haproxy-6d4f9f74d9-k5r9c   0/1     Pending            0             20m
aptos-node-16-validator-0                1/1     Running            0             20m
aptos-node-17-haproxy-7dd49d57f9-gk2wj   0/1     Pending            0             20m
aptos-node-17-validator-0                1/1     Running            0             20m
aptos-node-18-haproxy-7b897d9b99-h97x2   0/1     Pending            0             20m
aptos-node-18-validator-0                1/1     Running            0             20m
aptos-node-19-haproxy-57dcfdb89d-c8mxb   0/1     Pending            0             20m
aptos-node-19-validator-0                1/1     Running            0             20m
aptos-node-2-haproxy-85cfbc7db9-ckqhs    0/1     Pending            0             20m
aptos-node-2-validator-0                 1/1     Running            0             20m
aptos-node-20-haproxy-77c55cfd97-fw778   0/1     Pending            0             20m
aptos-node-20-validator-0                1/1     Running            0             20m
aptos-node-21-haproxy-5844789f79-5fx8k   0/1     Pending            0             20m
aptos-node-21-validator-0                1/1     Running            0             20m
aptos-node-22-haproxy-7d475fbbb8-jb5pw   0/1     CrashLoopBackOff   2 (5s ago)    20m
aptos-node-22-validator-0                1/1     Running            0             20m
aptos-node-23-haproxy-5896c89687-5lx4h   0/1     Pending            0             20m
aptos-node-23-validator-0                1/1     Running            0             20m
aptos-node-24-haproxy-7d9d85c5c8-vnjfs   0/1     Pending            0             20m
aptos-node-24-validator-0                1/1     Running            0             20m
aptos-node-25-haproxy-5f66fd75b6-96nwq   0/1     Pending            0             20m
aptos-node-25-validator-0                1/1     Running            0             20m
aptos-node-26-haproxy-6985cb7699-7njwk   0/1     Pending            0             20m
aptos-node-26-validator-0                1/1     Running            0             20m
aptos-node-27-haproxy-558849ff5-cqw7v    0/1     Pending            0             20m
aptos-node-27-validator-0                1/1     Running            0             20m
aptos-node-28-haproxy-7f8df87f4-jnzb8    0/1     Pending            0             20m
aptos-node-28-validator-0                1/1     Running            0             20m
aptos-node-29-haproxy-5f5c968f48-swjwc   0/1     Pending            0             20m
aptos-node-29-validator-0                1/1     Running            0             20m
aptos-node-3-haproxy-6945fc48cf-2f58h    0/1     Pending            0             20m
aptos-node-3-validator-0                 1/1     Running            0             20m
aptos-node-30-haproxy-694dc66d59-b78mh   0/1     Pending            0             20m
aptos-node-30-validator-0                1/1     Running            0             20m
aptos-node-31-haproxy-5445b5b856-vbrxv   0/1     Pending            0             20m
aptos-node-31-validator-0                1/1     Running            0             20m
aptos-node-32-haproxy-55bdcc7d8f-8rnq2   0/1     Pending            0             20m
aptos-node-32-validator-0                1/1     Running            0             20m
aptos-node-33-haproxy-5484f8d7dc-d2742   0/1     Pending            0             20m
aptos-node-33-validator-0                1/1     Running            0             20m
aptos-node-34-haproxy-7ccb4dfd4-9n54b    0/1     Pending            0             20m
aptos-node-34-validator-0                1/1     Running            0             20m
aptos-node-35-haproxy-76dd98744b-x4xnv   0/1     CrashLoopBackOff   1 (20s ago)   20m
aptos-node-35-validator-0                1/1     Running            0             20m
aptos-node-36-haproxy-85864c9c9b-m22fw   0/1     Pending            0             20m
aptos-node-36-validator-0                1/1     Running            0             20m
aptos-node-37-haproxy-67bfb55cfc-4lgqs   0/1     Pending            0             20m
aptos-node-37-validator-0                1/1     Running            0             20m
aptos-node-38-haproxy-849c665c67-6h2cc   0/1     Pending            0             20m
aptos-node-38-validator-0                1/1     Running            0             20m
aptos-node-39-haproxy-7457b6f4bc-4hqhg   0/1     Pending            0             20m
aptos-node-39-validator-0                1/1     Running            0             20m
aptos-node-4-haproxy-677c6749c4-nmszj    0/1     Pending            0             20m
aptos-node-4-validator-0                 1/1     Running            0             20m
aptos-node-40-haproxy-668f5c6f4b-jxhwh   0/1     Pending            0             20m
aptos-node-40-validator-0                1/1     Running            0             20m
aptos-node-41-haproxy-d99b459d6-p9fhs    0/1     Pending            0             20m
aptos-node-41-validator-0                1/1     Running            0             20m
aptos-node-42-haproxy-64db56cfd8-ndvgk   0/1     Pending            0             20m
aptos-node-42-validator-0                1/1     Running            0             20m
aptos-node-43-haproxy-95768c477-wl6nr    0/1     Pending            0             20m
aptos-node-43-validator-0                1/1     Running            0             20m
aptos-node-44-haproxy-85bfb5fc64-hsvx8   0/1     Pending            0             20m
aptos-node-44-validator-0                1/1     Running            0             20m
aptos-node-45-haproxy-6dccfbc4c4-ckbk2   0/1     Pending            0             20m
aptos-node-45-validator-0                1/1     Running            0             20m
aptos-node-46-haproxy-7dc5959d8b-p9287   0/1     Pending            0             20m
aptos-node-46-validator-0                1/1     Running            0             20m
aptos-node-47-haproxy-594bfc4764-42vpt   0/1     Pending            0             20m
aptos-node-47-validator-0                1/1     Running            0             20m
aptos-node-48-haproxy-86dcbcf7c9-5hmj5   0/1     Pending            0             20m
aptos-node-48-validator-0                1/1     Running            0             20m
aptos-node-49-haproxy-6f55c6bbdd-zpwxx   0/1     Pending            0             20m
aptos-node-49-validator-0                1/1     Running            0             20m
aptos-node-5-haproxy-7c57d96d76-tlzk8    0/1     Pending            0             20m
aptos-node-5-validator-0                 1/1     Running            0             20m
aptos-node-50-haproxy-85f9ff5d5d-67bbp   0/1     Pending            0             20m
aptos-node-50-validator-0                1/1     Running            0             20m
aptos-node-51-haproxy-5f88475df7-l4p4p   0/1     Pending            0             20m
aptos-node-51-validator-0                1/1     Running            0             20m
aptos-node-52-haproxy-84ff945b89-hv7tq   0/1     Pending            0             20m
aptos-node-52-validator-0                1/1     Running            0             20m
aptos-node-53-haproxy-d87d94b98-ntfwd    0/1     CrashLoopBackOff   2 (11s ago)   20m
aptos-node-53-validator-0                1/1     Running            0             20m
aptos-node-54-haproxy-5d88c7dc48-98cpw   0/1     Pending            0             20m
aptos-node-54-validator-0                1/1     Running            0             20m
aptos-node-55-haproxy-56878f6b4-l9r9v    0/1     Pending            0             20m
aptos-node-55-validator-0                1/1     Running            0             20m
aptos-node-56-haproxy-5bfc7c579b-7rl48   0/1     Pending            0             20m
aptos-node-56-validator-0                1/1     Running            0             20m
aptos-node-57-haproxy-6b7bc7cf58-94wxv   0/1     Pending            0             20m
aptos-node-57-validator-0                1/1     Running            0             20m
aptos-node-58-haproxy-7cb4c85789-9ntxt   0/1     Pending            0             20m
aptos-node-58-validator-0                1/1     Running            0             20m
aptos-node-59-haproxy-54fdc5b9d7-dl2kh   0/1     Pending            0             20m
aptos-node-59-validator-0                1/1     Running            0             20m
aptos-node-6-haproxy-c5d878ffb-5j9n6     0/1     Pending            0             20m
aptos-node-6-validator-0                 1/1     Running            0             20m
aptos-node-60-haproxy-745fb4f75b-jllct   0/1     Pending            0             20m
aptos-node-60-validator-0                1/1     Running            0             20m
aptos-node-61-haproxy-7f89859c4b-gffl2   0/1     Pending            0             20m
aptos-node-61-validator-0                1/1     Running            0             20m
aptos-node-62-haproxy-db5c8d585-tqq86    0/1     Pending            0             20m
aptos-node-62-validator-0                1/1     Running            0             20m
aptos-node-63-haproxy-6f4457489b-7wslq   0/1     Pending            0             20m
aptos-node-63-validator-0                1/1     Running            0             20m
aptos-node-64-haproxy-f898bffb7-txv97    0/1     Pending            0             20m
aptos-node-64-validator-0                1/1     Running            0             20m
aptos-node-65-haproxy-6b6c9cfff5-rr4xp   0/1     Pending            0             20m
aptos-node-65-validator-0                1/1     Running            0             20m
aptos-node-66-haproxy-5864588f9c-hvkx5   0/1     Pending            0             20m
aptos-node-66-validator-0                1/1     Running            0             20m
aptos-node-67-haproxy-54bd49c5c8-vvglf   0/1     Pending            0             20m
aptos-node-67-validator-0                1/1     Running            0             20m
aptos-node-68-haproxy-598b8fc589-f6prj   0/1     Pending            0             20m
aptos-node-68-validator-0                1/1     Running            0             20m
aptos-node-69-haproxy-689c954d9d-h4l6r   0/1     Pending            0             20m
aptos-node-69-validator-0                1/1     Running            0             20m
aptos-node-7-haproxy-68744c77dd-q77cd    0/1     Pending            0             20m
aptos-node-7-validator-0                 1/1     Running            0             20m
aptos-node-70-haproxy-54dc955d8c-vsf9k   0/1     Pending            0             20m
aptos-node-70-validator-0                1/1     Running            0             20m
aptos-node-71-haproxy-d496cfdc8-t9tvk    0/1     Pending            0             20m
aptos-node-71-validator-0                1/1     Running            0             20m
aptos-node-72-haproxy-5b7cc66f89-h8krm   0/1     Pending            0             20m
aptos-node-72-validator-0                1/1     Running            0             20m
aptos-node-73-haproxy-588db6d94c-zs2pj   0/1     Pending            0             20m
aptos-node-73-validator-0                1/1     Running            0             20m
aptos-node-74-haproxy-846f4ddc74-2gf5w   0/1     Pending            0             20m
aptos-node-74-validator-0                1/1     Running            0             20m
aptos-node-75-haproxy-6dc656cf85-wv5cg   0/1     Pending            0             20m
aptos-node-75-validator-0                1/1     Running            0             20m
aptos-node-76-haproxy-549cb7697c-ftwz6   0/1     Pending            0             20m
aptos-node-76-validator-0                1/1     Running            0             20m
aptos-node-77-haproxy-79cdd59988-5b9dm   0/1     Pending            0             20m
aptos-node-77-validator-0                1/1     Running            0             20m
aptos-node-78-haproxy-7d4c678b87-qm2t4   0/1     Pending            0             20m
aptos-node-78-validator-0                1/1     Running            0             20m
aptos-node-79-haproxy-67f6f9c968-rct85   0/1     Pending            0             20m
aptos-node-79-validator-0                1/1     Running            0             20m
aptos-node-8-haproxy-658bdd6465-5s8kg    0/1     Pending            0             20m
aptos-node-8-validator-0                 1/1     Running            0             20m
aptos-node-80-haproxy-7986455dcc-nbkq2   0/1     Pending            0             20m
aptos-node-80-validator-0                1/1     Running            0             20m
aptos-node-81-haproxy-6b77779bdc-gvptg   0/1     Pending            0             20m
aptos-node-81-validator-0                1/1     Running            0             20m
aptos-node-82-haproxy-7466c8945b-95n7p   0/1     Pending            0             20m
aptos-node-82-validator-0                1/1     Running            0             20m
aptos-node-83-haproxy-65c56fdc4c-87xfl   0/1     Pending            0             20m
aptos-node-83-validator-0                1/1     Running            0             20m
aptos-node-84-haproxy-8df49bddb-7752l    0/1     Pending            0             20m
aptos-node-84-validator-0                1/1     Running            0             20m
aptos-node-85-haproxy-7bc8d55fd8-dbjzw   0/1     Pending            0             20m
aptos-node-85-validator-0                1/1     Running            0             20m
aptos-node-86-haproxy-96684b59-plrcs     0/1     Pending            0             20m
aptos-node-86-validator-0                1/1     Running            0             20m
aptos-node-87-haproxy-89bc898bc-2687h    0/1     Pending            0             20m
aptos-node-87-validator-0                1/1     Running            0             20m
aptos-node-88-haproxy-9975b5977-gr8pd    0/1     Pending            0             20m
aptos-node-88-validator-0                1/1     Running            0             20m
aptos-node-89-haproxy-86456655c-rdvgc    0/1     Pending            0             20m
aptos-node-89-validator-0                1/1     Running            0             20m
aptos-node-9-haproxy-69c68c8c48-w9dt5    0/1     Pending            0             20m
aptos-node-9-validator-0                 1/1     Running            0             20m
aptos-node-90-haproxy-b86977c77-qt9ht    0/1     Pending            0             20m
aptos-node-90-validator-0                1/1     Running            0             20m
aptos-node-91-haproxy-7f9d4c8f5c-76wzq   0/1     Pending            0             20m
aptos-node-91-validator-0                1/1     Running            0             20m
aptos-node-92-haproxy-759c5c5b8-bx8d6    0/1     Pending            0             20m
aptos-node-92-validator-0                1/1     Running            0             20m
aptos-node-93-haproxy-555c66dbd9-lg7h8   0/1     Pending            0             20m
aptos-node-93-validator-0                1/1     Running            0             20m
aptos-node-94-haproxy-64f598c547-65686   0/1     Pending            0             20m
aptos-node-94-validator-0                1/1     Running            0             20m
aptos-node-95-haproxy-56bfc86b79-n7mwk   0/1     Pending            0             20m
aptos-node-95-validator-0                1/1     Running            0             20m
aptos-node-96-haproxy-9569b996d-97rfl    0/1     Pending            0             20m
aptos-node-96-validator-0                1/1     Running            0             20m
aptos-node-97-haproxy-78bb6f494b-z848s   0/1     Pending            0             20m
aptos-node-97-validator-0                1/1     Running            0             20m
aptos-node-98-haproxy-655d666b5f-74s8l   0/1     Pending            0             20m
aptos-node-98-validator-0                1/1     Running            0             20m
aptos-node-99-haproxy-7649597c78-f22q7   0/1     Pending            0             20m
aptos-node-99-validator-0                1/1     Running            0             20m
genesis-aptos-genesis-eforge141-2qhzx    0/1     Completed          0             22m

@Markuze Markuze closed this Oct 2, 2022
@davidiw davidiw deleted the markuze/haproxy_fixup_nr_connections branch October 3, 2022 07:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CICD:run-e2e-tests when this label is present github actions will run all land-blocking e2e tests from the PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant