Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Platform] Use the correct SSH user for doing on-prem node preflight checks #7268

Closed
streddy-yb opened this issue Feb 17, 2021 · 4 comments
Closed

[Platform] Use the correct SSH user for doing on-prem node preflight checks #7268

streddy-yb opened this issue Feb 17, 2021 · 4 comments
Assignees
@WesleyW
Labels
area/platform Yugabyte Platform priority/high High Priority
Milestone
2.5.x

Comments

@streddy-yb
Copy link
Contributor

Currently onprem preflight checks are always using centos user. This should be changed to whatever is configured in the OnpremProvider configuration.
@streddy-yb streddy-yb added this to the 2.5.x milestone Feb 17, 2021
@ssung-yugabyte
Copy link
Contributor

Yugaware onprem provider configuration
name: oel-bm-gcp
ssh user: yugabyte
ssh port: 22
ssh-key -> yugaware-host:/home/yugabyte/.ssh/id_rsa

LOG
[yugabyte@oel-ybware-0 ~]$ sudo docker logs yugaware
OpenJDK 64-Bit Server VM warning: ignoring option PermSize=1024m; support was removed in 8.0
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=1024m; support was removed in 8.0
2021-02-17 23:51:28.505 INFO DefaultDBApi.scala:70 [main] Database [default] initialized at jdbc:postgresql://192.168.1.2:5432/yugaware
2021-02-17 23:51:28.517 INFO HikariCPModule.scala:54 [main] Creating Pool for datasource 'default'
2021-02-17 23:51:30.883 WARN Configuration.scala:193 [main] conf/application.docker.conf: 1: play.crypto.secret is deprecated, use play.http.secret.key instead
2021-02-17 23:51:31.366 WARN ParameterMessageInterpolator.java:28 [main] HV000184: ParameterMessageInterpolator has been chosen, EL interpolation will not be supported
2021-02-17 23:51:32.209 INFO Commissioner.java:66 [main] Started Commissioner TaskPool.
2021-02-17 23:51:32.210 INFO Commissioner.java:72 [main] Started TaskProgressMonitor thread.
2021-02-17 23:51:34.215 INFO AppInit.java:51 [main] Yugaware Application has started
2021-02-17 23:51:34.703 INFO ExtraMigration.java:40 [main] Running migration 'V52__Update_Access_Key_Create_Extra_Migration'.
2021-02-17 23:51:34.707 INFO ExtraMigration.java:42 [main] Completed migration 'V52__Update_Access_Key_Create_Extra_Migration'.
2021-02-17 23:51:35.265 INFO CustomerTaskManager.java:54 [main] Failing incomplete tasks...
2021-02-17 23:51:35.273 INFO TaskGarbageCollector.java:128 [main] Scheduling TaskGC every PT24H
2021-02-17 23:51:35.277 INFO AppInit.java:136 [main] AppInit completed
2021-02-17 23:51:35.278 INFO CallHome.java:54 [main] Initialize callhome service
2021-02-17 23:51:35.279 INFO Scheduler.java:94 [main] Starting scheduling service
2021-02-17 23:51:35.280 INFO CallHome.java:75 [application-akka.actor.default-dispatcher-4] Running scheduler
2021-02-17 23:51:35.280 INFO Scheduler.java:116 [application-akka.actor.default-dispatcher-3] Running scheduler
2021-02-17 23:51:35.280 INFO HealthChecker.java:113 [main] Scheduling health checker every 300000 ms
2021-02-17 23:51:35.285 INFO HealthChecker.java:189 [application-akka.actor.default-dispatcher-5] Started running health checker
2021-02-17 23:51:35.294 INFO Play.scala:129 [main] Application started (Prod)
2021-02-17 23:51:35.344 INFO HealthChecker.java:200 [application-akka.actor.default-dispatcher-5] Completed running health checker.
2021-02-17 23:51:35.987 INFO AkkaHttpServer.scala:447 [main] Listening for HTTP on /0.0.0.0:9000
2021-02-17 23:52:11.412 INFO EncryptionAtRestController.java:111 [application-akka.actor.default-dispatcher-30] Listing KMS configurations for customer 88d5de2f-88e4-4419-8ddc-34982abcae45
2021-02-17 23:52:11.425 INFO ShellProcessHandler.java:79 [application-akka.actor.default-dispatcher-5] Starting proc (abbrev cmd) - bin/ybcloud.sh aws query current-host region
2021-02-17 23:52:12.586 INFO ShellProcessHandler.java:124 [application-akka.actor.default-dispatcher-5] Completed proc 'bin/ybcloud.sh aws query current-host region' status=success [ 1160 ms ]
2021-02-17 23:52:12.592 INFO ShellProcessHandler.java:79 [application-akka.actor.default-dispatcher-5] Starting proc (abbrev cmd) - bin/ybcloud.sh gcp query current-host
2021-02-17 23:52:13.332 INFO ShellProcessHandler.java:124 [application-akka.actor.default-dispatcher-5] Completed proc 'bin/ybcloud.sh gcp query current-host' status=success [ 740 ms ]
2021-02-17 23:53:35.297 INFO Scheduler.java:116 [application-akka.actor.default-dispatcher-36] Running scheduler
2021-02-17 23:53:37.984 INFO AccessKeyController.java:94 [application-akka.actor.default-dispatcher-37] Creating access key oel-bm-gcp-key for customer 88d5de2f-88e4-4419-8ddc-34982abcae45, provider f92b45a8-ab35-4550-902a-b98a6e03cb01.
2021-02-17 23:53:38.023 INFO ShellProcessHandler.java:79 [application-akka.actor.default-dispatcher-37] Starting proc (abbrev cmd) - bin/ybcloud.sh onprem --region central1 access create-vault /opt/yugabyte/yugaware/data/keys/f92b45a8-ab35-4550-902a-b98a6e03cb01/oel-bm-gcp-key.pem
2021-02-17 23:53:38.860 INFO ShellProcessHandler.java:124 [application-akka.actor.default-dispatcher-37] Completed proc 'bin/ybcloud.sh onprem --region central1 access create-vault /opt/yugabyte/yugaware/data/keys/f92b45a8-ab35-4550-902a-b98a6e03cb01/oel-bm-gcp-key.pem' status=success [ 837 ms ]
2021-02-17 23:53:38.917 INFO ShellProcessHandler.java:79 [application-akka.actor.default-dispatcher-37] Starting proc (abbrev cmd) - bin/ybcloud.sh onprem instance template prometheus
2021-02-17 23:53:39.763 INFO ShellProcessHandler.java:124 [application-akka.actor.default-dispatcher-37] Completed proc 'bin/ybcloud.sh onprem instance template prometheus' status=success [ 846 ms ]

yugaware add instances
zone: a,b,c
instance-type: oel-n2
region: central1
ip: 192.168.1.3, 192.168.0.2, 192.168.2.2

LOG
2021-02-17 23:55:35.296 INFO Scheduler.java:116 [application-akka.actor.default-dispatcher-37] Running scheduler
2021-02-17 23:56:21.291 INFO ShellProcessHandler.java:79 [application-akka.actor.default-dispatcher-37] Starting proc (abbrev cmd) - bin/ybcloud.sh aws query current-host region
2021-02-17 23:56:22.099 INFO ShellProcessHandler.java:124 [application-akka.actor.default-dispatcher-37] Completed proc 'bin/ybcloud.sh aws query current-host region' status=success [ 808 ms ]
2021-02-17 23:56:22.102 INFO ShellProcessHandler.java:79 [application-akka.actor.default-dispatcher-37] Starting proc (abbrev cmd) - bin/ybcloud.sh gcp query current-host
2021-02-17 23:56:22.850 INFO ShellProcessHandler.java:124 [application-akka.actor.default-dispatcher-37] Completed proc 'bin/ybcloud.sh gcp query current-host' status=success [ 748 ms ]
2021-02-17 23:56:27.309 INFO EncryptionAtRestController.java:111 [application-akka.actor.default-dispatcher-46] Listing KMS configurations for customer 88d5de2f-88e4-4419-8ddc-34982abcae45
2021-02-17 23:56:35.286 INFO HealthChecker.java:189 [application-akka.actor.default-dispatcher-68] Started running health checker
2021-02-17 23:56:35.289 INFO HealthChecker.java:208 [application-akka.actor.default-dispatcher-68] Skipping customer 88d5de2f-88e4-4419-8ddc-34982abcae45 due to missing alerting config...
2021-02-17 23:56:35.289 INFO HealthChecker.java:200 [application-akka.actor.default-dispatcher-68] Completed running health checker.
2021-02-17 23:56:48.199 INFO UniverseController.java:162 [application-akka.actor.default-dispatcher-46] Finding Universe with name ats-stan1.
2021-02-17 23:56:50.140 INFO UniverseController.java:162 [application-akka.actor.default-dispatcher-70] Finding Universe with name ats-stan1.
2021-02-17 23:56:56.179 INFO ShellProcessHandler.java:79 [application-akka.actor.default-dispatcher-37] Starting proc (abbrev cmd) - bin/ybcloud.sh aws query current-host region
2021-02-17 23:56:56.194 INFO EncryptionAtRestController.java:111 [application-akka.actor.default-dispatcher-47] Listing KMS configurations for customer 88d5de2f-88e4-4419-8ddc-34982abcae45
2021-02-17 23:56:56.897 INFO ShellProcessHandler.java:124 [application-akka.actor.default-dispatcher-37] Completed proc 'bin/ybcloud.sh aws query current-host region' status=success [ 718 ms ]
2021-02-17 23:56:56.915 INFO ShellProcessHandler.java:79 [application-akka.actor.default-dispatcher-37] Starting proc (abbrev cmd) - bin/ybcloud.sh gcp query current-host
2021-02-17 23:56:57.654 INFO ShellProcessHandler.java:124 [application-akka.actor.default-dispatcher-37] Completed proc 'bin/ybcloud.sh gcp query current-host' status=success [ 739 ms ]
2021-02-17 23:57:35.287 INFO Scheduler.java:116 [application-akka.actor.default-dispatcher-37] Running scheduler

yugaware create universe
name: ats-stan1
provider: oel-bm-gcp
regions: central1
nodes 3, RF 3

LOG
2021-02-17 23:59:35.288 INFO Scheduler.java:116 [application-akka.actor.default-dispatcher-47] Running scheduler
2021-02-18 00:00:18.428 INFO EncryptionAtRestController.java:111 [application-akka.actor.default-dispatcher-83] Listing KMS configurations for customer 88d5de2f-88e4-4419-8ddc-34982abcae45
2021-02-18 00:00:23.785 INFO UniverseController.java:162 [application-akka.actor.default-dispatcher-84] Finding Universe with name ats-stan1.
2021-02-18 00:00:25.243 INFO UniverseController.java:162 [application-akka.actor.default-dispatcher-84] Finding Universe with name ats-stan1.
2021-02-18 00:00:26.277 INFO PlacementInfoUtil.java:1917 [application-akka.actor.default-dispatcher-83] numRegions=1, numAzsInRegions=3, zonesIntended=3
2021-02-18 00:00:26.284 INFO PlacementInfoUtil.java:2020 [application-akka.actor.default-dispatcher-83] Incrementing RF for c to: 1
2021-02-18 00:00:26.285 INFO PlacementInfoUtil.java:2022 [application-akka.actor.default-dispatcher-83] Number of nodes in c: 1
2021-02-18 00:00:26.288 INFO PlacementInfoUtil.java:2020 [application-akka.actor.default-dispatcher-83] Incrementing RF for b to: 1
2021-02-18 00:00:26.288 INFO PlacementInfoUtil.java:2022 [application-akka.actor.default-dispatcher-83] Number of nodes in b: 1
2021-02-18 00:00:26.291 INFO PlacementInfoUtil.java:2020 [application-akka.actor.default-dispatcher-83] Incrementing RF for a to: 1
2021-02-18 00:00:26.291 INFO PlacementInfoUtil.java:2022 [application-akka.actor.default-dispatcher-83] Number of nodes in a: 1
2021-02-18 00:00:26.291 INFO PlacementInfoUtil.java:416 [application-akka.actor.default-dispatcher-83] Placement created=Cloud=onprem Region=central1 : (AZ=c, count=1, replication factor=1)(AZ=b, count=1, replication factor=1)(AZ=a, count=1, replication factor=1).
2021-02-18 00:00:26.300 INFO PlacementInfoUtil.java:977 [application-akka.actor.default-dispatcher-83] Adding 0/0/0 @ 0.
2021-02-18 00:00:26.302 INFO PlacementInfoUtil.java:977 [application-akka.actor.default-dispatcher-83] Adding 1/0/0 @ 1.
2021-02-18 00:00:26.303 INFO PlacementInfoUtil.java:977 [application-akka.actor.default-dispatcher-83] Adding 2/0/0 @ 2.
2021-02-18 00:00:26.304 INFO PlacementInfoUtil.java:1002 [application-akka.actor.default-dispatcher-83] Base placement indexes [[0:0:0:ADD], [0:0:1:ADD], [0:0:2:ADD]] for 3 nodes.
2021-02-18 00:00:26.306 INFO PlacementInfoUtil.java:1053 [application-akka.actor.default-dispatcher-83] Az Map {47249714-7f59-4196-add9-2b59a5a77032=1, 428696af-c72b-4349-9fe6-534243213062=1, 99d00076-eed7-4e97-a253-c1340d198205=1}
2021-02-18 00:00:26.306 INFO PlacementInfoUtil.java:660 [application-akka.actor.default-dispatcher-83] Update c 1 1.
2021-02-18 00:00:26.306 INFO PlacementInfoUtil.java:660 [application-akka.actor.default-dispatcher-83] Update b 1 1.
2021-02-18 00:00:26.306 INFO PlacementInfoUtil.java:660 [application-akka.actor.default-dispatcher-83] Update a 1 1.
2021-02-18 00:00:26.306 INFO PlacementInfoUtil.java:1458 [application-akka.actor.default-dispatcher-83] Set of nodes after node configure: [name: null, cloudInfo: c.central1.onprem, type: oel-n2, ip: null, isMaster: false, isTserver: true, state: ToBeAdded, azUuid: 99d00076-eed7-4e97-a253-c1340d198205, placementUuid: 951c934b-b104-4a8b-a57f-798849bb2dd0, name: null, cloudInfo: a.central1.onprem, type: oel-n2, ip: null, isMaster: false, isTserver: true, state: ToBeAdded, azUuid: 428696af-c72b-4349-9fe6-534243213062, placementUuid: 951c934b-b104-4a8b-a57f-798849bb2dd0, name: null, cloudInfo: b.central1.onprem, type: oel-n2, ip: null, isMaster: false, isTserver: true, state: ToBeAdded, azUuid: 47249714-7f59-4196-add9-2b59a5a77032, placementUuid: 951c934b-b104-4a8b-a57f-798849bb2dd0].
2021-02-18 00:00:26.307 INFO PlacementInfoUtil.java:556 [application-akka.actor.default-dispatcher-83] Setting per AZ replication factor.
2021-02-18 00:00:26.312 INFO PlacementInfoUtil.java:1460 [application-akka.actor.default-dispatcher-83] Final Placement info: Cloud=onprem Region=central1 : (AZ=c, count=1, replication factor=1)(AZ=b, count=1, replication factor=1)(AZ=a, count=1, replication factor=1).
2021-02-18 00:00:26.326 INFO PlacementInfoUtil.java:1917 [application-akka.actor.default-dispatcher-88] numRegions=1, numAzsInRegions=3, zonesIntended=3
2021-02-18 00:00:26.343 INFO PlacementInfoUtil.java:2020 [application-akka.actor.default-dispatcher-88] Incrementing RF for c to: 1
2021-02-18 00:00:26.343 INFO PlacementInfoUtil.java:2022 [application-akka.actor.default-dispatcher-88] Number of nodes in c: 1
2021-02-18 00:00:26.345 INFO PlacementInfoUtil.java:2020 [application-akka.actor.default-dispatcher-88] Incrementing RF for b to: 1
2021-02-18 00:00:26.346 INFO PlacementInfoUtil.java:2022 [application-akka.actor.default-dispatcher-88] Number of nodes in b: 1
2021-02-18 00:00:26.348 INFO PlacementInfoUtil.java:2020 [application-akka.actor.default-dispatcher-88] Incrementing RF for a to: 1
2021-02-18 00:00:26.348 INFO PlacementInfoUtil.java:2022 [application-akka.actor.default-dispatcher-88] Number of nodes in a: 1
2021-02-18 00:00:26.348 INFO PlacementInfoUtil.java:416 [application-akka.actor.default-dispatcher-88] Placement created=Cloud=onprem Region=central1 : (AZ=c, count=1, replication factor=1)(AZ=b, count=1, replication factor=1)(AZ=a, count=1, replication factor=1).
2021-02-18 00:00:26.355 INFO PlacementInfoUtil.java:977 [application-akka.actor.default-dispatcher-88] Adding 0/0/0 @ 0.
2021-02-18 00:00:26.356 INFO PlacementInfoUtil.java:977 [application-akka.actor.default-dispatcher-88] Adding 1/0/0 @ 1.
2021-02-18 00:00:26.358 INFO PlacementInfoUtil.java:977 [application-akka.actor.default-dispatcher-88] Adding 2/0/0 @ 2.
2021-02-18 00:00:26.358 INFO PlacementInfoUtil.java:1002 [application-akka.actor.default-dispatcher-88] Base placement indexes [[0:0:0:ADD], [0:0:1:ADD], [0:0:2:ADD]] for 3 nodes.
2021-02-18 00:00:26.358 INFO PlacementInfoUtil.java:1053 [application-akka.actor.default-dispatcher-88] Az Map {47249714-7f59-4196-add9-2b59a5a77032=1, 428696af-c72b-4349-9fe6-534243213062=1, 99d00076-eed7-4e97-a253-c1340d198205=1}
2021-02-18 00:00:26.359 INFO PlacementInfoUtil.java:660 [application-akka.actor.default-dispatcher-88] Update c 1 1.
2021-02-18 00:00:26.359 INFO PlacementInfoUtil.java:660 [application-akka.actor.default-dispatcher-88] Update b 1 1.
2021-02-18 00:00:26.359 INFO PlacementInfoUtil.java:660 [application-akka.actor.default-dispatcher-88] Update a 1 1.
2021-02-18 00:00:26.359 INFO PlacementInfoUtil.java:1458 [application-akka.actor.default-dispatcher-88] Set of nodes after node configure: [name: null, cloudInfo: c.central1.onprem, type: oel-n2, ip: null, isMaster: false, isTserver: true, state: ToBeAdded, azUuid: 99d00076-eed7-4e97-a253-c1340d198205, placementUuid: b748ec2b-b48e-4df0-8394-544165c4cc72, name: null, cloudInfo: b.central1.onprem, type: oel-n2, ip: null, isMaster: false, isTserver: true, state: ToBeAdded, azUuid: 47249714-7f59-4196-add9-2b59a5a77032, placementUuid: b748ec2b-b48e-4df0-8394-544165c4cc72, name: null, cloudInfo: a.central1.onprem, type: oel-n2, ip: null, isMaster: false, isTserver: true, state: ToBeAdded, azUuid: 428696af-c72b-4349-9fe6-534243213062, placementUuid: b748ec2b-b48e-4df0-8394-544165c4cc72].
2021-02-18 00:00:26.359 INFO PlacementInfoUtil.java:556 [application-akka.actor.default-dispatcher-88] Setting per AZ replication factor.
2021-02-18 00:00:26.359 INFO PlacementInfoUtil.java:1460 [application-akka.actor.default-dispatcher-88] Final Placement info: Cloud=onprem Region=central1 : (AZ=c, count=1, replication factor=1)(AZ=b, count=1, replication factor=1)(AZ=a, count=1, replication factor=1).
2021-02-18 00:01:03.374 INFO UniverseController.java:162 [application-akka.actor.default-dispatcher-85] Finding Universe with name ats-stan1.
2021-02-18 00:01:03.458 INFO UniverseController.java:463 [application-akka.actor.default-dispatcher-85] Create for 88d5de2f-88e4-4419-8ddc-34982abcae45.
2021-02-18 00:01:03.492 INFO PlacementInfoUtil.java:1053 [application-akka.actor.default-dispatcher-85] Az Map {47249714-7f59-4196-add9-2b59a5a77032=1, 428696af-c72b-4349-9fe6-534243213062=1, 99d00076-eed7-4e97-a253-c1340d198205=1}
2021-02-18 00:01:03.493 INFO PlacementInfoUtil.java:660 [application-akka.actor.default-dispatcher-85] Update c 1 1.
2021-02-18 00:01:03.493 INFO PlacementInfoUtil.java:660 [application-akka.actor.default-dispatcher-85] Update b 1 1.
2021-02-18 00:01:03.493 INFO PlacementInfoUtil.java:660 [application-akka.actor.default-dispatcher-85] Update a 1 1.
2021-02-18 00:01:03.495 INFO Universe.java:251 [application-akka.actor.default-dispatcher-85] Created db entry for universe ats-stan1 [56a7dfce-4ced-4010-81d9-cb69977d6a82]
2021-02-18 00:01:03.533 INFO UniverseController.java:518 [application-akka.actor.default-dispatcher-85] Created universe 56a7dfce-4ced-4010-81d9-cb69977d6a82 : ats-stan1.
2021-02-18 00:01:03.551 INFO UniverseController.java:524 [application-akka.actor.default-dispatcher-85] Added universe 56a7dfce-4ced-4010-81d9-cb69977d6a82 : ats-stan1 for customer [1].
2021-02-18 00:01:03.551 INFO Universe.java:102 [application-akka.actor.default-dispatcher-85] Setting config on universe ats-stan1 [ 56a7dfce-4ced-4010-81d9-cb69977d6a82 ]
2021-02-18 00:01:03.650 INFO TaskRunner.java:77 [application-akka.actor.default-dispatcher-85] Created task, details: task-info {taskType : CreateUniverse, taskState: Created}, task {CreateUniverse(56a7dfce-4ced-4010-81d9-cb69977d6a82)}
2021-02-18 00:01:03.654 INFO UniverseController.java:605 [application-akka.actor.default-dispatcher-85] Submitted create universe for 56a7dfce-4ced-4010-81d9-cb69977d6a82:ats-stan1, task uuid = b90e9be7-78ab-4a72-ab3b-eabfc8e13893.
2021-02-18 00:01:03.655 INFO TaskRunner.java:179 [TaskPool-0] Updating task [taskType : CreateUniverse, taskState: Created] to new state Running
2021-02-18 00:01:03.661 INFO CreateUniverse.java:37 [TaskPool-0] Started CreateUniverse(56a7dfce-4ced-4010-81d9-cb69977d6a82) task.
2021-02-18 00:01:03.677 INFO UniverseController.java:615 [application-akka.actor.default-dispatcher-85] Saved task uuid b90e9be7-78ab-4a72-ab3b-eabfc8e13893 in customer tasks table for universe 56a7dfce-4ced-4010-81d9-cb69977d6a82:ats-stan1
2021-02-18 00:01:03.742 INFO UniverseDefinitionTaskBase.java:275 [TaskPool-0] Node name yb-dev-ats-stan1-n1 at index 1
2021-02-18 00:01:03.742 INFO UniverseDefinitionTaskBase.java:275 [TaskPool-0] Node name yb-dev-ats-stan1-n2 at index 2
2021-02-18 00:01:03.742 INFO UniverseDefinitionTaskBase.java:275 [TaskPool-0] Node name yb-dev-ats-stan1-n3 at index 3
2021-02-18 00:01:03.743 INFO UniverseDefinitionTaskBase.java:370 [TaskPool-0] Current active master count = 0
2021-02-18 00:01:03.769 INFO UniverseDefinitionTaskBase.java:331 [TaskPool-0] Selecting prem nodes for universe ats-stan1 (56a7dfce-4ced-4010-81d9-cb69977d6a82).
2021-02-18 00:01:03.780 INFO NodeInstance.java:140 [TaskPool-0] Marking node yb-dev-ats-stan1-n3 (ip 192.168.1.3) as in-use.
2021-02-18 00:01:03.786 INFO NodeInstance.java:140 [TaskPool-0] Marking node yb-dev-ats-stan1-n1 (ip 192.168.0.2) as in-use.
2021-02-18 00:01:03.792 INFO NodeInstance.java:140 [TaskPool-0] Marking node yb-dev-ats-stan1-n2 (ip 192.168.2.2) as in-use.
2021-02-18 00:01:03.822 INFO SubTaskGroup.java:116 [TaskPool-0] Adding task #0: PrecheckNode(56a7dfce-4ced-4010-81d9-cb69977d6a82)(56a7dfce-4ced-4010-81d9-cb69977d6a82, yb-dev-ats-stan1-n1)
2021-02-18 00:01:03.849 INFO SubTaskGroup.java:116 [TaskPool-0] Adding task #1: PrecheckNode(56a7dfce-4ced-4010-81d9-cb69977d6a82)(56a7dfce-4ced-4010-81d9-cb69977d6a82, yb-dev-ats-stan1-n2)
2021-02-18 00:01:03.867 INFO SubTaskGroup.java:116 [TaskPool-0] Adding task #2: PrecheckNode(56a7dfce-4ced-4010-81d9-cb69977d6a82)(56a7dfce-4ced-4010-81d9-cb69977d6a82, yb-dev-ats-stan1-n3)
2021-02-18 00:01:03.918 INFO SubTaskGroup.java:116 [TaskPool-0] Adding task #0: AnsibleSetupServer(56a7dfce-4ced-4010-81d9-cb69977d6a82)(56a7dfce-4ced-4010-81d9-cb69977d6a82, yb-dev-ats-stan1-n1)
2021-02-18 00:01:04.056 INFO SubTaskGroup.java:116 [TaskPool-0] Adding task #1: AnsibleSetupServer(56a7dfce-4ced-4010-81d9-cb69977d6a82)(56a7dfce-4ced-4010-81d9-cb69977d6a82, yb-dev-ats-stan1-n2)
2021-02-18 00:01:04.095 INFO SubTaskGroup.java:116 [TaskPool-0] Adding task #2: AnsibleSetupServer(56a7dfce-4ced-4010-81d9-cb69977d6a82)(56a7dfce-4ced-4010-81d9-cb69977d6a82, yb-dev-ats-stan1-n3)
2021-02-18 00:01:04.144 INFO SubTaskGroup.java:116 [TaskPool-0] Adding task #0: AnsibleUpdateNodeInfo(56a7dfce-4ced-4010-81d9-cb69977d6a82)(56a7dfce-4ced-4010-81d9-cb69977d6a82, yb-dev-ats-stan1-n1)
2021-02-18 00:01:04.179 INFO SubTaskGroup.java:116 [TaskPool-0] Adding task #1: AnsibleUpdateNodeInfo(56a7dfce-4ced-4010-81d9-cb69977d6a82)(56a7dfce-4ced-4010-81d9-cb69977d6a82, yb-dev-ats-stan1-n2)
2021-02-18 00:01:04.198 INFO SubTaskGroup.java:116 [TaskPool-0] Adding task #2: AnsibleUpdateNodeInfo(56a7dfce-4ced-4010-81d9-cb69977d6a82)(56a7dfce-4ced-4010-81d9-cb69977d6a82, yb-dev-ats-stan1-n3)
2021-02-18 00:01:04.286 INFO SubTaskGroup.java:116 [TaskPool-0] Adding task #0: AnsibleConfigureServers(56a7dfce-4ced-4010-81d9-cb69977d6a82)(56a7dfce-4ced-4010-81d9-cb69977d6a82, yb-dev-ats-stan1-n1)
2021-02-18 00:01:04.347 INFO SubTaskGroup.java:116 [TaskPool-0] Adding task #1: AnsibleConfigureServers(56a7dfce-4ced-4010-81d9-cb69977d6a82)(56a7dfce-4ced-4010-81d9-cb69977d6a82, yb-dev-ats-stan1-n2)
2021-02-18 00:01:04.371 INFO SubTaskGroup.java:116 [TaskPool-0] Adding task #2: AnsibleConfigureServers(56a7dfce-4ced-4010-81d9-cb69977d6a82)(56a7dfce-4ced-4010-81d9-cb69977d6a82, yb-dev-ats-stan1-n3)
2021-02-18 00:01:04.400 INFO SubTaskGroup.java:116 [TaskPool-0] Adding task #0: AnsibleConfigureServers(56a7dfce-4ced-4010-81d9-cb69977d6a82)(56a7dfce-4ced-4010-81d9-cb69977d6a82, yb-dev-ats-stan1-n1)
2021-02-18 00:01:04.430 INFO SubTaskGroup.java:116 [TaskPool-0] Adding task #1: AnsibleConfigureServers(56a7dfce-4ced-4010-81d9-cb69977d6a82)(56a7dfce-4ced-4010-81d9-cb69977d6a82, yb-dev-ats-stan1-n2)
2021-02-18 00:01:04.449 INFO SubTaskGroup.java:116 [TaskPool-0] Adding task #2: AnsibleConfigureServers(56a7dfce-4ced-4010-81d9-cb69977d6a82)(56a7dfce-4ced-4010-81d9-cb69977d6a82, yb-dev-ats-stan1-n3)
2021-02-18 00:01:04.483 INFO SubTaskGroup.java:116 [TaskPool-0] Adding task #0: AnsibleClusterServerCtl(56a7dfce-4ced-4010-81d9-cb69977d6a82)(56a7dfce-4ced-4010-81d9-cb69977d6a82, yb-dev-ats-stan1-n1)(yb-dev-ats-stan1-n1, master: start)
2021-02-18 00:01:04.521 INFO SubTaskGroup.java:116 [TaskPool-0] Adding task #1: AnsibleClusterServerCtl(56a7dfce-4ced-4010-81d9-cb69977d6a82)(56a7dfce-4ced-4010-81d9-cb69977d6a82, yb-dev-ats-stan1-n2)(yb-dev-ats-stan1-n2, master: start)
2021-02-18 00:01:04.545 INFO SubTaskGroup.java:116 [TaskPool-0] Adding task #2: AnsibleClusterServerCtl(56a7dfce-4ced-4010-81d9-cb69977d6a82)(56a7dfce-4ced-4010-81d9-cb69977d6a82, yb-dev-ats-stan1-n3)(yb-dev-ats-stan1-n3, master: start)
2021-02-18 00:01:04.574 INFO SubTaskGroup.java:116 [TaskPool-0] Adding task #0: WaitForServer(56a7dfce-4ced-4010-81d9-cb69977d6a82, yb-dev-ats-stan1-n1, type=MASTER)
2021-02-18 00:01:04.581 INFO SubTaskGroup.java:116 [TaskPool-0] Adding task #1: WaitForServer(56a7dfce-4ced-4010-81d9-cb69977d6a82, yb-dev-ats-stan1-n2, type=MASTER)
2021-02-18 00:01:04.586 INFO SubTaskGroup.java:116 [TaskPool-0] Adding task #2: WaitForServer(56a7dfce-4ced-4010-81d9-cb69977d6a82, yb-dev-ats-stan1-n3, type=MASTER)
2021-02-18 00:01:04.601 INFO SubTaskGroup.java:116 [TaskPool-0] Adding task #0: AnsibleClusterServerCtl(56a7dfce-4ced-4010-81d9-cb69977d6a82)(56a7dfce-4ced-4010-81d9-cb69977d6a82, yb-dev-ats-stan1-n1)(yb-dev-ats-stan1-n1, tserver: start)
2021-02-18 00:01:04.631 INFO SubTaskGroup.java:116 [TaskPool-0] Adding task #1: AnsibleClusterServerCtl(56a7dfce-4ced-4010-81d9-cb69977d6a82)(56a7dfce-4ced-4010-81d9-cb69977d6a82, yb-dev-ats-stan1-n2)(yb-dev-ats-stan1-n2, tserver: start)
2021-02-18 00:01:04.653 INFO SubTaskGroup.java:116 [TaskPool-0] Adding task #2: AnsibleClusterServerCtl(56a7dfce-4ced-4010-81d9-cb69977d6a82)(56a7dfce-4ced-4010-81d9-cb69977d6a82, yb-dev-ats-stan1-n3)(yb-dev-ats-stan1-n3, tserver: start)
2021-02-18 00:01:04.678 INFO SubTaskGroup.java:116 [TaskPool-0] Adding task #0: WaitForServer(56a7dfce-4ced-4010-81d9-cb69977d6a82, yb-dev-ats-stan1-n1, type=TSERVER)
2021-02-18 00:01:04.681 INFO SubTaskGroup.java:116 [TaskPool-0] Adding task #1: WaitForServer(56a7dfce-4ced-4010-81d9-cb69977d6a82, yb-dev-ats-stan1-n2, type=TSERVER)
2021-02-18 00:01:04.683 INFO SubTaskGroup.java:116 [TaskPool-0] Adding task #2: WaitForServer(56a7dfce-4ced-4010-81d9-cb69977d6a82, yb-dev-ats-stan1-n3, type=TSERVER)
2021-02-18 00:01:04.694 INFO SubTaskGroup.java:116 [TaskPool-0] Adding task #0: SetNodeState(56a7dfce-4ced-4010-81d9-cb69977d6a82)(56a7dfce-4ced-4010-81d9-cb69977d6a82, yb-dev-ats-stan1-n1)
2021-02-18 00:01:04.710 INFO SubTaskGroup.java:116 [TaskPool-0] Adding task #1: SetNodeState(56a7dfce-4ced-4010-81d9-cb69977d6a82)(56a7dfce-4ced-4010-81d9-cb69977d6a82, yb-dev-ats-stan1-n2)
2021-02-18 00:01:04.724 INFO SubTaskGroup.java:116 [TaskPool-0] Adding task #2: SetNodeState(56a7dfce-4ced-4010-81d9-cb69977d6a82)(56a7dfce-4ced-4010-81d9-cb69977d6a82, yb-dev-ats-stan1-n3)
2021-02-18 00:01:04.750 INFO SubTaskGroup.java:116 [TaskPool-0] Adding task #0: WaitForMasterLeader(56a7dfce-4ced-4010-81d9-cb69977d6a82)
2021-02-18 00:01:04.757 INFO SubTaskGroup.java:116 [TaskPool-0] Adding task #0: UpdatePlacementInfo(56a7dfce-4ced-4010-81d9-cb69977d6a82)'(56a7dfce-4ced-4010-81d9-cb69977d6a82 null)'
2021-02-18 00:01:04.786 INFO UniverseTaskBase.java:139 [TaskPool-0] Setting encryption at rest status to UNDEFINED for universe 56a7dfce-4ced-4010-81d9-cb69977d6a82
2021-02-18 00:01:04.792 INFO SubTaskGroup.java:116 [TaskPool-0] Adding task #0: WaitForTServerHeartBeats(56a7dfce-4ced-4010-81d9-cb69977d6a82)
2021-02-18 00:01:04.807 INFO SubTaskGroup.java:116 [TaskPool-0] Adding task #0: SwamperTargetsFileUpdate(56a7dfce-4ced-4010-81d9-cb69977d6a82, Remove:false)
2021-02-18 00:01:04.817 INFO SubTaskGroup.java:116 [TaskPool-0] Adding task #0: UniverseUpdateSucceeded(56a7dfce-4ced-4010-81d9-cb69977d6a82)(56a7dfce-4ced-4010-81d9-cb69977d6a82)
2021-02-18 00:01:04.832 INFO SubTaskGroup.java:166 [TaskPool-0] Running task list AnsibleSetupServer.
2021-02-18 00:01:04.858 INFO PrecheckNode.java:40 [TaskPool-CreateUniverse(56a7dfce-4ced-4010-81d9-cb69977d6a82)-0] Running preflight checks for universe.
2021-02-18 00:01:04.871 INFO PrecheckNode.java:40 [TaskPool-CreateUniverse(56a7dfce-4ced-4010-81d9-cb69977d6a82)-1] Running preflight checks for universe.
2021-02-18 00:01:04.873 INFO PrecheckNode.java:40 [TaskPool-CreateUniverse(56a7dfce-4ced-4010-81d9-cb69977d6a82)-2] Running preflight checks for universe.
2021-02-18 00:01:04.906 INFO ShellProcessHandler.java:79 [TaskPool-CreateUniverse(56a7dfce-4ced-4010-81d9-cb69977d6a82)-0] Starting proc (abbrev cmd) - bin/ybcloud.sh onprem --region central1 instance precheck yb-dev-ats-stan1-n2
2021-02-18 00:01:04.935 INFO ShellProcessHandler.java:79 [TaskPool-CreateUniverse(56a7dfce-4ced-4010-81d9-cb69977d6a82)-1] Starting proc (abbrev cmd) - bin/ybcloud.sh onprem --region central1 instance precheck yb-dev-ats-stan1-n1
2021-02-18 00:01:04.938 INFO ShellProcessHandler.java:79 [TaskPool-CreateUniverse(56a7dfce-4ced-4010-81d9-cb69977d6a82)-2] Starting proc (abbrev cmd) - bin/ybcloud.sh onprem --region central1 instance precheck yb-dev-ats-stan1-n3
2021-02-18 00:01:32.596 INFO EncryptionAtRestController.java:111 [application-akka.actor.default-dispatcher-101] Listing KMS configurations for customer 88d5de2f-88e4-4419-8ddc-34982abcae45
2021-02-18 00:01:32.788 INFO ShellProcessHandler.java:124 [TaskPool-CreateUniverse(56a7dfce-4ced-4010-81d9-cb69977d6a82)-2] Completed proc 'bin/ybcloud.sh onprem --region central1 instance precheck yb-dev-ats-stan1-n3' status=success [ 27850 ms ]
2021-02-18 00:01:35.287 INFO Scheduler.java:116 [application-akka.actor.default-dispatcher-97] Running scheduler
2021-02-18 00:01:35.296 INFO HealthChecker.java:189 [application-akka.actor.default-dispatcher-98] Started running health checker
2021-02-18 00:01:35.300 INFO HealthChecker.java:208 [application-akka.actor.default-dispatcher-98] Skipping customer 88d5de2f-88e4-4419-8ddc-34982abcae45 due to missing alerting config...
2021-02-18 00:01:35.300 INFO HealthChecker.java:200 [application-akka.actor.default-dispatcher-98] Completed running health checker.
2021-02-18 00:01:37.875 INFO ShellProcessHandler.java:124 [TaskPool-CreateUniverse(56a7dfce-4ced-4010-81d9-cb69977d6a82)-1] Completed proc 'bin/ybcloud.sh onprem --region central1 instance precheck yb-dev-ats-stan1-n1' status=success [ 32940 ms ]
2021-02-18 00:01:37.971 INFO ShellProcessHandler.java:124 [TaskPool-CreateUniverse(56a7dfce-4ced-4010-81d9-cb69977d6a82)-0] Completed proc 'bin/ybcloud.sh onprem --region central1 instance precheck yb-dev-ats-stan1-n2' status=success [ 33065 ms ]
2021-02-18 00:01:37.992 ERROR SubTaskGroup.java:193 [TaskPool-0] Failed to execute task {"errorString":null,"nodeExporterUser":"prometheus","deviceInfo":{"volumeSize":200,"numVolumes":1,"diskIops":null,"storageClass":"standard","mountPoints":"/data","storageType":null},"universeUUID":"56a7dfce-4ced-4010-81d9-cb69977d6a82","expectedUniverseVersion":0,"cmkArn":null,"encryptionAtRestConfig":{"encryptionAtRestEnabled":false,"kmsConfigUUID":null,"opType":"UNDEFINED","type":"DATA_KEY"},"nodeDetailsSet":null,"communicationPorts":{"masterHttpPort":7000,"masterRpcPort":7100,"tserverHttpPort":9000,"tserverRpcPort":9100,"redisServerHttpPort":11000,"redisServerRpcPort":6379,"yqlServerHttpPort":12000,"yqlServerRpcPort":9042,"ysqlServerHttpPort":13000,"ysqlServerRpcPort":5433,"nodeExporterPort":9300},"extraDependencies":{"installNodeExporter":true},"firstTry":true,"clusters":[],"preflight_checks":null,"nodePrefix":null,"rootCA":null,"userAZSelected":false,"resetAZConfig":false,"updateInProgress":false,"backupInProgress":false,"updateSucceeded":true,"nextClusterIndex":1,"allowInsecure":true,"itestS3PackagePath":"","remotePackagePath":"","importedState":"NONE","capability":"EDITS_ALLOWED","azUuid":"99d00076-eed7-4e97-a253-c1340d198205","nodeName":"yb-dev-ats-stan1-n2","nodeUuid":null,"placementUuid":null,"instanceType":null,"properties":{},"region":{"uuid":"f47f2dac-63f5-4347-8cf2-306adaad0c4f","code":"central1","name":"central1","ybImage":null,"longitude":-99.0,"latitude":28.0,"zones":[{"uuid":"99d00076-eed7-4e97-a253-c1340d198205","code":"c","name":"c","active":true,"subnet":null},{"uuid":"47249714-7f59-4196-add9-2b59a5a77032","code":"b","name":"b","active":true,"subnet":null},{"uuid":"428696af-c72b-4349-9fe6-534243213062","code":"a","name":"a","active":true,"subnet":null}],"active":true,"details":null,"vnetName":null,"securityGroupId":null},"provider":{"uuid":"f92b45a8-ab35-4550-902a-b98a6e03cb01","code":"onprem","name":"oel-bm-gcp","active":true,"customerUUID":"88d5de2f-88e4-4419-8ddc-34982abcae45","awsHostedZoneId":null,"awsHostedZoneName":null,"cloudParams":{"errorString":null,"providerUUID":null,"perRegionMetadata":{},"keyPairName":null,"sshPrivateKeyContent":null,"sshUser":null,"airGapInstall":false,"sshPort":54422,"hostVpcId":null,"hostVpcRegion":null,"customHostCidrs":[],"destVpcId":null}},"az":{"uuid":"99d00076-eed7-4e97-a253-c1340d198205","code":"c","name":"c","active":true,"subnet":null}}, hit error java.lang.RuntimeException: {
"SSH Connection": false
}.
java.util.concurrent.ExecutionException: java.lang.RuntimeException: {
"SSH Connection": false
}
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at com.yugabyte.yw.commissioner.SubTaskGroup.waitFor(SubTaskGroup.java:182)
at com.yugabyte.yw.commissioner.SubTaskGroupQueue.run(SubTaskGroupQueue.java:43)
at com.yugabyte.yw.commissioner.tasks.CreateUniverse.run(CreateUniverse.java:154)
at com.yugabyte.yw.commissioner.TaskRunner.run(TaskRunner.java:144)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: {
"SSH Connection": false
}
at com.yugabyte.yw.commissioner.AbstractTaskBase.processShellResponse(AbstractTaskBase.java:112)
at com.yugabyte.yw.commissioner.tasks.subtasks.PrecheckNode.run(PrecheckNode.java:64)
... 5 common frames omitted
2021-02-18 00:01:37.999 ERROR SubTaskGroup.java:193 [TaskPool-0] Failed to execute task {"errorString":null,"nodeExporterUser":"prometheus","deviceInfo":{"volumeSize":200,"numVolumes":1,"diskIops":null,"storageClass":"standard","mountPoints":"/data","storageType":null},"universeUUID":"56a7dfce-4ced-4010-81d9-cb69977d6a82","expectedUniverseVersion":0,"cmkArn":null,"encryptionAtRestConfig":{"encryptionAtRestEnabled":false,"kmsConfigUUID":null,"opType":"UNDEFINED","type":"DATA_KEY"},"nodeDetailsSet":null,"communicationPorts":{"masterHttpPort":7000,"masterRpcPort":7100,"tserverHttpPort":9000,"tserverRpcPort":9100,"redisServerHttpPort":11000,"redisServerRpcPort":6379,"yqlServerHttpPort":12000,"yqlServerRpcPort":9042,"ysqlServerHttpPort":13000,"ysqlServerRpcPort":5433,"nodeExporterPort":9300},"extraDependencies":{"installNodeExporter":true},"firstTry":true,"clusters":[],"preflight_checks":null,"nodePrefix":null,"rootCA":null,"userAZSelected":false,"resetAZConfig":false,"updateInProgress":false,"backupInProgress":false,"updateSucceeded":true,"nextClusterIndex":1,"allowInsecure":true,"itestS3PackagePath":"","remotePackagePath":"","importedState":"NONE","capability":"EDITS_ALLOWED","azUuid":"47249714-7f59-4196-add9-2b59a5a77032","nodeName":"yb-dev-ats-stan1-n3","nodeUuid":null,"placementUuid":null,"instanceType":null,"properties":{},"region":{"uuid":"f47f2dac-63f5-4347-8cf2-306adaad0c4f","code":"central1","name":"central1","ybImage":null,"longitude":-99.0,"latitude":28.0,"zones":[{"uuid":"99d00076-eed7-4e97-a253-c1340d198205","code":"c","name":"c","active":true,"subnet":null},{"uuid":"47249714-7f59-4196-add9-2b59a5a77032","code":"b","name":"b","active":true,"subnet":null},{"uuid":"428696af-c72b-4349-9fe6-534243213062","code":"a","name":"a","active":true,"subnet":null}],"active":true,"details":null,"vnetName":null,"securityGroupId":null},"provider":{"uuid":"f92b45a8-ab35-4550-902a-b98a6e03cb01","code":"onprem","name":"oel-bm-gcp","active":true,"customerUUID":"88d5de2f-88e4-4419-8ddc-34982abcae45","awsHostedZoneId":null,"awsHostedZoneName":null,"cloudParams":{"errorString":null,"providerUUID":null,"perRegionMetadata":{},"keyPairName":null,"sshPrivateKeyContent":null,"sshUser":null,"airGapInstall":false,"sshPort":54422,"hostVpcId":null,"hostVpcRegion":null,"customHostCidrs":[],"destVpcId":null}},"az":{"uuid":"47249714-7f59-4196-add9-2b59a5a77032","code":"b","name":"b","active":true,"subnet":null}}, hit error java.lang.RuntimeException: {
"SSH Connection": false
}.
java.util.concurrent.ExecutionException: java.lang.RuntimeException: {
"SSH Connection": false
}
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at com.yugabyte.yw.commissioner.SubTaskGroup.waitFor(SubTaskGroup.java:182)
at com.yugabyte.yw.commissioner.SubTaskGroupQueue.run(SubTaskGroupQueue.java:43)
at com.yugabyte.yw.commissioner.tasks.CreateUniverse.run(CreateUniverse.java:154)
at com.yugabyte.yw.commissioner.TaskRunner.run(TaskRunner.java:144)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: {
"SSH Connection": false
}
at com.yugabyte.yw.commissioner.AbstractTaskBase.processShellResponse(AbstractTaskBase.java:112)
at com.yugabyte.yw.commissioner.tasks.subtasks.PrecheckNode.run(PrecheckNode.java:64)
... 5 common frames omitted
2021-02-18 00:01:38.008 ERROR SubTaskGroup.java:193 [TaskPool-0] Failed to execute task {"errorString":null,"nodeExporterUser":"prometheus","deviceInfo":{"volumeSize":200,"numVolumes":1,"diskIops":null,"storageClass":"standard","mountPoints":"/data","storageType":null},"universeUUID":"56a7dfce-4ced-4010-81d9-cb69977d6a82","expectedUniverseVersion":0,"cmkArn":null,"encryptionAtRestConfig":{"encryptionAtRestEnabled":false,"kmsConfigUUID":null,"opType":"UNDEFINED","type":"DATA_KEY"},"nodeDetailsSet":null,"communicationPorts":{"masterHttpPort":7000,"masterRpcPort":7100,"tserverHttpPort":9000,"tserverRpcPort":9100,"redisServerHttpPort":11000,"redisServerRpcPort":6379,"yqlServerHttpPort":12000,"yqlServerRpcPort":9042,"ysqlServerHttpPort":13000,"ysqlServerRpcPort":5433,"nodeExporterPort":9300},"extraDependencies":{"installNodeExporter":true},"firstTry":true,"clusters":[],"preflight_checks":null,"nodePrefix":null,"rootCA":null,"userAZSelected":false,"resetAZConfig":false,"updateInProgress":false,"backupInProgress":false,"updateSucceeded":true,"nextClusterIndex":1,"allowInsecure":true,"itestS3PackagePath":"","remotePackagePath":"","importedState":"NONE","capability":"EDITS_ALLOWED","azUuid":"428696af-c72b-4349-9fe6-534243213062","nodeName":"yb-dev-ats-stan1-n1","nodeUuid":null,"placementUuid":null,"instanceType":null,"properties":{},"region":{"uuid":"f47f2dac-63f5-4347-8cf2-306adaad0c4f","code":"central1","name":"central1","ybImage":null,"longitude":-99.0,"latitude":28.0,"zones":[{"uuid":"99d00076-eed7-4e97-a253-c1340d198205","code":"c","name":"c","active":true,"subnet":null},{"uuid":"47249714-7f59-4196-add9-2b59a5a77032","code":"b","name":"b","active":true,"subnet":null},{"uuid":"428696af-c72b-4349-9fe6-534243213062","code":"a","name":"a","active":true,"subnet":null}],"active":true,"details":null,"vnetName":null,"securityGroupId":null},"provider":{"uuid":"f92b45a8-ab35-4550-902a-b98a6e03cb01","code":"onprem","name":"oel-bm-gcp","active":true,"customerUUID":"88d5de2f-88e4-4419-8ddc-34982abcae45","awsHostedZoneId":null,"awsHostedZoneName":null,"cloudParams":{"errorString":null,"providerUUID":null,"perRegionMetadata":{},"keyPairName":null,"sshPrivateKeyContent":null,"sshUser":null,"airGapInstall":false,"sshPort":54422,"hostVpcId":null,"hostVpcRegion":null,"customHostCidrs":[],"destVpcId":null}},"az":{"uuid":"428696af-c72b-4349-9fe6-534243213062","code":"a","name":"a","active":true,"subnet":null}}, hit error java.lang.RuntimeException: {
"SSH Connection": false
}.
java.util.concurrent.ExecutionException: java.lang.RuntimeException: {
"SSH Connection": false
}
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at com.yugabyte.yw.commissioner.SubTaskGroup.waitFor(SubTaskGroup.java:182)
at com.yugabyte.yw.commissioner.SubTaskGroupQueue.run(SubTaskGroupQueue.java:43)
at com.yugabyte.yw.commissioner.tasks.CreateUniverse.run(CreateUniverse.java:154)
at com.yugabyte.yw.commissioner.TaskRunner.run(TaskRunner.java:144)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: {
"SSH Connection": false
}
at com.yugabyte.yw.commissioner.AbstractTaskBase.processShellResponse(AbstractTaskBase.java:112)
at com.yugabyte.yw.commissioner.tasks.subtasks.PrecheckNode.run(PrecheckNode.java:64)
... 5 common frames omitted
2021-02-18 00:01:38.024 ERROR SubTaskGroupQueue.java:53 [TaskPool-0] SubTaskGroup 'AnsibleSetupServer : completed 0 out of 3 tasks.' waitFor() returned failed status.
2021-02-18 00:01:38.037 ERROR CreateUniverse.java:156 [TaskPool-0] Error executing task CreateUniverse(56a7dfce-4ced-4010-81d9-cb69977d6a82), error='AnsibleSetupServer : completed 0 out of 3 tasks. failed.'
java.lang.RuntimeException: AnsibleSetupServer : completed 0 out of 3 tasks. failed.
at com.yugabyte.yw.commissioner.SubTaskGroupQueue.run(SubTaskGroupQueue.java:56)
at com.yugabyte.yw.commissioner.tasks.CreateUniverse.run(CreateUniverse.java:154)
at com.yugabyte.yw.commissioner.TaskRunner.run(TaskRunner.java:144)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-02-18 00:01:38.052 ERROR TaskRunner.java:150 [TaskPool-0] Error running task
java.lang.RuntimeException: AnsibleSetupServer : completed 0 out of 3 tasks. failed.
at com.yugabyte.yw.commissioner.SubTaskGroupQueue.run(SubTaskGroupQueue.java:56)
at com.yugabyte.yw.commissioner.tasks.CreateUniverse.run(CreateUniverse.java:154)
at com.yugabyte.yw.commissioner.TaskRunner.run(TaskRunner.java:144)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-02-18 00:01:38.055 INFO TaskRunner.java:179 [TaskPool-0] Updating task [taskType : CreateUniverse, taskState: Running] to new state Failure
2021-02-18 00:01:38.232 INFO Commissioner.java:167 [TaskProgressMonitor] Task task-info {taskType : CreateUniverse, taskState: Failure}, task {CreateUniverse(56a7dfce-4ced-4010-81d9-cb69977d6a82)} has failed.

@ssung-yugabyte
Copy link
Contributor

OS: oracle enterprise linux 7.7 with compatible kernel.
user: both yugabyte and centos are in wheel group with passwordless sudo)
yugabyte and centos users can passwordless ssh and sudo to yb-server nodes (ssh-copy-id).

When setup the provider with "centos" user, I can create a universe without issue.
When setup the provider with "yugabyte" user, I found the above error in yugaware docker logs.

@ssung-yugabyte
Copy link
Contributor

yugaware + replicated
version: latest 2.5.1.0-b153

@streddy-yb streddy-yb added the priority/high High Priority label Feb 18, 2021
WesleyW added a commit that referenced this issue Feb 18, 2021
@WesleyW
#7268: Use the correct SSH user for on-prem node preflight checks
e4dd7dd 
Summary:
The preflight checks are always using "centos" for the ssh_user because it is not being passed
down properly. Note that this already works for manual provisioning because we always specify
ssh user in the script.

Test Plan:
Create instance, add a new ssh-able sudo user, remove ssh access to centos.

Create universe and see if precheck passes the "SSH Connection" task.

Reviewers: sanketh, arnav, daniel, sb-yb, spotachev

Reviewed By: spotachev

Subscribers: jenkins-bot, yugaware

Differential Revision: https://phabricator.dev.yugabyte.com/D10650
WesleyW added a commit that referenced this issue Feb 18, 2021
@WesleyW
[Backport 2.4][#7268] Platform: Use the correct SSH user for on-prem …
83722b5 
…node preflight checks

Summary:
The preflight checks are always using "centos" for the ssh_user because it is not being passed
down properly. Note that this already works for manual provisioning because we always specify
ssh user in the script.

Test Plan:
Create instance, add a new ssh-able sudo user, remove ssh access to centos.

Create universe and see if precheck passes the "SSH Connection" task.

Reviewers: sanketh, arnav, daniel, sb-yb, spotachev

Reviewed By: spotachev

Subscribers: yugaware, jenkins-bot

Differential Revision: https://phabricator.dev.yugabyte.com/D10651
polarweasel pushed a commit to lizayugabyte/yugabyte-db that referenced this issue Mar 9, 2021
@WesleyW
yugabyte#7268: Use the correct SSH user for on-prem node preflight ch…
dc83e0d 
…ecks

Summary:
The preflight checks are always using "centos" for the ssh_user because it is not being passed
down properly. Note that this already works for manual provisioning because we always specify
ssh user in the script.

Test Plan:
Create instance, add a new ssh-able sudo user, remove ssh access to centos.

Create universe and see if precheck passes the "SSH Connection" task.

Reviewers: sanketh, arnav, daniel, sb-yb, spotachev

Reviewed By: spotachev

Subscribers: jenkins-bot, yugaware

Differential Revision: https://phabricator.dev.yugabyte.com/D10650
@streddy-yb
Copy link
Contributor Author

streddy-yb commented Apr 25, 2021 •
edited
Loading

  • Configured an Onprem Provider with yugabyte as the SSH user
  • Manually provisioned the node (** see below for an unrelated issue **)
  • Create the universe; verified that preflight checks were being performed using the correct user
2021-04-25 23:28:01,412 [INFO] from com.yugabyte.yw.commissioner.tasks.UniverseTaskBase in TaskPool-159 - Running preflight checks for node yb-dev-streddy-custom-ssh-user-n1.
2021-04-25 23:28:01,425 [INFO] from com.yugabyte.yw.common.DevopsBase in TaskPool-159 - Command to run: [bin/ybcloud.sh onprem --region us-west --zone A1 --node_metadata {"ip":"10.9.11.45","sshUser":"yugabyte","region":"us-west","zone":"A1","instanceType":"centos","instanceName":"node2","nodeName":"yb-dev-streddy-custom-ssh-user-n1"} instance precheck --vars_file /opt/yugabyte/yugaware/data/keys/78d7b9e7-c80d-46e7-8e35-0a2102ed405a/onprem-provider-key.vault --vault_password_file /opt/yugabyte/yugaware/data/keys/78d7b9e7-c80d-46e7-8e35-0a2102ed405a/onprem-provider-key.vault_password --private_key_file /opt/yugabyte/yugaware/data/keys/78d7b9e7-c80d-46e7-8e35-0a2102ed405a/onprem-provider-key.pem --custom_ssh_port 22 --precheck_type configure **--ssh_user yugabyte** --install_node_exporter --mount_points /storage --volume_size 100 yb-dev-streddy-custom-ssh-user-n1]

However, the manual provisioning step failed if I don't give sudo permissions to the yugabyte user. @WesleyW - does this mean the SSH user must always have SUDO permissions now?

yugaware]# /opt/yugabyte/yugaware/data/provision/78d7b9e7-c80d-46e7-8e35-0a2102ed405a/provision_instance.py --ip XX.XX. XX.XX --mount_points /storage

Failed preflight checks!
{
  "(Mount Point) /storage is writable": false, 
  "(Prometheus) No Pre-existing Node Exporter Running": true, 
  "Internet Connection": true, 
  "SSH Connection": true, 
  "(Prometheus) /var/run/prometheus is writable": false, 
  "(Prometheus) /etc/prometheus is writable": false, 
  "Sudo Access to Python": false, 
  "(Prometheus) /lib/systemd/system/node_exporter.service is writable": false, 
  "Yugabyte User in Yugabyte Group": true, 
  "(PAM Limits) /etc/security/limits.conf is writable": false, 
  "Try Ansible Command": true, 
  "(Prometheus) /opt/prometheus is writable": false, 
  "(Prometheus) /var/log/prometheus is writable": false, 
  "(Prometheus) /var/lib/prometheus is writable": false
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@WesleyW WesleyW

Labels
area/platform Yugabyte Platform priority/high High Priority
Projects
None yet
Milestone
2.5.x
Development

No branches or pull requests
3 participants
@WesleyW @streddy-yb @ssung-yugabyte