Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

failed to install stackstorm-ha on kubernetes using helm #364

Open
acharbha opened this issue Mar 16, 2023 · 1 comment
Open

failed to install stackstorm-ha on kubernetes using helm #364

acharbha opened this issue Mar 16, 2023 · 1 comment
Labels
question Further information is requested

Comments

@acharbha
Copy link

acharbha commented Mar 16, 2023 •
edited
Loading

More detailed information about failure:

PS C:\Users\user\repo> helm install stackstorm-ha .\stackstorm-ha-0.110.0.tgz --namespace=myspace --debug
install.go:194: [debug] Original chart version: ""
install.go:211: [debug] CHART PATH: C:\Users\user\repo\stackstorm-ha-0.110.0.tgz

client.go:133: [debug] creating 49 resource(s)
client.go:477: [debug] Starting delete for "stackstorm-ha-job-st2-apikey-load" Job
client.go:481: [debug] Ignoring delete failure for "stackstorm-ha-job-st2-apikey-load" batch/v1, Kind=Job: jobs.batch "stackstorm-ha-job-st2-apikey-load" not found
client.go:133: [debug] creating 1 resource(s)
client.go:703: [debug] Watching for changes to Job stackstorm-ha-job-st2-apikey-load with timeout of 5m0s
client.go:731: [debug] Add/Modify event for stackstorm-ha-job-st2-apikey-load: ADDED
client.go:770: [debug] stackstorm-ha-job-st2-apikey-load: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
Error: INSTALLATION FAILED: failed post-install: timed out waiting for the condition
helm.go:84: [debug] failed post-install: timed out waiting for the condition
INSTALLATION FAILED
main.newInstallCmd.func2
        helm.sh/helm/v3/cmd/helm/install.go:141
github.com/spf13/cobra.(*Command).execute
        github.com/spf13/[email protected]/command.go:916
github.com/spf13/cobra.(*Command).ExecuteC
        github.com/spf13/[email protected]/command.go:1044
github.com/spf13/cobra.(*Command).Execute
        github.com/spf13/[email protected]/command.go:968
main.main
        helm.sh/helm/v3/cmd/helm/helm.go:83
runtime.main
        runtime/proc.go:250
runtime.goexit
        runtime/asm_amd64.s:1571
PS C:\Users\user\repo>

helm.exe version
version.BuildInfo{Version:"v3.11.2", GitCommit:"912ebc1cd10d38d340f048efaf0abda047c3468e", GitTreeState:"clean", GoVersion:"go1.18.10"}

kubectl.exe version --short
Client Version: v1.25.0
Kustomize Version: v4.5.7
Server Version: v1.20.15

PS C:\Users\user> .\kubectl.exe get all -n myspace
NAME                                                    READY   STATUS             RESTARTS   AGE
pod/stackstorm-ha-mongodb-0                             1/1     Running            0          8m53s
pod/stackstorm-ha-rabbitmq-0                            1/1     Running            0          8m53s
pod/stackstorm-ha-st2actionrunner-5df8dfb44f-6p46k      0/1     CrashLoopBackOff   6          8m53s
pod/stackstorm-ha-st2actionrunner-5df8dfb44f-9bpvl      0/1     CrashLoopBackOff   6          8m53s
pod/stackstorm-ha-st2actionrunner-5df8dfb44f-dlfks      0/1     CrashLoopBackOff   6          8m53s
pod/stackstorm-ha-st2actionrunner-5df8dfb44f-hgjt2      0/1     CrashLoopBackOff   6          8m53s
pod/stackstorm-ha-st2actionrunner-5df8dfb44f-z9jw2      0/1     CrashLoopBackOff   6          8m53s
pod/stackstorm-ha-st2api-767d4f5847-55hpd               0/1     CrashLoopBackOff   6          8m53s
pod/stackstorm-ha-st2api-767d4f5847-kx7nl               0/1     CrashLoopBackOff   6          8m53s
pod/stackstorm-ha-st2auth-578c967cf8-r88lr              0/1     CrashLoopBackOff   6          8m53s
pod/stackstorm-ha-st2rulesengine-5f445b4d98-2n4hz       0/1     CrashLoopBackOff   6          8m53s
pod/stackstorm-ha-st2rulesengine-5f445b4d98-4qvmf       0/1     CrashLoopBackOff   6          8m53s
pod/stackstorm-ha-st2scheduler-ddb4b77b7-6xzsm          0/1     CrashLoopBackOff   6          8m53s
pod/stackstorm-ha-st2scheduler-ddb4b77b7-ndbr7          0/1     CrashLoopBackOff   6          8m53s
pod/stackstorm-ha-st2sensorcontainer-657cb59f7b-rt4c7   0/1     CrashLoopBackOff   6          8m54s
pod/stackstorm-ha-st2timersengine-6dcdbbc94c-nxf27      0/1     CrashLoopBackOff   6          8m53s
pod/stackstorm-ha-st2web-64644cf4c4-dtqmj               1/1     Running            0          8m54s
pod/stackstorm-ha-st2web-64644cf4c4-k669n               1/1     Running            0          8m54s
pod/stackstorm-ha-st2workflowengine-7d59b7c5c7-97lww    0/1     CrashLoopBackOff   6          8m54s
pod/stackstorm-ha-st2workflowengine-7d59b7c5c7-qw5wg    0/1     CrashLoopBackOff   6          8m54s

NAME                                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                 AGE
service/stackstorm-ha-mongodb-headless    ClusterIP   None             <none>        27017/TCP                               8m55s
service/stackstorm-ha-rabbitmq            ClusterIP   XX.XX.14.95     <none>        5672/TCP,4369/TCP,25672/TCP,15672/TCP   8m55s
service/stackstorm-ha-rabbitmq-headless   ClusterIP   None             <none>        4369/TCP,5672/TCP,25672/TCP,15672/TCP   8m55s
service/stackstorm-ha-redis               ClusterIP   XX.XX.232.114   <none>        6379/TCP,26379/TCP                      8m55s
service/stackstorm-ha-redis-headless      ClusterIP   None             <none>        6379/TCP,26379/TCP                      8m55s
service/stackstorm-ha-st2api              ClusterIP   XX.XX.192.205   <none>        9101/TCP                                8m55s
service/stackstorm-ha-st2auth             ClusterIP   XX.XX.163.153   <none>        9100/TCP                                8m55s
service/stackstorm-ha-st2stream           ClusterIP   XX.XX.104.135   <none>        9102/TCP                                8m55s
service/stackstorm-ha-st2web              NodePort    XX.XX.83.234    <none>        80:32282/TCP                            8m55s

NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/stackstorm-ha-st2actionrunner       0/5     5            0           8m55s
deployment.apps/stackstorm-ha-st2api                0/2     2            0           8m54s
deployment.apps/stackstorm-ha-st2auth               0/2     1            0           8m54s
deployment.apps/stackstorm-ha-st2client             0/1     0            0           8m54s
deployment.apps/stackstorm-ha-st2garbagecollector   0/1     0            0           8m54s
deployment.apps/stackstorm-ha-st2notifier           0/2     0            0           8m54s
deployment.apps/stackstorm-ha-st2rulesengine        0/2     2            0           8m54s
deployment.apps/stackstorm-ha-st2scheduler          0/2     2            0           8m54s
deployment.apps/stackstorm-ha-st2sensorcontainer    0/1     1            0           8m55s
deployment.apps/stackstorm-ha-st2stream             0/2     0            0           8m54s
deployment.apps/stackstorm-ha-st2timersengine       0/1     1            0           8m54s
deployment.apps/stackstorm-ha-st2web                2/2     2            2           8m55s
deployment.apps/stackstorm-ha-st2workflowengine     0/2     2            0           8m55s

NAME                                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/stackstorm-ha-st2actionrunner-5df8dfb44f      5         5         0       8m55s
replicaset.apps/stackstorm-ha-st2api-767d4f5847               2         2         0       8m54s
replicaset.apps/stackstorm-ha-st2auth-578c967cf8              2         1         0       8m54s
replicaset.apps/stackstorm-ha-st2client-c7d56bf9c             1         0         0       8m54s
replicaset.apps/stackstorm-ha-st2garbagecollector-bf8f5bbd6   1         0         0       8m54s
replicaset.apps/stackstorm-ha-st2notifier-76d676748c          2         0         0       8m54s
replicaset.apps/stackstorm-ha-st2rulesengine-5f445b4d98       2         2         0       8m54s
replicaset.apps/stackstorm-ha-st2scheduler-ddb4b77b7          2         2         0       8m54s
replicaset.apps/stackstorm-ha-st2sensorcontainer-657cb59f7b   1         1         0       8m55s
replicaset.apps/stackstorm-ha-st2stream-6b96d4f4              2         0         0       8m54s
replicaset.apps/stackstorm-ha-st2timersengine-6dcdbbc94c      1         1         0       8m54s
replicaset.apps/stackstorm-ha-st2web-64644cf4c4               2         2         2       8m55s
replicaset.apps/stackstorm-ha-st2workflowengine-7d59b7c5c7    2         2         0       8m55s

NAME                                        READY   AGE
statefulset.apps/stackstorm-ha-mongodb      0/3     8m54s
statefulset.apps/stackstorm-ha-rabbitmq     0/3     8m54s
statefulset.apps/stackstorm-ha-redis-node   0/3     8m54s

NAME                                          COMPLETIONS   DURATION   AGE
job.batch/stackstorm-ha-job-st2-apikey-load   0/1                      8m53s
PS C:\Users\user>


helm list -n myspace
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS  CHART                   APP VERSION
stackstorm-ha   myspace       1               2023-03-16 22:23:17.5061257 +0530 IST   failed  stackstorm-ha-0.110.0   3.8
@arm4b
Copy link
Member

arm4b commented Mar 18, 2023

The problem is with the backends not starting:

NAME                                        READY   AGE
statefulset.apps/stackstorm-ha-mongodb      0/3     8m54s
statefulset.apps/stackstorm-ha-rabbitmq     0/3     8m54s
statefulset.apps/stackstorm-ha-redis-node   0/3     8m54s

StackStorm HA relies on external helm charts for MongoDB, RabbitMQ, Redis which are used by the st2 cluster.

You'll need to debug those and find the reason why they're not starting.
I'd recommend looking into a storage as that was a frequent issue with the K8s cluster setup.

Once they start normally, stackstorm will start too.

@arm4b arm4b mentioned this issue Mar 18, 2023
Closed
@arm4b arm4b added the question Further information is requested label Mar 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@arm4b @acharbha