Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to delete pods, stuck in terminating state #220

Closed
cbluth opened this issue Jul 17, 2018 · 3 comments
Closed

Unable to delete pods, stuck in terminating state #220

cbluth opened this issue Jul 17, 2018 · 3 comments

Comments

@cbluth
Copy link

cbluth commented Jul 17, 2018

I am not sure the exact steps I took to land myself in this situation, but I cannot delete pods.
I cannot delete ArangoDeployment/arangodb-cluster either.

Here is some info:

user@laptop:~/arangodb$ kubectl delete pod -l app=arangodb
pod "arangodb-cluster-agnt-bnwzveev-4f7109" deleted
pod "arangodb-cluster-agnt-cvikovgy-4f7109" deleted
pod "arangodb-cluster-agnt-fc2fco8j-4f7109" deleted
pod "arangodb-cluster-prmr-lafvlwvb-4f7109" deleted
pod "arangodb-cluster-prmr-r40r3cdf-4f7109" deleted
pod "arangodb-cluster-prmr-skdk4xpp-4f7109" deleted

user@laptop:~/arangodb$ kubectl get pod -l app=arangodb
NAME                                    READY     STATUS        RESTARTS   AGE
arangodb-cluster-agnt-bnwzveev-4f7109   0/1       Terminating   0          22h
arangodb-cluster-agnt-cvikovgy-4f7109   0/1       Terminating   0          22h
arangodb-cluster-agnt-fc2fco8j-4f7109   0/1       Terminating   0          1d
arangodb-cluster-prmr-lafvlwvb-4f7109   0/1       Terminating   0          1d
arangodb-cluster-prmr-r40r3cdf-4f7109   0/1       Terminating   0          1d
arangodb-cluster-prmr-skdk4xpp-4f7109   0/1       Terminating   0          1d
user@laptop:~/arangodb$ 

user@laptop:~/arangodb$ kubectl -n platform describe ArangoDeployment/arangodb-cluster
Name:         arangodb-cluster
Namespace:    platform
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"database.arangodb.com/v1alpha","kind":"ArangoDeployment","metadata":{"annotations":{},"name":"arangodb-cluster","namespace":"platform"},...
API Version:  database.arangodb.com/v1alpha
Kind:         ArangoDeployment
Metadata:
  Cluster Name:                   
  Creation Timestamp:             2018-06-21T07:58:37Z
  Deletion Grace Period Seconds:  0
  Deletion Timestamp:             2018-07-17T14:31:04Z
  Finalizers:
    database.arangodb.com/remove-child-finalizers
  Generation:        2
  Resource Version:  11280293
  Self Link:         /apis/database.arangodb.com/v1alpha/namespaces/platform/arangodeployments/arangodb-cluster
  UID:               e7100671-7528-11e8-8e23-525400ac45ea
Spec:
  Agents:
    Count:  0
    Resources:
      Requests:
        Storage:  8Gi
  Auth:
    Jwt Secret Name:  arangodb-cluster-jwt
  Chaos:
    Interval:                  60000000000
    Kill - Pod - Probability:  50
  Coordinators:
    Count:  0
    Resources:
  Dbservers:
    Count:  0
    Resources:
      Requests:
        Storage:  8Gi
  Environment:    Production
  External Access:
  Image:              arangodb/arangodb:3.3.10
  Image Pull Policy:  IfNotPresent
  Mode:               Cluster
  Rocksdb:
    Encryption:
  Single:
    Resources:
      Requests:
        Storage:   8Gi
  Storage Engine:  RocksDB
  Sync:
    Auth:
      Client CA Secret Name:  arangodb-cluster-sync-client-auth-ca
      Jwt Secret Name:        arangodb-cluster-sync-jwt
    External Access:
    Monitoring:
      Token Secret Name:  arangodb-cluster-sync-mt
    Tls:
      Ca Secret Name:  arangodb-cluster-sync-ca
      Ttl:             2610h
  Syncmasters:
    Resources:
  Syncworkers:
    Resources:
  Tls:
    Ca Secret Name:  arangodb-cluster-ca
    Ttl:             2610h
Status:
  Accepted - Spec:
    Agents:
      Count:  0
      Resources:
        Requests:
          Storage:  8Gi
    Auth:
      Jwt Secret Name:  arangodb-cluster-jwt
    Chaos:
      Interval:                  60000000000
      Kill - Pod - Probability:  50
    Coordinators:
      Count:  0
      Resources:
    Dbservers:
      Count:  0
      Resources:
        Requests:
          Storage:  8Gi
    Environment:    Production
    External Access:
    Image:              arangodb/arangodb:3.3.10
    Image Pull Policy:  IfNotPresent
    Mode:               Cluster
    Rocksdb:
      Encryption:
    Single:
      Resources:
        Requests:
          Storage:   8Gi
    Storage Engine:  RocksDB
    Sync:
      Auth:
        Client CA Secret Name:  arangodb-cluster-sync-client-auth-ca
        Jwt Secret Name:        arangodb-cluster-sync-jwt
      External Access:
      Monitoring:
        Token Secret Name:  arangodb-cluster-sync-mt
      Tls:
        Ca Secret Name:  arangodb-cluster-sync-ca
        Ttl:             2610h
    Syncmasters:
      Resources:
    Syncworkers:
      Resources:
    Tls:
      Ca Secret Name:  arangodb-cluster-ca
      Ttl:             2610h
  Arangodb - Images:
    Arangodb - Version:  3.3.10
    Image:               arangodb/arangodb:3.3.10
    Image - Id:          arangodb/arangodb@sha256:fa7fe7c8f2ca3e058440b20b4910d10df9fc4488f52c4a75e4e71ed4f0194a28
  Conditions:
    Last Transition Time:  2018-06-21T07:58:42Z
    Last Update Time:      2018-06-21T07:58:42Z
    Status:                False
    Type:                  Ready
  Members:
    Agents:
      Conditions:
        Last Transition Time:        2018-07-16T15:58:54Z
        Last Update Time:            2018-07-16T15:58:54Z
        Reason:                      Pod Not Ready
        Status:                      False
        Type:                        Ready
      Created - At:                  2018-06-21T07:58:37Z
      Id:                            AGNT-cvikovgy
      Initialized:                   false
      Persistent Volume Claim Name:  arangodb-cluster-agent-cvikovgy
      Phase:                         Created
      Pod Name:                      arangodb-cluster-agnt-cvikovgy-4f7109
      Recent - Terminations:
        2018-07-16T15:56:21Z
      Conditions:
        Last Transition Time:        2018-07-16T15:58:54Z
        Last Update Time:            2018-07-16T15:58:54Z
        Reason:                      Pod Not Ready
        Status:                      False
        Type:                        Ready
      Created - At:                  2018-06-21T07:58:37Z
      Id:                            AGNT-bnwzveev
      Initialized:                   false
      Persistent Volume Claim Name:  arangodb-cluster-agent-bnwzveev
      Phase:                         Created
      Pod Name:                      arangodb-cluster-agnt-bnwzveev-4f7109
      Recent - Terminations:
        2018-07-16T15:56:21Z
      Conditions:
        Last Transition Time:        2018-07-16T12:38:17Z
        Last Update Time:            2018-07-16T12:38:17Z
        Reason:                      Pod Not Ready
        Status:                      False
        Type:                        Ready
      Created - At:                  2018-06-21T07:58:37Z
      Id:                            AGNT-fc2fco8j
      Initialized:                   false
      Persistent Volume Claim Name:  arangodb-cluster-agent-fc2fco8j
      Phase:                         Created
      Pod Name:                      arangodb-cluster-agnt-fc2fco8j-4f7109
      Recent - Terminations:
        2018-06-26T19:56:42Z
        2018-07-16T10:04:55Z
        2018-07-16T12:35:46Z
    Coordinators:
      Conditions:
        Last Transition Time:  2018-06-21T07:59:59Z
        Last Update Time:      2018-06-21T07:59:59Z
        Reason:                Pod Not Ready
        Status:                False
        Type:                  Ready
        Last Transition Time:  2018-06-26T19:56:42Z
        Last Update Time:      2018-06-26T19:56:42Z
        Reason:                Pod Terminated
        Status:                True
        Type:                  Terminated
      Created - At:            2018-06-21T07:58:37Z
      Id:                      CRDN-70m1ufys
      Initialized:             false
      Phase:                   Failed
      Pod Name:                arangodb-cluster-crdn-70m1ufys-4f7109
      Recent - Terminations:
        2018-06-26T19:56:42Z
      Conditions:
        Last Transition Time:  2018-06-21T07:59:59Z
        Last Update Time:      2018-06-21T07:59:59Z
        Reason:                Pod Not Ready
        Status:                False
        Type:                  Ready
        Last Transition Time:  2018-06-26T19:56:42Z
        Last Update Time:      2018-06-26T19:56:42Z
        Reason:                Pod Terminated
        Status:                True
        Type:                  Terminated
      Created - At:            2018-06-21T07:58:37Z
      Id:                      CRDN-lhzbloq9
      Initialized:             false
      Phase:                   Failed
      Pod Name:                arangodb-cluster-crdn-lhzbloq9-4f7109
      Recent - Terminations:
        2018-06-26T19:56:42Z
      Conditions:
        Last Transition Time:  2018-06-21T07:59:59Z
        Last Update Time:      2018-06-21T07:59:59Z
        Reason:                Pod Not Ready
        Status:                False
        Type:                  Ready
        Last Transition Time:  2018-07-16T10:55:09Z
        Last Update Time:      2018-07-16T10:59:04Z
        Reason:                Pod Terminated
        Status:                True
        Type:                  Terminated
      Created - At:            2018-06-21T07:58:37Z
      Id:                      CRDN-w8u5ely9
      Initialized:             false
      Phase:                   Failed
      Pod Name:                arangodb-cluster-crdn-w8u5ely9-4f7109
      Recent - Terminations:
        2018-07-16T10:55:09Z
    Dbservers:
      Conditions:
        Last Transition Time:        2018-07-16T13:37:26Z
        Last Update Time:            2018-07-16T13:37:26Z
        Reason:                      Pod Not Ready
        Status:                      False
        Type:                        Ready
      Created - At:                  2018-06-21T07:58:37Z
      Id:                            PRMR-skdk4xpp
      Initialized:                   false
      Persistent Volume Claim Name:  arangodb-cluster-dbserver-skdk4xpp
      Phase:                         Created
      Pod Name:                      arangodb-cluster-prmr-skdk4xpp-4f7109
      Recent - Terminations:
        2018-06-26T19:59:14Z
        2018-07-16T10:04:55Z
        2018-07-16T13:35:25Z
      Conditions:
        Last Transition Time:        2018-07-16T12:42:50Z
        Last Update Time:            2018-07-16T12:42:50Z
        Reason:                      Pod Not Ready
        Status:                      False
        Type:                        Ready
      Created - At:                  2018-06-21T07:58:37Z
      Id:                            PRMR-lafvlwvb
      Initialized:                   false
      Persistent Volume Claim Name:  arangodb-cluster-dbserver-lafvlwvb
      Phase:                         Created
      Pod Name:                      arangodb-cluster-prmr-lafvlwvb-4f7109
      Recent - Terminations:
        2018-07-16T10:04:55Z
        2018-07-16T12:40:48Z
      Conditions:
        Last Transition Time:        2018-07-16T12:44:22Z
        Last Update Time:            2018-07-16T12:44:22Z
        Reason:                      Pod Not Ready
        Status:                      False
        Type:                        Ready
      Created - At:                  2018-06-21T07:58:37Z
      Id:                            PRMR-r40r3cdf
      Initialized:                   false
      Persistent Volume Claim Name:  arangodb-cluster-dbserver-r40r3cdf
      Phase:                         Created
      Pod Name:                      arangodb-cluster-prmr-r40r3cdf-4f7109
      Recent - Terminations:
        2018-07-16T12:42:50Z
  Phase:  Running
  Plan:
    Creation Time:  2018-06-21T08:06:58Z
    Group:          4
    Id:             R32pMTPkJVj8gXiW
    Member ID:      CRDN-70m1ufys
    Type:           RemoveMember
    Creation Time:  2018-06-21T08:06:58Z
    Group:          4
    Id:             czD5LhtMcWGdXepC
    Type:           AddMember
  Secret - Hashes:
    Auth - Jwt:  dd5b638b4bd6a07c333ae638b7b2631c4074a1f0b300deeb56e0e9f45e7f6484
    Tls - Ca:    b22a4afa0e31708668d614ab69d8a92343baf037fd197a0d8ecd7eff7f4729ef
  Service Name:  arangodb-cluster
Events:          <none>
user@laptop:~/arangodb$

How can I delete these pods?
@cbluth
Copy link
Author

cbluth commented Jul 17, 2018

Here is a bit more info. not sure if its needed:


user@laptop:~/arangodb$ kubectl delete CustomResourceDefinition arangodeployments.database.arangodb.com --force --now=true
customresourcedefinition.apiextensions.k8s.io "arangodeployments.database.arangodb.com" deleted
user@laptop:~/arangodb$ kubectl delete CustomResourceDefinition arangodeployments.database.arangodb.com --force --now=true
customresourcedefinition.apiextensions.k8s.io "arangodeployments.database.arangodb.com" deleted
user@laptop:~/arangodb$ kubectl delete CustomResourceDefinition arangodeployments.database.arangodb.com --force --now=true
customresourcedefinition.apiextensions.k8s.io "arangodeployments.database.arangodb.com" deleted
user@laptop:~/arangodb$ kubectl describe CustomResourceDefinition arangodeployments.database.arangodb.com
Name:         arangodeployments.database.arangodb.com
Namespace:    
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"apiextensions.k8s.io/v1beta1","kind":"CustomResourceDefinition","metadata":{"annotations":{},"name":"arangodeployments.database.arangodb...
API Version:  apiextensions.k8s.io/v1beta1
Kind:         CustomResourceDefinition
Metadata:
  Creation Timestamp:             2018-06-21T07:40:55Z
  Deletion Grace Period Seconds:  0
  Deletion Timestamp:             2018-07-17T14:48:19Z
  Finalizers:
    customresourcecleanup.apiextensions.k8s.io
  Generation:        1
  Resource Version:  11282686
  Self Link:         /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/arangodeployments.database.arangodb.com
  UID:               6e192dab-7526-11e8-8e23-525400ac45ea
Spec:
  Group:  database.arangodb.com
  Names:
    Kind:       ArangoDeployment
    List Kind:  ArangoDeploymentList
    Plural:     arangodeployments
    Short Names:
      arangodb
      arango
    Singular:  arangodeployment
  Scope:       Namespaced
  Version:     v1alpha
Status:
  Accepted Names:
    Kind:       ArangoDeployment
    List Kind:  ArangoDeploymentList
    Plural:     arangodeployments
    Short Names:
      arangodb
      arango
    Singular:  arangodeployment
  Conditions:
    Last Transition Time:  2018-06-21T07:40:55Z
    Message:               no conflicts found
    Reason:                NoConflicts
    Status:                True
    Type:                  NamesAccepted
    Last Transition Time:  2018-06-21T07:40:55Z
    Message:               the initial names have been accepted
    Reason:                InitialNamesAccepted
    Status:                True
    Type:                  Established
    Last Transition Time:  2018-07-17T13:08:27Z
    Message:               CustomResource deletion is in progress
    Reason:                InstanceDeletionInProgress
    Status:                True
    Type:                  Terminating
Events:                    <none>
user@laptop:~/arangodb$

@cbluth
Copy link
Author

cbluth commented Jul 17, 2018

Deploying a new operator from scratch, and then re-creating the cluster with the same name arangodb-cluster fixed my issue.

@cbluth cbluth closed this as completed Jul 17, 2018
@ewoutp
Copy link
Contributor

ewoutp commented Jul 18, 2018

FYI The operator registers finalizers on the resources that is has created (such as Pods and PersistentVolumeClaims) in order to ensure a controlled removal of those resources.
If the operator itself is no longer running, there is no one to remove this finalizers (which prevents the resource from being removed)

@rglyons rglyons mentioned this issue Sep 7, 2018
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ewoutp @cbluth