Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Example is not working #99

Closed
cleveritcz opened this issue Jun 15, 2021 · 40 comments
Closed

Example is not working #99

cleveritcz opened this issue Jun 15, 2021 · 40 comments
Assignees
@shubham14bajpai

Comments

@cleveritcz
Copy link

What steps did you take and what happened:
[A clear and concise description of what the bug is, and what commands you ran.)
I followed manual for Quickstart deploy and I couldn't get running example app. The pvc didn't bound it.

What did you expect to happen:
I expect example app working

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)

[root@node1 opt]# kubectl logs -f jiva-operator-6b69b86894-5lmht -n openebs
I0615 19:16:01.931555 1 request.go:655] Throttling request took 1.046617453s, request: GET:https://10.233.0.1:443/apis/cert-manager.io/v1alpha2?timeout=32s
time="2021-06-15T19:16:02Z" level=info msg="Go Version: go1.14.7"
time="2021-06-15T19:16:02Z" level=info msg="Go OS/Arch: linux/amd64"
time="2021-06-15T19:16:02Z" level=info msg="Version of jiva-operator: 2.9.0"
time="2021-06-15T19:16:05Z" level=info msg="start bootstraping jiva componentsJivaVolume: pvc-9331eb72-6657-4145-8a97-0e7a2768d199"
time="2021-06-15T19:16:05Z" level=info msg="Creating a new serviceService.NamespaceopenebsService.Namepvc-9331eb72-6657-4145-8a97-0e7a2768d199-jiva-ctrl-svc"
time="2021-06-15T19:16:06Z" level=info msg="Updating JivaVolume with iscsi specISCSISpec{10.233.49.69 3260 iqn.2016-09.com.openebs.jiva:pvc-9331eb72-6657-4145-8a97-0e7a2768d199}"
time="2021-06-15T19:16:06Z" level=info msg="Creating a new deploymentDeploy.NamespaceopenebsDeploy.Namepvc-9331eb72-6657-4145-8a97-0e7a2768d199-jiva-ctrl"
time="2021-06-15T19:16:06Z" level=info msg="Creating a new StatefulsetStatefulset.NamespaceopenebsSts.Namepvc-9331eb72-6657-4145-8a97-0e7a2768d199-jiva-rep"
time="2021-06-15T19:16:07Z" level=info msg="Creating a new pod disruption budgetPdb.NamespaceopenebsPdb.Namepvc-9331eb72-6657-4145-8a97-0e7a2768d199-pdb"
time="2021-06-15T19:16:07Z" level=info msg="start bootstraping jiva componentsJivaVolume: pvc-5ea4b089-9617-402e-9a40-9a5d2d69be50"
time="2021-06-15T19:16:07Z" level=info msg="Creating a new serviceService.NamespaceopenebsService.Namepvc-5ea4b089-9617-402e-9a40-9a5d2d69be50-jiva-ctrl-svc"
time="2021-06-15T19:16:08Z" level=info msg="Updating JivaVolume with iscsi specISCSISpec{10.233.5.29 3260 iqn.2016-09.com.openebs.jiva:pvc-5ea4b089-9617-402e-9a40-9a5d2d69be50}"
time="2021-06-15T19:16:08Z" level=info msg="Creating a new deploymentDeploy.NamespaceopenebsDeploy.Namepvc-5ea4b089-9617-402e-9a40-9a5d2d69be50-jiva-ctrl"
time="2021-06-15T19:16:08Z" level=info msg="Creating a new StatefulsetStatefulset.NamespaceopenebsSts.Namepvc-5ea4b089-9617-402e-9a40-9a5d2d69be50-jiva-rep"
time="2021-06-15T19:16:08Z" level=info msg="Creating a new pod disruption budgetPdb.NamespaceopenebsPdb.Namepvc-5ea4b089-9617-402e-9a40-9a5d2d69be50-pdb"
time="2021-06-15T19:16:08Z" level=info msg="Failed to get volume stats errGet "http://10.233.49.69:9501/v1/stats\": dial tcp 10.233.49.69:9501: connect: connection refused"
time="2021-06-15T19:16:08Z" level=info msg="Failed to get volume stats errGet "http://10.233.5.29:9501/v1/stats\": dial tcp 10.233.5.29:9501: connect: connection refused"
time="2021-06-15T19:16:09Z" level=info msg="Failed to get volume stats errGet "http://10.233.49.69:9501/v1/stats\": dial tcp 10.233.49.69:9501: connect: connection refused"
time="2021-06-15T19:16:09Z" level=info msg="Failed to get volume stats errGet "http://10.233.5.29:9501/v1/stats\": dial tcp 10.233.5.29:9501: connect: connection refused"
time="2021-06-15T19:16:11Z" level=info msg="Failed to get volume stats errGet "http://10.233.5.29:9501/v1/stats\": dial tcp 10.233.5.29:9501: connect: connection refused"

apiVersion: openebs.io/v1alpha1
kind: JivaVolume
metadata:
  annotations:
    openebs.io/volume-policy: example-jivavolumepolicy
  creationTimestamp: "2021-06-15T19:16:05Z"
  generation: 5
  labels:
    openebs.io/component: jiva-volume
    openebs.io/persistent-volume: pvc-5ea4b089-9617-402e-9a40-9a5d2d69be50
  managedFields:
  - apiVersion: openebs.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:openebs.io/volume-policy: {}
        f:labels:
          .: {}
          f:openebs.io/component: {}
          f:openebs.io/persistent-volume: {}
      f:spec:
        .: {}
        f:accessType: {}
        f:capacity: {}
        f:iscsiSpec: {}
        f:mountInfo: {}
        f:policy:
          .: {}
          f:autoScaling: {}
          f:enableBufio: {}
          f:replica: {}
          f:target: {}
        f:pv: {}
      f:status: {}
      f:versionDetails:
        .: {}
        f:desired: {}
        f:status:
          .: {}
          f:current: {}
          f:dependentsUpgraded: {}
          f:lastUpdateTime: {}
    manager: jiva-csi
    operation: Update
    time: "2021-06-15T19:16:05Z"
  - apiVersion: openebs.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        f:desiredReplicationFactor: {}
        f:iscsiSpec:
          f:iqn: {}
          f:targetIP: {}
          f:targetPort: {}
        f:policy:
          f:replica:
            f:resources:
              .: {}
              f:limits:
                .: {}
                f:cpu: {}
                f:memory: {}
              f:requests:
                .: {}
                f:cpu: {}
                f:memory: {}
            f:tolerations: {}
          f:replicaSC: {}
          f:target:
            f:auxResources:
              .: {}
              f:limits:
                .: {}
                f:cpu: {}
                f:memory: {}
              f:requests:
                .: {}
                f:cpu: {}
                f:memory: {}
            f:replicationFactor: {}
            f:resources:
              .: {}
              f:limits:
                .: {}
                f:cpu: {}
                f:memory: {}
              f:requests:
                .: {}
                f:cpu: {}
                f:memory: {}
            f:tolerations: {}
      f:status:
        f:phase: {}
        f:status: {}
    manager: jiva-operator
    operation: Update
    time: "2021-06-15T19:16:11Z"
  name: pvc-5ea4b089-9617-402e-9a40-9a5d2d69be50
  namespace: openebs
  resourceVersion: "91179"
  uid: 5d514f46-185d-4e91-aa48-2f3b4edc0a51
spec:
  accessType: mount
  capacity: 4Gi
  desiredReplicationFactor: 1
  iscsiSpec:
    iqn: iqn.2016-09.com.openebs.jiva:pvc-5ea4b089-9617-402e-9a40-9a5d2d69be50
    targetIP: 10.233.5.29
    targetPort: 3260
  mountInfo: {}
  policy:
    autoScaling: false
    enableBufio: false
    replica:
      resources:
        limits:
          cpu: "0"
          memory: "0"
        requests:
          cpu: "0"
          memory: "0"
      tolerations:
      - effect: NoExecute
        key: node.kubernetes.io/notReady
        operator: Exists
      - effect: NoExecute
        key: node.cloudprovider.kubernetes.io/uninitialized
        operator: Exists
      - effect: NoExecute
        key: node.kubernetes.io/unreachable
        operator: Exists
      - effect: NoExecute
        key: node.kubernetes.io/not-ready
        operator: Exists
      - effect: NoExecute
        key: node.kubernetes.io/unschedulable
        operator: Exists
      - effect: NoExecute
        key: node.kubernetes.io/out-of-disk
        operator: Exists
      - effect: NoExecute
        key: node.kubernetes.io/memory-pressure
        operator: Exists
      - effect: NoExecute
        key: node.kubernetes.io/disk-pressure
        operator: Exists
      - effect: NoExecute
        key: node.kubernetes.io/network-unavailable
        operator: Exists
    replicaSC: openebs-hostpath
    target:
      auxResources:
        limits:
          cpu: "0"
          memory: "0"
        requests:
          cpu: "0"
          memory: "0"
      replicationFactor: 1
      resources:
        limits:
          cpu: "0"
          memory: "0"
        requests:
          cpu: "0"
          memory: "0"
      tolerations:
      - effect: NoExecute
        key: node.kubernetes.io/notReady
        operator: Exists
        tolerationSeconds: 0
      - effect: NoExecute
        key: node.kubernetes.io/unreachable
        operator: Exists
        tolerationSeconds: 0
      - effect: NoExecute
        key: node.kubernetes.io/not-ready
        operator: Exists
        tolerationSeconds: 0
  pv: pvc-5ea4b089-9617-402e-9a40-9a5d2d69be50
status:
  phase: Syncing
  status: RO
versionDetails:
  desired: 2.9.0
  status:
    current: 2.9.0
    dependentsUpgraded: true
    lastUpdateTime: null

apiVersion: v1
items:
- apiVersion: openebs.io/v1alpha1
  kind: JivaVolumePolicy
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"openebs.io/v1alpha1","kind":"JivaVolumePolicy","metadata":{"annotations":{},"name":"example-jivavolumepolicy","namespace":"openebs"},"spec":{"autoScaling":false,"enableBufio":false,"replicaSC":"openebs-hostpath","target":{"replicationFactor":1}}}
    creationTimestamp: "2021-06-15T18:51:32Z"
    generation: 1
    managedFields:
    - apiVersion: openebs.io/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:kubectl.kubernetes.io/last-applied-configuration: {}
        f:spec:
          .: {}
          f:autoScaling: {}
          f:enableBufio: {}
          f:replicaSC: {}
          f:target:
            .: {}
            f:replicationFactor: {}
      manager: kubectl-client-side-apply
      operation: Update
      time: "2021-06-15T18:51:32Z"
    name: example-jivavolumepolicy
    namespace: openebs
    resourceVersion: "83139"
    uid: 36b95f5f-3b79-48a6-bec3-1e5f54b76760
  spec:
    autoScaling: false
    enableBufio: false
    replicaSC: openebs-hostpath
    target:
      replicationFactor: 1
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:
@cleveritcz
Copy link
Author

[root@node1 opt]# kubectl get jv pvc-5ea4b089-9617-402e-9a40-9a5d2d69be50 -n openebs

NAME REPLICACOUNT PHASE STATUS
pvc-5ea4b089-9617-402e-9a40-9a5d2d69be50 Syncing RO

@cleveritcz
Copy link
Author

[root@node1 opt]# kubectl get po,pvc
NAME READY STATUS RESTARTS AGE
pod/fio-67664bfd9b-n9q4x 0/1 ContainerCreating 0 17m

NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/example-jiva-csi-pvc Bound pvc-5ea4b089-9617-402e-9a40-9a5d2d69be50 4Gi RWO openebs-jiva-csi-sc 18m

@shubham14bajpai
Copy link
Contributor

shubham14bajpai commented Jun 16, 2021 •
edited
Loading

Hi @cleveritcz Can you please share the output for kubectl get pods -n openebs. I suspect the replica pod is not up.

@cleveritcz
Copy link
Author

[root@node1 opt]# kubectl get pods -n openebs
NAME READY STATUS RESTARTS AGE
jiva-operator-6b69b86894-dr4p6 1/1 Running 0 3m24s
openebs-jiva-csi-controller-0 5/5 Running 0 3m23s
openebs-jiva-csi-node-jj9ff 3/3 Running 0 3m22s
openebs-jiva-csi-node-l9v6n 3/3 Running 0 3m22s
openebs-localpv-provisioner-7f85dcbb75-z7kdb 1/1 Running 0 3m44s

@shubham14bajpai
Copy link
Contributor

Yes I don't see the replica and controller pod here. Can you share the output for kubectl -n openebs get all

@cleveritcz
Copy link
Author

[root@node1 opt]# kubectl -n openebs get all
NAME READY STATUS RESTARTS AGE
pod/jiva-operator-6b69b86894-dr4p6 1/1 Running 0 6m21s
pod/openebs-jiva-csi-controller-0 5/5 Running 0 6m20s
pod/openebs-jiva-csi-node-jj9ff 3/3 Running 0 6m19s
pod/openebs-jiva-csi-node-l9v6n 3/3 Running 0 6m19s
pod/openebs-localpv-provisioner-7f85dcbb75-z7kdb 1/1 Running 0 6m41s

NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/openebs-jiva-csi-node 2 2 2 2 2 6m19s

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/jiva-operator 1/1 1 1 6m21s
deployment.apps/openebs-localpv-provisioner 1/1 1 1 6m41s

NAME DESIRED CURRENT READY AGE
replicaset.apps/jiva-operator-6b69b86894 1 1 1 6m21s
replicaset.apps/openebs-localpv-provisioner-7f85dcbb75 1 1 1 6m41s

NAME READY AGE
statefulset.apps/openebs-jiva-csi-controller 1/1 6m20s

@cleveritcz
Copy link
Author

I used these commands:

kubectl apply -f https://openebs.github.io/charts/hostpath-operator.yaml

kubectl apply -f https://openebs.github.io/charts/jiva-operator.yaml

I don't know why storageClass has to be twice

@shubham14bajpai
Copy link
Contributor

shubham14bajpai commented Jun 16, 2021 •
edited
Loading 

time="2021-06-15T19:16:07Z" level=info msg="Creating a new serviceService.NamespaceopenebsService.Namepvc-5ea4b089-9617-402e-9a40-9a5d2d69be50-jiva-ctrl-svc"
time="2021-06-15T19:16:08Z" level=info msg="Updating JivaVolume with iscsi specISCSISpec{10.233.5.29 3260 iqn.2016-09.com.openebs.jiva:pvc-5ea4b089-9617-402e-9a40-9a5d2d69be50}"
time="2021-06-15T19:16:08Z" level=info msg="Creating a new deploymentDeploy.NamespaceopenebsDeploy.Namepvc-5ea4b089-9617-402e-9a40-9a5d2d69be50-jiva-ctrl"
time="2021-06-15T19:16:08Z" level=info msg="Creating a new StatefulsetStatefulset.NamespaceopenebsSts.Namepvc-5ea4b089-9617-402e-9a40-9a5d2d69be50-jiva-rep"

Was the volume deleted later.

I see in the logs the resources got created but are missing from openebs namespace. The host-path storageclass is for the replica STS which is used for local hostpath storage for each replica
Also can we move this conversation to k8s slack #openebs channel.

@cleveritcz
Copy link
Author

I am sorry, I can't move to slack. I would like to get work it.

@shubham14bajpai
Copy link
Contributor

No problem. Can you delete the old volumes and apply the hostpath storageclass mentioned in the quickstart guide. Then it should work

@cleveritcz
Copy link
Author

I deleted pv, pvc and hostpath storage class and applied again hostpath storage class.

[root@node1 opt]# kubectl -n openebs get all
NAME READY STATUS RESTARTS AGE
pod/jiva-operator-6b69b86894-dr4p6 1/1 Running 0 40m
pod/openebs-jiva-csi-controller-0 5/5 Running 0 40m
pod/openebs-jiva-csi-node-jj9ff 3/3 Running 0 40m
pod/openebs-jiva-csi-node-l9v6n 3/3 Running 0 40m
pod/openebs-localpv-provisioner-7f85dcbb75-z7kdb 1/1 Running 0 40m

NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/openebs-jiva-csi-node 2 2 2 2 2 40m

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/jiva-operator 1/1 1 1 40m
deployment.apps/openebs-localpv-provisioner 1/1 1 1 40m

NAME DESIRED CURRENT READY AGE
replicaset.apps/jiva-operator-6b69b86894 1 1 1 40m
replicaset.apps/openebs-localpv-provisioner-7f85dcbb75 1 1 1 40m

NAME READY AGE
statefulset.apps/openebs-jiva-csi-controller 1/1 40m

@shubham14bajpai
Copy link
Contributor

I still don't see the ctrl and rep pods for the volume. Can you share the logs for jiva-operator pod.

@cleveritcz
Copy link
Author

[root@node1 opt]# kubectl logs -f jiva-operator-6b69b86894-dr4p6 -n openebs
time="2021-06-16T06:53:26Z" level=info msg="Go Version: go1.14.7"
time="2021-06-16T06:53:26Z" level=info msg="Go OS/Arch: linux/amd64"
time="2021-06-16T06:53:26Z" level=info msg="Version of jiva-operator: 2.9.0"
time="2021-06-16T07:37:44Z" level=info msg="start bootstraping jiva componentsJivaVolume: pvc-b657adee-3fde-4800-bafd-9d6cde98f8a1"
time="2021-06-16T07:37:44Z" level=info msg="Creating a new serviceService.NamespaceopenebsService.Namepvc-b657adee-3fde-4800-bafd-9d6cde98f8a1-jiva-ctrl-svc"
time="2021-06-16T07:37:45Z" level=info msg="Updating JivaVolume with iscsi specISCSISpec{10.233.36.39 3260 iqn.2016-09.com.openebs.jiva:pvc-b657adee-3fde-4800-bafd-9d6cde98f8a1}"
time="2021-06-16T07:37:45Z" level=info msg="Creating a new deploymentDeploy.NamespaceopenebsDeploy.Namepvc-b657adee-3fde-4800-bafd-9d6cde98f8a1-jiva-ctrl"
time="2021-06-16T07:37:45Z" level=info msg="Creating a new StatefulsetStatefulset.NamespaceopenebsSts.Namepvc-b657adee-3fde-4800-bafd-9d6cde98f8a1-jiva-rep"
time="2021-06-16T07:37:45Z" level=info msg="Creating a new pod disruption budgetPdb.NamespaceopenebsPdb.Namepvc-b657adee-3fde-4800-bafd-9d6cde98f8a1-pdb"
time="2021-06-16T07:37:45Z" level=info msg="Failed to get volume stats errGet "http://10.233.36.39:9501/v1/stats\": dial tcp 10.233.36.39:9501: connect: connection refused"
time="2021-06-16T07:37:45Z" level=info msg="Failed to get volume stats errGet "http://10.233.36.39:9501/v1/stats\": dial tcp 10.233.36.39:9501: connect: connection refused"
time="2021-06-16T07:37:45Z" level=info msg="Failed to get volume stats errGet "http://10.233.36.39:9501/v1/stats\": dial tcp 10.233.36.39:9501: connect: connection refused"
time="2021-06-16T07:37:46Z" level=info msg="Failed to get volume stats errGet "http://10.233.36.39:9501/v1/stats\": dial tcp 10.233.36.39:9501: connect: connection refused"
time="2021-06-16T07:37:46Z" level=info msg="Failed to get volume stats errGet "http://10.233.36.39:9501/v1/stats\": dial tcp 10.233.36.39:9501: connect: connection refused"
time="2021-06-16T07:37:46Z" level=info msg="Failed to get volume stats errGet "http://10.233.36.39:9501/v1/stats\": dial tcp 10.233.36.39:9501: connect: connection refused"
time="2021-06-16T07:37:48Z" level=info msg="Failed to get volume stats errGet "http://10.233.36.39:9501/v1/stats\": dial tcp 10.233.36.39:9501: connect: connection refused"
time="2021-06-16T07:37:51Z" level=info msg="Failed to get volume stats errGet "http://10.233.36.39:9501/v1/stats\": dial tcp 10.233.36.39:9501: connect: connection refused"
time="2021-06-16T07:37:51Z" level=info msg="Failed to get volume stats errGet "http://10.233.36.39:9501/v1/stats\": dial tcp 10.233.36.39:9501: connect: connection refused"
time="2021-06-16T07:37:51Z" level=info msg="Failed to get volume stats errGet "http://10.233.36.39:9501/v1/stats\": dial tcp 10.233.36.39:9501: connect: connection refused"
time="2021-06-16T07:37:53Z" level=info msg="Failed to get volume stats errGet "http://10.233.36.39:9501/v1/stats\": dial tcp 10.233.36.39:9501: connect: connection refused"
time="2021-06-16T07:37:55Z" level=info msg="Failed to get volume stats errGet "http://10.233.36.39:9501/v1/stats\": dial tcp 10.233.36.39:9501: connect: connection refused"
time="2021-06-16T07:37:56Z" level=info msg="Failed to get volume stats errGet "http://10.233.36.39:9501/v1/stats\": dial tcp 10.233.36.39:9501: connect: connection refused"
time="2021-06-16T07:37:56Z" level=info msg="Failed to get volume stats errGet "http://10.233.36.39:9501/v1/stats\": dial tcp 10.233.36.39:9501: connect: connection refused"
time="2021-06-16T07:37:57Z" level=info msg="Failed to get volume stats errGet "http://10.233.36.39:9501/v1/stats\": dial tcp 10.233.36.39:9501: connect: connection refused"
time="2021-06-16T07:38:00Z" level=info msg="Failed to get volume stats errGet "http://10.233.36.39:9501/v1/stats\": dial tcp 10.233.36.39:9501: connect: connection refused"
time="2021-06-16T07:38:01Z" level=info msg="Failed to get volume stats errGet "http://10.233.36.39:9501/v1/stats\": dial tcp 10.233.36.39:9501: connect: connection refused"
time="2021-06-16T07:38:01Z" level=info msg="Failed to get volume stats errGet "http://10.233.36.39:9501/v1/stats\": dial tcp 10.233.36.39:9501: connect: connection refused"
time="2021-06-16T07:38:02Z" level=info msg="Failed to get volume stats errGet "http://10.233.36.39:9501/v1/stats\": dial tcp 10.233.36.39:9501: connect: connection refused"
time="2021-06-16T07:38:05Z" level=info msg="Failed to get volume stats errGet "http://10.233.36.39:9501/v1/stats\": dial tcp 10.233.36.39:9501: connect: connection refused"
time="2021-06-16T07:38:06Z" level=info msg="Failed to get volume stats errGet "http://10.233.36.39:9501/v1/stats\": dial tcp 10.233.36.39:9501: connect: connection refused"
time="2021-06-16T07:38:06Z" level=info msg="Failed to get volume stats errGet "http://10.233.36.39:9501/v1/stats\": dial tcp 10.233.36.39:9501: connect: connection refused"
time="2021-06-16T07:38:07Z" level=info msg="Failed to get volume stats errGet "http://10.233.36.39:9501/v1/stats\": dial tcp 10.233.36.39:9501: connect: connection refused"

@cleveritcz
Copy link
Author

[root@node1 opt]# kubectl -n openebs get all
NAME READY STATUS RESTARTS AGE
pod/jiva-operator-6b69b86894-dr4p6 1/1 Running 0 47m
pod/openebs-jiva-csi-controller-0 5/5 Running 0 47m
pod/openebs-jiva-csi-node-jj9ff 3/3 Running 0 47m
pod/openebs-jiva-csi-node-l9v6n 3/3 Running 0 47m
pod/openebs-localpv-provisioner-7f85dcbb75-z7kdb 1/1 Running 0 47m
pod/pvc-b657adee-3fde-4800-bafd-9d6cde98f8a1-jiva-ctrl-7fbf7b9kmwdb 1/1 Running 0 2m40s
pod/pvc-b657adee-3fde-4800-bafd-9d6cde98f8a1-jiva-rep-0 1/1 Running 2 2m40s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/pvc-b657adee-3fde-4800-bafd-9d6cde98f8a1-jiva-ctrl-svc ClusterIP 10.233.36.39 3260/TCP,9501/TCP,9500/TCP 2m42s

NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/openebs-jiva-csi-node 2 2 2 2 2 47m

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/jiva-operator 1/1 1 1 47m
deployment.apps/openebs-localpv-provisioner 1/1 1 1 47m
deployment.apps/pvc-b657adee-3fde-4800-bafd-9d6cde98f8a1-jiva-ctrl 1/1 1 1 2m41s

NAME DESIRED CURRENT READY AGE
replicaset.apps/jiva-operator-6b69b86894 1 1 1 47m
replicaset.apps/openebs-localpv-provisioner-7f85dcbb75 1 1 1 47m
replicaset.apps/pvc-b657adee-3fde-4800-bafd-9d6cde98f8a1-jiva-ctrl-7fbf7b95c9 1 1 1 2m40s

NAME READY AGE
statefulset.apps/openebs-jiva-csi-controller 1/1 47m
statefulset.apps/pvc-b657adee-3fde-4800-bafd-9d6cde98f8a1-jiva-rep 1/1 2m40s

@cleveritcz
Copy link
Author

[root@node1 opt]# kubectl get jivavolume -n openebs
NAME REPLICACOUNT PHASE STATUS
pvc-b657adee-3fde-4800-bafd-9d6cde98f8a1 Syncing RO

@shubham14bajpai
Copy link
Contributor

Okay now the pods got created. Can you share the output for -oyaml of the jv resource? How many nodes do are there in this cluster? Also is the networking between the nodes working properly?

@cleveritcz
Copy link
Author 

apiVersion: v1
items:
- apiVersion: openebs.io/v1alpha1
  kind: JivaVolume
  metadata:
    annotations:
      openebs.io/volume-policy: example-jivavolumepolicy
    creationTimestamp: "2021-06-16T07:37:43Z"
    generation: 5
    labels:
      openebs.io/component: jiva-volume
      openebs.io/persistent-volume: pvc-b657adee-3fde-4800-bafd-9d6cde98f8a1
    managedFields:
    - apiVersion: openebs.io/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:openebs.io/volume-policy: {}
          f:labels:
            .: {}
            f:openebs.io/component: {}
            f:openebs.io/persistent-volume: {}
        f:spec:
          .: {}
          f:accessType: {}
          f:capacity: {}
          f:iscsiSpec: {}
          f:mountInfo: {}
          f:policy:
            .: {}
            f:autoScaling: {}
            f:enableBufio: {}
            f:replica: {}
            f:target: {}
          f:pv: {}
        f:status: {}
        f:versionDetails:
          .: {}
          f:desired: {}
          f:status:
            .: {}
            f:current: {}
            f:dependentsUpgraded: {}
            f:lastUpdateTime: {}
      manager: jiva-csi
      operation: Update
      time: "2021-06-16T07:37:43Z"
    - apiVersion: openebs.io/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        f:spec:
          f:desiredReplicationFactor: {}
          f:iscsiSpec:
            f:iqn: {}
            f:targetIP: {}
            f:targetPort: {}
          f:policy:
            f:replica:
              f:resources:
                .: {}
                f:limits:
                  .: {}
                  f:cpu: {}
                  f:memory: {}
                f:requests:
                  .: {}
                  f:cpu: {}
                  f:memory: {}
              f:tolerations: {}
            f:replicaSC: {}
            f:target:
              f:auxResources:
                .: {}
                f:limits:
                  .: {}
                  f:cpu: {}
                  f:memory: {}
                f:requests:
                  .: {}
                  f:cpu: {}
                  f:memory: {}
              f:replicationFactor: {}
              f:resources:
                .: {}
                f:limits:
                  .: {}
                  f:cpu: {}
                  f:memory: {}
                f:requests:
                  .: {}
                  f:cpu: {}
                  f:memory: {}
              f:tolerations: {}
        f:status:
          f:phase: {}
          f:status: {}
      manager: jiva-operator
      operation: Update
      time: "2021-06-16T07:38:08Z"
    name: pvc-b657adee-3fde-4800-bafd-9d6cde98f8a1
    namespace: openebs
    resourceVersion: "304562"
    uid: e6797bee-d4ac-4907-8c3d-2b73ff45c744
  spec:
    accessType: mount
    capacity: 4Gi
    desiredReplicationFactor: 1
    iscsiSpec:
      iqn: iqn.2016-09.com.openebs.jiva:pvc-b657adee-3fde-4800-bafd-9d6cde98f8a1
      targetIP: 10.233.36.39
      targetPort: 3260
    mountInfo: {}
    policy:
      autoScaling: false
      enableBufio: false
      replica:
        resources:
          limits:
            cpu: "0"
            memory: "0"
          requests:
            cpu: "0"
            memory: "0"
        tolerations:
        - effect: NoExecute
          key: node.kubernetes.io/notReady
          operator: Exists
        - effect: NoExecute
          key: node.cloudprovider.kubernetes.io/uninitialized
          operator: Exists
        - effect: NoExecute
          key: node.kubernetes.io/unreachable
          operator: Exists
        - effect: NoExecute
          key: node.kubernetes.io/not-ready
          operator: Exists
        - effect: NoExecute
          key: node.kubernetes.io/unschedulable
          operator: Exists
        - effect: NoExecute
          key: node.kubernetes.io/out-of-disk
          operator: Exists
        - effect: NoExecute
          key: node.kubernetes.io/memory-pressure
          operator: Exists
        - effect: NoExecute
          key: node.kubernetes.io/disk-pressure
          operator: Exists
        - effect: NoExecute
          key: node.kubernetes.io/network-unavailable
          operator: Exists
      replicaSC: openebs-hostpath
      target:
        auxResources:
          limits:
            cpu: "0"
            memory: "0"
          requests:
            cpu: "0"
            memory: "0"
        replicationFactor: 1
        resources:
          limits:
            cpu: "0"
            memory: "0"
          requests:
            cpu: "0"
            memory: "0"
        tolerations:
        - effect: NoExecute
          key: node.kubernetes.io/notReady
          operator: Exists
          tolerationSeconds: 0
        - effect: NoExecute
          key: node.kubernetes.io/unreachable
          operator: Exists
          tolerationSeconds: 0
        - effect: NoExecute
          key: node.kubernetes.io/not-ready
          operator: Exists
          tolerationSeconds: 0
    pv: pvc-b657adee-3fde-4800-bafd-9d6cde98f8a1
  status:
    phase: Syncing
    status: RO
  versionDetails:
    desired: 2.9.0
    status:
      current: 2.9.0
      dependentsUpgraded: true
      lastUpdateTime: null
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

3 nodes in cluster (1 master/2 workers). I don't know if networking is ok, I deployed it by kubespray with Cilium network plugin.

@shubham14bajpai
Copy link
Contributor

So the volume got created here. It is just that the ctrl and rep cannot connect due to network issue. dial tcp 10.233.36.39:9501: connect: connection refused". Can you share the output for kubectl -n kube-system get all

@cleveritcz
Copy link
Author

[root@node1 opt]# kubectl -n kube-system get all
NAME READY STATUS RESTARTS AGE
pod/cilium-2vzjl 1/1 Running 0 17h
pod/cilium-45w7c 1/1 Running 0 17h
pod/cilium-5zp7n 1/1 Running 0 17h
pod/cilium-operator-799486c86f-2tqts 1/1 Running 0 17h
pod/coredns-657959df74-ksmgw 1/1 Running 0 17h
pod/coredns-657959df74-lf782 1/1 Running 0 17h
pod/dns-autoscaler-b5c786945-4hsrs 1/1 Running 0 17h
pod/etcd-node1 1/1 Running 0 17h
pod/kube-apiserver-node1 1/1 Running 0 17h
pod/kube-controller-manager-node1 1/1 Running 0 17h
pod/kube-proxy-4dp4q 1/1 Running 0 17h
pod/kube-proxy-l6kpv 1/1 Running 0 17h
pod/kube-proxy-vtkfb 1/1 Running 0 17h
pod/kube-scheduler-node1 1/1 Running 0 17h
pod/metrics-server-5cd75b7749-jqq49 2/2 Running 0 17h
pod/nginx-proxy-node2 1/1 Running 0 17h
pod/nginx-proxy-node3 1/1 Running 0 17h
pod/nodelocaldns-9s5fm 1/1 Running 0 17h
pod/nodelocaldns-p5vfs 1/1 Running 0 17h
pod/nodelocaldns-pp2vh 1/1 Running 0 17h
pod/registry-proxy-c6zlj 1/1 Running 0 17h
pod/registry-proxy-dvzbc 1/1 Running 0 17h
pod/registry-proxy-gqql7 1/1 Running 0 17h
pod/registry-xxbbh 1/1 Running 0 17h

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/coredns ClusterIP 10.233.0.3 53/UDP,53/TCP,9153/TCP 17h
service/metrics-server ClusterIP 10.233.33.63 443/TCP 17h
service/registry ClusterIP 10.233.37.125 5000/TCP 17h

NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/cilium 3 3 3 3 3 17h
daemonset.apps/kube-proxy 3 3 3 3 3 kubernetes.io/os=linux 17h
daemonset.apps/nodelocaldns 3 3 3 3 3 17h
daemonset.apps/registry-proxy 2 2 2 2 2 17h

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/cilium-operator 1/1 1 1 17h
deployment.apps/coredns 2/2 2 2 17h
deployment.apps/dns-autoscaler 1/1 1 1 17h
deployment.apps/metrics-server 1/1 1 1 17h

NAME DESIRED CURRENT READY AGE
replicaset.apps/cilium-operator-799486c86f 1 1 1 17h
replicaset.apps/coredns-657959df74 2 2 2 17h
replicaset.apps/dns-autoscaler-b5c786945 1 1 1 17h
replicaset.apps/metrics-server-5cd75b7749 1 1 1 17h
replicaset.apps/registry 1 1 1 17h

@shubham14bajpai
Copy link
Contributor

I haven't used kubespray or cilium but was looking for similar issues. Found this kubernetes-sigs/kubespray#4357 (comment)
Setting the ingress_nginx_host_network: true within addons.yml may help.

@cleveritcz
Copy link
Author

ok, I will try to redeploy it.

@cleveritcz
Copy link
Author

I redeployed it with ingress_nginx_host_network: true and still is not working

@cleveritcz
Copy link
Author

[root@node1 ~]# kubectl logs -f jiva-operator-6b69b86894-wbkps -n openebs
I0616 08:45:37.132699 1 request.go:655] Throttling request took 1.046913956s, request: GET:https://10.233.0.1:443/apis/acme.cert-manager.io/v1alpha2?timeout=32s
time="2021-06-16T08:45:38Z" level=info msg="Go Version: go1.14.7"
time="2021-06-16T08:45:38Z" level=info msg="Go OS/Arch: linux/amd64"
time="2021-06-16T08:45:38Z" level=info msg="Version of jiva-operator: 2.9.0"
time="2021-06-16T08:47:05Z" level=info msg="start bootstraping jiva componentsJivaVolume: pvc-8a754aae-4a0f-48ca-a237-fd88f4089254"
time="2021-06-16T08:47:05Z" level=info msg="Creating a new serviceService.NamespaceopenebsService.Namepvc-8a754aae-4a0f-48ca-a237-fd88f4089254-jiva-ctrl-svc"
time="2021-06-16T08:47:06Z" level=info msg="Updating JivaVolume with iscsi specISCSISpec{10.233.17.180 3260 iqn.2016-09.com.openebs.jiva:pvc-8a754aae-4a0f-48ca-a237-fd88f4089254}"
time="2021-06-16T08:47:06Z" level=info msg="Creating a new deploymentDeploy.NamespaceopenebsDeploy.Namepvc-8a754aae-4a0f-48ca-a237-fd88f4089254-jiva-ctrl"
time="2021-06-16T08:47:06Z" level=info msg="Creating a new StatefulsetStatefulset.NamespaceopenebsSts.Namepvc-8a754aae-4a0f-48ca-a237-fd88f4089254-jiva-rep"
time="2021-06-16T08:47:07Z" level=info msg="Creating a new pod disruption budgetPdb.NamespaceopenebsPdb.Namepvc-8a754aae-4a0f-48ca-a237-fd88f4089254-pdb"
time="2021-06-16T08:47:07Z" level=info msg="Failed to get volume stats errGet "http://10.233.17.180:9501/v1/stats\": dial tcp 10.233.17.180:9501: connect: connection refused"
time="2021-06-16T08:47:07Z" level=info msg="Failed to get volume stats errGet "http://10.233.17.180:9501/v1/stats\": dial tcp 10.233.17.180:9501: connect: connection refused"
time="2021-06-16T08:47:07Z" level=error msg="failed to update JivaVolume, err: Operation cannot be fulfilled on jivavolumes.openebs.io "pvc-8a754aae-4a0f-48ca-a237-fd88f4089254": the object has been modified; please apply your changes to the latest version and try againfailed to update status"
time="2021-06-16T08:47:07Z" level=info msg="Failed to get volume stats errGet "http://10.233.17.180:9501/v1/stats\": dial tcp 10.233.17.180:9501: connect: connection refused"
time="2021-06-16T08:47:10Z" level=info msg="Failed to get volume stats errGet "http://10.233.17.180:9501/v1/stats\": dial tcp 10.233.17.180:9501: connect: connection refused"
time="2021-06-16T08:47:10Z" level=info msg="Failed to get volume stats errGet "http://10.233.17.180:9501/v1/stats\": dial tcp 10.233.17.180:9501: connect: connection refused"
time="2021-06-16T08:47:11Z" level=info msg="Failed to get volume stats errGet "http://10.233.17.180:9501/v1/stats\": dial tcp 10.233.17.180:9501: connect: connection refused"
time="2021-06-16T08:47:12Z" level=info msg="Failed to get volume stats errGet "http://10.233.17.180:9501/v1/stats\": dial tcp 10.233.17.180:9501: connect: connection refused"
time="2021-06-16T08:47:14Z" level=info msg="Failed to get volume stats errGet "http://10.233.17.180:9501/v1/stats\": dial tcp 10.233.17.180:9501: connect: connection refused"
time="2021-06-16T08:47:15Z" level=info msg="Failed to get volume stats errGet "http://10.233.17.180:9501/v1/stats\": dial tcp 10.233.17.180:9501: connect: connection refused"
time="2021-06-16T08:47:16Z" level=info msg="Failed to get volume stats errGet "http://10.233.17.180:9501/v1/stats\": dial tcp 10.233.17.180:9501: connect: connection refused"
time="2021-06-16T08:47:17Z" level=info msg="Failed to get volume stats errGet "http://10.233.17.180:9501/v1/stats\": dial tcp 10.233.17.180:9501: connect: connection refused"
time="2021-06-16T08:47:19Z" level=info msg="Failed to get volume stats errGet "http://10.233.17.180:9501/v1/stats\": dial tcp 10.233.17.180:9501: connect: connection refused"
time="2021-06-16T08:47:20Z" level=info msg="Failed to get volume stats errGet "http://10.233.17.180:9501/v1/stats\": dial tcp 10.233.17.180:9501: connect: connection refused"
time="2021-06-16T08:47:21Z" level=info msg="Failed to get volume stats errGet "http://10.233.17.180:9501/v1/stats\": dial tcp 10.233.17.180:9501: connect: connection refused"
time="2021-06-16T08:47:22Z" level=info msg="Failed to get volume stats errGet "http://10.233.17.180:9501/v1/stats\": dial tcp 10.233.17.180:9501: connect: connection refused"
time="2021-06-16T08:47:23Z" level=info msg="Failed to get volume stats errGet "http://10.233.17.180:9501/v1/stats\": dial tcp 10.233.17.180:9501: connect: connection refused"
time="2021-06-16T08:47:25Z" level=info msg="Failed to get volume stats errGet "http://10.233.17.180:9501/v1/stats\": dial tcp 10.233.17.180:9501: connect: connection refused"
time="2021-06-16T08:47:27Z" level=info msg="Failed to get volume stats errGet "http://10.233.17.180:9501/v1/stats\": dial tcp 10.233.17.180:9501: connect: connection refused"
time="2021-06-16T08:47:27Z" level=info msg="Failed to get volume stats errGet "http://10.233.17.180:9501/v1/stats\": dial tcp 10.233.17.180:9501: connect: connection refused"
time="2021-06-16T08:47:28Z" level=info msg="Failed to get volume stats errGet "http://10.233.17.180:9501/v1/stats\": dial tcp 10.233.17.180:9501: connect: connection refused"
time="2021-06-16T08:47:30Z" level=info msg="Failed to get volume stats errGet "http://10.233.17.180:9501/v1/stats\": dial tcp 10.233.17.180:9501: connect: connection refused"
time="2021-06-16T08:47:32Z" level=info msg="Failed to get volume stats errGet "http://10.233.17.180:9501/v1/stats\": dial tcp 10.233.17.180:9501: connect: connection refused"
time="2021-06-16T08:47:32Z" level=info msg="Failed to get volume stats errGet "http://10.233.17.180:9501/v1/stats\": dial tcp 10.233.17.180:9501: connect: connection refused"
time="2021-06-16T08:47:32Z" level=info msg="Failed to get volume stats errGet "http://10.233.17.180:9501/v1/stats\": dial tcp 10.233.17.180:9501: connect: connection refused"
time="2021-06-16T08:47:35Z" level=info msg="Failed to get volume stats errGet "http://10.233.17.180:9501/v1/stats\": dial tcp 10.233.17.180:9501: connect: connection refused"
time="2021-06-16T08:47:36Z" level=info msg="Failed to get volume stats errGet "http://10.233.17.180:9501/v1/stats\": dial tcp 10.233.17.180:9501: connect: connection refused"
time="2021-06-16T08:47:37Z" level=info msg="Failed to get volume stats errGet "http://10.233.17.180:9501/v1/stats\": dial tcp 10.233.17.180:9501: connect: connection refused"
time="2021-06-16T08:47:37Z" level=info msg="Failed to get volume stats errGet "http://10.233.17.180:9501/v1/stats\": dial tcp 10.233.17.180:9501: connect: connection refused"

@cleveritcz
Copy link
Author

maybe some rbac is missing

@shubham14bajpai
Copy link
Contributor

Can you describe the pod pvc-b657adee-3fde-4800-bafd-9d6cde98f8a1-jiva-rep-0 also check if it is getting restarted again and again.

@cleveritcz
Copy link
Author

root@node1 ~]# kubectl logs -f pvc-8a754aae-4a0f-48ca-a237-fd88f4089254-jiva-rep-0 -n openebs
time="2021-06-16T09:09:28Z" level=info msg="MAX_CHAIN_LENGTH env not set, default value is 512"
time="2021-06-16T09:09:28Z" level=info msg="Read log info"
time="2021-06-16T09:09:28Z" level=info msg="Configured logging with retentionPeriod: 180, maxLogFileSize: 100, maxBackups: 5"
time="2021-06-16T09:09:28Z" level=info msg="Starting replica having replicaType: , frontendIP: pvc-8a754aae-4a0f-48ca-a237-fd88f4089254-jiva-ctrl-svc.openebs.svc.cluster.local, size: 4294967296, dir: openebs"
time="2021-06-16T09:09:28Z" level=info msg="Setting replicaAddr: 10.233.65.129:9502, controlAddr: 10.233.65.129:9502, dataAddr: 10.233.65.129:9503, syncAddr: 10.233.65.129:9504"
time="2021-06-16T09:09:28Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2021-06-16T09:09:28Z" level=info msg="Listening on data 10.233.65.129:9503"
time="2021-06-16T09:09:28Z" level=info msg="Listening on control 10.233.65.129:9502"
time="2021-06-16T09:09:28Z" level=warning msg="Failed to check replica state, err: Get "http://pvc-8a754aae-4a0f-48ca-a237-fd88f4089254-jiva-ctrl-svc.openebs.svc.cluster.local:9501/v1/replicas\": dial tcp: lookup pvc-8a754aae-4a0f-48ca-a237-fd88f4089254-jiva-ctrl-svc.openebs.svc.cluster.local: no such host, will retry (9 retry left)"
time="2021-06-16T09:09:28Z" level=info msg="Listening on sync 10.233.65.129:9504 start: 9700 end: 9800"
time="2021-06-16T09:09:30Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2021-06-16T09:09:32Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2021-06-16T09:09:33Z" level=warning msg="Failed to check replica state, err: Get "http://pvc-8a754aae-4a0f-48ca-a237-fd88f4089254-jiva-ctrl-svc.openebs.svc.cluster.local:9501/v1/replicas\": dial tcp: lookup pvc-8a754aae-4a0f-48ca-a237-fd88f4089254-jiva-ctrl-svc.openebs.svc.cluster.local: no such host, will retry (8 retry left)"
time="2021-06-16T09:09:34Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2021-06-16T09:09:36Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2021-06-16T09:09:38Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2021-06-16T09:09:38Z" level=warning msg="Failed to check replica state, err: Get "http://pvc-8a754aae-4a0f-48ca-a237-fd88f4089254-jiva-ctrl-svc.openebs.svc.cluster.local:9501/v1/replicas\": dial tcp: lookup pvc-8a754aae-4a0f-48ca-a237-fd88f4089254-jiva-ctrl-svc.openebs.svc.cluster.local: no such host, will retry (7 retry left)"
time="2021-06-16T09:09:40Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2021-06-16T09:09:42Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2021-06-16T09:09:43Z" level=warning msg="Failed to check replica state, err: Get "http://pvc-8a754aae-4a0f-48ca-a237-fd88f4089254-jiva-ctrl-svc.openebs.svc.cluster.local:9501/v1/replicas\": dial tcp: lookup pvc-8a754aae-4a0f-48ca-a237-fd88f4089254-jiva-ctrl-svc.openebs.svc.cluster.local: no such host, will retry (6 retry left)"
time="2021-06-16T09:09:44Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2021-06-16T09:09:46Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2021-06-16T09:09:48Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2021-06-16T09:09:48Z" level=warning msg="Failed to check replica state, err: Get "http://pvc-8a754aae-4a0f-48ca-a237-fd88f4089254-jiva-ctrl-svc.openebs.svc.cluster.local:9501/v1/replicas\": dial tcp: lookup pvc-8a754aae-4a0f-48ca-a237-fd88f4089254-jiva-ctrl-svc.openebs.svc.cluster.local: no such host, will retry (5 retry left)"
time="2021-06-16T09:09:50Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2021-06-16T09:09:52Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2021-06-16T09:09:53Z" level=warning msg="Failed to check replica state, err: Get "http://pvc-8a754aae-4a0f-48ca-a237-fd88f4089254-jiva-ctrl-svc.openebs.svc.cluster.local:9501/v1/replicas\": dial tcp: lookup pvc-8a754aae-4a0f-48ca-a237-fd88f4089254-jiva-ctrl-svc.openebs.svc.cluster.local: no such host, will retry (4 retry left)"
time="2021-06-16T09:09:54Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2021-06-16T09:09:56Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2021-06-16T09:09:58Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2021-06-16T09:09:58Z" level=warning msg="Failed to check replica state, err: Get "http://pvc-8a754aae-4a0f-48ca-a237-fd88f4089254-jiva-ctrl-svc.openebs.svc.cluster.local:9501/v1/replicas\": dial tcp: lookup pvc-8a754aae-4a0f-48ca-a237-fd88f4089254-jiva-ctrl-svc.openebs.svc.cluster.local: no such host, will retry (3 retry left)"
time="2021-06-16T09:10:00Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2021-06-16T09:10:02Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2021-06-16T09:10:03Z" level=warning msg="Failed to check replica state, err: Get "http://pvc-8a754aae-4a0f-48ca-a237-fd88f4089254-jiva-ctrl-svc.openebs.svc.cluster.local:9501/v1/replicas\": dial tcp: lookup pvc-8a754aae-4a0f-48ca-a237-fd88f4089254-jiva-ctrl-svc.openebs.svc.cluster.local: no such host, will retry (2 retry left)"
time="2021-06-16T09:10:04Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2021-06-16T09:10:06Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2021-06-16T09:10:08Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2021-06-16T09:10:08Z" level=warning msg="Failed to check replica state, err: Get "http://pvc-8a754aae-4a0f-48ca-a237-fd88f4089254-jiva-ctrl-svc.openebs.svc.cluster.local:9501/v1/replicas\": dial tcp: lookup pvc-8a754aae-4a0f-48ca-a237-fd88f4089254-jiva-ctrl-svc.openebs.svc.cluster.local: no such host, will retry (1 retry left)"
time="2021-06-16T09:10:10Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2021-06-16T09:10:12Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2021-06-16T09:10:13Z" level=warning msg="Failed to check replica state, err: Get "http://pvc-8a754aae-4a0f-48ca-a237-fd88f4089254-jiva-ctrl-svc.openebs.svc.cluster.local:9501/v1/replicas\": dial tcp: lookup pvc-8a754aae-4a0f-48ca-a237-fd88f4089254-jiva-ctrl-svc.openebs.svc.cluster.local: no such host, will retry (0 retry left)"
time="2021-06-16T09:10:13Z" level=fatal msg="Retry count exceeded, Shutting down..."

it is crashing
pod/pvc-8a754aae-4a0f-48ca-a237-fd88f4089254-jiva-rep-0 0/1 CrashLoopBackOff 8 27m

@cleveritcz
Copy link
Author

I don't have domain cluster.local

@shubham14bajpai
Copy link
Contributor

shubham14bajpai commented Jun 16, 2021 •
edited
Loading

Lets try a thing. Can you exec into the replica pod and try to curl the service IP for the volume.

kubectl -n openebs exec -it pvc-8a754aae-4a0f-48ca-a237-fd88f4089254-jiva-rep-0 -- bash
curl <ip-for-the-svc>:9501

This will help us rule out service discovery problems

@cleveritcz
Copy link
Author

I changed domain to cluster.local and pod is running.

I am getting still this error:
[root@node1 ~]# kubectl logs -f jiva-operator-6b69b86894-kw8mm -n openebs
I0616 12:06:58.693772 1 request.go:655] Throttling request took 1.047247456s, request: GET:https://10.233.0.1:443/apis/certificates.k8s.io/v1beta1?timeout=32s
time="2021-06-16T12:07:00Z" level=info msg="Go Version: go1.14.7"
time="2021-06-16T12:07:00Z" level=info msg="Go OS/Arch: linux/amd64"
time="2021-06-16T12:07:00Z" level=info msg="Version of jiva-operator: 2.9.0"
time="2021-06-16T12:09:05Z" level=info msg="start bootstraping jiva componentsJivaVolume: pvc-2b697d2e-e561-4acf-b264-60fd6cd392e1"
time="2021-06-16T12:09:05Z" level=info msg="Creating a new serviceService.NamespaceopenebsService.Namepvc-2b697d2e-e561-4acf-b264-60fd6cd392e1-jiva-ctrl-svc"
time="2021-06-16T12:09:06Z" level=info msg="Updating JivaVolume with iscsi specISCSISpec{10.233.57.251 3260 iqn.2016-09.com.openebs.jiva:pvc-2b697d2e-e561-4acf-b264-60fd6cd392e1}"
time="2021-06-16T12:09:06Z" level=info msg="Creating a new deploymentDeploy.NamespaceopenebsDeploy.Namepvc-2b697d2e-e561-4acf-b264-60fd6cd392e1-jiva-ctrl"
time="2021-06-16T12:09:06Z" level=info msg="Creating a new StatefulsetStatefulset.NamespaceopenebsSts.Namepvc-2b697d2e-e561-4acf-b264-60fd6cd392e1-jiva-rep"
time="2021-06-16T12:09:07Z" level=info msg="Creating a new pod disruption budgetPdb.NamespaceopenebsPdb.Namepvc-2b697d2e-e561-4acf-b264-60fd6cd392e1-pdb"
time="2021-06-16T12:09:07Z" level=info msg="Failed to get volume stats errGet "http://10.233.57.251:9501/v1/stats\": dial tcp 10.233.57.251:9501: connect: connection refused"
time="2021-06-16T12:09:07Z" level=info msg="Failed to get volume stats errGet "http://10.233.57.251:9501/v1/stats\": dial tcp 10.233.57.251:9501: connect: connection refused"
time="2021-06-16T12:09:07Z" level=error msg="failed to update JivaVolume, err: Operation cannot be fulfilled on jivavolumes.openebs.io "pvc-2b697d2e-e561-4acf-b264-60fd6cd392e1": the object has been modified; please apply your changes to the latest version and try againfailed to update status"
time="2021-06-16T12:09:07Z" level=info msg="Failed to get volume stats errGet "http://10.233.57.251:9501/v1/stats\": dial tcp 10.233.57.251:9501: connect: connection refused"
time="2021-06-16T12:09:07Z" level=info msg="Failed to get volume stats errGet "http://10.233.57.251:9501/v1/stats\": dial tcp 10.233.57.251:9501: connect: connection refused"
time="2021-06-16T12:09:08Z" level=info msg="Failed to get volume stats errGet "http://10.233.57.251:9501/v1/stats\": dial tcp 10.233.57.251:9501: connect: connection refused"
time="2021-06-16T12:09:08Z" level=info msg="Failed to get volume stats errGet "http://10.233.57.251:9501/v1/stats\": dial tcp 10.233.57.251:9501: connect: connection refused"
time="2021-06-16T12:09:09Z" level=info msg="Failed to get volume stats errGet "http://10.233.57.251:9501/v1/stats\": dial tcp 10.233.57.251:9501: connect: connection refused"
time="2021-06-16T12:09:13Z" level=info msg="Failed to get volume stats errGet "http://10.233.57.251:9501/v1/stats\": dial tcp 10.233.57.251:9501: connect: connection refused"
time="2021-06-16T12:09:13Z" level=info msg="Failed to get volume stats errGet "http://10.233.57.251:9501/v1/stats\": dial tcp 10.233.57.251:9501: connect: connection refused"
time="2021-06-16T12:09:13Z" level=info msg="Failed to get volume stats errGet "http://10.233.57.251:9501/v1/stats\": dial tcp 10.233.57.251:9501: connect: connection refused"
time="2021-06-16T12:09:14Z" level=info msg="Failed to get volume stats errGet "http://10.233.57.251:9501/v1/stats\": dial tcp 10.233.57.251:9501: connect: connection refused"
time="2021-06-16T12:09:17Z" level=info msg="Failed to get volume stats errGet "http://10.233.57.251:9501/v1/stats\": dial tcp 10.233.57.251:9501: connect: connection refused"
time="2021-06-16T12:09:18Z" level=info msg="Failed to get volume stats errGet "http://10.233.57.251:9501/v1/stats\": dial tcp 10.233.57.251:9501: connect: connection refused"
time="2021-06-16T12:09:18Z" level=info msg="Failed to get volume stats errGet "http://10.233.57.251:9501/v1/stats\": dial tcp 10.233.57.251:9501: connect: connection refused"
time="2021-06-16T12:09:19Z" level=info msg="Failed to get volume stats errGet "http://10.233.57.251:9501/v1/stats\": dial tcp 10.233.57.251:9501: connect: connection refused"
time="2021-06-16T12:09:22Z" level=info msg="Failed to get volume stats errGet "http://10.233.57.251:9501/v1/stats\": dial tcp 10.233.57.251:9501: connect: connection refused"
time="2021-06-16T12:09:23Z" level=info msg="Failed to get volume stats errGet "http://10.233.57.251:9501/v1/stats\": dial tcp 10.233.57.251:9501: connect: connection refused"
time="2021-06-16T12:09:23Z" level=info msg="Failed to get volume stats errGet "http://10.233.57.251:9501/v1/stats\": dial tcp 10.233.57.251:9501: connect: connection refused"
time="2021-06-16T12:09:25Z" level=info msg="Failed to get volume stats errGet "http://10.233.57.251:9501/v1/stats\": dial tcp 10.233.57.251:9501: connect: connection refused"
time="2021-06-16T12:09:26Z" level=info msg="Failed to get volume stats errGet "http://10.233.57.251:9501/v1/stats\": dial tcp 10.233.57.251:9501: connect: connection refused"
time="2021-06-16T12:09:28Z" level=info msg="Failed to get volume stats errGet "http://10.233.57.251:9501/v1/stats\": dial tcp 10.233.57.251:9501: connect: connection refused"
time="2021-06-16T12:09:28Z" level=info msg="Failed to get volume stats errGet "http://10.233.57.251:9501/v1/stats\": dial tcp 10.233.57.251:9501: connect: connection refused"
time="2021-06-16T12:09:30Z" level=info msg="Failed to get volume stats errGet "http://10.233.57.251:9501/v1/stats\": dial tcp 10.233.57.251:9501: connect: connection refused"
time="2021-06-16T12:09:49Z" level=error msg="failed to update JivaVolume, err: Operation cannot be fulfilled on jivavolumes.openebs.io "pvc-2b697d2e-e561-4acf-b264-60fd6cd392e1": the object has been modified; please apply your changes to the latest version and try againfailed to update status"

[root@node1 ~]# kubectl logs -f pvc-2b697d2e-e561-4acf-b264-60fd6cd392e1-jiva-rep-0 -n openebs
time="2021-06-16T12:09:46Z" level=info msg="MAX_CHAIN_LENGTH env not set, default value is 512"
time="2021-06-16T12:09:46Z" level=info msg="Configured logging with retentionPeriod: 180, maxLogFileSize: 100, maxBackups: 5"
time="2021-06-16T12:09:46Z" level=info msg="Creating volume openebs, size 4294967296/512"
time="2021-06-16T12:09:46Z" level=info msg="Starting replica having replicaType: , frontendIP: pvc-2b697d2e-e561-4acf-b264-60fd6cd392e1-jiva-ctrl-svc.openebs.svc.cluster.local, size: 4294967296, dir: openebs"
time="2021-06-16T12:09:46Z" level=info msg="Setting replicaAddr: 10.233.65.185:9502, controlAddr: 10.233.65.185:9502, dataAddr: 10.233.65.185:9503, syncAddr: 10.233.65.185:9504"
time="2021-06-16T12:09:46Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2021-06-16T12:09:46Z" level=info msg="Listening on data 10.233.65.185:9503"
time="2021-06-16T12:09:46Z" level=info msg="Listening on control 10.233.65.185:9502"
time="2021-06-16T12:09:46Z" level=info msg="Closing replica"
time="2021-06-16T12:09:46Z" level=info msg="Skip closing replica, s.r not set"
time="2021-06-16T12:09:46Z" level=info msg="CheckAndResetFailedRebuild tcp://10.233.65.185:9502"
time="2021-06-16T12:09:46Z" level=info msg="Addreplica tcp://10.233.65.185:9502"
time="2021-06-16T12:09:46Z" level=info msg="Listening on sync 10.233.65.185:9504 start: 9700 end: 9800"
time="2021-06-16T12:09:46Z" level=info msg="Get Volume info from controller"
time="2021-06-16T12:09:46Z" level=info msg="Register replica at controller"
time="2021-06-16T12:09:46Z" level=info msg="StartReplica with value start"
10.233.65.220 - - [16/Jun/2021:12:09:46 +0000] "POST /v1/replicas/1?action=start HTTP/1.1" 200 1045
time="2021-06-16T12:09:46Z" level=info msg="Received start from controller"
time="2021-06-16T12:09:46Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:09:46Z" level=info msg="New connection from: 10.233.65.220:56000"
time="2021-06-16T12:09:46Z" level=info msg="Got signal: 'open', proceed to open replica"
time="2021-06-16T12:09:46Z" level=info msg="Opening volume openebs, size 4294967296/512"
time="2021-06-16T12:09:47Z" level=info msg="Update revison count: 1 of snapshot: volume-head-000.img"
time="2021-06-16T12:09:47Z" level=info msg="Start reading extents"
time="2021-06-16T12:09:47Z" level=info msg="Read extents successful"
10.233.65.220 - - [16/Jun/2021:12:09:46 +0000] "POST /v1/replicas/1?action=open HTTP/1.1" 200 1861
time="2021-06-16T12:09:47Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:09:47Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:09:47Z" level=info msg="SetReplicaMode to WO"
10.233.65.220 - - [16/Jun/2021:12:09:47 +0000] "POST /v1/replicas/1?action=setreplicamode HTTP/1.1" 200 1859
time="2021-06-16T12:09:47Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:09:48Z" level=info msg="Set clone status as NA"
time="2021-06-16T12:09:49Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:09:49Z" level=info msg="SetReplicaMode to RW"
10.233.65.220 - - [16/Jun/2021:12:09:49 +0000] "POST /v1/replicas/1?action=setreplicamode HTTP/1.1" 200 1861
time="2021-06-16T12:09:49Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:09:49Z" level=info msg="GetReplica for id 1"
2021-06-16T12:09:49.101Z INFO app/add_replica.go:71 {"eventcode": "jiva.volume.replica.add.success", "msg": "Successfully added Jiva volume replica", "rname": "tcp://10.233.65.185:9502"}
time="2021-06-16T12:09:49Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:09:49Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:09:49Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:09:49Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:09:50Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:09:52Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:09:53Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:09:54Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:09:56Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:09:56Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:09:57Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:09:58Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:09:59Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:01Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:02Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:02Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:03Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:04Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:06Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:07Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:07Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:09Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:11Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:12Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:12Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:14Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:16Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:16Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:17Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:19Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:21Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:21Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:22Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:24Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:26Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:27Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:27Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:29Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:30Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:32Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:32Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:34Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:35Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:37Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:37Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:39Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:39Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:42Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:42Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:44Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:45Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:47Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:47Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:48Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:49Z" level=error msg="checkpoint not found"
time="2021-06-16T12:10:50Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:52Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:52Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:53Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:55Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:56Z" level=info msg="GetReplica for id 1"
time="2021-06-16T12:10:58Z" level=info msg="GetReplica for id 1"

root@pvc-2b697d2e-e561-4acf-b264-60fd6cd392e1-jiva-rep-0:/# curl http://10.233.57.251:9501/v1/stats
{"IsClientConnected":true,"Name":"pvc-2b697d2e-e561-4acf-b264-60fd6cd392e1","ReadIOPS":"142","ReplicaCounter":1,"ReplicaInfo":[{"chain":["volume-head-000.img"],"checkpoint":"","clonestatus":"NA","dirty":true,"disks":{"volume-head-000.img":{"children":[],"created":"2021-06-16T12:09:46Z","name":"volume-head-000.img","parent":"","removed":false,"revisionCount":1,"size":"69386240","usercreated":false}},"head":"volume-head-000.img","parent":"","rebuilding":false,"remainsnapshots":1022,"replicamode":"RW","revisioncounter":"74","sectorSize":512,"size":"4294967296","state":"dirty","usedblocks":"16942","usedlogicalblocks":"16939"}],"Replicas":[{"Address":"tcp://10.233.65.185:9502","Mode":"RW"}],"RevisionCounter":74,"SCSIIOCount":{"0":56,"158":2,"160":1,"163":1,"18":26,"26":6,"40":142,"42":19,"53":4,"65":32,"66":3},"SectorSize":"4096","Size":"4294967296","Status":"RW","TotalReadBlockCount":"3343360","TotalReadTime":"160979549","TotalWriteBlockCount":"196608","TotalWriteTime":"593890439","UpTime":"304.776786","UsedBlocks":"16942","UsedLogicalBlocks":"16939","WriteIOPS":"19","actions":{},"links":{"self":"http://10.233.57.251:9501/v1/stats"},"type":"stats"}
root@pvc-2b697d2e-e561-4acf-b264-60fd6cd392e1-jiva-rep-0:/#```

[root@node1 ~]# kubectl get po
NAME                   READY   STATUS    RESTARTS   AGE
fio-5bd64c6b49-htgsz   1/1     Running   0          7m7s

@shubham14bajpai
Copy link
Contributor

shubham14bajpai commented Jun 16, 2021 •
edited
Loading

Can you share the -oyaml of the jv. The error will appear in the beginning because the repica pod take some time to come to Running state and should go away in jiva-operator after jv comes to Ready state

@cleveritcz
Copy link
Author 

items:
- apiVersion: openebs.io/v1alpha1
  kind: JivaVolume
  metadata:
    annotations:
      openebs.io/volume-policy: example-jivavolumepolicy
    creationTimestamp: "2021-06-16T12:09:06Z"
    generation: 8
    labels:
      nodeID: node2
      openebs.io/component: jiva-volume
      openebs.io/persistent-volume: pvc-2b697d2e-e561-4acf-b264-60fd6cd392e1
    managedFields:
    - apiVersion: openebs.io/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        f:spec:
          f:desiredReplicationFactor: {}
          f:iscsiSpec:
            f:iqn: {}
            f:targetIP: {}
            f:targetPort: {}
          f:policy:
            f:replica:
              f:resources:
                .: {}
                f:limits:
                  .: {}
                  f:cpu: {}
                  f:memory: {}
                f:requests:
                  .: {}
                  f:cpu: {}
                  f:memory: {}
              f:tolerations: {}
            f:replicaSC: {}
            f:target:
              f:auxResources:
                .: {}
                f:limits:
                  .: {}
                  f:cpu: {}
                  f:memory: {}
                f:requests:
                  .: {}
                  f:cpu: {}
                  f:memory: {}
              f:replicationFactor: {}
              f:resources:
                .: {}
                f:limits:
                  .: {}
                  f:cpu: {}
                  f:memory: {}
                f:requests:
                  .: {}
                  f:cpu: {}
                  f:memory: {}
              f:tolerations: {}
        f:status:
          f:phase: {}
          f:replicaCount: {}
          f:replicaStatus: {}
          f:status: {}
      manager: jiva-operator
      operation: Update
      time: "2021-06-16T12:09:49Z"
    - apiVersion: openebs.io/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:openebs.io/volume-policy: {}
          f:labels:
            .: {}
            f:nodeID: {}
            f:openebs.io/component: {}
            f:openebs.io/persistent-volume: {}
        f:spec:
          .: {}
          f:accessType: {}
          f:capacity: {}
          f:iscsiSpec: {}
          f:mountInfo:
            .: {}
            f:devicePath: {}
            f:fsType: {}
            f:stagingPath: {}
            f:targetPath: {}
          f:policy:
            .: {}
            f:autoScaling: {}
            f:enableBufio: {}
            f:replica: {}
            f:target: {}
          f:pv: {}
        f:status: {}
        f:versionDetails:
          .: {}
          f:desired: {}
          f:status:
            .: {}
            f:current: {}
            f:dependentsUpgraded: {}
            f:lastUpdateTime: {}
      manager: jiva-csi
      operation: Update
      time: "2021-06-16T12:10:03Z"
    name: pvc-2b697d2e-e561-4acf-b264-60fd6cd392e1
    namespace: openebs
    resourceVersion: "4902"
    uid: c50e27d4-d2e3-4457-8748-f0987b68c787
  spec:
    accessType: mount
    capacity: 4Gi
    desiredReplicationFactor: 1
    iscsiSpec:
      iqn: iqn.2016-09.com.openebs.jiva:pvc-2b697d2e-e561-4acf-b264-60fd6cd392e1
      targetIP: 10.233.57.251
      targetPort: 3260
    mountInfo:
      devicePath: /dev/disk/by-path/ip-10.233.57.251:3260-iscsi-iqn.2016-09.com.openebs.jiva:pvc-2b697d2e-e561-4acf-b264-60fd6cd392e1-lun-0
      fsType: ext4
      stagingPath: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-2b697d2e-e561-4acf-b264-60fd6cd392e1/globalmount
      targetPath: /var/lib/kubelet/pods/cc8a3bee-4996-4df0-9a70-384b82d07f20/volumes/kubernetes.io~csi/pvc-2b697d2e-e561-4acf-b264-60fd6cd392e1/mount
    policy:
      autoScaling: false
      enableBufio: false
      replica:
        resources:
          limits:
            cpu: "0"
            memory: "0"
          requests:
            cpu: "0"
            memory: "0"
        tolerations:
        - effect: NoExecute
          key: node.kubernetes.io/notReady
          operator: Exists
        - effect: NoExecute
          key: node.cloudprovider.kubernetes.io/uninitialized
          operator: Exists
        - effect: NoExecute
          key: node.kubernetes.io/unreachable
          operator: Exists
        - effect: NoExecute
          key: node.kubernetes.io/not-ready
          operator: Exists
        - effect: NoExecute
          key: node.kubernetes.io/unschedulable
          operator: Exists
        - effect: NoExecute
          key: node.kubernetes.io/out-of-disk
          operator: Exists
        - effect: NoExecute
          key: node.kubernetes.io/memory-pressure
          operator: Exists
        - effect: NoExecute
          key: node.kubernetes.io/disk-pressure
          operator: Exists
        - effect: NoExecute
          key: node.kubernetes.io/network-unavailable
          operator: Exists
      replicaSC: openebs-hostpath
      target:
        auxResources:
          limits:
            cpu: "0"
            memory: "0"
          requests:
            cpu: "0"
            memory: "0"
        replicationFactor: 1
        resources:
          limits:
            cpu: "0"
            memory: "0"
          requests:
            cpu: "0"
            memory: "0"
        tolerations:
        - effect: NoExecute
          key: node.kubernetes.io/notReady
          operator: Exists
          tolerationSeconds: 0
        - effect: NoExecute
          key: node.kubernetes.io/unreachable
          operator: Exists
          tolerationSeconds: 0
        - effect: NoExecute
          key: node.kubernetes.io/not-ready
          operator: Exists
          tolerationSeconds: 0
    pv: pvc-2b697d2e-e561-4acf-b264-60fd6cd392e1
  status:
    phase: Ready
    replicaCount: 1
    replicaStatus:
    - address: tcp://10.233.65.185:9502
      mode: RW
    status: RW
  versionDetails:
    desired: 2.9.0
    status:
      current: 2.9.0
      dependentsUpgraded: true
      lastUpdateTime: null
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

@cleveritcz
Copy link
Author

I don't know if it is working for production use case.

@cleveritcz
Copy link
Author

everything is up and running. I would like to use anothe domain than cluster.local. Where could I change it, please?

[root@node1 ~]# kubectl -n openebs get all
NAME READY STATUS RESTARTS AGE
pod/jiva-operator-6b69b86894-kw8mm 1/1 Running 0 18m
pod/openebs-jiva-csi-controller-0 5/5 Running 0 18m
pod/openebs-jiva-csi-node-4b4gz 3/3 Running 0 18m
pod/openebs-jiva-csi-node-kf4lh 3/3 Running 0 18m
pod/openebs-jiva-csi-node-wplxz 3/3 Running 0 18m
pod/openebs-localpv-provisioner-7f85dcbb75-vdwhn 1/1 Running 0 19m
pod/pvc-2b697d2e-e561-4acf-b264-60fd6cd392e1-jiva-ctrl-5dd65f4fxb5q 1/1 Running 0 16m
pod/pvc-2b697d2e-e561-4acf-b264-60fd6cd392e1-jiva-rep-0 1/1 Running 0 16m

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/pvc-2b697d2e-e561-4acf-b264-60fd6cd392e1-jiva-ctrl-svc ClusterIP 10.233.57.251 3260/TCP,9501/TCP,9500/TCP 16m

NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/openebs-jiva-csi-node 3 3 3 3 3 18m

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/jiva-operator 1/1 1 1 18m
deployment.apps/openebs-localpv-provisioner 1/1 1 1 19m
deployment.apps/pvc-2b697d2e-e561-4acf-b264-60fd6cd392e1-jiva-ctrl 1/1 1 1 16m

NAME DESIRED CURRENT READY AGE
replicaset.apps/jiva-operator-6b69b86894 1 1 1 18m
replicaset.apps/openebs-localpv-provisioner-7f85dcbb75 1 1 1 19m
replicaset.apps/pvc-2b697d2e-e561-4acf-b264-60fd6cd392e1-jiva-ctrl-5dd65f46b6 1 1 1 16m

NAME READY AGE
statefulset.apps/openebs-jiva-csi-controller 1/1 18m
statefulset.apps/pvc-2b697d2e-e561-4acf-b264-60fd6cd392e1-jiva-rep 1/1 16m

@shubham14bajpai shubham14bajpai self-assigned this Jun 16, 2021
@shubham14bajpai
Copy link
Contributor

Thanks for reporting this issue. I was able to reproduce it locally. I will add this to the next release tracker and try to fix it. It is a small fix we have to replace this hard coded value https://github.com/openebs/jiva-operator/blob/811a3af7c34aa13c487f7e4a84c4704cf8f94100/pkg/controllers/jivavolume_controller.go#L75

Also if you have a single replica use case you can directly use localpv hostpath

@cleveritcz
Copy link
Author

and what about this error?

time="2021-06-16T12:09:07Z" level=info msg="Failed to get volume stats errGet "http://10.233.57.251:9501/v1/stats\": dial tcp 10.233.57.251:9501: connect: connection refused"

@shubham14bajpai
Copy link
Contributor

This I need to look for a little more this is code is trying to watch the volume status by trying to png for the stats. This can happen because of a lot of reasons. But these errors are harmless in most cases

@shubham14bajpai
Copy link
Contributor

shubham14bajpai commented Jun 16, 2021 •
edited
Loading

After changing the cluster domain and the volume became Ready are they still appearing in the log?

@cleveritcz
Copy link
Author

yes

@shubham14bajpai shubham14bajpai mentioned this issue Jun 17, 2021
Merged
@shubham14bajpai
Copy link
Contributor

Hi @cleveritcz the error for the cluster domain got fixed in the latest release of jiva-operator. You can find the operator yamls here
openebs/charts#228

For the other error it is something that may happen due to some reachability issue. The volume is able to mount so the GET for the stats is successful but I still need to dig a little more.

@shubham14bajpai
Copy link
Contributor

Hi @cleveritcz I was able to find the cause of the second error that was flooding the logs and is fixed by the PR #112. These fixes are now available In 2.11.0 release. Closing this issue for now, please feel free to reopen if any errors occur.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@shubham14bajpai shubham14bajpai

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@shubham14bajpai @cleveritcz