PersistentVolumeClaim remains in Pending state

[puco]

When I try to create PV & PVC in Kubernetes with Terraform targeting a specific existing EBS volume the PVC never gets bound. It looks the PV created using Terraform does not create it in correct manner that it can be successfully bound.

Terraform Version

Terraform v0.10.7

Affected Resource(s)

• persistent_volume

Terraform Configuration Files

variable "vol_id" {
  # default = "aws://us-east-1a/vol-1111111111111"
}

provider "kubernetes" {
}

resource "kubernetes_persistent_volume" "pv" {
  metadata {
    name            = "test-volume"
  }
  spec {
    capacity {
      storage       = "1Gi"
    }
    
    access_modes    = ["ReadWriteOnce"]
    
    persistent_volume_source {
      aws_elastic_block_store {
        volume_id   = "${var.vol_id}"
        fs_type     = "ext4"
      }
    }
  }
}

resource "kubernetes_persistent_volume_claim" "pvc" {
  metadata {
    name            = "test-volume"
  }
  spec {
    resources {
      requests {
        storage     = "1Gi"
      }
    }
    access_modes    = ["ReadWriteOnce"]
    volume_name     = "test-volume"
    storage_class_name = "gp2"
  }
  wait_until_bound = false
  depends_on = ["kubernetes_persistent_volume.pv"]
}

Debug Output

https://gist.github.com/puco/23487f4d80001f3d44a41a48b53504cf

Expected Behavior

I should have a PV & bound PVC

Actual Behavior

I have a PV in Available state and PVC in Pending state

Steps to Reproduce

1. terraform apply

Important Factoids

Run of the mill Kubernetes 1.8 cluster w/o RBAC on AWS provisioned with kops. I created the EBS volume manually (as that part works w/o any issues).

What works

When I create the PV/PVC with kubectl directly they work. When I create the PV with terraform and PVC with kubectl it works as well, so there appears to be something wrong with the PV. I compared the definition (yaml) of both PVs (tf and kubectl) and there appears not to be any issues. Originally I assumed that the issue would be creating EBS volume, PV and PVC in one go but the issue still remains with an existing EBS volume.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: test-volume-yaml
spec:
  accessModes:
  - ReadWriteOnce
  awsElasticBlockStore:
    fsType: ext4
    volumeID: aws://us-east-1a/vol-1111111111111
  capacity:
    storage: 1Gi
  persistentVolumeReclaimPolicy: Retain
  storageClassName: gp2
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-volume-yaml
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: gp2
  volumeName: test-volume-yaml

[sam-at-luther]

I'm facing a similar issue on Azure, but I believe it's related to the storage class name on the persistent volume that terraform creates (the empty string). Because the storage class names don't match the PVC fails to bind to the PV. Could this be your problem (the storage class names don't match)?

There doesn't seem to be a way to specify the storage class name on the PV, while the PVC resource storage class name defaults to "default". When I manually edited the PV storage class name using kubectl from "" -> "default" then the PVC successfully bound to the PV.

[ccope]

I just hit the same problem on Google, and agree with @sam-at-luther's diagnosis.

[dionjwa]

Agree with @sam-at-luther
I'm trying to use minikube with persistent local volumes, and hitting the same issue, where the volume claim is stuck in a pending state.

There are workarounds and solutions in these issues, but they only work if you can make the changes in kubernetes config, but since terraform doesn't allow you to specify the storage class on the volume, they won't work.

kubernetes/minikube#214
kubernetes/minikube#1783
This also shows my observations:
kubernetes/minikube#1239

[plispe]

I have the same issue. But I don't understand, why terraform doesn't allow to specify storage class name on the persistent volume. Is there any reason for that?

[dionjwa]

@plispe I looked at the code, it's just a matter of adding the entry (which I would do if my go coding was up to it). Terraform explicitly maps parameters to the underlying drivers, so it cannot simply pass parameters it doesn't explicitly handle.

[plispe]

@dionjwa Thank you, I will try to make a pull request with the solution. I wrote already the code, but now I am trying to find how I should test the updated provider.

[ccope]

plispe There is already a pull request with a fix: #72

[plispe]

@ccope ok I will wait for merge and release. Thank you.

[ctrombet]

Hi all

any updates on this issue?

[radeksimko]

Hi folks,
thanks for the report and sorry for the delay in processing this.

The good news is that #111 which is fixing this bug was merged and will be part of the upcoming release which should come out 🔜.

Also #125 has improved the event log polling which should hopefully reduce any time needed to debug such issue.