Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PV Snapshot fails silently #353

Closed
cj-anthony opened this issue Mar 6, 2018 · 6 comments
Closed

PV Snapshot fails silently #353

cj-anthony opened this issue Mar 6, 2018 · 6 comments

Comments

@cj-anthony
Copy link

Hi! I'm trying to get persitent volume snapshops working on my ark install. The snapshot never seems to happen and the server doesn't log anything of note. Does anyone know how I might get more information? I am running an more recent image than latest as it supported the removal of the location configuration parameter in azure as per #344

The details are as follows:

Kubernetes Version: 1.8.1
ark cli version: 0.7.1
Persistent Storage: Azure
ark pod image name: ce59f80029fc
cli command: ark backup create mybackup --include-namespaces mynamespace --snapshot-volumes=true

server log:

time="2018-03-06T03:52:27Z" level=info msg="Got backups from object storage" backupCount=0 logSource="pkg/controller/backup_sync_controller.go:78"
time="2018-03-06T03:53:03Z" level=info msg="Starting backup" backup=heptio-ark/mybackup logSource="pkg/controller/backup_controller.go:329"
time="2018-03-06T03:53:05Z" level=info msg="Backup completed" backup=heptio-ark/mybackup logSource="pkg/controller/backup_controller.go:375"
@ncdc
Copy link
Contributor

ncdc commented Mar 6, 2018

Hi @cj-anthony, could you please share the output of ark backup logs mybackup?

@ncdc
Copy link
Contributor

ncdc commented Mar 8, 2018

Hi @cj-anthony, just checking in to see if you had any luck or can provide any more information?

@ncdc ncdc added the question label Mar 8, 2018
@cj-anthony
Copy link
Author

Hi Andy,

Thanks for your response, sorry I was out of the office yesterday. I have the log files but they're quite lengthy and contain some sensitive information. However I think I might have found the issue:

time="2018-03-06T04:36:07Z" level=info msg="Executing takePVSnapshot" backup=heptio-ark/entirecluster group=v1 groupResource=persistentvolumeclaims logSource="pkg/backup/item_backupper.go:280" name=myappplatform-datastore namespace=myapp
time="2018-03-06T04:36:07Z" level=info msg="label "failure-domain.beta.kubernetes.io/zone" is not present on PersistentVolume" backup=heptio-ark/entirecluster group=v1 groupResource=persistentvolumeclaims logSource="pkg/backup/item_backupper.go:299" name=myappplatform-datastore namespace=myapp
time="2018-03-06T04:36:07Z" level=info msg="PersistentVolume is not a supported volume type for snapshots, skipping." backup=heptio-ark/entirecluster group=v1groupResource=persistentvolumeclaims logSource="pkg/backup/item_backupper.go:307" name=myappplatform-datastore namespace=myapp

The persistent storage is an azure storage account file store. Is this type of storage supported for snapshots?

@cj-anthony
Copy link
Author

Sorry, also to add there is another error on a Persistent Storage type of Azure Managed disk. This one is interesting as it references a resource group called "dev-clusters" which is old and has since been removed and replaced when we redeployed the cluster.

time="2018-03-06T04:36:06Z" level=info msg="Backing up resource" backup=heptio-ark/entirecluster group=v1 groupResource=pods logSource="pkg/backup/item_backupper.go:163" name=pvc-77627e54-15cd-11e8-8cc7-0a58ac1f22c9 namespace=ci
time="2018-03-06T04:36:06Z" level=info msg="Executing pre hooks" backup=heptio-ark/entirecluster group=v1 groupResource=pods logSource="pkg/backup/item_backupper.go:168" name=pvc-77627e54-15cd-11e8-8cc7-0a58ac1f22c9 namespace=ci
time="2018-03-06T04:36:06Z" level=info msg="Executing takePVSnapshot" backup=heptio-ark/entirecluster group=v1 groupResource=pods logSource="pkg/backup/item_backupper.go:280" name=pvc-77627e54-15cd-11e8-8cc7-0a58ac1f22c9 namespace=ci
time="2018-03-06T04:36:06Z" level=info msg="label "failure-domain.beta.kubernetes.io/zone" is not present on PersistentVolume" backup=heptio-ark/entirecluster group=v1 groupResource=pods logSource="pkg/backup/item_backupper.go:299" name=pvc-77627e54-15cd-11e8-8cc7-0a58ac1f22c9 namespace=ci
time="2018-03-06T04:36:06Z" level=info msg="Snapshotting PersistentVolume" backup=heptio-ark/entirecluster group=v1 groupResource=pods logSource="pkg/backup/item_backupper.go:313" name=pvc-77627e54-15cd-11e8-8cc7-0a58ac1f22c9 namespace=ci volumeID=kubernetes-dynamic-pvc-77627e54-15cd-11e8-8cc7-0a58ac1f22c9
time="2018-03-06T04:36:06Z" level=error msg="error creating snapshot" backup=heptio-ark/entirecluster error="rpc error: code = Unknown desc = disk.DisksClient#Get: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceGroupNotFound" Message="Resource group 'dev-clusters' could not be found."" group=v1 groupResource=pods logSource="pkg/backup/item_backupper.go:317" name=pvc-77627e54-15cd-11e8-8cc7-0a58ac1f22c9 namespace=ci volumeID=kubernetes-dynamic-pvc-77627e54-15cd-11e8-8cc7-0a58ac1f22c9

@skriss
Copy link
Contributor

skriss commented Mar 9, 2018

hey @cj-anthony - re: the first comment, no, we do not support file store PVs for snapshot. For Azure we only support managed disks.

Re: the second comment - is it possible that you have the wrong value in AZURE_RESOURCE_GROUP within the cloud-credentials secret? That's being passed to the Azure client when trying to get info about the disk in order to take a snapshot. The value should be the one containing your disks (i.e. if an AKS cluster, the auto-generated one).

@cj-anthony
Copy link
Author

Thanks Andy, I think we'll close this for now as we're pretty embedded in azure file storage. Thanks for your help!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants