Upload iso failed

[MLintin]

What happened:
upload iso to dv which source is upload{} failed

What you expected to happen:
dv change phase to successed

How to reproduce it (as minimally and precisely as possible):
Set DV to Block And ReadWriteOnce

Additional context:

curl -v --insecure -H "Authorization: Bearer $TOKEN" --data-binary @../ubuntu-16.04.6-desktop-i386.iso  https://cdi-uploadproxy-cdi.$(minishift ip).nip.io/v1alpha1/upload-async

< HTTP/1.1 100 Continue
< HTTP/1.1 500 Internal Server Error
< content-length: 165
< content-type: text/plain; charset=utf-8
< date: Tue, 25 Oct 2022 08:47:22 GMT
< vary: Origin
* HTTP error before end of send, stop sending
< 
* Closing connection 0
Saving stream failed: Unable to transfer source data to target file: could not open file "/dev/cdi-block-volume": open /dev/cdi-block-volume: device or resource busy

kubectl logs  cdi-upload-upload 
I1025 07:33:53.417434       1 uploadserver.go:74] Running server on 0.0.0.0:8443
I1025 07:34:23.955269       1 uploadserver.go:325] Content type header is ""
I1025 07:34:23.955332       1 data-processor.go:379] Calculating available size
I1025 07:34:23.956476       1 data-processor.go:387] Checking out block volume size.
I1025 07:34:23.956492       1 data-processor.go:399] Request image size not empty.
I1025 07:34:23.956501       1 data-processor.go:404] Target size 50Gi.
I1025 07:34:23.970909       1 data-processor.go:282] New phase: TransferDataFile
E1025 07:34:23.973509       1 data-processor.go:278] open /dev/cdi-block-volume: device or resource busy
could not open file "/dev/cdi-block-volume"
kubevirt.io/containerized-data-importer/pkg/util.OpenFileOrBlockDevice
	pkg/util/util.go:180
kubevirt.io/containerized-data-importer/pkg/util.StreamDataToFile
	pkg/util/util.go:187
kubevirt.io/containerized-data-importer/pkg/importer.(*AsyncUploadDataSource).TransferFile
	pkg/importer/upload-datasource.go:158
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).initDefaultPhases.func4
	pkg/importer/data-processor.go:225
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause
	pkg/importer/data-processor.go:275
kubevirt.io/containerized-data-importer/pkg/uploadserver.newAsyncUploadStreamProcessor
	pkg/uploadserver/uploadserver.go:433
kubevirt.io/containerized-data-importer/pkg/uploadserver.(*uploadServerApp).uploadHandlerAsync.func1
	pkg/uploadserver/uploadserver.go:332
net/http.HandlerFunc.ServeHTTP
	GOROOT/src/net/http/server.go:2084
net/http.(*ServeMux).ServeHTTP
	GOROOT/src/net/http/server.go:2462
kubevirt.io/containerized-data-importer/pkg/uploadserver.(*uploadServerApp).ServeHTTP
	pkg/uploadserver/uploadserver.go:261
net/http.serverHandler.ServeHTTP
	GOROOT/src/net/http/server.go:2916
net/http.(*conn).serve
	GOROOT/src/net/http/server.go:1966
runtime.goexit
	GOROOT/src/runtime/asm_amd64.s:1571
Unable to transfer source data to target file
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).initDefaultPhases.func4
	pkg/importer/data-processor.go:227
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause
	pkg/importer/data-processor.go:275
kubevirt.io/containerized-data-importer/pkg/uploadserver.newAsyncUploadStreamProcessor
	pkg/uploadserver/uploadserver.go:433
kubevirt.io/containerized-data-importer/pkg/uploadserver.(*uploadServerApp).uploadHandlerAsync.func1
	pkg/uploadserver/uploadserver.go:332
net/http.HandlerFunc.ServeHTTP
	GOROOT/src/net/http/server.go:2084
net/http.(*ServeMux).ServeHTTP
	GOROOT/src/net/http/server.go:2462
kubevirt.io/containerized-data-importer/pkg/uploadserver.(*uploadServerApp).ServeHTTP
	pkg/uploadserver/uploadserver.go:261
net/http.serverHandler.ServeHTTP
	GOROOT/src/net/http/server.go:2916
net/http.(*conn).serve
	GOROOT/src/net/http/server.go:1966
runtime.goexit
	GOROOT/src/runtime/asm_amd64.s:1571
E1025 07:34:23.973717       1 uploadserver.go:337] Saving stream failed: Unable to transfer source data to target file: could not open file "/dev/cdi-block-volume": open /dev/cdi-block-volume: device or resource busy

When i set DV to FileSystem And ReadWriteOnce, then use upload-async method to upload , it can successed
When i set DV to Block And ReadWirteOnce, then set source to http, it can successed
In this two cases, i use the same iso.
Is there A problem with my StorageClass?

Environment:

• CDI version (use kubectl get deployments cdi-deployment -o yaml): 1.53
• Kubernetes version (use kubectl version): 1.22
• DV specification: 50Gi
• Cloud provider or hardware configuration: N/A
• OS (e.g. from /etc/os-release): linux
• Kernel (e.g. uname -a): N/A
• Install tools: N/A
• Others: okd environment

[awels]

Duplicate of #2433 ?

[MLintin]

It doesn't seem to be repeated， #2433 look like a Permission problem, his error log is different from mine

[mhenriks]

@MLintin are you uploading a qcow2 or raw file? #2433 is uploading a raw file which streams directly to the target pvc. With qcow2, the image is first written to a "scratch" FileSystem pvc, then transfered to the (Block) target. So the errors would be different for the two cases

[MLintin]

@mhenriks I just updated context and the file I want to upload is an ISO file. I have tried the suggestion in #2433 but did not worked(I set device_ownership_from_security_context = true). My problem is not 100% showing up. When my OKD just set up, the first time upload ususally would be successful, but after that this problem would show up and make upload failed.

[MLintin]

@mhenriks Duplicate of #2315 ?

[bc185174]

I think those are related issues. This is what I have found and the permissions seems to be a common problem in various scenarios. FYI using CDI version v1.55.0 and KubeVirt v0.57.1.

1. Upload image using virtctl, using RWX volume (FileSystem). Get permission denied converting qcow2 image to raw:

qemu-img: /data/disk.img: error while converting raw: Could not create '/data/disk.img': Permission denied

2. Upload image using virtctl, using RWX volume (Block). Get another permission error attempting to read size of device:

Calculating available size
exit status 1, blockdev: cannot open /dev/cdi-block-volume: Permission denied

3. Import image from container registry using RWX (FileSystem). For this used a DataVolume.

openfdat /data/lost+found: permission denied

4. Import image from container registry using RWX (Block). For this used a DataVolume.

exit status 1, blockdev: cannot open /dev/cdi-block-volume: Permission denied

5. Import image from container registry using RWO (FileSystem). For this used a DataVolume.

exit status 1, blockdev: cannot open /dev/cdi-block-volume: Permission denied

6. Import image from container registry using RWO (FileSystem). For this used a DataVolume. The image was successfully copied and we could successfully boot the vm.

7. Uploading the raw image using virtctl also worked to a RWO (FileSystem) volume.

8. The last thing I tried was using virtctl to upload the raw image to a RWX (FIleSystem) volume:

unexpected return value 500, Saving stream failed: Unable to transfer source data to target file: could not open file "/data/disk.img": open /data/disk.img: permission denied

So there seems to be permission issues with using block-volumes and another with the RWX volume?

Interesting, I tried to upload the qcow2 image again using virtctl onto a RWX (FileSystem) volume. /data directory is owned by root and not user id 107. Is that right, assume fsGroup should mount it with user id 107?

The RWX volume:

kubectl exec -ti cdi-upload-win10 -- stat /data
...
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
...

The RWO scratch volume:

kubectl exec -ti cdi-upload-win10 -- stat /scratch/tmpimage
...
Access: (0755/-rwxr-xr-x)  Uid: (  107/ UNKNOWN)   Gid: (  107/ UNKNOWN)
...

Followed this by doing the same but both the scratch and data volumes set to RWO.

kubectl exec -ti cdi-upload-win10 -- stat /data
...
Access: (2775/drwxrwsr-x)  Uid: (    0/    root)   Gid: (  107/ UNKNOWN)
...

kubectl exec -ti cdi-upload-win10 -- stat /scratch/tmpimage
...
Access: (0755/-rwxr-xr-x)  Uid: (  107/ UNKNOWN)   Gid: (  107/ UNKNOWN)
...

I've done a bit of research and it does not look like fsGroup is supported for RWX volumes. https://kubernetes-csi.github.io/docs/support-fsgroup.html.

[awels]

@bc185174 can you elaborate which RWX filesystem you are using? If it is straight NFS(CSI) then you need to make sure you configure the NFS exports properly otherwise there will be permission issues. Also would like to know your RWX block devices, and which CRI you are using (CRI-O or containerd) which apparently makes a difference in how the permissions are treated. And last but not least what RWO filesystem did you use that did work?

[kvaps]

Hi, I just faced with the same problem after importing dataVolume from the registry to RWX block volume. VM can't start:

  Warning  SyncFailed         1s (x16 over 74s)      virt-handler               server error. command SyncVMI failed: "LibvirtError(Code=38, Domain=18, Message='Cannot access storage file '/dev/boot': Operation not permitted')"

When I use RWO filesystem mode, there is no such issue

Storage provider: linstor-csi
Kubernetes version: v1.23.9
Kubevirt version: v0.57.0-dirty
OS: Ubuntu 22.04

upd: RWO block devices having the same problem

[awels]

I believe Ben's issue was the upload itself where the device didn't get the appropriate permissions when using containerd CRI. I don't know if KubeVirt itself is enforcing PSA yet and if that is the reason you are getting the error for the block devices. In case it is please that a look at the documentation update #2458 that explains how to configure the CRI to properly set permissions on the device.

[kvaps]

@awels thanks for pointing out, import was successful, but now virt-launcher can't start the vm with the same error:

{"component":"virt-launcher","level":"error","msg":"Cannot access storage file '/dev/boot': Operation not permitted","pos":"virStorageSourceReportBrokenChain:1248","subcomponent":"libvirt","thread":"32","timestamp":"2022-11-14T21:43:13.263000Z"}
{"component":"virt-launcher","kind":"","level":"error","msg":"Failed to start VirtualMachineInstance with flags 0.","name":"vm1","namespace":"default","pos":"manager.go:884","reason":"virError(Code=38, Domain=18, Message='Cannot access storage file '/dev/boot': Operation not permitted')","timestamp":"2022-11-14T21:43:13.264250Z","uid":"38133cb9-63f6-4af7-9d72-54ab33064a6d"}

any ideas?

[kvaps]

Setting option from #2458 solves the CDI issue

[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
    device_ownership_from_security_context = true

For KubeVirt issue let's switch to kubevirt/kubevirt#8699

/close

[kubevirt-bot]

@kvaps: Closing this issue.

In response to this:

Setting option from #2458 solves the CDI issue

[plugins]
 [plugins."io.containerd.grpc.v1.cri"]
   device_ownership_from_security_context = true

For KubeVirt let's switch to kubevirt/kubevirt#8699

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.