Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

virtctl image-upload xxx cannot open /dev/cdi-block-volume: Permission denied #2433

Closed
tghfly opened this issue Sep 22, 2022 · 18 comments
Closed

virtctl image-upload xxx cannot open /dev/cdi-block-volume: Permission denied #2433

tghfly opened this issue Sep 22, 2022 · 18 comments
Labels
kind/bug

Comments

@tghfly
Copy link

tghfly commented Sep 22, 2022

What happened:

virtctl image-upload --uploadproxy-url=https://10.105.77.138 \

--insecure
dv img-cirros-0.4.0
--size=2Gi
-n default
--storage-class=rook-ceph-block
--access-mode=ReadWriteMany
--block-volume
--image-path=./cirros-0.4.0-x86_64-disk.raw

PVC default/img-cirros-0.4.0 not found
DataVolume default/img-cirros-0.4.0 created
Waiting for PVC img-cirros-0.4.0 upload pod to be ready...
Pod now ready
Uploading data to https://10.105.77.138

1.50 MiB / 44.00 MiB [========>----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 3.41% 0s

unexpected return value 500, Saving stream failed: Unable to transfer source data to target file: error determining if block device exists: exit status 1, blockdev: cannot open /dev/cdi-block-volume: Permission denied

How to reproduce it (as minimally and precisely as possible):
Steps to reproduce the behavior.

Additional context:
Add any other context about the problem here.

Environment:

  • CDI version (use kubectl get deployments cdi-deployment -o yaml): 1.55.0
  • Kubernetes version (use kubectl version): v1.22.13
  • DV specification: N/A
  • Cloud provider or hardware configuration: vmware
  • OS (e.g. from /etc/os-release): CentOS7.9-x64
  • Kernel (e.g. uname -a): Linux master 5.4.211-1.el7.elrepo.x86_64 Start design doc #1 SMP Wed Aug 24 08:48:49 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux
@tghfly
Copy link
Author

tghfly commented Sep 22, 2022

virtctl version: v0.57.1

@tghfly
Copy link
Author

tghfly commented Sep 22, 2022

the default of namespace pods cdi-upload-xxx can not deleted, the pvc create immediately after deletion

@awels
Copy link
Member

awels commented Sep 22, 2022

We did some security related changes in 1.55, can you try 1.54 and let me know if it works?

@tghfly
Copy link
Author

tghfly commented Sep 23, 2022

1.54 works fine

@awels
Copy link
Member

awels commented Sep 24, 2022

Interesting @mhenriks any thoughts on why we would be getting permission denied on a block device backed by rook-ceph-rbd?

@mhenriks
Copy link
Member

@awels, No idea and I was unable to reproduce with k8s 1.22 in kubevirtci.

@tghfly how are you installing kubernetes? Can you upload to a filesystem pvc?

@tghfly
Copy link
Author

tghfly commented Sep 26, 2022

I have downgraded CDIversion to 1.54.0
My k8s install in RPM mode
storageClass: rook-ceph.rbd.csi.ceph.com
ceph version: 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable)

@mhenriks
Copy link
Member

Hi @tghfly we are testing with rook 1.8.9 and ceph 1.16.2.7. Perhaps the issue is related to the newer version of ceph. We can test that on our end. May be useful to see if older ceph works in your setup if possible

@mhenriks
Copy link
Member

@tghfly Just tested with latest rook and ceph 17.2.3. Could not reproduce

@tghfly
Copy link
Author

tghfly commented Sep 29, 2022 •
edited
Loading

In my environment, this problem must happen,And pod cdi-upload-xxx can't be removed

# kubectl logs -n default       cdi-upload-img-cirros-0.4.0
I0929 02:13:52.701618       1 uploadserver.go:74] Running server on 0.0.0.0:8443
I0929 02:13:54.313443       1 uploadserver.go:325] Content type header is ""
I0929 02:13:54.313502       1 data-processor.go:379] Calculating available size
E0929 02:13:54.319027       1 data-processor.go:383] exit status 1, blockdev: cannot open /dev/cdi-block-volume: Permission denied
I0929 02:13:54.319058       1 data-processor.go:391] Checking out file system volume size.
E0929 02:13:54.319092       1 data-processor.go:394] no such file or directory
I0929 02:13:54.319099       1 data-processor.go:399] Request image size not empty.
I0929 02:13:54.319110       1 data-processor.go:404] Target size -1.
I0929 02:13:54.319358       1 data-processor.go:282] New phase: TransferDataFile
E0929 02:13:54.320368       1 data-processor.go:278] exit status 1, blockdev: cannot open /dev/cdi-block-volume: Permission denied

kubevirt.io/containerized-data-importer/pkg/util.GetAvailableSpaceBlock
        pkg/util/util.go:140
kubevirt.io/containerized-data-importer/pkg/util.OpenFileOrBlockDevice
        pkg/util/util.go:168
kubevirt.io/containerized-data-importer/pkg/util.StreamDataToFile
        pkg/util/util.go:187
kubevirt.io/containerized-data-importer/pkg/importer.(*AsyncUploadDataSource).TransferFile
        pkg/importer/upload-datasource.go:158
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).initDefaultPhases.func4
        pkg/importer/data-processor.go:225
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause
        pkg/importer/data-processor.go:275
kubevirt.io/containerized-data-importer/pkg/uploadserver.newAsyncUploadStreamProcessor
        pkg/uploadserver/uploadserver.go:433
kubevirt.io/containerized-data-importer/pkg/uploadserver.(*uploadServerApp).uploadHandlerAsync.func1
        pkg/uploadserver/uploadserver.go:332
net/http.HandlerFunc.ServeHTTP
        GOROOT/src/net/http/server.go:2084
net/http.(*ServeMux).ServeHTTP
        GOROOT/src/net/http/server.go:2462
kubevirt.io/containerized-data-importer/pkg/uploadserver.(*uploadServerApp).ServeHTTP
        pkg/uploadserver/uploadserver.go:261
net/http.serverHandler.ServeHTTP
        GOROOT/src/net/http/server.go:2916
net/http.(*conn).serve
        GOROOT/src/net/http/server.go:1966
runtime.goexit
        GOROOT/src/runtime/asm_amd64.s:1571
error determining if block device exists
kubevirt.io/containerized-data-importer/pkg/util.OpenFileOrBlockDevice
        pkg/util/util.go:170
kubevirt.io/containerized-data-importer/pkg/util.StreamDataToFile
        pkg/util/util.go:187
kubevirt.io/containerized-data-importer/pkg/importer.(*AsyncUploadDataSource).TransferFile
        pkg/importer/upload-datasource.go:158
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).initDefaultPhases.func4
        pkg/importer/data-processor.go:225
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause
        pkg/importer/data-processor.go:275
kubevirt.io/containerized-data-importer/pkg/uploadserver.newAsyncUploadStreamProcessor
        pkg/uploadserver/uploadserver.go:433
kubevirt.io/containerized-data-importer/pkg/uploadserver.(*uploadServerApp).uploadHandlerAsync.func1
        pkg/uploadserver/uploadserver.go:332
net/http.HandlerFunc.ServeHTTP
        GOROOT/src/net/http/server.go:2084
net/http.(*ServeMux).ServeHTTP
        GOROOT/src/net/http/server.go:2462
kubevirt.io/containerized-data-importer/pkg/uploadserver.(*uploadServerApp).ServeHTTP
        pkg/uploadserver/uploadserver.go:261
net/http.serverHandler.ServeHTTP
        GOROOT/src/net/http/server.go:2916
net/http.(*conn).serve
        GOROOT/src/net/http/server.go:1966
runtime.goexit
        GOROOT/src/runtime/asm_amd64.s:1571
Unable to transfer source data to target file
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).initDefaultPhases.func4
        pkg/importer/data-processor.go:227
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause
        pkg/importer/data-processor.go:275
kubevirt.io/containerized-data-importer/pkg/uploadserver.newAsyncUploadStreamProcessor
        pkg/uploadserver/uploadserver.go:433
kubevirt.io/containerized-data-importer/pkg/uploadserver.(*uploadServerApp).uploadHandlerAsync.func1
        pkg/uploadserver/uploadserver.go:332
net/http.HandlerFunc.ServeHTTP
        GOROOT/src/net/http/server.go:2084
net/http.(*ServeMux).ServeHTTP
        GOROOT/src/net/http/server.go:2462
kubevirt.io/containerized-data-importer/pkg/uploadserver.(*uploadServerApp).ServeHTTP
        pkg/uploadserver/uploadserver.go:261
net/http.serverHandler.ServeHTTP
        GOROOT/src/net/http/server.go:2916
net/http.(*conn).serve
        GOROOT/src/net/http/server.go:1966
runtime.goexit
        GOROOT/src/runtime/asm_amd64.s:1571
E0929 02:13:54.320969       1 uploadserver.go:337] Saving stream failed: Unable to transfer source data to target file: error determining if block device exists: exit status 1, blockdev: cannot open /dev/cdi-block-volume: Permission denied

@awels
Copy link
Member

awels commented Sep 29, 2022

I suspect its the RunAsNonRoot, maybe in combination with centos 7.9?

@farcaller
Copy link
Contributor

This is broken because of #2410. The proxy runs as user 1001:

1001      166213  0.2  0.0 1775980 21648 ?       Ssl  Sep29   3:49 /usr/bin/cdi-uploadproxy -alsologtostderr -v=1

but the block device is 0660:

$ stat /dev/cdi-block-volume
  File: /dev/cdi-block-volume
  Size: 0               Blocks: 0          IO Block: 4096   block special file
Device: fbh/251d        Inode: 12          Links: 1     Device type: e6,10
Access: (0660/brw-rw----)  Uid: (    0/    root)   Gid: (    6/    disk)
Access: 2022-09-30 13:42:51.245507964 +0000
Modify: 2022-09-30 13:42:51.245507964 +0000
Change: 2022-09-30 13:42:51.245507964 +0000
 Birth: -

It results in a confused logic flow:

I0930 13:44:33.073755       1 data-processor.go:379] Calculating available size
E0930 13:44:33.077649       1 data-processor.go:383] exit status 1, blockdev: cannot open /dev/cdi-block-volume: Permission denied
I0930 13:44:33.078157       1 data-processor.go:391] Checking out file system volume size.
E0930 13:44:33.078195       1 data-processor.go:394] no such file or directory
I0930 13:44:33.078202       1 data-processor.go:399] Request image size not empty.
I0930 13:44:33.078539       1 data-processor.go:404] Target size -1.
I0930 13:44:33.078628       1 data-processor.go:282] New phase: TransferScratch

because here the blockdev fails with insufficient permissions, and CDI thinks it's not a block volume but an FS.

@awels
Copy link
Member

awels commented Sep 30, 2022

There are 2 components, the proxy (which runs in the cdi namespace) which simply forwards requests to the upload-server which runs in the user namespace. With #2410 the upload-server should run as the kvm user (uid 107). The block device should be readable/writable by that user. If it is not, can we get some more specifics about your storage? The original poster seems to be using rook-ceph, which we also use in our tests.

@farcaller
Copy link
Contributor

My storage backend is zfs-local-pv.

@mhenriks
Copy link
Member

mhenriks commented Oct 5, 2022

@farcaller I assume there is no issue with Filesystem PVC with same provisioner? What are the advantages of using block?

@awels
Copy link
Member

awels commented Oct 21, 2022

So we found this https://kubernetes.io/blog/2021/11/09/non-root-containers-and-devices/ and we were wondering if that might fix the issue for you, basically setting `device_ownership_from_security_context = true' for your CRI. Let us know if that fixed the issue so we can document it.

@awels awels mentioned this issue Oct 21, 2022
Closed
@bc185174
Copy link
Contributor

bc185174 commented Nov 10, 2022 •
edited
Loading

So we found this https://kubernetes.io/blog/2021/11/09/non-root-containers-and-devices/ and we were wondering if that might fix the issue for you, basically setting `device_ownership_from_security_context = true' for your CRI. Let us know if that fixed the issue so we can document it.

I tried this suggestion, setting device_ownership_from_security_context = true in CRI config and it worked. No blockdev: cannot open /dev/cdi-block-volume: Permission denied error. Should also resolve my comment in #2447 (comment) for what I think is the same issue.

@awels
Copy link
Member

awels commented Nov 10, 2022

We updated our documentation to include that particular fix with this PR #2458 closing this issue

@awels awels closed this as completed Nov 10, 2022
@kvaps kvaps mentioned this issue Nov 11, 2022
Open
89 tasks
@moshuipan moshuipan mentioned this issue Mar 16, 2023
Closed
@p-alik p-alik mentioned this issue Aug 22, 2023
Closed
@RoryDoherty RoryDoherty mentioned this issue Oct 18, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@farcaller @awels @mhenriks @tghfly @bc185174