Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can not create ceph primary storage? #5741

Closed
xuanyuanaosheng opened this issue Dec 2, 2021 · 14 comments
Closed

Can not create ceph primary storage? #5741

xuanyuanaosheng opened this issue Dec 2, 2021 · 14 comments

Comments

@xuanyuanaosheng
Copy link

xuanyuanaosheng commented Dec 2, 2021 •
edited
Loading

ISSUE TYPE
  • Bug Report

The doc is using:

  1. http://docs.ceph.org.cn/rbd/rbd-cloudstack/
  2. https://www.shapeblue.com/ceph-and-cloudstack-part-2/#top
  3. https://docs.ceph.com/en/latest/rbd/rbd-cloudstack/
  4. https://www.slideshare.net/ShapeBlue/ceph-with-cloudstack
  5. The related issue: 4.11.3.0 create ceph primary storage error #3523
  6. Ceph RBD primary storage fails connection and renders node unusable #2611
  7. The question about the cloudstack primary storage #5751
CLOUDSTACK VERSION

The OS is Red Hat Enterprise Linux release 8.3 (Ootpa)
The cloudstack management version: 4.15.2.0
The cloudstack agent version: cloudstack-agent-4.15.2.0-1.el8.x86_64
The ceph version: 15.2.13

The KVM host has ceph-common
# rpm -qa | grep librbd1
librbd1-15.2.13-0.el7.x86_64

# rpm -qa | grep librados
libradosstriper1-15.2.13-0.el7.x86_64
librados2-15.2.13-0.el7.x86_64

# rpm -qa | grep ceph 
libcephfs2-15.2.13-0.el7.x86_64
python3-ceph-common-15.2.13-0.el7.x86_64
ceph-common-15.2.13-0.el7.x86_64
python3-cephfs-15.2.13-0.el7.x86_64
python3-ceph-argparse-15.2.13-0.el7.x86_64

The ceph storage is

1.  ceph osd pool create cloudstack
2.  rbd pool init cloudstack
3.  ceph auth get-or-create client.cloudstack mon 'profile rbd' osd 'profile rbd pool=cloudstack'

Using cloudstack web UI:

image

The cloudstack management error is:

2021-12-02 17:42:57,562 DEBUG [c.c.a.t.Request] (AgentManager-Handler-10:null) (logid:) Seq 15-8895735163963595060: Processing:  { Ans: , MgmtId: 345052215515, via: 15, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: b90eae9d-973c-362c-8afc-af88f0743892
	at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:645)
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:329)
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:323)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1648)
	at com.cloud.agent.Agent.processRequest(Agent.java:661)
	at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079)
	at com.cloud.utils.nio.Task.call(Task.java:83)
	at com.cloud.utils.nio.Task.call(Task.java:29)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
","wait":"0","bypassHostMaintenance":"false"}}] }
2021-12-02 17:42:57,562 DEBUG [c.c.a.t.Request] (qtp769798433-969:ctx-91c6013f ctx-bc3c2687) (logid:731db75b) Seq 15-8895735163963595060: Received:  { Ans: , MgmtId: 345052215515, via: 15(whdckvm002.cn.prod), Ver: v1, Flags: 10, { Answer } }
2021-12-02 17:42:57,563 DEBUG [c.c.a.m.AgentManagerImpl] (qtp769798433-969:ctx-91c6013f ctx-bc3c2687) (logid:731db75b) Details from executing class com.cloud.agent.api.ModifyStoragePoolCommand: com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: b90eae9d-973c-362c-8afc-af88f0743892
	at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:645)
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:329)
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:323)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1648)
	at com.cloud.agent.Agent.processRequest(Agent.java:661)
	at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079)
	at com.cloud.utils.nio.Task.call(Task.java:83)
	at com.cloud.utils.nio.Task.call(Task.java:29)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

2021-12-02 17:42:57,565 WARN  [c.c.a.AlertManagerImpl] (qtp769798433-969:ctx-91c6013f ctx-bc3c2687) (logid:731db75b) AlertType:: 7 | dataCenterId:: 4 | podId:: 4 | clusterId:: null | message:: Unable to attach storage pool27 to the host15
2021-12-02 17:42:57,572 WARN  [o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl] (qtp769798433-969:ctx-91c6013f ctx-bc3c2687) (logid:731db75b) Unable to establish a connection between Host[-15-Routing] and org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@5ac383cb
com.cloud.utils.exception.CloudRuntimeException: Unable establish connection from storage head to storage pool 27 due to com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: b90eae9d-973c-362c-8afc-af88f0743892
	at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:645)
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:329)
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:323)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1648)
	at com.cloud.agent.Agent.processRequest(Agent.java:661)
	at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079)
	at com.cloud.utils.nio.Task.call(Task.java:83)
	at com.cloud.utils.nio.Task.call(Task.java:29)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
27
	at org.apache.cloudstack.storage.datastore.provider.DefaultHostListener.hostConnect(DefaultHostListener.java:82)
	at com.cloud.storage.StorageManagerImpl.connectHostToSharedPool(StorageManagerImpl.java:1036)
	at jdk.internal.reflect.GeneratedMethodAccessor7159.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
	at com.sun.proxy.$Proxy68.connectHostToSharedPool(Unknown Source)
	at org.apache.cloudstack.storage.datastore.lifecycle.CloudStackPrimaryDataStoreLifeCycleImpl.attachCluster(CloudStackPrimaryDataStoreLifeCycleImpl.java:451)
	at com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:749)
	at com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:218)
	at jdk.internal.reflect.GeneratedMethodAccessor6956.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
	at com.sun.proxy.$Proxy68.createPool(Unknown Source)
	at org.apache.cloudstack.api.command.admin.storage.CreateStoragePoolCmd.execute(CreateStoragePoolCmd.java:163)
	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
	at com.cloud.api.ApiServer.queueCommand(ApiServer.java:764)
	at com.cloud.api.ApiServer.handleRequest(ApiServer.java:588)
	at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:321)
	at com.cloud.api.ApiServlet$1.run(ApiServlet.java:134)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
	at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:131)
	at com.cloud.api.ApiServlet.doPost(ApiServlet.java:98)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
	at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1443)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:766)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
	at java.base/java.lang.Thread.run(Thread.java:834)
2021-12-02 17:42:57,573 WARN  [o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl] (qtp769798433-969:ctx-91c6013f ctx-bc3c2687) (logid:731db75b) No host can access storage pool org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@5ac383cb on cluster 6
2021-12-02 17:42:57,578 DEBUG [c.c.s.StorageManagerImpl] (qtp769798433-969:ctx-91c6013f ctx-bc3c2687) (logid:731db75b) Failed to add data store: Failed to access storage pool
com.cloud.utils.exception.CloudRuntimeException: Failed to access storage pool
	at org.apache.cloudstack.storage.datastore.lifecycle.CloudStackPrimaryDataStoreLifeCycleImpl.attachCluster(CloudStackPrimaryDataStoreLifeCycleImpl.java:464)
	at com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:749)
	at com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:218)
	at jdk.internal.reflect.GeneratedMethodAccessor6956.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
	at com.sun.proxy.$Proxy68.createPool(Unknown Source)
	at org.apache.cloudstack.api.command.admin.storage.CreateStoragePoolCmd.execute(CreateStoragePoolCmd.java:163)
	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
	at com.cloud.api.ApiServer.queueCommand(ApiServer.java:764)
	at com.cloud.api.ApiServer.handleRequest(ApiServer.java:588)
	at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:321)
	at com.cloud.api.ApiServlet$1.run(ApiServlet.java:134)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
	at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:131)
	at com.cloud.api.ApiServlet.doPost(ApiServlet.java:98)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
	at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1443)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:766)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
	at java.base/java.lang.Thread.run(Thread.java:834)
2021-12-02 17:42:57,583 DEBUG [c.c.s.StorageManagerImpl] (qtp769798433-969:ctx-91c6013f ctx-bc3c2687) (logid:731db75b) Failed to clean up storage pool: null
2021-12-02 17:42:57,583 INFO  [c.c.a.ApiServer] (qtp769798433-969:ctx-91c6013f ctx-bc3c2687) (logid:731db75b) Failed to add data store: Failed to access storage pool
2021-12-02 17:42:57,584 DEBUG [c.c.a.ApiServlet] (qtp769798433-969:ctx-91c6013f ctx-bc3c2687) (logid:731db75b) ===END===  10.30.148.74 -- POST  command=createStoragePool&response=json

The cloudstack client error is: failed to create the RBD IoCTX. Does the pool 'cloudstack' exist?: No such file or directory

The detail is:

2021-12-02 17:44:31,128 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:7fffda3e) <secret ephemeral='no' private='no'>
<uuid>b90eae9d-973c-362c-8afc-af88f0743892</uuid>
<usage type='ceph'>
<name>[email protected]:6789/cloudstack</name>
</usage>
</secret>

2021-12-02 17:44:31,130 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:7fffda3e) <pool type='rbd'>
<name>b90eae9d-973c-362c-8afc-af88f0743892</name>
<uuid>b90eae9d-973c-362c-8afc-af88f0743892</uuid>
<source>
<host name='10.29.44.1' port='6789'/>
<name>cloudstack</name>
<auth username='cloudstack' type='ceph'>
<secret uuid='b90eae9d-973c-362c-8afc-af88f0743892'/>
</auth>
</source>
</pool>

2021-12-02 17:44:44,631 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-5:null) (logid:) Executing: /usr/share/cloudstack-common/scripts/vm/network/security_group.py get_rule_logs_for_vms 
2021-12-02 17:44:44,633 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-5:null) (logid:) Executing while with timeout : 1800000
2021-12-02 17:44:44,705 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-5:null) (logid:) Execution is successful.
2021-12-02 17:44:44,705 DEBUG [kvm.resource.LibvirtConnection] (UgentTask-5:null) (logid:) Looking for libvirtd connection at: qemu:///system
2021-12-02 17:44:44,721 DEBUG [cloud.agent.Agent] (UgentTask-5:null) (logid:) Sending ping: Seq 15-10082:  { Cmd , MgmtId: -1, via: 15, Ver: v1, Flags: 11, [{"com.cloud.agent.api.PingRoutingWithNwGroupsCommand":{"newGroupStates":{},"_hostVmStateReport":{"v-249-VM":{"state":"PowerOn","host":"whdckvm002.cn.prod"},"s-250-VM":{"state":"PowerOn","host":"whdckvm002.cn.prod"},"r-254-VM":{"state":"PowerOn","host":"whdckvm002.cn.prod"}},"_gatewayAccessible":"true","_vnetAccessible":"true","hostType":"Routing","hostId":"15","wait":"0","bypassHostMaintenance":"false"}}] }
2021-12-02 17:44:44,745 DEBUG [cloud.agent.Agent] (Agent-Handler-2:null) (logid:c8942770) Received response: Seq 15-10082:  { Ans: , MgmtId: 345052215515, via: 15, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.PingAnswer":{"_command":{"hostType":"Routing","hostId":"15","wait":"0","bypassHostMaintenance":"false"},"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }
2021-12-02 17:44:52,586 DEBUG [kvm.resource.LibvirtConnection] (Thread-10967:null) (logid:) Looking for libvirtd connection at: qemu:///system
2021-12-02 17:44:52,601 DEBUG [kvm.resource.KVMHAMonitor] (Thread-10967:null) (logid:) Found NFS storage pool c8e9ca6a-c004-3851-a074-19f4948b28ff in libvirt, continuing
2021-12-02 17:44:52,601 DEBUG [kvm.resource.KVMHAMonitor] (Thread-10967:null) (logid:) Executing: /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh -i 10.26.246.6 -p /kvm-data -m /mnt/c8e9ca6a-c004-3851-a074-19f4948b28ff -h 10.26.246.6 
2021-12-02 17:44:52,602 DEBUG [kvm.resource.KVMHAMonitor] (Thread-10967:null) (logid:) Executing while with timeout : 60000
2021-12-02 17:44:52,616 DEBUG [kvm.resource.KVMHAMonitor] (Thread-10967:null) (logid:) Execution is successful.
2021-12-02 17:44:52,617 DEBUG [kvm.resource.LibvirtConnection] (Thread-10967:null) (logid:) Looking for libvirtd connection at: qemu:///system
2021-12-02 17:44:52,618 DEBUG [kvm.resource.KVMHAMonitor] (Thread-10967:null) (logid:) Found NFS storage pool 60b59087-7c53-3058-a50c-f50737e556bc in libvirt, continuing
2021-12-02 17:44:52,618 DEBUG [kvm.resource.KVMHAMonitor] (Thread-10967:null) (logid:) Executing: /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh -i 10.26.246.5 -p /kvm-data -m /mnt/60b59087-7c53-3058-a50c-f50737e556bc -h 10.26.246.6 
2021-12-02 17:44:52,618 DEBUG [kvm.resource.KVMHAMonitor] (Thread-10967:null) (logid:) Executing while with timeout : 60000
2021-12-02 17:44:52,631 DEBUG [kvm.resource.KVMHAMonitor] (Thread-10967:null) (logid:) Execution is successful.
2021-12-02 17:45:01,167 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:7fffda3e) Failed to create RBD storage pool: org.libvirt.LibvirtException: failed to create the RBD IoCTX. Does the pool 'cloudstack' exist?: No such file or directory
2021-12-02 17:45:01,167 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:7fffda3e) Failed to create the RBD storage pool, cleaning up the libvirt secret
2021-12-02 17:45:01,167 WARN  [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:7fffda3e) Caught: 
com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: b90eae9d-973c-362c-8afc-af88f0743892
	at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:645)
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:329)
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:323)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1648)
	at com.cloud.agent.Agent.processRequest(Agent.java:661)
	at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079)
	at com.cloud.utils.nio.Task.call(Task.java:83)
	at com.cloud.utils.nio.Task.call(Task.java:29)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
2021-12-02 17:45:01,169 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:7fffda3e) Seq 15-8895735163963595047:  { Ans: , MgmtId: 345052215515, via: 15, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: b90eae9d-973c-362c-8afc-af88f0743892
	at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:645)
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:329)
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:323)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1648)
	at com.cloud.agent.Agent.processRequest(Agent.java:661)
	at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079)
	at com.cloud.utils.nio.Task.call(Task.java:83)
	at com.cloud.utils.nio.Task.call(Task.java:29)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
","wait":"0","bypassHostMaintenance":"false"}}] }

Coud you please give some advice ?
@xuanyuanaosheng xuanyuanaosheng mentioned this issue Dec 3, 2021
Closed
@wido
Copy link
Contributor

wido commented Dec 3, 2021

6789 as port number, do the MONs run on that port?

New monitors might only bind on 3300

@xuanyuanaosheng
Copy link
Author

@wido I will use the 3300 port to do some test. thanks

@xuanyuanaosheng
Copy link
Author

xuanyuanaosheng commented Dec 3, 2021 •
edited
Loading

@wido I have tested on port 3300, But It is still not work.

The cloudstack client error message is Failed to create RBD storage pool: org.libvirt.LibvirtException: failed to connect to the RADOS monitor on: 10.29.44.1:3300,: No such file or directory

and the details is :


2021-12-03 18:38:26,318 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:53efb58d) Attempting to create storage pool b90eae9d-973c-362c-8afc-af88f0743892
2021-12-03 18:38:26,318 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:53efb58d) <secret ephemeral='no' private='no'>
<uuid>b90eae9d-973c-362c-8afc-af88f0743892</uuid>
<usage type='ceph'>
<name>[email protected]:3300/cloudstack</name>
</usage>
</secret>

2021-12-03 18:38:26,320 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:53efb58d) <pool type='rbd'>
<name>b90eae9d-973c-362c-8afc-af88f0743892</name>
<uuid>b90eae9d-973c-362c-8afc-af88f0743892</uuid>
<source>
<host name='10.29.44.1' port='3300'/>
<name>cloudstack</name>
<auth username='cloudstack' type='ceph'>
<secret uuid='b90eae9d-973c-362c-8afc-af88f0743892'/>
</auth>
</source>
</pool>

2021-12-03 18:38:32,197 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) (logid:2f5b05d6) Processing command: com.cloud.agent.api.GetHostStatsCommand
2021-12-03 18:38:56,354 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:53efb58d) Failed to create RBD storage pool: org.libvirt.LibvirtException: failed to connect to the RADOS monitor on: 10.29.44.1:3300,: No such file or directory
2021-12-03 18:38:56,354 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:53efb58d) Failed to create the RBD storage pool, cleaning up the libvirt secret
2021-12-03 18:38:56,355 WARN  [cloud.agent.Agent] (agentRequest-Handler-1:null) (logid:53efb58d) Caught: 
com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: b90eae9d-973c-362c-8afc-af88f0743892
	at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:645)
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:329)
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:323)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1648)
	at com.cloud.agent.Agent.processRequest(Agent.java:661)
	at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079)
	at com.cloud.utils.nio.Task.call(Task.java:83)
	at com.cloud.utils.nio.Task.call(Task.java:29)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
2021-12-03 18:38:56,357 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:null) (logid:53efb58d) Seq 15-9029435777901133923:  { Ans: , MgmtId: 345052215515, via: 15, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: b90eae9d-973c-362c-8afc-af88f0743892
	at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:645)
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:329)
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:323)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1648)
	at com.cloud.agent.Agent.processRequest(Agent.java:661)
	at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079)
	at com.cloud.utils.nio.Task.call(Task.java:83)
	at com.cloud.utils.nio.Task.call(Task.java:29)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
","wait":"0","bypassHostMaintenance":"false"}}] }

@weizhouapache
Copy link
Member

@xuanyuanaosheng
the error message asked Does the pool 'cloudstack' exist?, can you check it ?

@xuanyuanaosheng
Copy link
Author

xuanyuanaosheng commented Dec 4, 2021 •
edited
Loading

@weizhouapache I use the command on the kvm node:

# rbd map -p cloudstack cloudstack-primary --user cloudstack
rbd: warning: image already mapped as /dev/rbd0
/dev/rbd5

# ls /dev/rbd
rbd/  rbd0  rbd1  rbd2  rbd3  rbd4  rbd5

I think the pool 'cloudstack' is existed.

Cloud you please give some advice?

@xuanyuanaosheng
Copy link
Author

xuanyuanaosheng commented Dec 6, 2021 •
edited
Loading

@weizhouapache @wido Could you please give some advice?
The new info, I found in cloudstack UI:

image

@weizhouapache
Copy link
Member

@weizhouapache @wido Could you please give some advice? The new info, I found in cloudstack UI:

image

@xuanyuanaosheng
have you seen @leolleeooleo 's comment (#3523 (comment)) ?

@wido
Copy link
Contributor

wido commented Dec 6, 2021

@weizhouapache @wido Could you please give some advice? The new info, I found in cloudstack UI:

image

This seems like something outside CloudStack, double check:

  • All Hosts can reach the Ceph cluster nodes
  • RBD pool exists
  • User has access to the RBD pool
  • There is no / (slash) in the secret

@xuanyuanaosheng
Copy link
Author

xuanyuanaosheng commented Dec 6, 2021 •
edited
Loading

@wido @weizhouapache

I have check all the config as your advices.

  1. The label.rados.monitor is 10.29.44.1:6789 , The kvm node and the v-xxx-VM and s-xxx-VM can access the ceph node.
kvm001:~# telnet 10.29.44.1 6789
Trying 10.29.44.1...
Connected to 10.29.44.1.
Escape character is '^]'.
ceph v027 
, 
񗲨ell


root@s-252-VM:~#  telnet 10.29.44.1 6789
Trying 10.29.44.1...
Connected to 10.29.44.1.
Escape character is '^]'.
ceph v027 
,
  1. The RBD pool is exists: cloudstack

d3c6cea2-8e68-41f4-84dd-06d74869de06

  1. The user cloudstack has the access to the RBD pool: cloudstack

72b81e3f-5633-4d8d-9dc9-9f52cf0f93da

  1. The secret is 'AQDLxqlhIdOLJRAABPqps8O6eSGbFnyR7aSJwQ==' that has no / (slash) in the secret.

  2. The KVM node has ceph config: ceph.client.admin.keyring.

# ls /etc/ceph/
ceph.client.admin.keyring  ceph.client.cloudstack.keyring  ceph.conf  rbdmap

image

  1. The related info and The libvirt secrets has nothing:
# virsh pool-list
 Name                                   State    Autostart
------------------------------------------------------------
 60b59087-7c53-3058-a50c-f50737e556bc   active   no  
 c4355ed4-8833-381f-b3f7-2981782ee3fa   active   no
 c8e9ca6a-c004-3851-a074-19f4948b28ff   active   no
 d8dabcb0-1a57-4e13-8a82-339b2052dec1   active   no

# virsh secret-list
 UUID   Usage
---------------

# ls -a /etc/libvirt/secrets/
.  ..

image

image

image

image

But I can not find the storage pool d8dabcb0-1a57-4e13-8a82-339b2052dec1 on cloudstack UI and the storage pool d8dabcb0-1a57-4e13-8a82-339b2052dec1 will change when I reclick the add primary stotage button.

After checked all the config, I restart the management-server and cloudstack-agent services. The error is still the same: org.libvirt.LibvirtException: failed to create the RBD IoCTX. Does the pool 'cloudstack' exist?: No such file or directory

2021-12-06 17:54:17,921 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:96eddfd2) <pool type='rbd'>
<name>b90eae9d-973c-362c-8afc-af88f0743892</name>
<uuid>b90eae9d-973c-362c-8afc-af88f0743892</uuid>
<source>
<host name='10.29.44.1' port='6789'/>
<name>cloudstack</name>
<auth username='cloudstack' type='ceph'>
<secret uuid='b90eae9d-973c-362c-8afc-af88f0743892'/>
</auth>
</source>
</pool>

2021-12-06 17:54:39,461 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-5:null) (logid:) Executing: /usr/share/cloudstack-common/scripts/vm/network/security_group.py get_rule_logs_for_vms 
2021-12-06 17:54:39,463 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-5:null) (logid:) Executing while with timeout : 1800000
2021-12-06 17:54:39,534 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-5:null) (logid:) Execution is successful.
2021-12-06 17:54:39,535 DEBUG [kvm.resource.LibvirtConnection] (UgentTask-5:null) (logid:) Looking for libvirtd connection at: qemu:///system
2021-12-06 17:54:39,551 DEBUG [cloud.agent.Agent] (UgentTask-5:null) (logid:) Sending ping: Seq 15-6:  { Cmd , MgmtId: -1, via: 15, Ver: v1, Flags: 11, [{"com.cloud.agent.api.PingRoutingWithNwGroupsCommand":{"newGroupStates":{},"_hostVmStateReport":{"v-255-VM":{"state":"PowerOn","host":"whdckvm002.cn.prod"},"v-249-VM":{"state":"PowerOn","host":"whdckvm002.cn.prod"},"s-250-VM":{"state":"PowerOn","host":"whdckvm002.cn.prod"},"r-254-VM":{"state":"PowerOn","host":"whdckvm002.cn.prod"}},"_gatewayAccessible":"true","_vnetAccessible":"true","hostType":"Routing","hostId":"15","wait":"0","bypassHostMaintenance":"false"}}] }
2021-12-06 17:54:39,620 DEBUG [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Received response: Seq 15-6:  { Ans: , MgmtId: 345052215515, via: 15, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.PingAnswer":{"_command":{"hostType":"Routing","hostId":"15","wait":"0","bypassHostMaintenance":"false"},"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }
2021-12-06 17:54:47,958 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:96eddfd2) Failed to create RBD storage pool: org.libvirt.LibvirtException: failed to create the RBD IoCTX. Does the pool 'cloudstack' exist?: No such file or directory
2021-12-06 17:54:47,959 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:96eddfd2) Failed to create the RBD storage pool, cleaning up the libvirt secret
2021-12-06 17:54:47,961 WARN  [cloud.agent.Agent] (agentRequest-Handler-4:null) (logid:96eddfd2) Caught: 
com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: b90eae9d-973c-362c-8afc-af88f0743892
	at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:645)
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:329)
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:323)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1648)
	at com.cloud.agent.Agent.processRequest(Agent.java:661)
	at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079)
	at com.cloud.utils.nio.Task.call(Task.java:83)
	at com.cloud.utils.nio.Task.call(Task.java:29)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
2021-12-06 17:54:47,966 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:null) (logid:96eddfd2) Seq 15-6627046851675684885:  { Ans: , MgmtId: 345052215515, via: 15, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: b90eae9d-973c-362c-8afc-af88f0743892
	at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:645)
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:329)
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:323)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1648)
	at com.cloud.agent.Agent.processRequest(Agent.java:661)
	at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079)
	at com.cloud.utils.nio.Task.call(Task.java:83)
	at com.cloud.utils.nio.Task.call(Task.java:29)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
","wait":"0","bypassHostMaintenance":"false"}}] }
2021-12-06 17:54:52,709 DEBUG [kvm.resource.LibvirtConnection] (Thread-6:null) (logid:) Looking for libvirtd connection at: qemu:///system
2021-12-06 17:54:52,725 DEBUG [kvm.resource.KVMHAMonitor] (Thread-6:null) (logid:) Found NFS storage pool c8e9ca6a-c004-3851-a074-19f4948b28ff in libvirt, continuing
2021-12-06 17:54:52,725 DEBUG [kvm.resource.KVMHAMonitor] (Thread-6:null) (logid:) Executing: /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh -i 10.26.246.6 -p /kvm-data -m /mnt/c8e9ca6a-c004-3851-a074-19f4948b28ff -h 10.26.246.6 
2021-12-06 17:54:52,726 DEBUG [kvm.resource.KVMHAMonitor] (Thread-6:null) (logid:) Executing while with timeout : 60000
2021-12-06 17:54:52,737 DEBUG [kvm.resource.KVMHAMonitor] (Thread-6:null) (logid:) Execution is successful.

Any idears? Can you give some test scripts that can test the ceph storage is ok on the KVM node?

@xuanyuanaosheng xuanyuanaosheng mentioned this issue Dec 6, 2021
Closed
@weizhouapache
Copy link
Member

@xuanyuanaosheng

in my opinion, it is not a cloudstack issue.

Could you please try to add ceph storage by libvirt on kvm nodes ?

  1. virsh secret-define
  2. virsh secret-set-value
  3. virsh pool-create

@xuanyuanaosheng
Copy link
Author

xuanyuanaosheng commented Dec 8, 2021 •
edited
Loading

@weizhouapache @wido

I find the below error when create storage pool on virt-manager.

image

Error creating pool: Could not start storage pool: failed to connect to the RADOS monitor on: 10.29.44.1,: Success

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 75, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/createpool.py", line 378, in _async_pool_create
    poolobj = pool.install(create=True, meter=meter, build=build)
  File "/usr/share/virt-manager/virtinst/storage.py", line 415, in install
    raise RuntimeError(errmsg)
RuntimeError: Could not start storage pool: failed to connect to the RADOS monitor on: whdrcceph001.cn.prod,: Success

I use the methods as you advice, But the error info is too little, I don't know how to repair it. I google a lot, But failed.

# virsh pool-create --file pool.xml
error: Failed to create pool from pool.xml
error: failed to create the RBD IoCTX. Does the pool 'cloudstack' exist?: No such file or directory

The ceph error is

debug 2021-12-09T08:31:10.091+0000 7f2d4b620700  0 cephx server client.cloudstack:  attempt to reclaim global_id 392850 without presenting ticket
debug 2021-12-09T08:31:10.091+0000 7f2d4b620700  0 cephx server client.cloudstack:  could not verify old ticket

@wido
Copy link
Contributor

wido commented Dec 9, 2021

This seems like a client <> server issue with Ceph and does not have anything to do with CloudStack.

attempt to reclaim global_id 392850 without presenting ticket

That suggest that you have something misconfigured in Ceph, there have been recent changes around that. Please refer to the Release Notes of Ceph.

Once that is fixed it should also work in CloudStack.

@xuanyuanaosheng
Copy link
Author

@wido @weizhouapache: This is a problem in ceph.
The related issue is:

  1. CVE-2021-20288 - ceph-csi is using insecure global_id reclaim ceph/ceph-csi#2063
  2. ceph: Mons are allowing insecure global_id reclaim rook/rook#7746
  3. https://docs.ceph.com/en/latest/security/CVE-2021-20288/

To fix this problem:

# ceph config set mon auth_expose_insecure_global_id_reclaim false

# ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false

# ceph config set mon auth_allow_insecure_global_id_reclaim false

@xuanyuanaosheng
Copy link
Author

@wido @weizhouapache Thanks for you help. I will close this issue.

@xuanyuanaosheng xuanyuanaosheng mentioned this issue Dec 15, 2021
Closed
@rohityadavcloud rohityadavcloud mentioned this issue Aug 17, 2022
Closed
@kiranchavala kiranchavala mentioned this issue Oct 15, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@wido @xuanyuanaosheng @weizhouapache