Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gateway fails to load subsystems upon nvmeof service restart #255

Closed
caroav opened this issue Oct 5, 2023 · 1 comment · Fixed by #258
Closed

Gateway fails to load subsystems upon nvmeof service restart #255

caroav opened this issue Oct 5, 2023 · 1 comment · Fixed by #258
Assignees

Comments

@caroav
Copy link
Collaborator

caroav commented Oct 5, 2023

ceph-nvmeof setup with a subsystem and nearly 150 namespaces and able to see them with get_subsystems command.

Upon nvmeof service restart, restart is successful but GW is unable to load all configured targets(subsystem and namespaces)though OMAP holds all entries and updates and get_subsystems lists none

Version-Release number of selected component (if applicable):
ceph version 18.0.0-6175-g38f11f28 (38f11f28b05bfd80aaf2644cc1660ef8b51dd272) reef (dev)

How reproducible:
always

Steps to Reproduce:

  1. Deploy NVMeOF GW on ceph cluster and scale its entities - 1 subsystem and nearly 150 namespaces
  2. Restart nvmeof service - "ceph orch restart nvmeof.rbd"
  3. Upon get_subsystems, no entities are listed

Actual results:
GW fails to load OMAP entries upon a restart

Expected results:
GW should load OMAP entries correctly upon a restart

@caroav
Copy link
Collaborator Author

caroav commented Oct 5, 2023

INFO:control.grpc:Received request to create bdev U7U9-bdev243 from rbd/U7U9-image243 with block size 4096
INFO:control.grpc:create_bdev: []
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib64/python3.9/threading.py", line 980, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.9/threading.py", line 917, in run
self._target(*self._args, **self._kwargs)
File "/remote-source/ceph-nvmeof/app/control/state.py", line 420, in _update_caller
self.update()
File "/remote-source/ceph-nvmeof/app/control/state.py", line 465, in update
self._update_call_rpc(grouped_added, True, prefix_list)
File "/remote-source/ceph-nvmeof/app/control/state.py", line 487, in _update_call_rpc
self.gateway_rpc_caller(component_update, True)
File "/remote-source/ceph-nvmeof/app/control/server.py", line 297, in gateway_rpc_caller
self.gateway_rpc.create_bdev(req)
File "/remote-source/ceph-nvmeof/app/control/grpc.py", line 128, in create_bdev
return pb2.bdev(bdev_name=bdev_name, status=True)
TypeError: bad argument type for built-in operation
[2023-10-04 10:42:15.478168] bdev_rbd.c:1199:bdev_rbd_create: NOTICE: Add U7U9-bdev243 rbd disk to lun
INFO:control.grpc:get_subsystems: U7U9-bdev243
INFO:control.grpc:Received request to get subsystems
INFO:control.grpc:get_subsystems: []
INFO:control.grpc:Received request to get subsystems
INFO:control.grpc:get_subsystems: []
INFO:control.grpc:Received request to get subsystems
INFO:control.grpc:get_subsystems: []
INFO:control.grpc:Received request to get subsystems
INFO:control.grpc:get_subsystems: []
Stopping Ceph nvmeof.rbd.ceph-nvmf3-hrhd31-node5.begrve for 33fddd3a-626e-11ee-8bcb-fa163e0c7e19...

gbregman added a commit to gbregman/ceph-nvmeof that referenced this issue Oct 10, 2023
gbregman added a commit to gbregman/ceph-nvmeof that referenced this issue Oct 10, 2023
gbregman added a commit to gbregman/ceph-nvmeof that referenced this issue Oct 10, 2023
gbregman added a commit to gbregman/ceph-nvmeof that referenced this issue Oct 10, 2023
gbregman added a commit to gbregman/ceph-nvmeof that referenced this issue Oct 11, 2023
gbregman added a commit to gbregman/ceph-nvmeof that referenced this issue Oct 11, 2023
gbregman added a commit to gbregman/ceph-nvmeof that referenced this issue Oct 12, 2023
gbregman added a commit to gbregman/ceph-nvmeof that referenced this issue Oct 12, 2023
gbregman added a commit to gbregman/ceph-nvmeof that referenced this issue Oct 15, 2023
gbregman added a commit to gbregman/ceph-nvmeof that referenced this issue Oct 15, 2023
gbregman added a commit to gbregman/ceph-nvmeof that referenced this issue Oct 15, 2023
gbregman added a commit to gbregman/ceph-nvmeof that referenced this issue Oct 15, 2023
gbregman added a commit to gbregman/ceph-nvmeof that referenced this issue Oct 15, 2023
gbregman added a commit to gbregman/ceph-nvmeof that referenced this issue Oct 16, 2023
@github-project-automation github-project-automation bot moved this from 🆕 New to ✅ Done in NVMe-oF Oct 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
2 participants