ceph mimic iscsi issue

[fabianlv]

my environment:
os:CentOS Linux release 7.6.1810 (Core)
kernel:Linux ceph-all 4.19.10 #1 SMP Mon Jan 14 21:47:11 CST 2019 x86_64 x86_64 x86_64 GNU/Linux
ceph version:ceph -v
ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable)

rpm packages version:
[root@ceph-all log]# rpm -qa|grep rtslib
python-rtslib-2.1.67-1.noarch
[root@ceph-all log]# rpm -qa|grep targetcli
targetcli-2.1.fb47-0.1.20170815.git5bf3517.el7cp.noarch
[root@ceph-all log]# rpm -qa|grep targetcli
targetcli-2.1.fb47-0.1.20170815.git5bf3517.el7cp.noarch
[root@ceph-all log]# rpm -qa|grep ceph-iscsi
ceph-iscsi-config-2.6-2.6.el7.noarch
ceph-iscsi-tools-2.1-3.el7cp.noarch
ceph-iscsi-cli-2.7-2.7.el7.noarch

ceph -s :
[root@ceph-all log]# ceph -s
cluster:
id: bdd73746-31ff-4155-96d5-a2b7e1d933b2
health: HEALTH_WARN
2/3 objects misplaced (66.667%)
Reduced data availability: 160 pgs inactive
Degraded data redundancy: 243 pgs undersized

services:
mon: 1 daemons, quorum ceph-all
mgr: mgr1(active)
osd: 6 osds: 6 up, 6 in; 96 remapped pgs

data:
pools: 1 pools, 256 pgs
objects: 1 objects, 167 B
usage: 107 MiB used, 5.9 GiB / 6.0 GiB avail
pgs: 62.500% pgs not active
2/3 objects misplaced (66.667%)
160 undersized+peered
83 active+undersized+remapped
13 active+clean+remapped

[root@ceph-all log]# ceph osd lspools
5 rbd
[root@ceph-all log]# ceph osd pool stats
pool rbd id 5
2/3 objects misplaced (66.667%)

my problem:
rbd-target-api service can not start

log:
Jan 15 00:17:03 ceph-all polkitd[9498]: Registered Authentication Agent for unix-process:13912:126693 (system bus name :1.64 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)
Jan 15 00:17:03 ceph-all systemd[1]: Started Ceph iscsi target configuration API.
Jan 15 00:17:03 ceph-all polkitd[9498]: Unregistered Authentication Agent for unix-process:13912:126693 (system bus name :1.64, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)
Jan 15 00:17:03 ceph-all rbd-target-api[13918]: Started the configuration object watcher
Jan 15 00:17:03 ceph-all rbd-target-api[13918]: Checking for config object changes every 1s
Jan 15 00:17:03 ceph-all rbd-target-api[13918]: Secure API requested but the crt/key files missing/incompatible?
Jan 15 00:17:03 ceph-all rbd-target-api[13918]: Unable to start
Jan 15 00:17:03 ceph-all systemd[1]: rbd-target-api.service: main process exited, code=exited, status=16/n/a
Jan 15 00:17:03 ceph-all systemd[1]: Unit rbd-target-api.service entered failed state.
Jan 15 00:17:03 ceph-all systemd[1]: rbd-target-api.service failed.
Jan 15 00:17:03 ceph-all systemd[1]: rbd-target-api.service holdoff time over, scheduling restart.
Jan 15 00:17:03 ceph-all systemd[1]: Stopped Ceph iscsi target configuration API.
Jan 15 00:17:03 ceph-all systemd[1]: Started Ceph iscsi target configuration API.
Jan 15 00:17:03 ceph-all rbd-target-api[13961]: Started the configuration object watcher
Jan 15 00:17:03 ceph-all rbd-target-api[13961]: Secure API requested but the crt/key files missing/incompatible?
Jan 15 00:17:03 ceph-all rbd-target-api[13961]: Unable to start
Jan 15 00:17:03 ceph-all rbd-target-api[13961]: Checking for config object changes every 1s
Jan 15 00:17:03 ceph-all rbd-target-api[13961]: Exception in thread Thread-1:
Jan 15 00:17:03 ceph-all rbd-target-api[13961]: Traceback (most recent call last):
Jan 15 00:17:03 ceph-all rbd-target-api[13961]: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
Jan 15 00:17:03 ceph-all rbd-target-api[13961]: self.run()
Jan 15 00:17:03 ceph-all rbd-target-api[13961]: File "/usr/bin/rbd-target-api", line 1816, in run
Jan 15 00:17:03 ceph-all rbd-target-api[13961]: cluster = rados.Rados(conffile=settings.config.cephconf)
Jan 15 00:17:03 ceph-all rbd-target-api[13961]: AttributeError: 'NoneType' object has no attribute 'Rados'
Jan 15 00:17:03 ceph-all systemd[1]: rbd-target-api.service: main process exited, code=exited, status=16/n/a
Jan 15 00:17:03 ceph-all systemd[1]: Unit rbd-target-api.service entered failed state.
Jan 15 00:17:03 ceph-all systemd[1]: rbd-target-api.service failed.
Jan 15 00:17:04 ceph-all systemd[1]: rbd-target-api.service holdoff time over, scheduling restart.
Jan 15 00:17:04 ceph-all systemd[1]: Stopped Ceph iscsi target configuration API.
Jan 15 00:17:04 ceph-all systemd[1]: Started Ceph iscsi target configuration API.
Jan 15 00:17:04 ceph-all rbd-target-api[13990]: Started the configuration object watcher
Jan 15 00:17:04 ceph-all rbd-target-api[13990]: Secure API requested but the crt/key files missing/incompatible?
Jan 15 00:17:04 ceph-all rbd-target-api[13990]: Checking for config object changes every 1s
Jan 15 00:17:04 ceph-all rbd-target-api[13990]: Unable to start
Jan 15 00:17:04 ceph-all kernel: ms_dispatch[14025]: segfault at 7f3922695378 ip 00007f39248e06cf sp 00007f391a26b8d0 error 4 in libceph-common.so.0[7f39243ca000+7d9000]
Jan 15 00:17:04 ceph-all kernel: Code: 1f 84 00 00 00 00 00 0f 1f 40 00 41 57 41 56 41 55 41 54 4c 8d 6f 08 55 53 48 83 ec 08 48 8b 6f 10 48 85 ed 0f 84 ad 00 00 00 <4c> 8b 36 4d 89 ec 4d 8b 7e e8 eb 21 0f 1f 44 00 00 4c 89 f6 e8 28
Jan 15 00:17:04 ceph-all systemd[1]: rbd-target-api.service: main process exited, code=killed, status=11/SEGV
Jan 15 00:17:04 ceph-all systemd[1]: Unit rbd-target-api.service entered failed state.
Jan 15 00:17:04 ceph-all systemd[1]: rbd-target-api.service failed.
Jan 15 00:17:04 ceph-all systemd[1]: rbd-target-api.service holdoff time over, scheduling restart.
Jan 15 00:17:04 ceph-all systemd[1]: Stopped Ceph iscsi target configuration API.
Jan 15 00:17:04 ceph-all systemd[1]: start request repeated too quickly for rbd-target-api.service
Jan 15 00:17:04 ceph-all systemd[1]: Failed to start Ceph iscsi target configuration API.
Jan 15 00:17:04 ceph-all systemd[1]: Unit rbd-target-api.service entered failed state.
Jan 15 00:17:04 ceph-all systemd[1]: rbd-target-api.service failed.

gwcli can not execute
[root@ceph-all log]# gwcli -d
Adding ceph cluster 'ceph' to the UI
Fetching ceph osd information
Querying ceph for state information
Unable to access the configuration object : REST API failure, code : 500
Traceback (most recent call last):
File "/usr/bin/gwcli", line 191, in
main()
File "/usr/bin/gwcli", line 103, in main
root_node.refresh()
File "/usr/lib/python2.7/site-packages/gwcli/gateway.py", line 87, in refresh
raise GatewayError
gwcli.utils.GatewayError

can anyone give me advise?
thanks.

[dillaman]

It looks like you don't have python-rados installed (properly) given:

Jan 15 00:17:03 ceph-all rbd-target-api[13961]: File "/usr/bin/rbd-target-api", line 1816, in run
Jan 15 00:17:03 ceph-all rbd-target-api[13961]: cluster = rados.Rados(conffile=settings.config.cephconf)
Jan 15 00:17:03 ceph-all rbd-target-api[13961]: AttributeError: 'NoneType' object has no attribute 'Rados'

I would try to re-install librados2, librbd1, python-rados, and python-rbd (and make sure the RPM versions match).

[fabianlv]

It looks like you don't have python-rados installed (properly) given:

Jan 15 00:17:03 ceph-all rbd-target-api[13961]: File "/usr/bin/rbd-target-api", line 1816, in run
Jan 15 00:17:03 ceph-all rbd-target-api[13961]: cluster = rados.Rados(conffile=settings.config.cephconf)
Jan 15 00:17:03 ceph-all rbd-target-api[13961]: AttributeError: 'NoneType' object has no attribute 'Rados'

I would try to re-install librados2, librbd1, python-rados, and python-rbd (and make sure the RPM versions match).

[root@ceph-all ~]# yum list |grep python-rados
python-rados.x86_64 2:13.2.4-0.el7 @ceph-mimic-13.2.4
[root@ceph-all ~]# yum list |grep python-rbd
python-rbd.x86_64 2:13.2.4-0.el7 @ceph-mimic-13.2.4
[root@ceph-all ~]# yum list |grep librbd1
librbd1.x86_64 2:13.2.4-0.el7 @ceph-mimic-13.2.4
librbd1.i686 1:10.2.5-4.el7 os
librbd1-devel.i686 1:10.2.5-4.el7 os
librbd1-devel.x86_64 1:10.2.5-4.el7 os
[root@ceph-all ~]# yum list |grep librados2
librados2.x86_64 2:13.2.4-0.el7 @ceph-mimic-13.2.4
librados2.i686 1:10.2.5-4.el7 os
librados2-devel.i686 1:10.2.5-4.el7 os
librados2-devel.x86_64 1:10.2.5-4.el7 os
[root@ceph-all ~]# yum list |grep rados
librados2.x86_64 2:13.2.4-0.el7 @ceph-mimic-13.2.4
libradosstriper1.x86_64 2:13.2.4-0.el7 @ceph-mimic-13.2.4
python-rados.x86_64 2:13.2.4-0.el7 @ceph-mimic-13.2.4
ceph-radosgw.x86_64 2:13.2.2-0.el7 ceph
librados-devel.x86_64 2:13.2.2-0.el7 ceph
librados2.i686 1:10.2.5-4.el7 os
librados2-devel.i686 1:10.2.5-4.el7 os
librados2-devel.x86_64 1:10.2.5-4.el7 os
libradosstriper-devel.x86_64 2:13.2.2-0.el7 ceph
python34-rados.x86_64 2:13.2.2-0.el7 ceph
rados-objclass-devel.x86_64 2:13.2.2-0.el7 ceph

[fabianlv]

i confused how to set trusted_ip_list in iscsi-gateway.cfg
if i disabled trusted_ip_list in iscsi-gateway.cfg,i can execute gwcli
if i set trusted_ip_list in iscsi-gateway.cfg,i can't execute gwcli
my ip environment:
192.168.80.11 ceph1
192.168.80.12 ceph2
192.168.80.13 ceph3
192.168.80.180 centos1810
192.168.10.11 ceph1-iscsi
192.168.10.12 ceph2-iscsi
192.168.10.13 ceph3-iscsi

ceph.conf
[global]
fsid = 9a3d7e76-8ce1-46e2-940f-083be11712d3
public_network = 192.168.80.0/24
cluster_network = 192.168.10.0/24
mon_initial_members = ceph1,ceph2,ceph3
mon_host = 192.168.80.11,192.168.80.12,192.168.80.13
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
mon_allow_pool_delete=true

dumped monmap epoch 1
epoch 1
fsid 9a3d7e76-8ce1-46e2-940f-083be11712d3
last_changed 2019-01-15 15:54:48.676260
created 2019-01-15 15:54:48.676260
0: 192.168.80.11:6789/0 mon.ceph1
1: 192.168.80.12:6789/0 mon.ceph2
2: 192.168.80.13:6789/0 mon.ceph3

whatever i can set 80.x/10.x/1.x, i can't execute
only disable it i can execute gwcli

[dillaman]

@fabianlv

You appear to have both "python34-rados.x86_64 2:13.2.2-0.el7 ceph" and "python-rados.x86_64 2:13.2.4-0.el7 @ceph-mimic-13.2.4", which are different versions.

As for the trusted_ip_list, without providing more information, I really cannot help. The IP list is just additional IP addresses to allow to access the API, so populating it shouldn't ever restrict access -- it should only expand it.