Ceph : Error EINVAL: Failed to connect to ceph01 (ceph01)

[guidtz]

Hello,

do you have any problem to connect node to ceph master ? When I tried to add a node :

$ sudo ceph orch host add ceph01
Error EINVAL: Failed to connect to ceph01 (ceph01).
Check that the host is reachable and accepts connections using the cephadm SSH key

you may want to run:
> ceph cephadm get-ssh-config > ssh_config
> ceph config-key get mgr/cephadm/ssh_identity_key > key
> ssh -F ssh_config -i key root@ceph01

So I configured ssh access to node ceph01 with .ssh/config and It's works manually :

$ sudo ssh ceph01
Warning: Permanently added 'ceph01,192.168.85.31' (ECDSA) to the list of known hosts.
Linux ceph01 4.19.0-12-amd64 #1 SMP Debian 4.19.152-1 (2020-10-18) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sat Nov 21 18:48:29 2020 from 192.168.85.30
root@ceph01:~#

Do you have an idea ?

[rsoika]

Yes it could be, that your node is not accepting the public key from the root node.
Take a look at this tutorial. Here I explained the procedure in more detail in the section "Expending the Cluster".

Let me know if you can solve your issue so that we can improve the documentation.

[vitalyUstalu]

Hello,
I had the same problem with a manualy installed ceph cluster (via cephadm) and fixed it by installing python3 on target host.
I found it here https://github.com/ceph/ceph/blob/master/src/pybind/mgr/cephadm/module.py#L1008 Unfortunately this message is not displayed further.

[guidtz]

Ok I'll go to test this soon.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[thatsk]

this is worst case where it is not helping for beginneer even our dns is working fine we added in host file and still we get this error and not able to get started .

[thatsk]

single node cluster is not working also for quincy release hope fully better docs make better product

[rsoika]

@thatsk Which exact problem do you have? Do you have problems to install your ceph cluster? I have written a new tutorial how to install Ceph on Debian: https://ralph.blog.imixs.com/2021/10/03/ceph-pacific-running-on-debian-11-bullseye/

As I recommand you should run ceph independent from your Kubernetes cluster. So if it is a ceph problem you need to search first for help in the ceph community.

If your ceph cluster is up and running, but your imixs-cloud setup does not be able to create a static persistence volume connected to your Ceph than we can discuss it here.

[thatsk]

@rsoika I am not seeing any GitHub issues on the GitHub repo and don't know where we actually place raise the issue like for broader discussion.
https://github.com/ceph/ceph

[thatsk]

@rsoika i am dam sure that when we are trying to add the host it is trying to connect from container which is spinned up by cephadm and it not able to discover node from the container shell but its actually discovered from node and and when i see its attached to host network still i am facing issu.

[thatsk]

hoping this discussion will help lot of people like me to get this issue resolved

[thatsk]

Waiting for the mgr to restart...
Waiting for mgr epoch 5...
mgr epoch 5 is available
Setting orchestrator backend to cephadm...
Generating ssh key...
Wrote public SSH key to /etc/ceph/ceph.pub
Adding key to root@localhost authorized_keys...
Adding host csmgr01p1...
Non-zero exit code 22 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v17 -e NODE_NAME=csmgr01p1 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/84de6c14-00fa-11ed-91ec-0050568ec690:/var/log/ceph:z -v /tmp/ceph-tmpci7asyo_:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpaz74h5s9:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v17 orch host add csmgr01p1 10.174.206.41
/usr/bin/ceph: stderr Error EINVAL: Unable to reach remote host csmgr01p1. Process exited with non-zero exit status 1

[thatsk]

k i will try pacific on almalinux 8.6 and get back to you with result in next hour

[thatsk]

@rsoika you made my day all things are working on pacific version as per your blog

[thatsk]

seems like issues with Quincy will update if anything is required

[rsoika]

Great that it works now.
Yes, a new ceph release is always a kind of mystery. I will try Quincy when I next install a new cluster - and then I will write a new Blog about it ;-)

[thatsk]

Thanks @rsoika . Waiting for your post on Quincy releasse

[thatsk]

@rsoika whats this error mean and i am getting while adding osd.
and also i have seen in ceph-volume logs.

[2022-07-15 10:39:23,038][ceph_volume.util.system][INFO  ] /dev/sdb was not found as mounted
[2022-07-15 10:39:23,087][ceph_volume.util.system][INFO  ] /dev/sdc was not found as mounted

root@mgr01p1: ~# ceph orch device ls --wide
HOST                            PATH      TYPE  TRANSPORT  RPM  DEVICE ID   SIZE  HEALTH   IDENT  FAULT  AVAILABLE  REFRESHED  REJECT REASONS                                                 
mgr01p1  /dev/sdb  hdd   Unknown    -2               107G  Unknown  N/A    N/A               16m ago    Insufficient space (<10 extents) on vgs, LVM detected, locked  
mgr01p1  /dev/sdc  hdd   Unknown    -2               107G  Unknown  N/A    N/A               16m ago    Insufficient space (<10 extents) on vgs, LVM detected, locked  

[rsoika]

Do you mean with 'vms' virtual mashines? You are using something like VirtualBox or VMWare? I don't think that ceph will accept virtual disks.

[thatsk]

Yes i am using Venter and in that on esxi host i created 3 vms and each vm has 2 raw disk attached

[thatsk]

https://www.server-world.info/en/note?os=CentOS_8&p=ceph15&f=10 same issue like this

[rsoika]

I think it's better if you subscribe to mailing list [email protected]
Go here: https://lists.ceph.io/postorius/lists/ceph-users.ceph.io/

[rsoika]

I am really not the ceph expert on that level. But something with your disk is not as expeced. ceph provides a tool which allows your to erase (zap) a device so that it can be reused. zap calls ceph-volume zap on the remote host.

$ ceph orch device zap <hostname> <path>

Example command:

$ ceph orch device zap my_hostname /dev/sdx

Be careful with this command and make sure you set the correct host and device name.
Also note, that it can take 1 or 2 minutes until a ceph node signals that a device can be used. So check from time to time with

$ ceph orch device ls

before you continue! The column AVAILABLE must show 'YES'

[xuanyuanaosheng]

I using https://docs.ceph.com/en/quincy/cephadm/install/?#bootstrap-a-new-cluster:

Bug Report

Environment:

• System :

VM From Vcenter

• OS (e.g. from /etc/os-release):

# we using oracle linux, so i changed oracle to centos.
sed -i 's/ID="ol"/ID="centos"/g' /etc/os-release

# cat /etc/os-release
NAME="Oracle Linux Server"
VERSION="8.6"
ID="centos"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="8.6"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Oracle Linux Server 8.6"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:oracle:linux:8:6:server"
HOME_URL="https://linux.oracle.com/"
BUG_REPORT_URL="https://bugzilla.oracle.com/"

ORACLE_BUGZILLA_PRODUCT="Oracle Linux 8"
ORACLE_BUGZILLA_PRODUCT_VERSION=8.6
ORACLE_SUPPORT_PRODUCT="Oracle Linux"
ORACLE_SUPPORT_PRODUCT_VERSION=8.6

• Kernel (e.g. uname -a):

# uname -a
Linux whdcceph001.cn.prod 5.4.17-2102.203.6.el8uek.x86_64 #2 SMP Wed Jul 21 17:45:32 PDT 2021 x86_64 x86_64 x86_64 GNU/Linux

• Docker version if applicable (e.g. docker version):

# docker version
Client: Docker Engine - Community
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.17.11
 Git commit:        100c701
 Built:             Mon Jun  6 23:03:11 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.11
  Git commit:       a89b842
  Built:            Mon Jun  6 23:01:29 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.6
  GitCommit:        10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
 runc:
  Version:          1.1.2
  GitCommit:        v1.1.2-0-ga916309
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

• Ceph Repo and version (e.g. ceph -v):

# cephadm add-repo --release quincy

# ceph -v
ceph version 17.2.1 (ec95624474b1871a821a912b8c3af68f8f8e7aa1) quincy (stable)

• python version (e.g. python -V):

# python -V
Python 3.8.12

The init command

cephadm bootstrap --mon-ip 10.26.246.201 --allow-fqdn-hostname

The ceph status and install log error:

# docker ps
CONTAINER ID   IMAGE                   COMMAND                  CREATED          STATUS          PORTS     NAMES
8b86c3ecc9c0   quay.io/ceph/ceph:v17   "/usr/bin/ceph-mgr -…"   25 minutes ago   Up 25 minutes             ceph-98aab86e-089e-11ed-9fc8-005056b6e418-mgr-whdcceph001-cn-prod-actsgd
fbbd5602b688   quay.io/ceph/ceph:v17   "/usr/bin/ceph-mon -…"   25 minutes ago   Up 25 minutes             ceph-98aab86e-089e-11ed-9fc8-005056b6e418-mon-whdcceph001-cn-prod

# ceph -s
  cluster:
    id:     98aab86e-089e-11ed-9fc8-005056b6e418
    health: HEALTH_WARN
            OSD count 0 < osd_pool_default_size 3
 
  services:
    mon: 1 daemons, quorum whdcceph001.cn.prod (age 25m)
    mgr: whdcceph001.cn.prod.actsgd(active, since 25m)
    osd: 0 osds: 0 up, 0 in
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:     

# ceph orch device ls --wide

# ceph orch host add whdcceph001.cn.prod 10.26.246.201
Error EINVAL: Unable to reach remote host whdcceph001.cn.prod. Process exited with non-zero exit status 1

# The error logs is:

2022-07-19 20:46:32,840 7fcdf23acb80 INFO Waiting for mgr epoch 5...
2022-07-19 20:46:42,008 7fcdf23acb80 DEBUG /usr/bin/ceph: {
2022-07-19 20:46:42,008 7fcdf23acb80 DEBUG /usr/bin/ceph:     "mgrmap_epoch": 7,
2022-07-19 20:46:42,008 7fcdf23acb80 DEBUG /usr/bin/ceph:     "initialized": true
2022-07-19 20:46:42,008 7fcdf23acb80 DEBUG /usr/bin/ceph: }
2022-07-19 20:46:42,075 7fcdf23acb80 INFO mgr epoch 5 is available
2022-07-19 20:46:42,075 7fcdf23acb80 INFO Setting orchestrator backend to cephadm...
2022-07-19 20:46:43,277 7fcdf23acb80 DEBUG /usr/bin/ceph: value unchanged
2022-07-19 20:46:43,341 7fcdf23acb80 INFO Generating ssh key...
2022-07-19 20:46:44,523 7fcdf23acb80 DEBUG /usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCz1nvEshb9933GUpkObcn/M72TfCp4A/C5j14EkHx+jkCQxa6gJZQlaPz5bsHcPyHkS7vvW6dOOSfStPOCRQY27HLXETuv+b/+7lYDMC+xU95twLh5aGFWAg3bQI3z9LsTxJPgJU+2InHnW0ttSYZfw4BN0v2BUk4/eFN9+HCmmTrgprjU+5qtp8dz0UnoZN8Wyfv94o54pqoC2iqjo5b9KT2Ni5YHstHcLpeAvQ+9a0S90GO9PbUVbXeDKBadyvQRKELiurJseV4uTgJgABVXg1d2vX27IAPvbGMw/qmTKDD4PtYDoQtQp7uGo3kYnbRIBdAEzUCIg+hZDMYyG3CSVv4VGM2WLaQwAcshfCQv4vpBda/fqf6QkBUb4p52OjG33vZ5a2DK1RjztS1XepysmEqAa5ukI1/AuO/uaIUaNz3H6GPcZVVcqjNrG2FBE+UNsXgbaWNUkze+vyzArTJ+4clzql5yJeINdXh1FW1cwcbeW0Np75IpGRVtagD10xE= ceph-15d1ea8e-0754-11ed-b8cd-005056b63f21
2022-07-19 20:46:44,583 7fcdf23acb80 INFO Wrote public SSH key to /etc/ceph/ceph.pub
2022-07-19 20:46:44,583 7fcdf23acb80 INFO Adding key to root@localhost authorized_keys...
2022-07-19 20:46:44,585 7fcdf23acb80 INFO Adding host whdcceph002.cn.prod...
2022-07-19 20:46:45,369 7fcdf23acb80 DEBUG /usr/bin/ceph: Error EINVAL: Unable to reach remote host whdcceph002.cn.prod. Process exited with non-zero exit status 1
2022-07-19 20:46:45,420 7fcdf23acb80 INFO Non-zero exit code 22 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v17 -e NODE_NAME=whdcceph002.cn.prod -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/15d1ea8e-0754-11ed-b8cd-005056b63f21:/var/log/ceph:z -v /tmp/ceph-tmpvb29vcot:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp0rogw455:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v17 orch host add whdcceph002.cn.prod 10.26.246.202
2022-07-19 20:46:45,420 7fcdf23acb80 INFO /usr/bin/ceph: stderr Error EINVAL: Unable to reach remote host whdcceph002.cn.prod. Process exited with non-zero exit status 1
2022-07-19 20:46:45,420 7fcdf23acb80 ERROR ERROR: Failed to add host <whdcceph002.cn.prod>: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v17 -e NODE_NAME=whdcceph002.cn.prod -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/15d1ea8e-0754-11ed-b8cd-005056b63f21:/var/log/ceph:z -v /tmp/ceph-tmpvb29vcot:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp0rogw455:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v17 orch host add whdcceph002.cn.prod 10.26.246.202: Error EINVAL: Unable to reach remote host whdcceph002.cn.prod. Process exited with non-zero exit status 1

cephadm.log

[xuanyuanaosheng]

@xuanyuanaosheng I have test using cephadm add-repo --release octopus and it works fine.

[rsoika]

Yes, there seems to be some changes with the latest quincy release of ceph in the install procedure. I did not have found time to try it by myself. Did you know the ceph orch upgrade command? https://docs.ceph.com/en/quincy/cephadm/upgrade/

Maybe you can give it a try to upgrade later?