[secrets "ceph-bootstrap-osd-keyring" not found] ceph-osd fail on k8s 1.10.1 install by kubeadm #55
Comments
foxundermoon
changed the title
|
|
system centos
network:
public: 192.168.0.0/16
cluster: 192.168.0.0/16
port:
mon: 6789
rgw: 8088
enable_zap_and_potentially_lose_data: false
osd_devices:
- name: dev-sda
device: /dev/mapper/centos-root
# journal: /dev/sdf
zap: "1"
ceph_mgr_enabled_modules:
- restful
- status
- dashboard
# You can configure your mgr modules
# below. Each module has its own set
# of key/value. Refer to the doc
# above for more info. For example:
ceph_mgr_modules_config:
dashboard:
port: 7000
localpool:
failure_domain: host
subtree: rack
pg_num: "128"
num_rep: "3"
min_size: "2"
storageclass:
provision_storage_class: true
provisioner: ceph.com/rbd
name: ceph-rbd
monitors: null
pool: rbd
admin_id: admin
admin_secret_name: pvc-ceph-conf-combined-storageclass
admin_secret_namespace: ceph
user_id: admin
user_secret_name: pvc-ceph-client-key
image_format: "2"
image_features: layering |
|
I met that error before when I install it after uninstall a "failed |
|
BTW: It seems that you config the OSD device to use the OS device ??? I think you should add one or more bland disk for OSD device, NOT the OS root device |
|
@whmzsu thank you. |
|
@foxundermoon I try to install the ceph again after uninstall the success installation, delete the ceph-helm dir , and got the same error like your. So I think there must be something not cleared by the helm while uninstalling. |
|
We also found this issue on Kubernetes 1.9.2. As we try all is clear all helm environment/reinital k8s farm and try more than 3 times still issue this |
|
@whmzsu still issue. i am reinstall on other 3 servers. |
|
I am getting same problem.Problem is "/etc/ceph/ceph.mon.keyring" I do not have such a path "/etc/ceph" or "var/lib/ceph". How can I all do this with a VM ? I do not want to run a host with minikube. Installation (Kubernetes + Helm)
|
|
@githubfoam I have exactly same issue |
|
@githubfoam yes, 27.0 |
|
not merged yet I guess. |
|
Encountered with the same issue in discussion When i tried for the first time everything worked fine. As soon as i deleted the cluster using Also i do not have directories like Need help to get find @rootfs any suggestions ? |
|
@githubfoam i am running the fix you mentioned in your last comment. |
|
@foxundermoon @githubfoam did you ever managed to get this working. Whats the clean way to purge the cluster ? |
|
@ksingh7 did you find any workaround? |
|
@jasine unfortunately not |
|
You must remove the partitions on target devices before installing a new ceph cluster. |
|
Hi All, I have exactly the same issue with Kubernetes 1.11.2. Is there still no fix or workaround for this ? |
|
Just use |
|
The OSD Pods still not coming up, a "kubectl describe pod xxxxxx" shows: But all of these Secrets are existing in the same namespace. |
|
same issue for me in K8S v1.10.5 / RHEL7.5 / KERNEL 3.10.0. any ideas? |
Same issue occured on K8S v1.11.2 / Ubuntu 18.04. However after I deleted pods, they mounted secrets successfully. After that I have stumbled upon other error related to reinstall. At fault was storage disks with old information from previous install. @mamoit incorrectly suggested to use fdisk, where you should use gdisk to wipe/recreate partitions! Cephs expects GPT partition table and fdisk does not properly handle that. To verify, check logs from init container: I was getting the following error: After I have recreated GPT partition table (option 2) with |
|
cool @elvinasp ! |
|
I had the exact same issue. The last piece in the cleanup puzzle was to delete dangling docker volumes on all nodes:
|
Purge CephThe following will remove all remains of Ceph (after that I could install). This assumes that the helm del --purge ceph
kubectl delete namespaces ceph
while read host; do
echo $host; ssh -n $host -l root -- rm -rf /var/lib/ceph-helm /etc/ceph
done < <(kubectl get nodes -o name|sed 's!^node/!!')I also rebooted all nodes (not sure if this is a must). Verify that kubernetes is gone: kubectl get all --all-namespaces -o name|grep cephEDIT: Still fails :-( Deleting the OSD pods (works!)After deleting all OSD pods the cluster came up: kubectl delete pod -n ceph ceph-osd-dev-vdb-m28wr
... |
|
the same problem. |
|
I have same problem. I've tried purge twice, but it's the same problem |
the same, @xinghui-hou, did you find the solution? |
|
Hello, kubernetes 1.15.5 and the same things here... |
|
Last commit was on 26:th June 2019. |
I did not try but only way possible is to git clone/code/compile the solution and run it locally I guess.The solution seems merged.You can see the details of root cause analysis in the link I refer to. Manually typing "auth" in the "ceph.conf" could be last resort before pulling the source. |
Version of Helm and Kubernetes:
Which chart:
The text was updated successfully, but these errors were encountered: