Bosh stemcell default / mount option not compatible with k8s (node_exporter / longhorn)

[poblin-orange]

Context:

• bionic stemcell 1.1
• rancher 2.5.8
• k3s bosh release 15

Observed symptoms:

   Warning  Failed     11m                   kubelet            Error: failed to generate container "3db175e3aa6347acef7b87d4a1e488361f4f076759252ba2d0512f │
│ f4565aa755" spec: failed to generate spec: path "/" is mounted on "/" but it is not a shared or slave mount                                                │
│   Warning  Failed     10m (x3 over 11m)     kubelet            (combined from similar events): Error: failed to generate container "6611df6844dd9a14ca1c1d │
│ f5b6fd2a86d5685fb3256ef5ba00d1ac871dc25462" spec: failed to generate spec: path "/" is mounted on "/" but it is not a shared or slave mount                │
│   Normal   Pulled     2m47s (x50 over 12m)  kubelet            Container image "rancher/mirrored-prometheus-node-exporter:v1.1.2" already present on machi │
│ ne                                                                                                                                                         │

server/23576461-5263-47b7-ba4b-3450a7bfdc95:/var/vcap/store/terraform-apply# cat /proc/self/mountinfo |grep '/ / '
75 1 252:1 / / rw,relatime - ext4 /dev/vda1 rw

server/23576461-5263-47b7-ba4b-3450a7bfdc95:/var/vcap/store/terraform-apply# lsblk 
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda    252:0    0   40G  0 disk 
├─vda1 252:1    0  2.9G  0 part /
└─vda2 252:2    0 37.1G  0 part /var/vcap/data
vdb    252:16   0  108G  0 disk 
└─vdb1 252:17   0  108G  0 part /var/vcap/store
s

References:

• HostPath mounts failing with "Path is not a shared or slave mount" kubernetes/kubernetes#61058

Workaround (os level)

https://github.com/kubernetes/kubernetes/issues/61058#issuecomment-372764783

server/23576461-5263-47b7-ba4b-3450a7bfdc95:/var/vcap/store/terraform-apply# mount --make-rshared /
server/23576461-5263-47b7-ba4b-3450a7bfdc95:/var/vcap/store/terraform-apply# cat /proc/self/mountinfo |grep '/ / '
75 1 252:1 / / rw,relatime shared:42 - ext4 /dev/vda1 rw

Workaround (prometheus / node exporter level)

• [kube-prometheus-stack] node-exporter crashes at startup prometheus-community/helm-charts#467
• [kube-prometheus-stack] node-exporter crashes at startup prometheus-community/helm-charts#467 (comment)

[poblin-orange]

Same issue with longhorn pod, trying to mout /var/lib/longhorn
see https://rancher-users.slack.com/archives/CC2UQM49Y/p1620377479339000

[poblin-orange]

fixed in version 16