Skip to content

DirectPV Hardware issue and unable to discover data

Cesar Celis Hernandez edited this page Mar 7, 2024 · 1 revision

Links:

When you see **MountVolume.MountDevice failed for volume

  • umount this drive - umount /dev/sdc
  • you have to repeat few times to umount the lazy mounts
  • make sure the drive is completely umounted - lsblk
  • and restart directpv pod running on this node <------- restart pods
  • post restart, the drive should have been mounted <---- Solution to mount, drive is mounted on pod restarts.

NOTE:

  • if the drive is not mounted, then it should most likely be a bad disk and you will see an error in kubectl directpv list drives --all

Additional:

$ kubectl directpv list drives --all
┌──────┬─────────┬───────────────────────────┬─────────┬─────────┬─────────┬────────┐
│ NODE │ NAME    │ MAKE                      │ SIZE    │ FREE    │ VOLUMES │ STATUS │
├──────┼─────────┼───────────────────────────┼─────────┼─────────┼─────────┼────────┤
│ 25   │ sdb     │ SEAGATE XS1600LE70084     │ 1.5 TiB │ 1.5 TiB │ 1       │ Ready  │
│ 25   │ sdc     │ SEAGATE XS1600LE70084     │ 1.5 TiB │ 1.5 TiB │ -       │ Ready  │
│ 25   │ sdd     │ SEAGATE XS1600LE70084     │ 1.5 TiB │ 1.5 TiB │ 1       │ Ready  │
│ 25   │ sdf     │ SEAGATE XS1600LE70084     │ 1.5 TiB │ 1.5 TiB │ -       │ Ready  │
│ 25   │ nvme0n1 │ Micron_2200_MTFDHBA512TCK │ 477 GiB │ 477 GiB │ -       │ Ready  │
│ 25   │ nvme1n1 │ INTEL SSDPE2KX010T8       │ 932 GiB │ 931 GiB │ -       │ Ready  │
│ 25   │ nvme2n1 │ INTEL SSDPE2KX010T8       │ 932 GiB │ 931 GiB │ -       │ Ready  │
│ 26   │ sdb     │ SEAGATE XS1600LE70084     │ 1.5 TiB │ 1.5 TiB │ 1       │ Ready  │
│ 26   │ sdc     │ SEAGATE XS1600LE70084     │ 1.5 TiB │ 1.5 TiB │ -       │ Ready  │
│ 26   │ sdd     │ SEAGATE XS1600LE70084     │ 1.5 TiB │ 1.5 TiB │ 1       │ Ready  │
│ 26   │ sde     │ SEAGATE XS1600LE70084     │ 1.5 TiB │ 1.5 TiB │ -       │ Ready  │
│ 26   │ sdf     │ SEAGATE XS1600LE70084     │ 1.5 TiB │ 1.5 TiB │ -       │ Ready  │
│ 26   │ nvme1n1 │ INTEL SSDPE2KX010T8       │ 932 GiB │ 931 GiB │ -       │ Ready  │
│ 26   │ nvme2n1 │ INTEL SSDPE2KX010T8       │ 932 GiB │ 931 GiB │ -       │ Ready  │
│ 27   │ sdb     │ SEAGATE XS1600LE70084     │ 1.5 TiB │ 1.5 TiB │ -       │ Ready  │
│ 27   │ sdc     │ SEAGATE XS1600LE70084     │ 1.5 TiB │ 1.5 TiB │ 1       │ Ready  │
│ 27   │ sdd     │ SEAGATE XS1600LE70084     │ 1.5 TiB │ 1.5 TiB │ -       │ Ready  │
│ 27   │ sde     │ SEAGATE XS1600LE70084     │ 1.5 TiB │ 1.5 TiB │ -       │ Ready  │
│ 27   │ sdf     │ SEAGATE XS1600LE70084     │ 1.5 TiB │ 1.5 TiB │ 1       │ Ready  │
│ 27   │ nvme0n1 │ Micron_2200_MTFDHBA512TCK │ 477 GiB │ 477 GiB │ -       │ Ready  │
│ 27   │ nvme1n1 │ INTEL SSDPE2KX010T8       │ 932 GiB │ 931 GiB │ -       │ Ready  │
│ 27   │ nvme2n1 │ INTEL SSDPE2KX010T8       │ 932 GiB │ 931 GiB │ -       │ Ready  │
└──────┴─────────┴───────────────────────────┴─────────┴─────────┴─────────┴────────┘
$ kubectl directpv discover --all

 Discovered node '25' ✔
 Discovered node '26' ✔
 Discovered node '27' ✔

┌─────────────────────┬──────┬─────────┬─────────┬─────────────┬──────────────────────────────────────────────┬───────────┬─────────────────────────────────┐
│ ID                  │ NODE │ DRIVE   │ SIZE    │ FILESYSTEM  │ MAKE                                         │ AVAILABLE │ DESCRIPTION                     │
├─────────────────────┼──────┼─────────┼─────────┼─────────────┼──────────────────────────────────────────────┼───────────┼─────────────────────────────────┤
│ 253:0$K+LEXs8Pjr... │ 25   │ dm-0    │ 672 MiB │ -           │ vg1-thin--pool--1_tmeta                      │ NO        │ Held by other device            │
│ 253:1$XTA7h0zXBh... │ 25   │ dm-1    │ 1.3 TiB │ -           │ vg1-thin--pool--1_tdata                      │ NO        │ Held by other device            │
│ 253:2$4+u+L8dxWn... │ 25   │ dm-2    │ 1.3 TiB │ -           │ vg1-thin--pool--1-tpool                      │ NO        │ Held by other device            │
│ 253:3$kiwjkok/Wz... │ 25   │ dm-3    │ 1.3 TiB │ -           │ vg1-thin--pool--1                            │ NO        │ Read only                       │
│ 253:4$/I8u1daV8L... │ 25   │ dm-4    │ 30 GiB  │ -           │ vg1-35515cc0--9a18--4139--9623--ae44706c3b84 │ YES       │ -                               │
│ 253:5$oV9pKjQ0ML... │ 25   │ dm-5    │ 30 GiB  │ -           │ vg1-86e8b558--d625--476a--8086--f6b25118c66a │ YES       │ -                               │
│ 253:6$DYFWb8ZGT1... │ 25   │ dm-6    │ 30 GiB  │ -           │ vg1-8436f57f--65b6--4b83--8053--a1e0fc2e1367 │ YES       │ -                               │
│ 253:7$mWlarxxr0G... │ 25   │ dm-7    │ 30 GiB  │ -           │ vg1-3737e692--2bf8--4d5d--851c--2399f3c113c9 │ YES       │ -                               │
│ 253:8$zWxYuwBwn6... │ 25   │ dm-8    │ 30 GiB  │ -           │ vg1-698b2df0--6c1f--47d9--8210--77263ac55420 │ YES       │ -                               │
│ 253:9$M5/8sgyCsi... │ 25   │ dm-9    │ 30 GiB  │ -           │ vg1-9b0d258f--07e9--4277--b89d--d292b96a72d3 │ YES       │ -                               │
│ 259:2$1cYA9UdyQ8... │ 25   │ nvme0n1 │ 477 GiB │ xfs         │ Micron_2200_MTFDHBA512TCK                    │ NO        │ Mounted; Used by DirectPV       │
│ 259:0$iadvN6JMJY... │ 25   │ nvme1n1 │ 932 GiB │ xfs         │ INTEL SSDPE2KX010T8                          │ NO        │ Mounted; Used by DirectPV       │
│ 259:1$pQ8GI8wGVm... │ 25   │ nvme2n1 │ 932 GiB │ xfs         │ INTEL SSDPE2KX010T8                          │ NO        │ Mounted; Used by DirectPV       │
│ 8:0$100LgcjhLYzi... │ 25   │ sda     │ 1.5 TiB │ -           │ SEAGATE XS1600LE70084                        │ NO        │ Partitioned                     │
│ 8:1$kXaee14lqVPX... │ 25   │ sda1    │ 1.0 MiB │ -           │ SEAGATE XS1600LE70084 (Part 1)               │ NO        │ Too small                       │
│ 8:2$ZClcBtc9Fltl... │ 25   │ sda2    │ 127 MiB │ vfat        │ SEAGATE XS1600LE70084 (Part 2)               │ NO        │ Too small                       │
│ 8:3$gZ4PlDWPCXsi... │ 25   │ sda3    │ 384 MiB │ ext4        │ SEAGATE XS1600LE70084 (Part 3)               │ NO        │ Too small; Mounted              │
│ 8:4$9dFn36EXsBJZ... │ 25   │ sda4    │ 1.5 TiB │ xfs         │ SEAGATE XS1600LE70084 (Part 4)               │ NO        │ Mounted                         │
│ 8:16$0coF0p96NdO... │ 25   │ sdb     │ 1.5 TiB │ xfs         │ SEAGATE XS1600LE70084                        │ NO        │ Mounted; Used by DirectPV       │
│ 8:32$2AmeEJ5TfbE... │ 25   │ sdc     │ 1.5 TiB │ xfs         │ SEAGATE XS1600LE70084                        │ NO        │ Mounted; Used by DirectPV       │
│ 8:48$vHoVo2O8SIM... │ 25   │ sdd     │ 1.5 TiB │ xfs         │ SEAGATE XS1600LE70084                        │ NO        │ Mounted; Used by DirectPV       │
│ 8:64$tXbZoGp9E/S... │ 25   │ sde     │ 1.5 TiB │ LVM2_member │ SEAGATE XS1600LE70084                        │ NO        │ Held by other device            │
│ 8:80$iHWzf58d6BX... │ 25   │ sdf     │ 1.5 TiB │ xfs         │ SEAGATE XS1600LE70084                        │ NO        │ Mounted; Used by DirectPV       │
│ 253:0$z2IPsDgqkf... │ 26   │ dm-0    │ 216 MiB │ -           │ vg1-thin--pool--1_tmeta                      │ NO        │ Too small; Held by other device │
│ 253:1$Z7Ij6/pNmw... │ 26   │ dm-1    │ 429 GiB │ -           │ vg1-thin--pool--1_tdata                      │ NO        │ Held by other device            │
│ 253:2$IazGoqSX9s... │ 26   │ dm-2    │ 429 GiB │ -           │ vg1-thin--pool--1-tpool                      │ NO        │ Held by other device            │
│ 253:3$0AHiKO4OBy... │ 26   │ dm-3    │ 429 GiB │ -           │ vg1-thin--pool--1                            │ NO        │ Read only                       │
│ 253:4$dnQPu0BcWz... │ 26   │ dm-4    │ 30 GiB  │ -           │ vg1-bc91fcac--ff01--4e8a--a21d--7a6518a41683 │ YES       │ -                               │
│ 253:5$aAOHd5WXNN... │ 26   │ dm-5    │ 30 GiB  │ -           │ vg1-7069d247--9f2e--4084--9286--e7f99f3541c4 │ YES       │ -                               │
│ 253:6$Wrh2uiCAkr... │ 26   │ dm-6    │ 30 GiB  │ -           │ vg1-17c94729--95e7--477b--a8d1--b1ae4d22c32e │ YES       │ -                               │
│ 253:7$p0fzgWlCFi... │ 26   │ dm-7    │ 30 GiB  │ -           │ vg1-2ca4d503--b7ba--42c2--8598--831af2695b45 │ YES       │ -                               │
│ 253:8$72iyivDtHD... │ 26   │ dm-8    │ 30 GiB  │ -           │ vg1-e13c99e8--29ab--4169--94d1--333a2954aee1 │ YES       │ -                               │
│ 253:9$AVV4b0SrSs... │ 26   │ dm-9    │ 30 GiB  │ -           │ vg1-e3f3efe3--496d--47e4--90ae--8985af17057d │ YES       │ -                               │
│ 259:2$fbO4lYtUWw... │ 26   │ nvme0n1 │ 477 GiB │ LVM2_member │ Micron_2200_MTFDHBA512TCK                    │ NO        │ Held by other device            │
│ 259:1$os69Mc8uJ6... │ 26   │ nvme1n1 │ 932 GiB │ xfs         │ INTEL SSDPE2KX010T8                          │ NO        │ Mounted; Used by DirectPV       │
│ 259:0$GLMtkQjRwq... │ 26   │ nvme2n1 │ 932 GiB │ xfs         │ INTEL SSDPE2KX010T8                          │ NO        │ Mounted; Used by DirectPV       │
│ 8:0$sABSznucBOi1... │ 26   │ sda     │ 1.5 TiB │ -           │ SEAGATE XS1600LE70084                        │ NO        │ Partitioned                     │
│ 8:1$VM3APB52wevh... │ 26   │ sda1    │ 1.0 MiB │ -           │ SEAGATE XS1600LE70084 (Part 1)               │ NO        │ Too small                       │
│ 8:2$HRM6Wju67Edn... │ 26   │ sda2    │ 127 MiB │ vfat        │ SEAGATE XS1600LE70084 (Part 2)               │ NO        │ Too small                       │
│ 8:3$aYcmM44zO2WF... │ 26   │ sda3    │ 384 MiB │ ext4        │ SEAGATE XS1600LE70084 (Part 3)               │ NO        │ Too small; Mounted              │
│ 8:4$3yasG7k9iOHv... │ 26   │ sda4    │ 1.5 TiB │ xfs         │ SEAGATE XS1600LE70084 (Part 4)               │ NO        │ Mounted                         │
│ 8:16$di1IdjokXoD... │ 26   │ sdb     │ 1.5 TiB │ xfs         │ SEAGATE XS1600LE70084                        │ NO        │ Mounted; Used by DirectPV       │
│ 8:32$IW/tgu3JPnO... │ 26   │ sdc     │ 1.5 TiB │ xfs         │ SEAGATE XS1600LE70084                        │ NO        │ Mounted; Used by DirectPV       │
│ 8:48$RseCKQPQz/C... │ 26   │ sdd     │ 1.5 TiB │ xfs         │ SEAGATE XS1600LE70084                        │ NO        │ Mounted; Used by DirectPV       │
│ 8:64$gtaE4yAHwMp... │ 26   │ sde     │ 1.5 TiB │ xfs         │ SEAGATE XS1600LE70084                        │ NO        │ Mounted; Used by DirectPV       │
│ 8:80$giKrqNu/VZI... │ 26   │ sdf     │ 1.5 TiB │ xfs         │ SEAGATE XS1600LE70084                        │ NO        │ Mounted; Used by DirectPV       │
│ 259:2$whnuln8r7m... │ 27   │ nvme0n1 │ 477 GiB │ xfs         │ Micron_2200_MTFDHBA512TCK                    │ NO        │ Mounted; Used by DirectPV       │
│ 259:1$TWfAaGh4QH... │ 27   │ nvme1n1 │ 932 GiB │ xfs         │ INTEL SSDPE2KX010T8                          │ NO        │ Mounted; Used by DirectPV       │
│ 259:0$N3W929YCqh... │ 27   │ nvme2n1 │ 932 GiB │ xfs         │ INTEL SSDPE2KX010T8                          │ NO        │ Mounted; Used by DirectPV       │
│ 8:0$llNuIE/q65lp... │ 27   │ sda     │ 1.5 TiB │ -           │ SEAGATE XS1600LE70084                        │ NO        │ Partitioned                     │
│ 8:1$YxtCSqnEjAx8... │ 27   │ sda1    │ 1.0 MiB │ -           │ SEAGATE XS1600LE70084 (Part 1)               │ NO        │ Too small                       │
│ 8:2$VHe4QSv8tvSl... │ 27   │ sda2    │ 127 MiB │ vfat        │ SEAGATE XS1600LE70084 (Part 2)               │ NO        │ Too small                       │
│ 8:3$U1hn3TY50fZA... │ 27   │ sda3    │ 384 MiB │ ext4        │ SEAGATE XS1600LE70084 (Part 3)               │ NO        │ Too small; Mounted              │
│ 8:4$uIB2mkvlHGSl... │ 27   │ sda4    │ 1.5 TiB │ xfs         │ SEAGATE XS1600LE70084 (Part 4)               │ NO        │ Mounted                         │
│ 8:16$gWEYH2K2925... │ 27   │ sdb     │ 1.5 TiB │ xfs         │ SEAGATE XS1600LE70084                        │ NO        │ Mounted; Used by DirectPV       │
│ 8:32$6aEEgSbYIAr... │ 27   │ sdc     │ 1.5 TiB │ xfs         │ SEAGATE XS1600LE70084                        │ NO        │ Mounted; Used by DirectPV       │
│ 8:48$Xuxk57nirsX... │ 27   │ sdd     │ 1.5 TiB │ xfs         │ SEAGATE XS1600LE70084                        │ NO        │ Mounted; Used by DirectPV       │
│ 8:64$4lZq4gZrqFc... │ 27   │ sde     │ 1.5 TiB │ xfs         │ SEAGATE XS1600LE70084                        │ NO        │ Mounted; Used by DirectPV       │
│ 8:80$pIDiaHyrdyM... │ 27   │ sdf     │ 1.5 TiB │ xfs         │ SEAGATE XS1600LE70084                        │ NO        │ Mounted; Used by DirectPV       │
└─────────────────────┴──────┴─────────┴─────────┴─────────────┴──────────────────────────────────────────────┴───────────┴─────────────────────────────────┘

Generated 'drives.yaml' successfully.
Clone this wiki locally