feat(lib): Use lsblk directly, parse json output #255

jhermsmeier · 2018-03-09T13:18:48Z

This switches from using a bash script on linux to shelling out
to lsblk directly, using its --json option to obtain & parse
machine-readable output – removing the need for external script
files entirely.

Change-Type: minor
Connects To: #253, #231, #165, balena-io/etcher#2122

jhermsmeier · 2018-03-23T21:27:23Z

Updated to parse the key / value pairs that can be obtained through the --pairs option, should the --json option result in an error, or not be supported.

jviotti

Left some comments, but overall looks good, and it works fine on macOS

jviotti · 2018-03-23T21:39:45Z

lib/diskutil.js

+      exec('diskutil', [ 'info', '-plist', devicePath ], (error, stdout) => {
+        let data = null;
+        try {
+          data = plist.parse(stdout);


What if exec yields back an error? We will still attempt to parse stdout?

Yes, as diskutil may output error descriptions in plist format to stdout

I.e. see https://github.com/resin-io-modules/drivelist/pull/255/files#diff-c045d0de9bd65b6a0dd99604dd7b0742R114

jviotti · 2018-03-23T21:43:24Z

lib/lsblk/json.js

+      raw: device.kname,
+      description: getDescription(device),
+      error: null,
+      size: Number(device.size),


What happens if these evaluate to NaN? Should we handle this somehow and make them undefined or something like that?

Defaulted to null now, although that shouldn't happen.

jviotti · 2018-03-23T21:43:47Z

lib/lsblk/json.js

+      logicalBlockSize: Number(device['log-sec']),
+      mountpoints: device.children ? getMountpoints(device.children) : [],
+      isReadOnly: isReadOnly,
+      isSystem: !isRemovable && !isVirtual,


Can a system drive be virtual?

Not entirely sure; there might be contraptions where a system device could be virtual (apfs containers, encrypted volumes), not sure if they'd be labeled as virtual though; need to test.

resin-io-modules-versionbot · 2018-06-18T15:04:07Z

@jhermsmeier, status checks have failed for this PR. Please make appropriate changes and recommit.

lurch · 2018-06-18T22:53:46Z

Just as a heads-up, this appears to be non-functional on Ubuntu 14.04 :-(

With the master branch of this repo, node example/index.js gives me:

[ { enumerator: 'lsblk',
    busType: 'UNKNOWN',
    busVersion: '0.0',
    device: '/dev/sda',
    devicePath: null,
    raw: '/dev/sda',
    description: 'SanDisk SD8SN8U1',
    error: null,
    size: 1024209543168,
    blockSize: null,
    logicalBlockSize: null,
    mountpoints: 
     [ { path: '/' },
       { path: '/boot/efi' },
       { path: '/home' },
       { path: '/var/lib/docker/aufs' } ],
    isReadOnly: false,
    isSystem: true,
    isVirtual: null,
    isRemovable: null,
    isCard: null,
    isSCSI: null,
    isUSB: null,
    isUAS: null } ]

but on the lsblk-json branch of this repo, node example/index.js gives me:

[]

If it helps, this is the output of lsblk --bytes --pairs --all:

NAME="sda" MAJ:MIN="8:0" RM="0" SIZE="1024209543168" RO="0" TYPE="disk" MOUNTPOINT=""
NAME="sda1" MAJ:MIN="8:1" RM="0" SIZE="524288000" RO="0" TYPE="part" MOUNTPOINT="/boot/efi"
NAME="sda2" MAJ:MIN="8:2" RM="0" SIZE="41943040" RO="0" TYPE="part" MOUNTPOINT=""
NAME="sda3" MAJ:MIN="8:3" RM="0" SIZE="3221225472" RO="0" TYPE="part" MOUNTPOINT=""
NAME="sda4" MAJ:MIN="8:4" RM="0" SIZE="235291017216" RO="0" TYPE="part" MOUNTPOINT="/"
NAME="sda5" MAJ:MIN="8:5" RM="0" SIZE="16980639744" RO="0" TYPE="part" MOUNTPOINT="[SWAP]"
NAME="sda6" MAJ:MIN="8:6" RM="0" SIZE="549755813888" RO="0" TYPE="part" MOUNTPOINT="/home"
NAME="sda7" MAJ:MIN="8:7" RM="0" SIZE="218392166400" RO="0" TYPE="part" MOUNTPOINT=""
NAME="ram0" MAJ:MIN="1:0" RM="0" SIZE="67108864" RO="0" TYPE="disk" MOUNTPOINT=""
NAME="ram1" MAJ:MIN="1:1" RM="0" SIZE="67108864" RO="0" TYPE="disk" MOUNTPOINT=""
NAME="ram2" MAJ:MIN="1:2" RM="0" SIZE="67108864" RO="0" TYPE="disk" MOUNTPOINT=""
NAME="ram3" MAJ:MIN="1:3" RM="0" SIZE="67108864" RO="0" TYPE="disk" MOUNTPOINT=""
NAME="ram4" MAJ:MIN="1:4" RM="0" SIZE="67108864" RO="0" TYPE="disk" MOUNTPOINT=""
NAME="ram5" MAJ:MIN="1:5" RM="0" SIZE="67108864" RO="0" TYPE="disk" MOUNTPOINT=""
NAME="ram6" MAJ:MIN="1:6" RM="0" SIZE="67108864" RO="0" TYPE="disk" MOUNTPOINT=""
NAME="ram7" MAJ:MIN="1:7" RM="0" SIZE="67108864" RO="0" TYPE="disk" MOUNTPOINT=""
NAME="ram8" MAJ:MIN="1:8" RM="0" SIZE="67108864" RO="0" TYPE="disk" MOUNTPOINT=""
NAME="ram9" MAJ:MIN="1:9" RM="0" SIZE="67108864" RO="0" TYPE="disk" MOUNTPOINT=""
NAME="loop0" MAJ:MIN="7:0" RM="0" SIZE="" RO="0" TYPE="loop" MOUNTPOINT=""
NAME="loop1" MAJ:MIN="7:1" RM="0" SIZE="" RO="0" TYPE="loop" MOUNTPOINT=""
NAME="loop2" MAJ:MIN="7:2" RM="0" SIZE="" RO="0" TYPE="loop" MOUNTPOINT=""
NAME="loop3" MAJ:MIN="7:3" RM="0" SIZE="" RO="0" TYPE="loop" MOUNTPOINT=""
NAME="loop4" MAJ:MIN="7:4" RM="0" SIZE="" RO="0" TYPE="loop" MOUNTPOINT=""
NAME="loop5" MAJ:MIN="7:5" RM="0" SIZE="" RO="0" TYPE="loop" MOUNTPOINT=""
NAME="loop6" MAJ:MIN="7:6" RM="0" SIZE="" RO="0" TYPE="loop" MOUNTPOINT=""
NAME="loop7" MAJ:MIN="7:7" RM="0" SIZE="" RO="0" TYPE="loop" MOUNTPOINT=""
NAME="ram10" MAJ:MIN="1:10" RM="0" SIZE="67108864" RO="0" TYPE="disk" MOUNTPOINT=""
NAME="ram11" MAJ:MIN="1:11" RM="0" SIZE="67108864" RO="0" TYPE="disk" MOUNTPOINT=""
NAME="ram12" MAJ:MIN="1:12" RM="0" SIZE="67108864" RO="0" TYPE="disk" MOUNTPOINT=""
NAME="ram13" MAJ:MIN="1:13" RM="0" SIZE="67108864" RO="0" TYPE="disk" MOUNTPOINT=""
NAME="ram14" MAJ:MIN="1:14" RM="0" SIZE="67108864" RO="0" TYPE="disk" MOUNTPOINT=""
NAME="ram15" MAJ:MIN="1:15" RM="0" SIZE="67108864" RO="0" TYPE="disk" MOUNTPOINT=""

and this is the output of lsblk --help:

Usage:
 lsblk [options] [<device> ...]

Options:
 -a, --all            print all devices
 -b, --bytes          print SIZE in bytes rather than in human readable format
 -d, --nodeps         don't print slaves or holders
 -D, --discard        print discard capabilities
 -e, --exclude <list> exclude devices by major number (default: RAM disks)
 -f, --fs             output info about filesystems
 -h, --help           usage information (this)
 -i, --ascii          use ascii characters only
 -m, --perms          output info about permissions
 -l, --list           use list format ouput
 -n, --noheadings     don't print headings
 -o, --output <list>  output columns
 -P, --pairs          use key="value" output format
 -r, --raw            use raw output format
 -t, --topology       output info about topology

Available columns:
       NAME  device name
      KNAME  internal kernel device name
    MAJ:MIN  major:minor device number
     FSTYPE  filesystem type
 MOUNTPOINT  where the device is mounted
      LABEL  filesystem LABEL
       UUID  filesystem UUID
         RO  read-only device
         RM  removable device
      MODEL  device identifier
       SIZE  size of the device
      STATE  state of the device
      OWNER  user name
      GROUP  group name
       MODE  device node permissions
  ALIGNMENT  alignment offset
     MIN-IO  minimum I/O size
     OPT-IO  optimal I/O size
    PHY-SEC  physical sector size
    LOG-SEC  logical sector size
       ROTA  rotational device
      SCHED  I/O scheduler name
    RQ-SIZE  request queue size
       TYPE  device type
   DISC-ALN  discard alignment offset
  DISC-GRAN  discard granularity
   DISC-MAX  discard max bytes
  DISC-ZERO  discard zeroes data

For more information see lsblk(8).

jhermsmeier · 2018-06-19T18:40:12Z

@lurch what does lsblk --version output for you?

lurch · 2018-06-20T10:12:30Z

what does lsblk --version output for you?

lsblk: unrecognised option '--version'

😉

However doing man lsblk I see it says "The lsblk command is part of the util-linux package" and doing dpkg -l util-linux it tells me:

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-=================================
ii  util-linux     2.20.1-5.1ub amd64        Miscellaneous system utilities

See also https://packages.ubuntu.com/trusty-updates/util-linux

This switches from using a bash script on linux to shelling out to `lsblk` directly, using its `--json` option to obtain & parse machine-readable output – removing the need for external script files entirely. Change-Type: minor

jhermsmeier · 2018-07-06T20:35:31Z

Thanks @lurch – that's fixed now; was a bug in how disks & their child partitions were mapped to each other.

resin-io-modules-versionbot · 2018-07-06T20:42:24Z

@jhermsmeier, status checks have failed for this PR. Please make appropriate changes and recommit.

resin-io-modules-versionbot · 2018-07-06T21:27:32Z

@jhermsmeier, status checks have failed for this PR. Please make appropriate changes and recommit.

This fixes consolidation & mapping of children on systems with an old version of `lsblk` (like Ubuntu 14.04), where only the `--pairs` option is available and no `pkname` or `kname` fields are exposed in the output. Change-Type: patch

jhermsmeier requested review from Shou and jviotti March 9, 2018 13:18

jhermsmeier force-pushed the lsblk-json branch 4 times, most recently from 31e0bc4 to 9602025 Compare March 9, 2018 14:04

jhermsmeier mentioned this pull request Mar 9, 2018

feat(lib): Use lsblk directly, parse json output #254

Closed

lurch mentioned this pull request Mar 14, 2018

No such file or directory: undefined with version etcher-1.3.1-x86_64 balena-io/etcher#2122

Closed

jhermsmeier force-pushed the lsblk-json branch from 9602025 to e6c0cb3 Compare March 15, 2018 15:38

jhermsmeier requested a review from zvin March 15, 2018 16:16

jhermsmeier force-pushed the lsblk-json branch 2 times, most recently from 8a9bec8 to 6425068 Compare March 20, 2018 13:57

jviotti approved these changes Mar 23, 2018

View reviewed changes

jhermsmeier mentioned this pull request Apr 6, 2018

YAMLException when drive contains a new line character in its description balena-io/etcher#1325

Closed

jhermsmeier force-pushed the lsblk-json branch 2 times, most recently from ad20e15 to f1261b0 Compare May 9, 2018 09:52

jhermsmeier force-pushed the lsblk-json branch 2 times, most recently from 3087557 to b756bab Compare June 18, 2018 14:59

zvin approved these changes Jun 18, 2018

View reviewed changes

lurch mentioned this pull request Jun 18, 2018

fix(linux): Add flag to lsblk to list all devices #287

Merged

jhermsmeier force-pushed the lsblk-json branch 2 times, most recently from 4bb4d3f to 7397ee1 Compare July 5, 2018 23:38

feat(lib): Use lsblk directly, parse json output

f55c25f

This switches from using a bash script on linux to shelling out to `lsblk` directly, using its `--json` option to obtain & parse machine-readable output – removing the need for external script files entirely. Change-Type: minor

jhermsmeier force-pushed the lsblk-json branch from 3ff0d27 to 3e238f3 Compare July 6, 2018 20:32

jhermsmeier removed the request for review from Shou July 6, 2018 20:33

jhermsmeier requested review from zvin, jviotti and curcuz July 6, 2018 20:33

jhermsmeier force-pushed the lsblk-json branch 3 times, most recently from 6aa13a8 to cb93b23 Compare July 6, 2018 21:17

jhermsmeier force-pushed the lsblk-json branch from cb93b23 to bcd4118 Compare July 9, 2018 14:16

fix(linux): Fix pair consolidation for lsblk

fdee0f1

This fixes consolidation & mapping of children on systems with an old version of `lsblk` (like Ubuntu 14.04), where only the `--pairs` option is available and no `pkname` or `kname` fields are exposed in the output. Change-Type: patch

jhermsmeier force-pushed the lsblk-json branch from bcd4118 to fdee0f1 Compare July 9, 2018 14:19

zvin approved these changes Jul 9, 2018

View reviewed changes

jhermsmeier added the procbots/versionbot/ready-to-merge label Jul 9, 2018

v6.3.0

c653464

resin-io-modules-versionbot bot merged commit 24e6485 into master Jul 9, 2018

resin-io-modules-versionbot bot removed the procbots/versionbot/ready-to-merge label Jul 9, 2018

resin-io-modules-versionbot bot deleted the lsblk-json branch July 9, 2018 14:43

igo95862 mentioned this pull request May 29, 2021

Improve GetESP() Foxboron/sbctl#44

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(lib): Use lsblk directly, parse json output #255

feat(lib): Use lsblk directly, parse json output #255

jhermsmeier commented Mar 9, 2018 •

edited

Loading

jhermsmeier commented Mar 23, 2018

jviotti left a comment

jviotti Mar 23, 2018

jhermsmeier Jul 9, 2018

jhermsmeier Jul 9, 2018

jviotti Mar 23, 2018

jhermsmeier Jun 18, 2018

jviotti Mar 23, 2018

jhermsmeier Jun 18, 2018

resin-io-modules-versionbot bot commented Jun 18, 2018

lurch commented Jun 18, 2018

jhermsmeier commented Jun 19, 2018

lurch commented Jun 20, 2018

jhermsmeier commented Jul 6, 2018

resin-io-modules-versionbot bot commented Jul 6, 2018

resin-io-modules-versionbot bot commented Jul 6, 2018

feat(lib): Use lsblk directly, parse json output #255

feat(lib): Use lsblk directly, parse json output #255

Conversation

jhermsmeier commented Mar 9, 2018 • edited Loading

jhermsmeier commented Mar 23, 2018

jviotti left a comment

Choose a reason for hiding this comment

jviotti Mar 23, 2018

Choose a reason for hiding this comment

jhermsmeier Jul 9, 2018

Choose a reason for hiding this comment

jhermsmeier Jul 9, 2018

Choose a reason for hiding this comment

jviotti Mar 23, 2018

Choose a reason for hiding this comment

jhermsmeier Jun 18, 2018

Choose a reason for hiding this comment

jviotti Mar 23, 2018

Choose a reason for hiding this comment

jhermsmeier Jun 18, 2018

Choose a reason for hiding this comment

resin-io-modules-versionbot bot commented Jun 18, 2018

lurch commented Jun 18, 2018

jhermsmeier commented Jun 19, 2018

lurch commented Jun 20, 2018

jhermsmeier commented Jul 6, 2018

resin-io-modules-versionbot bot commented Jul 6, 2018

resin-io-modules-versionbot bot commented Jul 6, 2018

jhermsmeier commented Mar 9, 2018 •

edited

Loading