Skip to content

Commit

Permalink
growpart: Add support for overprovisioning (#35)
Browse files Browse the repository at this point in the history
Add option to 'growpart' to specify percentage of device that should be
left unallocated when growing partition. This is intended for consumer
SSDs and SD cards where the performance and/or lifetime of these devices
can be improved if some disk space (in addition to any the device "hides"
from users) is left unallocated.

Overprovisioning code caters for several distinct scenarios:

(1) MSDOS/MBR partitioned disk where the disk is >2TB and so MBR
    partitions cannot extend beyond 2TB - if disk is larger than
    (2TB + overprovisioning requirement) then nothing needs to be
    done.

(2) MSDOS/MBR partitioned disk where the disk is >2TB and so MBR
    partitions cannot extend beyond 2TB - if disk is not larger
    than (2TB + overprovisioning requirement) then *some*
    overprovisioning space still needs to be reserved.

(3) MSDOS/MBR partitioned disk <=2TB where overprovisioning space
    needs to be reserved.

(4) GPT partitioned disk where overprovisioning space needs to be
    reserved.

Also added a testcase script, test-growpart-overprovision.

Also correct some off-by-one errors in the existing growpart code and
correct some existing testcases affected by this.
  • Loading branch information
dermotbradley authored Apr 27, 2022
1 parent 5c24f6a commit c00ff0c
Show file tree
Hide file tree
Showing 5 changed files with 176 additions and 29 deletions.
123 changes: 97 additions & 26 deletions bin/growpart
Original file line number Diff line number Diff line change
Expand Up @@ -119,22 +119,32 @@ Usage() {
${0##*/} disk partition
rewrite partition table so that partition takes up all the space it can
options:
-h | --help print Usage and exit
--fudge F if part could be resized, but change would be
less than 'F' bytes, do not resize (default: ${FUDGE})
-N | --dry-run only report what would be done, show new 'sfdisk -d'
-v | --verbose increase verbosity / debug
-u | --update R update the the kernel partition table info after growing
this requires kernel support and 'partx --update'
R is one of:
- 'auto' : [default] update partition if possible
- 'force' : try despite sanity checks (fail on failure)
- 'off' : do not attempt
- 'on' : fail if sanity checks indicate no support
-h | --help print Usage and exit
--free-percent F resize so that specified percentage F of the disk is
not used in total (not just by this partition). This
is useful for consumer SSD or SD cards where a small
percentage unallocated can improve device lifetime.
--fudge F if part could be resized, but change would be less
than 'F' bytes, do not resize (default: ${FUDGE})
-N | --dry-run only report what would be done, show new 'sfdisk -d'
-v | --verbose increase verbosity / debug
-u | --update R update the the kernel partition table info after
growing this requires kernel support and
'partx --update'
R is one of:
- 'auto' : [default] update partition if possible
- 'force' : try despite sanity checks (fail on
failure)
- 'off' : do not attempt
- 'on' : fail if sanity checks indicate no
support
Example:
- ${0##*/} /dev/sda 1
Resize partition 1 on /dev/sda
- ${0##*/} --free-percent=10 /dev/sda 1
Resize partition 1 on /dev/sda so that 10% of the disk is unallocated
EOF
}

Expand Down Expand Up @@ -292,6 +302,7 @@ resize_sfdisk() {

local pt_start pt_size pt_end max_end new_size change_info dpart
local sector_num sector_size disk_size tot out
local excess_sectors free_percent_sectors remaining_free_sectors

LANG=C rqe sfd_list sfdisk --list --unit=S "$DISK" >"$tmp" ||
fail "failed: sfdisk --list $DISK"
Expand Down Expand Up @@ -350,7 +361,7 @@ resize_sfdisk() {
pt_start=$(awk '$1 == pt { print $4 }' "pt=${dpart}" <"${dump_mod}") &&
pt_size=$(awk '$1 == pt { print $6 }' "pt=${dpart}" <"${dump_mod}") &&
[ -n "${pt_start}" -a -n "${pt_size}" ] &&
pt_end=$((${pt_size}+${pt_start})) ||
pt_end=$((${pt_size} + ${pt_start} - 1)) ||
fail "failed to get start and end for ${dpart} in ${DISK}"

# find the minimal starting location that is >= pt_end
Expand All @@ -359,6 +370,8 @@ resize_sfdisk() {
min=${sector_num} pt_end=${pt_end} "${dump_mod}") &&
[ -n "${max_end}" ] ||
fail "failed to get max_end for partition ${PART}"
# As sector numbering starts from 0 need to reduce value by 1.
max_end=$((max_end - 1))

if [ "$format" = "gpt" ]; then
# sfdisk respects 'last-lba' in input, and complains about
Expand All @@ -378,11 +391,57 @@ resize_sfdisk() {

local gpt_second_size="33"
if [ "${max_end}" -gt "$((${sector_num}-${gpt_second_size}))" ]; then
# if mbr allow subsequent conversion to gpt without shrinking the
# partition. safety net at cost of 33 sectors, seems reasonable.
# if gpt, we can't write there anyway.
# if MBR, allow subsequent conversion to GPT without shrinking
# the partition and safety net at cost of 33 sectors seems
# reasonable. If GPT, we can't write there anyway.
debug 1 "padding ${gpt_second_size} sectors for gpt secondary header"
max_end=$((${sector_num}-${gpt_second_size}))
max_end=$((${sector_num} - ${gpt_second_size} - 1))
fi

if [ -n "${free_percent}" ]; then
free_percent_sectors=$((sector_num/100*free_percent))

if [ "$format" = "dos" ]; then
if [ $(($disk_size/512)) -ge $((mbr_max_512+free_percent_sectors)) ]; then
# If MBR partitioned disk larger than 2TB and
# remaining space over 2TB boundary is greater
# than the requested overprovisioning sectors
# then do not change max_end.
debug 1 "WARNING: Additional unused space on MBR/dos partitioned disk" \
"is larger than requested percent of overprovisioning."
elif [ $sector_num -gt $mbr_max_512 ]; then
# If only some of the overprovisioning sectors
# are over the 2TB boundary then reduce max_end
# by the remaining number of overprovisioning
# sectors.
excess_sectors=$((sector_num-mbr_max_512))
remaining_free_sectors=$((free_percent_sectors - excess_sectors))
debug 1 "reserving ${remaining_free_sectors} sectors from MBR maximum for overprovisioning"
max_end=$((max_end - remaining_free_sectors))
else
# Shrink max_end to keep X% of whole disk unused
# (for overprovisioning).
debug 1 "reserving ${free_percent_sectors} sectors (${free_percent}%) for overprovisioning"
max_end=$((max_end-free_percent_sectors))
fi

if [ ${max_end} -lt ${pt_end} ]; then
nochange "partition ${PART} could not be grown while leaving" \
"${free_percent}% (${free_percent_sectors} sectors) free on device"
return
fi
else
# Shrink max_end to keep X% of whole disk unused
# (for overprovisioning).
debug 1 "reserving ${free_percent_sectors} sectors (${free_percent}%) for overprovisioning"
max_end=$((max_end-free_percent_sectors))

if [ ${max_end} -lt ${pt_end} ]; then
nochange "partition ${PART} could not be grown while leaving" \
"${free_percent}% (${free_percent_sectors} sectors) free on device"
return
fi
fi
fi

debug 1 "max_end=${max_end} tot=${sector_num} pt_end=${pt_end}" \
Expand All @@ -397,9 +456,9 @@ resize_sfdisk() {
return
}

# now, change the size for this partition in ${dump_out} to be the
# new size
new_size=$((${max_end}-${pt_start}))
# Now, change the size for this partition in ${dump_out} to be the
# new size.
new_size=$((${max_end} - ${pt_start} + 1))
sed "\|^\s*${dpart} |s/\(.*\)${pt_size},/\1${new_size},/" "${dump_out}" \
>"${new_out}" ||
fail "failed to change size in output"
Expand Down Expand Up @@ -525,18 +584,17 @@ resize_sgdisk() {
pt_end=$(awk '$1 == '"${PART}"' { print $3 }' "${pt_data}") &&
[ -n "${pt_end}" ] ||
fail "${dev}: failed to get end sector"
# sgdisk start and end are inclusive. start 2048 length 10 ends at 2057.
pt_end=$((pt_end+1))
pt_size="$((${pt_end} - ${pt_start}))"
# Start and end are inclusive, start 2048 end 2057 is length 10.
pt_size="$((${pt_end} - ${pt_start} + 1))"

# Get the last usable sector
last=$(awk '/last usable sector is/ { print $NF }' \
"${pt_pretend}") && [ -n "${last}" ] ||
fail "${dev}: failed to get last usable sector"

# Find the minimal start sector that is >= pt_end
# Find the maximal end sector that is >= pt_end
pt_max=$(awk '{ if ($2 >= pt_end && $2 < min) { min = $2 } } END \
{ print min }' min="${last}" pt_end="${pt_end}" \
{ print min-1 }' min="${last}" pt_end="${pt_end}" \
"${pt_data}") && [ -n "${pt_max}" ] ||
fail "${dev}: failed to find max end sector"

Expand Down Expand Up @@ -569,7 +627,7 @@ resize_sgdisk() {
[ "$DRY_RUN" -ne 0 ] && wouldrun="would-run"

# Calculate the new size of the partition
new_size=$((${pt_max} - ${pt_start}))
new_size=$((${pt_max} - ${pt_start} + 1))
change_info="partition=${PART} start=${pt_start}"
change_info="${change_info} old: size=${pt_size} end=${pt_end}"
change_info="${change_info} new: size=${new_size} end=${pt_max}"
Expand Down Expand Up @@ -874,6 +932,19 @@ while [ $# -ne 0 ]; do
Usage
exit 0
;;
--free-percent|--free-percent=*)
if [ "${cur#--free-percent=}" != "$cur" ]; then
next="${cur#--free-percent=}"
else
shift
fi
if [ "$next" -gt 0 ] 2>/dev/null &&
[ "$next" -lt 100 ] 2>/dev/null; then
free_percent=$next
else
fail "unknown/invalid --free-percent option: $next"
fi
;;
--fudge)
FUDGE=${next}
shift
Expand Down
4 changes: 2 additions & 2 deletions test/test-growpart-fsimage-middle
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,8 @@ cleanup() {
TEMP_D=$(mktemp -d ${TMPDIR:-/tmp}/${0##*/}.XXXXXX)
trap cleanup EXIT

expected="CHANGED: partition=3 start=731136 old: size=819200 end=1550336"
expected="${expected} new: size=3330048 end=4061184"
expected="CHANGED: partition=3 start=731136 old: size=819200 end=1550335"
expected="${expected} new: size=3330048 end=4061183"
CR='
'
for resizer in sfdisk sgdisk; do
Expand Down
75 changes: 75 additions & 0 deletions test/test-growpart-overprovision
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
#!/bin/bash
#
# Just create an image in the filesystem, then grow it.

set -e

PT_TYPE="${PT_TYPE:-dos}" # dos or gpt
size=${DISK_SIZE_NEW:-100M}
osize=${DISK_SIZE_ORIG:-50M}
freepercent=${OVER_PROVISION_PERCENT:-10}

TEMP_D=""

cleanup() {
[ ! -d "${TEMP_D}" ] || rm -Rf "${TEMP_D}"
}
rq() {
local out="${TEMP_D}/out"
"$@" > "$out" 2>&1 || { echo "FAILED:" "$@"; cat "$out"; return 1; }
}

TEMP_D=$(mktemp -d ${TMPDIR:-/tmp}/${0##*/}.XXXXXX)
trap cleanup EXIT

img="${TEMP_D}/disk.img"

echo "Partitioning $PT_TYPE orig_size=$osize grow_size=$size, overprovisioning=$freepercent%."
echo "growpart is $(which growpart)"
rm -f $img

truncate --size $osize "$img"

label_flag="--label=${PT_TYPE}"
echo "2048," | rq sfdisk $label_flag --force --unit=S "$img"

truncate --size "$size" "$img"

echo "==== before ===="
sfdisk --list --unit=S "$img"

err="${TEMP_D}/gp.err"
out="${TEMP_D}/gp.out"
if ! growpart -v -v --free-percent "$freepercent" "$img" 1 2>"$err" > "$out"; then
cat "$err" "$out"
echo "failed"
exit 1
fi
echo "==== growpart-stderr ===="
cat "$err"
echo "==== growpart-stdout ===="
cat "$out"
grep -q "^CHANGED:" "$out" ||
{ echo "did not find 'CHANGED'"; exit 1; }

echo "==== after ===="
sfdisk --list --unit=S "$img"

enddevice=$(sfdisk --list --unit=S "$img" | grep "Disk $img:" | awk '{print $7}')
endpart=$(sfdisk --list --unit=S "$img" | grep "$img" | grep -v "Disk" | awk '{print $3}')
# Subtract the following from disk image end in sectors:
# - required number of overprovisioning sectors to leave unused
# - 33 sectors (MBR padding for GPT conversion)
# - 1 (as sector numbers start from 0)
# to calculate the expected end of resized partition in sectors.
expectedendpart=$((enddevice-(enddevice/100*freepercent)-33-1))
echo
if [ $endpart = $expectedendpart ]; then
echo "Final partition size matches expected partition size"
echo
else
echo "ERROR: final partition size of $endpart is different than expected size of $expectedendpart"
exit 1
fi

# vi: ts=4 noexpandtab
2 changes: 1 addition & 1 deletion test/test-growpart-start-matches-size
Original file line number Diff line number Diff line change
Expand Up @@ -92,4 +92,4 @@ trap cleanup EXIT
# the sfdisk and sgdisk resizers result in slightly different output,
# because of course they do.
test_resize sfdisk 1026048 3168223
test_resize sgdisk 1026048 3166208
test_resize sgdisk 1026048 3166207
1 change: 1 addition & 0 deletions tox.ini
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ shfiles =
test/test-growpart
test/test-growpart-fsimage
test/test-growpart-fsimage-middle
test/test-growpart-overprovision
test/test-growpart-lvm
test/test-growpart-start-matches-size
test/test-mic
Expand Down

0 comments on commit c00ff0c

Please sign in to comment.