Say you want to make an image file from of a disk or partition, and then restore it later. You can achieve it easily with dd:
sudo dd if=/dev/sdxx of=$HOME/some/dir/test.diskimage
sudo dd if=$HOME/some/dir/test.diskimage of=/dev/sdxxLet's add a few features:
Display a visual notification at the end.
Copying a disk may take a long time. We will use my background.sh tool, which displays a desktop notification at the end. This tool also lowers the disk priority, in order to reduce the performance impact on your desktop environment.
If you do not want to use background.sh, leave it out from the examples below.
Display a progress bar.
We could use dd's option status=progress, which prints something like this:
77594624 bytes (78 MB, 74 MiB) copied, 13 s, 5,9 MB/sHowever, the progress message does not provide an estimation of the time left even if the total size of the data is known, at least with GNU dd version 8.28.
Therefore, we will use pv instead. There are ways to make it display a dialog with a graphical progress bar, but we will just display a text progress bar in the current console. This is what it looks like:
49MiB 0:00:44 [6,07MiB/s] [===> ] 13% ETA 0:04:53pv's option "-f" ("--force") is necessary if you use background.sh. Otherwise, pv will think it is not outputting to a terminal (but to a tee pipe) and will not generate the progress bar.
The combined command-line option -ftpreb means:
f: --force t: --timer p: --progress r: --rate e: --eta b: --bytesSync the disk cache at the end.
This way, when dd has finished, we know that the all the data has been physically written to disk. Otherwise the data will land in the write-back cache first, and we must not forget to run sync before removing the physical media.
dd's option "oflag=sync" syncs after each output block, and "conv=fsync" does one sync at the end.
Avoid quickly overloading the system's file cache.
Linux' page cache is braindead and will agressively write-cache huge amounts of data at once, eventually dropping all other cached data. Your desktop environment may be rendered unresponsive afterwards, and you will have to wait a long time for the write cache to flush after the progress indication has reached 100%. This effect is especially noticable on slow USB 2.0 disks.
We will be limitting dd's block size with "bs=128M". Beware that this makes dd consume 128 MiB of RAM during the whole operation.
dd's "oflag=sync" option will make it sync to disk from time to time. This will will not eliminate the performance degradation, but will smooth it out over time. The final write-back flush time will also be reduced considerably, as write operations will be evenly spread during the whole operation.
Alternatively, dd's option "oflag=direct" bypasses the write caching and should avoid this issue completely.
Prevent the "partial read" warning.
If you do not use option iflag=fullblock, you risk reading less data than specified, and you get a warning like this:
dd: warning: partial read (131072 bytes); suggest iflag=fullblock 0% ETA 4:59:16Reliably report errors with a non-zero exit code.
By default, the shell ignores some of the error indications from pipe constructs. Therefore, we will be using command "set -o pipefail". This is especially important when using background.sh, as the final "success" or "error" indication depends on the exit code.
Compress the image file.
We will be using gzip to (hopefully) reduce the image file size. It is probably best to favour speed over compression level, so we will be using option "--fast". Otherwise, you will probably want to use bzip or xz instead.
Use TRIM / UNMAP for performance and wear leveling purposes.
Some disks support the TRIM / UNMAP command in order to improve performance and wear leveling.
If you want to store your disk image file on a FAT32 partition, you will easily hit the 4 GiB size limit. We could also use split to overcome this limitation.
The souped-up versions look as follows. You will always have to modify the parts in bold.
If the disk has writable partitions (like SATA disks or USB sticks), first of all, unmount any partitions on the source disk which are mounted as read/write.
If you do not compress the image file, then you can mount it directly later, for example for testing purposes.
In order to make a CD-ROM ISO image, the command would be:
background.sh bash -c "sudo pv -ftpreb /dev/cdrom >cdimage.iso"
In order to make a disk image and compress it with gzip:
background.sh bash -c "set -o pipefail && sudo pv -ftpreb /dev/sdxx | gzip --fast >\"$HOME/some/dir/test.diskimage.gz\""
Instead of gzip with fast but poor compression, for maximum compression use xz like this:
background.sh bash -c "set -o pipefail && sudo pv -ftpreb /dev/sdxx | xz -9 --threads=0 >\"$HOME/some/dir/test.diskimage.xz\""
Step 1) Unmount any partitions on the destination disk.
Step 2) In case the disk supports TRIM / UNMAP, now it is a good time to trim the whole disk:
sudo blkdiscard /dev/sdxx
Step 3) Prepare and run a command like this:
background.sh bash -c "set -o pipefail && pv -ftpreb \"$HOME/some/dir/test.diskimage.gz\" | gunzip --to-stdout | sudo dd bs=128M iflag=fullblock oflag=direct conv=fsync of=/dev/sdxx"
Note that the progress bar will not be very accurate, as pv will be measuring the amount of data read from the compressed file. Therefore, if the disk image has large empty areas (filled with zeros), the time estimates will be off by a large margin. For example, if the last disk sectors are empty, then the progress indicator will remain at 100% for a long time at the end. The trouble is, the disk may be bigger than the image, and finding out the size of the uncompressed data beforehand is not trivial.
For .xz files, use "xz --decompress --to-stdout" instead of the gunzip command.
If the source is a .zip file, you cannot pipe it to "unzip -p", because unzip cannot read .zip archives from stdin. I heard that zcat (the GNU version, equivalent to gunzip --to-stdout, not the BSD version of gunzip) can read from stdin and will then decompress only the first file inside the .zip archive, issuing a warning if there is more than one compressed file in the archive, but I haven't tested it yet. So, either decompress the .zip archive first, or use this alternative, which will not show an estimated time left:
background.sh bash -c "set -o pipefail && unzip -p \"$HOME/some/dir/test.diskimage.zip\" | pv -ftpreb | sudo dd bs=128M iflag=fullblock oflag=direct conv=fsync of=/dev/sdxx"
Step 4) Reload the partition table:
sudo partprobe --summary
Tool 'partprobe' is in the Ubuntu/Debian package named 'parted'.
First of all, you should probably unmount and remount the disk, or the system's disk cache may falsify the data verification results. Alternatively, you could drop the Linux system cache by writing to /proc/sys/vm/drop_caches, but that will affect the performance of the whole system.
Note that, if you remove and reattach the disk, the automounter will probably automatically mount all partitions with read/write access, and the disk will often be immediately written to, so the data verification will always fail. There are many reasons why a filesystem is written on first touch:
Filesystems tend to be mounted with mount option relatime.
Upon automatic mounting, most systems take a look at the root directory, which is often enough to update the "last access" timestamp on some files or directories.
As soon as Linux mounts a FAT partition as read/write, it sets its 'dirty' bit.
Upon a clean unmount, the 'dirty' bit is reset. Nevertheless, there is always room for trouble with this bit.
The ext4 journal is automatically replayed on mount.
This is in case the last write operation wasn't properly committed to disk.
ext4 has a superblock field called "last mount time" which gets unconditionally updated on mount.
You can see that timestamp with sudo "tune2fs -l /dev/<your device here>", look in the output for a text line which starts with "Last mount time:".
The best way to prevent unwanted modifications before comparing is to disable the automounter. Automounting is often performed by your desktop environment, so you may have to look there for the corresponding setting.
The easiest way to verify whether a disk image was written correctly is to read the data back from the disk and compare it with the image file. For example:
background.sh bash -c "set -o pipefail && pv -ftpreb \"$HOME/some/dir/test.diskimage.gz\" | gunzip --to-stdout | sudo cmp -- - /dev/sdxx"
In the cmp command above, '-' means stdin, and sudo is needed because cmp will be reading from a block device, instead of from a regular file.
The verification will probaby fail, because the disk is usually bigger than the image which was written to it. But as long as the error message looks like this, then it is fine:
cmp: EOF on - after byte 10989076480, in line 89840717Error message "EOF on '-'" just means that the image file was shorter than the disk, which was expected.
If you are using an uncompressed image file, you can prevent that error by telling cmp how long the data is:
IMAGE_FILE="$HOME/some/dir/test.diskimage"
background.sh bash -c "set -o pipefail && pv -ftpreb \"$IMAGE_FILE\" | sudo cmp --bytes=$(stat -c "%s" -- "$IMAGE_FILE") - /dev/sdxx"
If you are going to verify the same image often, it would be faster to calculate a data checksum, so that you do not have to read the original disk image every time.
BLOCK_DEVICE=/dev/sdxx background.sh bash -c "set -o pipefail && BLOCK_DEVICE_SIZE=\"\$(sudo blockdev --getsize64 \"\$BLOCK_DEVICE\")\" && dd bs=128M count=\$BLOCK_DEVICE_SIZE iflag=fullblock,count_bytes if=/dev/zero | pv -ftpreb --size \$BLOCK_DEVICE_SIZE | sudo dd bs=128M iflag=fullblock oflag=direct conv=fsync of=\"\$BLOCK_DEVICE\""
This one-liner just writes zeros to the whole disk or partition. In order to display an accurate progress bar, we have to find out beforehand (with blockdev) how big the disk is. We need to read with dd because there is no way to tell pv to stop reading after the given number of bytes.
In case the disk supports TRIM / UNMAP, now it is a good time to trim the whole disk:
sudo blkdiscard /dev/sdxx