layout.md

isohybrid images done right

'isohybrid' describes an image that can be both used as dvd (it has a valid iso9660 file system at offset 0) and as disk image (it also has a valid partition table).

The partition table type (gpt, mbr, or even apple) doesn't matter - the iso fs leaves enough unassigned space at its start (32 kB) to accomodate any of those types.

The catch is: Where should the data partition itself start? Obviously, at block 0 would be a choice. But such a partition would again enclose the partition table itself and partitioning tools really don't like such a layout.

Also, if the image should be EFI bootable, the EFI boot image needs its own partition. You can of course create a partition pointing to the EFI image included in the iso fs but then you get overlapping partitions and partitioning tools will eat you for lunch - this time for sure.

So, we do it differently. For isohybrid images we aim to create an image that

1. looks like an iso fs and
2. has a valid (sane) partition table with
   • one partition containing the same fs as 1. and
   • (optionally) another partition containing an EFI boot image

Using an iso9660 file system for the partition

To achieve this, two mkisofs runs are needed. To understand why the following works it's important to know that the iso fs layout starts with fs meta data, followed by directory data, followed by file data.

Now imagine you manage to put valid iso meta/directory data into the first data file: this basically means you have an iso fs (excluding this file) that doesn't start at offset 0 but at the offset of the first data file.

Let's see how this works in detail.

There are two special files in our iso fs we need below:

• efi: (optional) EFI (fat-) boot image file (full path on x86_64 is boot/x86_64/efi)
• glump: hidden(!) 2k-block containing a magic blob (as the file doesn't have a directory entry, we need the magic to find it in the image)

efi must be the 1st file, glump the 2nd (mkisofs lets you force the file order easily).

The fs layout looks like this after a 1st mkisofs run:

== 1st run ==
iso meta/dir 1
data 1
  efi 1
  glump 1
    magic 1
  other files 1

(Indentation is used to indicate 'contains'.)

So, glump contains the magic blob, and data consists of efi, glump, and other files.

The numbers are a reminder of the run the data was generated.

Now we copy iso meta/dir ... glump (inclusive) into a new glump and run mkisofs on the same tree again. We get this layout:

== 2nd run ==
ofs 0->   iso meta/dir 2
          data 2
part1->     efi 2
part2->     glump 2
              iso meta/dir 1
              data 1
                efi 1
                glump 1
                  magic 1
            other files 2

Note that other files 1 and other files 2 are identical (we didn't change any data there).

So, the result is:

• a valid iso fs at offset 0
• a valid iso fs at glump 2 == iso meta/dir 1

This means we can add a partition table where:

• part1 points to the EFI boot partition
• part2 points to a valid iso fs referencing the same files as via offset 0
• part1 and part2 don't overlap

Note that the EFI boot image exists twice - but efi 1 is unused (El Torito references efi 2 for booting). It would technically be possible to exclude it in the 1st run (efi 1 would then be gone) without loss of functionality. But this would mean the users see different things depending on whether they mount at offset 0 compared to partition 2. So, let's keep it and waste the few bytes.

Using a fat file system for the partition

Why not have an iso fs at offset 0 but a differnt fs type in partition 2?

The fat file system has a similar structure than iso fs: first fs meta data, then (mixed) directory and file data. But it's easy to disentangle directory and file data and ensure directories are first. So we can achieve the same basic fs layout and use the same trick as above again.

Let's start by running mkisofs:

== 1st run ==
iso meta/dir 1
data 1
  efi 1
  glump 1
    magic 1
  other files 1

Now we need to prepare the content of glump. This is a bit more tricky than with the iso fs above - but possible.

Here are the details:

1. create a new fat fs, with desired size (enough for all files, at least)
2. choose cluster size 2k, so files are aligned as in the iso fs (which also uses 2k blocks)
3. but: fat meta data can still be non-2k aligned - for this we will later move the start of the whole file system so the start of the file data is 2k-aligned
4. create all files in the order they are in the iso fs, but with size 0 - this will take care of creating all needed directory entries
5. then truncate all files to the correct size
6. while we do this, check if there are hidden files in the iso fs that would destroy the relative file alignment; if such a gap is detected, insert padding files into the fat fs
7. delete any inserted padding files
8. verify that the (relative) file offsets match the offsets in the iso fs
9. determine the alignment blocks needed (as explained in step 3.)
10. copy alignment blocks + fat fs up to the start of the file data into glump
11. add efi (full 2k-blocks) to glump, if it exists

Then, after a 2nd mkisofs run, the layout looks like this:

== 2nd run ==
ofs 0->   iso meta/dir 2
          data 2
part1->     efi 2
            glump 2
              alignment
part2->       fat meta/dir
              efi
            other files 2

So, the result is:

• a valid iso fs at offset 0
• a valid fat fs at fat meta/dir

This means we can add a partition table where:

• part1 points to the EFI boot partition
• part2 points to a valid fat fs referencing the same files as via offset 0
• part1 and part2 don't overlap

Note that the fat fs is in fact writable but modifying any files will disrupt the iso fs side. This shouldn't cause any problems in real life as users probably don't copy the image onto a usb stick, use it for a while, and then later burn a dvd with it.