Heads integration of wyng

[tlaurion]

Hello there again.

I'm really impressed by this project. Good work!

I would love to bundle wyng under Heads for different reasons:

1. Have a copy of /var/lib/wyng.backup under /boot. Heads creates a digest of everything being under /boot at each boot, and enforces verification of that autogenerated digest against ROM inserted, and TPM measured, gpg public key corresponding to USB security dongle private key. If it was possible to use wyng to verify that LVMs were not tampered with against volinfo/delta maps, Heads's OEM reownership wizard could then decrypt LUKS content and verify LVM integrity prior of reencrypting the LUKS volumes, while attesting of its state against OEM public key (Removing implicit trust of international distributors).
2. Have a whole wyng backup of LVMs content under /boot if (1) above is not enough to validate integrity and check against the backup directly. That would require resizing /boot so that the backups can sit there, while also empowering the user to restore their QubesOS installation in a clean, known, reversed state, for dom0, templates and OEM deployed AppVMs).
3. Heads OEMs could deploy a ssh public key to be used to connect to a read-only ssh repository to restore system to a known state for already existing templates and AppVMs, directly from Heads recovery shell.
4. Once Support metadata payload for VM config/other #18 is implemented, it would even be possible for the user to select new templates/VMs from ssh to be deployed directly from the recovery console.

My main interest would be to figure out the feasibility of 1 or if 2 would be required to obtain 1 goal's. And move incrementally from there.

If that would be possible, i would restart integration of lvm-provisioning-tools inside of Heads to make this go forward, since my main problem right now is to make OS integrity measurable by the user upon hardware reception against what I intend to be deployed from distributors (make sure they can't tamper with the LVM integrity inside of downloaded disk image they deploy).

• Measured boot against TOTP/HOTP (reprogrammer liability): done
• /boot integrity checked against Insurgo signature (my liability):done
• Firmware checked against /boot reproducible ROM, matching specific commit id signed version. Could even be redownloaded and checked against detached signature directly from Heads (my liability): todo
• QubesOS image (LVMs) integrity checked against wyng signed digest verification (my liability): todo.

That way, localized distribution of hardware would be possible, removing trust into remote hardware reprogrammers. Let me know what you think, else i'll investigate deeper the dm-verity way to provide needed LVMs verified integrity of volumes contained inside of LUKS encrypted container.

[tlaurion]

@tasket : any insight?

[tlaurion]

With better understanding, I think 2 above is the way to go, providing both integrity verification (diff returns 1 if different) and restoration of initial deployment state if desired (deleting all other LVMs, restoring dom0 and QubesOS known volumes).

@tasket Would it be possible to customize /var/lib/wyng.backup path from command line so that the archive.ini points to /boot backups directly?

The more I think about it and the more I'm thinking that those volumes should become the alternative to rpm template deployments from QubesOS, no?

[tasket]

I can't say just yet how well wyng could support this use case, but it sure sounds interesting. I'll try to get you a detailed response by Saturday, as I've only ever used AEM and don't want to sound stupid. ;) Initial thought is that right now wyng has value as a hash generator and fast updater; anything involving crypto verification has to come from another tool (e.g. Heads) which would operate on wyng hash manifests.

More later...

[tasket]

Short answer on custom path is yes the metadata path could be overridden if an option for it was added.

[tlaurion]

@taskset Let me know your ideas on 1, I will continue thin-provisioning-tools inclusion under heads and explore if micropython could fill wyng requirements.

[tlaurion]

My current strategy will be to use a QubesOS AppVM to store wyng backups inside of a thin provisioned LVM, not backuped by wyng, and access that thin LVM from Heads after unlocking the LUKS container for which the user receives temporary decryption passphrase when hardware is in his hands(transit tamper proofing). Once integrity of LVMs is validated/restored, the user will proceed to reencrypt received sdcard and SSD LUKS containers as per actual process.

Consequently, only wyng metadata would be present under /boot, for which Heads generates a digest at each boot and verifies this generated digest integrity against a signed one. This adds a 5 seconds delay at boot time, since the number of files is important, but would be less disastrous then having the backups also under /boot. (Need to test how long Heads would take to generate a digest for all backup files)

Not including the backups inside of /boot would saves a lot of time at each boot, since those files are not considered when generating the digest then verified against the signed one for all those files. The downside of this is that if the LVM pool is broken, the user would also loose his backup VM and wouldn't be able to restore dom0 and original AppVMs and templates.

Other possible strategies:

• Have a distinct factory restore partition outside of the LVM pool, mounted by Heads only when requested (reownership and on demand to restore LVMs)
• Others?

[tasket]

First comment: I'll limit myself to addressing 1 & 2 for now...

I think this is feasible for verification without storing actual backup data.

However:

• Wyng would need a new verification mode, a variation on diff that works a bit like send, reading and compressing the source lv.
• Performance penalty: Wyng currently cannot do hashes-to-volumes verification without either compressing or decompressing data. Impact depends on when and how often you want to verify lvs.

I plan to add Zstandard compression as an option in v0.3. This should greatly increase verification performance. But making wyng verify as fast as direct hashing would require a format change that almost doubles the metadata size on disk (currently, the trade-off being made is for a combination of space efficiency and security).

On metadata size: Its possible wyng will begin to compress manifest files in a future release, which would lessen boot delays.

You could forgo wyng and hash the volumes directly (b2sum is in coretools now and is very fast – b3sum should be faster when it arrives) which could provide simple verification, but not restore or update capabilities.

[tlaurion]

Consequently, only wyng metadata would be present under /boot, for which Heads generates a digest at each boot and verifies this generated digest integrity against a signed one. This adds a 5 seconds delay at boot time, since the number of files is important, but would be less disastrous then having the backups also under /boot. (Need to test how long Heads would take to generate a digest for all backup files)

@tasket Well. Generating a digest for backups takes 3minutes for standard templates and VMs. Acceptable if outside of /boot, but not in from Heads perspective. Detaced-signed digest on external volume to be checked again only when needed fits the bill as of right now.

[tlaurion]

@tasket having issues at integrating thin-provisioning-tools with Heads build system as of right now. WiP here.

[tasket]

That 3min run time sounds like you are hashing all the data with the metadata. What if you only hashed the volinfo, info and manifest files?

[tasket]

I'm still not clear on the use cases and workflow associated with the integration. For example, the need for thin-provisioning-tools in the Heads startup environment... Wyng only uses thin_delta from these tools, and only for monitor and send operations. I don't even think the thin-provisioning-tools are necessary when creating new thin volumes (i.e. when using receive).

If what you need is receive or verify capability in Heads, then I think its likely it could be done without thin-provisioning-tools. However, Wyng will have to be told not to exit when it doesn't find them.

[tlaurion]

I was effectively basing digest creation on all the data to make sure it was not tampered with. Heads measuring the public keys inserted in ROM in its TPM measurements, I was basing myself in that root of trust to detach sign that digest with USB security key on all the backup data, not just it's metadata. Unless wyng could fail if metadata doesn't match actual data in backups then it might not be a need.

[tlaurion]

The use cases would be multiple. First one would be to verify consistency of all lvm-volumes against backup once at reownership. Goal being to discourage distributed reprogrammers to tamper with disk image or even booting the OS while he has the decryption key prior of reencrypting content.

Another use case would be to add into heads a OEM ssh public key and a remote backup repository, where the user would basically be offered the possibility to deploy/restore from remotely available TemplateVMs and AppVMs from a list proposed to him.

[tasket]

Since Wyng keeps 256bit hashes for all the data chunks (these are the 'manifest' files), you can just create digests for those 3 file types (plus 'archive.ini') as I suggested earlier; digests could be in a single list that is signed by Heads. This allows you to quickly check the archive integrity on a basic level on each Heads startup, for example, and then you can have Wyng perform data verification if/when it is necessary. Data verification would probably take about the same amount of time, but doing it this way might give you more flexibility and better UX.

I just ran a test with thin-provisioning-tools removed, and indeed you can have LVM access and create thin LVs without them. Only caveat is that lvcreate will complain that it could not run thin_check as a precaution. I don't think it complains when accessing existing thin volumes.

[tasket]

Of course, current Wyng version will exit if thin_delta isn't present. I'm already making changes that will allow greater flexibility, so you can run most commands like verify and receive without thin tools present.

[tlaurion]

Will play with building micropython and python packaging tommorow. Thanks!

[tasket]

Based on a quick examination of micropython v1.12 in Debian, some critical libraries that are missing are: subprocess, stat, shutil, tmpfile, argparse and datetime.

Best strategy for conversion to micropython would be to isolate the receive_volume() function and the functions it calls; this would cover receive, verify and possibly diff commands. Converting commands like monitor, send and prune would not be realistic, however (at least, not for myself).

Another avenue is to script it in the shell. As you may know, early sparsebak had a compact shell script that could verify and extract a volume from an archive; adapting it should be possible.

[tlaurion]

Nah, goal would be to package wyng, not to rewrite it. Will look into python3

[tasket]

If shoe-horning CPython into Heads is a struggle, I can still provide a shell script for about 1/100 the effort. After adding xz/lzma support to wyng, I tested the ability of xzcat to extract the data into whole volumes and it works. A bonus is that the archive is smaller due to lzma's stronger (but slower) compression.

This cat-ing ability wasn't really an option with the default zlib compression bc I couldn't find a shell tool to decompress naked zlib chunks (gzip won't do it).

[tasket]

Well, that's interesting. The pigz tool can do the job with zlib, and it seems to come standard in Debian 10.

[tasket]

Here's a gist that can check and extract a volume (uses pigz):
https://gist.github.com/tasket/48b30124990e1c78c80c8716f819430a

[tlaurion]

Can we rely on standard gzip from busybox?

[tasket]

No, unless you use some tricks to pad gzip headers to the data files. Very messy and slow.

That's why I was trying out lzma/xz format. IIRC its found in most startup/rescue environments.

Another option is to make gzip a separate compression choice so the files are created in gzip format.

[tlaurion]

xz is supported AFAIK.

On March 3, 2020 5:20:24 PM UTC, tasket ***@***.***> wrote: No, unless you use some tricks to pad gzip headers to the data files. Very messy and slow. That's why I was trying out lzma/xz format. IIRC its found in most startup/rescue environments. Another option is to make gzip a separate compression choice so the files are created in gzip format. -- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: #54 (comment)

-- Sent from /e/ Mail

[tasket]

This is giving me second thoughts about using xz format:

Key findings include: (1) safe interoperability among xz implementations is not guaranteed; (2) xz's extensibility is unreasonable and problematic; (3) xz is vulnerable to unprotected flags and length fields; (4) LZMA2 is unsafe and less efficient than the original LZMA; (5) xz includes useless features that increase the number of false positives for corruption; (6) xz shows inconsistent behavior with respect to trailing data; (7) error detection in xz is several times less accurate than in bzip2, gzip and lzip.

---

Looking at the alternatives and the APIs in the old dom0 version of Python, I think bzip2 would be the best way to go because its API is more modern and easy to plug in here, and its respected as a high integrity format.

[tasket]

@tlaurion Commit c327968 adds a bz2 compression option. This is available in busybox.

lzma is also available in busybox, but the command names used to invoke it there varied wildly depending on which install I was looking at, and they didn't overlap.

[tlaurion]

i'll investigate deeper the dm-verity way to provide needed LVMs verified integrity of volumes contained inside of LUKS encrypted container.

@tasket small update while revising my notes: QubesOS/qubes-issues#4371 (comment)

[tasket]

FWIW, a factor is that the target partition /boot is small. It would be easy to run a dd from /dev/nvme0n1 (/boot dev) into a thin LV that is configured in Wyng; this could be done at intervals, or the mtimes of the partition vs LV could be compared in deciding when to run dd.

My own personal routine is before doing system update to dom0, to do an rsync -av --delete /boot /boot-bak where boot-bak is a dir in the root volume. Then root vol is backed up as usual. For anyone wanting to also be able to restore the original partition layout after a disaster, they would also need to backup that info to a small file in root, probably using sfdisk but it could also be done with dd sourced from the first blocks of the boot drive.

[tasket]

@tlaurion I know this has taken considerable time and seems like it may be out of steam, but here is what I would do to get this working in the short-term:

1. Include a specific version of zstandard
2. Also port GNU dd command to busybox
3. Add necessary remote support via ssh or sftp or rsync (and sshfs no longer an option)

Number 1, handled by you already I think.

Number 2, porting dd seems manageable and avoids a much bigger/messy re-write of the extraction script, allowing the recent edits to work as-is.

Number 3, handled by me; should take a few days. I would probably have it call another, generic file-fetching script where we can put any specific protocol details.

This approach produces the fastest, most efficient result from wyng-extract.sh with the least coding time if the dd port doesn't present special problems for you. Let me know what you would like to do!