qvm-backup-restore fails to restore from version 3 backups

[andrewdavidwong]

How to file a helpful issue

Qubes OS release

4.1.1

Brief summary

Attempting to restore from backups created between January 2016 and January 2019 (backup format version 3) fail. Restoring from backups created in October 2020 and later (backup format version 4) succeed.

(Note: I'm not saying these are the dates when the backup formats changed; these are just the dates of the backups I happened to try.)

Steps to reproduce

Try to restore from an older backup (backup format version 3, e.g., Jan 2016 - Jan 2019) on Qubes 4.1.1.

(Note: All my backups were successfully verified using the Qubes "verify-only" restore option at the time I created them.)

Expected behavior

qvm-backup-restore and its GUI equivalent are backward-compatible with all backups previously created on older Qubes OS systems, and restore succeeds normally.

Actual behavior

When attempting to restore from older backups, the errors always look similar to this:

[user@dom0 ~]$ qvm-backup-restore --rename-conflicting --skip-dom0-home -d backups /home/user/qubes-backup-2019-01-01 work
Please enter the passphrase to verify and (if encrypted) decrypt the backup:
app: Checking backup content...
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
bad decrypt
128286285820742:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:crypto/evp/evp_enc.c:643:
qubesadmin.backup.extract: ERROR: unable to extract files for qubes.xml.000, tar output:
 

  gzip: stdin: not in gzip format

  tar: Child returned status 1

  tar: Error is not recoverable: exiting now

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/qubesadmin/backup/core2.py", line 349, in load
    tree = lxml.etree.parse(fh)
  File "src/lxml/etree.pyx", line 3467, in lxml.etree.parse
  File "src/lxml/parser.pxi", line 1860, in lxml.etree._parseDocument
  File "src/lxml/parser.pxi", line 1880, in lxml.etree._parseFilelikeDocument
  File "src/lxml/parser.pxi", line 1775, in lxml.etree._parseDocFromFilelike
  File "src/lxml/parser.pxi", line 1187, in lxml.etree._BaseParser._parseDocFromFilelike
  File "src/lxml/parser.pxi", line 601, in lxml.etree._ParserContext._handleParseResultDoc
  File "src/lxml/parser.pxi", line 711, in lxml.etree._handleParseResult
  File "src/lxml/parser.pxi", line 640, in lxml.etree._raiseParseError
  File "/var/tmp/restoreyh0yrvx1/qubes-restored.xml", line 1
lxml.etree.XMLSyntaxError: Document is empty, line 1, column 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/qvm-backup-restore", line 5, in <module>
    sys.exit(main())
  File "/usr/lib/python3.8/site-packages/qubesadmin/tools/qvm_backup_restore.py", line 274, in main
    backup = BackupRestore(args.app, args.backup_location,
  File "/usr/lib/python3.8/site-packages/qubesadmin/backup/restore.py", line 958, in __init__
    self.backup_app = self._process_qubes_xml()
  File "/usr/lib/python3.8/site-packages/qubesadmin/backup/restore.py", line 1373, in _process_qubes_xml
    backup_app = Core2Qubes(qubes_xml_path)
  File "/usr/lib/python3.8/site-packages/qubesadmin/backup/core2.py", line 151, in __init__
    super().__init__(store)
  File "/usr/lib/python3.8/site-packages/qubesadmin/backup/__init__.py", line 33, in __init__
    self.load()
  File "/usr/lib/python3.8/site-packages/qubesadmin/backup/core2.py", line 351, in load
    xml.parsers.expat.ExpatError) as err:
AttributeError: module 'xml.parsers' has no attribute 'expat'

(Note: I also tried the same restores in the GUI backup restore tool, which simply causes the GUI tool to crash with no errors or notifications of any kind.)

According to the backup header, the compression filter is, indeed, gzip:

version=3
hmac-algorithm=SHA512
crypto-algorithm=aes-256-cbc
encrypted=True
compressed=True
compression-filter=gzip

(Note: I tried with both gzip- and bzip2-compressed backups. Same results regardless of compression filter.)

For comparison, this is from the backup header of the newer backup (from October 2020) that successfully restored:

version=4
hmac-algorithm=scrypt
encrypted=True
compression-filter=bzip2
compressed=True

[rustybird]

Can you confirm that this change in dom0 fixes it:

--- /usr/lib/python3.8/site-packages/qubesadmin/backup/restore.py.bak
+++ /usr/lib/python3.8/site-packages/qubesadmin/backup/restore.py
@@ -679,6 +679,8 @@
                         ["openssl", "enc",
                          "-d",
                          "-" + self.crypto_algorithm,
+                         "-md",
+                         "MD5",
                          "-pass",
                          "pass:" + self.passphrase],
                         stdin=subprocess.PIPE,

For context, OpenSSL 1.1.0+ changed the default digest algorithm that's used to derive a key from a password from MD5 to SHA256.

[andrewdavidwong]

Can you confirm that this change in dom0 fixes it:

--- /usr/lib/python3.8/site-packages/qubesadmin/backup/restore.py.bak
+++ /usr/lib/python3.8/site-packages/qubesadmin/backup/restore.py
@@ -679,6 +679,8 @@
                         ["openssl", "enc",
                          "-d",
                          "-" + self.crypto_algorithm,
+                         "-md",
+                         "MD5",
                          "-pass",
                          "pass:" + self.passphrase],
                         stdin=subprocess.PIPE,

Yes! After making this change, I can successfully restore from v3 backups on the command line using qvm-backup-restore, but for some reason trying to restore via the GUI tool still crashes it (and takes down the entire Qube Manager with it).

For context, OpenSSL 1.1.0+ changed the default digest algorithm that's used to derive a key from a password from MD5 to SHA256.

Good catch, thanks!

[rustybird]

trying to restore via the GUI tool still crashes it (and takes down the entire Qube Manager with it)

I forgot to mention to restart Qube Manager after applying the fix (if you're launching Restore from within it). Does that work?

But it's also interesting that Qube Manager crashes without showing any error. No idea how to fix that part. @marmarta

[andrewdavidwong]

I forgot to mention to restart Qube Manager after applying the fix (if you're launching Restore from within it).

Ah, I hadn't restarted it. I can't easily test again right now, but that's probably what it was.

[qubesos-bot]

Automated announcement from builder-github

The package core-admin-client has been pushed to the r4.2 testing repository for the Debian template.
To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing bookworm-testing (or appropriate equivalent for your template version), then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package core-admin-client has been pushed to the r4.2 testing repository for the CentOS centos-stream8 template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r4.2-current-testing

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package core-admin-client has been pushed to the r4.2 testing repository for the Debian template.
To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing bullseye-testing (or appropriate equivalent for your template version), then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The component core-admin-client (including package core-admin-client) has been pushed to the r4.2 testing repository for the Fedora template.
To test this update, please install it with the following command:

sudo dnf update --enablerepo=qubes-vm-r4.2-current-testing

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The component core-admin-client (including package core-admin-client) has been pushed to the r4.2 testing repository for the Fedora template.
To test this update, please install it with the following command:

sudo dnf update --enablerepo=qubes-vm-r4.2-current-testing

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package qubes-core-admin-client_4.1.27-1 has been pushed to the r4.1 testing repository for the Debian template.
To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing buster-testing (or appropriate equivalent for your template version), then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package core-admin-client has been pushed to the r4.1 stable repository for the CentOS centos-stream8 template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package qubes-core-admin-client_4.1.27-1+deb10u1 has been pushed to the r4.1 stable repository for the Debian template.
To install this update, please use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The component core-admin-client (including package python3-qubesadmin-4.1.27-1.fc32) has been pushed to the r4.1 stable repository for dom0.
To install this update, please use the standard update command:

sudo qubes-dom0-update

Or update dom0 via Qubes Manager.

Changes included in this update