tar fails to extract root.img from template.rpm

[sajkbflksadbfkasdjf]

Qubes OS release

4.1 with all updates applied (subscribed to testing)

Brief summary

Qubes fails to install templates because the root.img extraction fails

Steps to reproduce

Install template (I installed from commandline with dnf install ./qubes-template-...

Expected behavior

Template is installed correctly

Actual behavior

tar: root.img: Cannot seek to 212860928: Broken pipe
tar: Exiting with failure status due to previous errors
Traceback (most recent call last):
  File "/bin/qvm-template-postprocess", line 5, in <module>
    sys.exit(main())
  File "/usr/lib/python3.8/site-packages/qubesadmin/tools/qvm_template_postprocess.py", line 431, in main
    loop.run_until_complete(post_install(args))
  File "/usr/lib64/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/lib/python3.8/site-packages/qubesadmin/tools/qvm_template_postprocess.py", line 297, in post_install
    import_root_img(vm, args.dir)
  File "/usr/lib/python3.8/site-packages/qubesadmin/tools/qvm_template_postprocess.py", line 109, in import_root_img
    raise qubesadmin.exc.QubesException('root.img extraction failed')
qubesadmin.exc.QubesException: root.img extraction failed
warning: %post(qubes-template-archlinux-4.0.6-202201062107.noarch) scriptlet failed, exit status 1

Error in POSTIN scriptlet in rpm package qubes-template-archlinux
  Running scriptlet: qubes-template-archlinux-4.0.6-202201062107.noarch                   2/2 
  Cleanup          : qubes-template-archlinux-4.0.6-202201062107.noarch                   2/2 
  Verifying        : qubes-template-archlinux-4.0.6-202201062107.noarch                   1/2 
  Verifying        : qubes-template-archlinux-4.0.6-202201062107.noarch                   2/2 ```
This has been tested with self-built arch template and the gentoo template I downloaded manually and moved to dom0 to install. Can't confirm it happens with "normal" installs from qvm-template too.

[marmarek]

Definitely try with qvm-template. But also, check if you have enough disk space - you need few GB (the size of the template, uncompressed) in dom0 root filesystem.

[sajkbflksadbfkasdjf]

qvm-template doesn't accept my local rpms (some signature check failed), pulling from the repo isn't possible right now due to #7170
Disk space isn't the problem, df -h shows me 11 GB of free space for /dev/mapper/qubes_dom0-root

[marmarek]

qvm-template doesn't accept local rpms

It does, try qvm-template install --nogpgcheck

[sajkbflksadbfkasdjf]

root.img extraction fails too, the exception qubesadmin.exc.QubesException: root.img extraction failed appears in the python traceback. After that the qvm-template returns exit status 1 and quits.

[sajkbflksadbfkasdjf]

traceback for dnf:

tar: root.img: Cannot seek to 212860928: Broken pipe
tar: Exiting with failure status due to previous errors
Traceback (most recent call last):
  File "/bin/qvm-template-postprocess", line 5, in <module>
    sys.exit(main())
  File "/usr/lib/python3.8/site-packages/qubesadmin/tools/qvm_template_postprocess.py", line 431, in main
    loop.run_until_complete(post_install(args))
  File "/usr/lib64/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/lib/python3.8/site-packages/qubesadmin/tools/qvm_template_postprocess.py", line 297, in post_install
    import_root_img(vm, args.dir)
  File "/usr/lib/python3.8/site-packages/qubesadmin/tools/qvm_template_postprocess.py", line 109, in import_root_img
    raise qubesadmin.exc.QubesException('root.img extraction failed')
qubesadmin.exc.QubesException: root.img extraction failed
warning: %post(qubes-template-archlinux-4.0.6-202201062107.noarch) scriptlet failed, exit status 1

Error in POSTIN scriptlet in rpm package qubes-template-archlinux
  Running scriptlet: qubes-template-archlinux-4.0.6-202201062107.noarch                   2/2 
  Cleanup          : qubes-template-archlinux-4.0.6-202201062107.noarch                   2/2 
  Verifying        : qubes-template-archlinux-4.0.6-202201062107.noarch                   1/2 
  Verifying        : qubes-template-archlinux-4.0.6-202201062107.noarch                   2/2 

and traceback for qvm-template:

  File "/usr/bin/qvm-template-postprocess", line 5, in <module>
    sys.exit(main())
  File "/usr/lib/python3.8/site-packages/qubesadmin/tools/qvm_template_postprocess.py", line 431, in main
    loop.run_until_complete(post_install(args))
  File "/usr/lib64/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/lib/python3.8/site-packages/qubesadmin/tools/qvm_template_postprocess.py", line 297, in post_install
    import_root_img(vm, args.dir)
  File "/usr/lib/python3.8/site-packages/qubesadmin/tools/qvm_template_postprocess.py", line 109, in import_root_img
    raise qubesadmin.exc.QubesException('root.img extraction failed')
qubesadmin.exc.QubesException: root.img extraction failed
ERROR: Command '['qvm-template-postprocess', '--really', '--no-installed-by-rpm', 'post-install', 'gentoo', '/var/tmp/tmpwbvffcui/var/lib/qubes/vm-templates/gentoo']' returned non-zero exit status 1.

[marmarek]

Are you sure the template rpm isn't simply truncated? Try rpm -Kv on it.

[marmarek]

Or not rpm file itself, but the root.img file inside - see rpm -qpvl on it, and compare file sizes with what you'd expect (root.img may be in several parts there, you need to sum them).

[sajkbflksadbfkasdjf]

The sum of the (8) parts is ~960MB big (I think, wasn't in human-readable format). The first command returns a few signature OK statements.
I can append the results if you want but sadly I'm not yet knowledgeable enough about rpms to interpret the output

[fepitre]

traceback for dnf:

tar: root.img: Cannot seek to 212860928: Broken pipe
tar: Exiting with failure status due to previous errors
Traceback (most recent call last):
  File "/bin/qvm-template-postprocess", line 5, in <module>
    sys.exit(main())
  File "/usr/lib/python3.8/site-packages/qubesadmin/tools/qvm_template_postprocess.py", line 431, in main
    loop.run_until_complete(post_install(args))
  File "/usr/lib64/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/lib/python3.8/site-packages/qubesadmin/tools/qvm_template_postprocess.py", line 297, in post_install
    import_root_img(vm, args.dir)
  File "/usr/lib/python3.8/site-packages/qubesadmin/tools/qvm_template_postprocess.py", line 109, in import_root_img
    raise qubesadmin.exc.QubesException('root.img extraction failed')
qubesadmin.exc.QubesException: root.img extraction failed
warning: %post(qubes-template-archlinux-4.0.6-202201062107.noarch) scriptlet failed, exit status 1

Error in POSTIN scriptlet in rpm package qubes-template-archlinux
  Running scriptlet: qubes-template-archlinux-4.0.6-202201062107.noarch                   2/2 
  Cleanup          : qubes-template-archlinux-4.0.6-202201062107.noarch                   2/2 
  Verifying        : qubes-template-archlinux-4.0.6-202201062107.noarch                   1/2 
  Verifying        : qubes-template-archlinux-4.0.6-202201062107.noarch                   2/2 

and traceback for qvm-template:

  File "/usr/bin/qvm-template-postprocess", line 5, in <module>
    sys.exit(main())
  File "/usr/lib/python3.8/site-packages/qubesadmin/tools/qvm_template_postprocess.py", line 431, in main
    loop.run_until_complete(post_install(args))
  File "/usr/lib64/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/lib/python3.8/site-packages/qubesadmin/tools/qvm_template_postprocess.py", line 297, in post_install
    import_root_img(vm, args.dir)
  File "/usr/lib/python3.8/site-packages/qubesadmin/tools/qvm_template_postprocess.py", line 109, in import_root_img
    raise qubesadmin.exc.QubesException('root.img extraction failed')
qubesadmin.exc.QubesException: root.img extraction failed
ERROR: Command '['qvm-template-postprocess', '--really', '--no-installed-by-rpm', 'post-install', 'gentoo', '/var/tmp/tmpwbvffcui/var/lib/qubes/vm-templates/gentoo']' returned non-zero exit status 1.

I don't understand. I see that you try to install archlinux template where as there is post-install for gentoo. Can you please check that you have enough place inside dom0 like at least 7 or 8 GB.

[sajkbflksadbfkasdjf]

Oh, I see what you mean... I did try that too, but that might have been a failure in my build setup so I downloaded gentoo from the repos, manually transferred to dom0 and tried to install. df -h gives me ~9GB of free space, so I don't think the problem's about disk space

[sajkbflksadbfkasdjf]

With #7170 fixed I tried to use qvm-template to download the gentoo template. This works fine, but the issue with the root.img persists.

[sajkbflksadbfkasdjf]

(the download works fine, the install fails as usual)

[sajkbflksadbfkasdjf]

any updates on the source of this issue? otherwise I might try a reinstall sometime soon, to ascertain whether it's a software problem...

[DemiMarie]

Can you successfully install other templates, such as Fedora 35?

[sajkbflksadbfkasdjf]

Nope, issue confirmed with fedora-33

[sajkbflksadbfkasdjf]

any updates? I'd be happy to test stuff if it helps w/ troubleshooting

[unman]

Are you OP? Are there *any* templates that work for you? If you download a template rpm and install it manually, does that work for you? If so, remove the template and try installing it again with qvm-template. Obviously use one of the templates that didn't work - I would use Fedora-33, since we know that package has had extensive testing.

[brendanhoar]

qvm-template doesn't accept my local rpms (some signature check failed), pulling from the repo isn't possible right now due to #7170
Disk space isn't the problem, df -h shows me 11 GB of free space for /dev/mapper/qubes_dom0-root

What is the output of:

sudo lvs|grep -v vm-

?

B

[brendanhoar]

Looks like the rpm file is dropped into /home/admin/cache/qvm-template/ and then expanded into /var/tmp/tmp<random_8_alphanumeric>/var/lib/qubes/vm-templates/<template_shortname>

E.g. with centos-8-minimal the latter directory was:

/var/tmp/tmpy9p2sc4c/var/lib/qubes/vm-templates/centos-8-minimal/

Both paths should be on the filesystem on /dev/mapper/qubes_dom0-root

It's possible that watching the above two directories, as well as keeping an eye on df -h output and the lvs command output from the previous post might help to narrow down what is going on?

Lastly that bytecount in the OP is exactly 203MB.

B

PS: Before qvm-template, there was always a concern about sys-firewall having enough system storage for large template or multiple templates. It appears with R4.1, data is instead piped into dom0 from the curl running in sys-firewall and ends up being written under the /home/admin/.cache hierarchy instead. If that directory were, say, moved to a tmpfs mount, that could be a reason for failure. I've seen pages suggesting putting that in tmpfs, but as we all recall, tmpfs is basically a ram disk.

[sajkbflksadbfkasdjf]

Are you OP? Are there any templates that work for you? If you download a template rpm and install it manually, does that work for you? If so, remove the template and try installing it again with qvm-template. Obviously use one of the templates that didn't work - I would use Fedora-33, since we know that package has had extensive testing.

I tried manual install, actually the arch template is self-built. I tried removing a template I didn't need any longer and reinstalling it, but that failed as well.

[sajkbflksadbfkasdjf]

What is the output of:

sudo lvs|grep -v vm-

?

B

LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
root qubes_dom0 Vwi-aotz-- 20.00g root-pool 62.48
root-pool qubes_dom0 twi-aotz-- 24.00g 52.07 35.43
swap qubes_dom0 -wi-ao---- <3.95g

[sajkbflksadbfkasdjf]

well, I think with 4.1 out I'll probably try a reinstall, does anyone need any more infos before that?

[BEBF738VD]

I'm having the same issue. I've tried with multiple templates (fedora, debian, whonix) and they all end with the same exception error: qubesadmin.exc.QubesException: root.img extraction failed

I have tried on 4.1 (as downloaded on the official website) as well as 4.1 kernel-latest provided by fepitre (right after a complete reinstall).

Finally, I tried with sudo qubes-dom0-update and also qvm-template and qvm-template-gui.

df -h /: Size 20G, Used 8.7G, Avail 9.9G
rpm -Kv shows that digests are OK.

The output of rpm -qpvl is included below. marmarek mentioned that root.img may be divided into parts; in my case the size of root.img is 0 while root.img.part.00 is 984524800. I'm not sure if the former is supposed to be completely empty.

drwxrws---    2 root     qubes                       0 Aug 19  2021 /var/lib/qubes/vm-templates/debian-11-minimal
drwxrwxr-x    2 root     qubes                       0 Aug 19  2021 /var/lib/qubes/vm-templates/debian-11-minimal/apps
drwxrwxr-x    2 root     qubes                       0 Aug 19  2021 /var/lib/qubes/vm-templates/debian-11-minimal/apps.tempicons
drwxrwxr-x    2 root     qubes                       0 Aug 19  2021 /var/lib/qubes/vm-templates/debian-11-minimal/apps.templates
-rw-rw----    1 root     qubes                       0 Aug 19  2021 /var/lib/qubes/vm-templates/debian-11-minimal/clean-volatile.img.tar
-rw-rw-r--    1 root     qubes                      21 Aug 19  2021 /var/lib/qubes/vm-templates/debian-11-minimal/netvm-whitelisted-appmenus.list
-rw-rw----    1 root     qubes                       0 Aug 19  2021 /var/lib/qubes/vm-templates/debian-11-minimal/private.img
-rw-rw----    1 root     qubes                       0 Aug 19  2021 /var/lib/qubes/vm-templates/debian-11-minimal/root.img
-rw-rw----    1 root     qubes               984524800 Aug 19  2021 /var/lib/qubes/vm-templates/debian-11-minimal/root.img.part.00
-rw-rw-r--    1 root     qubes                       0 Aug 19  2021 /var/lib/qubes/vm-templates/debian-11-minimal/template.conf
-rw-rw-r--    1 root     qubes                      21 Aug 19  2021 /var/lib/qubes/vm-templates/debian-11-minimal/vm-whitelisted-appmenus.list
-rw-rw----    1 root     qubes                       0 Aug 19  2021 /var/lib/qubes/vm-templates/debian-11-minimal/volatile.img
-rw-rw-r--    1 root     qubes                      21 Aug 19  2021 /var/lib/qubes/vm-templates/debian-11-minimal/whitelisted-appmenus.list

I attempted to seek help on the forums with not much luck...
https://forum.qubes-os.org/t/cannot-install-templates-qvm-template-postprocess-error-root-img-extraction-failed-even-after-complete-qubesos-reinstall/11361/1

@sajkbflksadbfkasdjf sorry for tagging you, but have you found a solution?

[sajkbflksadbfkasdjf]

nope, sadly I had to reinstall and even now the issue persists. That's the reason I can't install fedora-35 on my machine...

[sajkbflksadbfkasdjf]

btw, issue confirmed for fully updated 4.1

[BEBF738VD]

Just an update to confirm that this issue still persists. I recently tried installing several templates with qvm-template as well as unman 's bookworm-minimal (with dnf install), but they all failed with the same error mentioned here.

I suspect the problem is insufficient disk space in dom0. Can you provide the output of df in dom0?

@DemiMarie the template is less than 1G and my /dev/mapper/qubes_dom0-root has 20G available. I ran df -h at .2 second intervals and it never came close to filling up the space.

I also ran journalctl through the entire process and no errors were thrown.

Any suggestion is greatly appreciated...

[BEBF738VD]

New case reported on the forums: https://forum.qubes-os.org/t/cant-recover-any-backup-and-and-cant-install-templates/14668

They reported 77G of free disk space in dom0

[DemiMarie]

@BEBF738VD This is definitely bad, but I am not sure how to debug this problem without either being able to reproduce the problem, or having shell access to a machine that has the problem.

[BEBF738VD]

@DemiMarie thank you very much for taking interest in this issue. I would be happy to run on my machine any test that you suggest, at your convenience.

[BEBF738VD]

Does the available space in the various tmpfs, and in xenstore matter? Or is it only /dev/mapper/qubes_dom0-root I have to look at?

[DemiMarie]

@DemiMarie thank you very much for taking interest in this issue. I would be happy to run on my machine any test that you suggest, at your convenience.

My first thought is to run strace on the tar command and see where it fails. This will produce a lot of output, so use -o to save it to a file somewhere.

Does the available space in the various tmpfs, and in xenstore matter? Or is it only /dev/mapper/qubes_dom0-root I have to look at?

Only the last one.

[sajkbflksadbfkasdjf]

https://github.com/sajkbflksadbfkasdjf/uploads/releases/download/uploads/out.txt

[sajkbflksadbfkasdjf]

I have run strace on the whole command (qvm-template install gentoo-minimal), hope this helps

[BEBF738VD]

I have found a solution!
When setting up Qubes, I removed vim. As it turns out, it is somehow necessary to the template script:

reinstalling vim fixed it for me.

I'm now able to install any template I wish.
Hope this helps.

[sajkbflksadbfkasdjf]

Weird. I have uninstalled vim but hard-linked neovim to vim...

[marmarek]

I guess this is related:

$ rpm -qf /usr/bin/xxd
vim-common-8.2.2787-1.fc32.x86_64

[sajkbflksadbfkasdjf]

Thanks. Then I'll try to only install vim-common, maybe that'll work. If so, I'd go ahead and change the dependencies of qvm-template to include this...

[sajkbflksadbfkasdjf]

Seems correct, vim-common solves the problem

[BEBF738VD]

Fantastic. Cheers everyone for pitching in.

[qubesos-bot]

Automated announcement from builder-github

The component core-admin (including package qubes-core-dom0-4.1.30-1.fc32) has been pushed to the r4.1 testing repository for dom0.
To test this update, please install it with the following command:

sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The component core-admin (including package qubes-core-dom0-4.1.30-1.fc32) has been pushed to the r4.1 stable repository for dom0.
To install this update, please use the standard update command:

sudo qubes-dom0-update

Or update dom0 via Qubes Manager.

Changes included in this update