Tool for performing multiple updates

[tasket]

Qubes needs a way to handle multiple OS updates automatically since managing these processes individually is disruptive to the user.

So far I have a python script qubes-multi-update that can handle Linux-based VMs which includes PVM templates, standalones and dom0:

https://github.com/tasket/Qubes-scripts/blob/master/qubes-multi-update

It is meant to be an improvement on various other update scripts seen online; it tries to intelligently handle VM states and status (such as 'updates available') and can optionally TRIM templates and standalones.

---

Related issues:

https://gist.github.com/andrewdavidwong/d0b109186de65835255d467ae103c289
https://gist.github.com/JimmyAx/818bcf11a14e85531516ef999c8c5765
#1378
#1760
#2150

[marmarek]

Can be handled by salt:
/srv/user_salt/update.sls:

update:
  pkg.uptodate:
    - refresh: True
    - dist_upgrade: True

Then add to /srv/user_salt/top.sls:

user:
  qubes:type:template:
    - match: pillar
    - update

And launch: qubesctl --all --show-output state.highstate

Maybe we should ship such configuration by default?

[tasket]

Does this handle only templates? And does it leave them running?

[marmarek]

Does this handle only templates?

In this case - yes, because I've specified it that way (qubes:type:template). You can also extend this to standalone VMs (qubes:type:standalone), or even AppVMs (qubes:type:app). The last one make very little sense, though.

And does it leave them running?

It will leave them in the same state - if template was running, it will be left as such, otherwise it will be stopped after applying state.

BTW, if you want to apply just this one action, you can do that in one line, without any additional file:

qubesctl --skip-dom0 --templates --show-output state.single pkg.uptodate refresh=True dist_upgrade=True

More info: https://www.qubes-os.org/doc/salt/

[tasket]

Since this appears to be pulling from Qubes API, it seems like ( qubes:type:updateable ) should be possible?

I'm reading the salt docs now.

[marmarek]

It should be easy to add such option (in practice: qubes:updateable:true), but it doesn't exist right now.

[tasket]

@marmarek This saltstack method tries to launch a VM for each template all at the same time. The result was spurious insufficient memory errors.

I also get this prompt for each template:

• Do you allow domain "disp-mgmt-fedora-24-minimal" to execute qubes.VMAuth operation on the domain "dom0"?

This is from update command not running as root with the vm-sudo config.

[marmarek]

On Sat, Apr 08, 2017 at 08:12:35AM -0700, tasket wrote: @marmarek This saltstack method tries to launch a VM for each template all at the same time. The result was spurious insufficient memory errors.

It execute 4 of them simultaneously. Currently the value is hardcoded here: https://github.com/QubesOS/qubes-mgmt-salt/blob/master/qubessalt/__init__.py#L191 But it shouldn't be hard to add an option.

I also get this prompt for each template: * Do you allow domain "disp-mgmt-fedora-24-minimal" to execute qubes.VMAuth operation on the domain "dom0"? This is from update command not running as root with the vm-sudo config.

Yes, salt and related scripts do use sudo. If you don't want those prompts, probably the easiest way is to add "allow" rules (before default ask) for those domains to /etc/qubes-rpc/policy/qubes.VMAuth. This applies to disp-mgmt-* VMs and templates itself. VM names disp-mgmt-* are not used anywhere else, so this shouldn't have side effects.

…

-- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing?

[tasket]

Currently having doubts about using a config management engine (salt) for routine maintenance tasks. The more I learn, the more it seems like a poor fit.

Adding some nice-to-have features to qubes-multi-update, but will definitely use saltstack for VPN setup, vm-sudo setup and more...

[J316]

@marmarek

Maybe we should ship such configuration by default?

Absolutely! It's a pain to upgrade all VMs one by one, a upgrade all VM button would be much, much appriciated.

[jpouellet]

Feature request: The ability to manually select some VMs for exclusion from bulk updates.

I have a couple VMs running intentionally outdated versions of stuff which is required to get some proprietary stuff running properly. Sure, I could probably just pin the specific packages necessary in my package manager, but this is not a workable solution for people who are not familiar with package managers and just want to say "I know this is vulnerable and terrible, but please don't touch it. It's not networked and I keep my data away from it, it's okay. I just need it to work."

[tasket]

@jpouellet Done :)

[jpouellet]

I had in mind more of some persistent config on a VM which says to exclude it (like existing options for "include in backups by default", "automatically start on boot", etc.), which you could set and forget, rather than some cmd line option which you need to remember to pass each time.

Looking forward, this seems to be exactly the kind of thing core3's "features" (essentially a dict of auxiliary metadata on each VM) enables cleanly.

[tasket]

No doubt, that would make it easy. But while we're running 3.x its good to have something that works.

And although there is no current way to tag or assign other metadata to VMs, keeping a separate list just for the purpose could also enable the 'set and forget' use case.

[jpouellet]

Indeed. Thanks for working on this :)

[tlaurion]

@marmarek
qubesctl --skip-dom0 --templates --show-output state.single pkg.uptodate refresh=True dist_upgrade=True

Gives a bunch of "SKIP (nothing to do)" while most of the templates require updates.

[marmarek]

It's #2451 - there is optimization to not start VMs not mentioned in top file. Until linked issue is resolved, you can use placeholder state, for example:

base:
  qubes:type:template:
    - match: pillar
    - topd

Put the above in /srv/salt/placeholder.top and enable with qubesctl top.enable placeholder.

[conorsch]

Confirming @tlaurion's "SKIP" results, even after implementing @marmarek's suggested workaround:

[user@dom0 ~]$ sudo cat /srv/salt/placeholder.top
base:
  qvm:type:template:
    - match: pillar
    - topd
[user@dom0 ~]$ sudo qubesctl top.enable placeholder
local:
    ----------
    placeholder.top:
        ----------
        status:
            unchanged
[user@dom0 ~]$ qubesctl --skip-dom0 --templates --show-output state.single pkg.uptodate refresh=True dist_upgrade=True
fedora-27: SKIP (nothing to do)
fedora-26: SKIP (nothing to do)
fedora-28: SKIP (nothing to do)
whonix-ws: SKIP (nothing to do)
whonix-gw: SKIP (nothing to do)
debian-9: SKIP (nothing to do)
[user@dom0 ~]$ qvm-start fedora-27 fedora-28
[user@dom0 ~]$ qubesctl --skip-dom0 --templates --show-output state.single pkg.uptodate refresh=True dist_upgrade=True
fedora-27: SKIP (nothing to do)
fedora-26: SKIP (nothing to do)
fedora-28: SKIP (nothing to do)
whonix-ws: SKIP (nothing to do)
whonix-gw: SKIP (nothing to do)
debian-9: SKIP (nothing to do)
[user@dom0 ~]$ qubesctl --skip-dom0 --templates --show-output pkg.upgrade
fedora-27: SKIP (nothing to do)
fedora-26: SKIP (nothing to do)
fedora-28: SKIP (nothing to do)
whonix-ws: SKIP (nothing to do)
whonix-gw: SKIP (nothing to do)
debian-9: SKIP (nothing to do)
[user@dom0 ~]$

To confirm it's not that there's really nothing to do, I manually started debian-9, and confirmed in a terminal window that there are indeed updates that can be installed there. I also tried running sudo qubesctl state.highstate in dom0, but the pkg.upgrade command still emits SKIP messages, regardless of whether the target TemplateVMs are started or halted.

[marmarek]

Oh, it should be qubes:type:template, not qvm:type:template.

[tlaurion]

@marmarek Still "SKIP (nothing to do)" warnings with qubes:type:template in /srv/salt/placeholder.top

When will a solution to upgrade all templates from GUI happens?

[unman]

@marmarek
The salt call works fine, and is how I update my systems
Is all that is needed here a separate GUI tool, a menu item from (gasp) Qube Manager, or an additional R-click menu item, "Update all qubes" ?

[conorsch]

Yes, with @marmarek 's tweak to the template I posted above, this is now a reliable method to "update all VMs." Still hoping to make this automatic, presumably via cron.

[marmarta]

There's a pull request for a gui tool for that - QubesOS/qubes-desktop-linux-manager#34 . Does this mean this issue is solved, @marmarek ? We have a CLI way to update VMs (which can be added to cron) and a gui tool for users to click through updates when they become available.

[andrewdavidwong]

Qubes needs a way to handle multiple OS updates automatically since managing these processes individually is disruptive to the user.

It looks like the decision about the "automatic" part was won't do.

Oops, missed this part:

We have a CLI way to update VMs (which can be added to cron)

Looks like it's considered done.