Some hosts don't have the /etc/cron.daily/apt file

[metakermit]

This /etc/cron.daily/apt file is supposed to trigger the unattended-upgrades command and it is missing in some cloud VMs. There was a lot written about this issue (on Stack Overflow, again, ServerFault, …).

This was solved in a related unattended_upgrades Ansible playbook, so maybe their solution would work here too.

[drybjed]

Do you have access to a system that is affected by this? Can you check if the script is diverted (dpkg-divert --list or just moved directly?

Can you check the md5sum of the moved file and compare it with the checksum in dpkg-query -s apt output?

[metakermit]

It's an Ubuntu VM on AWS. I don't see the file in dpkg-divert:

ubuntu@myhost:~$ dpkg-divert --list
diversion of /usr/bin/pod2latex to /usr/bin/pod2latex.bundled by libpod-latex-perl
diversion of /usr/share/man/man1/pod2latex.1.gz to /usr/share/man/man1/pod2latex.bundled.1.gz by libpod-latex-perl
diversion of /usr/sbin/grub-set-default to /usr/sbin/grub-set-default.real by grub-legacy-ec2
diversion of /usr/share/man/man1/sh.1.gz to /usr/share/man/man1/sh.distrib.1.gz by dash
diversion of /usr/share/vim/vim74/doc/tags to /usr/share/vim/vim74/doc/tags.vim-tiny by vim-runtime
diversion of /usr/share/vim/vim74/doc/help.txt to /usr/share/vim/vim74/doc/help.txt.vim-tiny by vim-runtime
diversion of /etc/init/ureadahead.conf to /etc/init/ureadahead.conf.disabled by cloud-init
diversion of /usr/share/dbus-1/system-services/org.freedesktop.systemd1.service to /usr/share/dbus-1/system-services/org.freedesktop.systemd1.service.systemd by systemd-shim
local diversion of /etc/apt/apt.conf.d/50unattended-upgrades to /etc/apt/apt.conf.d/50unattended-upgrades.dpkg-divert
diversion of /bin/sh to /bin/sh.distrib by dash
diversion of /usr/share/initramfs-tools/hooks/klibc to /usr/share/initramfs-tools/hooks/klibc^i-t by klibc-utils

Actually, looking at the dpkg-query -s apt output, I think that maybe the /etc/cron.daily/apt file was replaced by /etc/cron.daily/apt-compat in the package itself:

ubuntu@myhost:~$ dpkg-query -s apt
Package: apt
Status: install ok installed
Priority: important
Section: admin
Installed-Size: 3305
Maintainer: Ubuntu Developers <[email protected]>
Architecture: amd64
Version: 1.2.12~ubuntu16.04.1
Replaces: bash-completion (<< 1:2.1-4.2+fakesync1), manpages-it (<< 2.80-4~), manpages-pl (<< 20060617-3~), openjdk-6-jdk (<< 6b24-1.11-0ubuntu1~), sun-java5-jdk (>> 0), sun-java6-jdk (>> 0)
Depends: libapt-pkg5.0 (>= 1.2.2), libc6 (>= 2.15), libgcc1 (>= 1:3.0), libstdc++6 (>= 5.2), init-system-helpers (>= 1.18~), ubuntu-keyring, gpgv | gpgv2, gnupg | gnupg2, adduser
Suggests: aptitude | synaptic | wajig, dpkg-dev (>= 1.17.2), apt-doc, python-apt
Breaks: apt-utils (<< 1.1.3), manpages-it (<< 2.80-4~), manpages-pl (<< 20060617-3~), openjdk-6-jdk (<< 6b24-1.11-0ubuntu1~), sun-java5-jdk (>> 0), sun-java6-jdk (>> 0)
Conffiles:
 /etc/apt/apt.conf.d/01-vendor-ubuntu 5232396660502461fc834c0a1229dbe4
 /etc/apt/apt.conf.d/01autoremove 0b1391c01d75f95fa4ea5ac01219b515
 /etc/cron.daily/apt-compat c3446ad30001663bf5ad39ba59fae463
 /etc/kernel/postinst.d/apt-auto-removal dfebd32159f522e4d58f66cda864224e
 /etc/logrotate.d/apt 179f2ed4f85cbaca12fa3d69c2a4a1c3
Description: commandline package manager
 This package provides commandline tools for searching and
 managing as well as querying information about packages
 as a low-level access to all features of the libapt-pkg library.
 .
 These include:
  * apt-get for retrieval of packages and information about them
    from authenticated sources and for installation, upgrade and
    removal of packages together with their dependencies
  * apt-cache for querying available information about installed
    as well as installable packages
  * apt-cdrom to use removable media as a source for packages
  * apt-config as an interface to the configuration settings
  * apt-key as an interface to manage authentication keys
Original-Maintainer: APT Development Team <[email protected]>
ubuntu@myhost:~$ md5sum /etc/cron.daily/apt-compat
c3446ad30001663bf5ad39ba59fae463  /etc/cron.daily/apt-compat

[drybjed]

So in that case the solution proposed in the other role probably wouldn't work... Is the compat script comparable to the normal Debian version, ie. does it at least run unattended-upgrades? I suppose that the role could provide an additional small script that would manage the upgrades if the apt-compat does not do it.

Any differences in the unattended-upgrades package on that host?

[drybjed]

BTW, I just checked an Ubuntu 16.04 installation on one of my LXC containers and it's the same file looking at the checksum. So it's not an AWS issue, but an Ubuntu issue, it seems, at least that one release.

[drybjed]

Looks like the original script has been moved to /usr/lib/apt/apt.systemd.daily, can you check if you have that present and does it show up in the logs? The apt-compat script is now a compatibility layer between systemd timers and cron.

[metakermit]

The apt-compat script has a line exec /usr/lib/apt/apt.systemd.daily which in turn seems to run the script:

ubuntu@myhost:~$ cat /usr/lib/apt/apt.systemd.daily | grep unattended
#  - Run the "unattended-upgrade" security upgrade script
#    Requires the package "unattended-upgrades" and will write
#    a log in /var/log/unattended-upgrades
if which unattended-upgrade >/dev/null 2>&1 && check_stamp $UPGRADE_STAMP $UnattendedUpgradeInterval; then
    if unattended-upgrade $XUUPOPT; then
    debug_echo "unattended-upgrade (success)"
    debug_echo "unattended-upgrade (error)"
    debug_echo "unattended-upgrade (not run)"

[drybjed]

Check the status of the systemd services:

systemctl status apt-daily.service
systemctl status apt-daily.timer

This should tell you at least if the service is executed each day.

[metakermit]

I'm checking my syslog logs, but I see no trace that this apt-compat script was called … The apt-daily service seems to have been called:

ubuntu@myhost:~$ systemctl status apt-daily.service
● apt-daily.service - Daily apt activities
   Loaded: loaded (/lib/systemd/system/apt-daily.service; static; vendor preset: enabled)
   Active: inactive (dead) since Tue 2016-10-18 23:32:02 AEDT; 2h 15min ago
     Docs: man:apt(8)
 Main PID: 19097 (code=exited, status=0/SUCCESS)

ubuntu@myhost:~$ systemctl status apt-daily.timer
● apt-daily.timer - Daily apt activities
   Loaded: loaded (/lib/systemd/system/apt-daily.timer; enabled; vendor preset: enabled)
   Active: active (waiting) since Wed 2016-10-19 00:29:03 AEDT; 1h 18min ago

Oct 19 00:29:03 myhost systemd[1]: Started Daily apt activities.

The upgrades still seem to not have been installed, though:

28 packages can be updated.
5 updates are security updates.

Maybe I should give it another day to see if the necessary processes will kick in.

[drybjed]

Yeah, that would be a good way to check if it works unattended. Leave it for a bit and check back later.

[metakermit]

OK, after leaving everything to run for another day it seems that the unattended-upgrades program is actually run periodically via the apt-compat script, so this issue can be closed from that perspective.

My security upgrades are still not being installed, though:

2016-10-19 02:17:48,121 INFO No packages found that can be upgraded unattended and no pending auto-removals
2016-10-19 16:39:43,958 INFO Initial blacklisted packages:
2016-10-19 16:39:43,959 INFO Initial whitelisted packages:
2016-10-19 16:39:43,959 INFO Starting unattended upgrades script
2016-10-19 16:39:43,959 INFO Allowed origins are: ['o=Ubuntu,n=xenial,l=xenial-security']
2016-10-19 16:39:45,954 INFO No packages found that can be upgraded unattended and no pending auto-removals

After running the more verbose command unattended-upgrades -d I see this output (shortened):

Initial blacklisted packages:
Initial whitelisted packages:
Starting unattended upgrades script
Allowed origins are: ['o=Ubuntu,n=xenial,l=xenial-security']
Checking: distro-info-data ([<Origin component:'main' archive:'xenial-updates' origin:'Ubuntu' label:'Ubuntu' site:'ap-southeast-2.ec2.archive.ubuntu.com' isTrusted:True>])
Checking: initramfs-tools ([<Origin component:'main' archive:'xenial-updates' origin:'Ubuntu' label:'Ubuntu' site:'ap-southeast-2.ec2.archive.ubuntu.com' isTrusted:True>])
Checking: initramfs-tools-bin ([<Origin component:'main' archive:'xenial-updates' origin:'Ubuntu' label:'Ubuntu' site:'ap-southeast-2.ec2.archive.ubuntu.com' isTrusted:True>])
...
Checking: tzdata ([<Origin component:'main' archive:'xenial-updates' origin:'Ubuntu' label:'Ubuntu' site:'ap-southeast-2.ec2.archive.ubuntu.com' isTrusted:True>, <Origin component:'main' archive:'xenial-security' origin:'Ubuntu' label:'Ubuntu' site:'security.ubuntu.com' isTrusted:True>])
Checking: update-manager-core ([<Origin component:'main' archive:'xenial-updates' origin:'Ubuntu' label:'Ubuntu' site:'ap-southeast-2.ec2.archive.ubuntu.com' isTrusted:True>])
pkgs that look like they should be upgraded:
Fetched 0 B in 0s (0 B/s)
fetch.run() result: 0
blacklist: []
whitelist: []
No packages found that can be upgraded unattended and no pending auto-removals

But the OS is notifying me that I do indeed have some security upgrades pending.

Welcome to Ubuntu 16.04.1 LTS (GNU/Linux 4.4.0-42-generic x86_64)
...
29 packages can be updated.
5 updates are security updates.

After running some suggested commands to get just the security updates packages, it seems that at least packages like tzdata should be handled by unattended upgrades, no?

ubuntu@myhost:~$ sudo sh -c 'grep ^deb /etc/apt/sources.list |
      grep security > /etc/apt/sources.security.only.list'
ubuntu@myhost:~$ apt-get -s dist-upgrade -o Dir::Etc::SourceList=/etc/apt/sources.security.only.list -o Dir::Etc::SourceParts=/dev/null  |      grep "^Inst" | awk -F " " {'print $2'}
tzdata
libswresample-ffmpeg1
libavformat-ffmpeg56
libavcodec-ffmpeg56
libavutil-ffmpeg54

All of my debops settings are default, except that I enabled automatic rebooting.

unattended_upgrades__auto_reboot: True
unattended_upgrades__auto_reboot_time: "02:00"

[drybjed]

Can you check what is in the diverted configuration file (/etc/apt/apt.conf.d/50unattended-upgrades.dpkg-divert) in the Allowed-Origins and/or Origins-Pattern sections?

I see that on Debian Jessie the package uses Origins-Pattern, but on Ubuntu Xenial it's Allowed-Origins. I'm not sure why, and if the Ubuntu version supports Origins-Pattern at all. You could try and replace the diverted version (or just set unattended_upgrades__enabled: False which should do that as well) and see if that changes things.

[metakermit]

This is what I have in /etc/apt/apt.conf.d/50unattended-upgrades.dpkg-divert:

Unattended-Upgrade::Origins-Pattern {
      "origin=Ubuntu,archive=${distro_codename}-security";
      "o=Ubuntu,a=${distro_codename}";
      "o=Ubuntu,a=${distro_codename}-updates";
      "o=Ubuntu,a=${distro_codename}-proposed-updates";
  };

And this is what I have in /etc/apt/apt.conf.d/50unattended-upgrades:

Unattended-Upgrade::Origins-Pattern {
        "o=Ubuntu,n=${distro_codename},l=${distro_codename}-security";
};

When I placed the longer string from the diverted file into the normal file and rerun unattended-upgrades, the upgrades were indeed installed. How should we go about fixing this, then? Will you make some sort of special case for Ubuntu or should I manually override this file for my own VMs if this is a case that only happens on some hosts?

[drybjed]

Fixing it in the role sounds like a good idea. Do you have other Ubuntu releases to check if it's a Xenial issue, or Ubuntu in general?

[metakermit]

I have a 14.04 at hand where I will try it too now. I don't get it, though – the differences in these two configurations is simply adding core, updates and proposed:

      "o=Ubuntu,a=${distro_codename}";
      "o=Ubuntu,a=${distro_codename}-updates";
      "o=Ubuntu,a=${distro_codename}-proposed-updates";

I don't see why the security upgrades aren't installed for the first case when we only have security.

[metakermit]

OK, so I tried it on a 14.04 VM that hasn't been upgraded in a while:

67 packages can be updated.
34 updates are security updates.

Packages with security updates:

libgcrypt11
libssl1.0.0
ntpdate
libidn11
libcurl3-gnutls
curl
libcurl3
libfontconfig1
fontconfig-config
libgd3
linux-image-3.13.0-98-generic
ntp
gpgv
gnupg
tzdata
bind9-host
dnsutils
libisc95
libdns100
libisccc90
libisccfg90
liblwres90
libbind9-90
openssh-sftp-server
openssh-server
openssh-client
openssl
linux-headers-3.13.0-98
linux-headers-3.13.0-98-generic
linux-virtual
linux-image-virtual
linux-headers-virtual
linux-headers-generic
linux-libc-dev

Running the debops.unattended_upgrades playbook with only the security origin results in:

No packages found that can be upgraded unattended and no pending auto-removals

Adding the extra lines to the origins configuration (essentially I enabled unattended_upgrades__release: True) and rerunning unattended-upgrades installs all of these upgrades.

It seems that for now (on Ubuntu 14.04 and 16.04 at least), this "release" option should be true if you want to get any upgrades (including security upgrades) installed.