Debian package securedrop-app-code is not getting upgraded

[kushaldas]

Bug

Description

In my prod vm securedrop-app-code package is not getting upgraded.

Steps to Reproduce

Follow the QA steps to fetch the latest packages from apt-test repo.

Expected Behavior

We should get the 0.7~RC1 packages.

Actual Behavior

$ dpkg -al | grep secure
ii  openssh-client                       1:6.6p1-2ubuntu2.10                        amd64        secure shell (SSH) client, for secure access to remote machines
ii  openssh-server                       1:6.6p1-2ubuntu2.10                        amd64        secure shell (SSH) server, for secure access from remote machines
ii  openssh-sftp-server                  1:6.6p1-2ubuntu2.10                        amd64        secure shell (SSH) sftp server module, for SFTP access from remote machines
ii  secure-delete                        3.1-6                                      amd64        tools to wipe files, free disk space, swap and memory
ii  securedrop-app-code                  0.6                                        amd64        Packages the SecureDrop application code pip dependencies and apparmor profiles. This package will put the apparmor profiles in enforce mode. This package does use pip to install the pip wheelhouse
ii  securedrop-config                    0.1.1+0.7.0~rc1                            all          Establishes baseline system state for running SecureDrop.
ii  securedrop-grsec                     4.4.115+r1                                 amd64        Metapackage providing a grsecurity-patched Linux kernel for use
ii  securedrop-keyring                   0.1.1+0.7.0~rc1                            amd64        Provides an apt keyring for SecureDrop-related packages, so the master signing key used for SecureDrop packages can be updated via apt.
ii  securedrop-ossec-agent               2.8.2+0.7.0~rc1                            amd64        Installs the securedrop pre-configured OSSEC agent
ii  ssh-import-id                        3.21-0ubuntu1                              all          securely retrieve an SSH public key and install it locally

$ sudo apt-get upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
The following packages have been kept back:
  securedrop-app-code
0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.

[kushaldas]

It seems if I manually do a sudo apt-get update and then sudo apt upgrade, it pulls in the right dependencies and the upgraded package.

[redshiftzero]

Did you figure out why your manual intervention was necessary @kushaldas?

[kushaldas]

In my latest testing I could not reproduce this error. I am closing this issue.

[redshiftzero]

Reopening because I also see this same behavior while upgrade testing 0.7.0~rc3 on hardware - the securedrop-app-code package was held back at 0.6. All other packages upgraded without intervention. But if so it's unclear why you didn't see this in further testing @kushaldas. Hypothesis: this is due to the addition of the libjpeg-dev dependency in 9880a30

[redshiftzero]

It looks like @emkll and @dachary also performed upgrade testing - did you experience this issue?

[kushaldas]

@redshiftzero yes, it is missing libjpeg-dev package dependency and that is why it is holding back. In my last testing sudo cron-apt -s -i actually did a apt-get update, so it found the proper dependencies and upgraded to the right securedrop-app-code package.

[redshiftzero]

Do you understand why this only happens in some situations? cron-apt -s -i always does an apt-get update first, no?

[kushaldas]

Do you understand why this only happens in some situations? cron-apt -s -i always does an apt-get update first, no?

Yes (about doing that apt-get update by cron-apt -s -i), but, for some reason it failed for me once, I am guessing the same happened to you too.

[ghost]

did you experience this issue?

I did not. Could it be that something installs libjpeg-dev when running vagrant up app-prod?

[msheiny]

yeah im able to also reproduce this on my instance... digging

[msheiny]

Sooo I think this might be by design on debian's side ... I'm having trouble finding links explaining this behavior but as an analogy it seems to be similar to how permissions work on Android apps:

• You have an app Foo that you previously installed
• New updates to Foo come in without new permission changes and can be automatically updated
• Say one day Foo updates but is now requiring access to your Camera ... Android wants you to manually approve that change in underlying app dependency.

This article had a really good quote summing this up:

If I may try to rephrase: 'Apt-get dist-upgrade'ing also installs new packages brought in the chain of dependencies, whereas 'apt-get upgrade'ing only install newer versions of packages already installed. After a couple of years working with Debian, I never managed to exactly understand that difference. Would you believe I never found documentation who states this as simple?

Still diggin into solutions here

[redshiftzero]

Hmm interesting - but then how did we add dependencies before? For example, when we added the securedrop-keyring package as a dependency of the securedrop-app-code?

[msheiny]

To re-cap conversation that was conducted out of band.. the gist is this ..

• In order to add any dependency to an existing installed package, a dist-upgrade needs to be run
• We do in-fact run a dist-upgrade as part of cron-apt but its very narrow in scope: dist-upgrade -y -o APT::Get::Show-Upgraded=true -o Dir::Etc::SourceList=/etc/apt/security.list -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold.
• That security.list file from above has a very select number of repos that are whitelisted:

root@app-staging:/home/vagrant# cat /etc/apt/security.list
deb http://security.ubuntu.com/ubuntu trusty-security main
deb-src http://security.ubuntu.com/ubuntu trusty-security main
deb http://security.ubuntu.com/ubuntu trusty-security universe
deb-src http://security.ubuntu.com/ubuntu trusty-security universe
deb [arch=amd64] https://apt.freedom.press trusty main
deb https://tor-apt.freedom.press trusty main

the problem is that the newly added dependency, libjpeg-dev is not in one of those repos, its under archive.ubuntu.com/ubuntu trusty main ... so apt-cron won't upgrade here.

• This worked previously for new depdendencies like securedrop-keyring because those were hosted under apt.freedom.press which is on the security.list.

Now theoretically we could add those new libjpeg-dev deps to our internal repo and things should work good -- but the consensus, this close to release, is to roll back the necessary changes that required this new dependency and punt solving this issue to a later release.

[ghost]

Back when -o Dir::Etc::SourceList=/etc/apt/security.list was added in 2014, it does not look like it was for a specific reason. At least it was not explained in the commit history and there was not other discussions archived.

Unless someone has an issue with using Ubuntu repositories other than security repositories during upgrades, could we remove this option?

[conorsch]

@dachary I'm in favor of permitting upgrades for all installed packages. See #3376 for that discussion.

[conorsch]

Since we're excavating old commits, @dachary, I'll take this opportunity to point out that the decision to use cron-apt over the more typical unattended-upgades seems to have been motivated by the lack of a properly configured "Origin" field in the Release file: https://apt.freedom.press/dists/trusty/Release. That's a backend config setting, one which we can easily correct and therefore switch to using unattended-upgrades, if desired. The migration effort may not be worthwhile, but the problems reported here may have been more easily discoverable if we had been using unattended-upgrades, which enjoys much more robust documentation.

[redshiftzero]

Continued discussion over in #3376 (as this will get closed soon)