libxcrypt-devel.4.4.2-3.fc29 missing, but libxcrypt-4.4.2-3.fc29 is present

[edwintorok]

Host system details

State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: no runs since boot
Deployments:
  ostree://fedora-workstation:fedora/29/x86_64/silverblue
                   Version: 29.20181227.0 (2018-12-27T01:09:18Z)
                BaseCommit: a6d00a98f2bfb16e3cdae4f89f306a5035c7e67ae15012ccf3d757c11c4b8624
              GPGSignature: Valid signature by 5A03B4DD8254ECA02FDA1637A20AA56B429476B4
           LayeredPackages: ansible bcc-tools chromium dconf-editor evince evolution fedora-toolbox firefox-wayland gmp-devel gnome-photos gnome-tweak-tool kernel-tools
                            latencytop libreoffice mozilla-ublock-origin ncurses-term neovim pass perf perl-Pod-Html python3-psutil ripgrep stow strace tig weston zsh

● ostree://fedora-workstation:fedora/29/x86_64/silverblue
                   Version: 29.20181225.0 (2018-12-25T01:13:26Z)
                BaseCommit: 9c2273bb044c6b0a36c0e1549011091afd79f95700f27a568f5964ac12c7eb32
              GPGSignature: Valid signature by 5A03B4DD8254ECA02FDA1637A20AA56B429476B4
      ReplacedBasePackages: fuse-overlayfs 0.1-6.dev.git3d48bf9.fc29 -> 0.1-8.dev.git91bb401.fc29
           LayeredPackages: ansible bcc-tools chromium dconf-editor evince evolution fedora-toolbox firefox-wayland gmp-devel gnome-photos gnome-tweak-tool kernel-tools
                            latencytop libreoffice mozilla-ublock-origin ncurses-term neovim ocaml ocamldoc opam pass perf perl-Pod-Html python3-psutil ripgrep stow strace
                            tig weston zsh
                    Pinned: yes

Expected vs actual behavior

See #415 (comment), at first I thought this is bug 415, because I got an error about base package replacements, however it looks like the problem is that libxcrypt is at version 4.4.2-3, whereas libxcrypt-devel is at version 4.4.2-1 only:

# sudo rpm-ostree install libxcrypt-devel-4.4.2-3.fc29 libxcrypt-4.4.2-3.fc29 --allow-inactive
Inactive requests:
  libxcrypt-4.4.2-3.fc29 (already provided by libxcrypt-4.4.2-3.fc29.x86_64)
Checking out tree a6d00a9... done
Enabled rpm-md repositories: fedora updates
rpm-md repo 'fedora' (cached); generated: 2018-10-24T22:20:15Z
rpm-md repo 'updates' (cached); generated: 2018-12-26T02:39:50Z
Importing rpm-md... done
error: Packages not found: libxcrypt-devel-4.4.2-3.fc29
sudo rpm-ostree install libxcrypt-devel-4.4.2-1.fc29
Checking out tree a6d00a9... done
Enabled rpm-md repositories: fedora updates
rpm-md repo 'fedora' (cached); generated: 2018-10-24T22:20:15Z
rpm-md repo 'updates' (cached); generated: 2018-12-26T02:39:50Z
Importing rpm-md... done
⠁ Resolving dependencies... 
Forbidden base package replacements:
Resolving dependencies... done
error: Some base packages would be replaced
 sudo rpm-ostree install libxcrypt-devel-4.4.2-2.fc29
Checking out tree a6d00a9... done
Enabled rpm-md repositories: fedora updates
rpm-md repo 'fedora' (cached); generated: 2018-10-24T22:20:15Z
rpm-md repo 'updates' (cached); generated: 2018-12-26T02:39:50Z
Importing rpm-md... done
error: Packages not found: libxcrypt-devel-4.4.2-2.fc29

You can see that libxcrypt-devel-4.4.2-1.fc29 exists (but its not installable because I already have libxcrypt-4.4.2-3), and libxcrypt-devel 4.4.2-2.fc29 doesn't exist as expected, but libxcrypt-devel-4.4.2-3.fc29 doesn't exist either.

Expected:
libxcrypt and libxcrypt-devel to be promoted in the repository at same versions, together

Steps to reproduce it

libxcrypt-devel-4.4.2-3 is here part of same build as libxcrypt-4.4.2-3, so I don't know why rpm-ostree claims that it is not available:
https://koji.fedoraproject.org/koji/buildinfo?buildID=1175444
I tried rpm-ostree cleanup -m, rpm-ostree cleanup -b, rpm-ostree refresh-md, rpm-ostree uninstall --all, followed by rpm-ostree upgrade several times, but the package is still not shown as available.

Would you like to work on the issue?

No, I wouldn't know where to start, this seems to be an infrastructure problem, is it a bug in koji, bodhi, or rpm-ostree?

[dustymabe]

can you try rpm-ostree upgrade --install libxcrypt-devel ?

[cgwalters]

This is a dup of #415

[cgwalters]

EDIT: Nevermind I was wrong, that update was stable.

---

Though looks though like you're somehow getting the updates testing version.

Actually, I still see that too myself:

# rpm-ostree status -b
State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: last run 3h 14min ago
BootedDeployment:
● ostree://fedora-silverblue:fedora/29/x86_64/silverblue
                   Version: 29.20190104.0 (2019-01-04T01:19:14Z)
                BaseCommit: cd860b3aec37f997702d55251a3dcf5cba116e42dfe49543ad5410b8263d47b6
           LayeredPackages: emacs fedora-toolbox ffmpeg fuse-sshfs git-evtag gnome-tweak-tool gstreamer1-plugin-openh264 gstreamer1-plugins-ugly-free keepassx krb5-workstation libva-vdpau-driver libvirt
                            libvirt-client mozilla-openh264 opensc origin-clients pcsc-lite-ccid qemu-kvm strace tilix tmux virt-manager xsel ykclient ykpers
host:root@localhost /root
# rpm -q libxcrypt
libxcrypt-4.4.2-3.fc29.x86_64
#

We had a regression a while ago in Fedora infra where the updates-testing bits were making it into the main ref. /cc @dustymabe

[dustymabe]

/me looking

[dustymabe]

libxcrypt-4.4.2-3.fc29.x86_64 was pushed to stable on 12/28 - https://bodhi.fedoraproject.org/updates/FEDORA-2018-dc5207da89. It's expected that you would see it in the fedora/29/x86_64/silverblue ref right?

[owtaylor]

If we look at the initial comment, the ostree version where 4.4.2-3 was found is
Version: 29.20181227.0 (2018-12-27T01:09:18Z)

And the update wasn't pushed stable until:
"2018-12-28 05:12:54 (UTC)"

So, as far as I can tell @cgwalters comment "We had a regression a while ago in Fedora infra where the updates-testing bits were making it into the main ref." seems to be on target.

I'm not sure if:

rpm-md repo 'updates' (cached); generated: 2018-12-26T02:39:50Z

is at all relevant - if stale updates metadata is being used it could cause a problem like this, but the ostree vs. bodhi timestamps imply that isn't the cause here. Unfortunately, I don't think there are detailed compose logs available at this point, so it's hard to tell exactly.