Prepare R4.0 -> R4.1 upgrade procedure #5685

marmarek · 2020-02-23T18:49:46Z

marmarek · 2020-02-23T19:33:47Z

Here is example /etc/default/grub: https://gist.github.com/marmarek/b7717e71e3a72a7ab09aa3907b8afeb9
It needs to be adjusted to preserve Xen boot options (see xl info) and Linux boot options (see /proc/cmdline).

This set of packages will allow running rpm 4.14 in a fc25-based dom0 (R4.0), which is necessary to update to fc31-based dom0 (R4.1). QubesOS/qubes-issues#5685

marmarta · 2020-02-23T20:45:58Z

Procedure I followed to achieve successful upgrade:

For dom0 update itself:

Update all templates and standalone VMs (the updater widget works here)
Shut all unnecessary VMs down (sys-net, sys-firewall, sys-usb can stay)
Open terminal in dom0
Update dom0: sudo qubes-dom0-update
Update qubes-release: sudo qubes-dom0-update --releasever=4.1 --enablerepo=*testing* --enablerepo=*unstable* qubes-release
Fix a key that fails to import itself sudo rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-31-primary
Update everything: sudo qubes-dom0-update --action=distro-sync --allowerasing --enablerepo=*testing* --enablerepo=*unstable*
Reboot

And then to fix grub's not being there:

Open terminal in dom0, sudo -s for convenience
Copy /boot somewhere safe: cp -r /boot ~/
umount /boot/efi
fdisk /dev/sda (or whatever your disk is called) and there do the following:

check your current partition table (p)
delete EFI system partition (by default it will be the 1st partition, so perform d 1)
create a new partition 1 ( n, use default start, size +25M, remove the signature)
create another new partition (n, default start, default end, default the rest); it should get number 3 on a default install
change partition type of partition 1 to EFI system ( t, EFI system is type 1)
commit the changes and exit fdisk (w)

create a VFAT file system on the EFI partition you just created, for me mkfs.vfat /dev/sda1
create an ext4 file system on the other partition you just created, for me mkfs.ext4 /dev/sda3
mount the ext4 partition in /mnt/: mount /dev/sda3 /mnt
copy your safely stored /boot there: cp -r ~/boot/* /mnt/
umount /mnt
mount /dev/sda1 /boot/efi
now comes the annoying part: use blkid to find out the UUID of this partition, and then edit /etc/fstab to fix it there; then add another line to /etc/fstab that references the ext4 partition - I copied the one for /boot/efi, fixed its UUID with one taken fromblkid, changed the location to /boot and changed the contents of the third column to just defaults
qubes-dom0-update grub2-efi-x64 --enablerepo=*testing*
grub2-mkconfig -o boot/efi/EFI/qubes/grub.cfg
efibootmgr -v, fing the EFI line (it was 0003 for me)
efibootmgr -b 0003 -B;efibootmgr -c -L Qubes -l '\EFI\qubes\grubx64.efi (replace 0003 with your number of EFI partition if needed)
cp /usr/share/grub/unicode.pf2 /boot/efi/EFI/qubes/fonts/
plymouth-set-default-theme qubes-dark
dracut -f
and finally, vim /etc/default/grub and fix the UUID in GRUB_CMDLINE_LINUX

marmarek · 2020-02-23T20:52:05Z

@fepitre can you write a script that handle the grub part?
The above order works, but I'd recommend first installing grub2-efi-x64 and its configuration and only then repartition - otherwise you have a window where you have no bootloader at all and if the download fails, you're screwed.
Also, the script should check if there is really enough space for that /boot - default /boot/efi should be 500M, but on some early R4.0 builds it could be smaller - in that case better abort early.

fepitre · 2020-02-23T21:06:50Z

Sure.

This set of packages will allow running rpm 4.14 in a fc25-based dom0 (R4.0), which is necessary to update to fc31-based dom0 (R4.1). QubesOS/qubes-issues#5685

This allows using various --with/--without options (or rather --define="_with_* 1" and --define="_without_* 1" - for which --with/--without are aliases). For now it is used only for package rebuilt from a different Fedora version (linux-dom0-updates repo). Adjust also legacy build code that calls dnf builddep manually - here is why it needs to be --define. Change dependencies handling for src.rpm there, to extract spec file and operate on it directly, because otherwise --define options takes no effect (dnf normally looks at src.rpm dependencies in the header, not within the spec file). This issue does not apply to mock, because mock first rebuilds src.rpm package, including various --define. QubesOS/qubes-issues#5685

This set of packages will allow running rpm 4.14 in a fc25-based dom0 (R4.0), which is necessary to update to fc31-based dom0 (R4.1). QubesOS/qubes-issues#5685

New repositories require rpm 4.14 (packages compressed with zstd) and also new rpm requires qubes-core-dom0-linux >= 4.0.23 (different output in rpm -K). Prevent installing new qubes-release unless those other updates are installed too. Otherwise updating the system will stop working. QubesOS/qubes-issues#5685

QubesOS/qubes-issues#5685

0spinboson · 2020-02-26T05:49:52Z

I'm still using legacy boot, so updating dom0 went swimmingly, after I noticed I had to adjust the repo links in /etc/yum.repos.d/qubes-dom0.repo (to fc31) after step 5.
Only niggle: the grub boot screens were missing. :)
And I of course had to add gnttab_max_frames and gnttab_max_maptrack_frames values to the xen cmndline in /etc/default/grub, then regenerate grub.cfg, before rebooting.

updating fedora-30 template led to some issues as dnf refused to update xen-libs because of dependencies, didn't pursue that very far because I already had a fedora-31-xfce template configured.

marmarek · 2020-02-26T10:34:22Z

I'm still using legacy boot, so updating dom0 went swimmingly, after I noticed I had to adjust the repo links in /etc/yum.repos.d/qubes-dom0.repo (to fc31) after step 5.

That's probably because of not updated qubes-release package. Do you know what qubes-release package version you had at that step (was it updated in step 5 or not)?

0spinboson · 2020-02-26T10:50:58Z

I seem to recall seeing a package with 4.1 in the name, but not sure, sorry

marmarek · 2020-04-01T14:58:03Z

There are some upgrade problems described in a duplicate issue here: #4867

DemiMarie · 2020-06-01T01:57:00Z

I will add that the upgrade process should fail-safe ― if there is a failure during install, the system should still be usable.

marmarek · 2020-06-01T02:07:03Z

Generally yes (and the script already tries to do that), but making a backup before is still recommended.
The current WIP script: https://github.com/fepitre/qubes-migration

0spinboson · 2020-06-01T06:14:11Z

Left after upgrade to fc32:

fedora-obsolete-packages
pythons2-futures
python2-iniparse
python2-msgpack
python2-nose
python2-pycurl
python2-pygpgme
python2-systemd
python2-backports
python2-zmq
python2-backports
python2-chardet
python2-crypto
python2-qubesadmin
python2-singledispatch
python2-backgports_abc

All are listed under 'removing dependent packages' when running qubes-dom0-update, yet all are left in place.

marmarek · 2020-06-03T02:01:19Z

During --dist-upgrade phase I've got an error like below. I started from R4.0.3 default install, in UEFI mode:

Error: Transaction check error:
  file /usr/bin/rst2html from install of python3-docutils-0.15.2-4.fc32.noarch conflicts with file from package python2-docutils-0.13.1-3.fc25.noarch
  file /usr/bin/rst2html5 from install of python3-docutils-0.15.2-4.fc32.noarch conflicts with file from package python2-docutils-0.13.1-3.fc25.noarch
  file /usr/bin/rst2latex from install of python3-docutils-0.15.2-4.fc32.noarch conflicts with file from package python2-docutils-0.13.1-3.fc25.noarch
  file /usr/bin/rst2man from install of python3-docutils-0.15.2-4.fc32.noarch conflicts with file from package python2-docutils-0.13.1-3.fc25.noarch
  file /usr/bin/rst2odt from install of python3-docutils-0.15.2-4.fc32.noarch conflicts with file from package python2-docutils-0.13.1-3.fc25.noarch
  file /usr/bin/rst2odt_prepstyles from install of python3-docutils-0.15.2-4.fc32.noarch conflicts with file from package python2-docutils-0.13.1-3.fc25.noarch
  file /usr/bin/rst2pseudoxml from install of python3-docutils-0.15.2-4.fc32.noarch conflicts with file from package python2-docutils-0.13.1-3.fc25.noarch
  file /usr/bin/rst2s5 from install of python3-docutils-0.15.2-4.fc32.noarch conflicts with file from package python2-docutils-0.13.1-3.fc25.noarch
  file /usr/bin/rst2xetex from install of python3-docutils-0.15.2-4.fc32.noarch conflicts with file from package python2-docutils-0.13.1-3.fc25.noarch
  file /usr/bin/rst2xml from install of python3-docutils-0.15.2-4.fc32.noarch conflicts with file from package python2-docutils-0.13.1-3.fc25.noarch
  file /usr/bin/rstpep2html from install of python3-docutils-0.15.2-4.fc32.noarch conflicts with file from package python2-docutils-0.13.1-3.fc25.noarch
  file /usr/share/doc/python-systemd/NEWS from install of python3-systemd-234-12.fc32.x86_64 conflicts with file from package python2-systemd-232-1.fc25.x86_64
  file /usr/share/doc/python-systemd/README.md from install of python3-systemd-234-12.fc32.x86_64 conflicts with file from package python2-systemd-232-1.fc25.x86_64
  file /etc/sysconfig/network-scripts/ifdown-ppp from install of network-scripts-ppp-2.4.7-35.fc32.x86_64 conflicts with file from package ppp-2.4.7-9.fc24.x86_64
  file /etc/sysconfig/network-scripts/ifup-ppp from install of network-scripts-ppp-2.4.7-35.fc32.x86_64 conflicts with file from package ppp-2.4.7-9.fc24.x86_64
  file /etc/sysconfig/network-scripts/ifdown-Team from install of network-scripts-teamd-1.30-2.fc32.x86_64 conflicts with file from package teamd-1.27-1.fc25.x86_64
  file /etc/sysconfig/network-scripts/ifdown-TeamPort from install of network-scripts-teamd-1.30-2.fc32.x86_64 conflicts with file from package teamd-1.27-1.fc25.x86_64
  file /etc/sysconfig/network-scripts/ifup-Team from install of network-scripts-teamd-1.30-2.fc32.x86_64 conflicts with file from package teamd-1.27-1.fc25.x86_64
  file /etc/sysconfig/network-scripts/ifup-TeamPort from install of network-scripts-teamd-1.30-2.fc32.x86_64 conflicts with file from package teamd-1.27-1.fc25.x86_64
  file /usr/lib64/qt4/plugins/designer/libpyqt4.so from install of python3-PyQt4-4.12.3-11.fc32.x86_64 conflicts with file from package PyQt4-4.11.4-15.fc25.x86_64
  file /etc/dracut.conf.d/plymouth-missing-fonts.conf from install of qubes-artwork-plymouth-4.1.4-2.fc32.noarch conflicts with file from package qubes-artwork-4.0.1-2.fc25.noarch
  file /usr/share/plymouth/themes/qubes-dark/qubes-logo-outline.png from install of qubes-artwork-plymouth-4.1.4-2.fc32.noarch conflicts with file from package qubes-artwork-4.0.1-2.fc25.noarch
  file /usr/share/plymouth/themes/qubes-dark/qubes-logo-solid.png from install of qubes-artwork-plymouth-4.1.4-2.fc32.noarch conflicts with file from package qubes-artwork-4.0.1-2.fc25.noarch

Upgrade path to fc32 depends on a packaging change in python-systemd-234-4. Otherwise python2-systemd conflicts with python3-systemd on files, but lack appropriate package metadata about it. QubesOS/qubes-issues#5685

QubesOS/qubes-issues#5685

- legacy file pool on LVM - Debian (default one, and 10) as default template, including sys-net and sys-firewall QubesOS/qubes-issues#5685

The updatevm already downloads packages only (either using --downloadonly option internally, or using yumdownloader). The option from the user matters for the dnf call in dom0 only. Especially, sending --downloadonly breaks downloading updates via debian-10, which uses yumdownloader, as it doesn't have --downloadonly option. This is mostly relevant for R4.0 -> R4.1 upgrade (and maybe some future too?), as the process uses --downloadonly option internally. QubesOS/qubes-issues#5685

The updatevm already downloads packages only (either using --downloadonly option internally, or using yumdownloader). The option from the user matters for the dnf call in dom0 only. Especially, sending --downloadonly breaks downloading updates via debian-10, which uses yumdownloader, as it doesn't have --downloadonly option. This is mostly relevant for R4.0 -> R4.1 upgrade (and maybe some future too?), as the process uses --downloadonly option internally. QubesOS/qubes-issues#5685 (cherry picked from commit 0018597)

marmarek · 2022-01-17T19:44:09Z

Stage 4 completely fails with debian-10 updatevm templates. Error: yumdownloader: no such option: --downloadonly
This resulted in the system breaking (I couldn't switch the template anymore).

I tried to get it working with debian-10 as updatevm, but sadly the changes required are too severe. In short: yumdownloader cannot parse "rich dependencies", which are used in some newer packages, and there is no way to get dnf into debian-10. So, I'll adjust the tool to complain early about this case, and let the user change updatevm (template) while it is still possible.

QubesOS/qubes-issues#5685

DemiMarie · 2022-01-17T23:39:59Z

I recommend taking a (LVM or BTRFS) snapshot of the dom0 root filesystem, so that in the event of a disaster the user has a chance to recover.

QubesOS/qubes-issues#5685

marmarek · 2022-01-18T03:54:27Z

I recommend taking a (LVM or BTRFS) snapshot of the dom0 root filesystem, so that in the event of a disaster the user has a chance to recover.

It may help in some situations, but actually not that many. To restore from such snapshot, you'd need to reboot. At this point (when recovering from failed stage 4), you'll have all the templates and standalones already upgraded to R4.1, and it won't be compatible with reverted dom0 (we provide only backward compatibility, not forward compatibility, and both GUI and qrexec protocols have changed). So, you may get dom0 back to running, but not much more. You could try to restore templates from snapshots too, if you really want (and have one)...

The dist upgrade tool tries to check assumptions before performing the upgrade and refuse to continue when there is still a chance (this now includes all the cases reported in this issue too). And it also prints a huge warning about doing the backup before. The cases that are easy to recover are handled already, but otherwise the recommended way is to either retry directly (which should work if the failure was some intermittent download issue), or reinstall (possibly R4.1 already) and restore from backup - like in the other upgrade procedure.
With complex(*) fallback options, I would really want to avoid situation where mitigation that would help in very few cases anyway, would risk breaking the common case. And it is the case here, because for example extra snapshot could make the upgrade run out of disk space (especially when the VMs are stored in dom0 filesystem too, but it is a risk even without that).

DemiMarie · 2022-01-31T00:22:44Z

I recommend taking a (LVM or BTRFS) snapshot of the dom0 root filesystem, so that in the event of a disaster the user has a chance to recover.

It may help in some situations, but actually not that many. To restore from such snapshot, you'd need to reboot. At this point (when recovering from failed stage 4), you'll have all the templates and standalones already upgraded to R4.1, and it won't be compatible with reverted dom0 (we provide only backward compatibility, not forward compatibility, and both GUI and qrexec protocols have changed). So, you may get dom0 back to running, but not much more. You could try to restore templates from snapshots too, if you really want (and have one)...

I wonder if we should intentionally reboot dom0 after upgrading it, and only upgrade the templates and standalones afterwards. What do you think?

marmarek · 2022-01-31T10:51:12Z

I wonder if we should intentionally reboot dom0 after upgrading it, and only upgrade the templates and standalones afterwards. What do you think?

That could work, but it would make the upgrade process less convenient. Currently we exploit the fact that running VMs still works even after upgrading their templates, to perform the upgrade in the opposite order, requiring just one reboot instead of two. This way, you can call qubes-dist-upgrade --all --yes and leave it alone for some time, then just reboot and everything is done. With extra reboot, you'd need to sit there and do the reboot in the middle of the process.

DemiMarie · 2022-01-31T22:55:24Z

I wonder if we should intentionally reboot dom0 after upgrading it, and only upgrade the templates and standalones afterwards. What do you think?

That could work, but it would make the upgrade process less convenient. Currently we exploit the fact that running VMs still works even after upgrading their templates, to perform the upgrade in the opposite order, requiring just one reboot instead of two. This way, you can call qubes-dist-upgrade --all --yes and leave it alone for some time, then just reboot and everything is done. With extra reboot, you'd need to sit there and do the reboot in the middle of the process.

Personally, I consider the improved reliability to be worthwhile. Rebooting dom0 after a successful upgrade means that we can take a snapshot of the dom0 filesystem before the upgrade, and revert to it if the upgrade fails. We could also use Fedora’s offline system upgrade feature. We can also clone at least one of the templates before upgrading them, ensuring that the user will always be able to recover their system.

marmarek · 2022-02-01T19:31:52Z

While some of those ideas are okay, it is way too late to do major restructure of the upgrade process, that has been confirmed to work for many users already. Please propose such major structure changes earlier, when we'll prepare upgrades for R4.2.

QubesOS/qubes-issues#5685

ViliusSutkus89 · 2022-09-26T07:54:37Z

I think I've managed to upgrade qvm-appmenus before the 4.0 -> 4.1 dist upgrade, which breaks the dist upgrade.
qubes-dist-upgrade --post-reboot fails with an error:

usage: qvm-appmenus [--verbose] [--quiet] [--help] [--init] [--create]
                    [--remove] [--update] [--get-available] [--get-whitelist]
                    [--set-whitelist PATH | --set-default-whitelist PATH]
                    [--source SOURCE] [--force]
                    [--i-understand-format-is-unstable] [--file-field FIELDS]
                    VMNAME [VMNAME ...]
qvm-appmenus: error: the following arguments are required: VMNAME

andrewdavidwong · 2022-09-26T08:25:40Z

@ViliusSutkus89: Please note that the issue tracker (qubes-issues) is not intended to serve as a help desk or tech support center. Instead, we've set up other venues where you can ask for help and support, ask questions, and have discussions. (By contrast, the issue tracker is more of a technical tool intended to support our developers in their work.) Thank you for your understanding!

ViliusSutkus89 · 2022-09-26T09:10:08Z

@andrewdavidwong Oh no, I don't need help or tech support, I just wanted to report a possible failure scenario. No idea why I didn't open a new issue. Sorry about that.

BTW, Qubes is extra cool, thanks for working on this project.

marmarek added T: task Type: task. An action item that is neither a bug nor an enhancement. P: default Priority: default. Default priority for new issues, to be replaced given sufficient information. labels Feb 23, 2020

marmarek added this to the Release 4.1 milestone Feb 23, 2020

marmarek added a commit to QubesOS/qubes-release-configs that referenced this issue Feb 24, 2020

Add config for R4.0 -> R4.1 transitional packages build

c9df1cd

QubesOS/qubes-issues#5685

andrewdavidwong added C: doc C: other labels Feb 24, 2020

marmarek added a commit to QubesOS/qubes-release-configs that referenced this issue Feb 25, 2020

Fix typo in qubes-os-r4.1-dom0-update-from-r4.0.conf

6b011ce

QubesOS/qubes-issues#5685

marmarek added a commit to QubesOS/qubes-linux-yum that referenced this issue Feb 25, 2020

Add repositories for R4.0->R4.1 transitional update packages

4bac0a4

QubesOS/qubes-issues#5685

qubesos-bot mentioned this issue Feb 26, 2020

qubes-release v4.1-0.22 (r4.1) QubesOS/updates-status#1659

Closed

marmarek mentioned this issue Apr 19, 2020

Upgrading from QubesOS 4.x doesn't result in the same end user configuration (discards and tpool) #5643

Closed

andrewdavidwong added P: major Priority: major. Between "default" and "critical" in severity. and removed P: default Priority: default. Default priority for new issues, to be replaced given sufficient information. labels Jun 4, 2020

marmarek added a commit to marmarek/qubes-dist-upgrade that referenced this issue Nov 24, 2021

Skip template/standalone update in stage 1 too if requested

b55851f

QubesOS/qubes-issues#5685

marmarek added a commit to marmarek/qubes-dist-upgrade that referenced this issue Nov 24, 2021

Skip metadata resize on non-thin LVM setup

709c91e

QubesOS/qubes-issues#5685

marmarek added a commit to QubesOS/openqa-tests-qubesos that referenced this issue Nov 29, 2021

Add few more release upgrade tests

a2e9cf2

- legacy file pool on LVM - Debian (default one, and 10) as default template, including sys-net and sys-firewall QubesOS/qubes-issues#5685

marmarek mentioned this issue Nov 30, 2021

dom0-update: do not send --downloadonly option to the updatevm QubesOS/qubes-core-admin-linux#82

Merged

This was referenced Dec 6, 2021

core-admin-linux v4.1.16 (r4.1) QubesOS/updates-status#2746

Closed

core-admin-linux v4.0.32 (r4.0) QubesOS/updates-status#2747

Closed

core-admin-linux v4.0.33 (r4.0) QubesOS/updates-status#2751

Closed

marmarek added a commit to marmarek/qubes-dist-upgrade that referenced this issue Jan 17, 2022

Complain about too old updatevm, while it's still possible to change it

1ed8d25

QubesOS/qubes-issues#5685

marmarek added a commit to marmarek/qubes-dist-upgrade that referenced this issue Jan 17, 2022

Complain about too old updatevm, while it's still possible to change it

8358e67

QubesOS/qubes-issues#5685

marmarek added a commit to marmarek/qubes-dist-upgrade that referenced this issue Jan 18, 2022

Complain about too old updatevm, while it's still possible to change it

68dd91b

QubesOS/qubes-issues#5685

marmarek mentioned this issue Jan 30, 2022

Window resize problems (in particular in LibreOffice) #7179

Closed

qubesos-bot mentioned this issue Feb 3, 2022

dist-upgrade v4.0.4 (r4.0) QubesOS/updates-status#2808

Closed

marmarek added a commit to marmarek/qubes-dist-upgrade that referenced this issue Feb 3, 2022

Add another stage to convert LUKS1 to LUKS2

04b6013

QubesOS/qubes-issues#5685

marmarek closed this as completed Feb 4, 2022

andrewdavidwong added C: dist upgrade The code and tools that support upgrading in-place from one Qubes OS release to another and removed C: other labels Jul 27, 2022

rustybird mentioned this issue Oct 24, 2022

Support 4k storage #4974

Closed

marmarek mentioned this issue Oct 25, 2022

Prepare R4.1 -> R4.2 upgrade tool #7832

Closed

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prepare R4.0 -> R4.1 upgrade procedure #5685

Prepare R4.0 -> R4.1 upgrade procedure #5685

marmarek commented Feb 23, 2020 •

edited

Loading

marmarek commented Feb 23, 2020

marmarta commented Feb 23, 2020

marmarek commented Feb 23, 2020

fepitre commented Feb 23, 2020

0spinboson commented Feb 26, 2020 •

edited

Loading

marmarek commented Feb 26, 2020

0spinboson commented Feb 26, 2020

marmarek commented Apr 1, 2020

DemiMarie commented Jun 1, 2020

marmarek commented Jun 1, 2020

0spinboson commented Jun 1, 2020 •

edited

Loading

marmarek commented Jun 3, 2020

marmarek commented Jan 17, 2022 •

edited

Loading

DemiMarie commented Jan 17, 2022

marmarek commented Jan 18, 2022

DemiMarie commented Jan 31, 2022

marmarek commented Jan 31, 2022

DemiMarie commented Jan 31, 2022

marmarek commented Feb 1, 2022

ViliusSutkus89 commented Sep 26, 2022

andrewdavidwong commented Sep 26, 2022

ViliusSutkus89 commented Sep 26, 2022

Prepare R4.0 -> R4.1 upgrade procedure #5685

Prepare R4.0 -> R4.1 upgrade procedure #5685

Comments

marmarek commented Feb 23, 2020 • edited Loading

marmarek commented Feb 23, 2020

marmarta commented Feb 23, 2020

marmarek commented Feb 23, 2020

fepitre commented Feb 23, 2020

0spinboson commented Feb 26, 2020 • edited Loading

marmarek commented Feb 26, 2020

0spinboson commented Feb 26, 2020

marmarek commented Apr 1, 2020

DemiMarie commented Jun 1, 2020

marmarek commented Jun 1, 2020

0spinboson commented Jun 1, 2020 • edited Loading

marmarek commented Jun 3, 2020

marmarek commented Jan 17, 2022 • edited Loading

DemiMarie commented Jan 17, 2022

marmarek commented Jan 18, 2022

DemiMarie commented Jan 31, 2022

marmarek commented Jan 31, 2022

DemiMarie commented Jan 31, 2022

marmarek commented Feb 1, 2022

ViliusSutkus89 commented Sep 26, 2022

andrewdavidwong commented Sep 26, 2022

ViliusSutkus89 commented Sep 26, 2022

marmarek commented Feb 23, 2020 •

edited

Loading

0spinboson commented Feb 26, 2020 •

edited

Loading

0spinboson commented Jun 1, 2020 •

edited

Loading

marmarek commented Jan 17, 2022 •

edited

Loading