improvements to Steam Deck install experience

[sonic2kk]

Hihi! About a week ago, I started working on a PR to add the notifier to a few places during the Steam Deck installation process for SteamTinkerLaunch, and then the ideas I had kept growing and before I knew it, I ended up with a behemoth of a PR. Sorry about that, but the changes are all related to one thing: Improving the installation experience on Steam Deck!

The ideas came from my own experience installing and updating STL on Steam Deck, and the changes were written and tested on my Deck (I'm writing this with a keyboard+mouse+monitor attached in Desktop Mode, hi!)

Background

I've been using SteamTinkerLaunch on my Steam Deck since I got it, and it's worked great! But I found the installation process to be a little bit unclear at times. That gave me the idea to try and add a notifier or two to the install process, to give feedback that the install process started, that dependencies are being downloaded, and that installation finished.

When installing STL with ./steamtinkerlaunch, the output in the terminal makes it very clear what's going on - Though it's a bit messy. But at least on KDE, which the Steam Deck uses, you can double click a script and it will ask you if you want to execute the script. In this instance, it's unclear when the installation process really ends, or what the script is doing. It's not really a "problem", but it's something I wanted to try and fix.

As I was working on fixing the notifier, I had an idea for a lot of other things. They were all small ideas that at first I naively expected would be easy to implement. The things I wanted to improve essentially boil down into four main categories:

Visual Feedback

Using the notifier to show the user visually what's going on, and improving the echo output in the terminal to show more friendly messages such as "Preparing to install SteamTinkerLaunch" - or "Preparing to Update" if we already have some files in the ~stl/prefix.

Offline Installation

The big one, I wanted to add a lot of failsafes and catches to make sure STL didn't show any unnecessary warnings when trying to run offline, and to allow users to manually place dependencies in the relevant folders so STL can find them (like ~/stl/deps). I also wanted to be as generous as possible when trying to find these files, so a lot of time was spent trying to create regex patterns to match files that even somewhat match the dependency STL is searching for - if an exact filename match was not given.

I also wanted STL to be able to intelligently install/update when offline; use the current files with the install script if we don't have a prefix, and if we have a previous install, check to make sure an update is required.

Quality of Life

This really only consisted of two ideas:

• A "quiet" mode to hide the notifier - This idea came after I added the notifiers to the installation, as I recognized this could be unnecessary when using something like ProtonUp-Qt, but I also thought it would be a generally useful idea.
• Improve the language loading process so that the chosen language would be respected as soon as possible - Particularly useful on Steam Deck so the notifier translation strings are respected right away

Code Cleanup

Maybe "Cleanup" is the wrong word, but there was a comment about removing a hardcoded wget version that I wanted to address. I also noticed the code duplication for installing cabextract and innoextract, and wanted to try and address all of these problems with a generic function to fetch dependencies.

The code in steamdedeckt was getting a little "jumbled" after this refactoring, so I wanted to split some of the code out to make the function easier to follow and maintain.

Additional logging was also added around these changes, which should help with any potential debugging.

Solution

My solution in the end turned out to be a big overhaul to the Steam Deck installation experience in more than just the notifier. There were quite a lot of changes in the end, which came from my own experience using SteamTinkerLaunch on Steam Deck. This is virtually an implementation of my entire wishlist for the installation process with four main areas of focus once again:

• Visual Feedback, leting the user know what's going on during installation
• Offline Installation, adding some checks and catches to help SteamTinkerLaunch install offline for the users that might attempt it, without breaking functionality or behaviour for "online" installation
• Quality of Life, very minor touchups I noticed along the way that I tried to fix, basically just adding a tiny bit of "polish" onto the rest of these features :-)
• Code Cleanup, basically just moving some of the code around to break out logic that was becoming unwiedly in the ever-growing steamdedeckt function.

Changelog

As mentioned above, the changes fall into four main categories, so I'll break them down in a bullet pointed list.

Visual Feedback

• Added notifiers to the Steam Deck installation process to inform the user when SteamTinkerLaunch installation has started, if it's downloading dependencies, if it's pulling down changes from GitHub, warning them if they're offline, and notifying them of install/update success (with a different message for install/update success) and the version they updated to.
• Added checks to abort installation in various scenarios, such as missing dependencies, and added associated notifiers for these failures.
• Added echo statements throughout the code to mirror these notificiations, as well as provide some more in-depth information during the install that would be overkill for notifications (such as dependencies satisfied, dependency paths, etc).
• Added additional logging throughout the install process that logs in even more detail than the echoes, to help with transparency and debugging in the installation process.

Offline Installation

• Check if we're connected to the Internet early in the install process by checking if we can ping archlinux.org.
• Warn the user when they're attempting to install offline, as if they are missing various manually downloaded dependencies, installation might fail. But we attempt to install offline anyway.
• Check the version of the SteamTinkerLaunch script that the user is running to perform an install, and if the current directory looks like a valid SteamTinkerLaunch download, do one of the following:
  • If there is no existing installation, copy all the files in the script's directory to $PREFIX (currently $HOME/stl/prefix).
  • If there is an existing SteamTinkerLaunch installation, and the downloaded $PROGVERS is newer than the installed $PROGVERS, overwrite the $PREFIX with the downloaded files.
  • If there is an existing SteamTinkerLaunch installation, and the downloaded files $PROGVERS is older than the installed $PROGVERS, or the same, do nothing.
• Only attempt to download dependencies if we're connected to the Internet and they don't already exist (with an exact filename match) in the dependeencies directory (currently $HOME/stl/deps). Checking on the exact filename means we'll still download them again on a version bump
• If there is no exact name match in the dependencies folder, and if we fail to download it, we'll try to find it with a fuzzy search and do a quick check on the dependency version if we can find it, to make sure they're trying to install the same dependency version that the script expects.
  • This tries to extract the "pretty" depenency name from the desired dependency archive name, e.g. it will extract innoextract from innoextract-1.9-5.pkg.tar.zst, and then check if any files in the dependencies folder have innoextract in their name (with a -maxdepth 1).
  • Likewise it will also try to extract the "pretty" name for existing archives in the dependnecies folder to see if its pretty name exists in the original dependency's archive name.
  • We also check the version of dependencies from the archive name, if we can find it, and warn the user when the dependency version is mismatched. If there are any problems, they can delete the dependencies folder and reinstall, or connect to the Internet to download updated dependencies.
  • All of this is in an attempt to be as generous as possible when matching dependency archive names, with the aim of having installation failure be a last resort.
  • If the user is connected to the internet/if the repositories can be reached, this should really never happen. This is primarily intended for users that might want to place the archives in the dependencies folder themselves.
• Improve offline Yad installation by searching for exact filename matches in both $HOME/.config/steamtinkerlaunch/downloads/yadappimage/ and the STL dependencies directory. If we cannot find an exact filenamt match, we perform a case insensitive match for any .AppImage with yad in the name, in the same two directories. Only fail the installation if all four of these steps fail.

Quality of Life

• Respect the selected language as soon as possible in the installation process. This means SteamTinkerLaunch will load the correct language file for the notifier as early as possible in the installation process. Previously it was only loaded after the steamdedeckt function and some other code checks.
  • We first prioritise any incoming lang=<lang> argument, if that's not passed then we check if there are any language files matching the currently selected language (set in the STL global conf) in $HOME/.config/steamtinkerlaunch/lang. Failing all of these, we fall back to $STLDEFLANG (which is english.txt).
  • loadLangFile, which loads the default language file in the final case mentioned above, will now check the current script's lang dir for a matching language file, if it exists. This means the most up-to-date language file strings will be used as soon as possible, including during installation (which means the notifier will get the most up-to-date English strings for the notifier by default)
• Add a "Quiet" option with the -q flag to run SteamTinkerLaunch with the notifier disabled. This can be beneficial for users that find the notifier spammy but don't want to force disable it in their Desktop Environment's settings. The idea was thought up with ProtonUp-Qt in mind, so that the new installation notifier could be hidden when installing that way.

Code Cleanup

• Split logic for managing dependencies on Steam Deck from steamdedeckt into a separate function
• Split logic for downloading and extracting a dependency into a separate, more generic function, which should make it easier to add extra dependencies in future if needed, or to use the function elsewhere, if needed
  • NOTE: Yad is the exception here. The logic for downloading Yad in setYadBin and the code in dlCheck remain, though it does have extra checks for where to find Yad in Offline Mode as mentioned earlier
• Remove SSL_INIT output from the Yad download to clean up terminal output
• Remove the Curl LD_PRELOAD as it doesn't seem to be required anymore

Screenshots

Here's a couple of screenshots of how the install process would look now, with the notifier and logging:

Notifier

Terminal

TODO

There are a few things I still have to investigate with this PR.

• I have done extensive testing on my Steam Deck for this, but not as much on my PC or laptop. Some, but not a lot. I'll need to do some more testing around this to make sure everything still works as expected, checking as many edge cases as I can.
• Sometimes when I switch to Desktop Mode on my Steam Deck, the new notifiers pop up. I don't think any of my changes should've caused this, so I think maybe STL is running each time Steam opens? I'll need to look into this more, but if it's intentional behaviour I guess that means the notifier is working as expected, and we don't really want to hide that from the user unnecessarily (the whole point was to give them visual feedback when STL is updating 😃)

Considerations

I have a couple of reservations with the changes I have made:

• The check for the STL version in Offline Mode depends on the PROGVERS being accurate. This might add a maintenance cost if there are times when it's bumped twice a day or whatever. This likely would not affect very many users and I doubt most if any users will use STL offline for any length of time, but it's something that's been in the back of my mind.
• The fetchAndExtractDependencies function currently only uses tar and was only tested against .tar.zst files. This is because I didn't know if we could make any assumptions about what might be available to users, should we assume we have 7z for example. The other reason was that I'm not very familiar with tar and its options, and didn't think checking the archive type was important enough at least initially since this function only downloads the same file type right now from the Arch/SteamOS repos. If it's desired wwe can get a discussion going about how to best implement support for different archive types to help make this function more generic.

---

This PR ended up getting way bigger than I intended. Since this touches the actual installation of STL on Steam Deck, we should take our time with review and feature discussion, to ensure everything is up to a good standard. I'm still quite the beginner with Bash, and while I did take time to try and make sure the code was of the best standard I could make it, there could be ways to improve it that I'm not aware of. As such, especially with the size of this PR, all feedback is welcome. I put a lot of time into this in the hopes of having this benefit as many users as possible, in the best way possible, so as such I want all the input possible to make these changes the best they can be.

I also realise there is the mo2-rewrite branch in progress, and a couple of open issues related to testing MO2 and Vortex. As such, and given the big changes here, I'm in no rush for this to be merged. As mentioned we can take our time with review. There are a couple of merge conflicts, but if it's okay I'd prefer to leave resolving those until these changes are in a state that can be merged.

Thanks "code buddies"! 😉

[AliceDTRH]

Is there any way to verify if the dependencies are actually really the files we're looking for? Say using a hash or filesize?

[sonic2kk]

I believe setYadBin checks a hash (though it won't fail if it doesn't match afaik) but there was no check for the other dependencies. The current dependencies are pulled either from the Arch archives or the SteamOS Arch mirror, if either of these provide a hash it wouldn't be too hard to add a hash check to the dependency download, just download the hash to the same deps dir and check that.

Offline, I'm not sure how would be best to handle this. Maybe we would only check offline if the user provided one.

[sonic2kk]

I checked it out quickly before bed and it seems like the Arch archives provide a .sig file, which might allow for some kind of verification, but I'm not totally sure.

And just to be clear since I didn't really emphasise it before, I think this is a good idea. At least giving the user a warning when the sums don't match is just good stewardship when we're downloading packages for the user.

For offline, we could just check if a sum is in the same folder as the archive and assume the user provided a correct one. And if they didn't provide one we could just log saying no sum was provided so we didn't verify that the content of the file is correct, or something to that effect - Probably worded much more nicely than that 😅

So yes, a good idea for sure, and I'm open to suggestions and discussion on how to implement it! Thank you :-)

[frostworx]

thanks for the PR @sonic2kk and sorry, I still didn't have the time to review this in detail
(maybe we should discuss privately how to proceed generally - for example if you want to take over the project lead. I'm afraid time, patience and motivation likely won't increase any more here, so I consider to step back. thoughts welcome)

Not sure if it is known, but steamtinkerlaunch already has a generic download/extract/checksum function dlCheck
it is badly documented at best, but should generally work fine (it is used by most downloads)

will reply in detail soonish. sorry for the delay...

[sonic2kk]

I still didn't have the time to review this in detail

No worries at all, it's a big PR and there are other things going on. Take your time :)

maybe we should discuss privately how to proceed generally - for example if you want to take over the project lead. I'm afraid time, patience and motivation likely won't increase any more here, so I consider to step back. thoughts welcome

Oh wow, please send an email and we can get in touch and have a conversation about this, but if you're looking for a project lead I'm definitely willing! 😀

Not sure if it is known, but steamtinkerlaunch already has a generic download/extract/checksum function dlCheck
it is badly documented at best, but should generally work fine (it is used by most downloads)

Ah, I did see this function - late on into making the PR, when I was doing the Yad stuff which came towards the end. I thought maybe it was specifically for Yad but if it's for most other downloads we could try to find a way to use that at least partially for checking the checksums. I know you haven't done a review yet, but the function I created for downloading the dependencies separately does some other checks. From my understanding of how dlCheck works though it should be possible to just call it with an already downloaded file (that's pretty much what the new logic in setYadBin does I believe). Or I could try to merge the logic I have in the new function into dlCheck, my only concern there is potentially breaking other things that use dlCheck.

[frostworx]

sorry, still no review, but just emailed you at least 😀

[frostworx]

As already mentioned via mail, I suggest we simply merge the PR for a trouble-free project transfer :)

[frostworx]

:)

[sonic2kk]

@AliceDTRH I realise the merging of this was a bit sudden, but I haven't disregarded your suggestion on checking the sums of the dependencies. If you would like you are more than welcome to dig deeper and report back, or even open a PR yourself for this. Otherwise I'll take a look at implementing this at some point before the next major release. There are a couple of other things on my plate right now, so it's not an immediate priority, sorry!

[sonic2kk]

A sad news follow-up on this that I forgot to add last night: I can't find a way to verify the checksums of the downloaded files. The .sig file from the Arch archives doesn't seem to be adequate for that use case. So for now this is not something I'll be working on. Contributions would be welcome though :-)