Get Steam Linux Runtime from Compat Tool Manifest

[sonic2kk]

Background

If I understand correctly, the Steam Linux Runtime command used to be passed as part of the launch command for games. Somewhere along the way this behaviour changed, at least for games using compatibility tools. Maybe this changed with the introduction of the various different runtimes (Soldier, Sniper, Scout). Since then, as far as I know SteamTinkerLaunch's "Use Steam Linux Runtime" option has done pretty much nothing, since the change in Steam behaviour broke it.

Completely by accident, I noticed that in most recent Proton compatibility tools' toolmanifest.vdf file, they specify a require_tool_appid key. This holds an AppID that points at a required tool. I have only checked a few but I have only ever seen this point to a Steam Linux Runtime, and in fact, the same runtime at that - 1391110, which is Steam Linux Runtime - Soldier (fun fact, you can search for items in your library by AppID straight from Steam 😉).

Implementation

This PR restores passing the required SLR (if installed) to a given game. We check if the current RUNPROTON has a toolmanifest.vdf, check if it has a require_tool_appid field and get the AppID, then get the directory of that AppID. If it exists, we append /_v2-entry-point --verb=waitforexitandrun to the end of the path, and use that as our SLR command.

This currently makes a few assumptions that work for now, but may not work in future:

• Assumes all require_tool_appids are a variation of the Steam Linux Runtime
• Assumes _v2-entry-point script exists - This should probably have a separate check, as this could change in future
• Assumes the given tool with the AppID will always be available, when it might not be - The given tool AppID might download with vanilla Proton but probably would not download when using a community flavour of Proton like GE-Proton. If all vanilla Proton builds from Steam require one SLR, but a GE-Proton flavour doesn't, it probably won't download that SLR until game launch I would assume. That means there could be a case where a hypothetical GE-Proton7-69 requires Steam Linux Runtime - Sonic2kk, but all vanilla Proton builds stick to Steam Linux Runtime - Soldier. This is probably very unlikely and would more likely impact older GE builds that rely on older SLR runtimes than newer Proton releases, but it is still a problem. We can probably fix this by using some steam:// command to attempt to install a given AppID before game launch.
  • We do check if the path is valid, but we should investigate if we need to have a failsafe to ensure if a given compatibility tool will have its runtime. We want to use the required tool specified in the tool manifest when available, and if the AppID specified is not installed, we will need to install it basically.
• Possibly more things I have not considered yet?

A handy trick to confirm if a game is using the Steam Linux Runtime is to enable Proton logging. The Proton log will show some SLR information at the top:

======================
...
pressure-vessel: 0.20221215.0 scout
scripts: 0.20221215.0
soldier: 0.20230109.1 soldier 0.20230109.1
...
======================

Testing

This implementation also currently has an issue that causes GameScope to crash. Well, almost always. I launched Cookie Clicker about 5 times and only 1 of those times had a correct game launch. Most of the time the game didn't open and SteamTinkerLaunch showed its dialog and quit. This will need to be fixed The issue was to do with paths having spaces and how we built the launch command for the SLR. This has been fixed as per the "Fix SLR not working if not installed on path with spaces" checkbox.

Various other games with various different launch "features" e.g. games with launchers, will need to be tested to ensure launch functionality works. I will prioritise configuration launchers more than DRM launchers (STEINS;GATE, Sonic Adventure 2, etc).

Tools like MO2 will also need tested.

I tested GameMode and MangoHUD and they both worked correctly on each launch and did not cause any launch issues with Cookie Clicker at least.

TODO

• Fix SLR not working if not installed on path with spaces
• General game testing (games with launchers, etc)
• Force SLR should work fine, but needs tested
• Other functionality like MO2 GUI, etc, will need tested
• Test custom commands
• Test on Steam Deck
• Test native games
• Update langfiles
• Update wiki

[sonic2kk]

@frostworx You probably have more familiarity with the Steam Linux Runtime stuff than I would. No rush on review as this is still in an initial state but I wouldn't mind some input on this :-)

[sonic2kk]

Just checked the proton script for GE-Proton7-42 and Proton 7.0 (proton-7.0-6e) and there does not appear to be any logic for downloading the require_tool_appid when launching. I would assume then that the Steam client itself downloads the tool with the given AppID when one of the following happens:

• When a compatibility tool is selected for a title (e.g., when GE-ProtonX-XX is forced for a given title).
• When an official build of Proton is downloaded, the specified required tool is fetched alongside it.
• When a game is ran with a chosen compatibility tool, the Steam Linux Runtime is downloaded before startup

The last possibility, that it is downloaded on game run, is what I think the most likely case. I believe I have even seen this but I can't confirm and haven't tested this yet. Therefore, SteamTinkerLaunch will probably need to be responsible for downloading a required runtime. The chances are that a user will have ran a game before with vanilla Proton, and that vanilla Proton and GE-Proton builds will require the same SLR, but we can't guarantee this so perhaps we'll need to run something like steam://whatever to install the required tool before we can start the game.

I don't know if that command will wait until the tool is downloaded, if not we may have to find a way to "pause" the game launch while the given SLR downloads. On this subject, though we have a couple of ways of going about it, I personally prefer the idea of not downloading the SLR until after a user tries to start a game with STL. This mirrors Steam client behaviour and means we will always download the relevant tool specified for their selected Proton version (e.g. if they open a game and GE-ProtonX-YZ wants X tool, but they change to Proton 7.0 and it wants Y tool, two tools won't be downloaded).

---

Fun times :-)

[sonic2kk]

Steam command to download SLR works, however:

• It shows a prompt asking the user where to install it -- Not sure if we can do anything about this, I'll have to investigate the usage of the steam:// command, but I would guess only the Steam client could install apps like this.
• As I suspected it does not actually wait. It runs the command and then moves on. We'll need to find some kind of way to pause running until the SLR Is downloaded -- Not sure if we can even do this...

Well, progress is being made. The main outstanding bug I can see is with GameScope. The rest will be general testing that should hopefully not raise too many issues!

EDIT: Something seems to have broken with one of the recent commits, not sure what happened but SLR is stopping games from launching now. Will investigate.

EDIT 2: Even going back to known working commits the problem persists, maybe the SLR download got messed up. Doesn't seem like a bug with the implementation though, will look into it on the app side.

I'm dumb, the crashing issue was because the SLR was not installed to ~/.local/share/Steam/steamapps, it has to be installed here or it won't work for any game that uses it. No game with Proton 7.0 without STL would work and then I remembered I had this issue with RetroArch in the past.

Also from this, with my uninstalling and re-installing of the SLR, I can confirm that Steam will install the required tool(s) from the tool manifest when you try to launch the game.

That is something else we'll have to consider...

[frostworx]

I haven't spent any time within the last months in looking into SLR or any other proton related mechanics, so no idea what the current situation is.
basically, I'd guess that the situation is at best mostly the same as before. the SLR sandbox more or less tries to close any possibility to hook into a game launch process and interact with "undesirable" system programs and environments to "care for a clean and predefined" game start.
steamtinkerlaunch apparently requires 3rd party programs (and system paths) and therefore is not welcome in an SLR environment.
I think it will be very complicated and require lots of troubleshooting time to debug the current SLR process and develop a fully working steamtinkerlaunch/SLR combo.

The core requirement probably still is a 100% version "2" compatibility in the steamtinkerlaunch toolmanifest.vdf.
Seems like the version "1" part was removed in steamtinkerlaunch (I might have removed it myself(?)), so can't tell what the default internal behavior is.
version "1" has less (system path) restrictions than version "2", but likely won't work anymore in the future (or it already stopped working?)
steamtinkerlaunch and all its configs and 3rd party programs used will have to be run from the few SLR system paths which are available in the sandbox. Can't tell if this is even possible, maybe the bottleneck already closed.

My previous findings can be found in the wiki of course and this issue might have some more background information as well.

good luck!

I still hope for you that you will get some busy helping hands from the "community" 🤨
(sorry to say, but I'm afraid I'm mostly done with public linux gaming related things generally - still here for questions though)

[sonic2kk]

Hehe, no worries, thanks for the comment :-)

Perhaps a new option should be created so that SLR can be defaulted to off, with an option to enable it explicitly and a note that the user has to ensure the SLR is downloaded manually.

Not sure what impact this might have on native games, if any. Probably none.

---

I think re-working the to be opt-in is the solution that I'll go with here. That is basically the current behaviour anyway 😉

Will look into this when I have some time (got a couple of things to do and pending PRs for other projects)

[sonic2kk]

I think the solution we will go with is this:

• Keep the default SLR setting but change the description to specify that the SLR will only be used when it is available
• Log and show notifier if the SLR requested by a tool is not available
• Perhaps offer a button somewhere in the Main Menu to check/prompt a download for the current tool's SLR (if specified) - That way a user won't have to hunt for it themselves

I think most users will have a required tool's SLR installed. Unless they have not launched any game on their system before without STL and have only used STL to launch all of their games, they probably have the required SLR installed. And if they don't, no real problems should arise. Testing is needed to ensure nothing goes wrong if they do have the SLR, which is this PR. In other words we need to make sure this PR doesn't break anything, so I will do a lot of testing with it.

I was overthinking things a little I guess. If no SLR is available we can just skip and launch anyway like we currently do, and tell the user that we couldn't find a requested tool's SLR. Perhaps we should show this on the UI somewhere and offer a way to resolve this. On a related note, I wonder what steam:// does on Steam Deck Game Mode 🤔 Now that the new Big Picture UI is available on desktop, perhaps I could test this. I wonder if it shows an install dialog overlaid somewhere 😄

So this is looking feasible, it will just need testing and a couple of tweaks to fix the GameScope problem.

---

Update: The GameScope issue was related to the SLR failing if there are spaces in the path. This doesn't seem to affect regular Steam launches anymore, though it definitely used to. If we can solve the issue of spaces in the path, "regular" Proton launches should work.

[sonic2kk]

Progress update:

• Fixed the GameScope issue which was related to the issue with the SLR path having spaces.
• Tested MO2 with "Fallout: New Vegas" and "Oblivion", worked fine

Need to test native games and custom commands next. I think by default not passing the SLR to custom commands is the best choice, and having that fall under the checkbox introduced in #732. I'm not sure what happens with custom commands at the moment honestly when it comes to the SLR.

After that I will do some more general testing with other game variations (games with launchers, Origin games, etc) and see if they pose any problems.

---

Once testing on the desktop is finished, I'll move on to testing on my Steam Deck. I don't foresee any problems but I have been burned before here 😅

[sonic2kk]

Tested some games with launchers:

• Sonic Adventure 2 ✔️
• STEINS;GATE (vanilla) ✔️
• CHAOS;HEAD NOAH (Committee of Zero launcher) ✔️
• Mass Effect: Legendary Edition ❌

MELE did not work but it all with Origin, but when launching without STL it removed Origin, switched to the EA App. It still doesn't work with STL though even on master. Probably this is an entirely separate issue, thanks EA 🙄 Don't care enough to debug it for now but maybe in future.

Now onto testing some native games to see how they work. If the ones I test work then I will assume the SLR feature is working correctly and move on to testing other things around it like custom commands and force SLR.

EDIT: Turns out the EA App only crashes if the SLR is disabled. With it enabled on this PR, the EA App actually opens but says that the game failed to start. Probably the launch command has changed here and how we handle EA App games needs changing, but that is a separate issue to this PR and actually seems like it can only be investigated once this issue is resolved, i.e. when STL correctly uses the SLR.

[sonic2kk]

Just tested native games and had a "D'oh!" moment - This logic only gets the required SLR from the toolmanifest.vdf file! So of course, native games will not be able to grab it. Duh.

Not sure what to do here, I wonder if native games just default to the regular Steam Linux Runtime? I know some games can specify this somewhere, RetroArch does I think. Perhaps we need to check how a game specifies a specific SLR if it does. Perhaps in the App Manifest.

EDIT: Actually, on Linux Desktop, native games do not use the Steam Linux Runtime by default. This is only enforced for verified native games on the Steam Deck. And generally, games will work with the SLR on the Steam Deck. In cases where they don't, Steam Play None can be used I guess.

Hmm, I will need to think how we want to handle this for native games. I am thinking that we can check for and grab the regular Steam Linux Runtime to use for native games. I am a bit on the fence about forcing the SLR for native games, but actually in my experience it tends to fix native games more than break them, so it's probably fine (many of the "To the Moon" series games require you to enforce the SLR).

Note to self: This will surely be fun for custom commands 😅

[sonic2kk]

Well I rewrote the logic to try and accomodate native Linux games, but it did nothing it seems 😅

Running with the SLR forced as a compat tool for native games shows a pv-bwrap process. Proton games (with and without STL now that the SLR is enabled correctly) also shows the process. It is only native games that don't.

Running native games from the command line and passing the SLR launch command does not show the pv-bwrap process either. So I assume however this launch command is being used is incorrect? I confirmed that games which need the SLR still do not work with STL but do work when forcing "Steam Linux Runtime" as a compatibility tool. So we need to fix however we pass the SLR to native games.

---

Also, an interesting aside: The regular SLR (I guess named "Scout" internally) actually (currently) lists SLR Soldier as a required tool in its tool manifest. However the launch command is "commandline" "/scout-on-soldier-entry-point-v2 --". So I guess internally perhaps it relies on Soldier as the container? I don't see a good way for us to be able to manage this, so for native games, users will be on their own a little here. SteamTinkerLaunch will just attempt to use whatever is available.

I guess native games aren't broken by this, but they also aren't correctly getting the SLR. Fun!...

[sonic2kk]

Tested "Force Steam Linux Runtime" for a few Proton games, works fine.

[sonic2kk]

Inspecting the game launch process from Steam via the command line, it does something quite interesting:

Game process updated : AppID 206440 "/home/user/.local/share/Steam/ubuntu12_32/reaper SteamLaunch AppId=206440 -- /home/user/.local/share/Steam/ubuntu12_32/steam-launch-wrapper -- '/run/media/user/500GB SSD/Games/steamapps/common/SteamLinuxRuntime_soldier'/_v2-entry-point --verb=waitforexitandrun -- '/home/user/.local/share/Steam/steamapps/common/SteamLinuxRuntime'/scout-on-soldier-entry-point-v2 --  '/run/media/user/500GB SSD/Games/steamapps/common/To the Moon/steamshim/parent.amd64' ./ToTheMoon.bin.x86_64 --enableReset=false", ProcID 229025, IP 0.0.0.0:0

So it actually runs SLR Soldier (or if the require_tool_appid changes in future to a different SLR/tool, it will probably use that) and inside of that, it runs the scout on soldier script, and then finally the game. How interesting.

I guess for native games we'll need to check the SLR's required tool AppID and then fetch it pretty much like we do for Proton games. I will probably just strip out and re-create the logic again to avoid duplicating the logic, since the logic is pretty much identical to how Proton games do it, and we insert our scout-on-soldier after the "parent" SLR command.

[sonic2kk]

Woohoo, SLR now works for native and Proton games! Confirmed this by testing some games with Proton, and native games (one which I know requires SLR to even launch) and they all work, plus I checked to make sure the pv-bwrap process spawned. For Proton games I also checked their logs and they were reporting the SLR.

I also think I was able to make the code cleaner in this commit compared to before.

Now that native games seem to work, custom commands need testing, and then we need to ensure this still works on Steam Deck.

[sonic2kk]

Custom commands still work and of course do not use the SLR. Probably we can add a checkbox in a separate PR to enable SLR for custom commands.

[sonic2kk]

Langfiles are updated. Assuming things work fine on Steam Deck (fingers crossed), the wiki may need a light touch-up. I'll look into what might need updating there before merging this PR, and get an edit ready for it.

---

@zany130 You were the one that brought this to my attention, please feel free to give this a whirl and see if this runtime-funtime is solved :-)

[sonic2kk]

Initial testing on my Steam Deck seems to be working fine! The first time I tried to launch a game (Cookie Clicker) it took several minutes, but that was probably the Steam Deck itself as I was playing in Desktop Mode for the sake of checking if the SLR was running. pressure-vessel_adverb was there in the System Monitor processes!

Tested in Game Mode and everything seems to work fine there too. I can't verify very easily if it's actually using the SLR in Game Mode but given that it's not crashing, I'll assume it is 😅

Tested the following games on Steam Deck:

• Cookie Clicker (Desktop Mode & Game Mode)
• Katamari Damacy REROLL (Desktop Mode & Game Mode)
• Sonic Adventure 2 (Game Mode)

---

This seems to work fine in my tests. There is no major benefit that I can see, but it works! I will wait for further testing before merging.

[sonic2kk]

I have an updated wiki markdown file at the ready for when this is merged.

[sonic2kk]

Just realised this was not tested on Flatpak, but I don't foresee any issues that would be caused on Flatpak. It's already able to do a lot of the other filesystem stuff that STL does, so I don't foresee this introducing any other issues.

[zany130]

Wow nice work @sonic2kk !!! Sorry I haven't been active lately. I will go ahead and test this in the next few days

[sonic2kk]

Think it makes more sense to assign @zany130 as a reviewer instead :-)

No rush on this, I don't have a release planned in the near future or anything. Extra eyes are always welcome on the code I manage to hack together :-) It gives some reassurance before pushing the big green merge button!

[zany130]

Well haven't gotten to actually testing all the possibilities like what if a user has a fresh install with no SLR (i briefly looked over the conversation and there code for installing SLR in that senerio?)

But at the very least SLR is working for me with trails of cold steel with just the use slr and reaper option ticked (no force)

[sonic2kk]

what if a user has a fresh install with no SLR (i briefly looked over the conversation and there code for installing SLR in that senerio?)

If there's no SLR (i.e. the path doesn't exist) then STL should just skip it. I have a bunch of checks where it should basically ignore invalid paths 😄

But at the very least SLR is working for me with trails of cold steel with just the use slr and reaper option ticked (no force)

Awesome news!! 🥳

[zany130]

LGTM tested out a few games and it seems to be working correctly. Didn't get around to testing what happens with a clean steam install but that should be fine

EDIT: created a new user on my PC and tested that way it correctly logs that it couldn't find the requested SLR and that it needs to be downloaded. Game launch continued as normal so that's working

[sonic2kk]

Thanks a bunch for such thorough testing! 🚀 I'll merge this tomorrow more than likely :-)

[sonic2kk]

Heh, Proton 8.0 is out and has switched to using "Steam Linux Runtime - Sniper". SteamTinkerLaunch was successfully able to fetch the new SLR (was previously installed by a game I forced Proton 8.0 with):

Tue 18 Apr 01:24:11 BST 2023 INFO - setSLRReap - No SLR provided from command line, attempting to fetch required SLR from current compatibility tool's 'toolmanifest.vdf'
Tue 18 Apr 01:24:11 BST 2023 INFO - setSLRReap - Now executing Pressure Vessel Funtime 2nd Edition Ver. 2.31
Tue 18 Apr 01:24:11 BST 2023 INFO - getRequireToolAppidPath - Found tool manifest at '/run/media/username/500GB SSD/Games/steamapps/common/Proton 8.0/toolmanifest.vdf', attempting to get 'require_tool_appid' value...
Tue 18 Apr 01:24:11 BST 2023 INFO - getRequireToolAppidPath - Got 'require_tool_appid' from '/run/media/username/500GB SSD/Games/steamapps/common/Proton 8.0/toolmanifest.vdf' ('1628350') - Returning path to tool
Tue 18 Apr 01:24:11 BST 2023 INFO - getGameDir - Found matching App Manifest file for presumed entered AppID '1628350' - Manifest file is '/run/media/username/500GB SSD/Games/steamapps/appmanifest_1628350.acf'
Tue 18 Apr 01:24:11 BST 2023 INFO - setSLRReap - '/run/media/username/500GB SSD/Games/steamapps/common/SteamLinuxRuntime_sniper' exists - Path gotten from specified AppID looks valid
Tue 18 Apr 01:24:11 BST 2023 INFO - setSLRReap - Building Steam Linux Runtime command for Proton game
Tue 18 Apr 01:24:11 BST 2023 INFO - setSLRReap - Adding SLR '/run/media/username/500GB SSD/Games/steamapps/common/SteamLinuxRuntime_sniper/_v2-entry-point --verb=waitforexitandrun --' to the launch command

Pretty neat that it worked :-)