Add Non-Steam Game: Add Option To Select Compatibility Tool

[sonic2kk]

Implements #905.

Overview

This PR adds the option on the Add Non-Steam Game GUI to select a compatibility tool known to SteamTinkerLaunch via a dropdown menu populated with compatibility tools. This then gets written out to the config.vdf. if this file exists.

Right now, the list of tools is hardcoded, but this will be changed to display the Proton versions just like how the Proton version dropdowns across STL work (Main Menu, Game Menu, Vortex Proton, MO2 Proton). Then, some mapping logic will be done to map/fetch the internal name of the selected compatibility tool, since we can get the path from the compatibility tool name, then build the path to the compatibilitytool.vdf where we can extract the internal name. If, for some reason, this file is missing or we otherwise can't get the internal name, we can default to the name of the Proton version. Some tools do actually simply use this as their internal name, such as GE-Proton.

If no compatibility tool is passed from the commandline or selected on the UI, the config.vdf file is not touched.

Implementation

How Steam Stores Compatibility Tools in Use

Steam stores information about which game uses which compatibility tool in its ~/.local/share/Steam/config/config.vdf file, under a CompatToolMapping section. This section contains a list of blocks named after a game AppID.

Unlike with shortcuts.vdf and some others (appinfo.vdf I believe is the other one), this is a text-based VDF format, which Valve also re-use for things like their .acf files stored in library folders. This format is defined by Valve and pretty much only used with the Steam Client and I believe the Source engine.

Here's an example snippet of this section and some entries taken from my own config.vdf:

"CompatToolMapping"
{
    "0"
    {
        "name"		"proton_8"
        "config"		""
        "priority"		"75"
    }
    "22320"
    {
        "name"		"Proton-stl"
        "config"		""
        "priority"		"250"
    }
}

The "0" entry is the default global compatibility tool enforced by the user.

To set a compatibility tool for a given game, you must locate its AppID under CompatToolMapping, and then change the value of the name field to the internal name of the compatibility tool, such as proton_7 or Proton-stl.

Compatibility Tool Internal Names

A compatibility tool's internal name is a required field defined in a tool's compatibilitytool.vdf file. Thisi s Here is a snippet of how SteamTinkerLaunch's compatibilitytool.vdf file looks:

"compatibilitytools"
{
  "compat_tools"
  {
	"Proton-stl" // Internal name of this tool
	{
	  "install_path" "."
	  "display_name" "Steam Tinker Launch"

	  "from_oslist"  "windows"
	  "to_oslist"    "linux"
	}
  }
}

Due to the structure of this file, we can guarantee that the internal tool name will always be present, as it is required to set the entries inside of it, which are required for the compatibility tool to function (namely the from_oslist and to_oslist fields).

Manipulating VDF Files in Bash

The next step once you have a game's AppID and the internal name of the compatibility tool you want to set for use with it, you have to actually write out that block. For Non-Steam Games, we do now have the AppID and we're able to use this to set the game artwork. But we have no way of interacting with the VDF files to get the existing data or do any kind of manipulation.

To accomplish this, we needed a fleet of functions to interact with text-based VDF files, such as the config.vdf. As part of research into implementing the feature request for this PR, I generically wrote some utility methods over at sonic2kk/blush to accomplish this. The code was re-used for this PR.

The functions essentially boil down to extracting various parts of the text-based VDF file, and helper methods to go along with them. Then we have createVdfEntry, which is used for actually writing out the VDF entry and brings them all together. We can essentially then pass in the inforamtion we want to set for a VDF entry, tell the function whether we want it at the top of the bottom of the block, and write it out. A Bash array like ( "name!${NOSTCOMPATTOOL}" "config!" "priority!250" ) gets turned into this snippet:

"438958712"
{
    "name"		"Proton-stl"
    "config"		""
    "priority"		"250"
}

These VDF functions work based on grepping for block patterns, primarily relying on tabs. I checked my PC, Steam Deck, and a family member's PC to check that the VDF indentations were consistent, but that doesn't mean this is always the case. ProtonUp-Qt has encountered scenarios where the config.vdf keys casing was not consistent. Lots of testing will be needed to ensure this works as expected!

Remaining Work

Right now, only the basic skeleton UI and logic is in place. However, this is enough to select SteamTinkerLaunch. Proton 7, and Proton 8 as compatibility tools, as these are hardcoded. There is also logic to convert $NON into an empty string so it will be skipped. Commandline usage is also introduced since this is how the GUI function is orchestrated as well, meaning if you know the internal of a compatibility tool ahead of time, you can simply enter this internal name and it will get written out to config.vdf as expected.

There is plenty of remaining work though. There is no logic for mapping Proton names to their internal tool names, and thus the UI only displays a list of hardcoded compatibility tools. There needs to be logic for populating the compatibility tool dropdown with all Proton versions known to SteamTinkerLaunch (with STL added as well, since we don't normally list ourselves as a compatibility tool for obvious reasons 😉). We will need logic on the commandline side to default to the passed argument, since if it doesn't exist in our Proton list we should just assume the user knows what they are doing. If a non-existent compatibility tool is chosen, Steam won't let a user launch a game and it will simply show "Available on: <OS icon(s)>" instead of a play button, until a user forces a different compatibility tool. This is also what happens when you force a compatibility tool (such as GE-Proton), then remove that compatibility tool, from the Steam Client's perspective in this scenario, a removed and non-existent compatibility tool are the same. We should also add more code failsafes such as backing up the config.vdf, just in case anything goes wrong (such as a format change that no one catches, then a user tries to add a Non-Steam Game, write out and borks their config.vdf) Finally, we also need more logging around the VDF functions. Since this isn't present over on the Blush repository

---

This PR is in good shape right now with hopefully no massive bugs. If anyone wants to test this, please back up your config.vdf file!!!

TODO:

• Populate dropdown on UI with actual Proton versions (including STL itself)
• Map these Proton names to internal tool names, defaulting to the name we have if we can't find the compatibilitytool.vdf file and/or internal name for some reason (works for some tools such as GE-Proton)
• Fall back on the commandline side to the passed name in the -ct flag, if we can't find a corresponding Proton version in our CSV file, assume the user knows something we don't and allow this name as-is
• Add more checks and failsafes, such as backing up the config.vdf file before writing out to it
• More logging, mainly around the new VDF functions
• Use combobox entry field for Proton version name, in case the user wants to enter a custom name for some reason?
• Update langfiles

[trentondyck]

Testing looks ok on my deck:

(deck@steamdeck steamtinkerlaunch)$ diff config.vdf $config_vdf
(deck@steamdeck steamtinkerlaunch)$ ./steamtinkerlaunch addnonsteamgame --appname="deleteme" --exepath="/home/deck/horizon-xi/lib/net45/HorizonXI-Launcher.exe" --startdir="/home/deck/horizon-xi/lib/net45/" --iconpath="/home/deck/horizon-xi/icon.png" -ct="GE-Proton7-42"
Preparing to install SteamTinkerLaunch on Steam Deck
Updating SteamTinkerLaunch to latest git

Dependency 'innoextract' already installed, nothing to do.
Dependency 'cabextract' already installed, nothing to do.
Dependency 'yad' already installed, nothing to do.

Finished updating SteamTinkerLaunch ('v14.0.20230916-2')!
Finished setting game art for '3410598813'. Restart Steam for the changes to take effect.
(deck@steamdeck steamtinkerlaunch)$ diff config.vdf $config_vdf
974a975,980
>                                       "3410598813"
>                                       {
>                                               "name"          "GE-Proton7-42"
>                                               "config"                ""
>                                               "priority"              "250"
>                                       }

I didn't see anything in the help text though, and when I'm passing them command line looks like the convention was for double dashes?

[sonic2kk]

Woohoo, good signs all around!

I didn't see anything in the help text though

You're right, I forgot to update the help text. I usually do that at the end and I actually would have forgotten had you not brought this up.

and when I'm passing them command line looks like the convention was for double dashes?

If you're referring to the arguments, you can use -ct or --compatibilitytool:

steamtinkerlaunch/steamtinkerlaunch

Line 22466 in 0ee7690

-ct=*|--compatibilitytool=*)

This is fairly standard with a lot of STL commands including addnonsteamgame:

steamtinkerlaunch/steamtinkerlaunch

Lines 20584 to 20604 in 0ee7690

	echo " addnonsteamgame|ansg <args> Add a $NSGA to Steam"
	echo " opens gui without args"
	echo " <args>: for cli min arg is'-ep'"
	echo " -an=|--appname= App Name - optional"
	echo " -ep=|--exepath= Full gamepath required"
	echo " -sd=|--startdir= Start Dir - optional"
	echo " -ip=|--iconpath= Icon Path - optional"
	echo " -lo=|--launchoptions Game Launch Options - optional"
	echo " -hd=|--hide= Hide Game - optional*"
	echo " -adc=|--allowdesktopconf= Allow Desktop Conf - optional"
	echo " -ao=|--allowoverlay= Allow Overlay - optional"
	echo " -vr=|--openvr= OpenVR - optional*"
	echo " -t=|--tags= Tags - quoted, comma-separated"
	echo " -stllo|--stllaunchoption Use '${PROGCMD} %command%' as Launch Option"
	echo " -hr=|--hero= Hero Art path - Banner used on the Game Screen (3840x1240 recommended)"
	echo " -lg=|--logo= Logo Art path - Logo that gets displayed on Game Screen (16:9 recommended)"
	echo " -ba=|--boxart= Box Art path - Cover art used in the library (600x900 recommended)"
	echo " -tf=|--tenfoot= Tenfoot Art path - Small banner used for recently played game in library (600x350 recommended)"
	echo " --copy Copy art files to Steam Grid folder"
	echo " --link Symlink art files to Steam Grid folder"
	echo " --move Move art files to Steam Grid folder"

If there's anything that needs clarified or that needs changing for this specific command let me know :-)

---

Since this appears to work, the rest of the features here will be quality-of-life stuff to do with mapping and displaying the Proton versions using their tool names, and doing this for all compatibility tools that SteamTinkerLaunch knows about (everything in its ProtonCSV.txt).

I think as well, I will change the dropdown on the UI to be a combobox entry field. This way it still has a dropdown, but the user can enter their own tools if there is one SteamTinkerLaunch simply doesn't know about, or if they want to select a non-Proton compatibility tool such as Luxtorpeda (or related projects) or others like this.

On this note, I should also remember to add the native Steam Linux Runtime as a compatibility tool that a user can select as well as SteamTinkerLaunch.

It's not for this PR but I have a sneaking suspicion that the Steam launch options may not always get applied properly, I'll have to investigate that later on. Mostly writing this so I don't forget to investigate 😅

[sonic2kk]

Had a quick look into how Steam stores internal names.

Third-party tools use compatibilitytool.vdf, and in my checking they always have the same forrmat, meaning there is always a comment beside the internal tool name. This means we can use sed to get this line and just format it a little. I got an implementation for that working.

Now for the, well, not very happy part: Valve Proton versions (Proton 7/8/Experimental/Hotfix/etc, even Steam Linux Runtime 1.0 Scout (or the native Steam Linux Runtime)) have their internal names stored in the appinfo.vdf file. And this file, on top of just being huge, that file uses the binary VDF format. I've spent a bit of time working on ProtonUp-Qt, and from looking around its codebase, it gets Valve tool internal names afaik from this file.

This presents a little bit of a problem, because now I may have to figure out how to parse the appinfo file with Bash, but worst-case I'll just skip Valve Proton versions for now and address that in a separate PR. Another option, which is a bit hacky, is to simply grep for every string like "proton_" from config.vdf, though that would only fetch in-use Valve Proton compatibility tools (so if no game was using Proton Experimental, it wouldn't display it). Another very hacky solution would be to manually build the names of Valve Proton compatibility tools based on the Proton version name, i.e. get the major version from a Proton string like proton-8.0-3c, in this case 8, and append that to proton_. We'll have to do some hardcoded mapping things like this anyway, such as for the Steam Linux Runtime.

Option 3 might be a contender, we'll see. Differentiating between what's a Valve Proton version and what's everything else might be tricky, but we could probably make a good guess based on the directory structure and which files are and aren't in the Proton folder. So we could get all the known Proton versions from the ProtonCSV.txt file, and try to either fetch, guess, or hardcode what the internal name should be:

1. Non-Valve Proton builds, get it from the internal tool name in the compatibilitytool.vdf file -- If this fails for some reason, just fall back to the tool name.
2. For Valve Proton, if we're using Proton Experimental, just map to proton_experimental -- We can make a guess on what is Proton Experimental based on the structure of the folder (no compatibilitytool.vdf, has a dist folder and no files folder), this guess makes it less likely that we might mistake a third-party compatibility tool with "Experimental" in the name for Proton Experimental.
3. For Valve Proton again, make a guess whether something is or isn't Valve Proton based on the same directory structure logic above, and if we know it's a Valve Proton build, try to build a string with a given name (such as extracting the major version and appending that to proton_). For Proton Hotfix, this might get tricker, and again for versions like proton_411. We'll need to check if the minor version is a zero or not, so that proton-8.0-3c becomes proton_8 but proton-4.11-XX becomes proton_411.
4. If tool is not found in ProtonCSV.txt at all, just fall back to the name entered.

This guess approach is probably the most straightforward.

[sonic2kk]

We can now get the Internal name for Proton versions, including Valve Proton versions, in addNonSteamGameGuia. We fetch this from the name in the Proton CSV. The next step now is to list a nicer Proton name that is consistent with the rest of SteamTinkerLaunch, and once we take that in, we can map it.

I went with the approach described earlier. Basically: check compatibilitytool.vdf -> if no internal name found/no file exists, check for Valve Proton version -> if Valve Proton then (if Proton Experimental, hardcode to proton_experimental; elif Proton Hotfix, hardcode to proton_hotfix; else try build Proton version from version name (i.e. proton-8.0-3c becomes proton_8; proton-4.11-3d becomes proton_411) -> if internal name not in compatibilitytool.vdf AND we don't have a Valve Proton build, just return the passed version (just assume the version name matches the internal name, which is actually the case for some tools).

We'll have to do this in addNonSteamGame so the mapping works for commandline usage too. Will work on this next 👍

[sonic2kk]

Something that just occurred to me, is that the SteamTinkerLaunch Proton list includes SteamTinkerLaunch Proton versions, as in ones not available to Steam. Attempting to set these as compatibility tools will not work, we need to limit the compatibility tool selection to only those available to Steam.

This should be straightforward enough to do, we can skip one that have the SteamTinkerLaunch path in their ProtonCSV entry.

[sonic2kk]

Latest commit added some logic to generate a list of valid, Steam-accessible compatibility tools on the dropdown on the Add Non-Steam Game menu. Now the menu is displaying a more familiar list of available Proton versions. We also include the Steam Linux Runtime and Proton-stl here. These can be selected, and when sent to addNonSteamGame when accepting the changes on the UI (or when passing -ct="name" from the commandline), we create a full Proton name+path string and send this to getProtonInternalName This will return the internal name and then we can write this out to the config.vdf file using createVdfEntry.

Really now there are just three bits of outstanding work:

• Switch to combobox entry field, so the user can enter really whatever compatibility tool name they want.
• More checks, like backing up config.vdf before writing out to it - (probably we'll just create one backup and if we try to create another, check how old the existing backup is -- if it's over a certain threshold, allow it to be overwritten)
• Update langfiles, which I'm mentioning in case I forget (it's happened before ☹️)

[sonic2kk]

Added logic to back up the config.vdf before writing out to it. If the backup already exists and it's more than 1 day old, remove it and create a new backup, otherwise just skip if the backup already exists.

Also changed the dropdown to be a combobox entry field, so now custom tool names can be entered if the user knows what they're doing (such as entering luxtorpeda, which STL wouldn't pick up for example since it only tracks Proton tools).

Langfiles still need updating, but after that I think the main thing left for this PR is testing. I'll do a good amount before merging.

---

The core pieces for this PR are in place now:

• Functions for basic interaction with text-based VDF files (so we can write out the config.vdf entry for the tool)
• Functions to get the Internal name of a compatibility tool
• Functions for mapping a given Proton version name to its Internal name
• UI component for selecting the dropdown
• Logic to build and write out the config.vdf entry for the Non-Steam Game to the CompatToolMapping section, and the required tool internal name for it to use based on the selected compatibility tool

Most of the work for this was very general implementation of text VDF interaction and fetching the internal name of a compatibility too. After that, it was a case of wiring that up to the new UI. Both could be generally useful, and in future I may write out the Internal name to the ProtonCSV file so we have it in case it's useful for other purposes, but for now it's not necessary :-)

This PR should be mostly functional now, there may be problems/edge cases that I haven't accounted for yet that I'll try catch out and fix :-)

[sonic2kk]

Just tested and found out that if you add the shortcut with Steam opened and select a compatibility tool, when you close Steam it will remove that updated config.vdf entry. In other words, modification made to config.vdf file Steam are open appear to be discarded once you close it. The shortcut itself is still added, though, likely because those are stored in shortcuts.vdf.

It would be a good idea to add this to the notes on the addNonSteamGameGui, so users know to close Steam before adding games.

[trentondyck]

regarding the backup solution, maybe for another ticket but if something does actually go wrong users may have overwritten their vdf several times over already. Theres also no restore previous config button, which could be useful in a disaster situation. something to think about for later maybe..

[sonic2kk]

regarding the backup solution, maybe for another ticket but if something does actually go wrong users may have overwritten their vdf several times over already. Theres also no restore previous config button, which could be useful in a disaster situation. something to think about for later maybe..

Indeed, it was something I was worried about. I thought I wrote it in a comment but perhaps I didn't. I agree though, a way to restore the original VDF before STL ever wrote out to it would be very useful.

[trentondyck]

Just tested and found out that if you add the shortcut with Steam opened and select a compatibility tool, when you close Steam it will remove that updated config.vdf entry. In other words, modification made to config.vdf file Steam are open appear to be discarded once you close it. The shortcut itself is still added, though, likely because those are stored in shortcuts.vdf.

It would be a good idea to add this to the notes on the addNonSteamGameGui, so users know to close Steam before adding games.

oh this is super helpful info. I was wondering why we had to set this manually sometimes

[sonic2kk]

Yeah, this information doesn't seem to be too readily available. A lot of it is just me figuring stuff out as I go.

I'm hoping to update the Add Non-Steam Game Wiki Page to include the findings and changes in behaviour over the last while, for users and for anyone who might stumble across it with the same questions I had. It may even get referenced in other resources in future and help even more people :-) It happened with the GameScope wiki page which was referenced heavily on the Arch Linux wiki.

[sonic2kk]

Updated the warning on the UI.

Combining all the changes made to the UI since this PR was opened, It now looks like this:

Testing on my end so far is looking good. This is probably, hopefully, ready to be merged soon!

[sonic2kk]

One potential addition I thought of was a "default" option, which when selected will set the game to use the default Steam compatibility tool, the one set in the Steam Settings under "Compatibility":

In my case above, it would return proton_8.

This is stored in config.vdf, under "CompatToolMapping" and under the "0" option. We could check for/fetch this block and since it already uses the internal name, we can use that. If there is no tool found with this for some reason (maybe Steam Play is off or something?) then we can just set the tool name to blank and continue as if no tool was selected to begin with.

[trentondyck]

are they gauranteed to be windows games? could they be linux based non-steam games? are there cases where they select default and using proton somehow breaks their game? Or is it an edge case that almost no one will see anyway since theyre using stl?

[sonic2kk]

are they gauranteed to be windows games? could they be linux based non-steam games?

You can add any executable or really any file you want as the Non-Steam Game executable. Users don't have to select a compatibility tool if they don't want to, if they don't select one Steam will try to launch the executable like if the user double-clicked on it. If the user has Wine on their system and the system recognises that launching a Windows executable should be managed with Wine, then Steam will run the game with system Wine.

This is default Steam behaviour as well, then you add a Non-Steam Game from the Steam Client, by default, no compatibility tool is used at all.

If the user were to select a Linux executable/script here, and then select a compatibility tool, it would try to launch that program with the compatibility tool. It would work the same as if a user added a game manually, then went to Properties -> Compatibility -> Force the use of a Steam Play Compatibility Tool. In fact, modifying the config.vdf manually for a game's AppID and adding a compatibility tool's internal name will show that checkbox as enabled on the Steam UI, and the compatibility tool corresponding to the internal name entered on the config.vdf will be the one in the dropdown. That's how I've been checking to make sure the compatibility tools have been selected properly when adding from SteamTinkerLaunch.

are there cases where they select default and using proton somehow breaks their game? Or is it an edge case that almost no one will see anyway since theyre using stl?

Absolutely, though the global tool need not be Proton. It could just as easily be Luxtorpeda, SteamTinkerLaunch, or any other compatibility tool. For the SteamTinkerLaunch Add Non-Steam Game GUI, the default option will still be none on the dropdown. default would correspond to the default compatibility tool that all games on Steam will use if no compatibility tool is forced for the game.

On the Steam Client this does not happen at all with Non-Steam Games, they will always default to no compatibility tool and you have to enable it yourself. I think it could be useful to have the option to use the same default tool that Steam uses for non-native games :-)

[sonic2kk]

Implemented the logic for the above feature. The compatibility tool name default can be used, and SteamTinkerLaunch will fetch the internal name of the default global tool, if it exists.

[sonic2kk]

In my testing this appears to work, writing out to the config.vdf appears to be safe enough (still really surprised I got that working at all). This is ready from my perspective.

If no other issues crop up I will merge this tomorrow I think.

[trentondyck]

any chance at a release after this merge? when does that usually happen

[sonic2kk]

There's no hard-and-fast release schedule, I have only made two releases since taking over: "v12.0 - Still Alive" back on December 2022, and "v12.12 - Gate of Steiner".

Releases happen whenever there are enough of what I would consider "useful changes", along with when I have a name picked out, and when I feel like making a new release. For the last point it is just when I feel like it, but one major thing that plays into it is how much capacity I have to deal with a potential influx of new issues.

There's no ETA or even much desire on my end for a new release just yet, but there are some things I would like to implement and/or investigate implementing before the next release. Right now broadly speaking I have these in mind to at least make an attempt at adding:

• The ReShade+SpecialK improvements mentioned in ReShade compatibility with Special K #894 (can't exactly remember all what I had planned, it's in a comment and on a ticket on my personal Trello "scribble" board)
• SteamGridDB support for Non-Steam Games from SteamGridDB support for Non-Steam games #906 (probably will integrate somehow with the current SteamGridDB functionality but no idea yet, haven't even looked into it)
• Custom DXVK Versions in Custom DXVK versions #872 (due to how Proton works, this has evolved in my research+testing basically into creating a copy/symlink of an existing Proton version with custom SpecialK DLLs, and has also evolved into an idea to generally create custom Proton versions if that works out with custom DLLs)
  • The idea behind these is similar, so it would feed into implementing LatencyFleX in LatencyFlex 1&2 #794
• Looking into SteamOS 3.5 for ⚠️ SteamOS 3.5 Support #907 and seeing what's broken or what needs updated (I'm thinking innoextract may need bumped)
• Building a newer Yad AppImage version on steamtinkerlaunch-tweaks as part of Steam Deck Experience Improvements [Community Contributions Wanted] #859
• Investigating some commandline/UI options to clean up unused Non-Steam Game compatdata/config files (maybe an option to remove Non-Steam Games, like a Yad popup window with all Non-Steam Games where you can select the ones you want to remove, and it will remove the STL config/grid artwork/STL files etc)
• Refactoring how addNonSteamGame writes out Steam shortcuts, trying to abstract this out into a generic function to write out to shortcuts.vdf
• Option to pass custom Vortex and MO2 installer executable (to more easily use custom builds of either, but mainly I'm thinking of this for MO2 where using custom builds may be more useful). I have been meaning to implement this since Vortex: Add option to specify version #810.

That's a rough set of features I have in mind to implement before the next release. I've considered making an issue about it, but I don't really want to "commit" to adding these. Really, a next release will come when I feel like I've "done enough" to justify a new release.

[sonic2kk]

ShellCheck is still green on this, so I'll bump the version and merge momentarily 👍

Thank you a ton @trentondyck for all of your help with suggestions, testing, and review (and patience, some of these Non-Steam Game improvements have been over a year in the making!). Appropriate credit will be given in the changelog :-)