Use a specific compatibility tool with a Non-Steam Game and write it out to the config.vdf

[trentondyck]

Please see comments at the bottom of #738 for more context

We have the App ID now, so we should be able to configure proton compatibility to Non-Steam Games.

In my own script I'm doing something like the following (Obviously you wont be using vdf just including the example for context):

        /home/deck/.local/bin/pip install vdf 

        export horizon_dir="/home/deck/horizon-xi"
        export steam_dir="/home/deck/.local/share/Steam" 
        config_vdf=${steam_dir}/config/config.vdf
        cp -f ${config_vdf} ${horizon_dir}/bak.config_vdf
        # Documentation - https://github.com/ValvePython/vdf
        echo "Installing Proton layer to ${config_vdf}"

python << END

import vdf 
d=vdf.load(open('${config_vdf}')) 
 
if not 'CompatToolMapping' in d['InstallConfigStore']['Software']['Valve']['Steam']: 
  d['InstallConfigStore']['Software']['Valve']['Steam']['CompatToolMapping']={} 
 
ctm = d['InstallConfigStore']['Software']['Valve']['Steam']['CompatToolMapping'] 
ctm['${app_id}']={ 'name': 'GE-Proton7-42', 'config': '', 'priority': '250' } 
vdf.dump(d, open('${horizon_dir}/config.vdf','w'), pretty=True) 
 
END 
 
        cp -f ${horizon_dir}/config.vdf ${config_vdf} 
        echo "Should have copied ${horizon_dir}/config.vdf to $config_vdf"

        restart_steam
        echo "Successfully added nonsteam game"

[sonic2kk]

Though the script uses Python and the VDF library, since this VDF file is text-based and writeable, it should be possible to write out the compatibility tool.

There are two components to solving this: We need to be able to write out to the config.vdf file. Likely initially there is no entry added for the brand new shortcut, so we'll have to write one out ourselves. Once we can write out to this file, we also need to be able to get the internal name of the selected compatibility tool so that we can write that out to the name field. Solving both of these will allow us to write out a config.vdf entry with a selected compatibility tool using a specific internal name.

---

The name field has to match the internal name of the compatibility tool as defined in its compatibilitytool.vdf file (which is also text-based). This file, among other things, has both the internal name and a display name. The display name is what shows up in Steam, it's like a "pretty" name. The internal name is the actual name of the tool, which is written out to the name field in config.vdf. This file has to exist because it contains the to and from OS list, which means a compatibility tool cannot be used without these fields. and these fields have to go inside a block with the tool's internal name, meaning we should be able to assume the internal name will always exist.

In the dropdown on the UI, we should show a Proton version list identical to the one used to select a Proton version for a game. However, in terms of the logic behind-the-scenes, before writing out the chosen Proton version we should do some sort of mapping to make things more convenient when working from the commandline. For example, STL should be intelligent enough to see Proton Experimental in the arguments and map that to the actual Proton Experimental name (usually formatted as experimental_isodate). We'll also have to append SteamTinkerLaunch to this list, I hear a few people use that program, it's pretty cool imo 😉

When passing from the commandline, If the compat tool name is not recognised, we should check all Proton versions known to STL to see if it matches the internal name for any tools.

Once we have the internal name, we then need to figure out how to write out to a specific point in a file using Bash. At present, I have no idea how to do this. There's probably a way of doing this by grepping for a specific regex to indicate the end of a given section, and then inserting it there, but I'll need to look into how to do this. This part is probably going to be the most challenging, as getting the internal name may require some time but should be relatively straightforward (just a bunch of grepping around I feel like).

I am pretty positive it is possible to write out to a file like this using Bash, I just don't know off the top of my head how to do it just yet, and at time of writing I foresee this as the biggest blocker. When implementing this I will probably start by creating the functions to get the internal name and such from a given compatibility tool, then generating the actual text entry that will go into the VDF file. Once we have that, it's a matter of inserting that block into the VDF file.

---

And of course, like with any modification to Steam files like this, Steam will need to be restarted for these kinds of changes to take effect :-)

I am interested in this feature, I think this is a good proposal. I don't have an estimate though, I really just work on STL when I'm free and work on whatever I'm motivated too, but since this part of the codebase is fresh in my head I am hoping to get around to this sooner rather than later.

Thanks!

[trentondyck]

Maybe something like this for inserting content?

Example

#!/bin/bash

APP_ID="12345"
COMPATIBILITY_VERSION="GE-Proton7-42"

# Find the line number
LINE_NUMBER=$(grep -in "compattoolmapping" config.vdf -A1 | tail -n1 | sed 's/\([0-9]\+\).*$/\1/g')

# Construct content with spaces
CONTENT="                                        \"${APP_ID}\"\\n"
CONTENT+="                                        {\\n"
CONTENT+="                                                \"name\"          \"${COMPATIBILITY_VERSION}\"\\n"
CONTENT+="                                                \"config\"                \"\"\\n"
CONTENT+="                                                \"priority\"              \"250\"\\n"
CONTENT+="                                        }"

echo "inserting content..."
echo -e "$CONTENT"
echo ""

# Reverse the content lines
REVERSED_CONTENT=$(echo -e "$CONTENT" | tac)

# Insert each line preserving spaces
while IFS= read -r line || [[ -n $line ]]; do
    # Escape any characters that could be special to sed
    escaped_line=$(echo "$line" | sed -e 's/[&/\]/\\&/g')
    sed -i "${LINE_NUMBER}a\\
$escaped_line" config.vdf
done <<< "$REVERSED_CONTENT"

Example 2:

#!/bin/bash

APP_ID="12345"
COMPATIBILITY_VERSION="GE-Proton7-42"

# Find the line number
LINE_NUMBER=$(grep -in "compattoolmapping" config.vdf -A1 | tail -n1 | sed 's/\([0-9]\+\).*$/\1/g')

# Construct content with spaces
CONTENT="                                        \"${APP_ID}\"\\n"
CONTENT+="                                        {\\n"
CONTENT+="                                                \"name\"          \"${COMPATIBILITY_VERSION}\"\\n"
CONTENT+="                                                \"config\"                \"\"\\n"
CONTENT+="                                                \"priority\"              \"250\"\\n"
CONTENT+="                                        }"

echo "inserting content..."
echo -e "$CONTENT"
echo ""

# Insert each line preserving spaces
while IFS= read -r line || [[ -n $line ]]; do
    # Escape any characters that could be special to sed
    escaped_line=$(echo "$line" | sed -e 's/[&/\]/\\&/g')
    sed -i "${LINE_NUMBER}a\\
$escaped_line" config.vdf
    LINE_NUMBER=$((LINE_NUMBER+1)) # increment the line number
done <<< "$(echo -e "$CONTENT")"

[sonic2kk]

I came up with a solution mostly based on the second solution you provided. The text is... not very elegant, but this works so far:

compattoolmapping_line="$(( $( grep -n "CompatToolMapping" config.vdf | cut -d ':' -f1 | xargs ) + 1 ))"
new_block="\t\t\t\t\t\"123123123123\"\n\t\t\t\t\t{\n\t\t\t\t\t\t\"name\"\t\t\"Proton_stl\"\n\t\t\t\t\t\t\"config\"\t\t\"\"\n\t\t\t\t\t\t\"priority\"\t\t\"250\"\n\t\t\t\t\t}"
sed -i "${compattoolmapping_line}a\\${new_block}" config.vdf

This gets the first line with "CompatToolMapping" (returns in format linenum: filename, so we use cut to just get the line number) and then we add 1 to that. We don't have to add 2 because the sed in-place insertion uses append, which means it will automatically append to the next line.

The second line with new_block is the ugly part. This is basically the following block including the tabs, but with escape sequences, because sed is annoying and doesn't like newlines (and using escaped \ts is shorter than using tab characters).

                                        "123123123123"
                                        {
                                                "name"          "Proton_stl"
                                                "config"                ""
                                                "priority"              "250"
                                        }

The third line is the in-place edit, which your second solution taught me how to do. I had no idea what this was and had no idea sed allowed for in-place appending (it also has an i mode instead of a). The double-backslash tripped me up, because at first I didn't include it and didn't realise the significance. Without this, there would be a stray t at the start of each block (or n if we started with \n which I did during one revision, and so on).

So this above snippet does work, but it's not very readable. I guess it doesn't really have to be, though. I'd prefer create a more generic function for inserting sections into text-based VDF files, and specifying/guessing the indentation level required, and appending the tab characters as-needed. But at a core level, this will work, albeit it's ugly and not very flexible :-)

I don't think the indentation matters that much as Steam will probably correct it (didn't test it yet), I also assume Steam would re-order the entries. Steam is pretty good at keeping the internal structure of its files organised, but it's also very good at rejecting bad content in files to avoid corruption, so it could go either way without the indentation :-)

This code also does not do any checks for an existing entry in this VDF file, which we'd also probably want to do. Since Steam tries to be smart about its internal files it will probably just remove one (or both) of the entries, but we want to make sure there is only one entry and that it is valid. Maybe to solve this, another generic function that should be created is one to update an existing CompatToolMapping entry by AppID, and if it doesn't exist, create that entry. Might be a lot of work but I like this idea the most, it gives the greatest flexibility :-)

[trentondyck]

yeah if only they use json like sane human beings

[sonic2kk]

yeah if only they use json like sane human beings

JSON has been around since the early 2000s I believe, but likely when Steam was being developed, JSON was not as much of a standard and there was probably not as many standard libraries to parse JSON, especially for C++ back in the day. Steam launched in 2003 and was in use before that as DRM. Probably, Valve made their own format so that they would have that control. VDF was also probably used by Source and such so there was opportunity for reuse, where there wouldn't have been with JSON.

Having said that, they've also had 20 years to change that. Steam has also started using more and more web technologies (the whole client uses CEF now, and the library uses React). So they definitely have a lot more scope nowadays, unless Source 2 is still reliant on VDF files.

So historically I can see why JSON wasn't chosen, and maybe nowadays it's just too much work to change over to JSON. If it ain't broke, don't fix it, I guess. VDF is also used heavily throughout the Steam client; one look through ~/.local/share/Steam and you'll see it's used all over the place. I'm sure some engineers are Valve one day will decide it's time to use JSON though :-)

[sonic2kk]

Wrote a completely crazy, insane, messy function to do this. To say this is rough would be an understatement. Here goes:

## Generate entry in given VDF block with matching indentation
## TODO variable naming could be improved a LOT here
function generateVdfEntry {
    BLOCKNAME="$1"  # Block to start from, e.g. "CompatToolMapping"
    NEWBLOCK="$2"  # Name of new block, e.g. "<AppID>"

    # Arr will look like this, we use a '"' as the delimiter because we know names can't start with this!
    # ( 'name"Proton_stl' 'config"' 'priority"250' )
    KEYVALARR=(${@:3})  # Nutty stuff for Bash weirdness... Basically create array of all arguments and skip first two, since we know the third value is the array of key/value pairs for VDF block
    
    VDFENTRYSTR=""  # String to printf out in the end, this will be a string formatted version of the VDF block to write out
    
    # Indentation for entries in this block to start at (i.e. if BLOCKNAME is preceded with 4 tabs, this will make entries start at 5
    BASETAB="$(( $( grep "${BLOCKNAME}" config.vdf | awk '{print gsub(/\t/,"")}' ) + 1 ))"
    BLOCKTAB="$(( $BASETAB + 1 ))"  # Tabs for content inside the new block, which will be 1 more than the base, which is 1 more than its parent block
    
    BASETABSTR="$( printf '%.0s\t' $(seq 1 $BASETAB) )"
    BLOCKTABSTR="$( printf '%.0s\t' $(seq 1 $BLOCKTAB) )"

    # Line to start appending new entry from
    STARTLINE="$(( $( grep -n "${BLOCKNAME}" config.vdf | cut -d ':' -f1 | xargs ) + 1 ))"

    VDFENTRYSTR+="${BASETABSTR}\"${NEWBLOCK}\"\n"  # Beginning of new block
    VDFENTRYSTR+="${BASETABSTR}{\n"  # Opening brace
        
    for i in "${KEYVALARR[@]}"; do
        # New VDF block content
        # Assume '"' is our delimiter
        VDFDATAKEY="$( echo "$i" | cut -d '"' -f1 )"
        VDFDATAVAL="$( echo "$i" | cut -d '"' -f2 )"

        VDFENTRYSTR+="${BLOCKTABSTR}\"${VDFDATAKEY}\"\t\t\"${VDFDATAVAL}\"\n"
    done
    VDFENTRYSTR+="${BASETABSTR}}"  # Closing brace 
    
    printf "$VDFENTRYSTR"
    
    sed -i "${STARTLINE}a\\${VDFENTRYSTR}" config.vdf
}

# Associative arrays were not used because I wanted to preserve order here, not hash oder
NONSTEAMAPP_VDFENTRY=( 'name"Proton_stl' 'config"' 'priority"250' )
generateVdfEntry "CompatToolMapping" "3981288418" "${NONSTEAMAPP_VDFENTRY[@]}"

This function builds a string with escape characters to write out to a text-based VDF file. It takes the name of a block to insert a new block into (such as CompatToolMapping), then works out how many tabs precede that block. We use printf to add that many \t + 1 to the beginning of the new block's name (such as the Non-Steam AppID). We take an array of values to insert into this block, as key/value strings in an array delimited by " (since entries shouldn't be able to start with "). Inside the function we then loop through this array and split the string as key/value pairs, and build a string to append to our overall VDF block string. We make each line for the "content" of the VDF block with the correct number of preceding tabs, the two tab characters between keys and values, and then finally we append a newline. Finally, we close off our VDF string with a closing brace and write it out using sed.

A few changes will probably be made overtime before I even dream of implementing this in STL. The four most notable things are: variable naming here is rough (it's almost 3am for me 😴), I don't like how we pass the array (it makes the function call cleaner but the implementation messier, with how we slice the function arguments, $@), I think the writing out to the config.vdf should be left up to another function, and also we should pass a generic VDF file name to this function.

Well, this is a start on an implementation at least. This will write out to the VDF file, to the top of the given block. Though personally I would prefer to write this out to the bottom of the given block. I will try to find a way to do this as well.

[sonic2kk]

Well, Steam doesn't re-organize the VDF entries, and I don't feel too good about having those being at the top of the entries list. I would really rather have these at the bottom of the CompatToolMapping block. I also tested out using a separate function for writing out to the VDF and because we calculate the tabs in this function I don't see much value in having a function specifically to write out to a VDF file. Maybe in future, though.

For now, I'll focus on cleaning up this function and seeing if I can improve the things I don't like. I confirmed that this will actually write out a compatibility tool (though I made a small typo, the STL internal name is Proton-stl, with a hyphen and not an underscore). So this logic is on the right path, the implementation just needs fixing up. But glad I was able to go from having no idea how to do this, to having a working (if even less-than-ideal) proof-of-concept in less than 12 hours. At least I know how feasible this is to implement now!

[trentondyck]

I guess you'll find a different way but here's what I got for finding the last line before the end of the CompatToolMapping block (which you could then sed/append the resulting block) as an example:

#!/bin/bash

file="config.vdf"
curly_count=0
line_number=0
inside_block="false"

# Start processing from the line after "CompatToolMapping"
while IFS='' read -r line || [ -n "$line" ]
do

    if [[ "$line" == *"CompatToolMapping"* ]]; then
        export start_block=${line_number}
        export inside_block="true"
    fi

    if [[ ${inside_block} == "true" ]]; then

        [[ $line == *{* ]] && ((curly_count++))
        [[ $line == *}* ]] && ((curly_count--))

        if [[ "$line" == *"CompatToolMapping"* ]]; then
                # Skip one because curly count is zero at this point
                skip=true
        else
            if [[ $curly_count -eq 0 ]]; then
                end_block=${line_number}
                break
            fi
        fi

    fi

    ((line_number++))
done < config.vdf

echo "Start: $start_block"
echo "End: $end_block"

[sonic2kk]

Yeah I used sed to get the full CompatToolMapping block like this: sed -n '/"CompatToolMapping"/,/^[[:space:]][[:space:]][[:space:]][[:space:]]}/ p' config.vdf

Had to use [[:space:]] because sed is grumpy about what characters are used, didn't like \t.

This basically just starts a search with a start and end pattern. Here, the start pattern is "CompatToolMapping" and the end pattern is a curly brace preceded by four tabs. It'll match everything in here.

I haven't done it yet, but it should be possible from here to insert something between the last and second last lines. I appreciate posting these solutions though, I'm far from an expert with Bash so seeing other approaches is very cool and also good for anyone looking back on the history of this issue to see the discussion 👍

---

I used the knowledge here to write another little script which can parse config.vdf for a given account's UserID, and then convert that from the value stored in the VDF to the one used for the actual folder (over on a new experiment repo I created called blush).

[trentondyck]

How do you know other users config vdfs will all have the same indentation, is that required by how the file is created/managed by the vdf library?

[sonic2kk]

Yes, much like JSON you should be able to guarantee the indentation level. At least checking my Steam Deck, this matches. It may be worth my checking other PCs though just to make sure.

Casing, however, is not guaranteed, as ProtonUp-Qt often has this issue. For some strange, unknown reason, VDF files recently have inconsistent casing.

[sonic2kk]

Over the course of the day, I have been experimenting and creating some generic Bash functions to parse text-based VDF files, almost exclusively testing against config.vdf. Most of them can be seen over on sonic2kk/blush, as I think they could be generally useful for anyone else insane enough to want to use Bash to do these kinds of operations. Overtime I'll probably extend and re-organize that repository to have more general Steam utility functions, probably breaking out several from STL to be used generically.

The main one missing that I'm still tinkering around with locally is the method to write out the VDF entry. The function I have locally uses many of the methods in the linked repository, and while it has been refactored a fair bit from what I posted earlier, there are three main things I want to improve upon with it:

1. Check if new block already exists before writing out -- Should be possible with a method I wrote to check if a block already exists within a VDF block.
2. Verify case-insensitive matching, as VDF is not guaranteed to always have the same casing.
3. The big one: Add option to at least choose where the new VDF entry is inserted (either top/bottom, but above/below a given entry would also be really nice for general purposes).

I checked, and Steam appears to append new VDF entries to the bottom of a block, at least for CompatToolMapping. I tested by forcing a compat tool for Morrowind and it was appended to the bottom.

It should be possible to insert a block of text before the final closing brace of the section. Not sure how to do it yet, but I will work on it.

EDIT: Had a brainwave on this and feel dumb for not thinking of this yet, but we can choose where to insert by calculating the line to insert into. We already do this for the start line, so to insert into the end of the section, we can just take the startline, add the length of the CompatToolMapping block to it, and subtract a couple of lines. Just tested and this works.

I don't think inserting before/after a given entry is that useful having thought about it some more. I don't see a case where you would need one entry before/after another, but having the option to insert in the top/bottom of a block is useful imo.

[sonic2kk]

Okay, I've gotten things figured out for the most part now. I got the function to write out to the AppID working, and it can insert into either the top or the bottom of a given VDF block. By default, it appends to the bottom of the block.

There are probably some more safety checks that could/should be done before writing out to the VDF, but for now, this should work and shouldn't bork the CompatToolMapping section at the very least. During the "research" phase like how I did for most of this issue, I wrote it as a generic function for blush. You can see the function createVdfEntry over here: https://github.com/sonic2kk/blush/blob/main/config-vdf-utils.sh -- This is basically the implementation STL will go with, but adapted slightly. For STL, I will probably do some more strict things like backing up the config.vdf before we write into it, just in case of the worst-case scenario.

I have some more testing to do, and then I will open a draft PR, and it will be a very rough draft because I haven't figured out any mapping of the Proton names to the compatibilitytools.vdf internal names yet. Since we have the paths to the Proton versions already though, if we don't already do this it should be straightforward enough (should be able to parse a line with the same code that can get information like game name, install folder, etc from .acf files).

I didn't test the implementation I came up with outside of my own config.vdf, but I did check a family member's Steam installation (also on Linux) and their CompatToolMapping section indentation lined up with mine, so I assume this approach is safe.

More to come soon hopefully!

[sonic2kk]

Draft PR is up at #908 which has the VDF utility functions in place, and some basic logic to write out the selected compatibility tool to the VDF file. The list on the UI is hardcoded right now, but from the commandline, it will accept whatever value you pass in (though it has to be the tool internal name).

The flag for passing the compatibility tool is -ct, ex: -ct="Proton-stl".

If you or anyone else wants to test you're more than welcome to, keeping in mind that this PR is in draft status, and that it has only been tested on my PC.

If anyone is welling to test, PLEASE BACK UP YOUR config.vdf BEFORE TESTING!! It works fine for me, but you could lose any compatibility tool settings or launch options you have set for a game if something goes wrong!

---

I'm happy with the progress I was able to make on this today, we're most of the way to a working implementation.

[sonic2kk]

It has been a whirlwind this weekend but I've made good progress since my last comment. The linked PR should implement this feature now. I have implemented basic text-based VDF interaction, as well as logic to fetch a Proton version's internal tool name based on the name and path string from ProtonCSV.txt. The UI now displays a combobox entry dropdown list of known Proton compatibility tools (excluding ones that are only known to SteamTinkerLaunch and not Steam, i.e. Proton versions downloaded by STL for MO2/Vortex, etc), as well as steamlinuxruntime and Proton-stl added in there for the native Linux Steam Linux Runtime (or Steam Linux Runtime 1.0 (scout) as Steam calls it now) and SteamTinkerLaunch respectively. There is also logic to back up the config.vdf before writing out to it, to try and avoid data loss.

In my testing so far, everything appears to be smooth sailing. The Proton list and logging looks healthy, internal names are fetched properly, the config.vdf is written out to properly as well. One caveat with the last one is that Steam will overwrite any changes made to config.vdf once it is closed, so it is best to make these kinds of changes with Steam closed. A note about this will be added to the UI.

This feature should hopefully be implemented soon :-)

[sonic2kk]

This has been implemented with #908., and the changelog has been updated. Thanks!