Symbols
Oftentimes, you may hear this talk about "symbols" around these places. Here, you can learn what those are, how are they helpful to ExplorerPatcher and what to do in case of errors.
Symbol files represent files which tell various things about an executable in a human-readable form. These are very useful especially when debugging programs. They do not contain the source code, but rather pointers about what certain places in the executable file are and do. For example, a program's usually made up of a bunch of functions put together in an executable. A function is a block of organized, reusable instructions that is used to perform a single, related action. When debugging, it's often useful to have a general idea of what the functions in a program do, in order to know what to expect. Symbol files aid in this, containing the names the programmer used for those blocks of instructions when programming.
As I said before, symbol files can offer valuable information about certain places in a program. It's easier to talk when referencing an example, so let's take the Win+X menu as an example.
Windows still contains code that builds and manages the menu in a DLL called twinui.pcshell.dll. As it stands, the DLL does not contain any information to help identify where that code exactly is. Thus, its symbol file can be used to gather this information from. It contains information about where functions with names like CLauncherTipContextMenu::GetMenuItemsAsync and CLauncherTipContextMenu::_ExecuteCommand are located in the actual binary. Without symbols, identifying these places is much much harder, relying on techniques such as matching an expected pattern of how the instructions should look against the actual binary. This is more prone to breaking, but can sometimes be used to circumvent the need for symbols. In ExplorerPatcher, this is used in the code that enables the Windows 10 network flyout, for example, partly to avoid the need for introducing another dependency on symbol data.
So, ExplorerPatcher uses functions already present in Windows' libraries to enable certain functionality. The way it identifies where those functions are is by using symbol files.
Currently, symbol data is required for the following functionality:
-
twinui.pcshell.dll:Win+Xmenu -
twinui.pcshell.dll: skinning and unskinning some context menus -
twinui.pcshell.dll: legacy Windows 10 window switcher -
twinui.pcshell.dll: functionality that opens the calendar flyout instead of Microsoft Teams when pressingWin+C(until ExplorerPatcher version 22000.343.41.11) -
StartDocked.dll: open Start directly to "All apps" -
StartDocked.dll: maximum numbers of apps to show in the "Most used" section
For binaries (EXEs, DLLs, CPLs etc) shipped with certain builds of the Windows operating systems, Microsoft makes symbols available for public download on their public service for debugging purposes. The information about where to download the symbol file associated with a certain binary is contained in the binary itself. Currently, Microsoft publishes symbols for these types of OS builds:
- stable builds of the operating system (builds serviced during Patch Tuesday of each month, "B" releases)
- builds shipped via the Dev channel of the Windows Insider Program
Patch Tuesday refers to the first Tuesday of the month, when updates are usually delivered to Windows stable builds. Although usually they are published at the time the build is pushed via Windows Update, sometimes delays happen. There is nothing I can do on my side about this. It's just how Microsoft processes these things. It's usually a matter of maximum a couple of hours before they are finally published.
Symbols are usually NOT made available for the following types of builds:
- builds shipped via the Beta and Release Preview channel of the Windows Insider Program
- "C" and "D" stable OS builds - these are builds labeled as "Preview" and offered via Windows Update in the second part of a month, representative of what will generally arrive on the next Patch Tuesday; these are optional, the user has to explicitly click "Download and install" to have them applied on their computer, they are never automatically installed by Windows Update
When File Explorer starts up, ExplorerPatcher tries to load symbol data cached from a previous run of this check from the registry or from an internal table of symbol data for some known versions of the operating system. If ExplorerPatcher is unable determine this information, the program attempts to download the symbol files from Microsoft's servers, gather the necessary data from the files, and cache it in the registry for future use, so that this process is not repeated again at each File Explorer start-up for this OS build. If this fails, the program will run, but the functionality described above as dependent on symbol data from certain system files will not be available (i.e. will not work, the UI elements regarding it are still available in the "Properties" window).
First of all, you can tell this from the version of the release: it usually indicates the highest build on which it was tested and confirmed to work. It also usually contains hardcoded (built-in) data for the files in that build, so no download is necessary. For example, the current ExplorerPatcher version is 22000.348.39.0. That means it was tested and confirmed to work up to Windows 11 build 22000.348. It also contains built-in symbol data information up to that build, so no symbol data download is necessary when using that build, it will automatically use what it has built-in. Mainly, ExplorerPatcher is tested to work on the current stable build of Windows 11, as that is what I daily drive and also because testing against Insider builds is pretty difficult since things change a lot in there, so developing a patch this month might be rendered useless the next one when things are changed again or reverted altogether. So yeah, officially, ExplorerPatcher is supported only up to the builds that are listed in the release notes.
So, if you run 22000.348.39.0 on OS build 22000.348, everything will just work.
If you run 22000.348.39.0 on a newer build, ExplorerPatcher will try to download, parse and cache into the registry symbol data from Microsoft. Until that's done, the functionality described above as needing symbol data will be unavailable. If it succeeds, a one-time notification will show, informing you about the success of the operation and asking you to manually restart File Explorer to have the symbol data be used by ExplorerPatcher to enable the additional functionality. From this point on, the information for this build is cached in the registry and will be used the next times you run File Explorer on this build, bypassing the start-up symbol availability check and download.
If it fails, ExplorerPatcher will usually work without the functionality described above as requiring symbol data and no notification will be shown in this case. Telling if you're in this mode is easy: press Win+X for example. If nothing shows, you're likely in this mode.
I usually try to update ExplorerPatcher with any necessary stuff as soon as a new stable build is pushed. Most of the times, manual intervention is not needed: bumping up the version info, or maybe hardcoding the data for this build is kind of optional, as, so long as symbols are available from Microsoft, the application always has the fallback mechanism of extracting the data from symbols available online. With Microsoft being a US-based company, symbol files and new builds are usually published during the noon/afternoon in the United States. I'm in Europe, where at that point it's usually night time, so I might be asleep, hence the delay in updating this sometimes, or maybe I just cannot do the thing at that moment. As I said, for stable builds, this should mostly be a non-issue, or at most a couple of hours without some of the functionality.
The second method you can tell what state ExplorerPatcher is in is by watching the information printed in the debug console ("Properties" - "Advanced" - "Enable console"):
- If ExplorerPatcher uses built-in symbol information, you will see some lines like the following printed at the very beginning of the console output:
[Symbols] Identified known "twinui.pcshell.dll" with hash 8b23b02962856e89b8d8a3956de1d76c.
[Symbols] Identified known "StartDocked.dll" with hash b57bb94a48d2422de9a78c5fcba28f98.
Loaded symbols.
- If ExplorerPatcher does not have built-in symbol information for the build you are running and it uses data cached in the registry and gathered on a previous run, you will see output similar to this:
Loaded symbols.
- If ExplorerPatcher does not have built-in symbol information for the build you are running and it succeeds gathering symbol data, you will see output similar to this:
Attempting to download symbol data; for now, the program may have limited functionality.
[Symbols] Started "Download symbols" thread.
[Symbols] Attempting to download symbols for unknown OS version 10.0.22000.348.
[Symbols] Downloading to "C:\Users\root\AppData\Roaming\ExplorerPatcher\".
[Symbols] Downloading symbols for "C:\Windows\system32\twinui.pcshell.dll"...
[Symbols] Reading symbols...
[Symbols] Downloading symbols for "C:\Windows\SystemApps\Microsoft.Windows.StartMenuExperienceHost_cw5n1h2txyewy\StartDocked.dll"...
[Symbols] Reading symbols...
[Symbols] Finished gathering symbol data.
[Symbols] Finished "Download symbols" thread.
- If ExplorerPatcher does not have built-in symbol information for the build you are running and it failed to gather symbol data (due to various reasons, like symbols not being available from Microsoft, the computer not being connected to the Internet etc), you will see output similar to this:
Attempting to download symbol data; for now, the program may have limited functionality.
[Symbols] Started "Download symbols" thread.
[Symbols] Attempting to download symbols for unknown OS version 10.0.22000.348.
[Symbols] Downloading to "C:\Users\root\AppData\Roaming\ExplorerPatcher\".
[Symbols] Downloading symbols for "C:\Windows\system32\twinui.pcshell.dll"...
[Symbols] Symbols for "C:\Windows\system32\twinui.pcshell.dll" are not available - unable to download.
[Symbols] Please refer to "https://github.com/valinet/ExplorerPatcher/wiki/Symbols" for more information.
Not directly in ExplorerPatcher, but you can set up a system proxy and that should be picked up.
Although not recommended, you can uncheck this option: "Properties" - "Advanced" - "Enable symbols download *".
Yes and no. We can discuss this more in a separate thread, but to keep it short here:
- For things in
twinui.pcshell.dll, there would be 2 alternatives: reimplementing the functions used from there, like aWin+Xmenu builder, code that adds and removes owner drawing for contextual menus etc, or shipping a version of the DLL with known symbol data with the program. The former is doable, although it's not on the roadmap and would take a while, while the latter would be much more practical, but it's kind of impossible since I do not have a license to redistribute Microsoft's binaries. - For things in
StartDocked.dll, it's actually much more complicated, since I need the symbols to know where to hook and modify the behavior of certain of its function in order to achieve the functionality described above. Much much harder to do without symbols, almost impossible I'd say.
For offering certain capabilities, using symbols is mostly the way to go. For example, Start11 reimplemented the whole Windows 11 Start menu on their own just so that they did not have to bother with this, plus the OC Windows 11 menu is pretty opaque and developing something from scratch is much easier than spending endless days trying to figure out some obfuscated instructions in Microsoft's binaries.
No, as I said, it just contains some pointers, some very succinct information about various things in a program. Symbols are usually generated by the compiler when the binary is created, and are usually distributed in one form or another with it by the vendor, although that's not always the case. Debugging or exploring an executable without symbol information is harder because you have to deduct from the instructions alone what that block of code might do. Although, it's not impossible to figure out what happens then either.
No, but they definitely help ease the process of making sense of what happens in an executable. Symbols are related to final executables, those that contain machine code, or instructions that the CPU executes directly. The source code is a much more rich functional description of how the program is to operate, and is consumed by a compiler which turns it into an executable, a machine-specific representation of the description in the source code. This transformation is not a bijection, in other words, it's an irreversible process: theoretically different source code can produce the same executable and you cannot obtain back the source code from the executable. Why the latter? Well, the executable lacks some of the original information; an easy example of that are the function parameters: the information about how many parameters a function takes is not contained in the executable, as it is not needed for successful execution. This information may be deducted from the context (the instructions themselves), but what you deduct is not guaranteed to have been how the original description in the source code looked like. Not to mention, executable files lack valuable information which you can usually find in the source code in the form of comments.
Things of interests in a program are generically called "symbols". What makes a thing something of interest, is, again, dependent on the context, but symbol files usually contain information about "symbols" such as functions, global variables etc.