Multilib and layer names on Linux #155

djdeath · 2019-03-04T14:36:47Z

I couldn't find any guidance on this topic, maybe I didn't look hard enough.

On Linux multiple ABIs can be supported by a given system, the typical example is x86_64 (64bit x86) & i686 (32bit x86).
Quite a few games are running in 32bit and so a layer needs to be compiled twice for each ABI that a given binary might pick up.

My current problem is that I would like the name of the layer to remain the same for both ABIs and have the loader pick the right one.
As far as I can tell installing a layer_x86_64.json and layer_i686.json with each pointing to different library to load but with the same name doesn't work.

Is there any guidance on what I can do to avoid having 2 different names for a layer that essentially the same just recompiled for a different architecture?
Is it a feature worth adding to the loader?

The text was updated successfully, but these errors were encountered:

shmerl · 2019-06-27T15:22:48Z

Any progress on this?

lenny-lunarg · 2019-06-28T20:51:39Z

We don't currently have an official mechanism to support both 32 and 64 bit copies of the layers or ICDs. That being said, it is possible to use both architectures for ICDs because the loader will just hit an error when loading the wrong ICD, and it will continue with the correct one. Based on the inital comment from @djdeath it sounds like that isn't working for layers. I would guess this is because the loader has logic to remove duplicate layers, so the loader is probably only finding one of the JSON files for each layer.

As far as adding support for both 64 and 32 bit layers co-existing, I think that sounds like a good thing, but it's been low priority (particularly as Linux 32-bit has never had the level of support of other configurations) and I haven't given it much thought. I see a few basic ways of accomplishing this:

Create a separate directory for layers that apply to a certain architecture. This is similar to how this is handled on Windows, where there are separe 64 and 32 bit registry locations
Use the naming scheme described by @djdeath
Add some sort of architecture field to the JSON file. This could either define one architecture for the whole JSON and .so combination, or it could be an object within the JSON file containing both .so files like:

"library_path": {
    "i686": "some/path.so",
    "x86_64": "another/path.so"
}

I think I'd lean towards some variation of the last bullet point.

Also, if anyone wants to drive this forward, this is the kind of thing that is probably worth bringing to the Vulkan SI group. I don't know if there's other people who are interested in this, but the same basic pattern applies to both layers and drivers.

shmerl · 2019-06-28T20:59:50Z

As I explained here: https://bugs.freedesktop.org/show_bug.cgi?id=109807
ICD loader for drivers a least currently supports explicit way to specify the config JSON, such as VK_ICD_FILENAMES, in which you can reference JSON that points to the .so for needed architecture. I.e. there is one JSON for x86_32 and one for x86_64. You can load one as needed for your application if you are trying to customize things. There is no way to specify the layer JSON like that. I.e. there is only a way to set the PATH for all of them like VK_LAYER_PATH, not for a specific one.

At least providing a specific one to the loader can be a way to do it, until a more general solution is available. I.e. something like VK_LAYER_FILENAMES?

I think in general, support for this can be useful not for 32-bit Linux, but for 32-bit games on Linux, that would rely on 32-bit Vulkan driver (mostly for Wine+dxvk/d9vk case for older 32-bit Windows games, since it's highly unlikely anyone would make native 32-bit games for Linux these days, and old native ones aren't using Vulkan).

pchome · 2019-08-01T04:21:52Z

Maybe I misunderstood the problem, but I don't have one on Linux w/ multilib and a single Vulkan layer name.

Anyway, if this will help someone, I use the following options:

Installation

install x86 and x86_64 libraries system-wide, e.g. either
- into /usr/lib32 and /usr/lib64
- into /usr/lib32/<package> and /usr/lib64/<package>, and
  
  add paths to ld.so.conf
  
  add /etc/ld.so.conf.d/<package>.conf with two lines
  /usr/lib32/<package>
  /usr/lib64/<package>
  and use sudo ldconfig to regenerate cache
install into /your/x86/path/ and /your/x86_64/path, then
use LD_LIBRARY_PATH="/your/x86/path/:/your/x86_64/path" explicitly
(or add to user's env)

For my own needs I have the LD_LIBRARY_PATH="${HOME}/.local/lib64:${HOME}/.local/lib" environment variable set in ~/.profile.

Configure

So, when libraries are installed into one of mentioned paths, I can use single Vulkan layer manifest file. Just not specifying the path, neither full nor relative, just library name.

{
...
  "layer": {
    "name": "VK_LAYER_MY_test",
    "library_path": "libVkLayer_test.so"
...
}

Paths where loader looking for manifests are known

/etc/vulkan/{explicit,implicit}_layer.d/
/usr/local/etc/vulkan/{explicit,implicit}_layer.d/
/usr/share/vulkan/{explicit,implicit}_layer.d/
/usr/local/share/vulkan/{explicit,implicit}_layer.d/
$HOME/.local/share/vulkan/{explicit,implicit}_layer.d/ ($XDG_DATA_HOME/vulkan/{explicit,implicit}_layer.d/)

Now I have x86 and x86_64 libs named libVkLayer_test.so, and Vulkan Layer name VK_LAYER_MY_test.

By running e.g. VK_INSTANCE_LAYERS="VK_LAYER_MY_test" wine Game_x86.exe loader should load 32 bit layer.

In addition

vkconfig utility, which helps a lot.

Actually, it set's the implicit meta-layer, so this thing (or similar) could be used directly:

$XDG_DATA_HOME/vulkan/implicit_layer.d/VkLayer_override.json

{
    "file_format_version": "1.1.2",
    "layer": {
        "api_version": "1.1.106",
        "blacklisted_layers": [
            "VK_LAYER_MY_test_bad"
        ],
        "component_layers": [
            "VK_LAYER_MY_test"
        ],
        "description": "LunarG Override Layer",
        "disable_environment": {
            "DISABLE_VK_LAYER_LUNARG_override": "1"
        },
        "implementation_version": "1",
        "name": "VK_LAYER_LUNARG_override",
        "override_paths": [
        ],
        "type": "GLOBAL"
    }
}

With such example my fake VK_LAYER_MY_test_bad explicit layer shouldn't load at all, but my fake VK_LAYER_MY_test explicit layer should be loaded for any Vulkan application. Also, note override_paths, likely should work as append if all mentioned manifest paths and some custom paths will be added to this list.

shmerl · 2019-08-01T04:28:48Z

Looks like this requires placing .so for layers in the fixed location though. Any customization already won't work. The idea is to allow for JSON to point to a specific location for .so, and to be able to use one JSON for 32-bit case, and one for 64-bit case.

pchome · 2019-08-01T05:28:33Z

A couple of quick questions (based on example above), which maybe don't worth separate issues:

Trying to be more strict in Vulkan layer naming, but don't want to register VENDOR/AUTHOR ID or FQDN. Can I use "neutral" ID, like MY / CUSTOM / COMMUITY? Or maybe I should (at least) set up GitHub Pages, and use e.g. io_github_pchome_repo_name (can I)?
According to this guide: extensions-naming-conventions
Thinking about to split a Vulkan layer into several small layers, I want to make parts as separate implicit layers disabled by default, but each should be enabled (triggered) by setting corresponding environment variable to any value (not just to "1"), then this value will be used by the layer. Also I want to keep a single name for the layer, so likely should be an meta-layer enabled by default.
Is something like described above possible? Or maybe I trying to do something completely wrong?

pchome · 2019-08-01T05:43:31Z

@shmerl
I don't know what "customization" you mean, but e.g.

export LD_LIBRARY_PATH="${PWD}/x86/path/:${PWD}/x86_64/path"
export VK_LAYER_PATH="/path/to/manifest/"
export VK_INSTANCE_LAYERS="VK_LAYER_YOUR_layer_name"
./run_any_bit

should just work. (Note: ${PWD} just for "not fixed" location 😉 )

shmerl · 2019-08-01T05:50:12Z

I guess it might work, if you set LD_LIBRARY_PATH depending on 32-bit or 64-bit case. Mixing them up like that probably isn't a good idea.

kira-bruneau · 2020-07-06T03:33:23Z

@pchome That works, but it's not a great general solution. If you install a layer (eg. vkBasalt), you shouldn't have to specify LD_LIBRARY_PATH to use the layer.

Usually you can store the libraries in the respective 32bit and 64bit library paths to avoid needing LD_LIBRARY_PATH, but on NixOS, there are no default library search paths.

I've specifically run into this issue packaging vkBasalt on NixOS: NixOS/nixpkgs#92401.

smcv · 2020-12-01T15:34:55Z

I would guess this is because the loader has logic to remove duplicate layers, so the loader is probably only finding one of the JSON files for each layer.

See also #524 and https://gitlab.steamos.cloud/steamrt/steam-runtime-tools/-/issues/39.

It seems as though one simple way to avoid this would be to either stop removing duplicate layers at all (more like the behaviour seen for ICDs), or deduplicate by unique library_path or unique absolute path to the JSON file instead of deduplicating by unique name.

If you install a layer (eg. vkBasalt), you shouldn't have to specify LD_LIBRARY_PATH to use the layer.

This. In pressure-vessel (the container framework that the steam-runtime-tools issue talks about), we would prefer to be very careful about what we do and don't put on the LD_LIBRARY_PATH, to minimize hard-to-debug side-effects: altering the process-global library search path affects anything that uses dlopen(), not just Vulkan.

Some layers like MangoHud bypass this by using glibc's special ${PLATFORM} or ${LIB} tokens in their library_path (see ld.so(8)), but that has its own problems if you are writing tools like pressure-vessel that need to remap paths from one namespace to another and work in a cross-distro way, because different Linux distributions' glibc versions will look for different values for ${PLATFORM} and ${LIB} - so if we write out a remapped JSON file that mentions ${LIB}, we can't know where we need to put the actual libraries to get them to be loaded.

"library_path": {
"i686": "some/path.so",
"x86_64": "another/path.so"
}

I would suggest that this is a can of worms that Vulkan-Loader should try hard not to open, because it's a lot less simple than it appears at first glance.

Every Linux distribution and every build system seems to have a subtly different vocabulary of names for architectures and ABIs, and unfortunately it's not valid to assume that CPU types map 1:1 to ABIs. Even if you assume GNU/Linux (as in Linux with glibc), 64-bit x86 CPUs have at least the x86_64 and x32 ABIs, 32-bit ARM CPUs have OABI, EABI softfloat and EABI hardfloat, and MIPS has lots.

Here are some of the vocabularies I'm aware of:

Linux uname -m
- 32-bit PCs share an ABI but can be anything from i386 to i686
- 32-bit ARM can be called things like armv5tel or armv6bor armv7l, or even armv8l (denoting aarch64/ARM64 in 32-bit mode)
- conversely, ARM OABI, EABI softfloat and EABI hardfloat can have the same uname -m with incompatible ABIs
glibc ${PLATFORM}
- usually the same as uname -m but I don't think that's guaranteed
- inherits many of the same problems as uname -m
GNU Autotools, config.guess, config.sub
- tuples like i686-pc-linux-gnu, sometimes with the second (vendor) part omitted
- multiple GNU tuples often map to the same ABI: i386-linux-gnu is the same ABI as i686-linux-gnu, and arm-linux-gnueabi is the same ABI as armv5tel-linux-gnueabi
- the first (CPU) part is often but not always the same as uname -m
CMake ${CMAKE_SYSTEM_PROCESSOR}
- seems to be basically uname -m on at least Linux, but doesn't explicitly document that
Meson host_machine.cpu_family
- mostly similar to Linux uname -m, but 32-bit PCs are x86, and endianness is ignored
dpkg architecture names
- often the same as uname -m, but sometimes not, such as arm64 and amd64
RPM architecture names
- mostly the same as uname -m, but I think also sometimes not, similar to dpkg
Debian multiarch
- normalized GNU tuples like i386-linux-gnu (always i386-, never i686-) and arm-linux-gnueabihf (always arm-, never armv5tel-) that map 1:1 to ABIs
- on non-Debian-derived systems it isn't obvious how to figure out what the multiarch tuple should be
and so on...

If portability to non-GNU Linux (Android, musl, uclibc, etc.) is a goal, or if portability to non-Linux Unix platforms like *BSD is a goal, then it just gets worse.

I think Debian multiarch is probably the most comprehensive and least ambiguous of the various ways to name ABIs, but perhaps that's just my Debian-developer bias showing. It certainly seems like a source of complexity that would be better avoided if possible!

Following that proposal in, for example, Debian (which fully supports 9 ABIs and partially supports 13 more) would also require the JSON manifest to list 22 different library paths, because to avoid breaking the system when cross-compiling, files in /usr/share need to be the same for any pair of architectures. I'm sure the maintainers of "addon" layers like vkBasalt wouldn't particularly enjoy a series of merge requests like "please add riscv64 support to JSON manifest" as everyone adds their pet architecture.

Linux can support multiple ABIs, but Vulkan Layer manifest does not allow to specify different library paths, based on the ABI. As a solution, for ICDs, we can simply create multiple manifests, one per ABI, and the Loader will try them one by one until it finds the library that is compatible with the executable class. Instead, for Vulkan Layers, this method doesn't work because the Loader will discard the manifests that have a duplicated layer name. To add support for multiple ABIs to Vulkan Layers, and to make the behavior similar to the ICDs, with this commit we remove the duplicated layer name check. Fixes: KhronosGroup#155 Signed-off-by: Ludovico de Nittis <[email protected]>

Linux can support multiple ABIs, but Vulkan Layer manifest does not allow to specify different library paths, based on the ABI. As a solution, for ICDs, we can simply create multiple manifests, one per ABI, and the Loader will try them one by one until it finds the library that is compatible with the executable class. Instead, for Vulkan Layers, this method doesn't work because the Loader will discard the manifests that have a duplicated layer name. To add support for multiple ABIs to Vulkan Layers, and to make the behavior similar to the ICDs, with this commit we remove the duplicated layer name check. Instead we add to the output list all the Vulkan Layers that we find, and only when we are actually going to dlopen them, we discard the layers with the same name that we already successfully opened. Fixes: KhronosGroup#155 Signed-off-by: Ludovico de Nittis <[email protected]>

Linux can support multiple ABIs, but Vulkan Layer manifest does not allow to specify different library paths, based on the ABI. As a solution, for ICDs, we can simply create multiple manifests, one per ABI, and the Loader will try them one by one until it finds the library that is compatible with the executable class. Instead, for Vulkan Layers, this method doesn't work because the Loader will discard the manifests that have a duplicated layer name. To add support for multiple ABIs to Vulkan Layers, and to make the behavior similar to the ICDs, with this commit we remove the duplicated layer name check. Instead we add to the output list all the Vulkan Layers that we find, and only when we are actually going to dlopen them, we discard the layers with the same name that we already successfully opened. Fixes: #155 Signed-off-by: Ludovico de Nittis <[email protected]>

KarenGhavam-lunarG added the enhancement New feature or request label Mar 28, 2019

lenny-lunarg added this to the P2 milestone Apr 3, 2020

lenny-lunarg assigned charles-lunarg Apr 3, 2020

DadSchoorse mentioned this issue Nov 30, 2020

loader: skip layer if open_library fails #524

Closed

RyuzakiKK mentioned this issue Dec 1, 2020

loader: Do not remove duplicated layers #525

Merged

DadSchoorse mentioned this issue Jan 8, 2021

multiple issues with VK_LAYER_AMD_switchable_graphics GPUOpen-Drivers/AMDVLK#195

Closed

lenny-lunarg closed this as completed in #525 Jan 18, 2021

smcv mentioned this issue Mar 2, 2021

Proton 5.13-2 tries to use mesa/ANV before nVidia (Optimus laptops) ValveSoftware/steam-runtime#312

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multilib and layer names on Linux #155

Multilib and layer names on Linux #155

djdeath commented Mar 4, 2019 •

edited

Loading

shmerl commented Jun 27, 2019

lenny-lunarg commented Jun 28, 2019

shmerl commented Jun 28, 2019 •

edited

Loading

pchome commented Aug 1, 2019

shmerl commented Aug 1, 2019 •

edited

Loading

pchome commented Aug 1, 2019 •

edited

Loading

pchome commented Aug 1, 2019

shmerl commented Aug 1, 2019

kira-bruneau commented Jul 6, 2020 •

edited

Loading

smcv commented Dec 1, 2020 •

edited

Loading

Multilib and layer names on Linux #155

Multilib and layer names on Linux #155

Comments

djdeath commented Mar 4, 2019 • edited Loading

shmerl commented Jun 27, 2019

lenny-lunarg commented Jun 28, 2019

shmerl commented Jun 28, 2019 • edited Loading

pchome commented Aug 1, 2019

Installation

Configure

In addition

shmerl commented Aug 1, 2019 • edited Loading

pchome commented Aug 1, 2019 • edited Loading

pchome commented Aug 1, 2019

shmerl commented Aug 1, 2019

kira-bruneau commented Jul 6, 2020 • edited Loading

smcv commented Dec 1, 2020 • edited Loading

djdeath commented Mar 4, 2019 •

edited

Loading

shmerl commented Jun 28, 2019 •

edited

Loading

shmerl commented Aug 1, 2019 •

edited

Loading

pchome commented Aug 1, 2019 •

edited

Loading

kira-bruneau commented Jul 6, 2020 •

edited

Loading

smcv commented Dec 1, 2020 •

edited

Loading