Skip to content

Dynamic Linker

Jump to bottom Edit New page
WALDEMAR KOZACZUK edited this page Oct 18, 2020 · 31 revisions

Introduction

OSv is made of many components but the dynamic linker is probably the most essential one as it interacts with and ties all other components together and is responsible for bootstrapping an application. In essence, it involves locating ELF file on the filesystem, loading it into memory using mmap(), processing its headers and segments to relocate symbols, configuring TLS, executing its init functions, loading any dependant ELF objects and finally starting the app. Please note that unlike Linux, the dynamic linker is an integral part of the OSv kernel. Most of the dynamic linker code is located in core/elf.cc, arch/<arch>/arch-elf.cc and core/app.cc.

Concepts

  • app_registry

  • application - represents running program (_program member points to it)

  • elf::program - typically there is only one instance of it so effectively it is a singleton, but it is possible to create new programs for new ELF namespaces

    • symbol_module program::lookup(const char* name) - iterates over all objects in elf::program::modules_list and calls object::lookup_symbol(name) for each to finally return symbol_module for the first found occurrence

  • elf::object - represents an ELF object; implements logic to load an ELF file into memory and process its headers and relocations

    • void object::relocate_pltgot() -- ????; iterates over entries in DT_JMPREL and either calls object::arch_relocate_jump_slot() if bind_now or sets the jump slots (???) to resolve lazily later (PLT_GOT)
    • void* object::resolve_pltgot(unsigned index) - finds relocation info under dynamic_ptr<Elf64_Rela>(DT_JMPREL) and symbol index and finds symbol by calling object::symbol() and calls object::arch_relocate_jump_slot() to write the symbol`s relocated address
    • void object::relocate_rela() - iterates over the table of relocation entries (dynamic_ptr<Elf64_Rela>(DT_RELA)) and calls object::arch_relocate_rela() for each Elf64_Rela* and passes its relocation type (p->r_info & 0xffffffff), index in the symbol table of the object being relocated (p->r_info >> 32), address of the relocation (_base + p->r_offset, where write the relocation value to) and addend (p->r_addend)
    • bool object::arch_relocate_rela(u32 type, u32 sym, void *addr, Elf64_Sxword addend) - based on the relocation type (type argument) determines the relocation value (symbol relocated address or object module index or st_value (?) for TLS) and writes it to the relocation address (addr argument):
      • R_X86_64_COPY - calls object::symbol_other(sym) to find symbol in other objects
      • R_X86_64_64 - calls object::symbol(sym, true) to find symbol in all objects (see below) and calculates the value as symbol.relocated_addr() + addend
      • R_X86_64_RELATIVE - calculates the value as _base + addend
      • R_X86_64_JUMP_SLOT, R_X86_64_GLOB_DAT - calls object::symbol(sym, true) to find symbol in all objects (see below) and calculates the value as symbol.relocated_addr()
      • R_X86_64_DTPMOD64 - calls object::symbol(sym, true) to find symbol in all objects (see below) and calculates value as the module index of the object where symbol was found in; for STN_UNDEF uses index of this object
      • R_X86_64_DTPOFF64 - (TLS) ???
      • R_X86_64_TPOFF64 - (TLS)???
    • bool object::arch_relocate_jump_slot(u32 sym, void *addr, Elf64_Sxword addend, bool ignore_missing) - calls object::symbol(sym, true) to find symbol in all objects (see below) and writes symbol.relocated_addr() to the relocation jump slot address (addr argument)
    • symbol_module object::symbol(unsigned idx, bool ignore_missing) - entry point to symbol lookup; accepts symbol index, finds its name in the object symbols table (dynamic_ptr<Elf64_Sym>(DT_SYMTAB)) and searches for a symbol by name in all objects programs knows about by calling program::lookup(name); if symbol not found it aborts if ignore_missing is false otherwise just warns; returns symbol_module that is a tuple of the object the symbol resides and the symbol definition (Elf64_Sym *); called by following methods during relocation phase:
      • object::arch_relocate_rela()
      • object::arch_relocate_jump_slot()
      • object::resolve_pltgot()
    • Elf64_Sym* object::lookup_symbol(const char* name) - looks up symbol by name by delegating to either lookup_symbol_old or lookup_symbol_gnu; bails out if object not visible (during construction)
    • Elf64_Sym* object::lookup_symbol_old(const char* name) - ???
    • Elf64_Sym* object::lookup_symbol_gnu(const char* name) - uses GNU hashmap

  • elf::file

  • elf::memory_image

  • elf::symbol_module

Flow

The program::get_library() is the critical point where dynamic linker gets involved in instantiating new application.

The main program (kernel?) gets instantiated by elf::create_main_program() called from loader.cc

application::new_program() instantiates new program for new ELF namespace with new base address.

TLS (Thread Local Storage)

Thread local storage is what allows applications and shared libraries to use variables stored in memory specific to a given thread. These include variables marked with __thread and C++ thread_local modifiers. For TLS variables to work correctly, OSv dynamic linker needs to recognize TLS segments in an ELF file, construct static TLS blocks in memory, process relevant relocations and provide certain functions like __tls_get_addr among other things.

Before we delve into what OSv dynamic linker does to support TLS, it is important to understand two different formats of static TLS block layout - so-called Variant I and Variant II - and 4 different models of accessing TLS variables.

Static TLS block is an area of memory allocated for each thread independently intended to store thread-local variables and built from a template derived when loading the main application and its dependant libraries. The template, in essence, specifies the total size of the TLS block, the offsets in it for each ELF object including OSv kernel, and any initial values for the TLS variables in those objects. The static TLS does not change once the thread is created and running; that is why it is called 'static' after all. In Variant I (used in AArch64 port) the data is laid out from 'left-to-right' (local-exec for PIEs, kernel followed with other objects). In Variant II (used in X86_64 port) the data is laid out from 'right-to-left' which is exactly opposite to Variant I.

  • Variant I
    // (1) - TLS memory area layout with app as shared library
    // |------|--------------|-----|-----|-----|
    // |<NONE>|KERNEL        |SO_1 |SO_2 |SO_3 |
    // |------|--------------|-----|-----|-----|

    // (2) - TLS memory area layout with PIE or
    // position dependant executable
    // |------|--------------|-----|-----|
    // | EXE  |KERNEL        |SO_2 |SO_3 |
    // |------|--------------|-----|-----|
  • Variant II
    // (1) - TLS memory area layout with app shared library
    // |-----|-----|-----|--------------|------|
    // |SO_3 |SO_2 |SO_1 |KERNEL        |<NONE>|
    // |-----|-----|-----|--------------|------|

    // (2) - TLS memory area layout with pie or
    // position dependant executable
    //       |-----|-----|---------------------|
    //       |SO_3 |SO_2 |KERNEL        | EXE  |
    //       |-----|-----|--------------|------|

Flow

As OSv loads an ELF object (see core/elf.cc:void object::process_headers()), it detects any TLS segment and captures its size, alignment, and its address in memory. After std::shared_ptr<elf::object> program::load_object(..) loads ELF in memory (more specifically mmap-s it), OSv dynamic linker calls void object::init_static_tls() which in essence builds the TLS template for the main app object based on its own TLS segment and any of its dependencies. More specifically it calculates total TLS template size (stored in object::_initial_tls_size member variable) and iterates over its all dependant objects to copy their TLS data to object::_initial_tls buffer by delegating to the void object::prepare_initial_tls(..) or void object::prepare_local_tls(..) methods that are arch (AArch64 or X86_64) specific. At the same time, it also stores the offsets of TLS data for each object in object::_initial_tls_offsets vector. So at the end the init_static_tls() sets 3 key variables - object::_initial_tls, object::_initial_tls_size and object::_initial_tls_offsets.

References

Add a custom footer
Add a custom sidebar
Clone this wiki locally