Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mosquitto | Unsupported system call setresgid #44

Open
dzygann opened this issue Jan 18, 2022 · 23 comments
Open

Mosquitto | Unsupported system call setresgid #44

dzygann opened this issue Jan 18, 2022 · 23 comments

Comments

@dzygann
Copy link

dzygann commented Jan 18, 2022

Hi,

I got the following error, if I try to run the gsc container:

[P1:T1:mosquitto] trace: ---- shim_geteuid() = 0x0
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/etc/nsswitch.conf", 0x25a027630, 0) = 0x0
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/", 0x25a027750, 0) = 0x0
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/etc/nsswitch.conf", O_RDONLY|0x80000, 0000) = 0x4
[P1:T1:mosquitto] trace: ---- shim_newfstatat(4, "", 0x25a027550, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_read(4, 0x258342d70, 0x2000) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x1ee
[P1:T1:mosquitto] trace: ---- shim_read(4, 0x258342d70, 0x2000) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_newfstatat(4, "", 0x25a027630, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_close(4) = 0x0
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/etc/passwd", O_RDONLY|0x80000, 0000) = 0x4
[P1:T1:mosquitto] trace: ---- shim_newfstatat(4, "", 0x25a0276b0, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_lseek(4, 0x0, SEEK_SET) = 0x0
[P1:T1:mosquitto] trace: ---- shim_read(4, 0x258342d70, 0x2000) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x3d8
[P1:T1:mosquitto] trace: ---- shim_close(4) = 0x0
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/proc/sys/kernel/ngroups_max", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/etc/nsswitch.conf", 0x25a027690, 0) = 0x0
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/", 0x25a0277b0, 0) = 0x0
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/etc/nsswitch.conf", O_RDONLY|0x80000, 0000) = 0x4
[P1:T1:mosquitto] trace: ---- shim_newfstatat(4, "", 0x25a0275b0, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_read(4, 0x258342d70, 0x2000) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x1ee
[P1:T1:mosquitto] trace: ---- shim_read(4, 0x258342d70, 0x2000) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_newfstatat(4, "", 0x25a027690, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_close(4) = 0x0
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/etc/nsswitch.conf", 0x25a027690, 0) = 0x0
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/", 0x25a0277b0, 0) = 0x0
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/etc/nsswitch.conf", O_RDONLY|0x80000, 0000) = 0x4
[P1:T1:mosquitto] trace: ---- shim_newfstatat(4, "", 0x25a0275b0, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_read(4, 0x258342d70, 0x2000) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x1ee
[P1:T1:mosquitto] trace: ---- shim_read(4, 0x258342d70, 0x2000) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_newfstatat(4, "", 0x25a027690, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_close(4) = 0x0
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/etc/group", O_RDONLY|0x80000, 0000) = 0x4
[P1:T1:mosquitto] trace: ---- shim_newfstatat(4, "", 0x25a027240, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_lseek(4, 0x0, SEEK_SET) = 0x0
[P1:T1:mosquitto] trace: ---- shim_read(4, 0x258342d70, 0x2000) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x1cf
[P1:T1:mosquitto] trace: ---- shim_read(4, 0x258342d70, 0x2000) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_close(4) = 0x0
[P1:T1:mosquitto] trace: ---- shim_setgroups(1, 0x25a8ffce0) = 0x0
[P1:T1:mosquitto] warning: Unsupported system call setresgid
[P1:T1:mosquitto] trace: ---- shim_time(0) = 0x61e60300
[P1:T1:mosquitto] debug: ---- shim_exit_group (returning 1)
[P1:T1:mosquitto] debug: clearing POSIX locks for pid 1
[P1:T1:mosquitto] debug: sync client shutdown: closing handles
[P1:T1:mosquitto] debug: sync client shutdown: waiting for confirmation
[P1:T1:mosquitto] debug: sync client shutdown: finished
[P1:shim] debug: IPC worker: exiting worker thread
[P1:T1:mosquitto] debug: process 1 exited with status 1
debug: DkProcessExit: Returning exit code 1

This is my dockerfile:

From ubuntu:20.04

ENV LD_LIBRARY_PATH = "${LD_LIBRARY_PATH}:/usr/lib/x86_64-linux-gnu/"

RUN apt-get update -y && apt-get install -y \
        openssl \
        mosquitto \
        apt-utils 


EXPOSE 1883
EXPOSE 8883

COPY examples/mosquitto/ /etc/mosquitto/

ENTRYPOINT ["mosquitto"]

CMD ["-c", "/etc/mosquitto/mosquitto.conf", "-v"]

What is confusing me about this issue is that it is working fine if I run it in gramine-sgx. I don't run into the missing setresgid warning. What's the difference between them?

The example works if I don't use the -c /etc/mosquitto/mosquitto.conf argument. For some reason, this is causing the issue. mosquitto takes a default configuration, if the argument is missing.

I need the mosquitto.conf, because I want to add TLS to mosquitto and here is the place where to put the paths for the certificates.

Thanks in advance.

@dimakuv
Copy link

dimakuv commented Jan 18, 2022

What is confusing me about this issue is that it is working fine if I run it in gramine-sgx. I don't run into the missing setresgid warning. What's the difference between them?

You probably do run into this warning because setresgid() syscall is indeed not implemented in Gramine: https://github.com/gramineproject/gramine/blob/b2055ab935bd410829b262ef241b13e487fb3382/LibOS/shim/src/arch/x86_64/shim_table.c#L137

But since bare-metal gramine-sgx continues further, I don't think that the problem is in setresgid(). Mosquitto probably doesn't care about the result of this system call.

What actually happens, I think, is that your GSC Docker image is not exactly correct. You need to examine the original Docker image and the GSC-built Docker image.

  1. Are you sure your original Docker image (with the Dockerfile that you showed) actually works?
  2. Are you sure your Gramine log inside GSC doesn't contain any other error messages (you showed us only the last few dozen lines)?
  3. I suggest to get into your GSC container via docker run -it --entrypoint /bin/bash gsc-<your-mosquitto-image> and analyze the resulting .manifest.sgx file carefully -- you probably have something wrong there.
  4. Can you show your gsc build command and also the manifest file you're using with this command?

@dzygann
Copy link
Author

dzygann commented Jan 18, 2022

Hi @dimakuv

Are you sure your original Docker image (with the Dockerfile that you showed) actually works?

Yes, I double-checked it :)

Are you sure your Gramine log inside GSC doesn't contain any other error messages (you showed us only the last few dozen lines)?

Do I have overlooked sth?

warning: 'loader.preload' is deprecated; please switch to 'loader.entrypoint'
warning: DkVirtualMemoryProtect is unimplemented in Linux-SGX PAL
[::] debug: Gramine was built from commit: 32d277811b80e5884163994cf8e44ad59b35fd99
[::] debug: Host: Linux-SGX
[::] debug: LibOS xsave_enabled 1, xsave_size 0x340(832), xsave_features 0x7
[::] debug: Initial VMA region 0x1fd2b4000-0x1fd552000 (LibOS) bookkeeped
[::] debug: Initial VMA region 0x1ffcb4000-0x200000000 (manifest) bookkeeped
[::] debug: ASLR top address adjusted to 0xdf592000
[::] debug: Shim loaded at 0x1fd2b4000, ready to initialize
[::] debug: Mounting root as chroot filesystem: from file:/ to /
[::] debug: Mounting special proc filesystem: /proc
[::] debug: Mounting special dev filesystem: /dev
[::] debug: Mounting terminal device /dev/tty under /dev
[P1:T1:mosquitto] debug: Creating pipe: pipe.srv:6518a1a74f6958b6b08730d2446ce0a07a9021aaa120475f199ab9dc46b9f38f
[P1:T1:mosquitto] debug: Allocating stack at 0 (size = 2097152)
[P1:T1:mosquitto] debug: loading "file://mosquitto"
[P1:T1:mosquitto] debug: append_r_debug: adding file://mosquitto at 0xdf34b000
[P1:T1:mosquitto] debug: find_interp: searching for interpreter: /gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/ld-linux-x86-64.so.2
[P1:T1:mosquitto] debug: loading "file://gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/ld-linux-x86-64.so.2"
[P1:T1:mosquitto] debug: append_r_debug: adding file://gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/ld-linux-x86-64.so.2 at 0xdf11a000
[P1:T1:mosquitto] debug: Creating pipe: pipe.srv:f38079a31613587cdaa0ba57d7454c7568bdabaecd2a2ce657604621d394b1e5
[P1:T1:mosquitto] debug: Creating pipe: pipe.srv:1
[P1:T1:mosquitto] debug: Shim process initialized
[P1:shim] debug: IPC worker started
[P1:T1:mosquitto] debug: append_r_debug: adding file:[vdso_libos] at 0xdf119000
[P1:T1:mosquitto] trace: ---- shim_brk(0) = 0xe062d000
[P1:T1:mosquitto] trace: ---- shim_uname(0xdf591820) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mmap(0, 0x2000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdf117000
[P1:T1:mosquitto] trace: ---- shim_access("/etc/ld.so.preload", F_OK|R_OK) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/glibc-hwcaps/x86-64-v3/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/glibc-hwcaps/x86-64-v3", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/glibc-hwcaps/x86-64-v2/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/glibc-hwcaps/x86-64-v2", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/tls/haswell/x86_64/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/tls/haswell/x86_64", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/tls/haswell/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/tls/haswell", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/tls/x86_64/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/tls/x86_64", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/tls/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/tls", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/haswell/x86_64/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/haswell/x86_64", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/haswell/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/haswell", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/x86_64/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/x86_64", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc", 0xdf590e50, 0) = 0x0
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libfakeroot/glibc-hwcaps/x86-64-v3/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libfakeroot/glibc-hwcaps/x86-64-v3", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libfakeroot/glibc-hwcaps/x86-64-v2/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libfakeroot/glibc-hwcaps/x86-64-v2", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libfakeroot/tls/haswell/x86_64/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libfakeroot/tls/haswell/x86_64", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libfakeroot/tls/haswell/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libfakeroot/tls/haswell", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libfakeroot/tls/x86_64/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libfakeroot/tls/x86_64", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libfakeroot/tls/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libfakeroot/tls", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libfakeroot/haswell/x86_64/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libfakeroot/haswell/x86_64", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libfakeroot/haswell/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libfakeroot/haswell", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libfakeroot/x86_64/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libfakeroot/x86_64", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libfakeroot/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libfakeroot", 0xdf590e50, 0) = 0x0
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/local/lib/glibc-hwcaps/x86-64-v3/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/local/lib/glibc-hwcaps/x86-64-v3", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/local/lib/glibc-hwcaps/x86-64-v2/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/local/lib/glibc-hwcaps/x86-64-v2", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/local/lib/tls/haswell/x86_64/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/local/lib/tls/haswell/x86_64", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/local/lib/tls/haswell/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/local/lib/tls/haswell", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/local/lib/tls/x86_64/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/local/lib/tls/x86_64", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/local/lib/tls/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/local/lib/tls", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/local/lib/haswell/x86_64/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/local/lib/haswell/x86_64", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/local/lib/haswell/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/local/lib/haswell", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/local/lib/x86_64/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/local/lib/x86_64", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/local/lib/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/local/lib", 0xdf590e50, 0) = 0x0
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v3/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v3", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v2/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v2", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/lib/x86_64-linux-gnu/tls/haswell/x86_64/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/lib/x86_64-linux-gnu/tls/haswell/x86_64", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/lib/x86_64-linux-gnu/tls/haswell/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/lib/x86_64-linux-gnu/tls/haswell", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/lib/x86_64-linux-gnu/tls/x86_64/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/lib/x86_64-linux-gnu/tls/x86_64", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/lib/x86_64-linux-gnu/tls/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/lib/x86_64-linux-gnu/tls", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/lib/x86_64-linux-gnu/haswell/x86_64/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/lib/x86_64-linux-gnu/haswell/x86_64", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/lib/x86_64-linux-gnu/haswell/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/lib/x86_64-linux-gnu/haswell", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/lib/x86_64-linux-gnu/x86_64/libwrap.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/lib/x86_64-linux-gnu/x86_64", 0xdf590e50, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libwrap.so.0", O_RDONLY|0x80000, 0000) = 0x3
[P1:T1:mosquitto] trace: ---- shim_read(3, 0xdf590fb8, 0x340) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x340
[P1:T1:mosquitto] trace: ---- shim_newfstatat(3, "", 0xdf590e50, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mmap(0, 0xbc20, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdf10b000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xdf10e000, 0x5000, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdf10e000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xdf113000, 0x2000, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x8000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdf113000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xdf115000, 0x2000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x9000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdf115000
[P1:T1:mosquitto] trace: ---- shim_close(3) = 0x0
[P1:T1:mosquitto] debug: glibc register library /lib/x86_64-linux-gnu/libwrap.so.0 loaded at 0xdf10b000
[P1:T1:mosquitto] debug: append_r_debug: adding file://lib/x86_64-linux-gnu/libwrap.so.0 at 0xdf10b000
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libsystemd.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libfakeroot/libsystemd.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/local/lib/libsystemd.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libsystemd.so.0", O_RDONLY|0x80000, 0000) = 0x3
[P1:T1:mosquitto] trace: ---- shim_read(3, 0xdf590f88, 0x340) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x340
[P1:T1:mosquitto] trace: ---- shim_newfstatat(3, "", 0xdf590e20, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mmap(0, 0xae4f0, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdf05c000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xdf06c000, 0x75000, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x10000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdf06c000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xdf0e1000, 0x25000, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x85000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdf0e1000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xdf106000, 0x4000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xa9000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdf106000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xdf10a000, 0x4f0, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_FIXED, -1, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdf10a000
[P1:T1:mosquitto] trace: ---- shim_close(3) = 0x0
[P1:T1:mosquitto] debug: glibc register library /lib/x86_64-linux-gnu/libsystemd.so.0 loaded at 0xdf05c000
[P1:T1:mosquitto] debug: append_r_debug: adding file://lib/x86_64-linux-gnu/libsystemd.so.0 at 0xdf05c000
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libdlt.so.2", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libfakeroot/libdlt.so.2", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/local/lib/libdlt.so.2", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libdlt.so.2", O_RDONLY|0x80000, 0000) = 0x3
[P1:T1:mosquitto] trace: ---- shim_read(3, 0xdf590f58, 0x340) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x340
[P1:T1:mosquitto] trace: ---- shim_newfstatat(3, "", 0xdf590df0, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mmap(0, 0x24000, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdf038000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xdf03f000, 0x11000, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdf03f000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xdf050000, 0x7000, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x18000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdf050000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xdf057000, 0x2000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1e000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdf057000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xdf059000, 0x3000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_FIXED, -1, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdf059000
[P1:T1:mosquitto] trace: ---- shim_close(3) = 0x0
[P1:T1:mosquitto] debug: glibc register library /lib/x86_64-linux-gnu/libdlt.so.2 loaded at 0xdf038000
[P1:T1:mosquitto] debug: append_r_debug: adding file://lib/x86_64-linux-gnu/libdlt.so.2 at 0xdf038000
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libpthread.so.0", O_RDONLY|0x80000, 0000) = 0x3
[P1:T1:mosquitto] trace: ---- shim_read(3, 0xdf590f28, 0x340) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x340
[P1:T1:mosquitto] trace: ---- shim_newfstatat(3, "", 0xdf590dc0, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mmap(0, 0x201028, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdee36000
[P1:T1:mosquitto] trace: ---- shim_mprotect(0xdee37000, 0x1ff000, PROT_NONE) ...
[P1:T1:mosquitto] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mmap(0xdf036000, 0x2000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdf036000
[P1:T1:mosquitto] trace: ---- shim_close(3) = 0x0
[P1:T1:mosquitto] debug: glibc register library /gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libpthread.so.0 loaded at 0xdee36000
[P1:T1:mosquitto] debug: append_r_debug: adding file://gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libpthread.so.0 at 0xdee36000
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libssl.so.1.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libfakeroot/libssl.so.1.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/local/lib/libssl.so.1.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libssl.so.1.1", O_RDONLY|0x80000, 0000) = 0x3
[P1:T1:mosquitto] trace: ---- shim_read(3, 0xdf590ef8, 0x340) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x340
[P1:T1:mosquitto] trace: ---- shim_newfstatat(3, "", 0xdf590d90, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mmap(0, 0x92930, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdeda3000
[P1:T1:mosquitto] trace: ---- shim_mprotect(0xdedbf000, 0x6a000, PROT_NONE) ...
[P1:T1:mosquitto] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mmap(0xdedbf000, 0x4f000, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdedbf000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xdee0e000, 0x1a000, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6b000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdee0e000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xdee29000, 0xd000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x85000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdee29000
[P1:T1:mosquitto] trace: ---- shim_close(3) = 0x0
[P1:T1:mosquitto] debug: glibc register library /lib/x86_64-linux-gnu/libssl.so.1.1 loaded at 0xdeda3000
[P1:T1:mosquitto] debug: append_r_debug: adding file://lib/x86_64-linux-gnu/libssl.so.1.1 at 0xdeda3000
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libcrypto.so.1.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libfakeroot/libcrypto.so.1.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/local/lib/libcrypto.so.1.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libcrypto.so.1.1", O_RDONLY|0x80000, 0000) = 0x3
[P1:T1:mosquitto] trace: ---- shim_read(3, 0xdf590ec8, 0x340) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x340
[P1:T1:mosquitto] trace: ---- shim_newfstatat(3, "", 0xdf590d60, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mmap(0, 0x2000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdeda1000
[P1:T1:mosquitto] trace: ---- shim_mmap(0, 0x2d5fa0, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdeacb000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xdeb43000, 0x19b000, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x78000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdeb43000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xdecde000, 0x91000, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x213000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdecde000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xded6f000, 0x2e000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2a3000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xded6f000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xded9d000, 0x3fa0, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_FIXED, -1, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xded9d000
[P1:T1:mosquitto] trace: ---- shim_close(3) = 0x0
[P1:T1:mosquitto] debug: glibc register library /lib/x86_64-linux-gnu/libcrypto.so.1.1 loaded at 0xdeacb000
[P1:T1:mosquitto] debug: append_r_debug: adding file://lib/x86_64-linux-gnu/libcrypto.so.1.1 at 0xdeacb000
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libanl.so.1", O_RDONLY|0x80000, 0000) = 0x3
[P1:T1:mosquitto] trace: ---- shim_read(3, 0xdf590e98, 0x340) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x340
[P1:T1:mosquitto] trace: ---- shim_newfstatat(3, "", 0xdf590d30, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mmap(0, 0x201028, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xde8c9000
[P1:T1:mosquitto] trace: ---- shim_mprotect(0xde8ca000, 0x1ff000, PROT_NONE) ...
[P1:T1:mosquitto] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mmap(0xdeac9000, 0x2000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdeac9000
[P1:T1:mosquitto] trace: ---- shim_close(3) = 0x0
[P1:T1:mosquitto] debug: glibc register library /gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libanl.so.1 loaded at 0xde8c9000
[P1:T1:mosquitto] debug: append_r_debug: adding file://gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libanl.so.1 at 0xde8c9000
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libdl.so.2", O_RDONLY|0x80000, 0000) = 0x3
[P1:T1:mosquitto] trace: ---- shim_read(3, 0xdf590e68, 0x340) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x340
[P1:T1:mosquitto] trace: ---- shim_newfstatat(3, "", 0xdf590d00, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mmap(0, 0x201028, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xde6c7000
[P1:T1:mosquitto] trace: ---- shim_mprotect(0xde6c8000, 0x1ff000, PROT_NONE) ...
[P1:T1:mosquitto] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mmap(0xde8c7000, 0x2000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xde8c7000
[P1:T1:mosquitto] trace: ---- shim_close(3) = 0x0
[P1:T1:mosquitto] debug: glibc register library /gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libdl.so.2 loaded at 0xde6c7000
[P1:T1:mosquitto] debug: append_r_debug: adding file://gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libdl.so.2 at 0xde6c7000
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libm.so.6", O_RDONLY|0x80000, 0000) = 0x3
[P1:T1:mosquitto] trace: ---- shim_read(3, 0xdf590e38, 0x340) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x340
[P1:T1:mosquitto] trace: ---- shim_newfstatat(3, "", 0xdf590cd0, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mmap(0, 0x2d50f8, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xde3f1000
[P1:T1:mosquitto] trace: ---- shim_mprotect(0xde4c6000, 0x1ff000, PROT_NONE) ...
[P1:T1:mosquitto] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mmap(0xde6c5000, 0x2000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xd4000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xde6c5000
[P1:T1:mosquitto] trace: ---- shim_close(3) = 0x0
[P1:T1:mosquitto] debug: glibc register library /gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libm.so.6 loaded at 0xde3f1000
[P1:T1:mosquitto] debug: append_r_debug: adding file://gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libm.so.6 at 0xde3f1000
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libwebsockets.so.15", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libfakeroot/libwebsockets.so.15", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/local/lib/libwebsockets.so.15", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libwebsockets.so.15", O_RDONLY|0x80000, 0000) = 0x3
[P1:T1:mosquitto] trace: ---- shim_read(3, 0xdf590e08, 0x340) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x340
[P1:T1:mosquitto] trace: ---- shim_newfstatat(3, "", 0xdf590ca0, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mmap(0, 0x5b940, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xde395000
[P1:T1:mosquitto] trace: ---- shim_mprotect(0xde3a1000, 0x4d000, PROT_NONE) ...
[P1:T1:mosquitto] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mmap(0xde3a1000, 0x36000, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xc000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xde3a1000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xde3d7000, 0x16000, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x42000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xde3d7000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xde3ee000, 0x3000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x58000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xde3ee000
[P1:T1:mosquitto] trace: ---- shim_close(3) = 0x0
[P1:T1:mosquitto] debug: glibc register library /lib/x86_64-linux-gnu/libwebsockets.so.15 loaded at 0xde395000
[P1:T1:mosquitto] debug: append_r_debug: adding file://lib/x86_64-linux-gnu/libwebsockets.so.15 at 0xde395000
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libc.so.6", O_RDONLY|0x80000, 0000) = 0x3
[P1:T1:mosquitto] trace: ---- shim_read(3, 0xdf590dd8, 0x340) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x340
[P1:T1:mosquitto] trace: ---- shim_newfstatat(3, "", 0xdf590c70, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mmap(0, 0x2000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xde393000
[P1:T1:mosquitto] trace: ---- shim_mmap(0, 0x3f0c30, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xddfa2000
[P1:T1:mosquitto] trace: ---- shim_mprotect(0xde185000, 0x200000, PROT_NONE) ...
[P1:T1:mosquitto] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mmap(0xde385000, 0x6000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1e3000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xde385000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xde38b000, 0x7c30, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_FIXED, -1, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xde38b000
[P1:T1:mosquitto] trace: ---- shim_close(3) = 0x0
[P1:T1:mosquitto] debug: glibc register library /gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libc.so.6 loaded at 0xddfa2000
[P1:T1:mosquitto] debug: append_r_debug: adding file://gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libc.so.6 at 0xddfa2000
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libnsl.so.1", O_RDONLY|0x80000, 0000) = 0x3
[P1:T1:mosquitto] trace: ---- shim_read(3, 0xdf590da8, 0x340) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x340
[P1:T1:mosquitto] trace: ---- shim_newfstatat(3, "", 0xdf590c40, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mmap(0, 0x216978, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xddd8b000
[P1:T1:mosquitto] trace: ---- shim_mprotect(0xddd9f000, 0x1ff000, PROT_NONE) ...
[P1:T1:mosquitto] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mmap(0xddf9e000, 0x2000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x13000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xddf9e000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xddfa0000, 0x1978, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_FIXED, -1, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xddfa0000
[P1:T1:mosquitto] trace: ---- shim_close(3) = 0x0
[P1:T1:mosquitto] debug: glibc register library /gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libnsl.so.1 loaded at 0xddd8b000
[P1:T1:mosquitto] debug: append_r_debug: adding file://gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libnsl.so.1 at 0xddd8b000
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/librt.so.1", O_RDONLY|0x80000, 0000) = 0x3
[P1:T1:mosquitto] trace: ---- shim_read(3, 0xdf590d78, 0x340) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x340
[P1:T1:mosquitto] trace: ---- shim_newfstatat(3, "", 0xdf590c10, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mmap(0, 0x201038, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xddb89000
[P1:T1:mosquitto] trace: ---- shim_mprotect(0xddb8a000, 0x1ff000, PROT_NONE) ...
[P1:T1:mosquitto] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mmap(0xddd89000, 0x2000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xddd89000
[P1:T1:mosquitto] trace: ---- shim_close(3) = 0x0
[P1:T1:mosquitto] debug: glibc register library /gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/librt.so.1 loaded at 0xddb89000
[P1:T1:mosquitto] debug: append_r_debug: adding file://gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/librt.so.1 at 0xddb89000
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/liblzma.so.5", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libfakeroot/liblzma.so.5", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/local/lib/liblzma.so.5", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/lib/x86_64-linux-gnu/liblzma.so.5", O_RDONLY|0x80000, 0000) = 0x3
[P1:T1:mosquitto] trace: ---- shim_read(3, 0xdf590d48, 0x340) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x340
[P1:T1:mosquitto] trace: ---- shim_newfstatat(3, "", 0xdf590be0, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mmap(0, 0x28108, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xddb60000
[P1:T1:mosquitto] trace: ---- shim_mprotect(0xddb63000, 0x24000, PROT_NONE) ...
[P1:T1:mosquitto] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mmap(0xddb63000, 0x18000, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xddb63000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xddb7b000, 0xb000, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xddb7b000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xddb87000, 0x2000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x26000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xddb87000
[P1:T1:mosquitto] trace: ---- shim_close(3) = 0x0
[P1:T1:mosquitto] debug: glibc register library /lib/x86_64-linux-gnu/liblzma.so.5 loaded at 0xddb60000
[P1:T1:mosquitto] debug: append_r_debug: adding file://lib/x86_64-linux-gnu/liblzma.so.5 at 0xddb60000
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/liblz4.so.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libfakeroot/liblz4.so.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/local/lib/liblz4.so.1", O_RDONLY|0x80000, 0000) = -2
warning: Disallowing access to file '/lib/x86_64-linux-gnu/liblz4.so.1'; file is not protected, trusted or allowed.
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/lib/x86_64-linux-gnu/liblz4.so.1", O_RDONLY|0x80000, 0000) = -13
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/lib/glibc-hwcaps/x86-64-v3/liblz4.so.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/lib/glibc-hwcaps/x86-64-v3", 0xdf590bb0, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/lib/glibc-hwcaps/x86-64-v2/liblz4.so.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/lib/glibc-hwcaps/x86-64-v2", 0xdf590bb0, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/lib/tls/haswell/x86_64/liblz4.so.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/lib/tls/haswell/x86_64", 0xdf590bb0, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/lib/tls/haswell/liblz4.so.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/lib/tls/haswell", 0xdf590bb0, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/lib/tls/x86_64/liblz4.so.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/lib/tls/x86_64", 0xdf590bb0, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/lib/tls/liblz4.so.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/lib/tls", 0xdf590bb0, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/lib/haswell/x86_64/liblz4.so.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/lib/haswell/x86_64", 0xdf590bb0, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/lib/haswell/liblz4.so.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/lib/haswell", 0xdf590bb0, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/lib/x86_64/liblz4.so.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/lib/x86_64", 0xdf590bb0, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/lib/liblz4.so.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/lib", 0xdf590bb0, 0) = 0x0
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "= /glibc-hwcaps/x86-64-v3/liblz4.so.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "= /glibc-hwcaps/x86-64-v2/liblz4.so.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "= /tls/haswell/x86_64/liblz4.so.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "= /tls/haswell/liblz4.so.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "= /tls/x86_64/liblz4.so.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "= /tls/liblz4.so.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "= /haswell/x86_64/liblz4.so.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "= /haswell/liblz4.so.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "= /x86_64/liblz4.so.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "= /liblz4.so.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v3/liblz4.so.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v3", 0xdf590bb0, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v2/liblz4.so.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v2", 0xdf590bb0, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/tls/haswell/x86_64/liblz4.so.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/tls/haswell/x86_64", 0xdf590bb0, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/tls/haswell/liblz4.so.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/tls/haswell", 0xdf590bb0, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/tls/x86_64/liblz4.so.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/tls/x86_64", 0xdf590bb0, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/tls/liblz4.so.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/tls", 0xdf590bb0, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/haswell/x86_64/liblz4.so.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/haswell/x86_64", 0xdf590bb0, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/haswell/liblz4.so.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/haswell", 0xdf590bb0, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/x86_64/liblz4.so.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/x86_64", 0xdf590bb0, 0) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/liblz4.so.1", O_RDONLY|0x80000, 0000) = 0x3
[P1:T1:mosquitto] trace: ---- shim_read(3, 0xdf590d18, 0x340) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x340
[P1:T1:mosquitto] trace: ---- shim_newfstatat(3, "", 0xdf590bb0, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mmap(0, 0x20060, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xddb3f000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xddb41000, 0x1a000, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xddb41000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xddb5b000, 0x3000, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xddb5b000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xddb5e000, 0x2000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1e000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xddb5e000
[P1:T1:mosquitto] trace: ---- shim_close(3) = 0x0
[P1:T1:mosquitto] debug: glibc register library /usr/lib/x86_64-linux-gnu/liblz4.so.1 loaded at 0xddb3f000
[P1:T1:mosquitto] debug: append_r_debug: adding file://usr/lib/x86_64-linux-gnu/liblz4.so.1 at 0xddb3f000
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libgcrypt.so.20", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libfakeroot/libgcrypt.so.20", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/local/lib/libgcrypt.so.20", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libgcrypt.so.20", O_RDONLY|0x80000, 0000) = 0x3
[P1:T1:mosquitto] trace: ---- shim_read(3, 0xdf590ce8, 0x340) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x340
[P1:T1:mosquitto] trace: ---- shim_newfstatat(3, "", 0xdf590b80, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mmap(0, 0x11dfc8, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdda21000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xdda2d000, 0xce000, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xc000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdda2d000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xddafb000, 0x3d000, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xda000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xddafb000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xddb38000, 0x7000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x116000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xddb38000
[P1:T1:mosquitto] trace: ---- shim_close(3) = 0x0
[P1:T1:mosquitto] debug: glibc register library /lib/x86_64-linux-gnu/libgcrypt.so.20 loaded at 0xdda21000
[P1:T1:mosquitto] debug: append_r_debug: adding file://lib/x86_64-linux-gnu/libgcrypt.so.20 at 0xdda21000
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libev.so.4", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libfakeroot/libev.so.4", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/local/lib/libev.so.4", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libev.so.4", O_RDONLY|0x80000, 0000) = 0x3
[P1:T1:mosquitto] trace: ---- shim_read(3, 0xdf590c88, 0x340) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x340
[P1:T1:mosquitto] trace: ---- shim_newfstatat(3, "", 0xdf590b20, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mmap(0, 0x2000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdda1f000
[P1:T1:mosquitto] trace: ---- shim_mmap(0, 0x12cd0, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdda0c000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xdda0f000, 0xa000, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdda0f000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xdda19000, 0x4000, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xd000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdda19000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xdda1d000, 0x2000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x10000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdda1d000
[P1:T1:mosquitto] trace: ---- shim_close(3) = 0x0
[P1:T1:mosquitto] debug: glibc register library /lib/x86_64-linux-gnu/libev.so.4 loaded at 0xdda0c000
[P1:T1:mosquitto] debug: append_r_debug: adding file://lib/x86_64-linux-gnu/libev.so.4 at 0xdda0c000
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libuv.so.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libfakeroot/libuv.so.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/local/lib/libuv.so.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libuv.so.1", O_RDONLY|0x80000, 0000) = 0x3
[P1:T1:mosquitto] trace: ---- shim_read(3, 0xdf590c58, 0x340) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x340
[P1:T1:mosquitto] trace: ---- shim_newfstatat(3, "", 0xdf590af0, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mmap(0, 0x306e8, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdd9db000
[P1:T1:mosquitto] trace: ---- shim_mprotect(0xdd9e4000, 0x26000, PROT_NONE) ...
[P1:T1:mosquitto] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mmap(0xdd9e4000, 0x1b000, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x9000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdd9e4000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xdd9ff000, 0xa000, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x24000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdd9ff000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xdda0a000, 0x2000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2e000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdda0a000
[P1:T1:mosquitto] trace: ---- shim_close(3) = 0x0
[P1:T1:mosquitto] debug: glibc register library /lib/x86_64-linux-gnu/libuv.so.1 loaded at 0xdd9db000
[P1:T1:mosquitto] debug: append_r_debug: adding file://lib/x86_64-linux-gnu/libuv.so.1 at 0xdd9db000
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libcap.so.2", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libfakeroot/libcap.so.2", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/local/lib/libcap.so.2", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libcap.so.2", O_RDONLY|0x80000, 0000) = 0x3
[P1:T1:mosquitto] trace: ---- shim_read(3, 0xdf590c28, 0x340) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x340
[P1:T1:mosquitto] trace: ---- shim_newfstatat(3, "", 0xdf590ac0, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mmap(0, 0x8158, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdd9d2000
[P1:T1:mosquitto] trace: ---- shim_mprotect(0xdd9d4000, 0x5000, PROT_NONE) ...
[P1:T1:mosquitto] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mmap(0xdd9d4000, 0x3000, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdd9d4000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xdd9d7000, 0x1000, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdd9d7000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xdd9d9000, 0x2000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdd9d9000
[P1:T1:mosquitto] trace: ---- shim_close(3) = 0x0
[P1:T1:mosquitto] debug: glibc register library /lib/x86_64-linux-gnu/libcap.so.2 loaded at 0xdd9d2000
[P1:T1:mosquitto] debug: append_r_debug: adding file://lib/x86_64-linux-gnu/libcap.so.2 at 0xdd9d2000
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libgpg-error.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libfakeroot/libgpg-error.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/local/lib/libgpg-error.so.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libgpg-error.so.0", O_RDONLY|0x80000, 0000) = 0x3
[P1:T1:mosquitto] trace: ---- shim_read(3, 0xdf590bf8, 0x340) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x340
[P1:T1:mosquitto] trace: ---- shim_newfstatat(3, "", 0xdf590a90, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mmap(0, 0x22260, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdd9af000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xdd9b3000, 0x13000, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdd9b3000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xdd9c6000, 0xa000, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdd9c6000
[P1:T1:mosquitto] trace: ---- shim_mmap(0xdd9d0000, 0x2000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x20000) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdd9d0000
[P1:T1:mosquitto] trace: ---- shim_close(3) = 0x0
[P1:T1:mosquitto] debug: glibc register library /lib/x86_64-linux-gnu/libgpg-error.so.0 loaded at 0xdd9af000
[P1:T1:mosquitto] debug: append_r_debug: adding file://lib/x86_64-linux-gnu/libgpg-error.so.0 at 0xdd9af000
[P1:T1:mosquitto] trace: ---- shim_mmap(0, 0x2000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdd9ad000
[P1:T1:mosquitto] trace: ---- shim_mmap(0, 0x2000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdd9ab000
[P1:T1:mosquitto] trace: ---- shim_arch_prctl(4098, 0xdd9abc00) = 0x0
[P1:T1:mosquitto] trace: ---- shim_set_tid_address(0xdd9abed0) = 0x1
[P1:T1:mosquitto] trace: ---- shim_set_robust_list(0xdd9abee0, 0x18) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mprotect(0xde385000, 0x4000, PROT_READ) ...
[P1:T1:mosquitto] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mprotect(0xdd9d0000, 0x1000, PROT_READ) ...
[P1:T1:mosquitto] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mprotect(0xdd9d9000, 0x1000, PROT_READ) ...
[P1:T1:mosquitto] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mprotect(0xdf036000, 0x1000, PROT_READ) ...
[P1:T1:mosquitto] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mprotect(0xde8c7000, 0x1000, PROT_READ) ...
[P1:T1:mosquitto] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mprotect(0xdda0a000, 0x1000, PROT_READ) ...
[P1:T1:mosquitto] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mprotect(0xdda1d000, 0x1000, PROT_READ) ...
[P1:T1:mosquitto] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mprotect(0xddb38000, 0x2000, PROT_READ) ...
[P1:T1:mosquitto] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mprotect(0xddb5e000, 0x1000, PROT_READ) ...
[P1:T1:mosquitto] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mprotect(0xddb87000, 0x1000, PROT_READ) ...
[P1:T1:mosquitto] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mprotect(0xddd89000, 0x1000, PROT_READ) ...
[P1:T1:mosquitto] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mprotect(0xddf9e000, 0x1000, PROT_READ) ...
[P1:T1:mosquitto] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mprotect(0xded6f000, 0x2c000, PROT_READ) ...
[P1:T1:mosquitto] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mprotect(0xdee29000, 0x9000, PROT_READ) ...
[P1:T1:mosquitto] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mprotect(0xde3ee000, 0x2000, PROT_READ) ...
[P1:T1:mosquitto] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mprotect(0xde6c5000, 0x1000, PROT_READ) ...
[P1:T1:mosquitto] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mprotect(0xdeac9000, 0x1000, PROT_READ) ...
[P1:T1:mosquitto] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mprotect(0xdf057000, 0x1000, PROT_READ) ...
[P1:T1:mosquitto] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mprotect(0xdf106000, 0x3000, PROT_READ) ...
[P1:T1:mosquitto] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mprotect(0xdf115000, 0x1000, PROT_READ) ...
[P1:T1:mosquitto] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mprotect(0xdf38f000, 0x1000, PROT_READ) ...
[P1:T1:mosquitto] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_mprotect(0xdf347000, 0x2000, PROT_READ) ...
[P1:T1:mosquitto] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_prlimit64(0, 3, 0, 0xdf5919b0) = 0x0
[P1:T1:mosquitto] trace: ---- shim_getrandom(0xde3904b8, 0x8, GRND_NONBLOCK) = 0x8
[P1:T1:mosquitto] trace: ---- shim_brk(0) = 0xe062d000
[P1:T1:mosquitto] trace: ---- shim_brk(0xe064e000) = 0xe064e000
[P1:T1:mosquitto] warning: Unsupported system call prctl
[P1:T1:mosquitto] warning: Unsupported system call prctl
[P1:T1:mosquitto] warning: Unsupported system call prctl
[P1:T1:mosquitto] warning: Unsupported system call prctl
[P1:T1:mosquitto] warning: Unsupported system call prctl
[P1:T1:mosquitto] warning: Unsupported system call prctl
[P1:T1:mosquitto] trace: ---- shim_gettimeofday(0xdf591a00, 0) = 0x0
[P1:T1:mosquitto] trace: ---- shim_socket(INET, STREAM, 0) = 0x3
[P1:T1:mosquitto] trace: ---- shim_futex(0xded9f818, FUTEX_PRIVATE|FUTEX_WAKE, 2147483647, 0, 0xe062d2f0, 2) ...
[P1:T1:mosquitto] trace: ---- return from shim_futex(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_futex(0xded9f80c, FUTEX_PRIVATE|FUTEX_WAKE, 2147483647, 0, 0xde38aee0, 0) ...
[P1:T1:mosquitto] trace: ---- return from shim_futex(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_futex(0xded9f804, FUTEX_PRIVATE|FUTEX_WAKE, 2147483647, 0, 0xca, 1) ...
[P1:T1:mosquitto] trace: ---- return from shim_futex(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_futex(0xded9f8d8, FUTEX_PRIVATE|FUTEX_WAKE, 2147483647, 0, 0xe062d480, 2) ...
[P1:T1:mosquitto] trace: ---- return from shim_futex(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_futex(0xded9f7f0, FUTEX_PRIVATE|FUTEX_WAKE, 2147483647, 0, 0xe062d480, 0) ...
[P1:T1:mosquitto] trace: ---- return from shim_futex(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_futex(0xded9f7e8, FUTEX_PRIVATE|FUTEX_WAKE, 2147483647, 0, 0xe062d480, 0) ...
[P1:T1:mosquitto] trace: ---- return from shim_futex(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_futex(0xded9cc5c, FUTEX_PRIVATE|FUTEX_WAKE, 2147483647, 0, 0xe06316f0, 2) ...
[P1:T1:mosquitto] trace: ---- return from shim_futex(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_futex(0xded9f6c4, FUTEX_PRIVATE|FUTEX_WAKE, 2147483647, 0, 0xe0631810, 2) ...
[P1:T1:mosquitto] trace: ---- return from shim_futex(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_futex(0xded9f65c, FUTEX_PRIVATE|FUTEX_WAKE, 2147483647, 0, 0xde38bac0, 1) ...
[P1:T1:mosquitto] trace: ---- return from shim_futex(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_futex(0xded9f650, FUTEX_PRIVATE|FUTEX_WAKE, 2147483647, 0, 0xe062d3f0, 2) ...
[P1:T1:mosquitto] trace: ---- return from shim_futex(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_futex(0xded9f7fc, FUTEX_PRIVATE|FUTEX_WAKE, 2147483647, 0, 0xe0631b10, 0) ...
[P1:T1:mosquitto] trace: ---- return from shim_futex(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_futex(0xded9f7b8, FUTEX_PRIVATE|FUTEX_WAKE, 2147483647, 0, 0xe0631b10, 1) ...
[P1:T1:mosquitto] trace: ---- return from shim_futex(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_futex(0xded9f7b0, FUTEX_PRIVATE|FUTEX_WAKE, 2147483647, 0, 0xe06316f0, 0) ...
[P1:T1:mosquitto] trace: ---- return from shim_futex(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/ssl/openssl.cnf", O_RDONLY, 0000) = 0x4
[P1:T1:mosquitto] trace: ---- shim_newfstatat(4, "", 0xdf591570, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_read(4, 0xe0649b10, 0x2000) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x2000
[P1:T1:mosquitto] trace: ---- shim_brk(0xe066f000) = 0xe064e000
[P1:T1:mosquitto] trace: ---- shim_mmap(0, 0x100000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0x0) ...
[P1:T1:mosquitto] trace: ---- return from shim_mmap(...) = 0xdd8ab000
[P1:T1:mosquitto] trace: ---- shim_read(4, 0xe0649b10, 0x2000) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0xa9d
[P1:T1:mosquitto] trace: ---- shim_read(4, 0xe0649b10, 0x2000) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_close(4) = 0x0
[P1:T1:mosquitto] trace: ---- shim_futex(0xded9f7e0, FUTEX_PRIVATE|FUTEX_WAKE, 2147483647, 0, 0xe064bef0, -530279488) ...
[P1:T1:mosquitto] trace: ---- return from shim_futex(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/etc/mosquitto/mosquitto.conf", O_RDONLY, 0000) = 0x4
[P1:T1:mosquitto] trace: ---- shim_newfstatat(4, "", 0xdf591290, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_read(4, 0xdd8acd70, 0x2000) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x15c
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/etc/mosquitto/conf.d", O_RDONLY|0x90800, 0000) = 0x5
[P1:T1:mosquitto] trace: ---- shim_newfstatat(5, "", 0xdf591370, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_getdents64(5, 0xdd8aedb0, 0x8000) = 0x88
[P1:T1:mosquitto] trace: ---- shim_getdents64(5, 0xdd8aedb0, 0x8000) = 0x0
[P1:T1:mosquitto] trace: ---- shim_close(5) = 0x0
[P1:T1:mosquitto] trace: ---- shim_time(0) = 0x61e6e799
[P1:T1:mosquitto] trace: ---- shim_write(2, 0xdf58ec90, 0x43) ...
[P1:T1:mosquitto] trace: ---- return from shim_write(...) = 0x43
1642522521: Loading config file /etc/mosquitto/conf.d/default.conf
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/etc/mosquitto/conf.d/default.conf", O_RDONLY, 0000) = 0x5
[P1:T1:mosquitto] trace: ---- shim_newfstatat(5, "", 0xdf591150, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_read(5, 0xdd8aed80, 0x2000) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x11b
[P1:T1:mosquitto] trace: ---- shim_read(5, 0xdd8aed80, 0x2000) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_close(5) = 0x0
[P1:T1:mosquitto] trace: ---- shim_read(4, 0xdd8acd70, 0x2000) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_close(4) = 0x0
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/var/lib/mosquitto/mosquitto.db", O_RDONLY, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_geteuid() = 0x0
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/etc/nsswitch.conf", 0xdf591610, 0) = 0x0
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/", 0xdf591730, 0) = 0x0
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/etc/nsswitch.conf", O_RDONLY|0x80000, 0000) = 0x4
[P1:T1:mosquitto] trace: ---- shim_newfstatat(4, "", 0xdf591530, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_read(4, 0xdd8acd70, 0x2000) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x1ee
[P1:T1:mosquitto] trace: ---- shim_read(4, 0xdd8acd70, 0x2000) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_newfstatat(4, "", 0xdf591610, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_close(4) = 0x0
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/etc/passwd", O_RDONLY|0x80000, 0000) = 0x4
[P1:T1:mosquitto] trace: ---- shim_newfstatat(4, "", 0xdf591690, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_lseek(4, 0x0, SEEK_SET) = 0x0
[P1:T1:mosquitto] trace: ---- shim_read(4, 0xdd8acd70, 0x2000) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x3d8
[P1:T1:mosquitto] trace: ---- shim_close(4) = 0x0
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/proc/sys/kernel/ngroups_max", O_RDONLY|0x80000, 0000) = -2
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/etc/nsswitch.conf", 0xdf591670, 0) = 0x0
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/", 0xdf591790, 0) = 0x0
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/etc/nsswitch.conf", O_RDONLY|0x80000, 0000) = 0x4
[P1:T1:mosquitto] trace: ---- shim_newfstatat(4, "", 0xdf591590, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_read(4, 0xdd8acd70, 0x2000) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x1ee
[P1:T1:mosquitto] trace: ---- shim_read(4, 0xdd8acd70, 0x2000) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_newfstatat(4, "", 0xdf591670, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_close(4) = 0x0
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/etc/nsswitch.conf", 0xdf591670, 0) = 0x0
[P1:T1:mosquitto] trace: ---- shim_newfstatat(AT_FDCWD, "/", 0xdf591790, 0) = 0x0
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/etc/nsswitch.conf", O_RDONLY|0x80000, 0000) = 0x4
[P1:T1:mosquitto] trace: ---- shim_newfstatat(4, "", 0xdf591590, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_read(4, 0xdd8acd70, 0x2000) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x1ee
[P1:T1:mosquitto] trace: ---- shim_read(4, 0xdd8acd70, 0x2000) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_newfstatat(4, "", 0xdf591670, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_close(4) = 0x0
[P1:T1:mosquitto] trace: ---- shim_openat(AT_FDCWD, "/etc/group", O_RDONLY|0x80000, 0000) = 0x4
[P1:T1:mosquitto] trace: ---- shim_newfstatat(4, "", 0xdf591220, 4096) = 0x0
[P1:T1:mosquitto] trace: ---- shim_lseek(4, 0x0, SEEK_SET) = 0x0
[P1:T1:mosquitto] trace: ---- shim_read(4, 0xdd8acd70, 0x2000) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x1cf
[P1:T1:mosquitto] trace: ---- shim_read(4, 0xdd8acd70, 0x2000) ...
[P1:T1:mosquitto] trace: ---- return from shim_read(...) = 0x0
[P1:T1:mosquitto] trace: ---- shim_close(4) = 0x0
[P1:T1:mosquitto] trace: ---- shim_setgroups(1, 0xe064cce0) = 0x0
[P1:T1:mosquitto] warning: Unsupported system call setresgid
[P1:T1:mosquitto] trace: ---- shim_time(0) = 0x61e6e799
[P1:T1:mosquitto] debug: ---- shim_exit_group (returning 1)
[P1:T1:mosquitto] debug: clearing POSIX locks for pid 1
[P1:T1:mosquitto] debug: sync client shutdown: closing handles
[P1:T1:mosquitto] debug: sync client shutdown: waiting for confirmation
[P1:T1:mosquitto] debug: sync client shutdown: finished
[P1:shim] debug: IPC worker: exiting worker thread
[P1:T1:mosquitto] debug: process 1 exited with status 1
debug: DkProcessExit: Returning exit code 1

I suggest to get into your GSC container via docker run -it --entrypoint /bin/bash gsc- and analyze the resulting .manifest.sgx file carefully -- you probably have something wrong there.

I will compare the one in the docker with the bare metal one. Try to figure out some strange differences.

Can you show your gsc build command and also the manifest file you're using with this command?

This is the command:

./gsc build -d --insecure-args ubuntu20.04-mqtt test/ubuntu20.04-mqtt.manifest && ./gsc sign-image ubuntu20.04-mqtt enclave-key.pem

and this is the manifest:

sgx.enclave_size = "8G"
sgx.thread_num = 16
sys.stack.size = "2M"

loader.pal_internal_mem_size = "128M"

sys.enable_sigterm_injection = true


sgx.allowed_files = [
  "file:/etc/hosts.allow",
  "file:/etc/hosts.deny",
  "file:/etc/mosquitto/certs/docker.key",
  "file:/etc/mosquitto/certs/docker.crt",

  "file:/var/log/mosquitto/mosquitto.log",
  "file:/var/lib/mosquitto/mosquitto.db",

  "file:/lib/x86_64-linux-gnu/libwrap.so.0",
  "file:/lib/x86_64-linux-gnu/libsystemd.so.0",
  "file:/lib/x86_64-linux-gnu/libdlt.so.2",
  "file:/lib/x86_64-linux-gnu/libssl.so.1.1",
  "file:/lib/x86_64-linux-gnu/libcrypto.so.1.1",
  "file:/lib/x86_64-linux-gnu/libwebsockets.so.15",
  "file:/lib/x86_64-linux-gnu/liblzma.so.5",
  "file:/lib/x86_64-linux-gnu/libgcrypt.so.20",
  "file:/lib/x86_64-linux-gnu/liblz4.so.1",
  "file:/lib/x86_64-linux-gnu/libev.so.4",
  "file:/lib/x86_64-linux-gnu/libuv.so.1",
  "file:/lib/x86_64-linux-gnu/libcap.so.2",
  "file:/lib/x86_64-linux-gnu/libgpg-error.so.0",
]

@dimakuv
Copy link

dimakuv commented Jan 18, 2022

I don't see anything special... Can you try running Linux PAL instead of Linux-SGX PAL? There is a switch in GSC to do this: https://gramine.readthedocs.io/projects/gsc/en/latest/#execute-with-linux-pal-instead-of-linux-sgx-pal

@dzygann
Copy link
Author

dzygann commented Jan 19, 2022

Can you try running Linux PAL instead of Linux-SGX PAL?

If I use this command

docker run --name gsc-mqtt-broker -p 1883:1883 -p 8883:8883 --device=/dev/sgx_enclave       -v /var/run/aesmd/aesm.socket:/var/run/aesmd/aesm.socket --env GSC_PAL=Linux  --security-opt seccomp=docker_seccomp.json    gsc-ubuntu20.04-mqtt -c /etc/mosquitto/mosquitto.conf -v

This doesn't change the behaviour of the docker.

If I run the command without the --security-opt and seccomp=docker_seccomp.json flag:

docker run --name gsc-mqtt-broker -p 1883:1883 -p 8883:8883 --device=/dev/sgx_enclave       -v /var/run/aesmd/aesm.socket:/var/run/aesmd/aesm.socket --env GSC_PAL=Linux    gsc-ubuntu20.04-mqtt -c /etc/mosquitto/mosquitto.conf -v

I get this error:

error: PAL failed at ../Pal/src/host/Linux/db_main.c:pal_linux_main:229 (exitcode = 6, reason=setting personality failed)

Something else I tried is to run mosquitto inside the container.
It seems that it behaves similarly. I get the following output:

root@52b555db3a29:/etc/mosquitto# ./../../usr/sbin/mosquitto -v -c mosquitto.conf
1642631239: Loading config file /etc/mosquitto/conf.d/default.conf
[ 2302.540395]~DLT~   27~INFO     ~FIFO /tmp/dlt cannot be opened. Retrying later...

The next line after Loading config file /etc/mosquitto/conf.d/default.conf should be

Opening ipv4 listen socket on port 1883.

I add the RUN mkfifo /tmp/dlt to the dockerfile and add the /tmp/dlt to the allowed files, but it doesn't help.
Still the same output.

I found this issue in the mosquitto Issue 1735 github.
They recommend using a newer version of mosquitto to get rid of the tmp/dlterror.
So I changed the dockerfile:

From ubuntu:20.04
  
ENV LD_LIBRARY_PATH = "${LD_LIBRARY_PATH}:/usr/lib/x86_64-linux-gnu/:/lib/x86_64-linux-gnu/"

RUN apt-get update -y && apt-get install -y \
        openssl \
        apt-utils \
        software-properties-common

RUN add-apt-repository ppa:mosquitto-dev/mosquitto-ppa && apt-get update -y && apt install -y mosquitto

EXPOSE 1883
EXPOSE 8883

COPY examples/mosquitto/ca_certificates/ca.crt /etc/mosquitto/ca_certificates/
COPY examples/mosquitto/certs/docker.crt /etc/mosquitto/certs/
COPY examples/mosquitto/certs/docker.key /etc/mosquitto/certs/
COPY examples/mosquitto/conf.d/default.conf /etc/mosquitto/conf.d/

# RUN mkfifo /tmp/dlt

ENTRYPOINT ["/usr/sbin/mosquitto"]

CMD ["-c", "/etc/mosquitto/mosquitto.conf", "-v"]

Unfortunately, the gsc sign process runs into the following error:

Step 7/10 : RUN export PYTHONPATH="${PYTHONPATH}:$(find /gramine/meson_build_output/lib -type d -path '*/site-packages')"     && gramine-sgx-sign          --key /gsc-signer-key.pem          --manifest /entrypoint.manifest          --output /entrypoint.manifest.sgx

 ---> Running in 62b17d2ba5ee
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/toml/decoder.py", line 511, in loads
    ret = decoder.load_line(line, currentlevel, multikey,
  File "/usr/local/lib/python3.8/dist-packages/toml/decoder.py", line 778, in load_line
    value, vtype = self.load_value(pair[1], strictly_valid)
  File "/usr/local/lib/python3.8/dist-packages/toml/decoder.py", line 866, in load_value
    raise ValueError("Reserved escape sequence used")
ValueError: Reserved escape sequence used

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/gramine/meson_build_output/bin/gramine-sgx-sign", line 70, in <module>
    main() # pylint: disable=no-value-for-parameter
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/gramine/meson_build_output/bin/gramine-sgx-sign", line 43, in main
    sigstruct = get_tbssigstruct(output, today, libpal, verbose=verbose)
  File "/gramine/meson_build_output/lib/python3.6/site-packages/graminelibos/sgx_sign.py", line 532, in get_tbssigstruct
    mrenclave, manifest = get_mrenclave_and_manifest(manifest_path, libpal, verbose=verbose)
  File "/gramine/meson_build_output/lib/python3.6/site-packages/graminelibos/sgx_sign.py", line 457, in get_mrenclave_and_manifest
    manifest = Manifest.loads(manifest_data.decode('utf-8'))
  File "/gramine/meson_build_output/lib/python3.6/site-packages/graminelibos/manifest.py", line 163, in loads
    return cls(s)
  File "/gramine/meson_build_output/lib/python3.6/site-packages/graminelibos/manifest.py", line 85, in __init__
    manifest = toml.loads(manifest_str)
  File "/usr/local/lib/python3.8/dist-packages/toml/decoder.py", line 514, in loads
    raise TomlDecodeError(str(err), original, pos)
toml.decoder.TomlDecodeError: Reserved escape sequence used (line 34787 column 1 char 1321615)

What does this error mean?

@dimakuv
Copy link

dimakuv commented Jan 20, 2022

This error is interesting, I have never seen it before.

What happens is this:

  1. gsc build command line generates the gsc-<originalname>-unsigned Docker image. This Docker image contains the auto-generated-by-GSC .manifest.sgx file.
  2. gsc sign command line tries to parse this .manifest.sgx file from the gsc-<originalname>-unsigned Docker image. This parsing fails.

Apparently, there are some weird file names (or maybe weird environment variable values?) in the unsigned Docker image prepared in step 1. You'll need to find out what exactly is located in the .manifest.sgx file at line 34787. You can do this by entering the gsc-<originalname>-unsigned image, locating the .manifest.sgx file in this image, and reading through it.

To enter this unsigned Docker image, you should do something like:

docker run -it --entrypoint /bin/bash gsc-<originalname>-unsigned

@dzygann
Copy link
Author

dzygann commented Jan 20, 2022

Hi @dimakuv

I have checked the gsc-<originalname>-unsigned after the gsc build, but there is no entrypoint.manifest.sgx

image

Can I take the entrypoint.manifest and create a entrypoint.manifest.sgx by myself?

@dimakuv
Copy link

dimakuv commented Jan 20, 2022

Hm, maybe I'm mistaken, maybe it's just entrypoint.manifest in the unsigned version of the Docker image.

Then you should check for this error Reserved escape sequence used (line 34787 column 1 char 1321615) in this entrypoint.manifest file.

@dzygann
Copy link
Author

dzygann commented Jan 20, 2022

This doesn't work, because the entrypoint.manifest does have only 23733 lines. The lines with the hashes and so on are missing.
So, can I take the entrypoint.manifest and create a entrypoint.manifest.sgx by myself? Then I would be able to check which entry is causing the issue.

@dimakuv
Copy link

dimakuv commented Jan 20, 2022

So, can I take the entrypoint.manifest and create a entrypoint.manifest.sgx by myself? Then I would be able to check which entry is causing the issue.

Yes, please try this.

@dzygann
Copy link
Author

dzygann commented Jan 20, 2022

I had an older version of gsc. I pulled the new one from the repo.
The error message is still there, but on another line:

Step 7/10 : RUN export PYTHONPATH="${PYTHONPATH}:$(find /gramine/meson_build_output/lib -type d -path '*/site-packages')"     && gramine-sgx-sign          --key /gsc-signer-key.pem          --manifest /entrypoint.manifest          --output /entrypoint.manifest.sgx

 ---> Running in 47ba37c6473f
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/toml/decoder.py", line 511, in loads
    ret = decoder.load_line(line, currentlevel, multikey,
  File "/usr/local/lib/python3.8/dist-packages/toml/decoder.py", line 778, in load_line
    value, vtype = self.load_value(pair[1], strictly_valid)
  File "/usr/local/lib/python3.8/dist-packages/toml/decoder.py", line 880, in load_value
    return (self.load_array(v), "array")
  File "/usr/local/lib/python3.8/dist-packages/toml/decoder.py", line 1026, in load_array
    nval, ntype = self.load_value(a[i])
  File "/usr/local/lib/python3.8/dist-packages/toml/decoder.py", line 866, in load_value
    raise ValueError("Reserved escape sequence used")
ValueError: Reserved escape sequence used

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/gramine/meson_build_output/bin/gramine-sgx-sign", line 70, in <module>
    main() # pylint: disable=no-value-for-parameter
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/gramine/meson_build_output/bin/gramine-sgx-sign", line 28, in main
    manifest = Manifest.load(manifest_file)
  File "/gramine/meson_build_output/lib/python3.6/site-packages/graminelibos/manifest.py", line 167, in load
    return cls.loads(f.read())
  File "/gramine/meson_build_output/lib/python3.6/site-packages/graminelibos/manifest.py", line 163, in loads
    return cls(s)
  File "/gramine/meson_build_output/lib/python3.6/site-packages/graminelibos/manifest.py", line 85, in __init__
    manifest = toml.loads(manifest_str)
  File "/usr/local/lib/python3.8/dist-packages/toml/decoder.py", line 514, in loads
    raise TomlDecodeError(str(err), original, pos)
toml.decoder.TomlDecodeError: Reserved escape sequence used (line 6 column 1 char 884)

Line 6 contains the trusted_files and at char 884 is the d of "file:/etc/gshadow". This doesn't look like sth. reserved for me.

@dimakuv
Copy link

dimakuv commented Jan 20, 2022

You should probably analyze the whole Line 6... I guess it has thousands of entries. Sorry, but I don't see any other way.

I'm pretty sure some trusted file name is weird in this Line 6.

@dzygann
Copy link
Author

dzygann commented Jan 20, 2022

Do you know if there is a blacklist or whitelist for characters that are allowed or disallowed?

@dimakuv
Copy link

dimakuv commented Jan 20, 2022

We use TOML basic strings for sgx.trusted_files entries, so you can check the TOML spec: https://toml.io/en/v1.0.0#string

@dzygann
Copy link
Author

dzygann commented Jan 20, 2022

Hi @dimakuv

This one looks suspicious to me:

 "file:/usr/bin/[", 

 "file:/usr/lib/systemd/system/system-systemd\x2dcryptsetup.slice", <-I bet this one is the one which is causing the trouble

 "file:/usr/share/ca-certificates/mozilla/NetLock_Arany_=Class_Gold=_Főtanúsítvány.crt",

What would be the way to escape the backslash or one of the other characters?

---- Edit ----

I did a cross-check and indeed the file:/usr/lib/systemd/system/system-systemd\x2dcryptsetup.slice isn't in place, when I don't install the software-properties-common. The software-properties-common is necessary for the RUN add-apt-repository ppa:mosquitto-dev/mosquitto-ppa && apt-get update -y && apt install -y mosquitto ....

@dimakuv
Copy link

dimakuv commented Jan 20, 2022

@dzygann Awesome analysis! You rock :)

As you correctly mentioned, only entry 2 is a bad case (because it has the unescaped backslash \). The other two suspicious entries are actually fine.

I actually thought that GSC automatically escapes (by adding another backslash to each backslash, resulting in \\). But looks like we don't have this feature: https://github.com/gramineproject/gsc/blob/master/finalize_manifest.py#L67-L69. The only thing we do is we skip filenames that contain newline (\n).

@dzygann Would you be able to modify GSC (that particular finalize_manifest.py file) to add escaping of \ and " in filenames? You can check this for escaping (we use Python 3, so you can use the first answer): https://stackoverflow.com/questions/18935754/how-to-escape-special-characters-of-a-string-with-single-backslashes

I think after this fix everything will work.

@dimakuv
Copy link

dimakuv commented Jan 20, 2022

@dzygann Oh look at this:

gsc/gsc.py

Line 104 in 21b30f2

escaped_env_var = env_var.translate(str.maketrans({'\\': r'\\', '"': r'\"'}))

We actually have escaping, but only for environment variables, not for filenames :( So you can just take this one line and use it in finalize_manifest.py.

@mkow
Copy link
Member

mkow commented Jan 20, 2022

@dimakuv didn't we move to TOML in GSC? Seems like the move is incomplete and we're still handcrafting the strings?

@dzygann
Copy link
Author

dzygann commented Jan 20, 2022

Unfortunately, I get the following error if I add the second backslash to the filename:

Traceback (most recent call last):
  File "/gramine/meson_build_output/bin/gramine-sgx-sign", line 70, in <module>
    main() # pylint: disable=no-value-for-parameter
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/gramine/meson_build_output/bin/gramine-sgx-sign", line 30, in main
    expanded = manifest.expand_all_trusted_files()
  File "/gramine/meson_build_output/lib/python3.6/site-packages/graminelibos/manifest.py", line 193, in expand_all_trusted_files
    append_trusted_dir_or_file(trusted_files, tf, expanded)
  File "/gramine/meson_build_output/lib/python3.6/site-packages/graminelibos/manifest.py", line 57, in append_trusted_dir_or_file
    raise ManifestError(f'Cannot resolve {path}')
graminelibos.manifest.ManifestError: Cannot resolve /usr/lib/systemd/system/system-systemd\u002dcryptsetup.slice

In my case it was sufficient to exclude the /usr/lib/systemd/system/system-systemd\x2dcryptsetup.slice from the trusted files.

Finally, it works like a charm!

Thank you very much @dimakuv for your support!

@dimakuv
Copy link

dimakuv commented Jan 20, 2022

@dimakuv didn't we move to TOML in GSC? Seems like the move is incomplete and we're still handcrafting the strings?

But here we are adding strings to the TOML array. The strings must already be proper TOML (basic) strings. So it's not like TOML magically does the escaping for us.

Hmm, maybe there is a special "escape this string" function in the Python TOML package? We should check, instead of doing our own escaping.

@dimakuv
Copy link

dimakuv commented Jan 20, 2022

In my case it was sufficient to exclude the /usr/lib/systemd/system/system-systemd\x2dcryptsetup.slice from the trusted files.

@dzygann How did you exclude it? By manually changing some code in GSC? Or by tweaking the manifest file somehow?

@dzygann
Copy link
Author

dzygann commented Jan 20, 2022

I add an if condition to the finalize_manifest.py

if str.__contains__(filename, "\\"):
                print(f"Exclude {filename} file from trusted_files")
                continue

---- Edit ----

I just see you have an exclude_paths_regex variable. Maybe it would fit there also

excluded_paths_regex = (r'^/('

@mkow
Copy link
Member

mkow commented Jan 20, 2022

But here we are adding strings to the TOML array. The strings must already be proper TOML (basic) strings. So it's not like TOML magically does the escaping for us.

Wait, on the API level TOML should be able to serialize arbitrary Python dictionaries with standard types inside, and I think it already does this.

But there we do something really weird, we generate TOML data by just appending to a string and then calling toml.loads()?

gsc/gsc.py

Line 111 in c4d5c2d

base_image_environment += f'loader.env.{env_var_name} = "{env_var_value}"\n'

gsc/gsc.py

Lines 195 to 196 in c4d5c2d

base_image_environment = extract_environment_from_image_config(original_image.attrs['Config'])
base_image_dict = toml.loads(base_image_environment)

@dimakuv
Copy link

dimakuv commented Jan 21, 2022

But there we do something really weird, we generate TOML data by just appending to a string and then calling toml.loads()?

Yes, we do it in a few places in GSC.

I got your point now. You want to collect all TOML stuff in a Python dict and then do toml.dump(). This looks like the best solution, but I'm not sure how much we need to change in GSC for this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants