Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Podman Rootless Container Issue - failed to write to /proc/self/oom_score_adj: Permission denied #2961

Open
luckylinux opened this issue Dec 3, 2023 · 4 comments
Open

Podman Rootless Container Issue - failed to write to /proc/self/oom_score_adj: Permission denied #2961

luckylinux opened this issue Dec 3, 2023 · 4 comments

Comments

@luckylinux
Copy link

I am getting an error

[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

When trying to run the latest image of eclipse-mosquitto (e.g. from docker.io).

I originally thought this was a podman bug -> containers/podman#20886

However, after playing around a bit with mosquitto and trying to build using the Dockerfile, maybe the issue is within eclipse-mosquitto itself.

I tried to build using the following command

cd ~/build/mosquitto/git-eclipse-mosquitto/docker/$version
podman build --tag homelab:eclipse-mosquitto -f ./Dockerfile .

I tried:

  • 2.0 -> this results in [conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied
  • 2.0-openssl -> this results in [conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied
  • 1.6-openssl -> this works (container is running since 12 minutes now)

I tried a quick diff between the 2.0-openssl and 1.6-openssl Dockerfile

podman@Rock5B-01:~/build/mosquitto/git-eclipse-mosquitto/docker$ diff 2.0-openssl/Dockerfile 1.6-openssl/Dockerfile 
6,7c6,7
< ENV VERSION=2.0.18 \
<     DOWNLOAD_SHA256=d665fe7d0032881b1371a47f34169ee4edab67903b2cd2b4c083822823f4448a \
---
> ENV VERSION=1.6.15 \
>     DOWNLOAD_SHA256=5ff2271512f745bf1a451072cd3768a5daed71e90c5179fae12b049d6c02aa0f \
16d15
<         cjson-dev \
64c63
<         CFLAGS="-Wall -O2 -I/build/lws/include -I/build" \
---
>         CFLAGS="-Wall -O2 -I/build/lws/include" \
83,85c82
<     install -s -m755 /build/mosq/apps/mosquitto_ctrl/mosquitto_ctrl /usr/bin/mosquitto_ctrl && \
<     install -s -m755 /build/mosq/apps/mosquitto_passwd/mosquitto_passwd /usr/bin/mosquitto_passwd && \
<     install -s -m755 /build/mosq/plugins/dynamic-security/mosquitto_dynamic_security.so /usr/lib/mosquitto_dynamic_security.so && \
---
>     install -s -m755 /build/mosq/src/mosquitto_passwd /usr/bin/mosquitto_passwd && \
88c85
<     install -Dm644 /build/mosq/epl-v20 /usr/share/licenses/mosquitto/epl-v20 && \
---
>     install -Dm644 /build/mosq/epl-v10 /usr/share/licenses/mosquitto/epl-v10 && \
92,93c89
<         ca-certificates \
<         cjson && \
---
>         ca-certificates && \
100c96
< COPY docker-entrypoint.sh mosquitto-no-auth.conf /
---
> COPY docker-entrypoint.sh /

There doesn't seem to be a whole lot of differences. Any idea what could be causing the error ?
@luckylinux
Copy link
Author

Uhm, the error message is totally misleading when using Mosquitto 2.x.

Looking at the contents of /home/podman/log/mosquitto01/mosquitto.log seem to reveal the real culprit

(...)
1701606791: Error: Unable to load server certificate "/mosquitto/ssl/mosquitto01.cert". Check certfile.
1701606791: OpenSSL Error[0]: error:02FFF002:system library:func(4095):No such file or directory
1701606791: OpenSSL Error[1]: error:20FFF002:BIO routines:CRYPTO_internal:system lib
1701606791: OpenSSL Error[2]: error:14FFF002:SSL routines:(UNKNOWN)SSL_internal:system lib
1701606836: mosquitto version 2.0.18 starting
1701606836: Config loaded from /mosquitto/config/mosquitto.conf.
1701606836: Opening ipv4 listen socket on port 1883.
1701606836: Opening ipv6 listen socket on port 1883.
1701606836: Opening ipv4 listen socket on port 8885.
1701606836: Opening ipv6 listen socket on port 8885.
1701606836: Error: Unable to load server certificate "/mosquitto/ssl/mosquitto01.cert". Check certfile.
1701606836: OpenSSL Error[0]: error:80000002:system library::No such file or directory
1701606836: OpenSSL Error[1]: error:10080002:BIO routines::system lib
1701606836: OpenSSL Error[2]: error:0A080002:SSL routines::system lib
1701606844: mosquitto version 2.0.18 starting
1701606844: Config loaded from /mosquitto/config/mosquitto.conf.
1701606844: Opening ipv4 listen socket on port 1883.
1701606844: Opening ipv6 listen socket on port 1883.
1701606844: Opening ipv4 listen socket on port 8885.
1701606844: Opening ipv6 listen socket on port 8885.
1701606844: Error: Unable to load server certificate "/mosquitto/ssl/mosquitto01.cert". Check certfile.
1701606844: OpenSSL Error[0]: error:80000002:system library::No such file or directory
1701606844: OpenSSL Error[1]: error:10080002:BIO routines::system lib
1701606844: OpenSSL Error[2]: error:0A080002:SSL routines::system lib
1701607264: mosquitto version 1.6.15 starting
1701607264: Config loaded from /mosquitto/config/mosquitto.conf.
1701607264: Error: Invalid password hash for user admin, removing entry.
1701607264: Opening ipv4 listen socket on port 1883.
1701607264: Opening ipv6 listen socket on port 1883.
1701607264: Opening ipv4 listen socket on port 8885.
1701607264: Opening ipv6 listen socket on port 8885.
1701607264: mosquitto version 1.6.15 running
1701609064: Saving in-memory database to /mosquitto/data/mosquitto.db.
1701610865: Saving in-memory database to /mosquitto/data/mosquitto.db.
1701611515: mosquitto version 1.6.15 terminating
1701611515: Saving in-memory database to /mosquitto/data/mosquitto.db.
1701611575: mosquitto version 2.0.18 starting
1701611575: Config loaded from /mosquitto/config/mosquitto.conf.
1701611575: Opening ipv4 listen socket on port 1883.
1701611575: Opening ipv6 listen socket on port 1883.
1701611575: Opening ipv4 listen socket on port 8885.
1701611575: Opening ipv6 listen socket on port 8885.
1701611575: Error: Unable to load CA certificates. Check cafile "/mosquitto/ssl/ca/ca.pem".
1701611575: Error: Unable to load server certificate "/mosquitto/ssl/server/server.crt". Check certfile.
1701611575: OpenSSL Error[0]: error:02FFF002:system library:func(4095):No such file or directory
1701611575: OpenSSL Error[1]: error:20FFF080:BIO routines:CRYPTO_internal:no such file
1701611575: OpenSSL Error[2]: error:0BFFF002:x509 certificate routines:CRYPTO_internal:system lib
1701611575: OpenSSL Error[3]: error:09FFF06C:PEM routines:CRYPTO_internal:no start line
1701611575: OpenSSL Error[4]: error:14FFF009:SSL routines:(UNKNOWN)SSL_internal:PEM lib
1701611595: mosquitto version 2.0.18 starting
1701611595: Config loaded from /mosquitto/config/mosquitto.conf.
1701611595: Opening ipv4 listen socket on port 1883.
1701611595: Opening ipv6 listen socket on port 1883.
1701611595: Opening ipv4 listen socket on port 8885.
1701611595: Opening ipv6 listen socket on port 8885.
1701611595: Error: Unable to load CA certificates. Check cafile "/mosquitto/ssl/ca/ca.pem".
1701611595: Error: Unable to load server certificate "/mosquitto/ssl/server/server.crt". Check certfile.
1701611595: OpenSSL Error[0]: error:02FFF002:system library:func(4095):No such file or directory
1701611595: OpenSSL Error[1]: error:20FFF080:BIO routines:CRYPTO_internal:no such file
1701611595: OpenSSL Error[2]: error:0BFFF002:x509 certificate routines:CRYPTO_internal:system lib
1701611595: OpenSSL Error[3]: error:09FFF06C:PEM routines:CRYPTO_internal:no start line
1701611595: OpenSSL Error[4]: error:14FFF009:SSL routines:(UNKNOWN)SSL_internal:PEM lib

My mosquitto.conf

# MQTT
listener 1883
protocol mqtt
persistence true
persistence_location /mosquitto/data/
log_dest file /mosquitto/log/mosquitto.log

# MQTTS
listener 8885
cafile /mosquitto/ssl/ca/ca.pem
certfile /mosquitto/ssl/server/server.crt
keyfile /mosquitto/ssl/server/server.key
tls_version tlsv1.2

## Authentication ##
# By default, Mosquitto >=2.0 allows only authenticated connections. Change to true to enable anonymous connections.
allow_anonymous false
password_file /mosquitto/config/password.txt

And my podman / Docker compose.yml file is

version: '3'
networks:
  podman:
services:
  mosquitto01:
#    image: localhost/homelab:eclipse-mosquitto
    image: eclipse-mosquitto
    container_name: mosquitto01
    volumes:
      - ~/config/containers/mosquitto01:/mosquitto/config
      - ~/data/mosquitto01:/mosquitto/data
      - ~/log/mosquitto01:/mosquitto/log
      - ~/certificates/mosquitto01:/mosquitto/ssl
    ports:
      - 1883:1883
      - 8885:8885
      - 9001:9001
    networks:
      - podman
    capabilities: {CAP_NET_RAW,CAP_NET_BIND_SERVICE}

Am I using an unsupported cypher / library ? Is the openssl version I used too recent for Mosquitto ?

OpenSSL 3.0.11 19 Sep 2023 (Library: OpenSSL 3.0.11 19 Sep 2023)

@luckylinux
Copy link
Author

luckylinux commented Dec 3, 2023 •
edited
Loading

It's also generating this error during certificate creation

20C02385FFFF0000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:../crypto/evp/evp_fetch.c:373:Global default library context, Algorithm (nodes : 0), Properties (<null>)
20C02385FFFF0000:error:03000086:digital envelope routines:do_sigver_init:initialization error:../crypto/evp/m_sigver.c:253:

And server.crt is empty 😢.

I tried to generate it using

openssl x509 -provider legacy -provider default -nodes -req -in "${basefolder}/server/server.csr" -CA "${basefolder}/ca/ca.crt" -CAkey "${basefolder}/ca/ca.key" -CAcreateserial -out "${basefolder}/server/server.crt" -days $duration

Legacy procider should've make it work. I also added -provider default after -provider legacy but not difference.

@luckylinux
Copy link
Author

Actually the issue might be related to the "-nodes" option I used here (in some cases it's required, otherwise the prompt Enter the PEM Passphrase appears).

Updated to this generates the server.crt certificate correctly

openssl x509 -provider legacy -provider default -req -in "${basefolder}/server/server.csr" -CA "${basefolder}/ca/ca.crt" -CAkey "${basefolder}/ca/ca.key" -CAcreateserial -out "${basefolder}/server/server.crt" -days $duration

Onto the next error

1701612073: mosquitto version 2.0.18 running
1701612376: mosquitto version 2.0.18 terminating
1701612376: Saving in-memory database to /mosquitto/data//mosquitto.db.
1701612386: mosquitto version 2.0.18 starting
1701612386: Config loaded from /mosquitto/config/mosquitto.conf.
1701612386: Opening ipv4 listen socket on port 1883.
1701612386: Opening ipv6 listen socket on port 1883.
1701612386: Opening ipv4 listen socket on port 8885.
1701612386: Opening ipv6 listen socket on port 8885.
1701612386: Error: Unable to load CA certificates. Check cafile "/mosquitto/ssl/ca/ca.pem".
1701612386: Error: Unable to load server certificate "/mosquitto/ssl/server/server.crt". Check certfile.
1701612386: OpenSSL Error[0]: error:02FFF002:system library:func(4095):No such file or directory
1701612386: OpenSSL Error[1]: error:20FFF080:BIO routines:CRYPTO_internal:no such file
1701612386: OpenSSL Error[2]: error:0BFFF002:x509 certificate routines:CRYPTO_internal:system lib
1701612386: OpenSSL Error[3]: error:09FFF06C:PEM routines:CRYPTO_internal:no start line
1701612386: OpenSSL Error[4]: error:14FFF009:SSL routines:(UNKNOWN)SSL_internal:PEM lib
1701612442: mosquitto version 2.0.18 starting
1701612442: Config loaded from /mosquitto/config/mosquitto.conf.
1701612442: Opening ipv4 listen socket on port 1883.
1701612442: Opening ipv6 listen socket on port 1883.
1701612442: Opening ipv4 listen socket on port 8885.
1701612442: Opening ipv6 listen socket on port 8885.
1701612442: Error: Unable to load server certificate "/mosquitto/ssl/server/server.crt". Check certfile.
1701612442: OpenSSL Error[0]: error:09FFF06C:PEM routines:CRYPTO_internal:no start line
1701612442: OpenSSL Error[1]: error:14FFF009:SSL routines:(UNKNOWN)SSL_internal:PEM lib
1701613159: mosquitto version 2.0.18 starting
1701613159: Config loaded from /mosquitto/config/mosquitto.conf.
1701613159: Opening ipv4 listen socket on port 1883.
1701613159: Opening ipv6 listen socket on port 1883.
1701613159: Opening ipv4 listen socket on port 8885.
1701613159: Opening ipv6 listen socket on port 8885.
1701613159: Error: Unable to load server key file "/mosquitto/ssl/server/server.key". Check keyfile.
1701613159: OpenSSL Error[0]: error:0BFFF074:x509 certificate routines:CRYPTO_internal:extension value error

Probably some permission errors. But what worries me is the last line

OpenSSL Error[0]: error:0BFFF074:x509 certificate routines:CRYPTO_internal:extension value error

@luckylinux
Copy link
Author

Now it works but I do not know why ...

I regenerated all certificates, keys and CA ...

Then from outside the container I did

chown -R $user:$user "${basefolder}"
chmod 0755 "${basefolder}"
chmod 0755 "${basefolder}/ca"
chmod 0755 "${basefolder}/client"
chmod 0755 "${basefolder}/server"
chmod 0640 ${basefolder}/ca/*
chmod 0640 ${basefolder}/client/*
chmod 0640 ${basefolder}/server/*

And now it seems to work

CONTAINER ID  IMAGE                                         COMMAND               CREATED         STATUS        PORTS                                                                     NAMES
b9247e4d1504  docker.io/library/eclipse-mosquitto:latest    /usr/sbin/mosquit...  51 minutes ago  Up 2 minutes  0.0.0.0:1883->1883/tcp, 0.0.0.0:8885->8885/tcp, 0.0.0.0:9001->9001/tcp    mosquitto01

@luckylinux luckylinux mentioned this issue Dec 3, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@luckylinux