Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

K8S: pid is not 1, automatic detection of docker failed, inotify detection is not automatically enabled. #2824

Closed
jinleileiking opened this issue Dec 28, 2021 · 3 comments
Assignees
Labels
Enhancement Improvement or enhancement. Kubernetes For K8s, Prometheus, APM and Grafana. TransByAI Translated by AI/GPT. Won't fix We won't fix it.
Milestone

Comments

@jinleileiking
Copy link

jinleileiking commented Dec 28, 2021

Description'

Please ensure that the markdown structure is maintained.

  1. SRS版本(Version): 4.0release'

Please ensure that the markdown structure is maintained.

  1. SRS的日志如下(Log):

Please ensure that the markdown structure is maintained.

No.

Please ensure that the markdown structure is maintained.
  1. The configuration of SRS is as follows (Config):

Please ensure that the markdown structure is maintained.

Able to update:

Please ensure that the markdown structure is maintained.

inotify_auto_reload    on;
auto_reload_for_docker on;

Unable to update.

Please ensure that the markdown structure is maintained.

auto_reload_for_docker on;

Replay

Please ensure that the markdown structure is maintained.

1. You can start directly on ACK

Please ensure that the markdown structure is maintained.

Expected behavior (Expect)

Please ensure that the markdown structure is maintained.

srs_detect_docker works well.

Please ensure that the markdown structure is maintained.

Analysis:

Please ensure that the markdown structure is maintained.

I don't know the reason, but it caused the "ps" in the container to not use 1.

[root@srs-origin-0 srs]# ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
65535          1  0.0  0.0    956     4 ?        Ss   10:22   0:00 /pause
root           8  0.0  0.0  25296  5120 ?        Ss   10:23   0:00 ./objs/srs -c conf/srs.conf
root          14  0.0  0.0    788   248 ?        Ss   10:23   0:00 tini /usr/sbin/crond -f -L /dev/stdout
root          22  0.0  0.0   1568   448 ?        S    10:23   0:00 /usr/sbin/crond -f -L /dev/stdout
root          49  0.0  0.0  11828  1948 pts/0    Ss   10:26   0:00 /bin/bash
root          97  0.0  0.0  51732  1712 pts/0    R+   10:35   0:00 ps aux

Recently, we have been using containerd instead of Docker. I'm not sure if this is related to that.

That code: err = fr.open("/proc/1/cgroup") seems to be incorrect, right?

The phrase "enable auto reload for docker" is not printed in the log. There might be an issue with this sentence.

However, forcing inotify_auto_reload on; seems to directly detect configmap updates.

Updated log:

{"time":"2021-12-28T10:29:56.536+08:00", "level":"Trace", "pid":8, "cid":"g4944dug", "message":"inotify event wd=1, mask=0x40000100, len=32, name=..2021_12_28_10_29_56.500684997, reload=0"}
{"time":"2021-12-28T10:29:59.536+08:00", "level":"Trace", "pid":8, "cid":"g4944dug", "message":"inotify event wd=1, mask=0x100, len=16, name=..data_tmp, reload=0"}
{"time":"2021-12-28T10:29:59.536+08:00", "level":"Trace", "pid":8, "cid":"g4944dug", "message":"inotify event wd=1, mask=0x80, len=16, name=..data, reload=1"}
{"time":"2021-12-28T10:29:59.536+08:00", "level":"Trace", "pid":8, "cid":"g4944dug", "message":"reload config, signo=1"}
{"time":"2021-12-28T10:29:59.934+08:00", "level":"Trace", "pid":8, "cid":"2v02k593", "message":"config parse complete"}
{"time":"2021-12-28T10:29:59.934+08:00", "level":"Trace", "pid":8, "cid":"2v02k593", "message":"srs checking config..."}
{"time":"2021-12-28T10:29:59.934+08:00", "level":"Warn", "pid":8, "cid":"2v02k593","errno":11,  "message":"stats network use index=0, ip=10.33.0.34, ifname=eth0"}
{"time":"2021-12-28T10:29:59.934+08:00", "level":"Warn", "pid":8, "cid":"2v02k593","errno":11,  "message":"stats disk not configed, disk iops disabled."}
{"time":"2021-12-28T10:29:59.934+08:00", "level":"Trace", "pid":8, "cid":"2v02k593", "message":"you can check log by: tail -n 30 -f /var/log/srs/srs.log"}
{"time":"2021-12-28T10:29:59.934+08:00", "level":"Trace", "pid":8, "cid":"2v02k593", "message":"please check SRS by: ./etc/init.d/srs status"}
{"time":"2021-12-28T10:29:59.934+08:00", "level":"Trace", "pid":8, "cid":"2v02k593", "message":"reload max_connections success."}
{"time":"2021-12-28T10:29:59.934+08:00", "level":"Trace", "pid":8, "cid":"2v02k593", "message":"reload http_api success, nothing changed."}
{"time":"2021-12-28T10:29:59.934+08:00", "level":"Trace", "pid":8, "cid":"2v02k593", "message":"reload http stream success, nothing changed."}
{"time":"2021-12-28T10:29:59.934+08:00", "level":"Trace", "pid":8, "cid":"2v02k593", "message":"reload rtc server success, nothing changed."}
{"time":"2021-12-28T10:29:59.934+08:00", "level":"Trace", "pid":8, "cid":"2v02k593", "message":"vhost __defaultVhost__ maybe modified, reload its detail."}
{"time":"2021-12-28T10:29:59.934+08:00", "level":"Trace", "pid":8, "cid":"2v02k593", "message":"ingest nothing changed for vhost=__defaultVhost__"}
{"time":"2021-12-28T10:29:59.934+08:00", "level":"Trace", "pid":8, "cid":"2v02k593", "message":"reload config success."}


#1635

TRANS_BY_GPT3

@jinleileiking jinleileiking changed the title 在k8s环境下(aliyun ACK)conf文件通过configmap更新,srs程序无法感知 [bug maybe]在k8s环境下(aliyun ACK)conf文件通过configmap更新,srs程序无法感知 Dec 28, 2021
@jinleileiking jinleileiking changed the title [bug maybe]在k8s环境下(aliyun ACK)conf文件通过configmap更新,srs程序无法感知 [bug maybe]在k8s环境下(aliyun ACK)判断是否在docker里逻辑有问题 Dec 28, 2021
@winlinvip
Copy link
Member

winlinvip commented Dec 29, 2021

When automatic detection is not working, it is possible to manually specify it as "on". Please refer to the comments for details.

# Whether auto reload by watching the config file by inotify.
# default: off
inotify_auto_reload off;

# Whether enable inotify_auto_reload for docker.
# If on, it will set inotify_auto_reload to on in docker, even it's off.
# default: on
auto_reload_for_docker on;

As for the issue of PID not being 1, it needs to be investigated. It is possible that some automatic detection functions for Docker may not work properly, making it less convenient to use.

Since it is a non-blocking issue and not very common, if anyone encounters it, please feel free to submit a patch.

TRANS_BY_GPT3

@winlinvip winlinvip self-assigned this Dec 29, 2021
@winlinvip winlinvip added the Enhancement Improvement or enhancement. label Dec 29, 2021
@winlinvip winlinvip added this to the 5.0 milestone Dec 29, 2021
@winlinvip winlinvip changed the title [bug maybe]在k8s环境下(aliyun ACK)判断是否在docker里逻辑有问题 K8S: pid非1,自动检测docker失败,没有自动开启inotify检测 Dec 29, 2021
@winlinvip
Copy link
Member

winlinvip commented Jan 1, 2022

Found that Tencent Cloud's K8s (TKE) is not using PID 1. Reference:

https://github.com/ossrs/srs-tke-template

Added a new configuration item that can force the use of Docker, regardless of whether it is successfully detected or not.

# Whether in docker. When SRS starting, it will detect the docker, however
# it might detect failed, then read this config.
# Default: off
in_docker on;

This way, at least when the detection fails, it can be bypassed.

TRANS_BY_GPT3

@winlinvip winlinvip added the Kubernetes For K8s, Prometheus, APM and Grafana. label Sep 1, 2022
@winlinvip
Copy link
Member

winlinvip commented Jan 2, 2023

There is no good way to solve this problem.

TRANS_BY_GPT3

@winlinvip winlinvip closed this as not planned Won't fix, can't repro, duplicate, stale Jan 2, 2023
@winlinvip winlinvip added the Won't fix We won't fix it. label Jan 2, 2023
@winlinvip winlinvip changed the title K8S: pid非1,自动检测docker失败,没有自动开启inotify检测 K8S: pid is not 1, automatic detection of docker failed, inotify detection is not automatically enabled. Jul 28, 2023
@winlinvip winlinvip added the TransByAI Translated by AI/GPT. label Jul 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Enhancement Improvement or enhancement. Kubernetes For K8s, Prometheus, APM and Grafana. TransByAI Translated by AI/GPT. Won't fix We won't fix it.
Projects
None yet
Development

No branches or pull requests

2 participants