K8S: Support fsnotify and reload when ConfigMap update

[winlinvip]

K8S uses ConfigMap to store configuration files, for example:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: srs-config
data:
  srs.conf: |-
    listen              1935;
    max_connections     1000;
    daemon              off;
    http_api {
        enabled         on;
        listen          1985;
    }
    http_server {
        enabled         on;
        listen          8080;
    }
    vhost __defaultVhost__ {
        http_remux {
            enabled     on;
        }
        hls {
            enabled         on;
        }
    }
EOF

ConfigMap will be mounted as a volume and become a configuration file:

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: srs-deploy
  labels:
    app: srs
spec:
  replicas: 1
  selector:
    matchLabels:
      app: srs
  template:
    metadata:
      labels:
        app: srs
    spec:
      volumes:
      - name: config-volume
        configMap:
          name: srs-config
      containers:
      - name: srs
        image: ossrs/srs:3
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 1935
        - containerPort: 1985
        - containerPort: 8080
        volumeMounts:
        - name: config-volume
          mountPath: /usr/local/srs/conf
EOF

You can view the running pods:

Mac:trunk chengli.ycl$ pod=`kubectl get po|grep srs-deploy|awk '{print $1}'`
Mac:trunk chengli.ycl$ kubectl exec $pod -- cat conf/srs.conf && echo ""
listen              1935;
max_connections     1000;
daemon              off;
http_api {
    enabled         on;
    listen          1985;
}
http_server {
    enabled         on;
    listen          8080;
}
vhost __defaultVhost__ {
    http_remux {
        enabled     on;
    }
    hls {
        enabled         on;
    }
}

When ConfigMap changes, you can use fsnotify to receive notifications of file changes, thereby triggering the reload of SRS.

TRANS_BY_GPT3

[winlinvip]

The paragraph in Aliyun's SLS log service says link is as follows:

Logtail running in DaemonSet mode will periodically request the configuration server to obtain new or updated configurations and perform hot reloading.

Another solution is to listen for changes in the configuration file. However, there is an issue where the fsnotify signal may not be received due to symbolic links: link

Here is an example of Nginx's reload, which is implemented using fsnotify: link

watcher, watcherErr := fsnotify.NewWatcher()

go func() {
    event, ok := <-watcher.Events:
    if event.Op&fsnotify.Create != fsnotify.Create {
        return;
    }
    if filepath.Base(event.Name) != "..data" {
        return;
    }
    nginxProcess, nginxProcessErr := os.FindProcess(getMasterNginxPid())
    nginxProcess.Signal(syscall.SIGHUP)
}

pathToWatch = "/etc/nginx"
if err := watcher.Add(pathToWatch); err != nil {
    stderrLogger.Fatal(err)
}

You can see that it also listens for changes in the file directory and then sends the SIGHUP (reload) signal to Nginx.

TRANS_BY_GPT3

[winlinvip]

It is implemented using inotify under Linux.

INOTIFY_INIT(2)                                          Linux Programmer's Manual                                          INOTIFY_INIT(2)

NAME
       inotify_init, inotify_init1 - initialize an inotify instance

SYNOPSIS
       #include <sys/inotify.h>

       int inotify_init(void);
       int inotify_init1(int flags);

DESCRIPTION
       inotify_init() initializes a new inotify instance and returns a file descriptor associated with a new inotify event queue.

If flags is 0, then inotify_init1() is the same as inotify_init(). The following values can be bitwise ORed in flags to obtain different results.
       ferent behavior:

       IN_NONBLOCK Set the O_NONBLOCK file status flag on the new open file description.  Using this flag saves extra calls to fcntl(2)  to
                   achieve the same result.

       IN_CLOEXEC  Set  the  close-on-exec  (FD_CLOEXEC)  flag  on  the  new file descriptor.  See the description of the O_CLOEXEC flag in
                   open(2) for reasons why this may be useful.

Since inotify returns an fd and can be initialized as non-blocking mode using inotify_init1, reading it using read is an IO operation, so we should be able to use ST to read this fd.

/* the following are legal, implemented events that user-space can watch for */
#define IN_ACCESS               0x00000001      /* File was accessed */
#define IN_MODIFY               0x00000002      /* File was modified */
#define IN_ATTRIB               0x00000004      /* Metadata changed */
#define IN_CLOSE_WRITE          0x00000008      /* Writtable file was closed */
#define IN_CLOSE_NOWRITE        0x00000010      /* Unwrittable file closed */
#define IN_OPEN                 0x00000020      /* File was opened */
#define IN_MOVED_FROM           0x00000040      /* File was moved from X */
#define IN_MOVED_TO             0x00000080      /* File was moved to Y */
#define IN_CREATE               0x00000100      /* Subfile was created */
#define IN_DELETE               0x00000200      /* Subfile was deleted */
#define IN_DELETE_SELF          0x00000400      /* Self was deleted */
#define IN_MOVE_SELF            0x00000800      /* Self was moved */

/* the following are legal events.  they are sent as needed to any watch */
#define IN_UNMOUNT              0x00002000      /* Backing fs was unmounted */
#define IN_Q_OVERFLOW           0x00004000      /* Event queued overflowed */
#define IN_IGNORED              0x00008000      /* File was ignored */

/* helper events */
#define IN_CLOSE                (IN_CLOSE_WRITE | IN_CLOSE_NOWRITE) /* close */
#define IN_MOVE                 (IN_MOVED_FROM | IN_MOVED_TO) /* moves */

/* special flags */
#define IN_ONLYDIR              0x01000000      /* only watch the path if it is a directory */
#define IN_DONT_FOLLOW          0x02000000      /* don't follow a sym link */
#define IN_EXCL_UNLINK          0x04000000      /* exclude events on unlinked objects */
#define IN_MASK_ADD             0x20000000      /* add to the mask of an already existing watch */
#define IN_ISDIR                0x40000000      /* event occurred against dir */
#define IN_ONESHOT              0x80000000      /* only send event once */

Note: The event list can be referred to here.

TRANS_BY_GPT3

[winlinvip]

SRS has added two configurations, automatically enabling auto reload under Docker, and disabling auto reload outside of Docker.

# Whether auto reload by watching the config file by inotify.
# default: off
inotify_auto_reload off;
# Whether enable inotify_auto_reload for docker.
# If on, it will set inotify_auto_reload to on in docker, even it's off.
# default: on
auto_reload_for_docker on;

The added log prints are as follows:

[2020-03-12 10:10:50.020][Warn][2914][636][16] enable auto reload for docker
[2020-03-12 10:10:50.021][Trace][2914][636] auto reload watching console.conf, fd=14, wd=1
[2020-03-12 10:10:53.940][Trace][2914][642] inotify event wd=1, mask=0x2, len=0, name=
[2020-03-12 10:10:53.940][Trace][2914][642] reload config, signo=1
[2020-03-12 10:10:54.033][Trace][2914][636] config parse complete
[2020-03-12 10:10:54.033][Trace][2914][636] srs checking config...
[2020-03-12 10:10:54.033][Warn][2914][636][11] stats network use index=0, ip=172.17.0.2
[2020-03-12 10:10:54.033][Warn][2914][636][11] stats disk not configed, disk iops disabled.
[2020-03-12 10:10:54.033][Trace][2914][636] write log to console
[2020-03-12 10:10:54.033][Trace][2914][636] reload http_api success, nothing changed.
[2020-03-12 10:10:54.033][Trace][2914][636] reload http stream success, nothing changed.
[2020-03-12 10:10:54.033][Trace][2914][636] vhost __defaultVhost__ maybe modified, reload its detail.
[2020-03-12 10:10:54.034][Trace][2914][636] ingest nothing changed for vhost=__defaultVhost__
[2020-03-12 10:10:54.034][Trace][2914][636] reload config success.

You can see that there is an additional fd=10, a_inode(inotify):

[root@05181e679dc0 trunk]# lsof -p `ps aux|grep srs|grep objs|awk {'print $2}'`
COMMAND  PID USER   FD      TYPE DEVICE SIZE/OFF       NODE NAME
srs     2753 root    7u     IPv4  27833      0t0        TCP *:macromedia-fcs (LISTEN)
srs     2753 root    8u     IPv4  27836      0t0        TCP *:hsrp (LISTEN)
srs     2753 root    9u     IPv4  27839      0t0        TCP *:webcache (LISTEN)
srs     2753 root   10r  a_inode   0,13        0      12074 inotify

TRANS_BY_GPT3

[winlinvip]

If the file is deleted or moved, and then created again (as part of the ConfigMap update logic).

When a file is deleted, including the target of symbolic links being moved or deleted:

[2020-03-12 11:31:16.387][Trace][981][674] inotify watch 2.conf fd=10, watch=1, mask=0xfff, gone=0
[2020-03-12 11:31:20.405][Trace][981][674] inotify event wd=1, mask=0x4, len=0, name=, reload=0, moved=0
[2020-03-12 11:31:20.406][Trace][981][674] inotify event wd=1, mask=0x400, len=0, name=, reload=0, moved=1
[2020-03-12 11:31:20.406][Trace][981][674] inotify event wd=1, mask=0x8000, len=0, name=, reload=0, moved=1
[2020-03-12 11:31:20.406][Trace][981][674] inotify remove fd=10, watch=1, r0=-1

When a file is created, including the creation of the target of symbolic links:

[2020-03-12 11:31:26.412][Trace][981][674] inotify watch 2.conf fd=10, watch=2, mask=0xfff, gone=1
[2020-03-12 11:31:26.414][Trace][981][674] reload config, signo=1
[2020-03-12 11:31:26.437][Trace][981][668] config parse complete
[2020-03-12 11:31:26.437][Trace][981][668] srs checking config...
[2020-03-12 11:31:26.437][Warn][981][668][11] stats network use index=0, ip=172.17.0.2
[2020-03-12 11:31:26.437][Warn][981][668][11] stats disk not configed, disk iops disabled.
[2020-03-12 11:31:26.438][Trace][981][668] write log to console
[2020-03-12 11:31:26.438][Trace][981][668] reload http_api success, nothing changed.
[2020-03-12 11:31:26.438][Trace][981][668] reload http stream success, nothing changed.
[2020-03-12 11:31:26.438][Trace][981][668] vhost __defaultVhost__ maybe modified, reload its detail.
[2020-03-12 11:31:26.438][Trace][981][668] ingest nothing changed for vhost=__defaultVhost__
[2020-03-12 11:31:26.438][Trace][981][668] reload config success.

TRANS_BY_GPT3

[winlinvip]

K8S will have a ..data subdirectory:

[root@srs-deploy-698ff4c4b9-jh7rc conf]# ls -al
total 16
drwxrwxrwx 3 root root 4096 Mar 12 11:49 .
drwxr-xr-x 1 root root 4096 Mar 12 11:42 ..
drwxr-xr-x 2 root root 4096 Mar 12 11:49 ..2020_03_12_11_49_56.260663928
lrwxrwxrwx 1 root root   31 Mar 12 11:49 ..data -> ..2020_03_12_11_49_56.260663928
lrwxrwxrwx 1 root root   15 Mar 12 11:48 srs.conf -> ..data/srs.conf

All events of K8S are as follows:

// Startup phase
[2020-03-12 14:13:54.482][Trace][1][652] auto reload watching fd=9, watch=1, file=conf
[2020-03-12 14:13:54.482][Trace][1][652] auto reload watching fd=9, watch=2, file=conf/..data
[2020-03-12 14:13:54.990][Trace][1][656] inotify event wd=1, mask=0x40000020, len=32, name=..2020_03_12_14_13_51.188104428, reload=0
[2020-03-12 14:13:54.990][Trace][1][656] inotify event wd=2, mask=0x40000020, len=0, name=, reload=0
[2020-03-12 14:13:54.990][Trace][1][656] inotify event wd=1, mask=0x40000010, len=32, name=..2020_03_12_14_13_51.188104428, reload=0
[2020-03-12 14:13:54.990][Trace][1][656] inotify event wd=2, mask=0x40000010, len=0, name=, reload=0
[2020-03-12 14:13:57.990][Trace][1][656] inotify event wd=2, mask=0x20, len=16, name=srs.conf, reload=0
[2020-03-12 14:13:57.990][Trace][1][656] inotify event wd=2, mask=0x1, len=16, name=srs.conf, reload=0
[2020-03-12 14:13:57.990][Trace][1][656] inotify event wd=2, mask=0x10, len=16, name=srs.conf, reload=0
[2020-03-12 14:13:57.990][Trace][1][656] inotify event wd=1, mask=0x40000020, len=32, name=..2020_03_12_14_13_51.188104428, reload=0
[2020-03-12 14:13:57.990][Trace][1][656] inotify event wd=2, mask=0x40000020, len=0, name=, reload=0
[2020-03-12 14:13:57.990][Trace][1][656] inotify event wd=1, mask=0x40000010, len=32, name=..2020_03_12_14_13_51.188104428, reload=0
[2020-03-12 14:13:57.990][Trace][1][656] inotify event wd=2, mask=0x40000010, len=0, name=, reload=0
[2020-03-12 14:13:57.990][Trace][1][656] inotify event wd=2, mask=0x20, len=16, name=srs.conf, reload=0
[2020-03-12 14:13:57.990][Trace][1][656] inotify event wd=2, mask=0x1, len=16, name=srs.conf, reload=0
[2020-03-12 14:13:57.990][Trace][1][656] inotify event wd=2, mask=0x10, len=16, name=srs.conf, reload=0
[2020-03-12 14:14:00.990][Trace][1][656] inotify event wd=2, mask=0x20, len=16, name=srs.conf, reload=0
[2020-03-12 14:14:00.990][Trace][1][656] inotify event wd=2, mask=0x1, len=16, name=srs.conf, reload=0
[2020-03-12 14:14:00.990][Trace][1][656] inotify event wd=2, mask=0x10, len=16, name=srs.conf, reload=0

// Modify ConfigMap
[2020-03-12 14:19:03.551][Trace][1][656] inotify event wd=1, mask=0x40000020, len=32, name=..2020_03_12_14_17_54.925365984, reload=0
[2020-03-12 14:19:03.551][Trace][1][656] inotify event wd=1, mask=0x40000010, len=32, name=..2020_03_12_14_17_54.925365984, reload=0
[2020-03-12 14:19:03.551][Trace][1][656] inotify event wd=1, mask=0x40000100, len=32, name=..2020_03_12_14_19_03.505526463, reload=0
[2020-03-12 14:19:03.551][Trace][1][656] inotify event wd=1, mask=0x40000004, len=32, name=..2020_03_12_14_19_03.505526463, reload=0
[2020-03-12 14:19:03.551][Trace][1][656] inotify event wd=1, mask=0x100, len=16, name=..data_tmp, reload=0
[2020-03-12 14:19:03.551][Trace][1][656] inotify event wd=1, mask=0x40, len=16, name=..data_tmp, reload=0
[2020-03-12 14:19:03.551][Trace][1][656] inotify event wd=1, mask=0x80, len=16, name=..data, reload=0
[2020-03-12 14:19:03.551][Trace][1][656] inotify event wd=1, mask=0x40000020, len=0, name=, reload=0
[2020-03-12 14:19:03.551][Trace][1][656] inotify event wd=1, mask=0x40000020, len=32, name=..2020_03_12_14_17_54.925365984, reload=0
[2020-03-12 14:19:03.551][Trace][1][656] inotify event wd=1, mask=0x40000010, len=32, name=..2020_03_12_14_17_54.925365984, reload=0
[2020-03-12 14:19:03.551][Trace][1][656] inotify event wd=1, mask=0x40000200, len=32, name=..2020_03_12_14_17_54.925365984, reload=0
[2020-03-12 14:19:03.551][Trace][1][656] inotify event wd=1, mask=0x40000010, len=0, name=13_51.188104428, reload=0

Note: The configuration was updated at 18:23 and applied to the Pod at 19:03, taking a total of about 40 seconds.

The key events are as follows, it appears that K8S executed a command similar to ln -sf ..2020_03_12_14_17_54.925365984 ..data_tmp && mv ..data_tmp ..data:

• 0x100 ..data_tmp IN_CREATE - File creation.
• 0x40 ..data_tmp IN_MOVED_FROM - File movement.
• 0x80 ..data IN_MOVED_TO - File movement.

inotify should listen to the conf directory.

• The creation and modification of the conf directory, IN_MODIFY | IN_CREATE | IN_MOVED_TO, so that the creation of the ..data subdirectory can also be detected.

After receiving the event, it is necessary to determine the file name. Only if it is srs.conf or ..data, it is considered a reload, and others are ignored.

TRANS_BY_GPT3