how to recycle running logs (#215) (#2415)

* how to recycle running logs * Update 0.FAQ.md Update 0.FAQ.md * addons * Update docs-2.0-en/5.configurations-and-logs/2.log-management/logs.md * Update docs-2.0-en/5.configurations-and-logs/2.log-management/logs.md * Update docs-2.0-en/5.configurations-and-logs/2.log-management/logs.md * Update docs-2.0-en/5.configurations-and-logs/2.log-management/logs.md * comment fix --------- Co-authored-by: Chris Chen <[email protected]>
vesoft-inc · Jan 5, 2024 · 7f931c4 · 7f931c4
1 parent 5c1455e
commit 7f931c4
Show file tree

Hide file tree

Showing 4 changed files with 314 additions and 16 deletions.
diff --git a/docs-2.0-en/20.appendix/0.FAQ.md b/docs-2.0-en/20.appendix/0.FAQ.md
@@ -313,12 +313,7 @@ You can use [NebulaGraph Algorithm](../graph-computing/nebula-algorithm.md).
 
 ### "The runtime log files are too large. How to recycle the logs?"
 
-By default, the runtime logs of NebulaGraph are stored in  `/usr/local/nebula/logs/`. The INFO level log files are `nebula-graphd.INFO, nebula-storaged.INFO, nebula-metad.INFO`. If an alarm or error occurs, the suffixes are modified as `.WARNING` or `.ERROR`.
-
-NebulaGraph uses [glog](https://github.com/google/glog) to print logs. `glog` cannot recycle the outdated files. To rotate logs, you can:
-
--  Add the parameters `timestamp_in_logfile_name=true` (timestamp added to logfile) and `max_log_size=500` (log size limit in MB) to the configuration files of the three services, and then use crontab to delete logs periodically. For more information, see [`Glog should delete old log files automatically`](https://github.com/google/glog/issues/423).
--  Use [logrotate](https://github.com/logrotate/logrotate) to manage log files. Before using logrotate, modify the configurations of corresponding services and set `timestamp_in_logfile_name` to `false`.
+{{nebula.name}} uses [glog](https://github.com/google/glog) for log printing, which does not support log recycling. You can manage runtime logs by using cron jobs or the log management tool [logrotate](https://man7.org/linux/man-pages/man8/logrotate.8.html). For operational details, see [Log recycling](../5.configurations-and-logs/2.log-management/logs.md).
 
 ### "How to check the NebulaGraph version?"
 

diff --git a/docs-2.0-en/5.configurations-and-logs/2.log-management/logs.md b/docs-2.0-en/5.configurations-and-logs/2.log-management/logs.md
@@ -75,6 +75,160 @@ If the log level is changed while NebulaGraph is running, it will be restored to
 
 RocksDB runtime logs are usually used to debug RocksDB parameters and stored in `/usr/local/nebula/data/storage/nebula/$id/data/LOG`. `$id` is the ID of the example.
 
-## Log recycle
+## Log recycling
 
-How to recycle logs when the runtime log file is too large, see [FAQs](../../20.appendix/0.FAQ.md).
+Glog does not inherently support log recycling. To implement this feature, you can either use [cron jobs](https://man7.org/linux/man-pages/man1/crontab.1.html) in Linux to regularly remove old log files or use the log management tool, [logrotate](https://github.com/logrotate/logrotate), to rotate logs for regular archiving and deletion.
+
+### Log recycling using cron jobs
+
+This section provides an example of how to use cron jobs to regularly delete old log files from the Graph service's runtime logs.
+
+1. In the Graph service configuration file, apply the following settings and restart the service:
+
+  ```bash
+  timestamp_in_logfile_name = true
+  max_log_size = 500
+  ```
+
+  - By setting `timestamp_in_logfile_name` to `true`, the log file name includes a timestamp, allowing regular deletion of old log files.
+  - The `max_log_size` parameter sets the maximum size of a single log file in MB, such as `500`. Once this size is exceeded, a new log file is automatically created. The default value is `1800`.
+
+2. Use the following command to open the cron job editor.
+
+  ```bash
+  crontab -e
+  ```
+
+3. Add a cron job command to the editor to regularly delete old log files.
+
+  ```bash
+  * * * * * find <log_path> -name "<YourProjectName>" -mtime +7 -delete
+  ```
+
+  !!! caution
+
+        The `find` command in the above command should be executed by the root user or a user with sudo privileges.
+
+  - `* * * * *`: This cron job time field signifies that the task is executed every minute. For other settings, see [Cron Expression](https://crontab.cronhub.io/).
+  - `<log_path>`: The path of the service runtime log file, such as `/usr/local/nebula/logs`.
+  - `<YourProjectName>`: The log file name, such as `nebula-graphd.*`.
+  - `-mtime +7`: This deletes log files that are older than 7 days. Alternatively, use `-mmin +n` to delete log files older than `n` minutes. For details, see the find command.
+  - `-delete`: This deletes log files that meet the conditions.
+
+  For example, to automatically delete the Graph service runtime log files older than 7 days at 3 o'clock every morning, use:
+
+  ```bash
+  0 3 * * * find /usr/local/nebula/logs -name nebula-graphd.* -mtime +7 -delete
+  ```
+
+4. Save the cron job and exit the editor.
+
+
+### Log recycling using logrotate
+
+Logrotate is a tool that can rotate specified log files for archiving and recycling.
+
+!!! note
+
+    You must be the root user or a user with sudo privileges to install or run logrotate.
+
+This section provides an example of how to use logrotate to manage the Graph service's `INFO` level log file (`/usr/local/nebula/logs/nebula-graphd.INFO.impl`). 
+
+1. In the Graph service configuration file, set `timestamp_in_logfile_name` to `false` so that the logrotate tool can recognize the log file name. Then, restart the service.
+
+  ```bash
+  timestamp_in_logfile_name = false
+  ```
+
+2. Install logrotate.
+   
+  - For Debian/Ubuntu:
+
+    ```bash
+    sudo apt-get install logrotate
+    ```
+
+  - For CentOS/RHEL:
+
+    ```bash
+    sudo yum install logrotate
+    ```
+
+3. Create a logrotate configuration file, add log rotation rules, and save the configuration file.
+
+  In the `/etc/logrotate.d` directory, create a new logrotate configuration file `nebula-graphd.INFO`.
+
+    ```bash
+    sudo vim /etc/logrotate.d/nebula-graphd.INFO
+    ```
+
+  Then, add the following content:
+  
+    ```bash
+    # The absolute path of the log file needs to be configured
+    # And the file name cannot be a symbolic link file, such as `nebula-graph.INFO`
+    /usr/local/nebula/logs/nebula-graphd.INFO.impl {
+        daily
+        rotate 2
+        copytruncate
+        nocompress
+        missingok
+        notifempty
+        create 644 root root
+        dateext
+        dateformat .%Y-%m-%d-%s
+        maxsize 1k
+    }
+    ``` 
+
+  | Parameter       | Description                                                  |
+  | --------------- | ------------------------------------------------------------ |
+  | `daily`         | Rotate the log daily. Other available time units include `hourly`, `daily`, `weekly`, `monthly`, and `yearly`. |
+  | `rotate 2`      | Keep the most recent 2 log files before deleting the older one. |
+  | `copytruncate`  | Copy the current log file and then truncate it, ensuring no disruption to the logging process. |
+  | `nocompress`    | Do not compress the old log files.                           |
+  | `missingok`     | Do not report errors if the log file is missing.             |
+  | `notifempty`    | Do not rotate the log file if it's empty.                    |
+  | `create 644 root root` | Create a new log file with the specified permissions and ownership. |
+  | `dateext`       | Add a date extension to the log file name. <br/>The default is the current date in the format `-%Y%m%d`. <br/>You can extend this using the `dateformat` option. |
+  | `dateformat .%Y-%m-%d-%s` | This must follow immediately after `dateext` and defines the file name after log rotation. <br/>Before V3.9.0, only `%Y`, `%m`, `%d`, and `%s` parameters were supported. <br/>Starting from V3.9.0, the `%H` parameter is also supported.|
+  | `maxsize 1k`   | Rotate the log when it exceeds 1 kilobyte (`1024` bytes) in size or when the specified time unit (e.g., `daily`) passes. <br/>You can use size units like `k` and `M`, with the default unit being bytes. |
+
+  Modify the parameters in the configuration file according to actual needs. For more information about parameter configuration, see [logrotate](https://github.com/logrotate/logrotate).
+
+4. Test the logrotate configuration.
+
+  To verify whether the logrotate configuration is correct, use the following command for testing.
+
+  ```bash
+  sudo logrotate --debug /etc/logrotate.d/nebula-graphd.INFO
+  ```
+
+5. Execute logrotate.
+
+  Although `logrotate` is typically executed automatically by cron jobs, you can manually execute the following command to perform log rotation immediately.
+
+  ```bash
+  sudo logrotate -fv /etc/logrotate.d/nebula-graphd.INFO
+  ```
+
+  `-fv`: `f` stands for forced execution, `v` stands for verbose output.
+
+6. Verify the log rotation results.
+
+  After log rotation, new log files are found in the `/usr/local/nebula/logs` directory, such as `nebula-graphd.INFO.impl.2024-01-04-1704338204`. The original log content is cleared, but the file is retained for new log entries. When the number of log files exceeds the value set by `rotate`, the oldest log file is deleted. 
+
+  For example, `rotate `2` means keeping the 2 most recently generated log files. When the number of log files exceeds 2, the oldest log file is deleted.
+
+  ```bash
+  [test@test logs]$ ll
+  -rw-r--r-- 1 root root        0 Jan  4 11:18 nebula-graphd.INFO.impl 
+  -rw-r--r-- 1 root root     6894 Jan  4 11:16 nebula-graphd.INFO.impl.2024-01-04-1704338204 # This file is deleted when a new log file is generated
+  -rw-r--r-- 1 root root      222 Jan  4 11:18 nebula-graphd.INFO.impl.2024-01-04-1704338287
+  [test@test logs]$ ll
+  -rw-r--r-- 1 root root        0 Jan  4 11:18 nebula-graphd.INFO.impl
+  -rw-r--r-- 1 root root      222 Jan  4 11:18 nebula-graphd.INFO.impl.2024-01-04-1704338287
+  -rw-r--r-- 1 root root      222 Jan  4 11:18 nebula-graphd.INFO.impl.2024-01-04-1704338339 # The new log file is generated
+  ```
+
+If you need to rotate multiple log files, create multiple configuration files in the `/etc/logrotate.d` directory, with each configuration file corresponding to a log file. For example, to rotate the `INFO` level log file and the `WARNING` level log file of the Meta service, create two configuration files `nebula-metad.INFO` and `nebula-metad.WARNING`, and add log rotation rules in them respectively.
diff --git a/docs-2.0-zh/20.appendix/0.FAQ.md b/docs-2.0-zh/20.appendix/0.FAQ.md
@@ -331,12 +331,7 @@ nebula > MATCH (s)<-[e]-() WHERE id(s) == "given" RETURN count(e); #入度
 
 ### 运行日志文件过大时如何回收日志？
 
-{{nebula.name}} 的运行日志默认在 `/usr/local/nebula/logs/` 下，正常 INFO 级别日志文件为 `nebula-graphd.INFO, nebula-storaged.INFO, nebula-metad.INFO`，报警和错误级别后缀为 `.WARNING` 和 `.ERROR`。
-
-{{nebula.name}} 使用 [glog](https://github.com/google/glog) 打印日志。glog 没有日志回收的功能，用户可以：
-
-- 在三种服务的配置文件内添加参数`timestamp_in_logfile_name=true`（日志文件添加时间戳）和`max_log_size=500`（日志大小限制，单位 MB），然后使用 crontab 设置定期任务回收日志文件。详情请参见 [Glog should delete old log files automatically](https://github.com/google/glog/issues/423)。
-- 使用 [logrotate](https://github.com/logrotate/logrotate) 实现[日志轮询](https://discuss.nebula-graph.com.cn/t/topic/7803)。使用 logrotate 管理日志前需修改相应 {{nebula.name}} 服务的配置，将`timestamp_in_logfile_name`参数的值改成`false`。
+{{nebula.name}}使用 [glog](https://github.com/google/glog) 打印日志，glog 没有日志回收的功能。用户可以通过定时任务或日志管理工具 [logrotate](https://man7.org/linux/man-pages/man8/logrotate.8.html) 来管理运行日志。操作详情，参见[回收日志](../5.configurations-and-logs/2.log-management/logs.md)。
 
 ### 如何查看 {{nebula.name}} 版本
 

diff --git a/docs-2.0-zh/5.configurations-and-logs/2.log-management/logs.md b/docs-2.0-zh/5.configurations-and-logs/2.log-management/logs.md
@@ -69,12 +69,166 @@ $ curl -X PUT -H "Content-Type: application/json" -d '{"minloglevel":0,"v":3}' "
 
 ```
 
-如果在 {{nebula.name}} 运行时修改了运行日志级别，重启服务后会恢复为配置文件中设置的级别，如果需要永久修改，请修改[配置文件](../1.configurations/1.configurations.md)。
+如果在{{nebula.name}}运行时修改了运行日志级别，重启服务后会恢复为配置文件中设置的级别，如果需要永久修改，请修改[配置文件](../1.configurations/1.configurations.md)并重启服务。
 
 ## RocksDB 运行日志
 
 RocksDB 的运行日志通常在 `/usr/local/nebula/data/storage/nebula/$id/data/LOG`, 其中 `$id` 为实例号。该日志通常用于调试 RocksDB 参数。
 
 ## 回收日志
 
-运行日志文件过大时如何回收日志，请参见[常见问题](../../20.appendix/0.FAQ.md)。
+glog 本身不支持回收日志，如果需要回收日志，可以使用 Linux 系统中[定时任务（Cron Jobs）](https://man7.org/linux/man-pages/man1/crontab.1.html)来定期删除旧的日志文件。或者，使用日志管理工具 [logrotate](https://github.com/logrotate/logrotate) 来轮转日志以定期归档和删除日志。
+
+### 使用定时任务回收日志
+
+本文以回收 Graph 服务的运行日志为例，说明如何使用定时任务来定期删除旧的日志文件。操作步骤如下：
+
+1. 在 [Graph 服务配置文件](../1.configurations/3.graph-config.md)中，进行如下配置，然后重启服务。
+
+  ```bash
+  timestamp_in_logfile_name = true
+  max_log_size = 500
+  ```
+
+  - 设置`timestamp_in_logfile_name`为`true`，这样日志文件名中会包含时间戳，以定期删除旧的日志文件。
+  - 添加`max_log_size`参数，设置单个日志文件的最大大小，例如`500`。超过这个大小后，会自动创建新的日志文件，单位 MB，默认值为`1800`。
+
+2. 在 Linux 系统中，使用如下命令编辑定时任务：
+
+  ```bash
+  crontab -e
+  ```
+
+3. 在定时任务中添加命令，以定期删除旧的日志文件。
+
+  ```bash
+  * * * * * find <log_path> -name "<YourProjectName>" -mtime +7 -delete
+  ```
+
+  !!! caution
+
+        以上命令中的`find`命令需要使用 root 用户或者具有 sudo 权限的用户来执行。
+
+  - `* * * * *`：定时任务的时间字段，五个星号表示这个任务每分钟都会执行。其他设置，参见[Cron Expression](https://crontab.cronhub.io/)。
+  - `<log_path>`：服务运行日志文件的路径，例如`/usr/local/nebula/logs`。
+  - `<YourProjectName>`：日志文件名，例如`nebula-graphd.*`。
+  - `-mtime +7`：表示删除更新时间超过 7 天的日志文件。也可以使用`-mmin +n`，表示删除更新时间超过 n 分钟的日志文件。详情参见 [find 命令](https://man7.org/linux/man-pages/man1/find.1.html)。
+  - `-delete`：表示删除满足条件的日志文件。
+
+  例如，每天凌晨 3 点自动删除更新时间超过 7 天的 Graph 服务运行日志文件的命令：
+
+  ```bash
+  0 3 * * * find /usr/local/nebula/logs -name nebula-graphd.* -mtime +7 -delete
+  ```
+
+4. 保存定时任务。
+
+
+### 使用 logrotate 回收日志
+
+用户可以使用 logrotate 工具对指定的日志文件进行轮转，以达到归档和回收日志的目的。
+
+!!! note
+
+    需要使用 root 用户或者具有 sudo 权限的用户来安装 logrotate 或者运行 logrotate。
+
+本文以回收 Graph 服务`INFO`级别的日志文件（`/usr/local/nebula/logs/nebula-graphd.INFO.impl`）为例说明如何使用 logrotate 工具。操作步骤如下：
+
+1. 在 [Graph 服务配置文件](../1.configurations/3.graph-config.md)中，将`timestamp_in_logfile_name`设置为`false`，以便 logrotate 工具可以识别日志文件名。然后重启服务。
+
+  ```bash
+  timestamp_in_logfile_name = false
+  ```
+
+2. 安装 logrotate。
+
+  - Debian/Ubuntu：
+
+    ```bash
+    sudo apt-get install logrotate
+    ```
+
+  - CentOS/RHEL：
+
+    ```bash
+    sudo yum install logrotate
+    ```
+
+3. 创建 logrotate 配置文件，添加日志轮转规则，然后保存配置文件。
+
+  在`/etc/logrotate.d`目录下，创建一个新的 logrotate 配置文件`nebula-graphd.INFO`
+
+    ```bash
+    sudo vim /etc/logrotate.d/nebula-graphd.INFO
+    ```
+
+  添加以下内容：
+
+    ```bash
+    # 需配置日志文件的绝对路径
+    # 并且文件名不能为软链接文件，如不能为`nebula-graph.INFO`
+    /usr/local/nebula/logs/nebula-graphd.INFO.impl {
+        daily
+        rotate 2
+        copytruncate
+        nocompress
+        missingok
+        notifempty
+        create 644 root root
+        dateext
+        dateformat .%Y-%m-%d-%s
+        maxsize 1k
+    }
+    ``` 
+
+  |参数|说明|
+  |:--|:--|
+  |`daily`| 每天轮转日志。可用的时间单位有：`hourly`、`daily`、`weekly`、`monthly`、`yearly`。|
+  |`rotate 2`| 在删除前日志文件前，其被轮转的次数。即保留最近生成的 2 个日志文件。|
+  |`copytruncate`| 将当前日志文件复制一份，然后清空当前日志文件。|
+  |`nocompress`| 不压缩旧的日志文件。|
+  |`missingok`| 如果日志文件丢失，不报告错误。|
+  |`notifempty`| 如果日志文件为空，不进行轮转。|
+  |`create 644 root root`| 创建新的日志文件，并设置适当的权限和所有者。|
+  |`dateext`| 在日志文件名中添加日期后缀。<br/>默认是当前日期。默认是`-%Y%m%d`的后缀。可用`dateformat`选项扩展配置。|
+  |`dateformat .%Y-%m-%d-%s`| 必须配合`dateext`使用，紧跟在下一行出现，定义文件切割后的文件名。<br/>在V3.9.0 之前，只支持`%Y`、`%m`、`%d`、`%s`参数。在 V3.9.0 及之后，支持 %H 参数。|
+  |`maxsize 1k`| 当日志文件大小超过`1`千字节（`1024`字节）或者超过设定的周期（如`daily`）时，进行日志轮转。可用的大小单位有：`k`、`M`，默认单位为字节。|
+
+  用户可以根据实际需求修改配置文件中的参数。更多关于参数的配置及解释，参见 [logrotate](https://man7.org/linux/man-pages/man8/logrotate.8.html)。
+
+4. 测试 logrotate 配置。
+
+  为了验证 logrotate 的配置是否正确，可以使用以下命令来进行测试：
+
+  ```bash
+  sudo logrotate --debug /etc/logrotate.d/nebula-graphd.INFO
+  ```
+
+5. 运行 logrotate。
+
+  尽管`logrotate`通常由定时作业自动执行，但也可以手动执行以下命令，以立即进行日志轮转：
+
+  ```bash
+  sudo logrotate -fv /etc/logrotate.d/nebula-graphd.INFO
+  ```
+
+  `-fv`：`f`表示强制执行，`v`表示打印详细信息。
+
+6. 查看日志轮转结果。
+
+  日志轮转后，会在`/usr/local/nebula/logs`目录下看到新的日志文件，例如`nebula-graphd.INFO.impl.2024-01-04-1704338204`。原始日志内容会被清空，但文件会被保留，新日志继续写入。当日志文件数量超过`rotate`设置的值时，会删除最旧的日志文件。
+
+  例如，`rotate 2`表示保留最近生成的 2 个日志文件，当日志文件数量超过 2 个时，会删除最旧的日志文件。
+
+  ```bash
+  [test@test logs]$ ll
+  -rw-r--r-- 1 root root        0 Jan  4 11:18 nebula-graphd.INFO.impl 
+  -rw-r--r-- 1 root root     6894 Jan  4 11:16 nebula-graphd.INFO.impl.2024-01-04-1704338204 # 当新的日志文件生成时，此文件被删除
+  -rw-r--r-- 1 root root      222 Jan  4 11:18 nebula-graphd.INFO.impl.2024-01-04-1704338287
+  [test@test logs]$ ll
+  -rw-r--r-- 1 root root        0 Jan  4 11:18 nebula-graphd.INFO.impl
+  -rw-r--r-- 1 root root      222 Jan  4 11:18 nebula-graphd.INFO.impl.2024-01-04-1704338287
+  -rw-r--r-- 1 root root      222 Jan  4 11:18 nebula-graphd.INFO.impl.2024-01-04-1704338339 # 新生成的日志文件
+  ```
+
+如果用户需要对多个日志文件进行轮转，可以在`/etc/logrotate.d`目录下创建多个配置文件，每个配置文件对应一个日志文件。例如，用户需要对 Meta 服务的`INFO`级别日志文件和`WARNING`级别日志文件进行轮转，可以创建两个配置文件`nebula-metad.INFO`和`nebula-metad.WARNING`，并在其中分别添加日志轮转规则。