Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parse external request id from request headers, and print it in access log #22906

Merged
merged 24 commits into from
Mar 10, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
3ef5609
log reuqest id that parsed from request header
sillyguodong Feb 14, 2023
e23a9f6
fix
sillyguodong Feb 14, 2023
bca651f
update example and config cheat sheet
sillyguodong Feb 14, 2023
68ae309
put request id into access log
sillyguodong Feb 14, 2023
248ca3c
delete useless character
sillyguodong Feb 14, 2023
f15ce89
fix lint
sillyguodong Feb 14, 2023
e040fa8
set the maximum length of request id to 256 bits
sillyguodong Feb 15, 2023
55c2779
fix markdown
sillyguodong Feb 15, 2023
c5a5442
lint markdown
sillyguodong Feb 15, 2023
ec0d6c0
set maxRequestIDBtyeLength to 40
sillyguodong Feb 15, 2023
0c47e3f
fix comment misspelling
sillyguodong Feb 15, 2023
dbabb46
fix comment and the judgement of "needRequestID"
sillyguodong Feb 17, 2023
6f9d8da
Merge branch 'main' into feature/issue_22890
sillyguodong Feb 17, 2023
84a1a04
fix comment
sillyguodong Feb 17, 2023
20c36b3
Merge branch 'main' into feature/issue_22890
sillyguodong Feb 20, 2023
4a5bca7
Merge branch 'main' into feature/issue_22890
sillyguodong Feb 20, 2023
fc843f0
fix example
sillyguodong Feb 21, 2023
37d578c
Merge branch 'main' into feature/issue_22890
sillyguodong Feb 21, 2023
16d2d01
Merge branch 'main' into feature/issue_22890
lunny Mar 9, 2023
031f83e
Merge branch 'main' into feature/issue_22890
lunny Mar 9, 2023
29283b1
Merge branch 'main' into feature/issue_22890
jolheiser Mar 9, 2023
b387a1e
Update custom/conf/app.example.ini
zeripath Mar 9, 2023
f231c64
fix eample
sillyguodong Mar 10, 2023
a812d1d
Merge branch 'main' into feature/issue_22890
jolheiser Mar 10, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions custom/conf/app.example.ini
Original file line number Diff line number Diff line change
Expand Up @@ -576,6 +576,22 @@ ROUTER = console
;; The routing level will default to that of the system but individual router level can be set in
;; [log.<mode>.router] LEVEL
;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;
;; Print request id which parsed from request headers in access log, when access log is enabled.
;; * E.g:
;; * In request Header: X-Request-ID: test-id-123
;; * Configuration in app.ini: REQUEST_ID_HEADERS = X-Request-ID
;; * Print in log: 127.0.0.1:58384 - - [14/Feb/2023:16:33:51 +0800] "test-id-123"
;;
;; If you configure more than one in the .ini file, it will match in the order of configuration,
;; and the first match will be finally printed in the log.
;; * E.g:
;; * In reuqest Header: X-Trace-ID: trace-id-1q2w3e4r
;; * Configuration in app.ini: REQUEST_ID_HEADERS = X-Request-ID, X-Trace-ID, X-Req-ID
;; * Print in log: 127.0.0.1:58384 - - [14/Feb/2023:16:33:51 +0800] "trace-id-1q2w3e4r"
;;
;; REQUEST_ID_HEADERS =

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;
Expand Down
6 changes: 6 additions & 0 deletions docs/content/doc/advanced/config-cheat-sheet.en-us.md
Original file line number Diff line number Diff line change
Expand Up @@ -881,7 +881,13 @@ Default templates for project boards:
- `Identity`: the SignedUserName or `"-"` if not logged in.
- `Start`: the start time of the request.
- `ResponseWriter`: the responseWriter from the request.
- `RequestID`: the value matching REQUEST_ID_HEADERS(default: `-`, if not matched).
- You must be very careful to ensure that this template does not throw errors or panics as this template runs outside of the panic/recovery script.
- `REQUEST_ID_HEADERS`: **\<empty\>**: You can configure multiple values that are splited by comma here. It will match in the order of configuration, and the first match will be finally printed in the access log.
- e.g.
- In the Request Header: X-Request-ID: **test-id-123**
- Configuration in app.ini: REQUEST_ID_HEADERS = X-Request-ID
- Print in log: 127.0.0.1:58384 - - [14/Feb/2023:16:33:51 +0800] "**test-id-123**" ...

### Log subsections (`log.name`, `log.name.*`)

Expand Down
17 changes: 16 additions & 1 deletion docs/content/doc/advanced/config-cheat-sheet.zh-cn.md
Original file line number Diff line number Diff line change
Expand Up @@ -262,7 +262,22 @@ test01.xls: application/vnd.ms-excel; charset=binary

- `ROOT_PATH`: 日志文件根目录。
- `MODE`: 日志记录模式,默认是为 `console`。如果要写到多个通道,用逗号分隔
- `LEVEL`: 日志级别,默认为`Trace`。
- `LEVEL`: 日志级别,默认为 `Trace`。
- `DISABLE_ROUTER_LOG`: 关闭日志中的路由日志。
- `ENABLE_ACCESS_LOG`: 是否开启 Access Log, 默认为 false。
- `ACCESS_LOG_TEMPLATE`: `access.log` 输出内容的模板,默认模板:**`{{.Ctx.RemoteAddr}} - {{.Identity}} {{.Start.Format "[02/Jan/2006:15:04:05 -0700]" }} "{{.Ctx.Req.Method}} {{.Ctx.Req.URL.RequestURI}} {{.Ctx.Req.Proto}}" {{.ResponseWriter.Status}} {{.ResponseWriter.Size}} "{{.Ctx.Req.Referer}}\" \"{{.Ctx.Req.UserAgent}}"`**
模板支持以下参数:
- `Ctx`: 请求上下文。
- `Identity`: 登录用户名,默认: “`-`”。
- `Start`: 请求开始时间。
- `ResponseWriter`:
- `RequestID`: 从请求头中解析得到的与 `REQUEST_ID_HEADERS` 匹配的值,默认: “`-`”。
- 一定要谨慎配置该模板,否则可能会引起panic.
- `REQUEST_ID_HEADERS`: 从 Request Header 中匹配指定 Key,并将匹配到的值输出到 `access.log` 中(需要在 `ACCESS_LOG_TEMPLATE` 中指定输出位置)。如果在该参数中配置多个 Key, 请用逗号分割,程序将按照配置的顺序进行匹配。
- 示例:
- 请求头: X-Request-ID: **test-id-123**
- 配置文件: REQUEST_ID_HEADERS = X-Request-ID
- 日志输出: 127.0.0.1:58384 - - [14/Feb/2023:16:33:51 +0800] "**test-id-123**" ...

## Cron (`cron`)

Expand Down
34 changes: 34 additions & 0 deletions modules/context/access_log.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,9 @@ package context
import (
"bytes"
"context"
"fmt"
"net/http"
"strings"
"text/template"
"time"

Expand All @@ -20,20 +22,51 @@ type routerLoggerOptions struct {
Start *time.Time
ResponseWriter http.ResponseWriter
Ctx map[string]interface{}
RequestID *string
}

var signedUserNameStringPointerKey interface{} = "signedUserNameStringPointerKey"

const keyOfRequestIDInTemplate = ".RequestID"

// According to:
// TraceId: A valid trace identifier is a 16-byte array with at least one non-zero byte
// MD5 output is 16 or 32 bytes: md5-bytes is 16, md5-hex is 32
// SHA1: similar, SHA1-bytes is 20, SHA1-hex is 40.
// UUID is 128-bit, 32 hex chars, 36 ASCII chars with 4 dashes
// So, we accept a Request ID with a maximum character length of 40
const maxRequestIDByteLength = 40

func parseRequestIDFromRequestHeader(req *http.Request) string {
requestID := "-"
for _, key := range setting.Log.RequestIDHeaders {
if req.Header.Get(key) != "" {
requestID = req.Header.Get(key)
break
}
}
if len(requestID) > maxRequestIDByteLength {
requestID = fmt.Sprintf("%s...", requestID[:maxRequestIDByteLength])
}
return requestID
}

// AccessLogger returns a middleware to log access logger
func AccessLogger() func(http.Handler) http.Handler {
logger := log.GetLogger("access")
needRequestID := len(setting.Log.RequestIDHeaders) > 0 && strings.Contains(setting.Log.AccessLogTemplate, keyOfRequestIDInTemplate)
logTemplate, _ := template.New("log").Parse(setting.Log.AccessLogTemplate)
return func(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
start := time.Now()
identity := "-"
r := req.WithContext(context.WithValue(req.Context(), signedUserNameStringPointerKey, &identity))

var requestID string
if needRequestID {
requestID = parseRequestIDFromRequestHeader(req)
}

next.ServeHTTP(w, r)
rw := w.(ResponseWriter)

Expand All @@ -47,6 +80,7 @@ func AccessLogger() func(http.Handler) http.Handler {
"RemoteAddr": req.RemoteAddr,
"Req": req,
},
RequestID: &requestID,
})
if err != nil {
log.Error("Could not set up chi access logger: %v", err.Error())
Expand Down
2 changes: 2 additions & 0 deletions modules/setting/log.go
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ var Log struct {
EnableAccessLog bool
AccessLogTemplate string
BufferLength int64
RequestIDHeaders []string
}

// GetLogDescriptions returns a race safe set of descriptions
Expand Down Expand Up @@ -153,6 +154,7 @@ func loadLogFrom(rootCfg ConfigProvider) {
Log.AccessLogTemplate = sec.Key("ACCESS_LOG_TEMPLATE").MustString(
`{{.Ctx.RemoteAddr}} - {{.Identity}} {{.Start.Format "[02/Jan/2006:15:04:05 -0700]" }} "{{.Ctx.Req.Method}} {{.Ctx.Req.URL.RequestURI}} {{.Ctx.Req.Proto}}" {{.ResponseWriter.Status}} {{.ResponseWriter.Size}} "{{.Ctx.Req.Referer}}\" \"{{.Ctx.Req.UserAgent}}"`,
)
Log.RequestIDHeaders = sec.Key("REQUEST_ID_HEADERS").Strings(",")
// the `MustString` updates the default value, and `log.ACCESS` is used by `generateNamedLogger("access")` later
_ = rootCfg.Section("log").Key("ACCESS").MustString("file")

Expand Down