-
Notifications
You must be signed in to change notification settings - Fork 24.8k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Elasticsearch support to JSON logging (#36833)
In order to support JSON log format, a custom pattern layout was used and its configuration is enclosed in ESJsonLayout. Users are free to use their own patterns, but if smooth Beats integration is needed, they should use ESJsonLayout. EvilLoggerTests are left intact to make sure user's custom log patterns work fine. To populate additional fields node.id and cluster.uuid which are not available at start time, a cluster state update will have to be received and the values passed to log4j pattern converter. A ClusterStateObserver.Listener is used to receive only one ClusteStateUpdate. Once update is received the nodeId and clusterUUid are set in a static field in a NodeAndClusterIdConverter. Following fields are expected in JSON log lines: type, tiemstamp, level, component, cluster.name, node.name, node.id, cluster.uuid, message, stacktrace see ESJsonLayout.java for more details and field descriptions Docker log4j2 configuration is now almost the same as the one use for ES binary. The only difference is that docker is using console appenders, whereas ES is using file appenders. relates: #32850
- Loading branch information
Showing
31 changed files
with
1,624 additions
and
205 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,9 +1,43 @@ | ||
| status = error | ||
|
|
||
| appender.console.type = Console | ||
| appender.console.name = console | ||
| appender.console.layout.type = PatternLayout | ||
| appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n | ||
| # log action execution errors for easier debugging | ||
| logger.action.name = org.elasticsearch.action | ||
| logger.action.level = debug | ||
|
|
||
| appender.rolling.type = Console | ||
| appender.rolling.name = rolling | ||
| appender.rolling.layout.type = ESJsonLayout | ||
| appender.rolling.layout.type_name = server | ||
|
|
||
| rootLogger.level = info | ||
| rootLogger.appenderRef.console.ref = console | ||
| rootLogger.appenderRef.rolling.ref = rolling | ||
|
|
||
| appender.deprecation_rolling.type = Console | ||
| appender.deprecation_rolling.name = deprecation_rolling | ||
| appender.deprecation_rolling.layout.type = ESJsonLayout | ||
| appender.deprecation_rolling.layout.type_name = deprecation | ||
|
|
||
| logger.deprecation.name = org.elasticsearch.deprecation | ||
| logger.deprecation.level = warn | ||
| logger.deprecation.appenderRef.deprecation_rolling.ref = deprecation_rolling | ||
| logger.deprecation.additivity = false | ||
|
|
||
| appender.index_search_slowlog_rolling.type = Console | ||
| appender.index_search_slowlog_rolling.name = index_search_slowlog_rolling | ||
| appender.index_search_slowlog_rolling.layout.type = ESJsonLayout | ||
| appender.index_search_slowlog_rolling.layout.type_name = index_search_slowlog | ||
|
|
||
| logger.index_search_slowlog_rolling.name = index.search.slowlog | ||
| logger.index_search_slowlog_rolling.level = trace | ||
| logger.index_search_slowlog_rolling.appenderRef.index_search_slowlog_rolling.ref = index_search_slowlog_rolling | ||
| logger.index_search_slowlog_rolling.additivity = false | ||
|
|
||
| appender.index_indexing_slowlog_rolling.type = Console | ||
| appender.index_indexing_slowlog_rolling.name = index_indexing_slowlog_rolling | ||
| appender.index_indexing_slowlog_rolling.layout.type = ESJsonLayout | ||
| appender.index_indexing_slowlog_rolling.layout.type_name = index_indexing_slowlog | ||
|
|
||
| logger.index_indexing_slowlog.name = index.indexing.slowlog.index | ||
| logger.index_indexing_slowlog.level = trace | ||
| logger.index_indexing_slowlog.appenderRef.index_indexing_slowlog_rolling.ref = index_indexing_slowlog_rolling | ||
| logger.index_indexing_slowlog.additivity = false |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| [float] | ||
| [[breaking_70_logging_changes]] | ||
| === Logging changes | ||
|
|
||
| [float] | ||
| ==== New JSON format log files in `log` directory | ||
|
|
||
| Elasticsearch now will produce additional log files in JSON format. They will be stored in `*.json` suffix files. | ||
| Following files should be expected now in log directory: | ||
| * ${cluster_name}_server.json | ||
| * ${cluster_name}_deprecation.json | ||
| * ${cluster_name}_index_search_slowlog.json | ||
| * ${cluster_name}_index_indexing_slowlog.json | ||
| * ${cluster_name}.log | ||
| * ${cluster_name}_deprecation.log | ||
| * ${cluster_name}_index_search_slowlog.log | ||
| * ${cluster_name}_index_indexing_slowlog.log | ||
| * ${cluster_name}_audit.json | ||
| * gc.log | ||
|
|
||
| Note: You can configure which of these files are written by editing `log4j2.properties`. | ||
|
|
||
| [float] | ||
| ==== Log files ending with `*.log` deprecated | ||
| Log files with the `.log` file extension using the old pattern layout format | ||
| are now considered deprecated and the newly added JSON log file format with | ||
| the `.json` file extension should be used instead. | ||
| Note: GC logs which are written to the file `gc.log` will not be changed. | ||
|
|
||
| [float] | ||
| ==== Docker output in JSON format | ||
|
|
||
| All Docker console logs are now in JSON format. You can distinguish logs streams with the `type` field. |
Oops, something went wrong.