Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Convert Filebeat kafka.* to ECS #9297

Merged
merged 4 commits into from
Dec 20, 2018
Merged

Convert Filebeat kafka.* to ECS #9297

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
df67448
Map 2 fields to ECS:
Nov 29, 2018
e8dc178
Document kafka module file migrations in ecs-migration
Dec 19, 2018
7a4383b
Alias two Kafka fields. Remove unused field definition.
Dec 19, 2018
64316a1
Changelog
Dec 19, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha1...v7.0.0-alpha2[Check the
- Rename many `redis.log.*` fields to map to ECS. {pull}9315[9315]
- Rename many `icinga.*` fields to map to ECS. {pull}9294[9294]
- Rename many `postgresql.log.*` fields to map to ECS. {pull}9303[9303]
- Rename many `kafka.log.*` fields to map to ECS. {pull}9297[9297]

*Metricbeat*

Expand Down
10 changes: 10 additions & 0 deletions dev-tools/ecs-migration.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -376,6 +376,16 @@
to: network.forwarded_ip
alias: true

## Kafka module

- from: kafka.log.level
to: log.level
alias: true

- from: kafka.log.message
to: message
alias: true

## NGINX module

- from: nginx.access.user_name
Expand Down
18 changes: 4 additions & 14 deletions filebeat/docs/fields.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5444,31 +5444,21 @@ Kafka log lines.



*`kafka.log.timestamp`*::
+
--
The timestamp from the log line.


--

*`kafka.log.level`*::
+
--
example: WARN

The log level.
type: alias

alias to: log.level

--

*`kafka.log.message`*::
+
--
type: text

The logged message.
type: alias

alias to: message

--

Expand Down
2 changes: 1 addition & 1 deletion filebeat/module/kafka/fields.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

16 changes: 7 additions & 9 deletions filebeat/module/kafka/log/_meta/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,17 +3,15 @@
description: >
Kafka log lines.
fields:
- name: timestamp
description: >
The timestamp from the log line.
- name: level
example: "WARN"
description: >
The log level.
type: alias
path: log.level
migration: true
- name: message
type: text
description: >
The logged message.
type: alias
path: message
migration: true

- name: component
type: keyword
description: >
Expand Down
6 changes: 3 additions & 3 deletions filebeat/module/kafka/log/ingest/pipeline.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,18 +6,18 @@
"field": "message",
"trace_match": true,
"patterns": [
"(?m)%{TIMESTAMP_ISO8601:kafka.log.timestamp}. %{LOGLEVEL:kafka.log.level} +%{JAVALOGMESSAGE:kafka.log.message} \\(%{JAVACLASS:kafka.log.class}\\)$[ \\n]*(?'kafka.log.trace.full'.*)"
"(?m)%{TIMESTAMP_ISO8601:kafka.log.timestamp}. %{LOGLEVEL:log.level} +%{JAVALOGMESSAGE:message} \\(%{JAVACLASS:kafka.log.class}\\)$[ \\n]*(?'kafka.log.trace.full'.*)"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds like one day we should have something related to Java class.

@felixbarny Is there a standardised way Java APM agent is storing this info?

]
}
},
{
"grok": {
"field": "kafka.log.message",
"field": "message",
"pattern_definitions": {
"KAFKA_COMPONENT": "[^\\]]*"
},
"patterns": [
"\\[%{KAFKA_COMPONENT:kafka.log.component}\\][,:.]? +%{JAVALOGMESSAGE:kafka.log.message}"
"\\[%{KAFKA_COMPONENT:kafka.log.component}\\][,:.]? +%{JAVALOGMESSAGE:message}"
],
"on_failure": [
{
Expand Down
100 changes: 40 additions & 60 deletions filebeat/module/kafka/log/test/controller.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,9 @@
"input.type": "log",
"kafka.log.class": "kafka.controller.ControllerEventManager$ControllerEventThread",
"kafka.log.component": "controller-event-thread",
"kafka.log.level": "INFO",
"kafka.log.message": "Starting",
"log.level": "INFO",
"log.offset": 0,
"message": "[2017-08-04 10:48:21,048] INFO [controller-event-thread]: Starting (kafka.controller.ControllerEventManager$ControllerEventThread)"
"message": "Starting"
},
{
"@timestamp": "2017-08-04T10:48:21.063Z",
Expand All @@ -18,10 +17,9 @@
"input.type": "log",
"kafka.log.class": "kafka.controller.KafkaController",
"kafka.log.component": "Controller 0",
"kafka.log.level": "INFO",
"kafka.log.message": "0 successfully elected as the controller",
"log.level": "INFO",
"log.offset": 131,
"message": "[2017-08-04 10:48:21,063] INFO [Controller 0]: 0 successfully elected as the controller (kafka.controller.KafkaController)"
"message": "0 successfully elected as the controller"
},
{
"@timestamp": "2017-08-04T10:48:21.064Z",
Expand All @@ -30,10 +28,9 @@
"input.type": "log",
"kafka.log.class": "kafka.controller.KafkaController",
"kafka.log.component": "Controller 0",
"kafka.log.level": "INFO",
"kafka.log.message": "Broker 0 starting become controller state transition",
"log.level": "INFO",
"log.offset": 254,
"message": "[2017-08-04 10:48:21,064] INFO [Controller 0]: Broker 0 starting become controller state transition (kafka.controller.KafkaController)"
"message": "Broker 0 starting become controller state transition"
},
{
"@timestamp": "2017-08-04T10:48:21.082Z",
Expand All @@ -42,10 +39,9 @@
"input.type": "log",
"kafka.log.class": "kafka.controller.KafkaController",
"kafka.log.component": "Controller 0",
"kafka.log.level": "INFO",
"kafka.log.message": "Controller 0 incremented epoch to 1",
"log.level": "INFO",
"log.offset": 389,
"message": "[2017-08-04 10:48:21,082] INFO [Controller 0]: Controller 0 incremented epoch to 1 (kafka.controller.KafkaController)"
"message": "Controller 0 incremented epoch to 1"
},
{
"@timestamp": "2017-08-04T10:48:21.085Z",
Expand All @@ -54,10 +50,9 @@
"input.type": "log",
"kafka.log.class": "kafka.controller.KafkaController",
"kafka.log.component": "Controller 0",
"kafka.log.level": "DEBUG",
"kafka.log.message": "Registering IsrChangeNotificationListener",
"log.level": "DEBUG",
"log.offset": 507,
"message": "[2017-08-04 10:48:21,085] DEBUG [Controller 0]: Registering IsrChangeNotificationListener (kafka.controller.KafkaController)"
"message": "Registering IsrChangeNotificationListener"
},
{
"@timestamp": "2017-08-04T10:48:21.154Z",
Expand All @@ -66,10 +61,9 @@
"input.type": "log",
"kafka.log.class": "kafka.controller.ReplicaStateMachine",
"kafka.log.component": "Replica state machine on controller 0",
"kafka.log.level": "INFO",
"kafka.log.message": "Started replica state machine with initial state -> Map()",
"log.level": "INFO",
"log.offset": 632,
"message": "[2017-08-04 10:48:21,154] INFO [Replica state machine on controller 0]: Started replica state machine with initial state -> Map() (kafka.controller.ReplicaStateMachine)"
"message": "Started replica state machine with initial state -> Map()"
},
{
"@timestamp": "2017-08-04T10:48:21.156Z",
Expand All @@ -78,10 +72,9 @@
"input.type": "log",
"kafka.log.class": "kafka.controller.PartitionStateMachine",
"kafka.log.component": "Partition state machine on Controller 0",
"kafka.log.level": "INFO",
"kafka.log.message": "Started partition state machine with initial state -> Map()",
"log.level": "INFO",
"log.offset": 801,
"message": "[2017-08-04 10:48:21,156] INFO [Partition state machine on Controller 0]: Started partition state machine with initial state -> Map() (kafka.controller.PartitionStateMachine)"
"message": "Started partition state machine with initial state -> Map()"
},
{
"@timestamp": "2017-08-04T10:48:21.157Z",
Expand All @@ -90,10 +83,9 @@
"input.type": "log",
"kafka.log.class": "kafka.controller.KafkaController",
"kafka.log.component": "Controller 0",
"kafka.log.level": "INFO",
"kafka.log.message": "Broker 0 is ready to serve as the new controller with epoch 1",
"log.level": "INFO",
"log.offset": 976,
"message": "[2017-08-04 10:48:21,157] INFO [Controller 0]: Broker 0 is ready to serve as the new controller with epoch 1 (kafka.controller.KafkaController)"
"message": "Broker 0 is ready to serve as the new controller with epoch 1"
},
{
"@timestamp": "2017-08-04T10:48:21.165Z",
Expand All @@ -102,10 +94,9 @@
"input.type": "log",
"kafka.log.class": "kafka.controller.PartitionStateMachine",
"kafka.log.component": "Partition state machine on Controller 0",
"kafka.log.level": "INFO",
"kafka.log.message": "Invoking state change to OnlinePartition for partitions ",
"log.level": "INFO",
"log.offset": 1120,
"message": "[2017-08-04 10:48:21,165] INFO [Partition state machine on Controller 0]: Invoking state change to OnlinePartition for partitions (kafka.controller.PartitionStateMachine)"
"message": "Invoking state change to OnlinePartition for partitions "
},
{
"@timestamp": "2017-08-04T11:44:22.588Z",
Expand All @@ -114,10 +105,9 @@
"input.type": "log",
"kafka.log.class": "kafka.controller.KafkaController",
"kafka.log.component": "Controller 0",
"kafka.log.level": "DEBUG",
"kafka.log.message": "Live brokers: ",
"log.level": "DEBUG",
"log.offset": 1292,
"message": "[2017-08-04 11:44:22,588] DEBUG [Controller 0]: Live brokers: (kafka.controller.KafkaController)"
"message": "Live brokers: "
},
{
"@timestamp": "2017-08-04T11:44:25.094Z",
Expand All @@ -126,10 +116,9 @@
"input.type": "log",
"kafka.log.class": "kafka.controller.ControllerEventManager$ControllerEventThread",
"kafka.log.component": "controller-event-thread",
"kafka.log.level": "INFO",
"kafka.log.message": "Shutting down",
"log.level": "INFO",
"log.offset": 1390,
"message": "[2017-08-04 11:44:25,094] INFO [controller-event-thread]: Shutting down (kafka.controller.ControllerEventManager$ControllerEventThread)"
"message": "Shutting down"
},
{
"@timestamp": "2017-08-04T11:44:25.095Z",
Expand All @@ -138,10 +127,9 @@
"input.type": "log",
"kafka.log.class": "kafka.controller.ControllerEventManager$ControllerEventThread",
"kafka.log.component": "controller-event-thread",
"kafka.log.level": "INFO",
"kafka.log.message": "Stopped",
"log.level": "INFO",
"log.offset": 1526,
"message": "[2017-08-04 11:44:25,095] INFO [controller-event-thread]: Stopped (kafka.controller.ControllerEventManager$ControllerEventThread)"
"message": "Stopped"
},
{
"@timestamp": "2017-08-04T11:44:25.097Z",
Expand All @@ -150,10 +138,9 @@
"input.type": "log",
"kafka.log.class": "kafka.controller.ControllerEventManager$ControllerEventThread",
"kafka.log.component": "controller-event-thread",
"kafka.log.level": "INFO",
"kafka.log.message": "Shutdown completed",
"log.level": "INFO",
"log.offset": 1656,
"message": "[2017-08-04 11:44:25,097] INFO [controller-event-thread]: Shutdown completed (kafka.controller.ControllerEventManager$ControllerEventThread)"
"message": "Shutdown completed"
},
{
"@timestamp": "2017-08-04T11:44:25.099Z",
Expand All @@ -162,10 +149,9 @@
"input.type": "log",
"kafka.log.class": "kafka.controller.KafkaController",
"kafka.log.component": "Controller 0",
"kafka.log.level": "DEBUG",
"kafka.log.message": "Controller resigning, broker id 0",
"log.level": "DEBUG",
"log.offset": 1797,
"message": "[2017-08-04 11:44:25,099] DEBUG [Controller 0]: Controller resigning, broker id 0 (kafka.controller.KafkaController)"
"message": "Controller resigning, broker id 0"
},
{
"@timestamp": "2017-08-04T11:44:25.100Z",
Expand All @@ -174,10 +160,9 @@
"input.type": "log",
"kafka.log.class": "kafka.controller.KafkaController",
"kafka.log.component": "Controller 0",
"kafka.log.level": "DEBUG",
"kafka.log.message": "De-registering IsrChangeNotificationListener",
"log.level": "DEBUG",
"log.offset": 1914,
"message": "[2017-08-04 11:44:25,100] DEBUG [Controller 0]: De-registering IsrChangeNotificationListener (kafka.controller.KafkaController)"
"message": "De-registering IsrChangeNotificationListener"
},
{
"@timestamp": "2017-08-04T11:44:25.105Z",
Expand All @@ -186,10 +171,9 @@
"input.type": "log",
"kafka.log.class": "kafka.controller.PartitionStateMachine",
"kafka.log.component": "Partition state machine on Controller 0",
"kafka.log.level": "INFO",
"kafka.log.message": "Stopped partition state machine",
"log.level": "INFO",
"log.offset": 2042,
"message": "[2017-08-04 11:44:25,105] INFO [Partition state machine on Controller 0]: Stopped partition state machine (kafka.controller.PartitionStateMachine)"
"message": "Stopped partition state machine"
},
{
"@timestamp": "2017-08-04T11:44:25.111Z",
Expand All @@ -198,10 +182,9 @@
"input.type": "log",
"kafka.log.class": "kafka.controller.ReplicaStateMachine",
"kafka.log.component": "Replica state machine on controller 0",
"kafka.log.level": "INFO",
"kafka.log.message": "Stopped replica state machine",
"log.level": "INFO",
"log.offset": 2189,
"message": "[2017-08-04 11:44:25,111] INFO [Replica state machine on controller 0]: Stopped replica state machine (kafka.controller.ReplicaStateMachine)"
"message": "Stopped replica state machine"
},
{
"@timestamp": "2017-08-04T11:44:25.112Z",
Expand All @@ -210,10 +193,9 @@
"input.type": "log",
"kafka.log.class": "kafka.controller.RequestSendThread",
"kafka.log.component": "Controller-0-to-broker-0-send-thread",
"kafka.log.level": "INFO",
"kafka.log.message": "Shutting down",
"log.level": "INFO",
"log.offset": 2330,
"message": "[2017-08-04 11:44:25,112] INFO [Controller-0-to-broker-0-send-thread]: Shutting down (kafka.controller.RequestSendThread)"
"message": "Shutting down"
},
{
"@timestamp": "2017-08-04T11:44:25.112Z",
Expand All @@ -222,10 +204,9 @@
"input.type": "log",
"kafka.log.class": "kafka.controller.RequestSendThread",
"kafka.log.component": "Controller-0-to-broker-0-send-thread",
"kafka.log.level": "INFO",
"kafka.log.message": "Stopped",
"log.level": "INFO",
"log.offset": 2452,
"message": "[2017-08-04 11:44:25,112] INFO [Controller-0-to-broker-0-send-thread]: Stopped (kafka.controller.RequestSendThread)"
"message": "Stopped"
},
{
"@timestamp": "2017-08-04T11:44:25.113Z",
Expand All @@ -234,9 +215,8 @@
"input.type": "log",
"kafka.log.class": "kafka.controller.RequestSendThread",
"kafka.log.component": "Controller-0-to-broker-0-send-thread",
"kafka.log.level": "INFO",
"kafka.log.message": "Shutdown completed",
"log.level": "INFO",
"log.offset": 2568,
"message": "[2017-08-04 11:44:25,113] INFO [Controller-0-to-broker-0-send-thread]: Shutdown completed (kafka.controller.RequestSendThread)"
"message": "Shutdown completed"
}
]
Loading