timestamp missing with kinesis output

[UladzimirSemiankou]

Bug Report

Describe the bug
I am running fluent-bit 1.2.2 with systemd input and AWS kinesis output. Systemd does not generate the timestamp field inside the log, but put it prior to it like this:

[0] host.unknown: [1565706755.786588000, {"TRANSPORT"=>"syslog", "PRIORITY"=>"6", "SYSLOG_FACILITY"=>"10", "SYSLOG_IDENTIFIER"=>"sshd", "UID"=>"1000", "GID"=>"1000", "BOOT_ID"=>"5601a829777c459e845852afc9bcbeaf", "MACHINE_ID"=>"e14407097f0e4283a492f816e4bf6e79", "HOSTNAME"=>"ip-172-16-5-69.us-east-2.compute.internal", "SYSLOG_PID"=>"11971", "PID"=>"11971", "MESSAGE"=>"Disconnected from 172.16.12.137 port 51494", "SOURCE_REALTIME_TIMESTAMP"=>"1565706755768498"}]

When sending such a message to es output, it works fine since the output adds the timestamp, but Kinesis output seems to strip anything outside of json, so log messages look like this:

{"BOOT_ID":"53fb93cfe3664a0180672a247a4826cd","CAP_EFFECTIVE":"3fffffffff","CMDLINE":"/sbin/auditd","COMM":"auditd","EXE":"/usr/sbin/auditd","GID":"0","HOSTNAME":"ip-172-16-14-74.us-east-2.compute.internal","MACHINE_ID":"e14407097f0e4283a492f816e4bf6e79","MESSAGE":"Audit daemon rotating log files with keep option","PID":"15321","PRIORITY":"5","SOURCE_REALTIME_TIMESTAMP":"1565700926388220","SYSLOG_FACILITY":"3","SYSLOG_IDENTIFIER":"auditd","SYSLOG_PID":"15321","SYSTEMD_CGROUP":"/system.slice/auditd.service","SYSTEMD_SLICE":"system.slice","SYSTEMD_UNIT":"auditd.service","TRANSPORT":"syslog","UID":"0"}

As a result, i dont have a date field in AWS Elasticsearch. Parsing log messages for timestamp is not an option, since they dont have a consistent place to extract it from.
Can anything be done about this?

Version used: 1.2.2 with Kinesis output plugin
Configuration:
` fluent-bit-service.conf: |-
[SERVICE]
Flush 5
Daemon Off
Log_Level info
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_PORT 2020
Parsers_File parsers.conf

fluent-bit-input.conf: |-
[INPUT]
Name systemd
Tag host.*
Read_From_Tail true
Strip_Underscores true

fluent-bit-filter.conf: |-
[FILTER]
Name modify
Match host.*
Add SYSTEMD_UNIT unspecified

fluent-bit-output.conf: |-
[OUTPUT]
Name firehose
Match *
region us-west-2
delivery_stream my-stream

Running in EKS.

[PettitWesley]

@UladzimirSemiankou curious if you're using the aws-for-fluent-bit image or the fluent/fluent-bit image? Because the systemd input plugin is broken in aws-for-fluent-bit right now - aws/amazon-cloudwatch-logs-for-fluent-bit#10 - though we'll release a fix for it soon.

Can anything be done about this?

I've seen some Fluentd plugins that have a config option like include_time_key, for example, the CloudWatch Logs Fluentd plugin. We need to add something like this to the plugin.

I think this is a high priority feature request.

[raskad]

I second this request. I tried this plugin today with the ECS FireLens preview. A timestamp field would be really nice.
I dont know how feasable it is to add the ability to specify the timestamp format, but that would be really nice too.

[UladzimirSemiankou]

@PettitWesley i've built the plugin on top of fluent/fluent-bit image, systemd works fine there

[UladzimirSemiankou]

any updates on this?

[ScOut3R]

Ran into this the other day. Timestamp disappears or if I manage to pass it along with Time_Keep it is treated as string. Using AWS ElasticSearch as the destination for Firehose and ECS as input.

[arthurmauvezin]

I just tried with amazon-kinesis-for-fluent-bit plugin and amazon-kinesis-streams-for-fluent-bit and it has the same effect. Timestamp is present locally in stdout but there is no date or time received by AWS Elasticsearch.

I use both these plugins in amazon/aws-for-fluent-bit:latest Docker image.

My configuration file:

[SERVICE]
    Flush           5
    Daemon          off
    Log_Level       debug
    Parsers_File /fluent-bit/parsers/parsers.conf

[INPUT]
    Name        forward
    Listen      0.0.0.0
    Port        24224

[FILTER]
    Name parser
    Match *
    Parser nginx
    Key_Name log
    Preserve_Key True
    Reserve_Data True

[OUTPUT]
    Name   stdout
    Match  *

[OUTPUT]
    Name   firehose
    Match  *
    region eu-west-1
    delivery_stream my-log-test

[OUTPUT]
    Name            kinesis
    Match           *
    region          eu-west-1
    stream          my-log-test-stream

[PettitWesley]

@UladzimirSemiankou @ScOut3R @arthurmauvezin @raskad Apologies for how long it took us to get to this; see the comment here for the planned config options: #20 (comment)

[PettitWesley]

@UladzimirSemiankou @ScOut3R @arthurmauvezin @raskad Released in 2.2.0: https://github.com/aws/aws-for-fluent-bit/releases/tag/v2.2.0