in_dummy: support new property start_time_sec/nsec

[nokute78]

timestamp issue (especially filter plugin issue like #2015) is hard to reproduce.
(e.g. write dummy logfile and use in_tail to parse it or use in_forward and send dummy message pack event ...)

I added new property dummy_sec and dummy_nsec for debugging.
It makes timestamp of event configurable.

If dummy_sec and dummy_nsec are not set, in_dummy works as usual.
in_dummy uses current time as timestamp.

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

a.conf:

[INPUT]
    Name dummy
    dummy_sec  1582135200
    dummy_nsec 1000000

[OUTPUT]
    Name stdout
    Match *.*

output is like this.
Timestamp is overwritten.

taka@ubuntu:~/git/fluent-bit/build/dummy_timestamp$ ../bin/fluent-bit -c a.conf 
Fluent Bit v1.5.0
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2020/04/16 17:19:52] [ info] [storage] version=1.0.3, initializing...
[2020/04/16 17:19:52] [ info] [storage] in-memory
[2020/04/16 17:19:52] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2020/04/16 17:19:52] [ info] [engine] started (pid=85027)
[2020/04/16 17:19:52] [ info] [sp] stream processor started
[0] dummy.0: [1582135200.001000000, {"message"=>"dummy"}]
[1] dummy.0: [1582135201.001701210, {"message"=>"dummy"}]
[2] dummy.0: [1582135202.002102959, {"message"=>"dummy"}]
[3] dummy.0: [1582135203.001472454, {"message"=>"dummy"}]
^C[engine] caught signal (SIGINT)
taka@ubuntu:~/git/fluent-bit/build/dummy_timestamp$ 

valgrind :

taka@ubuntu:~/git/fluent-bit/build/dummy_timestamp$ valgrind ../bin/fluent-bit -c a.conf 
==84554== Memcheck, a memory error detector
==84554== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==84554== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==84554== Command: ../bin/fluent-bit -c a.conf
==84554== 
Fluent Bit v1.5.0
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2020/04/16 16:51:41] [ info] [storage] version=1.0.3, initializing...
[2020/04/16 16:51:41] [ info] [storage] in-memory
[2020/04/16 16:51:41] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2020/04/16 16:51:41] [ info] [engine] started (pid=84554)
[2020/04/16 16:51:42] [ info] [sp] stream processor started
[0] dummy.0: [1582135200.001000000, {"message"=>"dummy"}]
[1] dummy.0: [1582135200.986187041, {"message"=>"dummy"}]
[2] dummy.0: [1582135201.986513491, {"message"=>"dummy"}]
[3] dummy.0: [1582135202.985781083, {"message"=>"dummy"}]
^C[engine] caught signal (SIGINT)
==84554== 
==84554== HEAP SUMMARY:
==84554==     in use at exit: 0 bytes in 0 blocks
==84554==   total heap usage: 230 allocs, 230 frees, 750,696 bytes allocated
==84554== 
==84554== All heap blocks were freed -- no leaks are possible
==84554== 
==84554== For counts of detected and suppressed errors, rerun with: -v
==84554== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
taka@ubuntu:~/git/fluent-bit/build/dummy_timestamp$ 

Documentation

• Documentation required for this feature

Key	Description
Dummy_sec	Dummy base timestamp in seconds.
Dummy_nsec	Dummy base timestamp in nanoseconds.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[fujimotos]

I tested this feature with my dev server. I can confirm it works.

$ ./bin/fluent-bit -i dummy -p dummy_sec=1 -p dummy_nsec=500000000 -o stdout
...
[0] dummy.0: [1.500000000, {"message"=>"dummy"}]
[1] dummy.0: [2.499944670, {"message"=>"dummy"}]

[edsiper]

thanks for the contribution and review.

@nokute78 in order to follow the semantics of other plugins, would you please rename the configuration properties ?, e.g:

https://github.com/fluent/fluent-bit/blob/master/plugins/in_cpu/cpu.c#L521

[nokute78]

@fujimotos @edsiper Thank you for reviewing.

in order to follow the semantics of other plugins, would you please rename the configuration properties ?

This property is for the timestamp of first event, not for the interval of event.
And we already support the property rate for Interval same as fluentd.

Hmm, how about starttime_sec ? or any ideas?

[nokute78]

@edsiper Any ideas for naming ?

Currently, we can generate dummy event like this.
[ Dummy_sec.Dummy_nsec, Tag, Dummy ]

Dummy* are properties to edit event.

property	description	example
Dummy_sec	timestamp (sec)	1582135200
Dummy_nsec	timestamp(nsec)	1000000
Tag	tag	dummy.test
Dummy	record (in JSON)	{"dummy":"test"}

In this case, dummy event is
[1582135200.001000000, dummy.test, {"dummy":"test"}]

[edsiper]

ahh got it.

start_time_sec and start_time_nsec sounds good to me, but I just realized that the core is not using nanoseconds:

https://github.com/fluent/fluent-bit/blob/master/lib/monkey/mk_core/mk_event_epoll.c#L185

[nokute78]

@edsiper OK, I renamed the properties.

I just realized that the core is not using nanoseconds:

No problem.
These properties are just used to modify MessagePack timestamp.
https://github.com/nokute78/fluent-bit/blob/in_dummy_timestamp/plugins/in_dummy/in_dummy.c#L46
They are not passed to timer API.

[edsiper]

thanks!