Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

flb_aws_util: added function flb_aws_strftime_precision for time output #6472

Merged
merged 3 commits into from
Feb 13, 2023

Conversation

Claych
Copy link
Contributor

@Claych Claych commented Nov 28, 2022


Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
[INPUT]
    Name                         forward
    Listen                       0.0.0.0
    Port                         24224
[OUTPUT]
    Name                         kinesis_streams
    Match                        *
    region                       us-east-1
    stream                       test-clay-firehose
    time_key                     @timestamp
    time_key_format              %Y-%m-%dT%H:%M:%S.%3N.%L.%L
  • Debug log output from testing the change
    Normal time_key_format
[INPUT]
    Name                         forward
    Listen                       0.0.0.0
    Port                         24224

[OUTPUT]
    Name                         kinesis_streams
    Match                        *
    region                       us-east-1
    stream              test-clay-firehose
    time_key                     @timestamp
    time_key_format              %Y-%m-%dT%H:%M:%S.%3N.%L.%L
{"from":"userA","to":"userB","@timestamp":"2023-01-04T00:39:14.250.250161647.250161647.250161647"}
{"from":"userA","to":"userB","@timestamp":"2023-01-04T00:39:14.250.250229597.250229597.250229597"}
{"from":"userA","to":"userB","@timestamp":"2023-01-04T00:39:14.250.250403404.250403404.250403404"}
{"from":"userA","to":"userB","@timestamp":"2023-01-04T00:39:14.251.251213550.251213550.251213550"}
{"from":"userA","to":"userB","@timestamp":"2023-01-04T00:39:14.251.251456260.251456260.251456260"}
{"from":"userA","to":"userB","@timestamp":"2023-01-04T00:39:14.251.251516580.251516580.251516580"}
{"from":"userA","to":"userB","@timestamp":"2023-01-04T00:39:14.251.251573085.251573085.251573085"}
{"from":"userA","to":"userB","@timestamp":"2023-01-04T00:39:14.251.251810073.251810073.251810073"}
{"from":"userA","to":"userB","@timestamp":"2023-01-04T00:39:14.252.252052307.252052307.252052307"}
{"from":"userA","to":"userB","@timestamp":"2023-01-04T00:39:14.252.252310276.252310276.252310276"}
{"from":"userA","to":"userB","@timestamp":"2023-01-04T00:39:14.253.253333330.253333330.253333330"}

Wrong time_key_format

[INPUT]
    Name                         forward
    Listen                       0.0.0.0
    Port                         24224

[OUTPUT]
    Name                         kinesis_streams
    Match                        *
    region                       us-east-1
    stream              test-clay-firehose
    time_key                     @timestamp
    time_key_format              %Y-%m-%dT%H:%M:%S.%3N.%L.%L-%9N-9N~%9N-99N~%99N-99NN~%9%9NN-9NL~%9N%L

fluent-bit log

[2023/01/04 01:05:02] [ info] [output:kinesis_streams:kinesis_streams.0] worker #0 started
[2023/01/04 01:05:19] [error] [output:kinesis_streams:kinesis_streams.0] Failed to add time_key @timestamp to record, test-clay-firehose
[2023/01/04 01:05:19] [error] [output:kinesis_streams:kinesis_streams.0] Failed to add time_key @timestamp to record, test-clay-firehose
[2023/01/04 01:05:19] [error] [output:kinesis_streams:kinesis_streams.0] Failed to add time_key @timestamp to record, test-clay-firehose
[2023/01/04 01:05:19] [error] [output:kinesis_streams:kinesis_streams.0] Failed to add time_key @timestamp to record, test-clay-firehose
[2023/01/04 01:05:19] [error] [output:kinesis_streams:kinesis_streams.0] Failed to add time_key @timestamp to record, test-clay-firehose
{"from":"userA","to":"userB"}
{"from":"userA","to":"userB"}
{"from":"userA","to":"userB"}
{"from":"userA","to":"userB"}
{"from":"userA","to":"userB"}
{"from":"userA","to":"userB"}
{"from":"userA","to":"userB"}
{"from":"userA","to":"userB"}
  • Attached Valgrind output that shows no leaks or memory corruption was found
==8834== 
==8834== LEAK SUMMARY:
==8834==    definitely lost: 0 bytes in 0 blocks
==8834==    indirectly lost: 0 bytes in 0 blocks
==8834==      possibly lost: 0 bytes in 0 blocks
==8834==    still reachable: 102,912 bytes in 3,432 blocks
==8834==         suppressed: 0 bytes in 0 blocks
==8834== 
==8834== For lists of detected and suppressed errors, rerun with: -s
==8834== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

Documentation

  • Documentation required for this feature

fluent/fluent-bit-docs#981 fluent/fluent-bit-docs#980
Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@Claych Claych changed the title flb_aws_util: added function flb_aws_strftime_precision for time outp… flb_aws_util: added function flb_aws_strftime_precision for time output can support millisecond and nanosecond by adding %3N and %L to time_format. Nov 28, 2022
@Claych Claych changed the title flb_aws_util: added function flb_aws_strftime_precision for time output can support millisecond and nanosecond by adding %3N and %L to time_format. flb_aws_util: added function flb_aws_strftime_precision for time output can support millisecond and nanosecond by adding %3N and %9N or %L to time_format. Nov 28, 2022
@Claych Claych force-pushed the clay-aws_strftime_precision-1.9 branch 4 times, most recently from 7dbf7fd to 8b9ebb2 Compare November 29, 2022 18:02
PettitWesley
PettitWesley previously approved these changes Nov 30, 2022
@Claych Claych temporarily deployed to pr November 30, 2022 19:59 Inactive
@Claych Claych temporarily deployed to pr November 30, 2022 19:59 Inactive
@lecaros lecaros added this to the Fluent Bit v1.9.11 milestone Nov 30, 2022
@Claych Claych temporarily deployed to pr November 30, 2022 20:16 Inactive
@Claych Claych changed the title flb_aws_util: added function flb_aws_strftime_precision for time output can support millisecond and nanosecond by adding %3N and %9N or %L to time_format. flb_aws_util: added function flb_aws_strftime_precision for time output Dec 5, 2022
"%" PRIu64, (uint64_t) tms->tm.tv_nsec / 1000000);
snprintf(nanosecond_str, FLB_AWS_NANOSECOND_FORMATTER_LENGTH+1,
"%" PRIu64, (uint64_t) tms->tm.tv_nsec);
for (int i = 0; i < time_format_len; i++) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

declare variable at top of function not in for loop

@matthewfala
Copy link
Contributor

I get a Segfault with the following test:

[INPUT]
    Name                        dummy
    Tag                         log

[OUTPUT]
    Name                         kinesis_firehose
    Match                        *
    region                       us-west-2
    delivery_stream              firehose-compression-4-0
    time_key                     @timestamp
    time_key_format              %Y-%m-%dT%H:%M:%S.%3N.%L.%L-%9N-9N~%9N-99N~%99N-99NN~%9%9NN-9NL~%9N%L

@Claych Claych force-pushed the clay-aws_strftime_precision-1.9 branch 2 times, most recently from ffcb6fa to 11f03c2 Compare December 12, 2022 19:58
@Claych Claych force-pushed the clay-aws_strftime_precision-1.9 branch from 11f03c2 to 97579cc Compare December 14, 2022 03:28
matthewfala
matthewfala previously approved these changes Dec 22, 2022
Copy link
Contributor

@matthewfala matthewfala left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay. This looks better. As long as it is well tested, it has my approval!

Comment on lines 224 to 223
/*
* when function flb_aws_strftime_precision's return value is -1,
* time_key will not be added to record.
*/
flb_plg_error(ctx->ins, "Failed to add time_key %s to record, %s",
ctx->time_key, ctx->delivery_stream);
return -1;
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please read the comment above this function, it says:

-1 = failure, record not added

We discussed this and agreed we want to still add the record to the buffer, and just not add the timekey. This corresponds with a 0 return value.

*/
flb_plg_error(ctx->ins, "Failed to add time_key %s to record, %s",
ctx->time_key, ctx->stream_name);
return -1;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment as in firehose

@Claych Claych force-pushed the clay-aws_strftime_precision-1.9 branch from 97579cc to e225ff7 Compare December 28, 2022 01:25
@Claych Claych temporarily deployed to pr January 4, 2023 01:14 — with GitHub Actions Inactive
@Claych Claych temporarily deployed to pr January 4, 2023 01:14 — with GitHub Actions Inactive
@PettitWesley
Copy link
Contributor

@Claych I am concerned with the testing results:

{"from":"userA","to":"userB","@timestamp":"2022-11-18T04:58:45.649.649094581"}

This came from?

time_key_format              %Y-%m-%dT%H:%M:%S.%3N.%L.%L

How come nanoseconds aren't shown twice at the end of the timestamp above?

@Claych Claych temporarily deployed to pr January 4, 2023 01:31 — with GitHub Actions Inactive
@edsiper edsiper merged commit 0b0653a into fluent:1.9 Feb 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants