Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bump 2.31.1 #535

Merged
merged 1 commit into from
Feb 3, 2023
Merged

bump 2.31.1 #535

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions AWS_FLB_CHERRY_PICKS
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,3 +23,13 @@ https://github.com/PettitWesley/fluent-bit.git s3-log-key-warn-1_9 308b73558fb2d

# ECS Filter
https://github.com/PettitWesley/fluent-bit.git ecs-empty-metadata-fix-one-commit 24934c938e8f63700edf1230a09e4483ac5df6a3

# Add back Datadog fixes that do not trigger segfault
https://github.com/matthewfala/fluent-bit.git ecs-datadog-sequential-revert acc01a4bf4ce656023d82943bec2683b5b7755e5
https://github.com/matthewfala/fluent-bit.git ecs-datadog-sequential-revert ac30b7c876a95d44c33a406b2b58ce08b978587e

# Datadog Partial Fix Patch
https://github.com/matthewfala/fluent-bit.git datadog-ecs-patch 3c1ad69ada5bb6f2e448c6f39a1a0ea6a6f4ff17

# S3 retry limit configuration option
https://github.com/Claych/fluent-bit.git clay-retry-limit-1.9 76ae393e565ccf37c131afc58902c9fc3d15de12
2 changes: 1 addition & 1 deletion AWS_FOR_FLUENT_BIT_VERSION
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2.31.0
2.31.1
21 changes: 21 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,26 @@
# Changelog

### 2.31.1
This release includes:
* Fluent Bit [1.9.10](https://fluentbit.io/announcements/v1.9.10/)
* Amazon CloudWatch Logs for Fluent Bit 1.9.1
* Amazon Kinesis Streams for Fluent Bit 1.10.1
* Amazon Kinesis Firehose for Fluent Bit 1.7.1

Compared to `2.31.0`, this release adds the following feature that we are working on getting accepted upstream:
* Feature - Add a configuration option for S3 retry attempts [fluent-bit:6475](https://github.com/fluent/fluent-bit/pull/6475)
* Bug - Resolve a rare Datadog segfault that occurs when remapping tags [aws-for-fluent-bit:491](https://github.com/aws/aws-for-fluent-bit/issues/491)

Same as `2.31.0`, this release includes the following fixes and features that we are working on getting accepted upstream:
* Feature - Add `kinesis_firehose` and `kinesis_streams` support for `time_key_format` milliseconds with `%3N` option, and nanoseconds `9N` and `%L` options [fluent-bit:2831](https://github.com/fluent/fluent-bit/issues/2831)
* Feature - Support OpenSearch Serverless data ingestion via OpenSearch plugin [fluent-bit:6448](https://github.com/fluent/fluent-bit/pull/6448)
* Enhancement - Transition S3 to fully synchronous file uploads to improve plugin stability [fluent-bit:6573](https://github.com/fluent/fluent-bit/pull/6573)
* Bug - Mitigate Datadog output plugin issue by reverting recent PR [aws-for-fluent-bit:491](https://github.com/aws/aws-for-fluent-bit/issues/491)
* Bug - Format S3 filename with timestamp from the first log in uploaded file, rather than the time the first log was buffered by the s3 output [aws-for-fluent-bit:459](https://github.com/aws/aws-for-fluent-bit/issues/459)
* Bug - Resolve S3 logic to display `log_key` missing warning message if the configured `log_key` field is missing from log payload [fluent-bit:6557](https://github.com/fluent/fluent-bit/pull/6557)
* Bug - ECS Metadata filter gracefuly handle task metadata query errors and cache metadata processing state to improve performance [aws-for-fluent-bit:505](https://github.com/aws/aws-for-fluent-bit/issues/505)


### 2.31.0
This release includes:
* Fluent Bit [1.9.10](https://fluentbit.io/announcements/v1.9.10/)
Expand Down
2 changes: 1 addition & 1 deletion linux.version
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"linux": {
"version": "2.31.0",
"version": "2.31.1",
"build": "1",
"fluent-bit": "1.9.10",
"kinesis-plugin": "v1.10.1",
Expand Down
14 changes: 13 additions & 1 deletion windows.versions
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"windows": [
{
"version": "2.31.0",
"version": "2.31.1",
"build": "1",
"fluent-bit": "1.9.10",
"kinesis-plugin": "v1.10.1",
Expand All @@ -12,6 +12,18 @@
"latest": true,
"stable": false
},
{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remember to run the new Makefile target for validating this file which is in our runbook now.

@matthewfala @rawahars should we make that target run in the release target? that way we fail fast.

make validate-version-file-format

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ran the script. It works.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what do you think about just making it part of the release target so we already run it a fail fast?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good idea. I can submit a pr for it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

.PHONY: validate-version-file-format
validate-version-file-format:
	jq -e . windows.versions && true || false
	jq -e . linux.version && true || false

Actually, it's not clear what the point of this script is,

Is it supposed to fail if the following is not in JSON format?

{
  "linux": {
    "version": "2.31.1",
    "build": "1",
    "fluent-bit": "1.9.10",
    "kinesis-plugin": "v1.10.1",
    "firehose-plugin": "v1.7.1",
    "cloudwatch-plugin": "v1.9.1"
  }
}

the make validate-version-file-format does not, if that is the case.

Meanwhile it appears that the build linux plugins does fail if the version document is not JSON regardless of whether the validate-version-file-format command is run or not

./scripts/build_plugins.sh \
        --KINESIS_PLUGIN_CLONE_URL= \
        --KINESIS_PLUGIN_TAG= \
        --KINESIS_PLUGIN_BRANCH= \
        --FIREHOSE_PLUGIN_CLONE_URL= \
        --FIREHOSE_PLUGIN_TAG= \
        --FIREHOSE_PLUGIN_BRANCH= \
        --CLOUDWATCH_PLUGIN_CLONE_URL= \
        --CLOUDWATCH_PLUGIN_TAG= \
        --CLOUDWATCH_PLUGIN_BRANCH=
latest version not found for linux
make: *** [linux-plugins] Error 1

Our runbook specifies:
The above changes go in the linux.version and windows.versions file. Run make validate-version-file-format to check that the update was successful. If formatted incorrectly, it will lead to a release pipeline failure.

If the JSON file is not formatted correctly, it won't be printed out, so that may be a way to manually check for issues.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@PettitWesley I think it would be. great if we add it to the release target.

@matthewfala Yes that is correct. The make target would fail if the json files for Windows and Linux are not formatted properly. It would be a fast fail to ensure that our Pipelines do not fail due to invalid JSON.

I am not clear what you mean when you say the make validate-version-file-format does not, if that is the case.

For example-

{
  "linux": {
    "version": "2.31.1",
    "build": "1",
    "fluent-bit": "1.9.10",
    "kinesis-plugin": "v1.10.1",
    "firehose-plugin": "v1.7.1",
    "cloudwatch-plugin": "v1.9.1"
  }
}

This would succeed since the format of json is correct.

{
  "linux": {
    "version": "2.31.1",
    "build": "1",
    "fluent-bit": "1.9.10",
    "kinesis-plugin": "v1.10.1",
    "firehose-plugin": "v1.7.1",
    "cloudwatch-plugin": "v1.9.1",
  }
}

This would fail with the following error-

jq -e . linux.version && true || false
parse error: Expected another key-value pair at line 9, column 3
make: *** [validate-version-file-format] Error 1

This is because the above JSON is not formatted properly and has an extra comma at "cloudwatch-plugin": "v1.9.1",. Such errors can be easily identified before we commit the changes leading to pipeline failures.

"version": "2.31.0",
"build": "1",
"fluent-bit": "1.9.10",
"kinesis-plugin": "v1.10.1",
"firehose-plugin": "v1.7.1",
"cloudwatch-plugin": "v1.9.1",
"openssl": "3.0.7",
"flexBison": "2.5.22",
"latest": false,
"stable": false
},
{
"version": "2.30.0",
"build": "1",
Expand Down