Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Turned off data integrity check #633

Merged
merged 9 commits into from
Jun 13, 2022
Merged

Turned off data integrity check #633

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
baca244
Turned off data integrity check when writing and reading logs from th…
avilevy18 May 26, 2022
f9285a1
Replaced outdated comment
avilevy18 May 27, 2022
f83c15d
Merge branch 'master' into storage-checksum-off
avilevy18 May 30, 2022
adf27bc
Addressed PR comments
avilevy18 Jun 3, 2022
0e81b1f
Merge branch 'master' into storage-checksum-off
avilevy18 Jun 3, 2022
d177711
Fixed typo in comment
avilevy18 Jun 8, 2022
83444c8
Merge branch 'master' into storage-checksum-off
avilevy18 Jun 8, 2022
842b255
Merge branch 'master' into storage-checksum-off
qingling128 Jun 9, 2022
c48072a
Merge branch 'master' into storage-checksum-off
avilevy18 Jun 13, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions confgenerator/fluentbit/service.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,8 @@ func (s Service) Component() Component {
// https://docs.fluentbit.io/manual/administration/buffering-and-storage#service-section-configuration
// storage.path is set by Fluent Bit systemd unit (e.g. /var/lib/google-cloud-ops-agent/fluent-bit/buffers).
"storage.sync": "normal",
// Enable the data integrity check when writing and reading data from the filesystem.
"storage.checksum": "on",
// Turned off due to empty spaces being considered as corrupted. Most users do not enforce checksum. See:b/233109163 for more info
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about:

Suggested change
// Turned off due to empty spaces being considered as corrupted. Most users do not enforce checksum. See:b/233109163 for more info
// Disable data integrity check to avoid spurious corruption errors due to empty spaces.
// Most users do not enforce checksum.

I don't believe there's a need to mention the bug in the code — git blame can trace it to this PR, which already references it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed in: adf27bc

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: Disable, not Disabled (because it describes what the option does, not what this PR did)…

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the feedback.

Changed in: d177711

"storage.checksum": "off",
// The maximum amount of data to load into the memory when processing old chunks from the backlog that is from previous Fluent Bit processes (e.g. Fluent Bit may have crashed or restarted).
"storage.backlog.mem_limit": "50M",
// Enable storage metrics in the built-in HTTP server.
Expand Down
2 changes: 1 addition & 1 deletion ...a/valid/linux/all-backward_compatible_with_explicit_exporters/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion confgenerator/testdata/valid/linux/all-built_in_config/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion confgenerator/testdata/valid/linux/all-user_config_file_deleted/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion confgenerator/testdata/valid/linux/logging-custom_log_level/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level warn
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion ...or/testdata/valid/linux/logging-default_overrides_disable_all/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion ...ator/testdata/valid/linux/logging-pipeline_multiple_pipelines/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion ...generator/testdata/valid/linux/logging-processor_exclude_logs/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion ...enerator/testdata/valid/linux/logging-processor_modify_fields/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion confgenerator/testdata/valid/linux/logging-processor_order/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion ...alid/linux/logging-processor_parse_json_and_parse_regex_types/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion ...erator/testdata/valid/linux/logging-processor_parse_multiline/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion .../linux/logging-processor_parse_regex_type_on_default_pipeline/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion confgenerator/testdata/valid/linux/logging-receiver_apache/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion ...generator/testdata/valid/linux/logging-receiver_apache_custom/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion confgenerator/testdata/valid/linux/logging-receiver_cassandra/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion ...erator/testdata/valid/linux/logging-receiver_cassandra_custom/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion confgenerator/testdata/valid/linux/logging-receiver_couchdb/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion ...generator/testdata/valid/linux/logging-receiver_elasticsearch/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion ...or/testdata/valid/linux/logging-receiver_elasticsearch_custom/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion .../testdata/valid/linux/logging-receiver_files_refresh_interval/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion ...ta/valid/linux/logging-receiver_files_type_multiple_receivers/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion confgenerator/testdata/valid/linux/logging-receiver_forward/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion ...ux/logging-receiver_forward_multiple_receivers_conflicting_id/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion ...id/linux/logging-receiver_forward_multiple_receivers_with_dot/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion ...d/linux/logging-receiver_forward_omitting_optional_parameters/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion confgenerator/testdata/valid/linux/logging-receiver_hadoop/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion ...testdata/valid/linux/logging-receiver_hadoop_refresh_interval/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion confgenerator/testdata/valid/linux/logging-receiver_hbase/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion confgenerator/testdata/valid/linux/logging-receiver_jetty/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion confgenerator/testdata/valid/linux/logging-receiver_kafka/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion confgenerator/testdata/valid/linux/logging-receiver_kafka_custom/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion confgenerator/testdata/valid/linux/logging-receiver_mongodb/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion confgenerator/testdata/valid/linux/logging-receiver_mysql/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion confgenerator/testdata/valid/linux/logging-receiver_mysql_custom/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion confgenerator/testdata/valid/linux/logging-receiver_nginx/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion confgenerator/testdata/valid/linux/logging-receiver_nginx_custom/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion confgenerator/testdata/valid/linux/logging-receiver_postgresql/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion ...rator/testdata/valid/linux/logging-receiver_postgresql_custom/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion confgenerator/testdata/valid/linux/logging-receiver_rabbitmq/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion confgenerator/testdata/valid/linux/logging-receiver_redis/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion confgenerator/testdata/valid/linux/logging-receiver_redis_custom/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion confgenerator/testdata/valid/linux/logging-receiver_solr/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
2 changes: 1 addition & 1 deletion ...a/valid/linux/logging-receiver_syslog_type_multiple_receivers/golden_fluent_bit_main.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Log_Level info
dns.resolver legacy
storage.backlog.mem_limit 50M
storage.checksum on
storage.checksum off
storage.max_chunks_up 128
storage.metrics on
storage.sync normal
Expand Down
Loading