Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

out_es: support write_operation option #4079

Merged
merged 8 commits into from
Jan 7, 2022
Merged

out_es: support write_operation option #4079

merged 8 commits into from
Jan 7, 2022

Conversation

liljaylj
Copy link
Contributor

@liljaylj liljaylj commented Sep 10, 2021

Adds write_operation option as in fluentd-plugin-elasticsearch (https://github.com/uken/fluent-plugin-elasticsearch#write_operation)

Fixes: #1430


Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
  • Debug log output from testing the change
  • Attached Valgrind output that shows no leaks or memory corruption was found

Documentation

  • Documentation required for this feature

fluent/fluent-bit-docs#603


Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@liljaylj
Copy link
Contributor Author

liljaylj commented Sep 10, 2021

Configuration example

elasticsearch.conf.txt

@liljaylj
Copy link
Contributor Author

liljaylj commented Sep 10, 2021

Test result

result of command (in ./build dir):

(cmake -DFLB_DEV=On -DFLB_TESTS_RUNTIME=On -DFLB_TESTS_INTERNAL=On .. && make && valgrind bin/flb-rt-out_elasticsearch) 2>&1 | tee /tmp/fluent-test.log

fluent-test.log

@liljaylj
Copy link
Contributor Author

Valgrind output

result of command (in ./build dir):

(cmake -DFLB_DEV=On -DFLB_TESTS_RUNTIME=On -DFLB_TESTS_INTERNAL=On .. && make && valgrind bin/fluent-bit -c /tmp/elasticsearch.conf.txt) 2>&1 | tee /tmp/fluent-valgrind.log

fluent-valgrind.log

Signed-off-by: Zhanibek Adilbekov <[email protected]>
@liljaylj liljaylj marked this pull request as ready for review September 11, 2021 03:20
Copy link
Collaborator

@nokute78 nokute78 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you attach valgrind output again ?

Below file is incomplete.
There is no summary information.
#4079 (comment)

plugins/out_es/es.c Outdated Show resolved Hide resolved
plugins/out_es/es.c Outdated Show resolved Hide resolved
plugins/out_es/es.c Outdated Show resolved Hide resolved
plugins/out_es/es_conf.c Outdated Show resolved Hide resolved
plugins/out_es/es.c Outdated Show resolved Hide resolved
@github-actions
Copy link
Contributor

github-actions bot commented Nov 3, 2021

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

@github-actions github-actions bot added the Stale label Nov 3, 2021
@edsiper
Copy link
Member

edsiper commented Dec 12, 2021

@liljaylj ping

@liljaylj
Copy link
Contributor Author

liljaylj commented Dec 16, 2021

Could you attach valgrind output again ?

Below file is incomplete. There is no summary information. #4079 (comment)

here you go) new valgrind log after latest commit
fluent-valgrind.log

@liljaylj
Copy link
Contributor Author

@liljaylj ping

Sorry, it was tough lately. Multiple projects was going to production)

Copy link
Collaborator

@nokute78 nokute78 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@liljaylj Thank you for updating.
I added some requests. Could you check them?

plugins/out_es/es_conf.c Show resolved Hide resolved
plugins/out_es/es.h Outdated Show resolved Hide resolved
@liljaylj liljaylj requested a review from nokute78 December 19, 2021 00:25
Copy link
Collaborator

@nokute78 nokute78 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for fixing and sorry for several requests.

  1. Could you fix conflicts?
  2. Could you modify commit message ?
    It should be out_es: something not fix: something

@liljaylj
Copy link
Contributor Author

Thank you for fixing and sorry for several requests.

1. Could you fix conflicts?

2. Could you modify commit message ?
   It should be `out_es: something` not `fix: something`

oh, sorry, my bad.
done)

@liljaylj liljaylj requested a review from nokute78 December 19, 2021 01:28
@nokute78
Copy link
Collaborator

@liljaylj Thank you for quick response.

DCO error is reported for latest commit.
Could you check it? Sorry for multiple requests.
https://probot.github.io/apps/dco/

Signed-off-by: Zhanibek Adilbekov <[email protected]>
@liljaylj
Copy link
Contributor Author

@liljaylj Thank you for quick response.

DCO error is reported for latest commit. Could you check it? Sorry for multiple requests. https://probot.github.io/apps/dco/

done

Copy link
Collaborator

@nokute78 nokute78 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! LGTM.

@edsiper edsiper merged commit d9a9196 into fluent:master Jan 7, 2022
0Delta pushed a commit to 0Delta/fluent-bit that referenced this pull request Jan 20, 2022
* out_es: support write_operation option

Signed-off-by: Zhanibek Adilbekov <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add the write_operation and upsert mode of the ES Fluentd plugin
3 participants