Loki Logstash Plugin #1822

adityacs · 2020-03-19T12:47:47Z

What this PR does / why we need it:
Logstash plugin for Loki

Which issue(s) this PR fixes:
Fixes #1192

Checklist

Documentation added
Tests updated

adityacs · 2020-03-19T12:48:26Z

@cyriltovena @slim-bean Need your thoughts and guidance on this

cyriltovena · 2020-03-19T14:13:14Z

Can you remove the .class and .gem files, basically all artefacts. Feel free to add an .gitignore for that folter.

logstash/build.gradle

logstash/logstash-output-java_output_example.gemspec

cyriltovena · 2020-03-19T14:28:34Z

Have you considered writing the plugin only in ruby ? I think it would make our life easier since we also have a ruby plugin for fluentd.

adityacs · 2020-03-19T14:39:42Z

@cyriltovena Ruby is new to me. It would take little more time for me to write the plugin in Ruby. I totally agree that it would be lot easier to maintain the code since we already have fluentd in Ruby.

What do you think? If you say let's stick to Ruby, let's go with Ruby. I will close this and create new PR.

slim-bean · 2020-03-19T15:01:17Z

Hey @adityacs, firstly can't thank you enough for all your contributions lately. You are a pleasure to work with and we really appreciate everything you are doing!!

I think in this case Ruby would be preferred as adding the java toolchain I think is adding another amount of complication we may not want. It looks like most other logstash plugins are ruby based.

adityacs · 2020-03-19T15:06:44Z

@slim-bean Yeah Ruby would be better. I just took an easy route since I am familiar with Java 😜

I will work on the Ruby plugin and update this PR.

cyriltovena · 2020-03-19T15:08:37Z

I'm not the best but I've been involved a lot lately with the fluentd plugin in Ruby, so don't hesitate to ask question on #loki-dev by pinging me or @slim-bean.

adityacs · 2020-03-19T15:10:23Z

Sure @cyriltovena

adityacs · 2020-03-19T15:11:37Z

@slim-bean @cyriltovena Thanks for the feedback. It's a great experience to work with you guys.
Thanks again for all the guidance.

adityacs · 2020-03-26T14:12:26Z

Pushed plugin with Ruby code. Have to add add few tests.

adityacs · 2020-03-27T13:25:07Z

@cyriltovena @slim-bean Have added tests and also updated the docs. Kindly requesting you guys to review this

Also, let me know if you are able to build and test the plugin.

codecov-io · 2020-03-27T13:35:34Z

Codecov Report

Merging #1822 into master will decrease coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #1822      +/-   ##
==========================================
- Coverage   64.75%   64.72%   -0.03%     
==========================================
  Files         122      122              
  Lines        9271     9271              
==========================================
- Hits         6003     6001       -2     
- Misses       2853     2854       +1     
- Partials      415      416       +1

Impacted Files	Coverage Δ
pkg/promtail/targets/tailer.go	`76.13% <0.00%> (-2.28%)`	⬇️

cyriltovena · 2020-03-30T15:10:29Z

logstash/LICENSE

@@ -0,0 +1,202 @@
+


@slim-bean do we need to have a license ?

No separate license, this would fall under the license for the project itself

Will remove

cyriltovena · 2020-03-30T15:12:21Z

Can you remove binary artifacts. Feel free to add a .gitignore to this new folder if required.

cyriltovena · 2020-03-30T22:23:01Z

logstash/lib/logstash/outputs/loki.rb

+    data_labels = lbls.merge(@external_labels)
+
+    entry_hash = {
+      "ts" => event.get("@timestamp").to_i * (10**9) +  Time.new.nsec,


What's going on here ?

This converts the timestamp to nano seconds

I have doubts about the new here, may be a test since we know what timestamp are using logstash ;)

logstash/lib/logstash/outputs/loki.rb

cyriltovena · 2020-03-31T01:19:05Z

logstash/lib/logstash/outputs/loki.rb

+      "line" => event.get("message").to_s
+    }
+
+    @Channel.go do


I don’t think you need a go here ?

Not required. Will remove

Actually Channel.go is required here. Since we are using unbuffered channel(i.e without specifying capacity), if we don't use Channel.go it hangs there without proceeding with the code

Hanging is fine, if it is hanging it is probably waiting for the previous batch to be sent.

If you keep it this way, it means for each event, (million of them) create a go routine. That can be huge for the system.

You may want to use buffered but I think it's better to wait, this also signal, don't send more we're busy to the plugin.

Yeah, got your point. Will make the change

Signed-off-by: Cyril Tovena <[email protected]>

adityacs · 2020-07-14T03:49:24Z

cmd/logstash/README.md

+git clone [email protected]:elastic/logstash.git
+cd logstash
+git checkout tags/v7.6.2
+export LOGSTASH_PATH=`pwd`


adityacs · 2020-07-14T06:55:13Z

We are pending with Makefile changes to build this plugin. I will take care of this

Signed-off-by: Cyril Tovena <[email protected]>

adityacs · 2020-07-15T03:15:13Z

cmd/logstash/lib/logstash/outputs/loki/entry.rb

+            event.to_hash.each { |key,value|
+                next if key.start_with?('@')
+                next if value.is_a?(Hash)


We should consider the nested fields as well right? Any reason to skip it?

They can already create new field out of current field if they want to.

I think logstash is good enough for managing fields that we shouldn't try to do anything.

Now testing/writing docs and charts. Signed-off-by: Cyril Tovena <[email protected]>

cyriltovena · 2020-07-15T20:34:21Z

One sec I've just finished the code.

…loki_logstash_plugin Signed-off-by: Cyril Tovena <[email protected]>

cmd/logstash/lib/logstash/outputs/loki.rb

Signed-off-by: Cyril Tovena <[email protected]>

* Logstash plugin * include_labels * include_labels * Removes binary. Signed-off-by: Cyril Tovena <[email protected]> * Improve documentation and remove the push path. Signed-off-by: Cyril Tovena <[email protected]> * Move to cmd. Signed-off-by: Cyril Tovena <[email protected]> * Add more precision for jruby. Signed-off-by: Cyril Tovena <[email protected]> * Update docs/clients/logstash/README.md * p Signed-off-by: Cyril Tovena <[email protected]> * ignore Signed-off-by: Cyril Tovena <[email protected]> * remove ignore file/ Signed-off-by: Cyril Tovena <[email protected]> * More precision for installing jruby Signed-off-by: Cyril Tovena <[email protected]> * Rename without Grafana Signed-off-by: Cyril Tovena <[email protected]> * A lot of refactoring and testing. Signed-off-by: Cyril Tovena <[email protected]> * change delay logic * Fully tested version. Now testing/writing docs and charts. Signed-off-by: Cyril Tovena <[email protected]> * Forgot to save merge. Signed-off-by: Cyril Tovena <[email protected]> * working version. Signed-off-by: Cyril Tovena <[email protected]> * Makefile + easier docker build. Signed-off-by: Cyril Tovena <[email protected]> * adds ci to build logstash image. Signed-off-by: Cyril Tovena <[email protected]> * Fix build for logstash. Signed-off-by: Cyril Tovena <[email protected]> * Adds example with helm charts. Signed-off-by: Cyril Tovena <[email protected]> * Fix target to send 10 logs with logstash. Signed-off-by: Cyril Tovena <[email protected]> * Improved documentation. Signed-off-by: Cyril Tovena <[email protected]> * add missing helm add repo for external repo Signed-off-by: Cyril Tovena <[email protected]> * Review comment. Signed-off-by: Cyril Tovena <[email protected]> * Fixes loki service in Promtail. Signed-off-by: Cyril Tovena <[email protected]> * Update loki-stack version Signed-off-by: Cyril Tovena <[email protected]> Co-authored-by: Cyril Tovena <[email protected]>

adityacs mentioned this pull request Mar 19, 2020

Logstash plugin #1192

Closed

adityacs force-pushed the loki_logstash_plugin branch from 499a68a to c19554b Compare March 19, 2020 12:53

cyriltovena reviewed Mar 19, 2020

View reviewed changes

logstash/build.gradle Outdated Show resolved Hide resolved

cyriltovena reviewed Mar 19, 2020

View reviewed changes

logstash/build.gradle Outdated Show resolved Hide resolved

cyriltovena reviewed Mar 19, 2020

View reviewed changes

logstash/logstash-output-java_output_example.gemspec Outdated Show resolved Hide resolved

cyriltovena reviewed Mar 19, 2020

View reviewed changes

logstash/logstash-output-java_output_example.gemspec Outdated Show resolved Hide resolved

adityacs force-pushed the loki_logstash_plugin branch 2 times, most recently from 4754a74 to 9b1d683 Compare March 26, 2020 14:07

adityacs force-pushed the loki_logstash_plugin branch 4 times, most recently from 92b9c50 to 6acd391 Compare March 27, 2020 13:24

cyriltovena reviewed Mar 30, 2020

View reviewed changes

cyriltovena requested a review from wardbekker March 30, 2020 20:21

cyriltovena reviewed Mar 30, 2020

View reviewed changes

logstash/lib/logstash/outputs/loki.rb Outdated Show resolved Hide resolved

cyriltovena reviewed Mar 31, 2020

View reviewed changes

pull-request-size bot added the size/XL label Jul 13, 2020

Rename without Grafana

c2b33e3

Signed-off-by: Cyril Tovena <[email protected]>

adityacs commented Jul 14, 2020

View reviewed changes

A lot of refactoring and testing.

cbef9a4

Signed-off-by: Cyril Tovena <[email protected]>

adityacs commented Jul 15, 2020

View reviewed changes

adityacs and others added 2 commits July 15, 2020 16:49

change delay logic

d10f042

Fully tested version.

26875cb

Now testing/writing docs and charts. Signed-off-by: Cyril Tovena <[email protected]>

Merge branch 'loki_logstash_plugin' of github.com:adityacs/loki into …

5a1af98

…loki_logstash_plugin Signed-off-by: Cyril Tovena <[email protected]>

cyriltovena reviewed Jul 15, 2020

View reviewed changes

cmd/logstash/lib/logstash/outputs/loki.rb Outdated Show resolved Hide resolved

cyriltovena added 4 commits July 15, 2020 16:36

Forgot to save merge.

1366ef8

Signed-off-by: Cyril Tovena <[email protected]>

working version.

99f504d

Signed-off-by: Cyril Tovena <[email protected]>

Makefile + easier docker build.

4776178

Signed-off-by: Cyril Tovena <[email protected]>

adds ci to build logstash image.

425d375

Signed-off-by: Cyril Tovena <[email protected]>

pull-request-size bot added size/XXL and removed size/XL labels Jul 16, 2020

cyriltovena added 9 commits July 16, 2020 11:05

Fix build for logstash.

0d2acee

Signed-off-by: Cyril Tovena <[email protected]>

Adds example with helm charts.

37508d1

Signed-off-by: Cyril Tovena <[email protected]>

Fix target to send 10 logs with logstash.

a8cde47

Signed-off-by: Cyril Tovena <[email protected]>

Improved documentation.

0a266ec

Signed-off-by: Cyril Tovena <[email protected]>

add missing helm add repo for external repo

4ae8cc7

Signed-off-by: Cyril Tovena <[email protected]>

Review comment.

c87581a

Signed-off-by: Cyril Tovena <[email protected]>

Fixes loki service in Promtail.

3e6a5cf

Signed-off-by: Cyril Tovena <[email protected]>

Merge remote-tracking branch 'upstream/master' into loki_logstash_plugin

29ee215

Update loki-stack version

e9fb63a

Signed-off-by: Cyril Tovena <[email protected]>

cyriltovena approved these changes Jul 17, 2020

View reviewed changes

cyriltovena merged commit 4c60907 into grafana:master Jul 17, 2020

steven-sheehy mentioned this pull request Jul 23, 2020

Helm: Fix regression in chart name #2406

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Loki Logstash Plugin #1822

Loki Logstash Plugin #1822

adityacs commented Mar 19, 2020 •

edited

Loading

adityacs commented Mar 19, 2020

cyriltovena commented Mar 19, 2020 •

edited

Loading

cyriltovena commented Mar 19, 2020

adityacs commented Mar 19, 2020

slim-bean commented Mar 19, 2020

adityacs commented Mar 19, 2020

cyriltovena commented Mar 19, 2020

adityacs commented Mar 19, 2020

adityacs commented Mar 19, 2020

adityacs commented Mar 26, 2020 •

edited

Loading

adityacs commented Mar 27, 2020

codecov-io commented Mar 27, 2020 •

edited

Loading

cyriltovena Mar 30, 2020

slim-bean Mar 30, 2020

adityacs Mar 31, 2020

cyriltovena commented Mar 30, 2020

cyriltovena Mar 30, 2020

adityacs Mar 31, 2020

cyriltovena Mar 31, 2020

cyriltovena Mar 31, 2020

adityacs Mar 31, 2020

adityacs Mar 31, 2020

cyriltovena Mar 31, 2020 •

edited

Loading

adityacs Apr 1, 2020

adityacs Jul 14, 2020

adityacs commented Jul 14, 2020

adityacs Jul 15, 2020

cyriltovena Jul 15, 2020

cyriltovena commented Jul 15, 2020

Loki Logstash Plugin #1822

Loki Logstash Plugin #1822

Conversation

adityacs commented Mar 19, 2020 • edited Loading

adityacs commented Mar 19, 2020

cyriltovena commented Mar 19, 2020 • edited Loading

cyriltovena commented Mar 19, 2020

adityacs commented Mar 19, 2020

slim-bean commented Mar 19, 2020

adityacs commented Mar 19, 2020

cyriltovena commented Mar 19, 2020

adityacs commented Mar 19, 2020

adityacs commented Mar 19, 2020

adityacs commented Mar 26, 2020 • edited Loading

adityacs commented Mar 27, 2020

codecov-io commented Mar 27, 2020 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cyriltovena commented Mar 30, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cyriltovena Mar 31, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adityacs commented Jul 14, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cyriltovena commented Jul 15, 2020

adityacs commented Mar 19, 2020 •

edited

Loading

cyriltovena commented Mar 19, 2020 •

edited

Loading

adityacs commented Mar 26, 2020 •

edited

Loading

codecov-io commented Mar 27, 2020 •

edited

Loading

cyriltovena Mar 31, 2020 •

edited

Loading