Java Entrypoint #8161

original-brownbear · 2017-09-07T14:40:46Z

Java entry point:

Created org.logstash.Logstash as entrypoint
Safely handle Ruby runtime (which sadly is still a singleton, moving away from that will require a few iterations on top of this)
Adjusted bat and sh entry point wrappers
Verified manually that performance is unchanged (i.e. all Java opts are still loaded properly)
Flattened .jar path to make it a little less bothersome to build the -cp string
Retained ability to load jars from Ruby via the global $LS_JARS_LOADED variable hack, to keep plugin specs that load LS as a .gem functional (like e.g. the ITs in LS itself)
No need for the gem jars magic anymore, the downloading and moving into place of jars is now all handled by Gradle

Description incoming

Works on Windows:

original-brownbear · 2017-09-07T15:02:29Z

should work out of the box with /bin/logstash after running ./gradlew clean assemble btw

jordansissel · 2017-09-07T15:11:40Z

The last time we had lots of ruby code (aka jruby-complete) living in a jar, it had really bad effects on Logstash startup time because (at the time) require from JRuby loading ruby code from a .jar was very slow. We accidentally worked around it by having jruby ruby source as files when we moved away from the monolithic jar (years ago).

jordansissel · 2017-09-07T15:06:45Z

logstash-core/src/main/java/org/logstash/Main.java

+import org.jruby.exceptions.RaiseException;
+import org.jruby.runtime.builtin.IRubyObject;
+
+public final class Main implements Runnable {


Let's call this Logstash so that when folks do jps "Logstash" is what shows up instead of "Main"

andrewvc · 2017-09-11T13:21:53Z

This is awesome.

What I think this needs to move to completion is a wrapper for arbitrary ruby tasks like rake and bin/rspec, + adaptation of the Gradle stuff to use this.

I'd rather move the whole thing over than just the main logstash entrypoint for environment consistency.

original-brownbear · 2017-09-11T13:25:42Z

@andrewvc sure we can do that, already figured out how and played with that a little, will add that here soon. @jordansissel 's point is valid though, we probably shouldn't pack the .rb files into the .jar to not make the start-up crazy slow.
Also, just as a note for myself, I need to add the jvm.properties file handling to this thing, those settings are currently ignored making performance pretty meh :)

original-brownbear · 2017-10-25T12:30:47Z

@andrewvc just to summarize what we talked about:

I think moving the rake tasks to Gradle will be tough to do at this point. I'm seeing some conflicts between Gradle and the Ruby Maven gem (at least I think that's the root cause of all issues) that make building LS from only the Gradle JVM very tricky and hard to maintain.
The solution here has some issues with the load order, that only do not cause harm accidentally. I will have to do some more work here and at least fix We have an annoying cyclic dependency (Ruby <-> Java) because of LogStash::Json and org.logstash.ext.JrubyEventExtLibrary#load #8154 before we will get value (mainly thinking about JUnit tests here) out of moving to a Java entrypoint.

original-brownbear · 2017-10-26T12:06:41Z

@jordansissel hah I learned a new thing :) Apparently the only reason loading .rbs from a Jar comes with a hit is the compression. Turning off the Jar compression via https://docs.gradle.org/current/javadoc/org/gradle/api/tasks/bundling/ZipEntryCompression.html#STORED should make packing everything into a single jar as fast as reading straight from the FS (if not faster, since it means only loading a single file instead of hundreds).

original-brownbear · 2017-11-11T11:53:13Z

@andrewvc this version packages and works just fine (but there are a few open questions around the effects this will have on the IT and plugin test infrastructures ... ITs are red because loading Java first breaks the transitive including of jars for the ITs that include core a gem. But easy to fix, just a question of what we want to achieve).

Let's look into this whenever you have some time next week:)

jordansissel · 2017-11-13T17:49:42Z

@original-brownbear I"m open to reviewing this; lemme know when you think it's ready for testing and review :)

original-brownbear · 2017-11-13T17:50:59Z

@jordansissel thanks, hopefully today, will ping you then :)

original-brownbear · 2017-11-14T18:32:06Z

@jordansissel alright, now it should be good to review :)

original-brownbear · 2017-11-15T17:21:47Z

Jenkins test this please

original-brownbear · 2017-11-15T17:35:25Z

Sorry, for some reason the Kafka IT is broken now (was all green before a few trivial adjustments), will have to look into that and ping once it's green.

jordansissel · 2017-12-28T18:28:53Z

This is on my todo list for this week to test/review further.

jordansissel

Did a brief code review to check for user experience or deployment issues.

I'll test this separately.

jordansissel · 2017-12-28T18:48:38Z

bin/logstash

@@ -58,5 +58,8 @@ if [ "$1" = "-V" ] || [ "$1" = "--version" ]; then
  fi
  echo "logstash $LOGSTASH_VERSION"
 else
-  ruby_exec "${LOGSTASH_HOME}/lib/bootstrap/environment.rb" "logstash/runner.rb" "$@"
+  function join_cp { local IFS=":"; echo "$*"; }
+  jars=(${LOGSTASH_HOME}/logstash-core/lib/jars/*.jar)


All systems may not have bash. For example, docker containers or other similarly-minimal Linux environments.

Alternatives

Option 1: Only use a single .jar

This means -cp logstash-core/lib/jars/logstash.jar and any build changes required to achieve it.

Option 2: Compute classpath at build-time, not runtime.

This would have the build process compute the classpath so that we wouldn't need to compute the classpath during bin/logstash execution.

Option 3: Same method as you, but with bourne shell (not bash)

If you need to build a class path, you can do this without Bash's array features using $@ which has special meaning in bourne shell.

There may be other ways to achieve this, but here is what I came up with without testing:

Example:

sh-4.4$ ls /tmp/*.jar /tmp/a.jar /tmp/b.jar '/tmp/c c.jar' /tmp/d.jar sh-4.4$ function classpath() { > for i in "$@" ; do > echo -n "${i}:" > done | sed -e 's/:$//' > } sh-4.4$ classpath /tmp/*.jar; echo /tmp/a.jar:/tmp/b.jar:/tmp/c c.jar:/tmp/d.jar ## Alternative shell implementation function classpath() { echo -n "$1" shift while [ $# -gt 0 ] ; do echo -n ":${1}" shift done }

@jordansissel I'd go for 3 here.

Is really hacky (and I think even problematic legally in some cases) since we have to flatten all jars (or do some other hack that deals with nested jars) into one which entails removing the checksums from their META-INF paths => not something we want to do imo.

I don't have a good theoretical argument against this, but the build is already quite complicated isn't it? I'd rather not pile onto that :) Plus I was just able to remove the generated files like logstash_core_jars.rb from the production code paths, I'd rather remove these hacks and not just add a new generated piece of code.

=> sh is def more portable than bash, let me try your code :) thanks

@jordansissel hmm your solution works like a charm, but unfortunately only on newer sh. On vanilla OSX with sh 3.2 the -n is not understood by echo and the solution will only work with bash.
But you know what bash must be fine for us ... we are currently calling jruby/bin/jruby from our scripts and it's a bash script too => we're only compatible if bash is available right now anyways.
=> Use your solution but with bash to work on systems with older sh? :)

jordansissel · 2017-12-28T18:59:56Z

logstash-core/lib/logstash-core/logstash-core.rb

-    require "logstash-core/logstash-core.jar"
-  rescue Exception => e
-    raise("Error loading logstash-core/logstash-core.jar file, cause: #{e.message}")
+unless $LS_JARS_LOADED


Hmm.. If we have two places (bin/logstash and this file) where we compute some classpath things, can we move this to a single place or at least a single language (ruby vs shell vs java)?

Maybe have org.logstash.Logstash have a method compute this dynamically for use in various places (testing, main method, etc) ?

@jordansissel can't do it :( org.logstash.Logstash can't load without loading the jars beforehand (doing this kind of thing from Java and then dynamically loading the jars would require a massive hack I think). This we really only need for having the bin/rspec tool still work.
If we are fine with dropping that tool we don't need it, but I'd rather keep it around so I can debug Ruby comfortably from the IDE :)

This we really only need for having the bin/rspec tool still work.

Can you note this in a code comment?

jordansissel · 2017-12-28T19:04:02Z

logstash-core/src/main/java/org/logstash/Logstash.java

+    public static void main(final String... args) {
+        final String lsHome = System.getenv("LS_HOME");
+        if (lsHome == null) {
+            throw new IllegalStateException("LS_HOME environment variable must be set.");


Can we make this error message more actionable? If a user sees this exception, what should they do to resolve it, and can we include hints for resolving this in the message?

@jordansissel hmm I think the user can't do anything they see this message (other than open an issue here). This being thrown means there is a bug in the bin/logstash or bin/logstash.bat. I simply put this check here for our convenience to have a clean failure instead of some other failure with a strange null in some string.

other than open an issue here

Ok cool. Let's tell them it's probably a bug in the error message.

Let's include an action to tell them to file an issue or otherwise note that this is a bug and not a user configuration problem.

@jordansissel isn't probably a bug implied in this being a RuntimeException? :)

As a user, I don't personally associate "this is a bug" with RuntimeException. I'd rather be explicit. Saying "must be set" implies to me that the user should take action to set LS_HOME, which likely isn't the case. Otherwise, a user will end up struggling with this action trying to figure out how, and what value, to set LS_HOME.

jordansissel · 2017-12-28T19:12:18Z

logstash-core/src/main/java/org/logstash/Logstash.java

+            resolved = resolved.resolve(element);
+        }
+        if (!resolved.toFile().exists()) {
+            throw new IllegalArgumentException(String.format("Missing: %s.", resolved));


Can we make this more actionable, or would a user never expect to see this?

@jordansissel jup same here => user should never ever see this except for when there's a bug :)

jordansissel · 2017-12-28T19:12:56Z

logstash-core/src/main/java/org/logstash/Logstash.java

+        if (lsHome == null) {
+            throw new IllegalStateException("LS_HOME environment variable must be set.");
+        }
+        final Path home = Paths.get(lsHome).toAbsolutePath();


Can we verify some properties about this path before using it: Does the path exist? Does it have things we want in it? etc..

My thought is we can do a quick check here and provide actionable response to the user if the check fails.

@jordansissel same as the above, this will actually throw something like "file not found" if the path is wrong which we will understand, for the user this means there is a bug in the startup script in 100% of cases imo => nothing actionable they can do right?

I'm OK with this, no change required here.

jordansissel · 2017-12-28T19:35:15Z

should work out of the box with /bin/logstash after running ./gradlew clean assemble btw

% ./gradlew clean assemble
...
% bin/logstash -e ''
Unable to find JRuby.
If you are a user, this is a bug.
If you are a developer, please run 'rake bootstrap'. Running 'rake' requires the 'ruby' program be available.

jordansissel · 2017-12-28T19:46:14Z

If I try to work around the above error (can't find jruby) with this patch:

diff --git a/bin/logstash.lib.sh b/bin/logstash.lib.sh
index bb9090808..7f01bcd32 100755
--- a/bin/logstash.lib.sh
+++ b/bin/logstash.lib.sh
@@ -120,7 +120,7 @@ setup_vendored_jruby() {

 setup() {
   setup_java
-  setup_vendored_jruby
+  #setup_vendored_jruby
 }

 ruby_exec() {

Then run bin/logstash, it appears to do nothing?

% time bin/logstash --log.level debug -e 'input { generator { count => 1 } }'
bin/logstash --log.level debug -e 'input { generator { count => 1 } }'  11.93s user 0.84s system 292% cpu 4.366 total

Running bin/logstash with sh -x to see what's going on, I see this as the command, if this helps:

+ exec /bin/java -Xms1g -Xmx1g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -cp /home/jls/projects/logstash/logstash-core/lib/jars/commons-compiler-3.0.7.jar:/home/jls/projects/logstash/logstash-core/lib/jars/jackson-annotations-2.9.1.jar:/home/jls/projects/logstash/logstash-core/lib/jars/jackson-core-2.9.1.jar:/home/jls/projects/logstash/logstash-core/lib/jars/jackson-databind-2.9.1.jar:/home/jls/projects/logstash/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.1.jar:/home/jls/projects/logstash/logstash-core/lib/jars/janino-3.0.7.jar:/home/jls/projects/logstash/logstash-core/lib/jars/jruby-complete-9.1.13.0.jar:/home/jls/projects/logstash/logstash-core/lib/jars/log4j-api-2.9.1.jar:/home/jls/projects/logstash/logstash-core/lib/jars/log4j-core-2.9.1.jar:/home/jls/projects/logstash/logstash-core/lib/jars/log4j-slf4j-impl-2.9.1.jar:/home/jls/projects/logstash/logstash-core/lib/jars/logstash-core.jar:/home/jls/projects/logstash/logstash-core/lib/jars/slf4j-api-1.7.25.jar org.logstash.Logstash --log.level debug -e 'input { generator { count => 1 } }'

I'm not sure how to test this as bin/logstash appears to do nothing. Any ideas?

original-brownbear · 2017-12-28T19:48:06Z

@jordansissel hmm don't waste your time apparently there is an issue here now. It was green a few weeks ago when I pinged, but I only rebased today without rerunning tests. Seems Jenkins isn't going anywhere as well. Probably just some trivial packaging issue. Will fix tomorrow morning :)

jordansissel · 2017-12-28T21:11:30Z

@original-brownbear noted! :)

original-brownbear · 2017-12-29T05:55:27Z

@jordansissel with your change to the startup script it works fine for me + on Jenkins (failure is unrelated OOM 05:50:14 +Cannot allocate memory - git ls-files from ).
Should be fine now. Making assemble work out of the box isn't so easy though I'm afraid :( We are currently calling that target from rake and making that work would require cleaning up the gradle and rake interaction some more first (we should do that asap imo, but not here, this one already is pretty long).
For now ./gradlew assembleTarDistribution should work fine though :)

jordansissel · 2018-01-02T18:24:10Z

@original-brownbear noted, will test again. Thanks!

jordansissel · 2018-01-02T18:29:33Z

Manual test:

% rake artifact:tar
...
% bin/logstash -e ''
...

[separate terminal]
% jps
56479 Logstash

It's good to see Logstash here. Woo :)

jordansissel · 2018-01-02T19:12:26Z

Tests passing locally.

jordansissel · 2018-01-02T18:27:10Z

logstash-core/lib/logstash-core/logstash-core.rb

-    require "logstash-core/logstash-core.jar"
-  rescue Exception => e
-    raise("Error loading logstash-core/logstash-core.jar file, cause: #{e.message}")
+unless $LS_JARS_LOADED


This we really only need for having the bin/rspec tool still work.

Can you note this in a code comment?

jordansissel · 2018-01-02T18:27:27Z

logstash-core/logstash-core.gemspec

@@ -53,7 +53,7 @@ Gem::Specification.new do |gem|

  gem.add_runtime_dependency "sinatra", '~> 1.4', '>= 1.4.6'
  gem.add_runtime_dependency 'puma', '~> 2.16'
-  gem.add_runtime_dependency "jruby-openssl", ">= 0.9.20" # >= 0.9.13 Required to support TLSv1.2
+  gem.add_runtime_dependency "jruby-openssl", ">= 0.9.21" # >= 0.9.13 Required to support TLSv1.2


Intended? Seems unrelated to the PR.

@jordansissel ha good question so many weeks later :D I think I fixed the problem that this solved elsewhere already => let me revert this :)

jordansissel · 2018-01-02T18:28:08Z

logstash-core/src/main/java/org/logstash/Logstash.java

+    public static void main(final String... args) {
+        final String lsHome = System.getenv("LS_HOME");
+        if (lsHome == null) {
+            throw new IllegalStateException("LS_HOME environment variable must be set.");


Let's include an action to tell them to file an issue or otherwise note that this is a bug and not a user configuration problem.

jordansissel · 2018-01-02T18:28:41Z

logstash-core/src/main/java/org/logstash/Logstash.java

+        if (lsHome == null) {
+            throw new IllegalStateException("LS_HOME environment variable must be set.");
+        }
+        final Path home = Paths.get(lsHome).toAbsolutePath();


I'm OK with this, no change required here.

jordansissel · 2018-01-03T18:56:46Z

bin/logstash

@@ -1,4 +1,4 @@
-#!/bin/sh
+#!/bin/bash


this should still be /bin/sh

@jordansissel bash is more portable here imo :( Your code only works with sh 4.x => bash 3.x works fine though => standard OSX and old Linux will break if we go with sh => bash wins, especially since JRuby requires bash anyways in the way we currently used to invoke it? :)

Hmm, fair. I didn't realize that bin/jruby invoked bash.

* Created `org.logstash.Logstash` as entrypoint * Safely handle `Ruby` runtime (which sadly is still a singleton, moving away from that will require a few iterations on top of this) * Adjusted `bat` and `sh` entry point wrappers * Verified manually that performance is unchanged (i.e. all Java opts are still loaded properly) * Flattened `.jar` path to make it a little less bothersome to build the `-cp` string * Retained ability to load jars from Ruby via the global `$LS_JARS_LOADED` variable hack, to keep plugin specs that load LS as a `.gem` functional (like e.g. the ITs in LS itself) * No need for the gem jars magic anymore, the downloading and moving into place of jars is now all handled by Gradle

original-brownbear · 2018-01-03T19:07:10Z

@jordansissel all points addressed I think/hope :)

jordansissel · 2018-01-03T19:09:01Z

Yep! I still need to test on windows. Should be done building shortly :)

original-brownbear · 2018-01-03T19:11:04Z

@jordansissel see PR description at least for me Windows worked fine :D *fingers crossed :P

jordansissel · 2018-01-03T19:11:44Z

@original-brownbear 🤦‍♂️ I missed the screenshot. LGTM

original-brownbear · 2018-01-03T19:12:56Z

@jordansissel thanks! :)

elasticsearch-bot · 2018-01-03T19:13:30Z

Armin Braun merged this into the following branches!

Branch	Commits
master	`543b722`
6.x	`dff953d`

* Created `org.logstash.Logstash` as entrypoint * Safely handle `Ruby` runtime (which sadly is still a singleton, moving away from that will require a few iterations on top of this) * Adjusted `bat` and `sh` entry point wrappers * Verified manually that performance is unchanged (i.e. all Java opts are still loaded properly) * Flattened `.jar` path to make it a little less bothersome to build the `-cp` string * Retained ability to load jars from Ruby via the global `$LS_JARS_LOADED` variable hack, to keep plugin specs that load LS as a `.gem` functional (like e.g. the ITs in LS itself) * No need for the gem jars magic anymore, the downloading and moving into place of jars is now all handled by Gradle Fixes #8161

* Created `org.logstash.Logstash` as entrypoint * Safely handle `Ruby` runtime (which sadly is still a singleton, moving away from that will require a few iterations on top of this) * Adjusted `bat` and `sh` entry point wrappers * Verified manually that performance is unchanged (i.e. all Java opts are still loaded properly) * Flattened `.jar` path to make it a little less bothersome to build the `-cp` string * Retained ability to load jars from Ruby via the global `$LS_JARS_LOADED` variable hack, to keep plugin specs that load LS as a `.gem` functional (like e.g. the ITs in LS itself) * No need for the gem jars magic anymore, the downloading and moving into place of jars is now all handled by Gradle Fixes elastic#8161

original-brownbear added the work in progress label Sep 7, 2017

jordansissel suggested changes Sep 7, 2017

View reviewed changes

original-brownbear mentioned this pull request Sep 14, 2017

[discuss] unify launch scripts options support #8261

Closed

original-brownbear mentioned this pull request Oct 31, 2017

#8442 Java Pipeline Compiler #8543

Closed

andrewvc mentioned this pull request Nov 4, 2017

TESTS: Improve Gradle Test Configuration #8583

Closed

original-brownbear force-pushed the cleanup-ruby-load-simpler branch from c7c3195 to aece1fc Compare November 11, 2017 11:47

original-brownbear force-pushed the cleanup-ruby-load-simpler branch from 399eb57 to a10ba72 Compare November 13, 2017 13:15

original-brownbear mentioned this pull request Nov 13, 2017

[Meta] Logstash Javafication #8614

Closed

17 tasks

original-brownbear force-pushed the cleanup-ruby-load-simpler branch from a10ba72 to 4f9ae50 Compare November 14, 2017 11:04

original-brownbear mentioned this pull request Nov 14, 2017

#7135 add Gradle task for bootstrapping and running all integration tests #8599

Closed

original-brownbear force-pushed the cleanup-ruby-load-simpler branch from d7944b5 to 3524c70 Compare November 14, 2017 16:19

original-brownbear force-pushed the cleanup-ruby-load-simpler branch from 56e78c7 to f77c914 Compare November 15, 2017 10:27

original-brownbear added enhancement v6.1.0 v7.0.0-alpha1 and removed work in progress labels Nov 15, 2017

original-brownbear changed the title ~~POC: Java Entrypoint~~ Java Entrypoint Nov 15, 2017

original-brownbear force-pushed the cleanup-ruby-load-simpler branch from f77c914 to cacc790 Compare November 15, 2017 18:02

jordansissel suggested changes Dec 28, 2017

View reviewed changes

original-brownbear force-pushed the cleanup-ruby-load-simpler branch 2 times, most recently from 8d328dd to 6d3dfe3 Compare December 29, 2017 05:28

original-brownbear added the v6.2.0 label Dec 29, 2017

jordansissel reviewed Jan 3, 2018

View reviewed changes

original-brownbear force-pushed the cleanup-ruby-load-simpler branch from 6d3dfe3 to 166c88c Compare January 3, 2018 19:06

jordansissel approved these changes Jan 3, 2018

View reviewed changes

elasticsearch-bot closed this in 543b722 Jan 3, 2018

original-brownbear deleted the cleanup-ruby-load-simpler branch January 3, 2018 19:13

jakelandis mentioned this pull request Mar 19, 2018

Logstash 6.2.1 and jdbc 4.3.3 Java::JavaSql::SQLException: No suitable driver found for jdbc:sybase:Tds:server:port logstash-plugins/logstash-input-jdbc#263

Closed

colinsurprenant mentioned this pull request Jul 10, 2018

fix broken classpath when whitespaces are in the path #9832

Merged

		@@ -1,4 +1,4 @@
		#!/bin/sh
		#!/bin/bash

Java Entrypoint #8161

Java Entrypoint #8161

Conversation

original-brownbear commented Sep 7, 2017 • edited Loading

original-brownbear commented Sep 7, 2017

jordansissel commented Sep 7, 2017

Choose a reason for hiding this comment

andrewvc commented Sep 11, 2017

original-brownbear commented Sep 11, 2017

original-brownbear commented Oct 25, 2017

original-brownbear commented Oct 26, 2017

original-brownbear commented Nov 11, 2017

jordansissel commented Nov 13, 2017

original-brownbear commented Nov 13, 2017

original-brownbear commented Nov 14, 2017

original-brownbear commented Nov 15, 2017

original-brownbear commented Nov 15, 2017 • edited Loading

jordansissel commented Dec 28, 2017

jordansissel left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Alternatives

Option 1: Only use a single .jar

Option 2: Compute classpath at build-time, not runtime.

Option 3: Same method as you, but with bourne shell (not bash)

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jordansissel commented Dec 28, 2017

jordansissel commented Dec 28, 2017

original-brownbear commented Dec 28, 2017

jordansissel commented Dec 28, 2017

original-brownbear commented Dec 29, 2017 • edited Loading

jordansissel commented Jan 2, 2018

jordansissel commented Jan 2, 2018 • edited Loading

jordansissel commented Jan 2, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

original-brownbear commented Jan 3, 2018

jordansissel commented Jan 3, 2018

original-brownbear commented Jan 3, 2018

jordansissel commented Jan 3, 2018

original-brownbear commented Jan 3, 2018

elasticsearch-bot commented Jan 3, 2018

original-brownbear commented Sep 7, 2017 •

edited

Loading

original-brownbear commented Nov 15, 2017 •

edited

Loading

jordansissel left a comment •

edited

Loading

original-brownbear commented Dec 29, 2017 •

edited

Loading

jordansissel commented Jan 2, 2018 •

edited

Loading