Autodiscover #5245

exekias · 2017-09-26T10:56:38Z

This PR adds support for Autodiscover, and implements a first provider for Docker.

Autodiscover

Autodiscover allows the user to define different providers to watch for system changes and instantiate new modules as these happen.

Providers watch for changes and emit events to a common bus, then the autodiscovery process detect situations when there is something we can monitor and instantiate a modules for it.

Bus messages contain useful information about what providers see, with some common well-known optional fields (provider specific fields may also be present):

host - service host
port - service port
config - recommended config for this instance (from the user or environment)

Docker provider

Docker provider watches for docker container events, it supports config mapping from container metadata -> config templates, so new modules are created when a container starts. The available data for a container (that you can use in the config template) is:

host
port
docker.container.id
docker.container.name
docker.container.image
docker.container.labels

Testing

This should be enough to monitor any present and future redis instances that run on docker:

metricbeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            equals.docker.container.image: redis
          config:
            - module: redis
              metricsets: ["info", "keyspace"]
              hosts: "${data.host}:${data.port}"

Or reading redis logs only:

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            equals.docker.container.image: redis
          config:
            - type: docker
              conatiners.ids:
                -${data.docker.container.id}

You can launch it like this:

docker run -p 6379:6379 redis

This PR is in progress, with some missing parts:

Revisit Docker IP/port mapping, do we really need exposed ports? probably not
Add support in filebeat
Revisit bus implementation, use interfaces (instead of MapStr)?
Add docker metadata to modules discovered by docker provider
System tests
Update metricbeat and filebeat YAML files
Docs

kvch · 2017-09-26T13:26:35Z

libbeat/autodiscover/provider.go

+
+// AddProvider registers a new ProviderBuilder
+func (r *registry) AddProvider(name string, provider ProviderBuilder) error {
+	r.lock.Lock()


Why do you lock here? Are init functions called concurrently?

Nope, or not at the moment 😇, but some defensive code there won't hurt I think, this may be a thing in the future?

kvch · 2017-09-26T13:29:10Z

libbeat/common/bus/bus.go

+// Bus provides a common channel to emit and listen for Events
+type Bus interface {
+	// Emit an event to the bus
+	Emit(Event)


I think Publish would be a more expressive name than Emit, as it is a kind of Publish-Subscribe model.

ruflin

Great start for this feature.

ruflin · 2017-09-28T08:11:58Z

libbeat/autodiscover/common/config.go

@@ -0,0 +1,63 @@
+package common


I worry a bit that this will conflict with our libbeat common package in some places.

ruflin · 2017-09-28T08:14:14Z

libbeat/autodiscover/providers/docker/config.go

+
+import (
+	"github.com/elastic/beats/libbeat/autodiscover/common"
+	"github.com/elastic/beats/libbeat/processors/add_docker_metadata"


Feels strange to include here a processor. Perhaps we should extract the TLS config part to a docker package.

ruflin · 2017-09-28T08:14:28Z

libbeat/autodiscover/providers/docker/docker.go

+	"github.com/elastic/beats/libbeat/common/bus"
+	"github.com/elastic/beats/libbeat/logp"
+
+	// TODO move add_docker_metadata.Watcher to a common place


ruflin · 2017-09-28T08:17:34Z

libbeat/processors/add_docker_metadata/watcher.go

@@ -32,6 +33,12 @@ type Watcher interface {

 	// Containers returns the list of known containers
 	Containers() map[string]*Container
+
+	// ListenStart returns a bus listener to receive container started events, with a `container` key holding it
+	ListenStart() bus.Listener


I wonder if we should put these in the same interface or one more specific to autodiscovery which is composeo f this Watcher interface.

ruflin · 2017-09-28T08:41:04Z

@exekias I stumbled over the config a bit because of all the indentation it has. Here an alternative suggestion:

metricbeat.autodiscover:
  providers:
    - type: docker
      filter:
        docker-image: redis
        port: 6309
      config:
        - module: redis
          metricsets: ["info"]
          hosts: "${data.host}:${data.port}"
        - module: redis
          metricsets: ["info", "keyspace"]
          hosts: "${data.host}:${data.port}"
          period: 60s
    - type: docker
      conditions:
        - when.equals.docker-image: mysql
        - when.range.port.gte: 6000
      config:
        - module: mysql
          metricsets: ["stats"]
          hosts: "${data.host}:${data.port}"

The basic idea is to make it more obvious out of what the config will be generated and allow multiple and criterias / filters in a simple way. But at the same time provide an option to use our more powerful conditions coming from the processors for matching.

I don't think we should provide the conditions in the first implementation but wanted to add it here to take into account where this could go. I'm not happy with filter naming in the first config part. Perhaps criteria or match could be an alternative?

I'm pretty sure the above config brings up other issues as I only focused on the config and not potential implications on the implementation itself.

urso · 2017-10-05T12:51:54Z

libbeat/autodiscover/common/config.go

+					ucfg.VarExp,
+				}
+				for _, config := range configs {
+					c, err := ucfg.NewFrom(config, opts...)


Why do you need ucfg here?

Minor problem with this approach is, you are loosing the full setting paths. If some config unpacking fails, the setting it's full path is reported with the error message.

I pass provider event data for variable expansion, so config templates can use data from the event. Agree on the minor issue (would affect logs mainly?)

Yes, it's for logs only. As we can't store file positions (by line) yet, not having the correct names can make it more difficult to track-down misconfigurations.

discovery can be more automatic than providing docker-image: redis. if we use docker labels, we should be able to pass annotation like information same like collectbeat to be able to discover workloads automatically. i could run 10 containers in a host. it would be a huge config if i have to say for each via config how it has to be discovered.

I agree @vjsamuel, this is a first PR, intended to bring the basic functionality and building blocks to grow from there. Definitely that's something we want to have

vjsamuel · 2017-10-12T17:54:09Z

@exekias on general thought. with autodiscover we should avoid using add_docker_metadata and add_kubernetes_metadata, we should provide all metadata upfront as part of the module configuration. This will improve performance tremendously as we would not need to do append of metadata at each event.

we should also have some plugability in place for folks who want to run something custom on top of auto discovery. this is similar to collectbeat appenders where we have our own internal appenders more custom tailored for our needs.

fin09pcap · 2017-10-22T07:07:39Z

subscribe

exekias · 2017-10-26T10:11:08Z

@ruflin I updated config format to take advantage of processor conditions, let me know what you think about it:

metricbeat.autodiscover:
  providers:
    - type: docker
      templates:
        - when.equals.docker.image: redis
          use:
            - module: redis
              metricsets: ["info", "keyspace"]
              hosts: "${data.host}:${data.port}"

christiangalsterer · 2017-11-04T07:20:43Z

This is a really great feature as this really great for mid to large scale Docker/Kubernetes installations to auto-configure Beats when container come and go.
Do you have by chance any idea in which version and when this will be available? A first indication would already help if this is something we can expect in the next month, next quarter, etc.

exekias · 2017-11-06T10:46:29Z

Hi @christiangalsterer, I'm glad you like it! this is something we want to have in the 6.X timeline, probably in a few months

ruflin · 2017-11-06T21:02:43Z

@exekias How would your example look if multiple conditions are used? I'm still stumbling over the template part but lets separate these discussions :-)

ruflin · 2017-11-06T23:35:28Z

We had a discussion with @exekias and @urso about the config and that is what we came up with. I added some comments to explain why the decision was made this way:

metricbeat.autodiscover:
  # Each Provider can be defined multiple times.
  # Example: Assuming we have a port scanner provider, different scan intervals are applied for different port ranges
  providers:
    - type: docker
    
      # Each provider can have it's specific config options
      host: docker/socket/path
      
      # Multiple templates can be used per provider with different condtions
      templates:
      
          # One condition is used per template. Sub conditions can be added with and to nest conditions
        - condition:
            and:
              - equals.docker.container.image: redis
        
          # The configuration fields that will be created
          config:
            - module: redis
              metricsets: ["info"]
              hosts: "${data.host}:${data.port}"
              frequency: 1m
            - module: redis
              metricsets: ["keyspace"]
              hosts: "${data.host}:${data.port}"
              frequency: 10s
              
          # Same config as above just starting with config instead of condition
        - config:
            - module: redis
              metricsets: ["info", "keyspace"]
              hosts: "${data.host}:${data.port}"
          condition:
            when.equals.docker.container.image: redis

We couldn't decide if it should be called config or configs but sticked with config for now.

ruflin

Change LGTM in general. I would appreciate if we would have some automate tests to see if the discovery works as expected with docker but that could become tricky.

I left some minor comments and some questions mainly for my understanding.

ruflin · 2017-11-07T00:07:15Z

filebeat/crawler/crawler.go

@@ -21,6 +21,8 @@ type Crawler struct {
 	prospectorConfigs   []*common.Config
 	out                 channel.Factory
 	wg                  sync.WaitGroup
+	ProspectorsFactory  cfgfile.RunnerFactory


Note for myself: Need to understand why we need to store the factory now in the Crawler struct.

Short answer: we use this factory from Autodiscovery. In the long term, we should unify all prospector/module initialization in a common place

ruflin · 2017-11-07T00:08:14Z

filebeat/fileset/factory.go

@@ -110,6 +110,7 @@ func (p *prospectorsRunner) Stop() {
 		prospector.Stop()
 	}
 }
-func (p *prospectorsRunner) ID() uint64 {


Was the ID still used somewhere? If not, perhaps remove it in a different PR?

It wasn't used, it's a spurious implementation, probably forgot from a previous removal

ruflin · 2017-11-07T00:08:51Z

filebeat/prospector/prospector.go

@@ -141,3 +144,7 @@ func (p *Prospector) stop() {
 		p.prospectorer.Stop()
 	}
 }
+
+func (p *Prospector) String() string {


Is that a required method or just for convenience / logging?

It's for logging purposes only, autodiscover will tell about new spawned prospectors/modules in the log output

ruflin · 2017-11-07T00:10:04Z

libbeat/autodiscover/autodiscover.go

+	// RunnerFactory provides runner creation by feeding valid configs
+	cfgfile.RunnerFactory
+
+	// StartFilter returns the bus filter to retrieve runner start triggering events


So far it seems these are all the same for the current implementation. How will these look for other adapters?

I don't know yet 😇, I think this code can be used in other parts of beats, not only to detect new modules to launch, so wanted to keep this open

Do we even need the filters methods on the interface? Can the filters be passed when registering the adapter, so to keep the interface somewhat 'smaller'?

The adapter is not registered but passed to Autodiscover Manager. If you look at is implementation, it's very small generally, I think it makes sense to keep these filters together with the functions that use it. If in the future we use something different from config we need to update the filters too https://github.com/exekias/beats/blob/411d13a878d64dbe62b6008057bbaa1c207f9e4e/metricbeat/beater/autodiscover.go

ruflin · 2017-11-07T00:15:11Z

libbeat/autodiscover/autodiscover.go

+}
+
+// Start autodiscovery process
+func (a *Autodiscover) Start() {


Some idea that popped up in my head when reading this: Will it be needed to enabled / disable autodiscovery dynamically? For example on startup you want the beat to autodiscover all the services that are out there and then stop the discovery for example? Or only use autodiscovery for some time enabled through centralised config? We definitively not need that now, just thinking out loud.

this is a good point, something like modules reload could make sense for autodiscover providers

ruflin · 2017-11-07T00:49:18Z

libbeat/common/bus/bus.go

+	Subscribe(filter ...string) Listener
+}
+
+// Listener retrieves Events from a Bus subscription until Close is called


until Stop is called?

ruflin · 2017-11-07T00:49:54Z

libbeat/common/bus/bus.go

+}
+
+// New initializes a new bus with the given name and returns it
+func New(name string) Bus {


What are the names of the buses you have in mind?

I don't know, this name is used just for logging

ruflin · 2017-11-07T00:53:48Z

libbeat/common/docker/watcher_test.go

 	// event list to send on Events call
 	events []interface{}

 	done chan interface{}
 }

 func (m *MockClient) ContainerList(ctx context.Context, options types.ContainerListOptions) ([]types.Container, error) {
-	return m.containers, nil
+	res := m.containers[0]


Can you elaborate on this change?

I added support for several requests to this mock, each request returns the next response and pops it out the list

ruflin · 2017-11-07T00:56:48Z

libbeat/common/mapstr_pointer.go

+	"unsafe"
+)
+
+// MapStrPointer stores a pointer to atomically get/set a MapStr object


I would have prefered to have "optimizations" like this in a follow up PR to have the code we put in as simple as possible to start with and then when we do optimizations we also add some benchmark tests for it. Makes it easier to later detect in case these optimizations causes some issues, if it happened with the optimization or the original code.

Agreed, but as I try to get a stable API to introduce new providers as soon as possible this somehow made sense while I waited for reviews. Once this is merged, autodiscover API should have everything we need

ruflin · 2017-11-07T00:58:02Z

libbeat/processors/condition.go

@@ -42,6 +42,10 @@ type WhenProcessor struct {
 	p         Processor
 }

+type ValuesMap interface {


Note to myself: Figure out why this change is needed and if it could happen in a separate PR.

urso · 2017-11-08T12:29:10Z

libbeat/beat/pipeline.go

@@ -34,6 +34,9 @@ type ClientConfig struct {
 	// Fields provides additional 'global' fields to be added to every event
 	Fields common.MapStr

+	// DynamicMeta provides additional fields to be added to every event, supporting live updates
+	DynamicMeta *common.MapStrPointer


To reduce confusion, rather use DynamicFields. The beat event has Meta and Fields. The Meta are somewhat internal (like LS @metadata). But DynamicMeta is updating the event Fields, not Meta.

Btw. instead of adding the functionality to the publisher pipeline, you can just pass a processor doing the magic.

Instead of passing DynamicMeta, we could add atomic values support to the normalizer of common.MapStr. This would allow you to pass updatable fields via Fields (given you don't need them on top-level).

Thanks for the input, I agree DynamicFields is the right name.

I thought about the processor but that would cause Copy to be called for all cases, I guess that's not too bad?

urso · 2017-11-08T12:39:49Z

libbeat/common/mapstr_pointer.go

+// replacing it.
+type MapStrPointer struct {
+	p unsafe.Pointer
+}


How about changing the type to:

type VolatileMapStr struct { p *unsafe.Pointer } func MakeVolatileMapStr(m MapStr) VolatileMapStr { return VolatileMapStr{ p: &unsafe.Pointer(&m) } } func (m VolatileMapStr) Get() MapStr { return *((*MapStr)(atomic.LoadPointer(m.p))) } func (m VolatileMapStr) Set(new MapStr) { atomic.StorePointer(m.p, unsafe.Pointer(&new)) }

Then you can use VolatileMapStr and MapStr by value, just like MapStr is already passed by value.

LGTM, but don't really like the Volatile name, somehow it hides the real thing?

houndci-bot · 2017-11-08T16:41:24Z

metricbeat/mb/module/factory.go

@@ -27,14 +27,14 @@ func NewFactory(maxStartDelay time.Duration, p beat.Pipeline) *Factory {
 	}
 }

-func (r *Factory) Create(c *common.Config) (cfgfile.Runner, error) {
+func (r *Factory) Create(c *common.Config, meta *common.MapStrPointer) (cfgfile.Runner, error) {


exported method Factory.Create should have comment or be unexported

houndci-bot · 2017-11-08T16:41:24Z

metricbeat/mb/module/connector.go

 }

 type connectorConfig struct {
 	Processors           processors.PluginConfig `config:"processors"`
 	common.EventMetadata `config:",inline"`      // Fields and tags to add to events.
 }

-func NewConnector(pipeline beat.Pipeline, c *common.Config) (*Connector, error) {
+func NewConnector(pipeline beat.Pipeline, c *common.Config, meta *common.MapStrPointer) (*Connector, error) {


exported function NewConnector should have comment or be unexported

houndci-bot · 2017-11-08T16:41:24Z

metricbeat/beater/autodiscover.go

+	return []string{"start", "config"}
+}
+
+func (m *AutodiscoverAdapter) StopFilter() []string {


exported method AutodiscoverAdapter.StopFilter should have comment or be unexported

houndci-bot · 2017-11-08T16:41:25Z

metricbeat/beater/autodiscover.go

+	return m.factory.Create(c, meta)
+}
+
+func (m *AutodiscoverAdapter) StartFilter() []string {


exported method AutodiscoverAdapter.StartFilter should have comment or be unexported

houndci-bot · 2017-11-08T16:41:25Z

metricbeat/beater/autodiscover.go

+	return nil
+}
+
+func (m *AutodiscoverAdapter) Create(c *common.Config, meta *common.MapStrPointer) (cfgfile.Runner, error) {


exported method AutodiscoverAdapter.Create should have comment or be unexported

houndci-bot · 2017-11-08T16:41:25Z

libbeat/common/mapstr_pointer.go

+	return (*MapStr)(atomic.LoadPointer(&m.p))
+}
+
+func (m *MapStrPointer) Set(p *MapStr) {


exported method MapStrPointer.Set should have comment or be unexported

houndci-bot · 2017-11-08T16:41:25Z

libbeat/common/mapstr_pointer.go

+	}
+}
+
+func (m *MapStrPointer) Get() *MapStr {


exported method MapStrPointer.Get should have comment or be unexported

houndci-bot · 2017-11-08T16:41:25Z

libbeat/common/mapstr_pointer.go

+	p unsafe.Pointer
+}
+
+func NewMapStrPointer(p *MapStr) *MapStrPointer {


exported function NewMapStrPointer should have comment or be unexported

houndci-bot · 2017-11-08T16:41:26Z

filebeat/beater/autodiscover.go

+	return []string{"start", "config"}
+}
+
+func (m *AutodiscoverAdapter) StopFilter() []string {


exported method AutodiscoverAdapter.StopFilter should have comment or be unexported

houndci-bot · 2017-11-08T16:41:26Z

filebeat/beater/autodiscover.go

+	return m.prospectorFactory.Create(c, meta)
+}
+
+func (m *AutodiscoverAdapter) StartFilter() []string {


exported method AutodiscoverAdapter.StartFilter should have comment or be unexported

houndci-bot · 2017-11-08T17:28:28Z

libbeat/cfgfile/registry.go


 	_, ok := r.List[hash]
 	return ok
 }

+func (r *Registry) Get(hash uint64) Runner {


exported method Registry.Get should have comment or be unexported

houndci-bot · 2017-11-08T17:28:28Z

libbeat/autodiscover/template/config.go

+// Event adapts MapStr to processors.ValuesMap interface
+type Event common.MapStr
+
+func (e Event) GetValue(key string) (interface{}, error) {


exported method Event.GetValue should have comment or be unexported

houndci-bot · 2017-11-12T18:16:59Z

libbeat/autodiscover/providers/docker/docker.go

+	autodiscover.ProviderRegistry.AddProvider("docker", AutodiscoverBuilder)
+}
+
+// Autodiscover implements autodiscover provider for docker containers


comment on exported type Provider should be of the form "Provider ..." (with optional leading article)

tsg · 2017-11-13T10:19:59Z

metricbeat/metricbeat.yml

+# Autodiscover allows you to detect changes in the system and spawn new modules
+# as they happen.
+
+metricbeat.autodiscover:


I think Autodiscovery doesn't need to be in the short config (which we try to keep minimal), but only in the reference one. Especially since Autodiscovery is going to be in the K8s deployment manifersts, right?

👍 that works for me, will remove it from short configs

exekias · 2017-11-14T15:09:00Z

This PR is ready for a (final?) review, I'm currently opening side PRs to shrink this one

ruflin · 2017-11-14T22:47:01Z

@exekias just merged the other two. Could you rebase on top? If it reduces the hassle of rebasing, feel free to squash it into 1 commit already (or as many as you want in the end).

tsg · 2017-11-16T14:43:35Z

@exekias Merged, but please keep in mind to add docs before the 6.1 GA.

ruflin · 2017-11-16T23:09:38Z

@exekias Awesome stuff. Thanks for making this happen.

dedemorton · 2018-01-22T19:02:00Z

Autodiscover docs were added with #5868, so I'm removing the needs_docs label.

exekias added in progress Pull request is currently in progress. libbeat Metricbeat Metricbeat review labels Sep 26, 2017

exekias force-pushed the autodiscover branch 2 times, most recently from 7344322 to b7674c4 Compare September 26, 2017 13:06

kvch reviewed Sep 26, 2017

View reviewed changes

exekias force-pushed the autodiscover branch from c946667 to 8eb747a Compare September 26, 2017 15:15

ruflin reviewed Sep 28, 2017

View reviewed changes

urso reviewed Oct 5, 2017

View reviewed changes

exekias mentioned this pull request Oct 16, 2017

filebeat add_kubernetes_metadata kicks in too late #5377

Closed

vjsamuel mentioned this pull request Oct 23, 2017

Docker Support eBay/collectbeat#3

Closed

exekias force-pushed the autodiscover branch from 326c3db to c139c6c Compare October 24, 2017 20:58

exekias force-pushed the autodiscover branch from 97749fc to 0b26fcc Compare October 26, 2017 11:15

exekias mentioned this pull request Oct 27, 2017

Refactor add_kubernetes_metadata to support autodiscovery #5434

Merged

exekias added the Filebeat Filebeat label Oct 27, 2017

ruflin reviewed Nov 7, 2017

View reviewed changes

urso reviewed Nov 8, 2017

View reviewed changes

houndci-bot reviewed Nov 8, 2017

View reviewed changes

houndci-bot reviewed Nov 12, 2017

View reviewed changes

exekias force-pushed the autodiscover branch 6 times, most recently from f10f5cb to 34c9aa0 Compare November 12, 2017 22:41

tsg reviewed Nov 13, 2017

View reviewed changes

exekias force-pushed the autodiscover branch 2 times, most recently from 8f63a35 to 21a0563 Compare November 13, 2017 18:25

exekias removed the in progress Pull request is currently in progress. label Nov 13, 2017

exekias mentioned this pull request Nov 14, 2017

Add Get method to cfgfile.Registry #5581

Merged

exekias force-pushed the autodiscover branch from a147ee6 to 9f3b4d8 Compare November 14, 2017 23:44

exekias added the needs_docs label Nov 15, 2017

exekias force-pushed the autodiscover branch from 9f3b4d8 to 26ce229 Compare November 15, 2017 12:55

exekias added 3 commits November 16, 2017 09:53

Add autodiscover framework and initial docker provider

4c4c223

Enable autodiscover for Filebeat

6b9761f

Enable autodiscover for Metricbeat

2f59c0b

exekias force-pushed the autodiscover branch from 26ce229 to 2f59c0b Compare November 16, 2017 08:54

tsg merged commit 4bbecab into elastic:master Nov 16, 2017

ruflin mentioned this pull request Dec 8, 2017

Enhancement Request: Multiple Multiline Statements per file #4626

Closed

This was referenced Dec 12, 2017

Getting Logs from Docker Containers #918

Closed

Add docs for autodiscover #5868

Merged

Cherry-pick #5868 to 6.1: Add docs for autodiscover #5879

Merged

dedemorton removed the needs_docs label Jan 22, 2018

Autodiscover #5245

Autodiscover #5245

Conversation

exekias commented Sep 26, 2017 • edited by dedemorton Loading

Autodiscover

Docker provider

Testing

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin commented Sep 28, 2017

Choose a reason for hiding this comment

exekias Oct 6, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vjsamuel commented Oct 12, 2017

fin09pcap commented Oct 22, 2017

exekias commented Oct 26, 2017 • edited Loading

christiangalsterer commented Nov 4, 2017

exekias commented Nov 6, 2017

ruflin commented Nov 6, 2017

ruflin commented Nov 6, 2017

ruflin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

exekias Nov 8, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

exekias Nov 13, 2017 • edited Loading

Choose a reason for hiding this comment

exekias commented Nov 14, 2017 • edited Loading

ruflin commented Nov 14, 2017

tsg commented Nov 16, 2017

ruflin commented Nov 16, 2017

dedemorton commented Jan 22, 2018

exekias commented Sep 26, 2017 •

edited by dedemorton

Loading

exekias Oct 6, 2017 •

edited

Loading

exekias commented Oct 26, 2017 •

edited

Loading

exekias Nov 8, 2017 •

edited

Loading

exekias Nov 13, 2017 •

edited

Loading

exekias commented Nov 14, 2017 •

edited

Loading