[docker-compose] Add Filebeat container to ship logs from Mordred to ES

[canasdiaz]

Filebeat container was added to the compose file to ship the logs written by Mordred to the Elasticsearch cluster. A pretty simple configuration file was added to the "default-grimoirelab-settings" folder. As a result Filebeat creates daily indexes in the cluster with the content of the Mordred log file.

The README file was also modified to help the user know how to add the index pattern needed to visualize the logs.

Fixes #276

Signed-off-by: Luis Cañas-Díaz [email protected]

[valeriocos]

Thank you @sanacl for the PR, I'll have a look at it today

[valeriocos]

@sanacl the idea of having the logs in the instance is really nice! The risk I see is that sensible information (credentials, local paths, db names, etc.) might be exposed in the logs.

Logstash seems to provide a mechanism to filter part of the text in the log messages (https://discuss.elastic.co/t/remove-part-of-message-string/66809). I wasn't able to find the same feature for filebeat. Is there a specific reason to use filebeat? Would like to investigate how to hide parts of the text in the log messages?

[canasdiaz]

Logstash seems to provide a mechanism to filter part of the text in the log messages (https://discuss.elastic.co/t/remove-part-of-message-string/66809). I wasn't able to find the same feature for filebeat. Is there a specific reason to use filebeat? Would like to investigate how to hide parts of the text in the log messages?

@valeriocos

Logstash was split years ago into different components. Filebeat is just a component that reads a file a puts its content somewhere else (ES cluster or Logstash). I could have a look to see whether Logstash is still able of reading a file by itself, so we won't need a second container for this dropping. The Logstash filter is able to detect a field a drop the item, but as far as I know it does not replace the content.

In any case, do u think hiding these parameters is important for our community members?

[valeriocos]

In any case, do u think hiding these parameters is important for our community members?

I would say so. Community members may deploy their own instance, thus providing the means to hide these params (or understand if this is possible) can be useful for the community.

[canasdiaz]

I've been having a look at this and in order to modify all the places where strings like "https://user:pass" appear, we would need to deploy filebeat + logstash, which from my point of view is overkill for our average community members.

In some cases this is already done by Mordred, but it is not done for all of them.

I see two options:

1. add a new container with logstash and add there a "mutation" to overwrite strings like "https://user:pass"
2. include a warning in the documention before adding the "mordred" index pattern, making clear some credentials could be exposed (as it happens with the file logs BTW)

What do u think @valeriocos ?

[sduenas]

Closing this due to inactivity .