Document Sematext Logsene. Add rsyslog.conf overwrite feature.

[fernviridian]

Tozny is interested in using this container for remote logging of files on it's servers. We use Sematext Logsene, so this container was very interesting to us as this feature was already built. We dug through the configuration, and got it working with our Logsene app.

This PR includes documentation on the environment variables needed for making Logsene work. Also included, is the addition of the RSYSLOG_CONFIG_BASE64 environment variable. This allows injection of a rsyslog.conf file into the container, without mounting a volume or file into the container. If the environment variable is not set, the starter.sh script does not overwrite the /etc/rsyslog.conf file and the script continues.

[rgerhards]

Thx for the PR, much appreciated.

Question on the RSYSLOG_CONFIG_BASE64 var: do you consider it essential? I ask because one design goal is to have the capability to run the container read-only, what obviously is not possible if that feature is used. Of course, we can document this, but I wonder if it is worth the deviation from intended use case.

[rgerhards]

@tgjamin I have another question, and it is primarily for me to understand the usability of this container (remember, we are in the early stages).

The idea behind the container setup is to have an easy to use config file that shall be modified. This here https://github.com/rsyslog/rsyslog-docker/blob/master/appliance/alpine/internal/container_config. The container provides a command to modify the config. The meta config happens to use environment variables. Thus you can also overwrite the config by setting and importing them, I guess much like you do.

Questions for me:

1. why is this method superior or at least a viable alternative? I guess it relates to scripted deployments?
2. wouldn't it be better to keep the doc for those meta config environment variables in a single place, namely the meta config? If so, I guess it would at least be required to link from the main readme to it.

Again, I am trying to create the best possible usability for this container and so this are actually honest questions. Anyone else is also invited to provide feedback, of course. @deoren @radu-gheorghe @otisg @portante would be great if you could find time to comment

[radu-gheorghe]

Let me see if I get the options right: one is to provide environment variables for essential things (such as Logsene token - or Elasticsearch index name, same thing), and the other one is to provide the whole config as an environment variable?

If I got it right, are the two options mutually exclusive? Or is it just the concern that if the whole config is provided then the original rsyslog becomes irrelevant, other environment variables are overridden (or not?) and in general things would be more complex/confusing?

[rgerhards]

Let me see if I get the options right: one is to provide environment variables for essential things (such as Logsene token - or Elasticsearch index name, same thing), and the other one is to provide the whole config as an environment variable?

The design of the rsyslog appliance container is as follows: we want an easy config layer for "standard things" like logging to logsense. So that users do not need to know all the details about queues, rulesets etc to get things going. We call this "meta config". It is actually a single file. You can see this file via

docker run ... tools/show-config

and edit it via

docker run ... tools/edit-config

Or alternatively by editing the file inside the volume.

More details on the meta config are here (slide 15 is maybe a good start): https://www.slideshare.net/rainergerhards1/rsyslog-meets-docker

In the meta config, we use environment variables. These were originally not meant to be overwritten by something else. This PR now introduces/recommend to set the environment variables outside of the container config. This works, but is a bit dangerous, as the meta config sets some default values, just not for logsene. That is because the meta config right now assumes it has full control over the environment variables - what was the actual design. Of course, we can change that easily if so desired.

So the first question is if we need to support passing environment variables directly into the container rather than using the meta config layer. If so, that requires some internal changes to be reliable. I would also like to understand why this may be beneficial.

If I got it right, are the two options mutually exclusive? Or is it just the concern that if the whole config is provided then the original rsyslog becomes irrelevant, other environment variables are overridden (or not?) and in general things would be more complex/confusing?

They are mutually exclusive, as the passed-in config overwrites the container one and so disables the meta config. Presentation slide 19 (best 15..19) shows the relationship between meta config and rsyslog.conf.

But my second question is a bit different: if we permit to rewrite rsyslog.conf via a container variable, we cannot execute the container in read-only mode. One of the design goals was to make this possible, because

• I read everywhere this is a recommended best practice and
• I have seen that some folks (accidently?) store log data into the upper container fs layer, instead of a bind mount or volume. By making this layer read-only, we prevent this mistake

So I ask if it is worth giving up the ability, at least when using this feature.

Side-note: we ourselves demand the the container is non-read only, but if and only if it is put into debug mode. Here, we need it writable because we need to install debug symbols and other helpers which are not present by default, because really not needed other than for debug.

I hope that explains my thoughts and concerns. Again, I do not object the proposed changes, but would like to understand the reasons in order to craft the best possible container. Any questions please let me know.

[radu-gheorghe]

Thanks for explaining this so clearly, Rainer! I am familiar with the presentation, but now I understand things better :)

It sounds like the meta config layer is a nicer way to do things. Whether we should allow overriding it or not - I'm really not sure. I would guess it's down to what is easier to use, in case someone needs to change the provided rsyslog.conf beyond what the container provides. Environment variables (as provided in this PR) are one solution, I suppose changing rsyslog.conf from the container definition is another (though it won't be as easy to run the container from Docker Hub). I don't have a clear preference here, though it sounds like allowing to override the whole config is a slightly better option.

I still don't understand why providing a different file excludes the meta config layer. Can't the provided config file say

index=`echo $LOGSENE_APP_TOKEN`

and, if the LOGSENE_APP_TOKEN is defined, to replace it?

I have a feeling I'm missing some important stuff here still, so maybe others can provide more valuable feedback.

[rgerhards]

I am on the run, so for today just this:

I still don't understand why providing a different file excludes the meta config layer. Can't the provided config file say

index=echo $LOGSENE_APP_TOKEN
and, if the LOGSENE_APP_TOKEN is defined, to replace it?

Assuming that someone copies all of the provided rsyslog.conf support for the meta config, it would continue to work. My assumption was that it is really replaced by a custom config. IMHO rsyslog.conf get's far less readable when the meta config is applied, so I would have expected that if someone really wants to considerably depart from the build-in config use cases, the full config is replaced. Probably a wrong assumption on my side?

[radu-gheorghe]

I think I got it now. Then it sounds like the two aren't mutually exclusive. We would want people to use the meta config layer (it's easier, as you said), but if they need more power, this PR seems to give them that.

Unless I'm missing something, it sounds like a good idea. We do a similar thing with Sematext Docker Agent here: https://sematext.com/docs/sematext-docker-agent/configuration/ (look for LOGAGENT_PATTERNS and LOGAGENT_PATTERNS_BASE64). @megastef did that part, and because of that I assume it's a good Docker practice :)

[fernviridian]

I am new to rsyslog configuration, so still getting a feel for the project and configuration methods. I appreciate the feedback on the PR and will do my best to address the questions.

As @radu-gheorghe pointed out, the sematext-docker-agent has an environment variable LOGAGENT_PATTERNS_BASE64 that allows the overwrite of logagent pattern yaml file. This was an addition I wrote for our needs at Tozny, that was merged into that project with the help of @megastef. We do large deployments of the sematext agent through container orchestration tools, and wanted a way to easily overwrite the logagent pattern file, without pre-populating a file on every host, and then mounting it into the sematext-docker-agent container, as this does not scale well. Instead, we can define all of the necessary configuration for the sematext-docker-agent with environment variables in the orchestration layer yaml files, and scale it as much as needed without manual configuration management. This was a huge need for us when we started using dynamically allocated servers with AWS Autoscaling groups. We no longer could use typical manual configuration, or even configuration management as each server has no guaranteed lifetime, and needs to be provisioned 100% automatically.

@radu-gheorghe pointed out:

I think I got it now. Then it sounds like the two aren't mutually exclusive. We would want people to use the meta config layer (it's easier, as you said), but if they need more power, this PR seems to give them that.

This is correct. A user of the container would not be required to inject environment variables but could choose to (if they wanted). The default tools/edit-config meta would continue to work as intended. This would be an extension for users with specific needs, like us.

On slide 30 of https://www.slideshare.net/rainergerhards1/rsyslog-meets-docker I noticed the future work of "imfile rewrite, permit to monitor "external" files (on host machine)". We accomplished this with the current container by mounting logs into the container read-only during docker run ... -v "/var/log/auth.log:/var/log/auth.log:ro" commands. We overwrite the rsyslog.conf` file, with the same defaults as the one in the container, only to add the following extra:

• module(load="imfile") -- to load the imfile module
• add input(type="imfile" tag="testtag" file="/var/log/*.log") -- to configure the mounted file as input.

Question on the RSYSLOG_CONFIG_BASE64 var: do you consider it essential? I ask because one design goal is to have the capability to run the container read-only, what obviously is not possible if that feature is used. Of course, we can document this, but I wonder if it is worth the deviation from intended use case.

We would need a way to overwrite the default rsyslog.conf file in an automated way, without manually running the tools/edit-config helper wrappers for meta config. The easiest way to accomplish this (in our experience of running docker in production for 2 years) is to overwrite configuration with environment variables. This is automation friendly, particularly for container orchestrators. While there may be a better way to overwrite configs, this is generally the simplest and least messy way in our experience that supports as many uses as possible (manual docker run commands, docker-compose, Rancher, Kubernetes, $orchestration-layer, etc).

But my second question is a bit different: if we permit to rewrite rsyslog.conf via a container variable, we cannot execute the container in read-only mode.

Perhaps the RSYSLOG_CONFIG_BASE64 environment variable should instead write the config to the /config/ directory, to utilize the docker volume there. This would have the same effect, and overwrite the default /etc/rsyslog.conf file. This would maintain the read-only property of the container's root filesystem. How does that sound?

Here is an example of running with logsene enabled and configured to log /var/log/auth.log mounted from the host system to Sematext Logsene:

docker run -it --rm \
  -e LOGSENE_TOKEN=<write only logsene token> \
  -e LOGSENE_URL=logsene-receiver.sematext.com \
  -e RSYSLOG_CONFIG_BASE64="Z2xvYmFsKHByb2Nlc3NJbnRlcm5hbE1lc3NhZ2VzPSJvbiIpCgojbW9kdWxlKGxvYWQ9ImltdGNwIiBTdHJlYW1Ecml2ZXIuQXV0aE1vZGU9ImFub24iIFN0cmVhbURyaXZlci5Nb2RlPSIxIikKbW9kdWxlKGxvYWQ9ImltcHN0YXRzIikgIyBjb25maWcuZW5hYmxlZD1gZWNobyAkRU5BQkxFX1NUQVRJU1RJQ1NgKQptb2R1bGUobG9hZD0iaW1yZWxwIikKbW9kdWxlKGxvYWQ9ImltcHRjcCIpCm1vZHVsZShsb2FkPSJpbXVkcCIgVGltZVJlcXVlcnk9IjUwMCIpCm1vZHVsZShsb2FkPSJpbWZpbGUiKQoKbW9kdWxlKGxvYWQ9Im9tc3Rkb3V0IikKbW9kdWxlKGxvYWQ9Im9tZWxhc3RpY3NlYXJjaCIpCgptb2R1bGUobG9hZD0ibW1qc29ucGFyc2UiKQptb2R1bGUobG9hZD0ibW11dGY4Zml4IikKCgppbnB1dCh0eXBlPSJpbXB0Y3AiIHBvcnQ9IjUxNCIpCmlucHV0KHR5cGU9ImltdWRwIiBwb3J0PSI1MTQiKQppbnB1dCh0eXBlPSJpbXJlbHAiIHBvcnQ9IjE2MDEiKQoKIyBpbmNsdWRlcyBkb25lIGV4cGxpY2l0ZWx5CmluY2x1ZGUoZmlsZT0iL2V0Yy9yc3lzbG9nLmNvbmYuZC9sb2dfdG9fbG9nc2VuZS5jb25mIiBjb25maWcuZW5hYmxlZD1gZWNobyAkRU5BQkxFX0xPR1NFTkVgKQppbmNsdWRlKGZpbGU9Ii9ldGMvcnN5c2xvZy5jb25mLmQvbG9nX3RvX2ZpbGVzLmNvbmYiIGNvbmZpZy5lbmFibGVkPWBlY2hvICRFTkFCTEVfTE9HRklMRVNgKQoKIyMjIyMjIyMjIyMjIyMjIyMjIyMgZGVmYXVsdCBydWxlc2V0IGJlZ2lucyAjIyMjIyMjIyMjIyMjIyMjIyMjIwoKIyB3ZSBlbWl0IG91ciBvd24gbWVzc2FnZXMgdG8gZG9ja2VyIGNvbnNvbGU6CnN5c2xvZy4qIDpvbXN0ZG91dDoKCmluY2x1ZGUoZmlsZT0iL2NvbmZpZy9kcm9wcnVsZXMuY29uZiIgbW9kZT0ib3B0aW9uYWwiKSAgIyB0aGlzIHBlcm1pdHMgdGhlIHVzZXIgdG8gZWFzaWx5IGRyb3AgdW53YW50ZWQgbWVzc2FnZXMKCmFjdGlvbihuYW1lPSJtYWluX3V0ZjhmaXgiIHR5cGU9Im1tdXRmOGZpeCIgcmVwbGFjZW1lbnRDaGFyPSI/IikKCmluY2x1ZGUodGV4dD1gZWNobyAkQ05GX0NBTExfTE9HX1RPX0xPR0ZJTEVTYCkKaW5jbHVkZSh0ZXh0PWBlY2hvICRDTkZfQ0FMTF9MT0dfVE9fTE9HU0VORWApCgppbnB1dCh0eXBlPSJpbWZpbGUiIHRhZz0idGVzdHRhZyIgZmlsZT0iL3Zhci9sb2cvKi5sb2ciKQo=" \
  -v "/var/log/auth.log:/var/log/auth.log:ro" \
  --hostname=$(hostname) \
  -m 50m \
  rsyslog/syslog_appliance_alpine \
  rsyslog

Please let me know if I failed to address any questions.

[rgerhards]

@tgjamin Thanks a lot for your very clear response. It is extremely helpful to me. Just note that I am finalizing things for the next rsyslog release, so a proper reply from me may take a short while, but it for sure will happen. I would appreciate if you could also advise from time to time on how we proceed with the container. Based on your feedback, I am definitely able to provide an even more useful container!

[fernviridian]

Hi @rgerhards just wanted to check in to see how review/discussion on this PR is going since it's been a little over two weeks.

[rgerhards]

@tgjamin sorry, we are a relatively small project and lots of things are going on. I need to change a bit of the overall structure of doc etc as the PR only addresses part of it. If I just merge as-is without changing the rest (all the other config options), things become inconsistent. If it is really urgent for you, I can do so, but I would prefer to have on consistent step.

side-note, you probably know this much better than I do: my plan is to add a tag previous_version to the current container, and then update. So we have a fallback if something goes wrong with the update. It's a general question I have thought about, but could not sufficiently research (you see, it's more complex to merge this for me than it looks externally). Same question when I push new rsyslog versions into the container. Help would appreciated.

[fernviridian]

@rgerhards Totally understand the small team and lots of work. I'm not in an urgent rush to get this merged. If I need the feature sooner, I will just work off of a fork of the project. I know I mentioned changing the code a bit to overwrite the rsyslog.conf in the /config/ directory instead of the /etc/rsyslog.conf file here:

Perhaps the RSYSLOG_CONFIG_BASE64 environment variable should instead write the config to the /config/ directory, to utilize the docker volume there. This would have the same effect, and overwrite the default /etc/rsyslog.conf file. This would maintain the read-only property of the container's root filesystem. How does that sound?

Should I update the PR sometime to reflect this, or wait until later?

As for tagging of the docker images, having the previous and latest tags on DockerHub seems ok to me, considering this is a work-in-progress. When the project is more stable, perhaps having DockerHub image tags that are the same as the Rsyslog version would be helpful, should someone want to use a specific version of Rsyslog. Looking at other DockerHub projects, this seems ok to me. I looked at https://hub.docker.com/r/rancher/server/tags/ which is a orchestrator container we use, where they have tagged images for each version of their software, and ones for latest, and master.

[rgerhards]

I would appreciate if you could update the PR. I'll try to do some more work on the container over the weekend or during next week.

[fernviridian]

Changed the PR to create a /config/rsyslog.conf file instead of overwriting the /etc/rsyslog.conf file in c29d8d9 . This should allow the container's root filesystem to be read only. I also noticed the RSYSLOG_CONF flag was not being used when starting rsyslog with the tools/rsyslog start script. I fixed this issue so if not set, the script in starter.sh will default it to /etc/rsyslog.conf.

[rgerhards]

@tgjamin many thanks again for your contribution, much appreciated. Will merge now and update the container. Tag previous already added, in case a fallback is required.

Any more suggestions, comments, hints or actual code you could contribute is highly appreciated!

[rgerhards]

container images has now also been updated and is using 8.34 now