Rsyslog Support

[thetuxinator]

Hi, as logstash is very slow and ressource intensive compared to syslog (which is by nature as Java vs C) what about supporting Rsyslog with the omelastic method?

regards
tuxinator

[enotspe]

Can you elaborate a little bit more please??

We use logstash mainly because it is the main tool for writing data into Elasticsearch

[thetuxinator]

Hi, well i know it comes with it, however we had syslog in place and one of our consultants recommended logstash, while our rsyslog servers where never busy and still now face a load of 0.14 at max, when we evaluated logstash, for the same amount of data it generated a load of 2 on the same server :D this is easy explainable as java vs C

[enotspe]

Thank you for sharing your experience @thetuxinator

To be honest, I did not know Rsyslog could output to ES. I had always thought on Kafka though, more for scalability and resilience. I know Kafka is still Java, so maybe it won't be such a gain in performance. But making any move on any of them would be very down far the road.

I do get that logstash is not very efficient, but I think we all are saving lot of money by using Fortinet+ES, so throwing a couple more cores and RAM to logstash should not hurt the wallet.

[thetuxinator]

Well you save even more money and time by just using syslog. After working 20 years as a Linux System Engineer i don't get why the new generation always wants to reinvent the wheel ;-) let me compare logstash vs syslog/rsyslog options in short:

Rsyslog:

• Fast as hell as written in C
• Always part of almost all distributions
• needs almost no additional setup
• 90% of the infrastructure is already there on Linux (and forti) no need to install logstash
• needs 1 line of configuration to send everything as a copy to a central syslog server
• almost all products/operating systems and hardware devices since 1980 support it
• Most software on Linux already sends everything to the local syslog so no configuration needed
• Proven standard (RFC) so nobody can do some ugly hacks, change everything on their own opinion as it would break with RFC
• Known to be rock-solid
• Works together with journald (future proven)
• Rsyslog supports many plugins
• Already comes with features like handling properly duplicate flood messages and not filling up the disk in those cases and overloading the syslog server
• You always send to the local syslog which then forwards so network-problems are no problems and your software keeps running
• If you are out of space everything keeps running, as it will begin to discard logs and your software keeps running
• Passed many code reviews by the time
• Needs only omelasticsearch module setup to send to elastic
• the logs are in text-format to easy to compress and rotate (logrotate is also part of every Linux)

Logstash:

• Needs to be installed additionaly
• Not known to have had a code-review
• No RFC standard
• No wide support and compatibility from almost all vendors/devices/software yet
• Slow in Text Processing (Java)
• Not rock solid - we had major downtime because of flooding messages which caused so high load, logstash discarded new connections and containers died because they could no more log
• no log deduplication (in case of flooding)
• Complicated additional non standard-setup

Correct if anything of the above is no more up2date.

IMHO:
As a last word, it is like with containers and Virtualization, just because "you can" does not mean "you should". Also especially in ICT where everybody claims "we lack human resources" people should maybe keep the effort needed down by using existing stuff instead of reinventing everything because they read something fancy somewhere ;-) .

Long story short, maybe we could try to get a working omelasticsearch configuration for this additionaly to the logstash option. i will check if i get anything when time allows.

Regards

[thetuxinator]

So far already having a look on it, https://chabik.com/2019/03/rsyslog-to-elasticsearch/ looks interesting and also https://gist.github.com/gmoon/3a778bf6948eb485e8fa5b87f625a3ff actually https://sematext.com/blog/recipe-rsyslog-elasticsearch-kibana/#toc-getting-all-the-ingredients-0 seems to provide a complete solution to rsyslog -> Elasic -> Kibana

[thetuxinator]

So far i have a working configuration for rsyslog. Still working on Kibana at the moment. Following your guide i can't get passed the part 3. "Load component templates. the referenced script seems not to be there and there is no real hint on how to do it in bulk or 1 by 1

[thetuxinator]

ok finally also got the part component templates working, however on a fortiweb i now get almost for all entries a "Field not found" for example for source.ip

[enotspe]

Well you save even more money and time by just using syslog. After working 20 years as a Linux System Engineer i don't get why the new generation always wants to reinvent the wheel ;-) let me compare logstash vs syslog/rsyslog options in short:

Rsyslog:

• Fast as hell as written in C
• Always part of almost all distributions
• needs almost no additional setup
• 90% of the infrastructure is already there on Linux (and forti) no need to install logstash
• needs 1 line of configuration to send everything as a copy to a central syslog server
• almost all products/operating systems and hardware devices since 1980 support it
• Most software on Linux already sends everything to the local syslog so no configuration needed
• Proven standard (RFC) so nobody can do some ugly hacks, change everything on their own opinion as it would break with RFC
• Known to be rock-solid
• Works together with journald (future proven)
• Rsyslog supports many plugins
• Already comes with features like handling properly duplicate flood messages and not filling up the disk in those cases and overloading the syslog server
• You always send to the local syslog which then forwards so network-problems are no problems and your software keeps running
• If you are out of space everything keeps running, as it will begin to discard logs and your software keeps running
• Passed many code reviews by the time
• Needs only omelasticsearch module setup to send to elastic
• the logs are in text-format to easy to compress and rotate (logrotate is also part of every Linux)

Logstash:

• Needs to be installed additionaly
• Not known to have had a code-review
• No RFC standard
• No wide support and compatibility from almost all vendors/devices/software yet
• Slow in Text Processing (Java)
• Not rock solid - we had major downtime because of flooding messages which caused so high load, logstash discarded new connections and containers died because they could no more log
• no log deduplication (in case of flooding)
• Complicated additional non standard-setup

Correct if anything of the above is no more up2date.

IMHO: As a last word, it is like with containers and Virtualization, just because "you can" does not mean "you should". Also especially in ICT where everybody claims "we lack human resources" people should maybe keep the effort needed down by using existing stuff instead of reinventing everything because they read something fancy somewhere ;-) .

Long story short, maybe we could try to get a working omelasticsearch configuration for this additionaly to the logstash option. i will check if i get anything when time allows.

Regards

I fully agree on everything you mention. It is out of discussion the benefits of syslog as a logging parading now. About Rsyslog, as it's own name, it is specially crafted for syslog so it is no surprise that its performance is tailored for it. Regarding Logstash, syslog is just one of its usecases. Logstash is a multipurpose ETL, which has great features as well. I tried to do SNMP with logstash and it was even worse, but for other usecases is very suitable. As always, in Cyber, everything just "depends". I have suffered outages with logstash like the ones you had. Once my logstash server was even bigger than my Elastic servers in order to handle load.

I am by no means and "expert" on linux or even logstash, my knowhow is more on firewalls, so I am very glad you have shared your experience and will be glad as well if you can share a config of Rsyslog+ES. I will be happy to accept a PR and work together on it.

The overall vision of the project is not restricted to logstash, nor Fortinet. Someday we will be just Dragons and not FortiDragons, Palo Alto is for sure the next firewall I would like to explore. But this is still a side project to me, so for now the next priority steps will be to work on ECS compatibility on pipelines (which I think are gonna improve performance), and FortiOS v7+ testing (I think they have changed the syslog message a little bit). We have also done some work on SNMP on Fortinet with Zabbix, we will be sharing our templates once they are more polished.

At the end, the main goal is to have a platform that can help us extract more info about the logs and perform better analysis and threat hunting, regarding of the tools (logstash, kafka, rsyslog, kibana, grafana, etc.) we use beneath. Hopefully FortiDragon will be the field where we all engage as a community.

[thetuxinator]

I fully agree on everything you mention. It is out of discussion the benefits of syslog as a logging parading now. About Rsyslog, as it's own name, it is specially crafted for syslog so it is no surprise that its performance is tailored for it. Regarding Logstash, syslog is just one of its usecases. Logstash is a multipurpose ETL, which has great features as well. I tried to do SNMP with logstash and it was even worse, but for other usecases is very suitable. As always, in Cyber, everything just "depends". I have suffered outages with logstash like the ones you had. Once my logstash server was even bigger than my Elastic servers in order to handle load.

Agree

I am by no means and "expert" on linux or even logstash, my knowhow is more on firewalls, so I am very glad you have shared your experience and will be glad as well if you can share a config of Rsyslog+ES. I will be happy to accept a PR and work together on it.

Well i am a Linux expert, so if i may help, just tell.

The overall vision of the project is not restricted to logstash, nor Fortinet. Someday we will be just Dragons and not FortiDragons, Palo Alto is for sure the next firewall I would like to explore. But this is still a side project to me, so for now the next priority steps will be to work on ECS compatibility on pipelines (which I think are gonna improve performance), and FortiOS v7+ testing (I think they have changed the syslog message a little bit). We have also done some work on SNMP on Fortinet with Zabbix, we will be sharing our templates once they are more polished.

Sounds nice, btw we also use Zabbix

At the end, the main goal is to have a platform that can help us extract more info about the logs and perform better analysis and threat hunting, regarding of the tools (logstash, kafka, rsyslog, kibana, grafana, etc.) we use beneath. Hopefully FortiDragon will be the field where we all engage as a community.

Sounds good

regards

Mike

[thetuxinator]

ok finally also got the part component templates working, however on a fortiweb i now get almost for all entries a "Field not found" for example for source.ip

Any hint on above?

[enotspe]

ok finally also got the part component templates working, however on a fortiweb i now get almost for all entries a "Field not found" for example for source.ip

Any hint on above?

You mean on Kibana??? But do you see your data on the index?? I mean, do you have those fields indeed??? I would delete the index and let it recreate again, fields should be populated automatically

[enotspe]

You can open another issue for that. Or contact on the discord channel as well if you prefer.

[thetuxinator]

Let me create a fork where i document syslog stuff i did (enhance README with it) so we can work on that and merge later if you want so