Comprehensive testing for OSSEC rules

[redshiftzero]

Description

We should have automated testing to ensure that particular log events do or do not produce OSSEC alerts. We can begin by adding regression tests when patching the OSSEC rules.

It appears that the right tool for testing if particular log events do or do not produce OSSEC alerts is ossec-logtest. This is currently installed on mon. For example, if I want to test whether the log event described in PR #871 produces an alert, I can simply pass the event to ossec-logtest, it will read the rules that are currently configured, and inform me whether or not an OSSEC rule would be produced, and if so, what the alert level would be:

root@mon-staging:/home/vagrant# sudo echo "Feb 10 23:34:40 app-prod kernel: [  124.188641] grsec: denied RWX mmap of <anonymous mapping> by /usr/sbin/apache2[apache2:1328] uid/euid:33/33 gid/egid:33/33, parent /usr/sbin/apache2[apache2:1309] uid/euid:0/0 gid/egid:0/0" | /var/ossec/bin/ossec-logtest
2017/08/16 22:28:25 ossec-testrule: INFO: Reading local decoder file.
2017/08/16 22:28:25 ossec-testrule: INFO: Started (pid: 18973).
ossec-testrule: Type one log per line.



**Phase 1: Completed pre-decoding.
       full event: 'Feb 10 23:34:40 app-prod kernel: [  124.188641] grsec: denied RWX mmap of <anonymous mapping> by /usr/sbin/apache2[apache2:1328] uid/euid:33/33 gid/egid:33/33, parent /usr/sbin/apache2[apache2:1309] uid/euid:0/0 gid/egid:0/0'
       hostname: 'app-prod'
       program_name: 'kernel'
       log: '[  124.188641] grsec: denied RWX mmap of <anonymous mapping> by /usr/sbin/apache2[apache2:1328] uid/euid:33/33 gid/egid:33/33, parent /usr/sbin/apache2[apache2:1309] uid/euid:0/0 gid/egid:0/0'

**Phase 2: Completed decoding.
       decoder: 'iptables'

**Phase 3: Completed filtering (rules).
       Rule id: '100101'
       Level: '7'
       Description: 'grsec error was detected'
**Alert to be generated.

We can use this ossec-logtest utility to create new testinfra tests for our OSSEC rules.

Eventually, we could also incorporate testing for OSSEC into our integration testing: when user interaction occurs, we should verify that OSSEC alerts are not produced. This would catch bugs like #1331 and #1240.

[conorsch]

Relevant for the testing story here is that OSSEC 2.9 added support for JSON logging. We'll need to evaluate whether it's worth changing the output format given that we also pipe the events to mail for transport.

Before we upgrade OSSEC we should tackle #1756. I also don't think we have a discrete issue for bumping the OSSEC version, so I'll add that presently.

[redshiftzero]

Writing automated tests for syscheck (OSSEC's file integrity checking process) is not as straightforward as one cannot use ossec-logtest: in an ideal world syscheck would create a log that could be fed into ossec-logtest for testing purposes, but this does not appear to be the case. However, see these comments and the testing steps in PR #1691 for how one can manually verify fixes to syscheck. One could reproduce the manual steps described there to automate these tests, but there must be a more straightforward option.