Upgrade ossec to v3.0.0 #3724

emkll · 2018-08-21T21:08:41Z

Status

Ready for review

Fixes #3701:

Updates ossec to v3.0.0, modifies build logic to use GPG signatures for verifying source.

Testing

Clean install scenario

make build-debs
Provision staging vms on this branch
./var/ossec/bin/ossec-control status returns running for all except ossec-execd
Observe ./var/ossec/bin/list-agents -c on mon server returns the app agent
Observe that ossec mails are sent if email is configured is prod

Upgrade scenario

make build-debs
copy and install securedrop-ossec-* and ossec-* to app and mon servers
./var/ossec/bin/ossec-control status returns running for all except ossec-execd
observe ./var/ossec/bin/list-agents -c on mon server returns the app agent
Observe that ossec mails are sent if email is configured is prod

Deployment

Updates to securedrop-ossec-{agent,server} and ossec-{agent,server} will be served by apt and will be upgraded automatically.

Checklist

If you made changes to the server application code:

Linting (make ci-lint) and tests (make -C securedrop test) pass in the development container

If you made changes to `securedrop-admin`:

Linting and tests (make -C admin test) pass in the admin development container

If you made changes to the system configuration:

Configuration tests pass

If you made non-trivial code changes:

I have written a test plan and validated it for this PR

If you made changes to documentation:

Doc linting (make docs-lint) passed locally

msheiny · 2018-09-28T22:23:34Z

So far I'm testing on an upgrade scenario path...

I'm actually installing from a 0.9.1 stock.. and then i ran the apt install of the latest ossec packages on both servers... Seems like mon comes up fine but app showed this status

root@app-staging:/home/vagrant# /etc/init.d/ossec status
ossec-logcollector: Process 5822 not used by ossec, removing ..
ossec-logcollector not running...
ossec-syscheckd not running...
ossec-agentd: Process 5818 not used by ossec, removing ..
ossec-agentd not running...
ossec-execd not running..

i tried to reboot that service and i get

root@app-staging:/home/vagrant# /etc/init.d/ossec restart
ossec-logcollector not running ..
ossec-syscheckd not running ..
ossec-agentd not running ..
ossec-execd not running ..
OSSEC HIDS v3.0.0 Stopped
Starting OSSEC HIDS v3.0.0 (by Trend Micro Inc.)...
Started ossec-execd...
2018/09/28 22:15:42 ossec-agentd: INFO: Using notify time: 600 and max time to reconnect: 1800
Started ossec-agentd...
Started ossec-logcollector...
2018/09/28 22:15:45 ossec-syscheckd(1210): ERROR: Queue '/var/ossec/queue/ossec/queue' not accessible: 'Connection refused'.
2018/09/28 22:15:45 rootcheck(1210): ERROR: Queue '/var/ossec/queue/ossec/queue' not accessible: 'Connection refused'.
2018/09/28 22:15:53 ossec-syscheckd(1210): ERROR: Queue '/var/ossec/queue/ossec/queue' not accessible: 'Connection refused'.
2018/09/28 22:15:53 rootcheck(1210): ERROR: Queue '/var/ossec/queue/ossec/queue' not accessible: 'Connection refused'.
2018/09/28 22:16:06 ossec-syscheckd(1210): ERROR: Queue '/var/ossec/queue/ossec/queue' not accessible: 'Connection refused'.
2018/09/28 22:16:06 rootcheck(1211): ERROR: Unable to access queue: '/var/ossec/queue/ossec/queue'. Giving up..
ossec-syscheckd did not start

Ya want to sync on this next week? I'd love to take a more active role in ushering this ticket thru if ya want some help. Maybe I just did something silly though -- so I'm also happy to get yelled at next week :)

msheiny · 2018-10-01T18:56:55Z

rebased on develop 71ca2ed58

emkll · 2018-10-02T15:24:10Z

After spending some time debugging, it seems like the agent and server debs that are compiled with the build-ossec-deb-pkg role (https://github.com/freedomofpress/securedrop/blob/develop/install_files/ansible-base/roles/build-ossec-deb-pkg/tasks/main.yml#L95) now provide an empty /var/ossec/etc/client.key, which causes the ossec agent to fail startup. This was not the case with debs build from upstream, nor with version 2.8.2 debs.

This is also why the clean install works, but the upgrade scenario does not, as the client key is overwritten in the configuration.

emkll · 2018-10-02T19:06:21Z

@conorsch do you have any clue as to where the contents of /var/ossec/ comes from on this line ?https://github.com/freedomofpress/securedrop/blob/develop/install_files/ansible-base/roles/build-ossec-deb-pkg/tasks/main.yml#L95 ? The contents of this are likely overwriting the client.keys

EDIT: Not sure how I missed it, but https://github.com/freedomofpress/securedrop/blob/develop/install_files/ansible-base/roles/build-ossec-deb-pkg/tasks/main.yml#L84 installs ossec on the build host and then contents of /var/ossec are moved into the build dir. Removing the empty client.keys appears to do the trick !

Ossec 3.0.0 contains several bug and security fixes, as such we should upgrade securedrop ossec agents and servers to this release.

As of 2.9.3, ossec source tarballs are now signed with a GPG key, instead of providing checksums. The following should now verify source code tarballs when packages are built/

- Generate and use shared secret which is required for agent registration - agent-auth now returns 0 when registration failure occurs

Due to issues with ossec 2.8.2+ and disabling of the ipv6 stack, name lookups can't `getaddrinfo: Name or service not known`. Using ip addresses in lieu of aliases sidesteps the issue. - Since ossec.conf is not templated, securedrop-ossec agent and server will replace these values as part of the postinst.

An empty client.keys was overwriting /var/ossec/etc/client.keys with an empty one, breaking the registration between client and server. Removing the empty client.keys prior to building will ensure the keys are preserved during an upgrade.

conorsch · 2018-10-03T12:11:37Z

installs ossec on the build host and then contents of /var/ossec are moved into the build dir

That's right, and it's specifically the USER_DIR value in the vars template that sets /var/ossec as the dest.

Removing the empty client.keys appears to do the trick !

Glad to hear it! Given the cycles we're spending on pulling off the upgrade, now seems a fine time to bump mention of #2140, in case it would reduce our maintenance burden.

emkll · 2018-10-03T14:36:17Z

Thanks for the help @msheiny, this is ready for re-review: based on my testing in prod VMs the bug you experienced should be fixed, emails now flow in.
Since the existing postinst scrips work with 3.0.0, I've opted for the smallest diff for this PR, perhaps we should maybe revisit the build strategy and/or using wazuh in an upcoming engineering meeting.

zenmonkeykstop · 2018-10-03T14:46:57Z

@emkll is there much if any change so far to the user experience (email formatting, sensitivity of triggers etc) compared to 2.8.2?

emkll · 2018-10-03T14:58:16Z

@zenmonkeykstop: good question. The emails look the same, and based on < 24h of observations, the alerts appear to be the same as before, but we should keep an eye out for new alerts and update the docs if necessary. Release notes state that were some changes in 2.8.3 for Apache/Postfix, but a closer look at changes in https://github.com/ossec/ossec-hids/tree/master/etc/rules indicates they are minor.

msheiny

Looking good on a second pass of an upgrade test! 🎉

Confirmed services were up and running on 3.x on both servers.. connection maintained .. and alerts were being processed by tailing the logs. I also rebooted boxes and confirmed behavior remained.

Doing scrach test now.

msheiny

Yep looking good on the from scratch front as well! Hooraaaaaayyyyyyyyyyyyyyyyy

emkll requested review from conorsch and msheiny as code owners August 21, 2018 21:08

emkll mentioned this pull request Aug 21, 2018

Update OSSEC to v3.0 #3701

Closed

emkll force-pushed the 3701-bump-ossec-3.0.0 branch 3 times, most recently from 76ec489 to 146b813 Compare September 11, 2018 16:28

emkll force-pushed the 3701-bump-ossec-3.0.0 branch from 146b813 to f7eb0d3 Compare September 28, 2018 15:06

emkll changed the title ~~WIP - Do not merge - Upgrade ossec to v3.0.0~~ Upgrade ossec to v3.0.0 Sep 28, 2018

emkll force-pushed the 3701-bump-ossec-3.0.0 branch from f7eb0d3 to f531d03 Compare September 28, 2018 15:36

msheiny force-pushed the 3701-bump-ossec-3.0.0 branch from f531d03 to 6a562fd Compare October 1, 2018 18:55

emkll force-pushed the 3701-bump-ossec-3.0.0 branch 2 times, most recently from 27ac0f0 to b8f97d2 Compare October 2, 2018 21:25

emkll added 5 commits October 2, 2018 17:46

Bump ossec version strings to 3.0.0

2c79f0b

Ossec 3.0.0 contains several bug and security fixes, as such we should upgrade securedrop ossec agents and servers to this release.

Use GPG to verify ossec tarballs

7ef2613

As of 2.9.3, ossec source tarballs are now signed with a GPG key, instead of providing checksums. The following should now verify source code tarballs when packages are built/

Update ossec registration logic for 3.0

36a989a

- Generate and use shared secret which is required for agent registration - agent-auth now returns 0 when registration failure occurs

Remove client.keys

bddc30d

An empty client.keys was overwriting /var/ossec/etc/client.keys with an empty one, breaking the registration between client and server. Removing the empty client.keys prior to building will ensure the keys are preserved during an upgrade.

emkll force-pushed the 3701-bump-ossec-3.0.0 branch from b8f97d2 to bddc30d Compare October 2, 2018 21:47

msheiny reviewed Oct 3, 2018

View reviewed changes

msheiny approved these changes Oct 3, 2018

View reviewed changes

msheiny merged commit bcc624d into develop Oct 3, 2018

msheiny deleted the 3701-bump-ossec-3.0.0 branch October 3, 2018 20:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade ossec to v3.0.0 #3724

Upgrade ossec to v3.0.0 #3724

emkll commented Aug 21, 2018 •

edited

Loading

msheiny commented Sep 28, 2018

msheiny commented Oct 1, 2018

emkll commented Oct 2, 2018 •

edited

Loading

emkll commented Oct 2, 2018 •

edited

Loading

conorsch commented Oct 3, 2018

emkll commented Oct 3, 2018

zenmonkeykstop commented Oct 3, 2018

emkll commented Oct 3, 2018

msheiny left a comment

msheiny left a comment

Upgrade ossec to v3.0.0 #3724

Upgrade ossec to v3.0.0 #3724

Conversation

emkll commented Aug 21, 2018 • edited Loading

Status

Testing

Clean install scenario

Upgrade scenario

Deployment

Checklist

If you made changes to the server application code:

If you made changes to securedrop-admin:

If you made changes to the system configuration:

If you made non-trivial code changes:

If you made changes to documentation:

msheiny commented Sep 28, 2018

msheiny commented Oct 1, 2018

emkll commented Oct 2, 2018 • edited Loading

emkll commented Oct 2, 2018 • edited Loading

conorsch commented Oct 3, 2018

emkll commented Oct 3, 2018

zenmonkeykstop commented Oct 3, 2018

emkll commented Oct 3, 2018

msheiny left a comment

Choose a reason for hiding this comment

msheiny left a comment

Choose a reason for hiding this comment

emkll commented Aug 21, 2018 •

edited

Loading

If you made changes to `securedrop-admin`:

emkll commented Oct 2, 2018 •

edited

Loading

emkll commented Oct 2, 2018 •

edited

Loading