Initial start on parsing PAM messages #8756
Conversation
|
Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually? |
|
Need to figure out how to make the mappings... |
There was a problem hiding this comment.
Thanks @opoplawski for this contribution! This is looking good, could you also add some example logs for testing?
| @@ -15,6 +15,8 @@ | |||
| "%{SYSLOGTIMESTAMP:system.auth.timestamp} %{SYSLOGHOST:system.auth.hostname} sudo(?:\\[%{POSINT:system.auth.pid}\\])?: \\s*%{DATA:system.auth.user} :( %{DATA:system.auth.sudo.error} ;)? TTY=%{DATA:system.auth.sudo.tty} ; PWD=%{DATA:system.auth.sudo.pwd} ; USER=%{DATA:system.auth.sudo.user} ; COMMAND=%{GREEDYDATA:system.auth.sudo.command}", | |||
| "%{SYSLOGTIMESTAMP:system.auth.timestamp} %{SYSLOGHOST:system.auth.hostname} groupadd(?:\\[%{POSINT:system.auth.pid}\\])?: new group: name=%{DATA:system.auth.groupadd.name}, GID=%{NUMBER:system.auth.groupadd.gid}", | |||
| "%{SYSLOGTIMESTAMP:system.auth.timestamp} %{SYSLOGHOST:system.auth.hostname} useradd(?:\\[%{POSINT:system.auth.pid}\\])?: new user: name=%{DATA:system.auth.useradd.name}, UID=%{NUMBER:system.auth.useradd.uid}, GID=%{NUMBER:system.auth.useradd.gid}, home=%{DATA:system.auth.useradd.home}, shell=%{DATA:system.auth.useradd.shell}$", | |||
| "%{SYSLOGTIMESTAMP:system.auth.timestamp} %{SYSLOGHOST:system.auth.hostname}? %{DATA:system.auth.program}(?:\\[%{POSINT:system.auth.pid}\\])?: pam_%{DATA:system.auth.pam.module}\\(%{DATA:system.auth.pam.service}:%{DATA:system.auth.pam.type}\\): %{DATA:system.auth.pam.result}; logname=%{DATA:system.auth.pam.logname} uid=%{INT:system.auth.pam.uid} euid=%{INT:system.auth.pam.euid} tty=%{DATA:system.auth.pam.tty} ruser=%{DATA:system.auth.pam.ruser} rhost=%{DATA:system.auth.pam.rhost} user=%{DATA:system.auth.pam.user}", | |||
| "%{SYSLOGTIMESTAMP:system.auth.timestamp} %{SYSLOGHOST:system.auth.hostname}? %{DATA:system.auth.program}(?:\\[%{POSINT:system.auth.pid}\\])?: pam_%{DATA:system.auth.pam.module}\\(%{DATA:system.auth.pam.service}:%{DATA:system.auth.pam.type}\\): received for user %{DATA:system.auth.pam.user}: %{POSINT:system.auth.pam.errno} \\(%{DATA:system.auth.pam.errmsg}\\)", | |||
There was a problem hiding this comment.
Could you also add some examples of the log lines these patterns are parsing? If you add them to filebeat/module/system/auth/test they will be used on testing.
There was a problem hiding this comment.
I'm adding them to test.log, hope that's right.
There was a problem hiding this comment.
As I mention in the other comment, you can also add them to an specific file.
| The PAM result text, e.g. "authentication success". | ||
| - name: logname | ||
| description: > | ||
| The PAM logname. |
There was a problem hiding this comment.
What is this logname? maybe it'd be good to add an example here too.
| - name: rhost | ||
| type: ip | ||
| description: > | ||
| The remote host IP address. |
There was a problem hiding this comment.
These fields could be remote.user and remote.ip.
| type: long | ||
| description: > | ||
| The PAM result error number. | ||
| - name: errmsg |
There was a problem hiding this comment.
You may want to index error message for full text search with type: text.
|
At this point I can figure out what isn't matching in expected.json output. |
|
@opoplawski thanks for adding the test entries, they will be very helpful! You can also add them to a new file instead of using the existing one, all files that exist in test directories with a corresponding expected.json file are used in tests. Completing the expected.json file can be a bit tricky, there is an option that helps on that, if you run the test with If you have problems with this let me know and I think we can leave it for another PR. |
| type: long | ||
| description: > | ||
| The PAM result error number. | ||
| - name: message |
There was a problem hiding this comment.
type: text also for error.message?
| The requesting user name. | ||
| - name: rhost | ||
| description: > | ||
| The requesting hostname or IP address. |
There was a problem hiding this comment.
remote.user and remote.host instead of ruser and rhost? Just proposing, I am ok also with ruser and rhost if they are usual names for these values in the PAM context 🙂
|
This will need a changelog entry |
| @@ -15,22 +15,26 @@ | |||
| "%{SYSLOGTIMESTAMP:system.auth.timestamp} %{SYSLOGHOST:system.auth.hostname} sudo(?:\\[%{POSINT:system.auth.pid}\\])?: \\s*%{DATA:system.auth.user} :( %{DATA:system.auth.sudo.error} ;)? TTY=%{DATA:system.auth.sudo.tty} ; PWD=%{DATA:system.auth.sudo.pwd} ; USER=%{DATA:system.auth.sudo.user} ; COMMAND=%{GREEDYDATA:system.auth.sudo.command}", | |||
| "%{SYSLOGTIMESTAMP:system.auth.timestamp} %{SYSLOGHOST:system.auth.hostname} groupadd(?:\\[%{POSINT:system.auth.pid}\\])?: new group: name=%{DATA:system.auth.groupadd.name}, GID=%{NUMBER:system.auth.groupadd.gid}", | |||
| "%{SYSLOGTIMESTAMP:system.auth.timestamp} %{SYSLOGHOST:system.auth.hostname} useradd(?:\\[%{POSINT:system.auth.pid}\\])?: new user: name=%{DATA:system.auth.useradd.name}, UID=%{NUMBER:system.auth.useradd.uid}, GID=%{NUMBER:system.auth.useradd.gid}, home=%{DATA:system.auth.useradd.home}, shell=%{DATA:system.auth.useradd.shell}$", | |||
| "%{SYSLOGTIMESTAMP:system.auth.timestamp} %{SYSLOGHOST:system.auth.hostname}? %{DATA:system.auth.program}(?:\\[%{POSINT:system.auth.pid}\\])?: %{GREEDYMULTILINE:system.auth.message}" | |||
| "%{SYSLOGTIMESTAMP:system.auth.timestamp} %{SYSLOGHOST:system.auth.hostname}? %{DATA:system.auth.program}(?:\\[%{POSINT:system.auth.pid}\\])?: pam_%{DATA:system.auth.pam.module}\\(%{DATA:system.auth.pam.service}:%{DATA:system.auth.pam.type}\\): %{DATA:system.auth.pam.result}; logname=%{DATA:system.auth.pam.auth.logname} uid=%{INT:system.auth.pam.auth.uid} euid=%{INT:system.auth.pam.auth.euid} tty=%{DATA:system.auth.pam.auth.tty} ruser=%{DATA:system.auth.pam.auth.ruser} rhost=%{DATA:system.auth.pam.auth.rhost} user=%{DATA:system.auth.pam.user}$", | |||
There was a problem hiding this comment.
@original-brownbear I wonder if there are any recommendations from the ES ingest side on how we should handle that many ingest pipelines to make sure they stay performant.
There was a problem hiding this comment.
@ruflin I'm not sure I understand the question 100% but:
- The number of configured ingest pipelines should not affect performance in and of itself I think.
- @jakelandis merge per processor stats in ingest: processor stats elasticsearch#34202 so that's incoming as a tool to measure the impact of the various individual GROK configs that may help you?
Is that what you're looking for? :)
There was a problem hiding this comment.
Sorry, I should have mentioned "multiple grok patterns" affect perfomance as we use many different grok patterns above.
Looking forward to the stats.
There was a problem hiding this comment.
@ruflin ah got it :) So I think you don't have to worry about this too much.
Grok compiles all the patterns into a single Regex for matching. So it's more a question of what the individual Regex pieces look like than how many you have.
For non-backtracking Regex things should scale better than linear with the length of the single Regex we compile.
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
botelastic
bot
added
Stalled
needs_team
|
It may make sense to just start over, but I'll try to poke at this some more |
❕ Build Aborted
Expand to view the summary
Build stats
Steps errorsExpand to view the steps failures
Log outputExpand to view the last 100 lines of log output
|
andresrc
added
the
Team:Services
|
Pinging @elastic/integrations-services (Team:Services) |
botelastic
bot
removed
the
needs_team
|
Hi! We're labeling this issue as |
|
Hey @opoplawski, I think we can go on with this change, but could you update the branch with master and run We should review if it would make sense to use ECS for some of the fields, for example Also this will need a changelog entry. |
|
Pinging @elastic/siem (Team:SIEM) |
|
Hi! We're labeling this issue as |
|
Hi! |
Is this a reasonable approach?