[Auditbeat] Reduce system/socket logging noise

[adriansr]

What does this PR do?

This PR updates the logging of system/socket dataset to be less noisy, especially when -d '*' is used. Also updates some code that could introduce a difficult to spot bug in the future.

• Make ThreadEnter errors less noisy

  The socket dataset uses ThreadEnter(event) / ThreadLeave(event) as a
  single-event per-thread state tracking, useful for correlating function
  calls and their return values.

  However, in some cases functions are stacked, like sys_execve() calling
  itself recursively or inet6_create() calling inet_create(). This results
  in an existing stored event to be evicted, which is not a problem but
  currently is causing a warning to be printed to the logs.

  This patch makes two changes to this situation:

  • Only print warnings from the state machine when socketdetailed selector
    is enabled. The state machine currently only generates warnings for
    ThreadEnter/ThreadExit issues.

  • Change ThreadEnter errors to be constructed on demand by their Error()
    method, so that the somewhat expensive fmt.Sprintf() / event.String()
    is only invoked if the error is going to be printed to the log.

  This is a huge CPU saving in systems where this benign eviction is
  happening frequently.

• socketdetailed selector has to be enabled explicitly

  This selector is extremely noisy. This change excludes it from being
  enabled when debug is enabled with -d '*' and requires it to be
  explicitly defined: -d '*,socketdetailed'.

• Print template variables when socket debug is enabled

  It makes no sense to print guesses' progress in regular debug (socket)
  and the resulting template variables only when socketdetailed is set.

• Fix syscall arguments usage in guesses

  The syscall parameter templates (SYS_Pn) were not valid until
  guess_syscall_args runs. As the variables SYS_Pn already existed,
  another guess using them could use the wrong values because the
  dependency mechanism only checks if the variable exists.

  The fix for this is to have temporary variables (_SYS_Pn) and have
  guess_syscall_args create the definitive ones.

  This didn't cause any bug as the only guess that used syscall
  args is the new guess_deref which is enabled on demand via an
  environment variable for diagnostic purposes.

Why is it important?

Two reasons:

• In some cases this dataset would generate noisy and misleading messages in the logs at INFO level. This causes unnecessary concern and increased CPU usage.
• The socketdetailed selector being activated by default when -d '*' is specified caused the logs to be full of kprobe events at rates of 10.000 per second or more. Log files would rotate every few minutes.

Checklist

• My code follows the style guidelines of this project

• I have commented my code, particularly in hard-to-understand areas

• I have made corresponding changes to the documentation

• I have made corresponding change to the default configuration files

• I have added tests that prove my fix is effective or that my feature works

• I haven't added a changelog entry because I don't think these changes have entity to deserve it.

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[andrewkroh]

LGTM