Formatting output of alerts generated by k8saudit plugin does not work

[sboschman]

The -p flag (docs) seems to only work with syscall alerts, but not with alerts from plugins (k8saudit).

How to reproduce it

Install Falco 0.33.0 (tested on AlmaLinux, with rpm install).

Enable the k8saudit plugin in falco.yaml:

load_plugins: [k8saudit, json]

Copy the k8s_audit_rules.yaml to rules.d and add a catch all rule for testing purposes:

- rule: Everything
  desc: Yes give us everything
  condition: >
    k8s_audit_always_true
  output: Audit event (user=%ka.user.name, groups=%ka.user.groups)
  priority: INFO
  source: k8s_audit
  tags: [k8s]

Run falco:

falco -p "This is Some Extra"

Expected behaviour

Expected to have the text 'This is Some Extra' appended to the output field of all alerts.

Screenshots

Edit a file under /etc/ to trigger a syscall alert:

$ vi /etc/yum.conf. -> don't have to edit anything, just saving is enough to trigger the syscall event

falco output:
13:51:44.559614255: Error File below /etc opened for writing (user=root user_loginuid=2001 command=vim /etc/yum.conf pid=1489 parent=bash pcmdline=bash file=/etc/yum.conf program=vim gparent=su ggparent=sudo gggparent=bash container_id=host image=<NA>) This is Some Extra

Indeed, as expected, the text 'This is Some Extra' is appended to the output.

Create a file event.json with. a k8s audit event blob:

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Request",
  "auditID": "547daa8a-bed4-33a2-7bd4-43341aab43ae",
  "stage": "RequestReceived",
  "requestURI": "/apis/extensions/v1beta1/namespaces/default/deployments/soluble-deployment/scale",
  "verb": "update",
  "user": {
    "username": "kubernetes-admin",
    "groups": ["system:masters", "system:authenticated"]
  },
  "sourceIPs": ["172.20.10.88"],
  "userAgent": "kubectl/v1.14.0 (linux/amd64) kubernetes/641856d",
  "objectRef": {
    "resource": "deployments",
    "namespace": "default",
    "name": "soluble-deployment",
    "apiGroup": "extensions",
    "apiVersion": "v1beta1",
    "subresource": "scale"
  },
  "requestReceivedTimestamp": "2020-02-10T08:41:23.177243Z",
  "stageTimestamp": "2020-02-10T08:41:23.177243Z"
}

Sent the audit event to falco using curl:

curl -H "Content-Type: application/json" --data @event.json http://localhost:9765/k8s-audit -v

falco output:

08:41:23.177243000: Informational Audit event (user=kubernetes-admin, groups=(system:masters,system:authenticated))

Unfortunately no 'This is Some Extra' text added to the alert from the k8saudit plugin.

Environment

• Falco version:
  Falco version: 0.33.0
  Libs version: 0.9.0
  Plugin API: 2.0.0
  Driver:
  API version: 2.0.0
  Schema version: 2.0.0
  Default driver: 3.0.1+driver
• System info:
  {
  "machine": "x86_64",
  "nodename": "xxx",
  "release": "4.18.0-425.3.1.el8.x86_64",
  "sysname": "Linux",
  "version": "# 1 SMP Tue Nov 8 14:08:25 EST 2022"
  }
• Cloud provider or hardware configuration:
  Virtual Machine
• OS:
  AlmaLinux 8.7 (Stone Smilodon)
• Kernel:
  Linux 4.18.0-425.3.1.el8.x86_64 # 1 SMP Tue Nov 8 14:08:25 EST 2022 x86_64 x86_64 x86_64 GNU/Linux
• Installation method:
  RPM

Additional context
My goal is to run a central falco (cluster) ingesting k8saudit events (or other cloud based events supported by falco plugins). A k8saudit event though misses important metadata to be able to trace the event back to the cluster (related discussion), like a cluster name/id.
The only way I see to do this, is by using the client authentication to provide additional metadata. A (reverse)proxy/ingress controller is responsible for handling authentication and pass additional metadata based on the authentication to falco by setting additional http headers. This is usually a pretty standard feature of a proxy (e.g. X-Forward-For headers). For example we can use mtls with k8s cluster specific client certs to authenticate. The proxy can forward the certificate subject field to the k8saudit plugin, and we can encode the cluster name in the subject for example.
I am looking into extending the k8saudit plugin with support to grab additional fields from http headers. Passing these values along the plugin event processing pipeline gets a bit ugly though. Only option seems to be altering the event from the http body with the header metadata. This unfortunately seems to mean the field names are fixed, as required by the extractor part.
Last challenge is to actually get the extra fields in the output, apart from editing all rules, the -p options seems atm the only way (related comment). See issue as well for additional output fields.
Downside of this approach is that each plugin requires a dedicated falco cluster, as the -p flag (with %field markers) is specific to each plugin. Falco validates the -p argument very early on, even before excluding rules with the -T/-t flags. E.g. adding a '%ka.auditid' with -p fails even if you only enable k8s rules (-t k8s).
Another downside is that the extra metadata is added to the output field. A better alternative is to only add these fields to the 'output_fields', not the 'output'. Just like falcosidekick allows to set 'custom fields', which are only added to the 'output_fields'.
Furthermore it would be ideal to be able to add 'custom' named fields to the output_fields of the alert. The additional fields a user wants to set, based on authentication/headers, is completely up to the user. Especially if you are forwarding the alerts with falcosidekick to other tools, the extra output_fields basically become 'labels' standardised across the organisation. E.g. alerts are forwarded as log lines to a log aggregation system. Each log line has a number of fixed labels so its origin can easily be found. The kubernetes cluster of origin is just one possibility, but one could add labels like division, project or team.

[jasondellaluce]

/milestone 0.34.0

[jasondellaluce]

/milestone 0.35.0

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

[sboschman]

Milestone is set to 0.35.0, I can confirm it is not working in 0.34.1

/remove-lifecycle stale

[FedeDP]

/milestone 0.36.0

[incertum]

Checking in here, a PR is up for #2127, need to check if this solves problem for plugins as well or if this is a slightly different request meaning the relevant code for plugins that supports this feature is elsewhere (haven't looked yet)? @jasondellaluce

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

[Andreagit97]

/remove-lifecycle stale