[filebeat][aws][cloudtrail] accessKeyId should be searchable

[nhnicwaller]

Describe the enhancement:

I'm using Filebeat with the aws module to ingest AWS CloudTrail logs into Elastic Cloud. This works mostly okay, except some of the most interesting pieces of data are not searchable. I am particularly interested in the AssumeRole event emitted by AWS CloudTrail (a full example is shown below). In this event, the field aws.cloudtrail.response_elements contains a stringified object, which unfortunately makes the contents opaque and impossible to search for in Kibana.

What would be super helpful is if credentials.accessKeyId and credentials.sessionToken were made searchable. Maybe this is possible by changing the tokenizer used on this field, but more likely it seems like it would require parsing this JSON and capturing those into new sub-fields.

Here is the line of code where the event body is stringified:

https://github.com/elastic/beats/blob/v7.7.0/x-pack/filebeat/module/aws/cloudtrail/ingest/pipeline.yml#L132

Describe a specific use case for the enhancement or feature:

I am attempting to use Elastic Cloud as a SIEM, and indeed Elastic is positioning itself as a SIEM offering. However, this gap in functionality effectively makes it impossible for me to use Elastic to track down the person that performed a specific action, which makes it substantially less useful as a SIEM.

---

Example AssumeRole event from aws.cloudtrail:

{
  "_id": "7UVlYnIBw0kYlt9zPYqS",
  "_index": "filebeat-7.6.2-2020.05.07-000001",
  "_score": 1,
  "_source": {
    "@timestamp": "2020-05-29T21:33:22.000Z",
    "agent": {
      "ephemeral_id": "0f5c9338-cf98-4145-8596-f87b6fd640f0",
      "hostname": "c23583bc9eac",
      "id": "87297c4e-bd47-4236-902d-5d4b0dfb5545",
      "type": "filebeat",
      "version": "7.6.2"
    },
    "aws": {
      "cloudtrail": {
        "event_type": "AwsApiCall",
        "event_version": "1.05",
        "recipient_account_id": "REDACTED",
        "request_parameters": "{durationSeconds=900, roleArn=arn:aws:iam::REDACTED:role/myrole, roleSessionName=1590788002139749100}",
        "response_elements": "{assumedRoleUser={arn=arn:aws:sts::REDACTED:assumed-role/myrole/1590788002139749100, assumedRoleId=AROACDRML13PHK3X7J1UL:1590788002139749100}, credentials={accessKeyId=ASIAREDACTED, sessionToken=REDACTED}}",
        "user_identity": {
          "access_key_id": "AKIAREDACTED",
          "arn": "arn:aws:iam::REDACTED:user/REDACTED@REDACTED",
          "type": "IAMUser"
        }
      },
      "s3": {
        "bucket": {
          "arn": "arn:aws:s3:::REDACTED-bucket",
          "name": "REDACTED-bucket"
        },
        "object.key": "AWSLogs/REDACTED/CloudTrail/us-east-1/2020/05/29/REDACTED_CloudTrail_us-east-1_20200529T2135Z_0p9TAh8rnLNQ28t0.json.gz"
      }
    },
    "cloud": {
      "account": {
        "id": "REDACTED"
      },
      "provider": "aws",
      "region": "ca-central-1"
    },
    "ecs": {
      "version": "1.4.0"
    },
    "event": {
      "action": "AssumeRole",
      "dataset": "aws.cloudtrail",
      "id": "fededed9-2b0a-4bbb-bd1b-40270e6a3a19",
      "kind": "event",
      "module": "aws",
      "original": "{\"awsRegion\":\"us-east-1\",\"eventID\":\"b4a0e081-42de-4118-b694-985167c867e4\",\"eventName\":\"AssumeRole\",\"eventSource\":\"sts.amazonaws.com\",\"eventTime\":\"2020-05-29T21:33:22Z\",\"eventType\":\"AwsApiCall\",\"eventVersion\":\"1.05\",\"recipientAccountId\":\"REDACTED\",\"requestID\":\"5c53445c-6665-4519-8ace-6c8f444c654a\",\"requestParameters\":{\"durationSeconds\":900,\"roleArn\":\"arn:aws:iam::REDACTED:role/myrole\",\"roleSessionName\":\"1590788002139749100\"},\"resources\":[{\"ARN\":\"arn:aws:iam::REDACTED:role/myrole\",\"accountId\":\"REDACTED\",\"type\":\"AWS::IAM::Role\"}],\"responseElements\":{\"assumedRoleUser\":{\"arn\":\"arn:aws:sts::REDACTED:assumed-role/myrole/1590788002139749100\",\"assumedRoleId\":\"AROACDRML13PHK3X7J1UL:1590788002139749100\"},\"credentials\":{\"accessKeyId\":\"ASIAREDACTED\",\"expiration\":\"May 29, 2020 9:48:22 PM\",\"sessionToken\":\"FwoGZXIvYXdzEK///////////wEaDM6q1RJMI+laZ3P+1yK3AaSVw/zm5hdBIsnYUgebG5oCISqrQJ+/X8rbqwuUj71MWgBf1vv9nDzv39QhMskyCdDCEsTaXKrpblVxmpOSPotfLYwSgYzY+PIiOnBpZPd7mVQchhNdwqAO8iANQS8Aly7ypUUEA59Wpp2AY+RiEVVMFYeXPpqWTquCUoToSwY/KlhuJeawVnTAeNNrfipaEBolBPLqo2CEe3Uq6WMBekcByWdfZM3bZC7qTaYhzdctCfsIg7+5Myii/8X2BTItWLnqFcxPBxORuDp/9l1/2kREqZmESvJtABso9VtPCJ0gH6oCig0g/65Iz6qk\"}},\"sourceIPAddress\":\"207.6.233.58\",\"userAgent\":\"aws-sdk-go/1.25.38 (go1.13.4; linux; amd64)\",\"userIdentity\":{\"accessKeyId\":\"AKIAREDACTED\",\"accountId\":\"REDACTED\",\"arn\":\"arn:aws:iam::REDACTED:user/REDACTED@REDACTED\",\"principalId\":\"AIDAREDACTED\",\"type\":\"IAMUser\",\"userName\":\"REDACTED@REDACTED\"}}",
      "outcome": "success",
      "provider": "sts.amazonaws.com",
      "type": "info"
    },
    "fileset": {
      "name": "cloudtrail"
    },
    "host": {
      "name": "c23583bc9eac"
    },
    "input": {
      "type": "s3"
    },
    "log": {
      "file.path": "https://REDACTED-bucket.s3-ca-central-1.amazonaws.com/AWSLogs/REDACTED/CloudTrail/us-east-1/2020/05/29/REDACTED_CloudTrail_us-east-1_20200529T2135Z_0p9TAh8rnLNQ28t0.json.gz",
      "offset": 1593
    },
    "service": {
      "type": "aws"
    },
    "source": {
      "address": "555.555.555.555",
      "geo": {
        "city_name": "Vancouver",
        "continent_name": "North America",
        "country_iso_code": "CA",
        "location": {
          "lat": 49.2824,
          "lon": -123.0399
        },
        "region_iso_code": "CA-BC",
        "region_name": "British Columbia"
      }
    },
    "user": {
      "id": "AIDAREDACTED",
      "name": "REDACTED@REDACTED"
    },
    "user_agent": {
      "device": {
        "name": "Other"
      },
      "name": "aws-sdk-go",
      "original": "aws-sdk-go/1.25.38 (go1.13.4; linux; amd64)",
      "version": "1.25.38"
    }
  },
  "_type": "_doc",
  "fields": {
    "@timestamp": [
      "2020-05-29T21:33:22.000Z"
    ],
    "suricata.eve.timestamp": [
      "2020-05-29T21:33:22.000Z"
    ]
  }
}

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[andrewkroh]

it seems like it would require parsing this JSON and capturing those into new sub-fields.

👍 That's probably the best approach.

It might also be helpful to use a multi-field mapping (keyword+text) on the aws.cloudtrail.response_elements field to allow searches.

[leehinman]

Coded up 3 options

• [Filebeat] change field types in aws cloudtrail fileset #18909 changes field type to object, this relies on dynamic mappings and runs the risk of a mapping explosion. The 30 API calls we have in the test suite, resulted in 78 new mappings.
• [Filebeat] Change field type in aws cloudtrail fileset to flattened #19020 changes field type to flattened, the objects are mapped to a single field, but can be searched like keywords flattened doc
• [Filebeat] add multi_fields in aws cloudtrail fileset #19021 Adds a text multi_field for the fields in question, makes searching easier

[nhnicwaller]

The option put forth in #19020 would work best in my scenario because I only expect to need keyword-style searches.

I was also concerned about mapping explosion, so I ran some tests on logs in my environment and came to a similar conclusion about magnitude (about 70-70 new mappings). However a more serious concern is that in some other places I've seen AWS logs emit different types for the same field name (I think from EventBridge, not GuardDuty) so the first option would carry an additional risk of potential data loss, especially as AWS continues to evolve the schemas of their emitted logs.

[leehinman]

New PR #19121 up that has both flattened & text multi_field. This will be backwards compatible but should offer the most flexibility.