S3 server access logs are inherently incomplete

[nadove-ucsc]

https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerLogs.html#LogDeliveryBestEffort

The completeness and timeliness of server logging is not guaranteed. The log record for a particular request might be delivered long after the request was actually processed, or it might not be delivered at all.

• Security design review completed; the Resolution of this issue does not …
  • … affect authentication; for example:
    • OAuth 2.0 with the application (API or Swagger UI)
    • Authentication of developers with Google Cloud APIs
    • Authentication of developers with AWS APIs
    • Authentication with a GitLab instance in the system
    • Password and 2FA authentication with GitHub
    • API access token authentication with GitHub
    • Authentication with
  • … affect the permissions of internal users like access to
    • Cloud resources on AWS and GCP
    • GitLab repositories, projects and groups, administration
    • an EC2 instance via SSH
    • GitHub issues, pull requests, commits, commit statuses, wikis, repositories, organizations
  • … affect the permissions of external users like access to
    • TDR snapshots
  • … affect permissions of service or bot accounts
    • Cloud resources on AWS and GCP
  • … affect audit logging in the system, like
    • adding, removing or changing a log message that represents an auditable event
    • changing the routing of log messages through the system
  • … affect monitoring of the system
  • … introduce a new software dependency like
    • Python packages on PYPI
    • Command-line utilities
    • Docker images
    • Terraform providers
  • … add an interface that exposes sensitive or confidential data at the security boundary
  • … affect the encryption of data at rest
  • … require persistence of sensitive or confidential data that might require encryption at rest
  • … require unencrypted transmission of data within the security boundary
  • … affect the network security layer; for example by
    • modifying, adding or removing firewall rules
    • modifying, adding or removing security groups
    • changing or adding a port a service, proxy or load balancer listens on
• Documentation on any unchecked boxes is provided in comments below

[dsotirho-ucsc]

@hannes-ucsc to fill in details.

[hannes-ucsc]

The two logging options are 1) CloudTrail data event logging and 2) S3 server access logs. We currently use the latter and will make them searchable by forwarding them to CloudWatch.

A comparison of the two options can be found here: https://docs.aws.amazon.com/AmazonS3/latest/userguide/logging-with-S3.html

For compliance reasons (scoping of information spillage) we need the ability to definitively say that an object was NOT accessed. Due to the best effort implementation of S3 server access logs, the absence of a log record does not necessarily mean the absence of a request. The CloudTrail logging option does not come with that "best-effort" caveat. On the other hand, CloudTrail does not log authentication failures and the log does not contain certain fields. It is more expensive. We should implement both logging options.

Enable data event logging for all buckets by specifying an event selector on the trail. See Logging All S3 Object Events By Using Basic Event Selectors on https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudtrail

[dsotirho-ucsc]

@hannes-ucsc: "Ideally we'd want all events in one trail, that is all existing events that are currently logged plus the S3 data events. If that is not possible, we should create a second trail just for S3 data events, but try to send both trails to the same log group."

[hannes-ucsc]

For demo, show that both management and data events are logged concurrently in the shared trail in dev. For extra credit, pick a data event from the trail and find the corresponding event from the forwarded S3 server access logs. Compare the events.

[dsotirho-ucsc]

@hannes-ucsc to provide design to prevent excessive Access Denied notifications while still satisfying the SecurityHub rule that requires that specific alarm.

[hannes-ucsc]

Modify the api_unauthorized filter with a refined pattern that excludes the noisy events originating from CloudFront. Below I've included an example of an event that should be excluded. The exclusion pattern should target the eventSource, eventName and userIdentity.invokedBy fields of the event.

This changes the pattern from what the SecurityHub best-practices rules expect, and will result in a finding. We'll address that finding again in #5255.

{
  "@timestamp": "2023-05-24 18:03:30.575",
  "@message": {
    "eventVersion": "1.08",
    "userIdentity": {
      "type": "AWSService",
      "invokedBy": "cloudfront.amazonaws.com"
    },
    "eventTime": "2023-05-24T18:02:27Z",
    "eventSource": "s3.amazonaws.com",
    "eventName": "GetObject",
    "awsRegion": "us-east-1",
    "sourceIPAddress": "cloudfront.amazonaws.com",
    "userAgent": "cloudfront.amazonaws.com",
    "errorCode": "AccessDenied",
    "errorMessage": "Access Denied",
    "requestParameters": {
      "bucketName": "anvil.gi.ucsc.edu",
      "Host": "anvil.gi.ucsc.edu.s3.us-east-1.amazonaws.com",
      "key": "apidocs/swagger.json"
    },
    "responseElements": null,
    "additionalEventData": {
      "SignatureVersion": "SigV4",
      "CipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
      "bytesTransferredIn": 0,
      "AuthenticationMethod": "AuthHeader",
      "x-amz-id-2": "EIZYDF+R3WH7fai/lDigrFG5u7otASV0h3jEzCoPnWGMeD08Oo0PHOWouwSBxAswyJqok7kqurs=",
      "bytesTransferredOut": 243
    },
    "requestID": "DN11SE4RA2MT18Q3",
    "eventID": "b935d261-f9ea-407e-b61b-ec61d606d89e",
    "readOnly": true,
    "resources": [
      {
        "type": "AWS::S3::Object",
        "ARN": "arn:aws:s3:::anvil.gi.ucsc.edu/apidocs/swagger.json"
      },
      {
        "accountId": "289950828509",
        "type": "AWS::S3::Bucket",
        "ARN": "arn:aws:s3:::anvil.gi.ucsc.edu"
      }
    ],
    "eventType": "AwsApiCall",
    "managementEvent": false,
    "recipientAccountId": "289950828509",
    "sharedEventID": "778e1726-50c7-4704-9047-fdb41fa24e82",
    "eventCategory": "Data"
  }
}