Review audit log retention

[nolunwa-ucsc]

What is the Retention Period for the following Audit Capabilities that is enabled:

• Cloudwatch
• cloudtrail
• S3
• SecurityHub
• Guardduty
• AWS Config
• ElasticSearch
• VPN
• GitLab instance
• GitLab job logs
• and ?

[nolunwa-ucsc]

CloudWatch logs retention is currently set to 180 days for most logs

azul/src/azul/init.py

Line 257 in d50be3b

audit_log_retention_days = 180 # FedRAMP mandates 90 days

and 30 days for ElasticSearch error logs
azul/src/azul/init.py

Line 259 in d50be3b

verbose_log_retention_days = 30

[edit @danielsotirhos]
From: #4043 (comment)

[dsotirho-ucsc]

CloudWatch

https://aws.amazon.com/cloudwatch/faqs/

By default, CloudWatch Logs will store your log data indefinitely.

Terraform resource

aws_cloudwatch_log_group.retention_in_days

Production values

$ aws logs describe-log-groups | grep -e logGroupName -e retentionInDays

/aws/aes/domains/{domain}/index-logs    180 days (config.audit_log_retention_days)
/aws/aes/domains/{domain}/search-logs   180 days (config.audit_log_retention_days)
/aws/aes/domains/{domain}/error-logs    30 days (config.verbose_log_retention_days)
/aws/apigateway/azul-indexer-prod       180 days (config.audit_log_retention_days)
/aws/apigateway/azul-service-prod       180 days (config.audit_log_retention_days)
/aws/vpc/azul-default-prod              180 days (config.audit_log_retention_days)
/aws/vpc/azul-gitlab                    180 days (config.audit_log_retention_days)
/aws/vpn/azul-gitlab                    180 days (config.audit_log_retention_days)
azul-trail-prod                         180 days (config.audit_log_retention_days)

[Edit (@hannes-ucsc): Determine if error log retention can be extended to 180 days, if the error log is currently populated and what the expected size of 180 days worth of logging is.]

[Edit (@dsotirho-ucsc): Response in comment below]

[Edit (@dsotirho-ucsc): Created ticket #5557 PR #5558]

---

CloudTrail

https://docs.aws.amazon.com/awscloudtrail/latest/userguide/best-practices-security.html

The CloudTrail trail default is to store log files indefinitely in the Amazon S3 bucket configured for the trail. You can use the Amazon S3 object lifecycle management rules to define your own retention policy to better meet your business and auditing needs

Terraform resource

aws_s3_bucket_lifecycle_configuration

Production values

(no expiration lifecycle policy set on trail bucket)

$ aws s3api get-bucket-lifecycle --bucket edu-ucsc-gi-platform-hca-prod-trail.us-east-1

{
    "Rules": [
        {
            "ID": "mandatory_tag_enforcer_lifecycle_s3",
            "Status": "Enabled",
            "Transition": {
                "Days": 1,
                "StorageClass": "INTELLIGENT_TIERING"
            }
        }
    ]
}

[Edit (@hannes-ucsc): We don't want this storage class transition. We disabled the tag enforcer. Determine what needs to be done in each of our AWS accounts to remove that rule or post evidence that the rule is gone from all buckets in all accounts.]

[Edit (@dsotirho-ucsc): There are 16 buckets with the mandatory_tag_enforcer_lifecycle_s3 rule, details posted in comment below. Ticket #5136 removed the rule custodian-mandatory_tag_enforcer_lifecycle_s3, however mandatory_tag_enforcer_lifecycle_s3 is not mentioned. The rule can be removed via the AWS Console.]

[Edit (@dsotirho-ucsc): Created ticket #5561 to remove the rule from these buckets by explicitly setting no rule in the TF config.]

---

S3

https://stackoverflow.com/questions/26891607/s3-default-lifecycle-expiration

By default, an object on S3 doesn't has a expiration date. If you don't specify a lifecycle rule, it will be there forever.

Terraform resource

aws_s3_bucket_lifecycle_configuration

Production values

$ aws s3api get-bucket-lifecycle-configuration --bucket edu-ucsc-gi-platform-hca-prod-storage-prod.us-east-1
$ aws s3api get-bucket-lifecycle-configuration --bucket edu-ucsc-gi-platform-hca-prod-shared.us-east-1
$ aws s3api get-bucket-lifecycle-configuration --bucket edu-ucsc-gi-platform-hca-prod-logs.us-east-1

edu-ucsc-gi-platform-hca-prod-storage-prod.us-east-1
- Expire `manifests/` prefix objects after 7 days (config.manifest_expiration)

edu-ucsc-gi-platform-hca-prod-shared.us-east-1
- Expire noncurrent object versions after 30 days (hard coded in TF config)

edu-ucsc-gi-platform-hca-prod-logs.us-east-1
- Expire objects 90 days after upload (hard coded in TF config)

[Edit (@hannes-ucsc): File PR to increase retention of non-current object versions in shared bucket to 90 days]

[Edit (@dsotirho-ucsc): Created ticket #5552 PR #5553]

---

SecurityHub

https://docs.aws.amazon.com/securityhub/latest/userguide/securityhub_limits.html

Findings are deleted 90 days after the most recent update or 90 days after the creation date if no update occurs.
To store findings for longer than 90 days, you can configure a rule in EventBridge that routes findings to your Amazon S3 bucket.

Terraform resource

aws_cloudwatch_event_rule

Production values

$ aws events list-rules | grep -i security

(no EventBridge rule to route findings to an S3 bucket)

---

GuardDuty

https://docs.aws.amazon.com/guardduty/latest/ug/guardduty_limits.html

Finding retention period = 90 days
You cannot request a quota increase.

---

AWS Config

https://docs.aws.amazon.com/config/latest/developerguide/delete-config-data-with-retention-period.html

In the AWS Management Console, if you do not select a data retention period, the default period is 7 years or 2557 days.

Terraform resource

aws_config_configuration_recorder^*

^*Not currently possibe to set a retention_period via aws_config_configuration_recorder
https://www.github.com/hashicorp/terraform-provider-aws/issues/13305

Production values

$ aws configservice describe-retention-configurations

{
    "RetentionConfigurations": []
}

---

Elasticsearch

See CloudWatch section for retention of log groups /aws/aes/domains/…

---

VPN

See CloudWatch section for retention of log group /aws/vpn/azul-gitlab

---

GitLab logs

(This information pertains to logs stored on the GitLab instance & will need to be updated once #3894 is implemented to forward GitLab logs to Cloudwatch)

List of logs managed by logrotate vs svlogd/runit:
https://docs.gitlab.com/ee/administration/logs/#log-rotation

Log retention

The default values for logging are being used.

GitLab config file:
/etc/gitlab/gitlab.rb

Logging settings default values:
https://docs.gitlab.com/omnibus/settings/logs.html#runit-logs
https://docs.gitlab.com/omnibus/settings/logs.html#logrotate

logging['svlogd_size']      = 200 * 1024 * 1024 # rotate after 200 MB of log data
logging['svlogd_num']       = 30 # keep 30 rotated log files
logging['svlogd_timeout']   = 24 * 60 * 60 # rotate after 24 hours
logging['svlogd_filter']    = "gzip" # compress logs with gzip
logging['svlogd_udp']       = nil # transmit log messages via UDP
logging['svlogd_prefix']    = nil # custom prefix for log messages

logging['logrotate_frequency']  = "daily" # rotate logs daily
logging['logrotate_maxsize']    = nil # logs will be rotated when they grow bigger than size specified for `maxsize`, even before the specified time interval (daily, weekly, monthly, or yearly)
logging['logrotate_size']       = nil # do not rotate by size by default
logging['logrotate_rotate']     = 30 # keep 30 rotated logs
logging['logrotate_compress']   = "compress" # see 'man logrotate'
logging['logrotate_method']     = "copytruncate" # see 'man logrotate'
logging['logrotate_postrotate'] = nil # no postrotate command by default
logging['logrotate_dateformat'] = nil # use date extensions for rotated files rather than numbers e.g. a value of "-%Y-%m-%d" would give rotated files like production.log-2016-03-09.gz

[Edit (@hannes-ucsc): I believe that all these logs are now forwarded to CloudWatch. Please confirm this and link to the relevant ticket(s) and PR(s).]

[Edit (@dsotirho-ucsc): Relevant ticket #3894 PR #5187 , comment below lists the GitLab host logs that are now forwarded to CloudWatch.]

---

GitLab job logs

https://docs.gitlab.com/ee/administration/job_logs.html

Log location

When a job is running:
/var/opt/gitlab/gitlab-ci/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log

After a job is finished:
/var/opt/gitlab/gitlab-rails/shared/artifacts/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log

Log retention

https://docs.gitlab.com/ee/administration/job_logs.html#how-to-remove-job-logs

There isn’t a way to automatically expire old job logs, but it’s safe to remove them if they’re taking up too much space.

[Edit (@hannes-ucsc): We are OK with retaining job logs in perpetuity until we run into space issues.]

[hannes-ucsc]

I added edits with action items directly to @dsotirho-ucsc's comment above.

[Edit (@dsotirho-ucsc): Response to action items added as inline edits in comment above.]

[dsotirho-ucsc]

CloudWatch

Determine if error log retention can be extended to 180 days…

Yes, there are three log types (INDEX_SLOW_LOGS, SEARCH_SLOW_LOGS, & ES_APPLICATION_LOGS) published from ES to their own CloudWatch log group. While we currently set the retention of the ES_APPLICATION_LOGS (error-logs) to 30 days, the other two log types have a retention set of 180 days.

…if the error log is currently populated…

/aws/aes/domains/azul-index-dev/error-logs - 10 records
/aws/aes/domains/azul-index-prod/error-logs - 7 records
/aws/aes/domains/azul-index-anvildev/error-logs - 0 records
/aws/aes/domains/azul-index-anvilprod/error-logs - 2 records

and what the expected size of 180 days worth of logging is.

Stored bytes:

/aws/aes/domains/azul-index-dev/error-logs: 11292       * 6 months = 67,752
/aws/aes/domains/azul-index-prod/error-logs: 1849       * 6 months = 11,094
/aws/aes/domains/azul-index-anvildev/error-logs: 0      * 6 months = 0
/aws/aes/domains/azul-index-anvilprod/error-logs: 448   * 6 months = 2,688

Code used to fetch the stored bytes

from azul.deployment import aws
client = aws.client('logs')
paginator = client.get_paginator('describe_log_groups')
for page in paginator.paginate():
    for group in page['logGroups']:
        print(f"{group['logGroupName']}: {group['storedBytes']}")

Note:

• The metric IncomingBytes shows a spike of 107MB in azul-index-prod/error-logs on 2023-01-23. (Metric query link)
• The IncomingBytes is the uncompressed value while the actual stored bytes will be compressed.

[dsotirho-ucsc]

GitLab logs

GitLab host logs forwarded to CloudWatch Agent:

/var/log/amazon/ssm/amazon-ssm-agent.log
/var/log/audit/audit.log
/var/log/cloud-init.log
/var/log/cron
/var/log/maillog
/var/log/messages
/var/log/secure
/mnt/gitlab/logs/gitaly/gitaly_ruby_json.log
/mnt/gitlab/logs/gitlab-shell/gitlab-shell.log
/mnt/gitlab/logs/nginx/gitlab_access.log
/mnt/gitlab/logs/nginx/gitlab_error.log
/mnt/gitlab/logs/nginx/gitlab_registry_access.log
/mnt/gitlab/logs/puma/puma_stderr.log
/mnt/gitlab/logs/puma/puma_stdout.log
/mnt/gitlab/logs/reconfigure/*.log
/mnt/gitlab/logs/gitlab-rails/api_json.log
/mnt/gitlab/logs/gitlab-rails/application_json.log
/mnt/gitlab/logs/gitlab-rails/application.log
/mnt/gitlab/logs/gitlab-rails/audit_json.log
/mnt/gitlab/logs/gitlab-rails/auth.log
/mnt/gitlab/logs/gitlab-rails/database_load_balancing.log
/mnt/gitlab/logs/gitlab-rails/exceptions_json.log
/mnt/gitlab/logs/gitlab-rails/graphql_json.log
/mnt/gitlab/logs/gitlab-rails/migrations.log
/mnt/gitlab/logs/gitlab-rails/production_json.log
/mnt/gitlab/logs/gitlab-rails/production.log
/mnt/gitlab/logs/gitlab-rails/sidekiq_client.log

[dsotirho-ucsc]

The following buckets currently have the mandatory_tag_enforcer_lifecycle_s3 lifecycle rule.

platform-hca-dev

• data-browser.dev.lungmap.net
• dev.archive-preview.singlecell.gi.ucsc.edu
• dev.explore.lungmap.net
• dev.explore.singlecell.gi.ucsc.edu
• dev.singlecell.gi.ucsc.edu
• edu-ucsc-gi-platform-hca-dev-awsconfig.us-east-1
• edu-ucsc-gi-platform-hca-dev-trail.us-east-1

platform-hca-prod

• archive-preview.humancellatlas.org
• data-browser.explore.lungmap.net
• data-browser.lungmap.net
• edu-ucsc-gi-platform-hca-prod-awsconfig.us-east-1
• edu-ucsc-gi-platform-hca-prod-trail.us-east-1
• org-humancellatlas-data-browser-dcp2-prod
• org-humancellatlas-data-portal-dcp2-prod

platform-anvil-dev

• edu-ucsc-gi-platform-anvil-dev-awsconfig.us-east-1
• edu-ucsc-gi-platform-anvil-dev-trail.us-east-1

platform-anvil-prod

• (none)

[achave11-ucsc]

@hannes-ucsc: "@dsotirho-ucsc comments above list the retention for the various logs in the system. Any necessary changes to the retention are tracked in dedicated tickets. These tickets are linked from the comments. @nolunwa-ucsc to review and decide on next steps."

[nolunwa-ucsc]

with NIST SP 800 53 Rev 5, retain audit records for a time period in compliance with M-21-31

12 month active storage and 18 month cold data storage: AWS Cloudtrail Log• Amazon Cloudwatch Logs • AWS Config • Amazon S3 Access Logs • Amazon VPC Flow Logs • AWS WAF Logs • AWS Shield • AWS Guardduty • AWS Security Hub

[hannes-ucsc]

Executive orders apply to agencies in the federal government. Why would this be relevant for us?

[nolunwa-ucsc]

FedRAMP SSP rev 5 Template for Moderate system says

AU-11 Audit Record Retention (L)(M)(H)
Retain audit records for [FedRAMP Assignment: a time period in compliance with M-21-31] to provide support for after-the-fact investigations of incidents and to meet regulatory and organizational information retention requirements.

  	AU-11 Additional FedRAMP Requirements and Guidance:
  	Guidance: The service provider is encouraged to align with M-21-31 where possible

Requirement: The service provider retains audit records on-line for at least ninety days and further preserves audit records off-line for a period that is in accordance with NARA requirements.
Requirement: The service provider must support Agency requirements to comply with M-21-31 (https://www.whitehouse.gov/wp-content/uploads/2021/08/M-21-31-Improving-the-Federal-Governments-Investigative-and-Remediation-Capabilities-Related-to-Cybersecurity-Incidents.pdf).

[dsotirho-ucsc]

Assignee to come up with design for increasing retention in compliance with that executive order.

[hannes-ucsc]

Implementation of that executive order is tracked in #6070.