From 667ba9c86692b64a40f9a037f317cc622ec0bc99 Mon Sep 17 00:00:00 2001 From: JaseKoonce <122409379+JaseKoonce@users.noreply.github.com> Date: Thu, 9 Mar 2023 16:49:01 -0500 Subject: [PATCH] Bastion logging (#66) * added and tested managed node group Signed-off-by: jase koonce * rebasing Signed-off-by: jase koonce * pre-commit Signed-off-by: jase koonce * bastion logging wip Signed-off-by: jase koonce * adding eventbridge rules to aggregate logs Signed-off-by: jase koonce * tested/adjusted formating Signed-off-by: jase koonce * rebase Signed-off-by: jase koonce * WIP Signed-off-by: jase koonce * adds username logging/one concurrent session Signed-off-by: jase koonce * cleanup/precommit Signed-off-by: jase koonce * add parameters to example modules and update readmes (#84) * make optional * update readme * update env vars * unable to iterate over tuple if it doesn't exist * default null * parameterize instance type * refactor * add * parameterize instance type * parameterize instance type * update readme * parameterize * add parameters to example * add backend.tf.example * update lockfile thing * add backend.tf.example * DRYify admin username parameters * update variables * DRYify input vars * add -foce-copy * Remove variable, do not need defaults at the moment * added cluster_name * update readmes * updating self-managed/managed readmes Signed-off-by: jase koonce * fix some pah and var logic * update readme * ignore build dir * update readme * Empty commit * testing sign * testing sign again * testing sign again again * update readme * update env vars * unable to iterate over tuple if it doesn't exist * default null * parameterize instance type * refactor * add * parameterize instance type * parameterize instance type * update readme * Move the tfstate-backend module to its own repo (#77) * Update README.md (#78) * Use new remote tfstate-backend module (and delete the one in this repo) (#80) * eks output fix (#83) * parameterize * add parameters to example * add backend.tf.example * update lockfile thing * add backend.tf.example * DRYify admin username parameters * update variables * DRYify input vars * add -foce-copy * Remove variable, do not need defaults at the moment * added cluster_name * update readmes * fix some pah and var logic * update readme * update readme * updating self-managed/managed readmes Signed-off-by: jase koonce * Empty commit * testing sign * testing sign again * testing sign again again --------- Signed-off-by: jase koonce Co-authored-by: jase koonce Co-authored-by: Andy Roth Co-authored-by: Gabe <70963120+ntwkninja@users.noreply.github.com> * Update CODEOWNERS to use groups (#87) * Add GitHub Actions workflows for enabling test automation (#86) * change kc db output from endpoint to address (#92) * added and tested managed node group Signed-off-by: jase koonce * rebase Signed-off-by: jase koonce * Fixing cloudwatch error Signed-off-by: jase koonce * adjusting userdata to log out newest session Signed-off-by: jase koonce * Fixing failed checks Signed-off-by: jase koonce * bastion refactor Signed-off-by: jase koonce * adding newline for checks Signed-off-by: jase koonce * add checkov comments/adjust startup script behavior Signed-off-by: jase koonce * formating adjustment Signed-off-by: jase koonce --------- Signed-off-by: jase koonce Co-authored-by: Gabe <70963120+ntwkninja@users.noreply.github.com> Co-authored-by: brian.rexrode Co-authored-by: Zack A <24322023+zack-is-cool@users.noreply.github.com> Co-authored-by: Andy Roth Co-authored-by: brianrexrode <90282218+brianrexrode@users.noreply.github.com> --- modules/bastion/README.md | 9 +- modules/bastion/iam.tf | 57 ++++- modules/bastion/logging.tf | 171 +++++++++++++ modules/bastion/main.tf | 2 + modules/bastion/s3-buckets.tf | 75 +++++- modules/bastion/ssm.tf | 50 +--- modules/bastion/templates/user_data.sh.tpl | 281 +++++++++++++++++++++ 7 files changed, 593 insertions(+), 52 deletions(-) create mode 100644 modules/bastion/logging.tf diff --git a/modules/bastion/README.md b/modules/bastion/README.md index 74196f56..ebcaa4ab 100644 --- a/modules/bastion/README.md +++ b/modules/bastion/README.md @@ -31,7 +31,12 @@ No modules. | Name | Type | |------|------| +| [aws_cloudtrail.ssh-access](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudtrail) | resource | +| [aws_cloudwatch_event_rule.ssh-access](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_event_rule) | resource | +| [aws_cloudwatch_event_target.ssm-target](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_event_target) | resource | +| [aws_cloudwatch_log_group.ec2_cloudwatch_logs](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_log_group) | resource | | [aws_cloudwatch_log_group.session_manager_log_group](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_log_group) | resource | +| [aws_cloudwatch_log_group.ssh-access-log-group](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_log_group) | resource | | [aws_iam_instance_profile.bastion_ssm_profile](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_instance_profile) | resource | | [aws_iam_policy.custom](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_policy) | resource | | [aws_iam_policy.s3_logging_policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_policy) | resource | @@ -56,7 +61,6 @@ No modules. | [aws_network_interface_attachment.attach](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/network_interface_attachment) | resource | | [aws_s3_bucket.access_log_bucket](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket) | resource | | [aws_s3_bucket.session_logs_bucket](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket) | resource | -| [aws_s3_bucket_acl.access_log_bucket](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_acl) | resource | | [aws_s3_bucket_acl.session_logs_bucket](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_acl) | resource | | [aws_s3_bucket_lifecycle_configuration.access_log_bucket](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_lifecycle_configuration) | resource | | [aws_s3_bucket_lifecycle_configuration.session_logs_bucket](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_lifecycle_configuration) | resource | @@ -64,6 +68,7 @@ No modules. | [aws_s3_bucket_logging.session_logs_bucket](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_logging) | resource | | [aws_s3_bucket_notification.access_log_bucket_notification](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_notification) | resource | | [aws_s3_bucket_notification.session_logs_bucket_notification](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_notification) | resource | +| [aws_s3_bucket_policy.cloudwatch-s3-policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_policy) | resource | | [aws_s3_bucket_public_access_block.access_log_bucket](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_public_access_block) | resource | | [aws_s3_bucket_public_access_block.session_logs_bucket](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_public_access_block) | resource | | [aws_s3_bucket_server_side_encryption_configuration.access_log_bucket](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_server_side_encryption_configuration) | resource | @@ -73,10 +78,12 @@ No modules. | [aws_security_group.sg](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/security_group) | resource | | [aws_sqs_queue.queue](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sqs_queue) | resource | | [aws_ssm_document.session_manager_prefs](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ssm_document) | resource | +| [aws_ssm_parameter.cloudwatch_configuration_file](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ssm_parameter) | resource | | [tls_private_key.bastion_key](https://registry.terraform.io/providers/hashicorp/tls/latest/docs/resources/private_key) | resource | | [aws_ami.from_filter](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ami) | data source | | [aws_caller_identity.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/caller_identity) | data source | | [aws_iam_policy.AmazonSSMManagedInstanceCore](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy) | data source | +| [aws_iam_policy_document.cloudwatch-policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source | | [aws_iam_policy_document.kms_access](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source | | [aws_iam_policy_document.ssm_ec2_access](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source | | [aws_iam_policy_document.ssm_s3_cwl_access](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source | diff --git a/modules/bastion/iam.tf b/modules/bastion/iam.tf index 0b4b4edf..d74747ad 100644 --- a/modules/bastion/iam.tf +++ b/modules/bastion/iam.tf @@ -32,6 +32,7 @@ data "aws_iam_policy" "AmazonSSMManagedInstanceCore" { arn = "arn:${data.aws_partition.current.partition}:iam::aws:policy/AmazonSSMManagedInstanceCore" } + resource "aws_iam_role_policy_attachment" "bastion-ssm-amazon-policy-attach" { role = aws_iam_role.bastion_ssm_role.name policy_arn = data.aws_iam_policy.AmazonSSMManagedInstanceCore.arn @@ -129,7 +130,7 @@ data "aws_iam_policy_document" "ssm_ec2_access" { ] } } - +# Create a custom policy for the bastion and attachment resource "aws_iam_policy" "ssm_ec2_access" { name = "ssm-${var.name}-${var.aws_region}" path = "/" @@ -393,3 +394,57 @@ resource "aws_iam_policy" "terraform_policy" { } EOF } +# Create custom policy for KMS +data "aws_iam_policy_document" "kms_access" { + # checkov:skip=CKV_AWS_111: todo reduce perms on key + # checkov:skip=CKV_AWS_109: todo be more specific with resources + statement { + sid = "KMS Key Default" + principals { + type = "AWS" + identifiers = [ + "arn:${data.aws_partition.current.partition}:iam::${data.aws_caller_identity.current.account_id}:root" + ] + } + + actions = [ + "kms:*", + ] + + resources = ["*"] + } + statement { + sid = "CloudWatchLogsEncryption" + principals { + type = "Service" + identifiers = ["logs.${var.aws_region}.amazonaws.com"] + } + actions = [ + "kms:Encrypt*", + "kms:Decrypt*", + "kms:ReEncrypt*", + "kms:GenerateDataKey*", + "kms:Describe*", + ] + + resources = ["*"] + } + statement { + sid = "Cloudtrail KMS permissions" + principals { + type = "Service" + identifiers = [ + "cloudtrail.amazonaws.com" + ] + } + actions = [ + "kms:Encrypt*", + "kms:Decrypt*", + "kms:ReEncrypt*", + "kms:GenerateDataKey*", + "kms:Describe*", + ] + resources = ["*"] + } + +} diff --git a/modules/bastion/logging.tf b/modules/bastion/logging.tf new file mode 100644 index 00000000..c530627c --- /dev/null +++ b/modules/bastion/logging.tf @@ -0,0 +1,171 @@ +# Create a log group for ssh accesss +resource "aws_cloudwatch_log_group" "ssh-access-log-group" { + name = "/aws/events/ssh-access" + retention_in_days = 60 + kms_key_id = aws_kms_key.ssmkey.arn +} + +# Create a cloudtrail and event rule to monitor bastion access over ssh +resource "aws_cloudtrail" "ssh-access" { + # checkov:skip=CKV_AWS_252: SNS not currently needed + # checkov:skip=CKV2_AWS_10: Cloudwatch logs already being used with cloudtrail + name = "ssh-access" + s3_bucket_name = var.access_log_bucket_name + kms_key_id = aws_kms_key.ssmkey.arn + is_multi_region_trail = true + enable_log_file_validation = true + event_selector { + read_write_type = "All" + include_management_events = true + } + depends_on = [ + aws_s3_bucket_policy.cloudwatch-s3-policy, + aws_kms_key.ssmkey, + aws_cloudwatch_log_group.ssh-access-log-group + ] +} + +resource "aws_cloudwatch_event_rule" "ssh-access" { + name = "ssh-access" + description = "filters ssm access logs and sends usable data to a cloudwatch log group" + + event_pattern = < /etc/profile.d/startupscript.sh +#!/bin/bash + +{ + trap '' 2 #disable ctrl+c + + ###Script to check if exceeded maximum Session Manager Sessions and takes action + { + ###Configuration Options + MAX_SESSIONS=1 #Number of maximum sessions allowed + TERMINATE_SESSIONS=false #This will terminate the sessions starting from the oldest; if set to false, it will list out the sessions IDs, but not terminate them + TERMINATE_OLDEST=false #true/false - if true, script will terminate the oldest session first. if false, the newest session will be terminated. + #Terminating the newest session may result in poor experiance as there will be no message provided to the user. + + + ###Logic + MESSAGE="" #clears out message variable (mainly for debugging purposes in case script is run multiple times) + + ##Configure Reverse Logic + REVERSE_LOGIC='| reverse' + if [[ "$TERMINATE_OLDEST" = false ]] + then + REVERSE_LOGIC='' + fi + + ##Get Instance details and configure aws region + + EC2_INSTANCE_ID=$(TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600" ) +curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id ) + + REGION=$(TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600" ) +curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/placement/region ) + aws configure set default.region $REGION + + + ##Get All sessions for the instance and group by owner + + SESSION_INFO=$(aws ssm describe-sessions --state "Active" --filter "key=Target,value=$EC2_INSTANCE_ID" 2>&1) + + if [[ $? -gt 0 ]] #An error has occured + then + MESSAGE="An Error has occured; ExitCode: $?, Details: $SESSION_INFO" + else + SESSIONS=$(jq '.Sessions | group_by(.Owner)' <<< $SESSION_INFO) + SESSIONS_GROUP=$(jq 'length' <<< $SESSIONS) + + if [[ $SESSIONS_GROUP -gt 0 ]] + then + COUNTER=0 + MESSAGE_HEADER="Too many sessions found:" + while [ $COUNTER -lt $SESSIONS_GROUP ] + do + SESSION_COUNT=$(jq ".[$COUNTER] | length" <<< $SESSIONS) + if [ $SESSION_COUNT -gt $MAX_SESSIONS ] + then + SORTED=$(jq ".[$COUNTER] | sort_by(.StartDate) $REVERSE_LOGIC" <<< $SESSIONS) + while [ $SESSION_COUNT -gt $MAX_SESSIONS ] + do + TERMINATE_ROW=$(($SESSION_COUNT-1)) + TERMINATE_SESSION=$(jq -r ".[$TERMINATE_ROW].SessionId" <<< $SORTED) + + if [[ "$TERMINATE_SESSIONS" = true ]] + then + TERMINATOR=$(aws ssm terminate-session --session-id $TERMINATE_SESSION 2>&1) + echo "new line 233" + if [[ $? -gt 0 ]] #An error has occured + then + MESSAGE="An Error has occured; ExitCode: $?, Details: $TERMINATOR" + break 2 + fi + MESSAGE="$MESSAGE\n Terminated Session $TERMINATE_SESSION" + else + MESSAGE="$MESSAGE\n$TERMINATE_SESSION" + fi + + + SESSION_COUNT=$(($SESSION_COUNT-1)) + done + fi + COUNTER=$((COUNTER+1)) + done + if [[ ! -z "$MESSAGE" ]] + then + MESSAGE=$MESSAGE_HEADER$MESSAGE + fi + else + MESSAGE="No active sessions for this instance" + fi + fi + } + trap 2 #enable ctrl+c + clear && echo -e $MESSAGE + + +} + + + +_EOF_ +sudo chmod +x /etc/profile.d/startupscript.sh + +#Adjusting SSHD Config + +sudo cat << '_EOF_' > /etc/ssh/sshd_config +# $OpenBSD: sshd_config,v 1.100 2016/08/15 12:32:04 naddy Exp $ + +# This is the sshd server system-wide configuration file. See +# sshd_config(5) for more information. + +# This sshd was compiled with PATH=/usr/local/bin:/usr/bin + +# The strategy used for options in the default sshd_config shipped with +# OpenSSH is to specify options with their default value where +# possible, but leave them commented. Uncommented options override the +# default value. + +# If you want to change the port on a SELinux system, you have to tell +# SELinux about this change. +# semanage port -a -t ssh_port_t -p tcp #PORTNUMBER +# +#Port 22 +#AddressFamily any +#ListenAddress 0.0.0.0 +#ListenAddress :: + +HostKey /etc/ssh/ssh_host_rsa_key +#HostKey /etc/ssh/ssh_host_dsa_key +HostKey /etc/ssh/ssh_host_ecdsa_key +# HostKey /etc/ssh/ssh_host_ed25519_key + +# Ciphers and keying +#RekeyLimit default none + +# Logging +#SyslogFacility AUTH +SyslogFacility AUTHPRIV +#LogLevel INFO + +# Authentication: + +LoginGraceTime 15m +#PermitRootLogin yes +StrictModes yes +MaxAuthTries 3 +MaxSessions 1 +MaxStartups 1:100:100 +#PubkeyAuthentication yes + +# The default is to check both .ssh/authorized_keys and .ssh/authorized_keys2 +# but this is overridden so installations will only check .ssh/authorized_keys +AuthorizedKeysFile .ssh/authorized_keys + +#AuthorizedPrincipalsFile none + + +# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts +#HostbasedAuthentication no +# Change to yes if you don't trust ~/.ssh/known_hosts for +# HostbasedAuthentication +IgnoreUserKnownHosts yes +# Don't read the user's ~/.rhosts and ~/.shosts files +#IgnoreRhosts yes + +# To disable tunneled clear text passwords, change to no here! +PasswordAuthentication yes +#PermitEmptyPasswords no +#PasswordAuthentication no + +# Change to no to disable s/key passwords +#ChallengeResponseAuthentication yes +ChallengeResponseAuthentication no + +# Kerberos options +KerberosAuthentication no +#KerberosOrLocalPasswd yes +#KerberosTicketCleanup yes +#KerberosGetAFSToken no +#KerberosUseKuserok yes + +# GSSAPI options +GSSAPIAuthentication no +GSSAPICleanupCredentials no +#GSSAPIStrictAcceptorCheck yes +#GSSAPIKeyExchange no +#GSSAPIEnablek5users no + +# Set this to 'yes' to enable PAM authentication, account processing, +# and session processing. If this is enabled, PAM authentication will +# be allowed through the ChallengeResponseAuthentication and +# PasswordAuthentication. Depending on your PAM configuration, +# PAM authentication via ChallengeResponseAuthentication may bypass +# the setting of "PermitRootLogin without-password". +# If you just want the PAM account and session checks to run without +# PAM authentication, then enable this but set PasswordAuthentication +# and ChallengeResponseAuthentication to 'no'. +# WARNING: 'UsePAM no' is not supported in Red Hat Enterprise Linux and may cause several +# problems. +UsePAM yes + +#AllowAgentForwarding yes +#AllowTcpForwarding yes +#GatewayPorts no +X11Forwarding no +#X11DisplayOffset 10 +#X11UseLocalhost yes +#PermitTTY yes +#PrintMotd yes +PrintLastLog yes +#TCPKeepAlive yes +#UseLogin no +UsePrivilegeSeparation sandbox +#PermitUserEnvironment no +#Compression delayed +#ClientAliveInterval 0 +#ClientAliveCountMax 3 +#ShowPatchLevel no +#UseDNS yes +#PidFile /var/run/sshd.pid +#MaxStartups 10:30:100 +#PermitTunnel no +#ChrootDirectory none +#VersionAddendum none + +# no default banner path +#Banner none + +# Accept locale-related environment variables +AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES +AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT +AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE +AcceptEnv XMODIFIERS + +# override default of no subsystems +Subsystem sftp /usr/libexec/openssh/sftp-server + +# Example of overriding settings on a per-user basis +#Match User anoncvs +# X11Forwarding no +# AllowTcpForwarding no +# PermitTTY no +# ForceCommand cvs server + +AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys %u %f +AuthorizedKeysCommandUser ec2-instance-connect +Ciphers aes256-ctr,aes192-ctr,aes128-ctr +MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-256 +# RhostsRSAAuthentication no +Compression no +X11UseLocalhost yes +MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-256 +ClientAliveInterval 14400 +ClientAliveCountMax 0 + + +_EOF_ + +#Restart SSHD + +sudo service sshd restart