Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support cross-region replication #128

Closed
wants to merge 7 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# Unix-style newlines with a newline ending every file
[*]
charset = utf-8
end_of_line = lf
indent_size = 2
indent_style = space
insert_final_newline = true
trim_trailing_whitespace = true

[*.go]
indent_size = 2
indent_style = tab

[*.{tf,tfvars}]
indent_size = 2
indent_style = space

[*.md]
max_line_length = 0
trim_trailing_whitespace = false

# Override for Makefile
[{Makefile, makefile, GNUmakefile, Makefile.*}]
tab_width = 2
indent_style = tab
indent_size = 4

[COMMIT_EDITMSG]
max_line_length = 0
34 changes: 25 additions & 9 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,12 +1,28 @@
# Compiled files
# Local .terraform directories
**/.terraform
**/.terraform.d

# .tfstate files
*.tfstate
*.tfstate.backup
**/.terraform.lock.hcl
*.tfstate.*
.terraform
.terraform.tfstate.lock.info
.terraform.lock.hcl

**/.idea
**/*.iml
**/.vscode

# Cloud Posse Build Harness https://github.com/cloudposse/build-harness
**/.build-harness
**/build-harness

# Crash log files
crash.log
test.log

# Module directory
.terraform/
.idea
terraform-aws-tfstate-backend.iml
# Editor backups
*.orig
*.draft
*~

.build-harness
build-harness
2 changes: 1 addition & 1 deletion LICENSE
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,7 @@
same "printed page" as the copyright notice for easier
identification within third-party archives.

Copyright 2018-2023 Cloud Posse, LLC
Copyright 2020 Cloud Posse, LLC

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
Expand Down
117 changes: 53 additions & 64 deletions README.md

Large diffs are not rendered by default.

20 changes: 9 additions & 11 deletions README.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -50,22 +50,20 @@ related:

# Short description of this project
description: |-
Terraform module to provision an S3 bucket to store `terraform.tfstate` file and a DynamoDB table to lock the state file
to prevent concurrent modifications and state corruption.
Terraform module to provision an S3 backend to hold Terraform state.

The module supports the following:
The module supports the following options:

1. Forced server-side encryption at rest for the S3 bucket
2. S3 bucket versioning to allow for Terraform state recovery in the case of accidental deletions and human errors
3. State locking and consistency checking via DynamoDB table to prevent concurrent operations
4. DynamoDB server-side encryption

https://www.terraform.io/docs/backends/types/s3.html
1. Creation of DynamoDB table to provide consistency checking and state locking to prevent concurrent operations
2. Replication of state and locks to a second region, to ensure continued availability in the event of a region outage

The module enforces the following:

__NOTE:__ The operators of the module (IAM Users) must have permissions to create S3 buckets and DynamoDB tables when performing `terraform plan` and `terraform apply`
1. Forced server-side encryption at rest for the S3 bucket and DynamoDB table
2. S3 bucket versioning to allow for Terraform state recovery in the case of accidental deletions and human errors
3. Point-in-time recovery enabled for DynamoDB table

__NOTE:__ This module cannot be used to apply changes to the `mfa_delete` feature of the bucket. Changes regarding mfa_delete can only be made manually using the root credentials with MFA of the AWS Account where the bucket resides. Please see: https://github.com/terraform-providers/terraform-provider-aws/issues/62
https://www.terraform.io/docs/backends/types/s3.html

# How to use this project
usage: |-
Expand Down
93 changes: 93 additions & 0 deletions docs/migration-v0-v1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
# Migration Notes for Terraform State Backend v1

## Key changes in v1.0
- Support added for bi-directional replication of 2 S3 buckets and DynamoDB lock table.
This maintains availability of the Terraform state backend in the event of a region
outage, although you will need to manually reconfigure your Terraform backend to
target the secondary region.
- Because of the above, this module now requires 2 AWS providers: `aws.blue`
and `aws.green`, configured for the primary and secondary regions respectively.
If you do not want to enable replication and a second bucket, you still must
pass in providers for both colors, but you can use the same provider for both.
- When replication is enabled, an SNS topic is created to publish notifications
of replication events. You must subscribe to this topic to receive notifications.
- Server-side encryption is enabled for the S3 buckets and cannot be disabled.
There is therefore no longer a block against unencrypted uploads, since they
will be immediately encrypted anyway.
Comment on lines +15 to +16
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't this mean we're no longer enforcing the following?:

 "Condition": {
        "Bool": {
          "aws:SecureTransport": "false"
        }
      },

If so, some of the compliance checks are going to fail. This means that the object isn't required to be encrypted in transit.

- S3 ACLs are no longer set on the S3 buckets. Instead, the buckets are
configured with S3 Object Ownership set to `BucketOwnerEnforced`.
- All public access to the S3 bucket is blocked.
- DynamoDB table is billed in `PAY_PER_REQUEST` mode instead of `PROVISIONED`.
AWS highly recommends this mode to support replication. In practice, the
costs are negligible either way.
- Point-in-time recovery is enabled for the DynamoDB table and cannot be disabled.
- This module will no longer create an S3 bucket for S3 Server Access Logging.
If you want to enable logging, you must create a bucket and pass in the
logging configfuration in the `blue_bucket_logging` and `green_bucket_logging` inputs.
Note that you can also log access to the bucket via CloudTrail.
See [Logging options for Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/logging-with-S3.html) for details about the
differences.
- Support added for attaching a permission boundary to the IAM role that is
created for the S3 bucket replication.
- Many options have been removed and the rest have been renamed.
This module is now much simpler, and correspondingly more opinionated.
- Terraform version 1.3.0 or later required
- Terraform AWS Provider 4.0 or later required

## Migration Guide

Version 0.x of this module will remain available and we will consider pull requests
to update it, so you can continue to use it if you prefer. However, we recommend
upgrading to version 1.0, with the caveat that the migration can range from
easy to difficult.

### Easy

If you were already using encryption everywhere and not using replication
or logging, then the migration should be easy. Your new module will be
like this (if you are using Cloud Posse's "null-label" `context.tf`:

```hcl
module "tfstate" {
source = "cloudposse/tfstate-backend/aws"
version = "1.0.0"

providers = {
aws.blue = aws
aws.green = aws
}

replication_enabled = false

context = module.this.context
}
```

If you were not using `context.tf` then you would continue to provide
whatever inputs to [null-label](https://github.com/cloudposse/terraform-null-label/)
you were providing before.

The S3 bucket and DynamoDB table will automatically "moved" by
`terraform` to their new "Terraform addresses". If Terraform wants
to delete and recreate either of them because of name changes,
then you can use the `blue_s3_bucket_name` and `dynamodb_table_name`
inputs to force the names to be the same as before.

The Terraform plan will show something like this:

- DynamoDB table will be updated in place:
- `billing_mode` changed from `PROVISIONED` to `PAY_PER_REQUEST`
- `point_in_time_recovery` changed from `false` to `true`
- `aws_s3_bucket`, `aws_s3_bucket_policy`, and `aws_s3_bucket_public_access_block` will be moved to new Terraform addresses.
- `aws_s3_bucket_ownership_controls`, `aws_s3_bucket_server_side_encryption_configuration`,
and `aws_s3_bucket_versioning` will be created.

The conversion of the DynamoDB table from `PROVISIONED` to `PAY_PER_REQUEST`
may take over 10 minutes. Otherwise, these changes will be very quick.

The module no longer outputs a `terraform_backend_config` file, so that
you do not need to input things like profile and role ARN. Instead, the
module provides `blue_backend_config` as a map of the backend configuration
generated by the module. You can add profile, role ARN, workspace key
prefix, and any other settings you want to this map to generate a complete
backend configuration.
Loading