Transition Micromasters from Heroku Managed Postgres to Pulumi Manage…

…d RDS (#1380)

#1090 - Code to transition bespoke Heroku managed database instances and associated s3 buckets under Pulumi management.

docs/runbooks/transition_heroku_postgres_db_to_rds.md

@@ -0,0 +1,151 @@
+# Moving a Heroku Managed Postgres DB to Pulumi Managed AWS RDS
+
+## Preparation
+
+You will need to gather some information about the Heroku managed application and its database before you start. You'll need the
+[Heroku CLI](https://devcenter.heroku.com/articles/heroku-cli) with auehenticated access to the application in question to continue.
+
+You'll also need the postgresql tools, specifically the `pg_dump` and `psql` tools.
+
+Record the following information somewhere persistent that you'll be able to refer back to through this process. I like using my [notes](https://joplinapp.org/).
+
+- The Heroku application's name. Our convention is generally <application>-<environment> so for the CI environment of the micromasters
+application, you'd use `micromasters-ci`.
+- The applications's `DATABASE_URL`. You can obtain this with the following invocation: `heroku config:get DATABASE_URL -a <your app>`
+- The currently attached Heroku database name as well as the alias they have assigned to it by default. You can get this with:
+`heroku addons -a <application> | grep -i postgres` for example:
+```
+╰ ➤ heroku addons -a micromasters-ci | grep -i postgres
+heroku-postgresql (postgresql-rigid-71273)  mini   $5/month   created
+ └─ as HEROKU_POSTGRES_YELLOW
+```
+so in this case we see that postgresql-rigid-71273 is attached as HEROKU_POSTGRES_YELLOW
+- A set of URLs to browse when the transition is done to ensure everything is working properly. You should also browse them before the transition to note how
+everything looks.
+
+## Building the Infrastructure with Pulumi
+
+<!-- TODO: My branch might vanish. Change this to a CR when one exists. -->
+Describing in detail how to code the necessary resources to build the AWS RDS Postgres instance and associated S3 bucket, IAM rules, VPC peerings
+etc. is beyond the scope of this ducument. You can see what I did in my [Github branch](https://github.com/mitodl/ol-infrastructure/tree/cpatti_micromasters_pulumi).
+
+Once the infrastructure is properly built, you'll need to record the new AWS RDS Postgres instance's endpoint. You can do this from within
+the directory for the application you're working on. For my current project, that's `ol-infrastructure/src/ol_infrastructure/applications/micromasters`
+with this: 'poetry run pulumi stack export -s applications.micromasters.CI | grep -i endpoint' but obviously sub your app in for micromasters.
+
+You'll also need to retrieve the database password from Vault using Pulumi. You can do that with the following invocation:
+`poetry run pulumi config get  "micromasters:db_password"`
+
+## Dump Heroku Managed DB
+
+Use something like the following invocation to dump the contents of the current application database.
+
+`pg_dump -x -O $(heroku config:get DATABASE_URL -a micromasters-rc) > micromasters_qa_db_dump.sql`
+
+Obviously, substitute your app for micromasters and your environment for rc/qa.
+
+(Aside: We use rc and qa interchangably here).
+
+Examine the dump in your editor (read-only to be safe) and ensure that all the necessary components are present: Schema, data, foreign keys, and the like.
+
+## Construct A New DATABASE_URL
+
+I suggest doing this in a text file you can source easily since you'll be working with this database a bit for this project. I keep such things in an 'envsnips' folder
+in my home directory.
+
+The file should look something like:
+```
+export DATABASE_URL=postgresql://oldevops:<password you pulled from Pulumi config>@micromasters-ci-app-db.cbnm7ajau6mi.us-east-1.rds.amazonaws.com:5432/micromasters
+```
+
+Make sure the URL has the following components:
+
+- `postgresql://` is the protocol identifier followed by a :.
+- 'oldevops' is the database user, then another :.
+- the database password we pulled from Pulumi above, followed by an @ sign.
+- The endpoint hostname we retrieved from Pulumi earlier, followed by a :.
+- The port number. We usually use 5432. Then a /.
+- The database name.
+
+If your URL is missing any of these it will not work. Once you've finished write out your file and source it in your shell.
+
+Now, test that you can connect using the URL you just built with:
+
+`psql $DATABASE_URL`
+
+If you get an access denied message, make sure you got the correct password for the app and environment (e.g. CI, QA or production) and check the
+other components.
+
+We'll assume $DATABASE_URL is set to to the new RDS database we've created for the rest of the runbook.
+
+## Restore Dump Into AWS RDS DB
+
+Using the DATABASE_URL we just created and tested, we can now restore the data we dumped in the prior step into the new DB:
+
+`psql $DATABASE_URL < micromasters_qa_db_dump.sql`
+
+You will see a lot of output representing each statement as it's processed by the DB. You shouldn't see any errors here.
+
+## Coordinate Transition
+
+In the process of changing the database out from under a running application, there will be some small period of down time, so it's important to coordinate with
+all the appropriate stakeholders and leadership before you do.
+
+## Perform The Final Transition
+
+At the time, it's important that you perform the following steps quickly in succession, because once you detach the current DB, the application will be down.
+Keep this as brief as possible.
+
+You may wish to cue up the commands you want to run in a text file somewhere you can eaily review them, and then cut and paste them into your shell when the
+time comes.
+
+### Create An Additional DB Attachment
+
+You'll need to create an additional attachment for the current DB:
+`heroku addons:attach postgresql-amorphous-36035 --as HEROKU_POSTGRES_DB`
+
+Substitute your db instance you gathered above. HEROKU_POSTGRES_DB is just an alias we can use if we should need to roll back.
+
+### Detach The Current Database
+
+This is where you'll need to use the Heroku managed database instance above, along with the Heroku application name we collected. Substitute accordingly
+into the following invocation:
+
+`heroku addons:detach postgresql-amorphous-36035 -a micromasters-rc`
+
+### Change the DATABASE_URL to the New RDS Instance
+
+Ensuring that your DATABASE_URL environment variable is properly set to your new RDS from the above steps, use it to set DATABASE_URL in the heroku app:
+
+`heroku config:set -a micromasters-rc DATABASE_URL=$DATABASE_URL`
+
+Now immediately print out the value you just set to ensure that all looks good:
+`heroku config:get -a micromasters-rc DATABASE_URL`
+
+## Test Your Work
+
+You should carefully test the application you just transitioned to ensure everything works using the set of URLs you gathered at the beginning.
+- Do the pages have all the elements they should?
+- Are images loading?
+
+## How To Roll Back
+
+If something goes wrong and you need to roll back, don't panic!
+
+All you need to do is promote the old Heroku managed DB back into use:
+
+`heroku pg:promote --app micromasters-ci postgresql-rigid-71273`
+
+Obviously substitute your db and application for the ones above.
+
+Re-run your tests as defined above to make sure everything's working right post-rollback.
+
+## S3 Buckets
+
+Our applications use S3 buckets for CMS asset storage and backup among other things.
+
+You will need to either continue using the existing buckets by using `pulumi import` or creating new onnes. You should create
+new ones if the old ones don't conform to naming conventions. You'll also need to ensure that IAM permissions are properly
+set in your Pulumi code.
+
+To sync the bucket contents, use the [AWS CLI](https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html) `aws s3 sync` command.

src/ol_infrastructure/applications/micromasters/Pulumi.applications.micromasters.CI.yaml

@@ -0,0 +1,12 @@
+---
+secretsprovider: awskms://alias/infrastructure-secrets-ci
+encryptedkey: AQICAHi+npazf3LfzV9oCtcYyCMYLOzaQhbo9xt6lJVVpz9tkQHmbQbdOIGG4Jt34XVtsKrHAAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMzjNghqk4vTeob3lJAgEQgDt7H0yPqnjaQpouv/pngrLocorB4cYIcu+1zjTT95OxLKYWG6n4zAOokfTG64Ut0fyLkxA2EvI7vytTgg==
+config:
+  aws:region: us-east-1
+  consul:address: https://consul-micromasters-ci.odl.mit.edu
+  consul:scheme: https
+  micromasters:db_password:
+    secure: v1:DTEttuHYUMFQ5AJM:FSsYgItu3JT8hNcO/kz/JJn3t/dSHEWl5RSNUzlErvov6GDajBte9cvhMjrWi1itHopHJCkiseHdBxjz8Iulaodo9eeHQwSMrAq3+HuWgXE=
+  micromasters:domain: micromasters-ci.odl.mit.edu
+  vault:address: https://vault-ci.odl.mit.edu
+  vault_server:env_namespace: operations.ci

src/ol_infrastructure/applications/micromasters/Pulumi.applications.micromasters.QA.yaml

@@ -0,0 +1,14 @@
+---
+secretsprovider: awskms://alias/infrastructure-secrets-qa
+encryptedkey: AQICAHijXuVxVlAL6bY9xCOrzO3YYhFlQBPt6jNyJGkhYu+q4QEsTzqLr3gfTn1G3A6pkrEbAAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQM1rdw6SA8KJIVsrqgAgEQgDtFgUTp7OitAkIB79LlqX1C9uN9VEph+bqsa1Q0VNT1TP0pqNpCo2rzs7zmr3iUcvpAMn/1Y4q9TNPmHg==
+config:
+  aws:region: us-east-1
+  consul:address: https://consul-micromasters-qa.odl.mit.edu
+  consul:http_auth:
+    secure: v1:cFN5rfbLYKJOLKko:EbolfR0aA+3QNuLKU4ixNwZVdSuSB5UIl0gAPICG4mrprmQSDFabC/VkxrJGDgaAbELMxpskbvN0qj2y9SMX0IlKJYD2JHUZsfYqd8FX1QcxAxeSL00jJ5Zkks+mP8c=
+  consul:scheme: https
+  micromasters:db_password:
+    secure: v1:72AVczMFP7U0adTg:qiVKioH1VjNf38HfLkNIZsCdc8RqKs0bkaou+fU6SSRK9W4kbS1do2PwdD3JWYlc37eojd+sdAL8npp+tyEeQ66q5mHmWDQK3WlqgdWbf8G41fizyMwoy2AQdZmtZtJe40IoeqaSX3ntfJvhI0uP7YWmpge4G9e6KMNBmhZ0wl0T6qHU4z180aId1ZpjCM4=
+  micromasters:domain: micromasters-rc.odl.mit.edu
+  vault:address: https://vault-qa.odl.mit.edu
+  vault_server:env_namespace: operations.qa

src/ol_infrastructure/applications/micromasters/Pulumi.yaml

@@ -0,0 +1,7 @@
+---
+name: ol-infrastructure-micromasters-application
+runtime: python
+description: Pulumi project for deploying the stack of services needed by the micromasters
+  application
+backend:
+  url: s3://mitol-pulumi-state/

src/ol_infrastructure/applications/micromasters/__main__.py

@@ -0,0 +1,178 @@
+"""Create the infrastructure and services needed to support the
+MicroMasters application.
+
+- Create a PostgreSQL database in AWS RDS for production environments
+- Create an IAM policy to grant access to S3 and other resources
+"""
+
+import json
+
+import pulumi_vault as vault
+from pulumi import Config, StackReference, export
+from pulumi_aws import ec2, iam, s3
+
+from bridge.lib.magic_numbers import DEFAULT_POSTGRES_PORT
+from ol_infrastructure.components.aws.database import OLAmazonDB, OLPostgresDBConfig
+from ol_infrastructure.components.services.vault import (
+    OLVaultDatabaseBackend,
+    OLVaultPostgresDatabaseConfig,
+)
+from ol_infrastructure.lib.aws.iam_helper import lint_iam_policy
+from ol_infrastructure.lib.ol_types import AWSBase
+from ol_infrastructure.lib.pulumi_helper import parse_stack
+from ol_infrastructure.lib.stack_defaults import defaults
+from ol_infrastructure.lib.vault import setup_vault_provider
+
+setup_vault_provider()
+micromasters_config = Config("micromasters")
+stack_info = parse_stack()
+network_stack = StackReference(f"infrastructure.aws.network.{stack_info.name}")
+micromasters_vpc = network_stack.require_output("applications_vpc")
+operations_vpc = network_stack.require_output("operations_vpc")
+micromasters_environment = f"micromasters-{stack_info.env_suffix}"
+aws_config = AWSBase(
+    tags={
+        "OU": "micromasters",
+        "Environment": micromasters_environment,
+        "Application": "micromasters",
+    }
+)
+
+# Create S3 bucket
+
+# Bucket used to store files from MicroMasters app.
+micromasters_bucket_name = f"ol-micromasters-app-{stack_info.env_suffix}"
+micromasters_audit_bucket_name = f"odl-micromasters-audit-{stack_info.env_suffix}"
+micromasters_bucket = s3.Bucket(
+    f"micromasters-{stack_info.env_suffix}",
+    bucket=micromasters_bucket_name,
+    versioning=s3.BucketVersioningArgs(
+        enabled=True,
+    ),
+    tags=aws_config.tags,
+    acl="private",
+    policy=json.dumps(
+        {
+            "Version": "2012-10-17",
+            "Statement": [
+                {
+                    "Effect": "Allow",
+                    "Action": [
+                        "s3:GetObject",
+                        "s3:ListAllMyBuckets",
+                        "s3:ListBucket",
+                        "s3:ListObjects",
+                        "s3:PutObject",
+                        "s3:DeleteObject",
+                    ],
+                    "Resource": [f"arn:aws:s3:::{micromasters_bucket_name}/*"],
+                }
+            ],
+        }
+    ),
+    cors_rules=[{"allowedMethods": ["GET", "HEAD"], "allowedOrigins": ["*"]}],
+)
+
+
+micromasters_iam_policy = iam.Policy(
+    f"micromasters-{stack_info.env_suffix}-policy",
+    description="AWS access controls for the MicroMasters application in the "
+    f"{stack_info.name} environment",
+    path=f"/ol-applications/micromasters/{stack_info.env_suffix}/",
+    name_prefix=f"micromasters-{stack_info.env_suffix}-application-policy-",
+    policy=lint_iam_policy(
+        {
+            "Version": "2012-10-17",
+            "Statement": [
+                {
+                    "Effect": "Allow",
+                    "Action": "s3:ListAllMyBuckets",
+                    "Resource": "*",
+                },
+                {
+                    "Effect": "Allow",
+                    "Principal": "*",
+                    "Action": [
+                        "s3:ListBucket*",
+                        "s3:PutObject",
+                        "s3:PutObjectAcl",
+                        "s3:GetObject*",
+                        "s3:DeleteObject*",
+                    ],
+                    "Resource": [
+                        f"arn:aws:s3:::{micromasters_bucket_name}",
+                        f"arn:aws:s3:::{micromasters_bucket_name}/*",
+                        f"arn:aws:s3:::{micromasters_audit_bucket_name}",
+                        f"arn:aws:s3:::{micromasters_audit_bucket_name}/*",
+                    ],
+                },
+            ],
+        },
+        stringify=True,
+        parliament_config={
+            "PERMISSIONS_MANAGEMENT_ACTIONS": {
+                "ignore_locations": [{"actions": ["s3:putobjectacl"]}]
+            }
+        },
+    ),
+)
+
+micromasters_vault_backend_role = vault.aws.SecretBackendRole(
+    "micromasters-app",
+    name="micromasters",
+    backend="aws-mitx",
+    credential_type="iam_user",
+    policy_arns=[micromasters_iam_policy.arn],
+)
+
+# Create RDS instance
+micromasters_db_security_group = ec2.SecurityGroup(
+    f"micromasters-db-access-{stack_info.env_suffix}",
+    description=f"Access control for the MicroMasters App DB in {stack_info.name}",
+    ingress=[
+        ec2.SecurityGroupIngressArgs(
+            protocol="tcp",
+            from_port=DEFAULT_POSTGRES_PORT,
+            to_port=DEFAULT_POSTGRES_PORT,
+            cidr_blocks=["0.0.0.0/0"],
+            ipv6_cidr_blocks=["::/0"],
+            description="Allow access over the public internet from Heroku",
+        )
+    ],
+    egress=[
+        ec2.SecurityGroupEgressArgs(
+            from_port=0,
+            to_port=0,
+            protocol="-1",
+            cidr_blocks=["0.0.0.0/0"],
+            ipv6_cidr_blocks=["::/0"],
+        )
+    ],
+    tags=aws_config.merged_tags(
+        {"Name": "micromasters-db-access-applications-{stack_info.env_suffix}"}
+    ),
+    vpc_id=micromasters_vpc["id"],
+)
+
+micromasters_db_config = OLPostgresDBConfig(
+    instance_name=f"micromasters-{stack_info.env_suffix}-app-db",
+    password=micromasters_config.require("db_password"),
+    subnet_group_name=micromasters_vpc["rds_subnet"],
+    security_groups=[micromasters_db_security_group],
+    tags=aws_config.tags,
+    db_name="micromasters",
+    public_access=True,
+    **defaults(stack_info)["rds"],
+)
+micromasters_db = OLAmazonDB(micromasters_db_config)
+
+micromasters_vault_backend_config = OLVaultPostgresDatabaseConfig(
+    db_name=micromasters_db_config.db_name,
+    mount_point=f"{micromasters_db_config.engine}-micromasters",
+    db_admin_username=micromasters_db_config.username,
+    db_admin_password=micromasters_db_config.password.get_secret_value(),
+    db_host=micromasters_db.db_instance.address,
+)
+micromasters_vault_backend = OLVaultDatabaseBackend(micromasters_vault_backend_config)
+
+export("micromasters_app", {"rds_host": micromasters_db.db_instance.address})

src/ol_infrastructure/infrastructure/aws/network/__main__.py

@@ -514,26 +514,3 @@ def vpc_exports(vpc: OLVPC, peers: Optional[list[str]] = None) -> dict[str, Any]
     operations_vpc,
     xpro_vpc,
 )
-
-if stack_info.env_suffix == "production":
-    # TODO: Delete this once we migrate the Micromasters RDS into the applications VPC
-    # (TMM 2021-05-19)
-    # This is necessary in order for Redash to be able to access the
-    # MicroMasters read replica
-
-    # Update 2022-08-23 (TMM): Micromasters is being actively deprecated and the Redash
-    # instance has been migrated into the Data VPC. The result is that we need to keep
-    # this hack in place until Micromasters is taken out of operation, because it's not
-    # worth the effort to actually migrate the RDS instance.
-    ec2.Route(
-        "operations-to-micromasters-peer-route",
-        route_table_id=operations_vpc.route_table.id,
-        destination_cidr_block="10.10.0.0/16",
-        vpc_peering_connection_id="pcx-0d1b9264",
-    )
-    ec2.Route(
-        "data-to-micromasters-peer-route",
-        route_table_id=data_vpc.route_table.id,
-        destination_cidr_block="10.10.0.0/16",
-        vpc_peering_connection_id="pcx-0cfc129dda49e516c",
-    )