DPR2-893: Glue connection and placeholder operational datastore secret #6670
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
tom-ogle-moj
commented
Jun 19, 2024
•
tom-ogle-moj
commented
- Limit created resources to Dev/Sandbox only
- Create a Glue JDBC Data Connection for attachment to DataHub Glue jobs to give them Operational Datastore access
- Temporarily populate it with the manually created Dev/Sandbox RDS details which will be replaced by the actual Operational Datastore details
- Create a secret for the credentials used by the DataHub Glue jobs to write to the Operational Datastore
- Create a Security Group associated with the Glue JDBC Data Connection
- it will be associated with a glue job automatically when the connection is attached to it
- this is a requirement of Glue and it allows Glue to communicate with itself by allowing TCP ingress on all ports from the security group. See https://docs.aws.amazon.com/glue/latest/dg/setup-vpc-for-glue-access.html
- the same security group is attached to the RDS as well as the glue job so the same ingress rule also allows access to the RDS instance. Instead, a separate security group could be attached to the RDS allowing Ingress from this security group.
- it allows egress on all ports which allows it to communicate with services such as S3 without having an S3 service endpoint in the VPC.
- I've tested this by attaching the connection to the activities glue batch job, manually giving the job access to read the secret and verifying it can write to the RDS database using the connection and secret provided here
…ts for use in datahub jobs.
github-actions
bot
added
the
environments-repository
tom-ogle-moj
had a problem deploying
to
digital-prison-reporting-test
GitHub Actions
Error
tom-ogle-moj
had a problem deploying
to
digital-prison-reporting-development
GitHub Actions
Error
tom-ogle-moj
had a problem deploying
to
digital-prison-reporting-development
GitHub Actions
Failure
tom-ogle-moj
had a problem deploying
to
digital-prison-reporting-test
GitHub Actions
Error
tom-ogle-moj
temporarily deployed
to
digital-prison-reporting-development
GitHub Actions
Inactive
tom-ogle-moj
temporarily deployed
to
digital-prison-reporting-development
GitHub Actions
Inactive
tom-ogle-moj
temporarily deployed
to
digital-prison-reporting-development
GitHub Actions
Inactive
tom-ogle-moj
had a problem deploying
to
digital-prison-reporting-development
GitHub Actions
Error
…elated resources to development environment only for now.
tom-ogle-moj
temporarily deployed
to
digital-prison-reporting-development
GitHub Actions
Inactive
tom-ogle-moj
changed the title
tom-ogle-moj
temporarily deployed
to
digital-prison-reporting-test
GitHub Actions
Inactive
koladeadewuyi-moj
approved these changes
Jun 20, 2024
gavriil-g-moj
approved these changes
Jun 20, 2024
It seems that it fails for our project continuously for every PR |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.