diff --git a/runbooks/source/node-group-changes.html.md.erb b/runbooks/source/node-group-changes.html.md.erb index 95434f3f..e6e56e32 100644 --- a/runbooks/source/node-group-changes.html.md.erb +++ b/runbooks/source/node-group-changes.html.md.erb @@ -1,7 +1,7 @@ --- title: Handling Node Group and Instance Changes weight: 54 -last_reviewed_on: 2023-12-13 +last_reviewed_on: 2023-12-15 review_in: 6 months --- @@ -16,13 +16,16 @@ You may need to make a change to an EKS [cluster node group] or [instance type c To avoid bringing down all the nodes at once is to follow these steps: 1. add a new node group with your [updated changes] -1. lookup the old node group name (you can find this in the aws gui) -1. once merged in you can drain the old node group using the command below: -1. raise a new [pr deleting] the old node group +2. re-run the [infrastructure-account/terraform-apply] pipeline to update the Modsecurity Audit logs cluster to map roles to both old and new node group IAM Role + This is to avoid losing modsec audit logs from the new node group +3. lookup the old node group name (you can find this in the aws gui) +4. once merged in you can drain the old node group using the command below: -> cloud-platform pipeline cordon-and-drain --cluster-name --node-group - -[script source] because this command runs remotely in concourse you can't use this command to drain default ng on the manager cluster. + > cloud-platform pipeline cordon-and-drain --cluster-name --node-group + [script source] because this command runs remotely in concourse you can't use this command to drain default ng on the manager cluster. +5. raise a new [pr deleting] the old node group +6. re-run the [infrastructure-account/terraform-apply] pipeline to again to update the Modsecurity Audit logs cluster to map roles with only the new node group IAM Role +7. run the integration tests to ensure the cluster is healthy ### Notes: @@ -36,3 +39,4 @@ To avoid bringing down all the nodes at once is to follow these steps: [updated changes]: https://github.com/ministryofjustice/cloud-platform-infrastructure/pull/2657 [cordons and drains nodes]: https://github.com/ministryofjustice/cloud-platform-terraform-concourse/blob/main/pipelines/manager/main/cordon-and-drain-nodes.yaml [script source]: https://github.com/ministryofjustice/cloud-platform-terraform-concourse/blob/7851f741e6c180ed868a97d51cec0cf1e109de8d/pipelines/manager/main/cordon-and-drain-nodes.yaml#L50 +[infrastructure-account/terraform-apply]: https://concourse.cloud-platform.service.justice.gov.uk/teams/main/pipelines/infrastructure-account/jobs/terraform-apply