Unable to update GitLab instance

[amarjandu]

Unable to update GitLab from 14.7.4 to 14.7.5, Terraform attempts to update the ec2 instance in place, when the instance starts up again the instance version has not been updated.

See Terraform output for more details.

The patch used to apply the update was

From fd2fbc0f6b9ae67e3e8c3544d45c5d5cca4dc1c7 Mon Sep 17 00:00:00 2001
From: amar jandu <[email protected]>
Date: Mon, 14 Mar 2022 11:03:57 -0700
Subject: [PATCH] Update GitLab to 14.7.5

---
 terraform/gitlab/gitlab.tf.json.template.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/terraform/gitlab/gitlab.tf.json.template.py b/terraform/gitlab/gitlab.tf.json.template.py
index 7aa9261f..da6e3388 100644
--- a/terraform/gitlab/gitlab.tf.json.template.py
+++ b/terraform/gitlab/gitlab.tf.json.template.py
@@ -1240,7 +1240,7 @@ emit_tf({} if config.terraform_component != 'gitlab' else {
                                --volume /mnt/gitlab/config:/etc/gitlab \
                                --volume /mnt/gitlab/logs:/var/log/gitlab \
                                --volume /mnt/gitlab/data:/var/opt/gitlab \
-                               gitlab/gitlab-ce:14.7.4-ce.0
+                               gitlab/gitlab-ce:14.7.5-ce.0
                         docker run \
                                --detach \
                                --name gitlab-runner \
@@ -1248,7 +1248,7 @@ emit_tf({} if config.terraform_component != 'gitlab' else {
                                --volume /mnt/gitlab/runner/config:/etc/gitlab-runner \
                                --network gitlab-runner-net \
                                --env DOCKER_HOST=tcp://gitlab-dind:2375 \
-                               gitlab/gitlab-runner:v14.7.0
+                               gitlab/gitlab-runner:v14.7.1
                     '''[1:]),  # trim newline char at the beginning as dedent() only removes indent common to all lines
                 'tags': {
                     'Name': 'azul-gitlab',
-- 
2.24.3 (Apple Git-128)

[amarjandu]

Prior to the VPN addition (#3605), when updating the GitLab the ec2 instance was destroyed then created again, this time when updated Terraform performed the changes in-place. Perhaps this is why the modifications are not reflected to the instance.

[hannes-ucsc]

hashicorp/terraform-provider-aws#23315

The provider update from #3605 is probably the cause.

From https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/instance#user_data

• user_data - (Optional) User data to provide when launching the instance. Do not pass gzip-compressed data via this argument; see user_data_base64 instead. Updates to this field will trigger a stop/start of the EC2 instance.

But it it didn't even reboot the instance. I suspect that this is because we use the most recent version of the provider with a rather old version of TF.

[hannes-ucsc]

I could see the updated user data referring to version 14.7.5 on the instance. Rebooting the instance did not seem to have an effect: the instance came back with the old image version 14.7.4. Stopping and starting also does not help, which is really odd.

[hannes-ucsc]

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-add-user-data.html

If you stop an instance, modify its user data, and start the instance, the updated user data is not run when you start the instance.

[hannes-ucsc]

Spike to run the following experiment.

1. Get back to clean state by terminating the instance in the console or with terraform destroy and redeploying from a clean unmodified working copy with develop checked out. Post screenshot of user data (like the one above) to prove that user data refers to 14.7.4. Wait until instance is fully up and reachable via web UI.

2. Stop the instance in the EC2 console

3. Apply above patch and deploy

4. Post screenshot of user data

5. Start the instance via the EC2 console

6. Wait until it comes back up

7. Post screenshot of user data and of https://gitlab.dev.singlecell.gi.ucsc.edu/help

[achave11-ucsc]

Step four and seven are merged into one, since five is inapplicable, deploying the changes (step three) automatically starts the instance.

[theathorn]

@hannes-ucsc :"Based on the experiment it appears that the only way for user data to take effect is terminating and recreating the instance, something that the current version of the AWS provider does not appear to be doing automatically. We need to workaround this by updating the operator manual to include terraform destroy on the instance".

[hannes-ucsc]

PR #3627 only added a workaround for this issue. If a PR is a partial fix for one issue and a complete fix for another, it should have the partial label (to alert us that something is special about the PR) but it should only be connected to the issue for which it is a complete fix.

FYI: @theathorn, @achave11

[hannes-ucsc]

Turns out that the root cause is a regression introduced by the fix for hashicorp/terraform-provider-aws#23. Before that fix a modification of the user_data property of the aws_instance resource resulted in a recreation of the instance. After that fix, which was included in the 4.2.0 release of the AWS provider, the same modification resulted only in a start/stop cycle. Cloud-init runs write_files once per instance (not per boot) so restarting the instance does not cause any files to be rewritten.

hashicorp/terraform-provider-aws#23315 fixes the regression by adding an attribute with which users can revert back to the pre-4.2.0 behavior. It's included in the 4.7.0 release.

[hannes-ucsc]

Think about how to best demo this.

[hannes-ucsc]

For demo, update the GitLab version on dev. It should not require any manual action.