feat: add helper module to determine code_uri

This commit replaces how we map "version" to code URI for our lambda functions. Previously we maintained a hardcoded mapping of `region` to `code_uri` for each binary, e.g: ``` region,code_uri ap-south-1,s3://observeinc-ap-south-1/apps/forwarder/1.19.2/faceca9e757102324a1fa722b95f8546 eu-north-1,s3://observeinc-eu-north-1/apps/forwarder/1.19.2/e4bfefb742254a45b1db834cb9e70c89 ``` This was successful in ensuring that a given terraform module version had a pinned binary version, but allowed no flexibility for overriding version. We now dynamically resolve the code URI by downloading the cloudformation template and parsing out the expected property. This has the disadvantage that it tightly couples us to S3 and our cloudformation template, but arguably both must work for the Lambda install process to work.
observeinc · Jun 6, 2024 · 58894dd · 58894dd
1 parent f6dcb49
commit 58894dd
Show file tree

Hide file tree

Showing 25 changed files with 176 additions and 74 deletions.
diff --git a/modules/forwarder/README.md b/modules/forwarder/README.md
@@ -60,7 +60,7 @@ module "forwarder" {
 |------|---------|
 | <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.3 |
 | <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 5.0 |
-| <a name="requirement_random"></a> [random](#requirement\_random) | >= 3.0.0 |
+| <a name="requirement_http"></a> [http](#requirement\_http) | >= 3.0 |
 
 ## Providers
 
@@ -70,7 +70,9 @@ module "forwarder" {
 
 ## Modules
 
-No modules.
+| Name | Source | Version |
+|------|--------|---------|
+| <a name="module_samversion"></a> [samversion](#module\_samversion) | ../samversion | n/a |
 
 ## Resources
 
@@ -99,6 +101,8 @@ No modules.
 
 | Name | Description | Type | Default | Required |
 |------|-------------|------|---------|:--------:|
+| <a name="input_code_uri"></a> [code\_uri](#input\_code\_uri) | S3 URI for lambda binary. If set, takes precedence over code\_version. | `string` | `""` | no |
+| <a name="input_code_version"></a> [code\_version](#input\_code\_version) | Binary version as released on github.com/observeinc/aws-sam-apps. | `string` | `""` | no |
 | <a name="input_content_type_overrides"></a> [content\_type\_overrides](#input\_content\_type\_overrides) | A list of key value pairs. The key is a regular expression which is<br>applied to the S3 source (<bucket>/<key>) of forwarded files. The value<br>is the content type to set for matching files. For example,<br>`\.json$=application/x-ndjson` would forward all files ending in `.json`<br>as newline delimited JSON | <pre>list(object({<br>    pattern      = string<br>    content_type = string<br>  }))</pre> | `[]` | no |
 | <a name="input_debug_endpoint"></a> [debug\_endpoint](#input\_debug\_endpoint) | Endpoint to send debugging telemetry to. Sets the OTEL\_EXPORTER\_OTLP\_ENDPOINT environment variable for the lambda function. | `string` | `""` | no |
 | <a name="input_destination"></a> [destination](#input\_destination) | Destination filedrop | <pre>object({<br>    arn    = optional(string, "")<br>    bucket = optional(string, "")<br>    prefix = optional(string, "")<br>    # exclusively for backward compatible HTTP endpoint<br>    uri = optional(string, "")<br>  })</pre> | n/a | yes |

diff --git a/modules/forwarder/main.tf b/modules/forwarder/main.tf
@@ -1,6 +1,6 @@
 locals {
-  s3_uri          = one([for item in csvdecode(file("${path.module}/uris.csv")) : item["code_uri"] if item["region"] == data.aws_region.current.name])
-  parsed_s3_uri   = regex("s3://(?P<bucket>[^/]+)/(?P<key>.+)", local.s3_uri)
+  code_uri        = var.code_uri != "" ? var.code_uri : module.samversion[0].code_uri
+  parsed_s3_uri   = regex("s3://(?P<bucket>[^/]+)/(?P<key>.+)", local.code_uri)
   destination_uri = var.destination.uri != "" ? var.destination.uri : "s3://${var.destination.bucket}/${var.destination.prefix}"
   name_prefix     = "${substr(var.name, 0, 37)}-"
 
@@ -20,3 +20,11 @@ data "aws_caller_identity" "current" {}
 data "aws_region" "current" {}
 
 data "aws_partition" "current" {}
+
+module "samversion" {
+  count    = var.code_uri != "" ? 0 : 1
+  source   = "../samversion"
+  app      = "forwarder"
+  function = "Forwarder"
+  release  = var.code_version
+}
diff --git a/modules/forwarder/sam.version b/modules/forwarder/sam.version
@@ -0,0 +1 @@
+1.19.3
diff --git a/modules/forwarder/uris.csv b/modules/forwarder/uris.csv
diff --git a/modules/forwarder/variables.tf b/modules/forwarder/variables.tf
@@ -188,3 +188,17 @@ variable "verbosity" {
   nullable    = false
   default     = 1
 }
+
+variable "code_version" {
+  description = "Binary version as released on github.com/observeinc/aws-sam-apps."
+  type        = string
+  default     = ""
+  nullable    = false
+}
+
+variable "code_uri" {
+  description = "S3 URI for lambda binary. If set, takes precedence over code_version."
+  type        = string
+  default     = ""
+  nullable    = false
+}
diff --git a/modules/forwarder/versions.tf b/modules/forwarder/versions.tf
@@ -7,9 +7,9 @@ terraform {
       version = ">= 5.0"
     }
 
-    random = {
-      source  = "hashicorp/random"
-      version = ">= 3.0.0"
+    http = {
+      source  = "hashicorp/http"
+      version = ">= 3.0"
     }
   }
 }
diff --git a/modules/logwriter/README.md b/modules/logwriter/README.md
@@ -94,6 +94,8 @@ module "logwriter" {
 | <a name="input_bucket_arn"></a> [bucket\_arn](#input\_bucket\_arn) | S3 Bucket ARN to write log records to. | `string` | n/a | yes |
 | <a name="input_buffering_interval"></a> [buffering\_interval](#input\_buffering\_interval) | Buffer incoming data for the specified period of time, in seconds, before<br>delivering it to S3. | `number` | `60` | no |
 | <a name="input_buffering_size"></a> [buffering\_size](#input\_buffering\_size) | Buffer incoming data to the specified size, in MiBs, before delivering it<br>to S3. | `number` | `1` | no |
+| <a name="input_code_uri"></a> [code\_uri](#input\_code\_uri) | S3 URI for lambda binary. If set, takes precedence over code\_version. | `string` | `null` | no |
+| <a name="input_code_version"></a> [code\_version](#input\_code\_version) | Binary version as released on github.com/observeinc/aws-sam-apps. | `string` | `null` | no |
 | <a name="input_debug_endpoint"></a> [debug\_endpoint](#input\_debug\_endpoint) | Endpoint to send debugging telemetry to. Sets the OTEL\_EXPORTER\_OTLP\_ENDPOINT environment variable for the lambda function. | `string` | `null` | no |
 | <a name="input_discovery_rate"></a> [discovery\_rate](#input\_discovery\_rate) | EventBridge rate expression for periodically triggering discovery. If not<br>set, no eventbridge rules are configured. | `string` | `null` | no |
 | <a name="input_exclude_log_group_name_patterns"></a> [exclude\_log\_group\_name\_patterns](#input\_exclude\_log\_group\_name\_patterns) | Exclude any log group names matching the provided set of regular<br>expressions. | `list(string)` | `null` | no |

diff --git a/modules/logwriter/subscriber.tf b/modules/logwriter/subscriber.tf
@@ -17,4 +17,6 @@ module "subscriber" {
   lambda_timeout                  = var.lambda_timeout
   debug_endpoint                  = var.debug_endpoint
   verbosity                       = var.verbosity
+  code_uri                        = var.code_uri
+  code_version                    = var.code_version
 }
diff --git a/modules/logwriter/variables.tf b/modules/logwriter/variables.tf
@@ -148,3 +148,15 @@ variable "verbosity" {
   type        = number
   default     = null
 }
+
+variable "code_version" {
+  description = "Binary version as released on github.com/observeinc/aws-sam-apps."
+  type        = string
+  default     = null
+}
+
+variable "code_uri" {
+  description = "S3 URI for lambda binary. If set, takes precedence over code_version."
+  type        = string
+  default     = null
+}
diff --git a/modules/samversion/README.md b/modules/samversion/README.md
@@ -0,0 +1,15 @@
+# SAM Version
+
+This is a helper module that given a SAM App name and Lambda function resource ID returns the code URI.
+
+## Usage
+
+```hcl
+module "samversion" {
+    app             = "forwarder"
+    function        = "Forwarder"
+    release_version = "1.19.3"
+}
+```
+
+If `release_version` is omitted, we will default to a value that is updated by `updatecli`.
diff --git a/modules/samversion/main.tf b/modules/samversion/main.tf
@@ -0,0 +1,18 @@
+locals {
+  latest_release = "1.19.3"
+  code_version   = var.release != "" ? var.release : local.latest_release
+  code_uri       = yamldecode(data.http.manifest.response_body)["Resources"][var.function]["Properties"]["CodeUri"]
+}
+
+data "aws_region" "current" {}
+
+data "http" "manifest" {
+  url = "https://observeinc-${data.aws_region.current.name}.s3.amazonaws.com/aws-sam-apps/${local.code_version}/${var.app}.yaml"
+
+  lifecycle {
+    postcondition {
+      condition     = self.status_code == 200 && can(yamldecode(self.response_body))
+      error_message = "Unable to retrieve manifest"
+    }
+  }
+}
diff --git a/modules/samversion/outputs.tf b/modules/samversion/outputs.tf
@@ -0,0 +1,4 @@
+output "code_uri" {
+  description = "Code URI for binary."
+  value       = local.code_uri
+}
diff --git a/modules/samversion/variables.tf b/modules/samversion/variables.tf
@@ -0,0 +1,18 @@
+variable "app" {
+  description = "App name"
+  type        = string
+  nullable    = false
+}
+
+variable "function" {
+  description = "Function name"
+  type        = string
+  nullable    = false
+}
+
+variable "release" {
+  description = "Release version on github.com/observeinc/aws-sam-apps."
+  type        = string
+  default     = ""
+  nullable    = false
+}
diff --git a/modules/samversion/versions.tf b/modules/samversion/versions.tf
@@ -0,0 +1,15 @@
+terraform {
+  required_version = ">= 1.3"
+
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = ">= 5.0"
+    }
+
+    http = {
+      source  = "hashicorp/http"
+      version = ">= 3.0"
+    }
+  }
+}
diff --git a/modules/stack/README.md b/modules/stack/README.md
@@ -166,12 +166,13 @@ You can additionally configure other submodules in this manner:
 
 | Name | Description | Type | Default | Required |
 |------|-------------|------|---------|:--------:|
+| <a name="input_code_version"></a> [code\_version](#input\_code\_version) | Binary versions as released on github.com/observeinc/aws-sam-apps. | `string` | `null` | no |
 | <a name="input_config"></a> [config](#input\_config) | Variables for AWS Config collection. | <pre>object({<br>    include_resource_types        = list(string)<br>    exclude_resource_types        = optional(list(string))<br>    delivery_frequency            = optional(string)<br>    include_global_resource_types = optional(bool)<br>  })</pre> | `null` | no |
 | <a name="input_configsubscription"></a> [configsubscription](#input\_configsubscription) | Variables for AWS Config subscription. | <pre>object({<br>    delivery_bucket_name = string<br>  })</pre> | `null` | no |
 | <a name="input_debug_endpoint"></a> [debug\_endpoint](#input\_debug\_endpoint) | Endpoint to send debugging telemetry to. Sets OTEL\_EXPORTER\_OTLP\_ENDPOINT environment variable for supported lambda functions. | `string` | `null` | no |
 | <a name="input_destination"></a> [destination](#input\_destination) | Destination filedrop | <pre>object({<br>    arn    = optional(string, "")<br>    bucket = optional(string, "")<br>    prefix = optional(string, "")<br>    # exclusively for backward compatible HTTP endpoint<br>    uri = optional(string, "")<br>  })</pre> | n/a | yes |
-| <a name="input_forwarder"></a> [forwarder](#input\_forwarder) | Variables for forwarder module. | <pre>object({<br>    source_bucket_names                      = optional(list(string), [])<br>    source_topic_arns                        = optional(list(string), [])<br>    content_type_overrides                   = optional(list(object({ pattern = string, content_type = string })), [])<br>    max_file_size                            = optional(number)<br>    lambda_memory_size                       = optional(number)<br>    lambda_timeout                           = optional(number)<br>    lambda_env_vars                          = optional(map(string))<br>    retention_in_days                        = optional(number)<br>    queue_max_receive_count                  = optional(number)<br>    queue_delay_seconds                      = optional(number)<br>    queue_message_retention_seconds          = optional(number)<br>    queue_batch_size                         = optional(number)<br>    queue_maximum_batching_window_in_seconds = optional(number)<br>  })</pre> | `{}` | no |
-| <a name="input_logwriter"></a> [logwriter](#input\_logwriter) | Variables for AWS CloudWatch Logs collection. | <pre>object({<br>    log_group_name_patterns         = optional(list(string))<br>    log_group_name_prefixes         = optional(list(string))<br>    exclude_log_group_name_prefixes = optional(list(string))<br>    buffering_interval              = optional(number)<br>    buffering_size                  = optional(number)<br>    filter_name                     = optional(string)<br>    filter_pattern                  = optional(string)<br>    num_workers                     = optional(number)<br>    discovery_rate                  = optional(string, "24 hours")<br>    lambda_memory_size              = optional(number)<br>    lambda_timeout                  = optional(number)<br>  })</pre> | `null` | no |
+| <a name="input_forwarder"></a> [forwarder](#input\_forwarder) | Variables for forwarder module. | <pre>object({<br>    source_bucket_names                      = optional(list(string), [])<br>    source_topic_arns                        = optional(list(string), [])<br>    content_type_overrides                   = optional(list(object({ pattern = string, content_type = string })), [])<br>    max_file_size                            = optional(number)<br>    lambda_memory_size                       = optional(number)<br>    lambda_timeout                           = optional(number)<br>    lambda_env_vars                          = optional(map(string))<br>    retention_in_days                        = optional(number)<br>    queue_max_receive_count                  = optional(number)<br>    queue_delay_seconds                      = optional(number)<br>    queue_message_retention_seconds          = optional(number)<br>    queue_batch_size                         = optional(number)<br>    queue_maximum_batching_window_in_seconds = optional(number)<br>    code_uri                                 = optional(string)<br>    code_version                             = optional(string)<br>  })</pre> | `{}` | no |
+| <a name="input_logwriter"></a> [logwriter](#input\_logwriter) | Variables for AWS CloudWatch Logs collection. | <pre>object({<br>    log_group_name_patterns         = optional(list(string))<br>    log_group_name_prefixes         = optional(list(string))<br>    exclude_log_group_name_prefixes = optional(list(string))<br>    buffering_interval              = optional(number)<br>    buffering_size                  = optional(number)<br>    filter_name                     = optional(string)<br>    filter_pattern                  = optional(string)<br>    num_workers                     = optional(number)<br>    discovery_rate                  = optional(string, "24 hours")<br>    lambda_memory_size              = optional(number)<br>    lambda_timeout                  = optional(number)<br>    code_uri                        = optional(string)<br>    code_version                    = optional(string)<br>  })</pre> | `null` | no |
 | <a name="input_metricstream"></a> [metricstream](#input\_metricstream) | Variables for AWS CloudWatch Metrics Stream collection. | <pre>object({<br>    include_filters    = optional(list(object({ namespace = string, metric_names = optional(list(string)) })))<br>    exclude_filters    = optional(list(object({ namespace = string, metric_names = optional(list(string)) })))<br>    buffering_interval = optional(number)<br>    buffering_size     = optional(number)<br>  })</pre> | `null` | no |
 | <a name="input_name"></a> [name](#input\_name) | Name of role. Since this name must be unique within the<br>account, it will be reused for most of the resources created by this<br>module. | `string` | n/a | yes |
 | <a name="input_s3_bucket_lifecycle_expiration"></a> [s3\_bucket\_lifecycle\_expiration](#input\_s3\_bucket\_lifecycle\_expiration) | Expiration in days for S3 objects in collection bucket | `number` | `4` | no |

diff --git a/modules/stack/forwarder.tf b/modules/stack/forwarder.tf
@@ -22,4 +22,6 @@ module "forwarder" {
   queue_maximum_batching_window_in_seconds = var.forwarder.queue_maximum_batching_window_in_seconds
   debug_endpoint                           = var.debug_endpoint
   verbosity                                = var.verbosity
+  code_uri                                 = var.forwarder.code_uri
+  code_version                             = try(coalesce(var.forwarder.code_version, var.code_version), null)
 }
diff --git a/modules/stack/logwriter.tf b/modules/stack/logwriter.tf
@@ -18,6 +18,8 @@ module "logwriter" {
   lambda_timeout          = var.logwriter.lambda_timeout
   debug_endpoint          = var.debug_endpoint
   verbosity               = var.verbosity
+  code_uri                = var.logwriter.code_uri
+  code_version            = try(coalesce(var.logwriter.code_version, var.code_version), null)
 
   depends_on = [aws_s3_bucket_notification.this]
 }
diff --git a/modules/stack/variables.tf b/modules/stack/variables.tf
@@ -42,6 +42,8 @@ variable "forwarder" {
     queue_message_retention_seconds          = optional(number)
     queue_batch_size                         = optional(number)
     queue_maximum_batching_window_in_seconds = optional(number)
+    code_uri                                 = optional(string)
+    code_version                             = optional(string)
   })
   default  = {}
   nullable = false
@@ -86,6 +88,8 @@ variable "logwriter" {
     discovery_rate                  = optional(string, "24 hours")
     lambda_memory_size              = optional(number)
     lambda_timeout                  = optional(number)
+    code_uri                        = optional(string)
+    code_version                    = optional(string)
   })
   default = null
 }
@@ -120,3 +124,9 @@ variable "verbosity" {
   type        = number
   default     = null
 }
+
+variable "code_version" {
+  description = "Binary versions as released on github.com/observeinc/aws-sam-apps."
+  type        = string
+  default     = null
+}
diff --git a/modules/subscriber/README.md b/modules/subscriber/README.md
@@ -9,6 +9,7 @@ This app is specifically to register new cloudwatch log groups for the `logwrite
 |------|---------|
 | <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.1 |
 | <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 4.0 |
+| <a name="requirement_http"></a> [http](#requirement\_http) | >= 3.0 |
 
 ## Providers
 
@@ -18,7 +19,9 @@ This app is specifically to register new cloudwatch log groups for the `logwrite
 
 ## Modules
 
-No modules.
+| Name | Source | Version |
+|------|--------|---------|
+| <a name="module_samversion"></a> [samversion](#module\_samversion) | ../samversion | n/a |
 
 ## Resources
 
@@ -41,12 +44,13 @@ No modules.
 | [aws_iam_policy_document.scheduler_assume_role_policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
 | [aws_iam_policy_document.scheduler_queue](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
 | [aws_iam_policy_document.subscription_policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
-| [aws_region.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/region) | data source |
 
 ## Inputs
 
 | Name | Description | Type | Default | Required |
 |------|-------------|------|---------|:--------:|
+| <a name="input_code_uri"></a> [code\_uri](#input\_code\_uri) | S3 URI for lambda binary. If set, takes precedence over code\_version. | `string` | `""` | no |
+| <a name="input_code_version"></a> [code\_version](#input\_code\_version) | Binary version as released on github.com/observeinc/aws-sam-apps. | `string` | `"1.19.3"` | no |
 | <a name="input_debug_endpoint"></a> [debug\_endpoint](#input\_debug\_endpoint) | Endpoint to send debugging telemetry to. Sets the OTEL\_EXPORTER\_OTLP\_ENDPOINT environment variable for the lambda function. | `string` | `""` | no |
 | <a name="input_destination_iam_arn"></a> [destination\_iam\_arn](#input\_destination\_iam\_arn) | ARN for destination iam policy | `string` | n/a | yes |
 | <a name="input_discovery_rate"></a> [discovery\_rate](#input\_discovery\_rate) | EventBridge scheduler rate expression for periodically triggering discovery. If not set, no scheduler is configured. | `string` | `""` | no |