New Bindplane cloud-config-container setup (#2272)

* new bindplane cloud-config-container setup

modules/cloud-config-container/README.md

@@ -11,6 +11,7 @@ These modules are designed for several use cases:
 
 ## Available modules
 
+- [Bindplane](./bindplane)
 - [CoreDNS](./coredns)
 - [MySQL](./mysql)
 - [Nginx](./nginx)

modules/cloud-config-container/bindplane/README.md

@@ -0,0 +1,95 @@
+# Containerized Bindplane on Container Optimized OS
+
+This module manages a `cloud-config` configuration that starts a containerized [Bindplane](https://observiq.com/solutions) service on Container Optimized OS, using the official [Bindplane EE image](https://hub.docker.com/r/observiq/bindplane-ee) provided by observIQ and documented in the [official documentation page](https://observiq.com/download) when selecting "Docker" as target platform.
+
+The resulting `cloud-config` can be customized in a number of ways:
+
+- a custom Bindplane configuration can be set in Docker compose file available in `/run/bindplane/docker-compose.yml` using the `bindplane_config` variable
+- additional files can be passed in via the `files` variable
+- a completely custom `cloud-config` can be passed in via the `cloud_config` variable, and additional template variables can be passed in via `config_variables`
+
+The default instance configuration inserts iptables rules to allow traffic on port 3001.
+
+Logging and monitoring are enabled via the [Google Cloud Logging agent](https://cloud.google.com/container-optimized-os/docs/how-to/logging) configured for the instance via the `google-logging-enabled` metadata property, and the [Node Problem Detector](https://cloud.google.com/container-optimized-os/docs/how-to/monitoring) service started by default on boot.
+
+The module renders the generated cloud config in the `cloud_config` output, to be used in instances or instance templates via the `user-data` metadata.
+
+## Setup
+
+Please refer to the examples below for a sample terraform code for deploying a Bindplane server on GCP Compute VM with COS. After setting up the terraform code run the following commands:
+
+```bash
+terraform init
+terraform apply
+```
+
+Wait for a couple of minutes for the VM to be bootstrapped then connect via IAP tunnel using the following command (substitute the VM name, project and zone according to your configuration):
+
+```bash
+gcloud compute ssh $VM_NAME --project $PROJECT --zone $ZONE -- -L 3001:127.0.0.1:3001 -N -q -f
+```
+
+Navigate to http://localhost:3001 to access the Bindplane console, the following login page should be displayed.
+
+<p align="center">
+  <img src="./images/login.png" alt="Bindplane Login page">
+</p>
+
+## Examples
+
+### Default configuration
+
+This example will create a `cloud-config` that uses the module's defaults, creating a simple bindplane server with default (latest) docker image versions and setting localhost as remote url (suited only for local development).
+
+```hcl
+module "cos-nginx" {
+  source   = "./fabric/modules/cloud-config-container/bindplane"
+  password = "secret"
+}
+
+module "vm-nginx-tls" {
+  source     = "./fabric/modules/compute-vm"
+  project_id = "my-project"
+  zone       = "europe-west8-b"
+  name       = "cos-nginx"
+  network_interfaces = [{
+    network    = "default"
+    subnetwork = "gce"
+  }]
+  metadata = {
+    user-data              = module.cos-nginx.cloud_config
+    google-logging-enabled = true
+  }
+  boot_disk = {
+    initialize_params = {
+      image = "projects/cos-cloud/global/images/family/cos-stable"
+      type  = "pd-ssd"
+      size  = 10
+    }
+  }
+  tags = ["http-server", "ssh"]
+}
+# tftest modules=2 resources=2
+```
+
+<!-- BEGIN TFDOC -->
+## Variables
+
+| name | description | type | required | default |
+|---|---|:---:|:---:|:---:|
+| [password](variables.tf#L63) | Default admin user password. | <code>string</code> | ✓ |  |
+| [bindplane_config](variables.tf#L17) | Bindplane configurations. | <code title="object&#40;&#123;&#10;  remote_url                      &#61; optional&#40;string, &#34;localhost&#34;&#41;&#10;  bindplane_server_image          &#61; optional&#40;string, &#34;us-central1-docker.pkg.dev&#47;observiq-containers&#47;bindplane&#47;bindplane-ee:latest&#34;&#41;&#10;  bindplane_transform_agent_image &#61; optional&#40;string, &#34;us-central1-docker.pkg.dev&#47;observiq-containers&#47;bindplane&#47;bindplane-transform-agent:latest&#34;&#41;&#10;  bindplane_prometheus_image      &#61; optional&#40;string, &#34;us-central1-docker.pkg.dev&#47;observiq-containers&#47;bindplane&#47;bindplane-prometheus:1.56.0&#34;&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> |  | <code>&#123;&#125;</code> |
+| [cloud_config](variables.tf#L29) | Cloud config template path. If null default will be used. | <code>string</code> |  | <code>null</code> |
+| [config_variables](variables.tf#L35) | Additional variables used to render the cloud-config and Nginx templates. | <code>map&#40;any&#41;</code> |  | <code>&#123;&#125;</code> |
+| [file_defaults](variables.tf#L41) | Default owner and permissions for files. | <code title="object&#40;&#123;&#10;  owner       &#61; string&#10;  permissions &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> |  | <code title="&#123;&#10;  owner       &#61; &#34;root&#34;&#10;  permissions &#61; &#34;0644&#34;&#10;&#125;">&#123;&#8230;&#125;</code> |
+| [files](variables.tf#L53) | Map of extra files to create on the instance, path as key. Owner and permissions will use defaults if null. | <code title="map&#40;object&#40;&#123;&#10;  content     &#61; string&#10;  owner       &#61; string&#10;  permissions &#61; string&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> |  | <code>&#123;&#125;</code> |
+| [runcmd_post](variables.tf#L68) | Extra commands to run after starting nginx. | <code>list&#40;string&#41;</code> |  | <code>&#91;&#93;</code> |
+| [runcmd_pre](variables.tf#L74) | Extra commands to run before starting nginx. | <code>list&#40;string&#41;</code> |  | <code>&#91;&#93;</code> |
+| [users](variables.tf#L80) | List of additional usernames to be created. | <code title="list&#40;object&#40;&#123;&#10;  username &#61; string,&#10;  uid      &#61; number,&#10;&#125;&#41;&#41;">list&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> |  | <code title="&#91;&#10;&#93;">&#91;&#8230;&#93;</code> |
+
+## Outputs
+
+| name | description | sensitive |
+|---|---|:---:|
+| [cloud_config](outputs.tf#L17) | Rendered cloud-config file to be passed as user-data instance metadata. |  |
+<!-- END TFDOC -->

modules/cloud-config-container/bindplane/cloud-config.yaml

@@ -0,0 +1,128 @@
+#cloud-config
+
+# Copyright 2024 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# https://hub.docker.com/r/nginx/nginx/
+# https://nginx.io/manual/toc/#installation
+
+users:
+  - name: bindplane
+    uid: 2000
+  %{ for user in users }
+  - name: ${user.username}
+    uid: ${user.uid}
+  %{ endfor }
+
+write_files:
+  - path: /var/lib/docker/daemon.json
+    permissions: 0644
+    owner: root
+    content: |
+      {
+        "live-restore": true,
+        "storage-driver": "overlay2",
+        "log-opts": {
+          "max-size": "1024m"
+        }
+      }
+
+  - path: /run/bindplane/docker-compose.yml
+    permissions: 0644
+    owner: root
+    content: |
+      version: "3"
+
+      volumes:
+        bindplane:
+        prometheus:
+      
+      services:
+        prometheus:
+          container_name: bindplane-prometheus
+          restart: always
+          image: ${bindplane_prometheus_image}
+          ports:
+            - "9090:9090"
+          volumes:
+            - prometheus:/prometheus
+      
+        transform:
+          container_name: bindplane-transform-agent
+          restart: always
+          image: ${bindplane_transform_agent_image}
+          ports:
+            - "4568:4568"
+      
+        bindplane:
+          container_name: bindplane-server
+          restart: always
+          image: ${bindplane_server_image}
+          ports:
+            - "3001:3001"
+          environment:
+            - BINDPLANE_USERNAME=admin
+            - BINDPLANE_PASSWORD=${password}
+            - BINDPLANE_REMOTE_URL=http://${remote_url}:3001
+            - BINDPLANE_SESSION_SECRET=${uuid}
+            - BINDPLANE_LOG_OUTPUT=stdout
+            - BINDPLANE_ACCEPT_EULA=true
+            - BINDPLANE_PROMETHEUS_ENABLE=true
+            - BINDPLANE_PROMETHEUS_ENABLE_REMOTE=true
+            - BINDPLANE_PROMETHEUS_HOST=prometheus
+            - BINDPLANE_PROMETHEUS_PORT=9090
+            - BINDPLANE_TRANSFORM_AGENT_ENABLE_REMOTE=true
+            - BINDPLANE_TRANSFORM_AGENT_REMOTE_AGENTS=transform:4568
+          volumes:
+            - bindplane:/data
+          depends_on:
+            - prometheus
+            - transform
+
+  # bindplane container service
+  - path: /etc/systemd/system/bindplane.service
+    permissions: 0644
+    owner: root
+    content: |
+      [Unit]
+      Description=Start bindplane containers
+      After=gcr-online.target docker.socket
+      Wants=gcr-online.target docker.socket docker-events-collector.service
+      [Service]
+      Environment="HOME=/home/bindplane"
+      ExecStartPre=/usr/bin/docker-credential-gcr configure-docker
+      ExecStart=/usr/bin/docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /var:/var -v /run:/run -w=/var cryptopants/docker-compose-gcr -f /run/bindplane/docker-compose.yml up
+      ExecStop=/usr/bin/docker rm -f $(docker ps -a -q)
+
+  %{ for path, data in files }
+  - path: ${path}
+    owner: ${lookup(data, "owner", "root")}
+    permissions: ${lookup(data, "permissions", "0644")}
+    content: |
+      ${indent(6, data.content)}
+  %{ endfor }
+
+bootcmd:
+  - systemctl start node-problem-detector
+
+runcmd:
+%{ for cmd in runcmd_pre ~}
+  - ${cmd}
+%{ endfor ~}
+  - iptables -I INPUT 1 -p tcp -m tcp --dport 3001 -m state --state NEW,ESTABLISHED -j ACCEPT
+  - systemctl daemon-reload
+  - systemctl start bindplane
+%{ for cmd in runcmd_post ~}
+  - ${cmd}
+%{ endfor ~}

modules/cloud-config-container/bindplane/main.tf

@@ -0,0 +1,48 @@
+/**
+ * Copyright 2022 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+locals {
+  cloud_config = templatefile(local.template, merge(var.config_variables, {
+    bindplane_prometheus_image      = var.bindplane_config.bindplane_prometheus_image
+    bindplane_server_image          = var.bindplane_config.bindplane_server_image
+    bindplane_transform_agent_image = var.bindplane_config.bindplane_transform_agent_image
+    files                           = local.files
+    password                        = var.password
+    remote_url                      = var.bindplane_config.remote_url
+    runcmd_pre                      = var.runcmd_pre
+    runcmd_post                     = var.runcmd_post
+    users                           = var.users
+    uuid                            = random_uuid.uuid_secret.result
+  }))
+  files = {
+    for path, attrs in var.files : path => {
+      content = attrs.content,
+      owner   = attrs.owner == null ? var.file_defaults.owner : attrs.owner,
+      permissions = (
+        attrs.permissions == null
+        ? var.file_defaults.permissions
+        : attrs.permissions
+      )
+    }
+  }
+  template = (
+    var.cloud_config == null
+    ? "${path.module}/cloud-config.yaml"
+    : var.cloud_config
+  )
+}
+
+resource "random_uuid" "uuid_secret" {}

modules/cloud-config-container/bindplane/outputs.tf

@@ -0,0 +1,20 @@
+/**
+ * Copyright 2024 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+output "cloud_config" {
+  description = "Rendered cloud-config file to be passed as user-data instance metadata."
+  value       = local.cloud_config
+}

modules/cloud-config-container/bindplane/variables.tf

@@ -0,0 +1,88 @@
+/**
+ * Copyright 2022 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+variable "bindplane_config" {
+  description = "Bindplane configurations."
+  type = object({
+    remote_url                      = optional(string, "localhost")
+    bindplane_server_image          = optional(string, "us-central1-docker.pkg.dev/observiq-containers/bindplane/bindplane-ee:latest")
+    bindplane_transform_agent_image = optional(string, "us-central1-docker.pkg.dev/observiq-containers/bindplane/bindplane-transform-agent:latest")
+    bindplane_prometheus_image      = optional(string, "us-central1-docker.pkg.dev/observiq-containers/bindplane/bindplane-prometheus:1.56.0")
+  })
+  default  = {}
+  nullable = false
+}
+
+variable "cloud_config" {
+  description = "Cloud config template path. If null default will be used."
+  type        = string
+  default     = null
+}
+
+variable "config_variables" {
+  description = "Additional variables used to render the cloud-config and Nginx templates."
+  type        = map(any)
+  default     = {}
+}
+
+variable "file_defaults" {
+  description = "Default owner and permissions for files."
+  type = object({
+    owner       = string
+    permissions = string
+  })
+  default = {
+    owner       = "root"
+    permissions = "0644"
+  }
+}
+
+variable "files" {
+  description = "Map of extra files to create on the instance, path as key. Owner and permissions will use defaults if null."
+  type = map(object({
+    content     = string
+    owner       = string
+    permissions = string
+  }))
+  default = {}
+}
+
+variable "password" {
+  description = "Default admin user password."
+  type        = string
+}
+
+variable "runcmd_post" {
+  description = "Extra commands to run after starting nginx."
+  type        = list(string)
+  default     = []
+}
+
+variable "runcmd_pre" {
+  description = "Extra commands to run before starting nginx."
+  type        = list(string)
+  default     = []
+}
+
+variable "users" {
+  description = "List of additional usernames to be created."
+  type = list(object({
+    username = string,
+    uid      = number,
+  }))
+  default = [
+  ]
+}