Merge pull request #56 from terraform-google-modules/cos-squid

Add squid container to cloud-config module

modules/cloud-config-container/squid/README.md

@@ -0,0 +1,80 @@
+# Containerized Squid on Container Optimized OS
+
+This module manages a `cloud-config` configuration that starts a containerized [Squid](http://www.squid-cache.org/) proxy on Container Optimized OS. The default configuration creates a filtering proxy that only allows connection to a whitelisted set of domains.
+
+The resulting `cloud-config` can be customized in a number of ways:
+
+- a custom squid.conf configuration can be set using the `squid_config` variable
+- additional files (e.g. additional acls) can be passed in via the `files` variable
+- a completely custom `cloud-config` can be passed in via the `cloud_config` variable, and additional template variables can be passed in via `config_variables`
+
+The default instance configuration inserts iptables rules to allow traffic on TCP port 3128.
+
+Logging and monitoring are enabled via the [Google Cloud Logging driver](https://docs.docker.com/config/containers/logging/gcplogs/) configured for the Squid container, and the [Node Problem Detector](https://cloud.google.com/container-optimized-os/docs/how-to/monitoring) service started by default on boot.
+
+The module renders the generated cloud config in the `cloud_config` output, to be used in instances or instance templates via the `user-data` metadata.
+
+For convenience during development or for simple use cases, the module can optionally manage a single instance via the `test_instance` variable. If the instance is not needed the `instance*tf` files can be safely removed. Refer to the [top-level README](../README.md) for more details on the included instance.
+
+## Examples
+
+### Default Squid configuration
+
+This example will create a `cloud-config` that allows any client in the 10.0.0.0/8 CIDR to use the proxy to connect github.com or any subdomain of github.com.
+
+```hcl
+module "cos-squid" {
+  source           = "./modules/cos-container/squid"
+  whitelist = [".github.com"]
+  clients   = ["10.0.0.0/8"]
+}
+
+# use it as metadata in a compute instance or template
+resource "google_compute_instance" "default" {
+  metadata = {
+    user-data = module.cos-squid.cloud_config
+  }
+```
+
+### Test Squid instance
+
+This example shows how to create the single instance optionally managed by the module, providing all required attributes in the `test_instance` variable. The instance is purposefully kept simple and should only be used in development, or when designing infrastructures.
+
+```hcl
+module "cos-squid" {
+  source           = "./modules/cos-container/squid"
+  whitelist = ["github.com"]
+  clients   = ["10.0.0.0/8"]
+  test_instance = {
+    project_id = "my-project"
+    zone       = "europe-west1-b"
+    name       = "cos-squid"
+    type       = "f1-micro"
+    network    = "default"
+    subnetwork = "https://www.googleapis.com/compute/v1/projects/my-project/regions/europe-west1/subnetworks/my-subnet"
+  }
+}
+```
+
+<!-- BEGIN TFDOC -->
+## Variables
+
+| name | description | type | required | default |
+|---|---|:---: |:---:|:---:|
+| *clients* | List of CIDRs from which Squid will allow connections | <code title="list&#40;string&#41;">list(string)</code> |  | <code title="">[]</code> |
+| *cloud_config* | Cloud config template path. If null default will be used. | <code title="">string</code> |  | <code title="">null</code> |
+| *config_variables* | Additional variables used to render the cloud-config and Squid templates. | <code title="map&#40;any&#41;">map(any)</code> |  | <code title="">{}</code> |
+| *file_defaults* | Default owner and permissions for files. | <code title="object&#40;&#123;&#10;owner       &#61; string&#10;permissions &#61; string&#10;&#125;&#41;">object({...})</code> |  | <code title="&#123;&#10;owner       &#61; &#34;root&#34;&#10;permissions &#61; &#34;0644&#34;&#10;&#125;">...</code> |
+| *files* | Map of extra files to create on the instance, path as key. Owner and permissions will use defaults if null. | <code title="map&#40;object&#40;&#123;&#10;content     &#61; string&#10;owner       &#61; string&#10;permissions &#61; string&#10;&#125;&#41;&#41;">map(object({...}))</code> |  | <code title="">{}</code> |
+| *squid_config* | Squid configuration path, if null default will be used. | <code title="">string</code> |  | <code title="">null</code> |
+| *test_instance* | Test/development instance attributes, leave null to skip creation. | <code title="object&#40;&#123;&#10;project_id &#61; string&#10;zone       &#61; string&#10;name       &#61; string&#10;type &#61; string&#10;network    &#61; string&#10;subnetwork &#61; string&#10;&#125;&#41;">object({...})</code> |  | <code title="">null</code> |
+| *test_instance_defaults* | Test/development instance defaults used for optional configuration. If image is null, COS stable will be used. | <code title="object&#40;&#123;&#10;disks &#61; map&#40;object&#40;&#123;&#10;read_only &#61; bool&#10;size      &#61; number&#10;&#125;&#41;&#41;&#10;image                 &#61; string&#10;metadata              &#61; map&#40;string&#41;&#10;nat                   &#61; bool&#10;service_account_roles &#61; list&#40;string&#41;&#10;tags                  &#61; list&#40;string&#41;&#10;&#125;&#41;">object({...})</code> |  | <code title="&#123;&#10;disks    &#61; &#123;&#125;&#10;image    &#61; null&#10;metadata &#61; &#123;&#125;&#10;nat      &#61; false&#10;service_account_roles &#61; &#91;&#10;&#34;roles&#47;logging.logWriter&#34;,&#10;&#34;roles&#47;monitoring.metricWriter&#34;&#10;&#93;&#10;tags &#61; &#91;&#34;ssh&#34;&#93;&#10;&#125;">...</code> |
+| *whitelist* | List of domains Squid will allow connections to | <code title="list&#40;string&#41;">list(string)</code> |  | <code title="">[]</code> |
+
+## Outputs
+
+| name | description | sensitive |
+|---|---|:---:|
+| cloud_config | Rendered cloud-config file to be passed as user-data instance metadata. |  |
+| test_instance | Optional test instance name and address |  |
+<!-- END TFDOC -->

modules/cloud-config-container/squid/cloud-config.yaml

@@ -0,0 +1,88 @@
+#cloud-config
+
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# TODO: switch to the gcplogs logging driver, and set driver labels
+
+users:
+- name: squid
+  uid: 2000
+
+write_files:
+  - path: /var/lib/docker/daemon.json
+    permissions: 0644
+    owner: root
+    content: |
+      {
+        "live-restore": true,
+        "storage-driver": "overlay2",
+        "log-opts": {
+          "max-size": "1024m"
+        }
+      }
+
+  - path: /etc/squid/squid.conf
+    permissions: 0644
+    owner: root
+    content: |
+      ${indent(6, squid_config)}
+
+  - path: /etc/squid/whitelist.txt
+    permissions: 0644
+    owner: root
+    content: |
+      ${indent(6, join("\n", whitelist))}
+
+  - path: /etc/squid/clients.txt
+    permissions: 0644
+    owner: root
+    content: |
+      ${indent(6, join("\n", clients))}
+
+  # squid container service
+  - path: /etc/systemd/system/squid.service
+    permissions: 0644
+    owner: root
+    content: |
+      [Unit]
+      Description=Start squid container
+      After=gcr-online.target docker.socket
+      Wants=gcr-online.target docker.socket docker-events-collector.service
+
+      [Service]
+      Environment="HOME=/home/squid"
+      ExecStartPre=/usr/bin/docker-credential-gcr configure-docker
+      ExecStart=/usr/bin/docker run --rm --name=squid  \
+        --log-driver=gcplogs --network host \
+        -v /etc/squid:/etc/squid \
+        gcr.io/pso-cft-fabric/squid:0.10
+      ExecStop=/usr/bin/docker stop squid
+      ExecStopPost=/usr/bin/docker rm squid
+
+  %{ for path, data in files }
+  - path: ${path}
+    owner: ${lookup(data, "owner", "root")}
+    permissions: ${lookup(data, "permissions", "0644")}
+    content: |
+      ${indent(4, data.content)}
+  %{ endfor }
+
+bootcmd:
+  - systemctl start node-problem-detector
+
+runcmd:
+  - iptables -I INPUT 1 -p tcp -m tcp --dport 3128 -m state --state NEW,ESTABLISHED -j ACCEPT
+  - systemctl daemon-reload
+  - systemctl start squid

modules/cloud-config-container/squid/instance.tf

@@ -0,0 +1 @@
+../instance.tf

modules/cloud-config-container/squid/main.tf

@@ -0,0 +1,45 @@
+/**
+ * Copyright 2019 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+locals {
+  cloud_config = templatefile(local.template, merge(local.config_variables, {
+    squid_config = templatefile(local.squid_config, local.config_variables)
+    files        = local.files
+  }))
+  squid_config = (
+    var.squid_config == null ? "${path.module}/squid.conf" : var.squid_config
+  )
+  files = {
+    for path, attrs in var.files : path => {
+      content = attrs.content,
+      owner   = attrs.owner == null ? var.file_defaults.owner : attrs.owner,
+      permissions = (
+        attrs.permissions == null
+        ? var.file_defaults.permissions
+        : attrs.permissions
+      )
+    }
+  }
+  template = (
+    var.cloud_config == null
+    ? "${path.module}/cloud-config.yaml"
+    : var.cloud_config
+  )
+  config_variables = merge(var.config_variables, {
+    whitelist = var.whitelist
+    clients   = var.clients
+  })
+}

modules/cloud-config-container/squid/outputs-instance.tf

@@ -0,0 +1 @@
+../outputs-instance.tf

modules/cloud-config-container/squid/outputs.tf

@@ -0,0 +1,20 @@
+/**
+ * Copyright 2019 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+output "cloud_config" {
+  description = "Rendered cloud-config file to be passed as user-data instance metadata."
+  value       = local.cloud_config
+}

modules/cloud-config-container/squid/squid.conf

@@ -0,0 +1,34 @@
+# bind to port 3128
+http_port 0.0.0.0:3128
+
+# only proxy, don't cache
+cache deny all
+
+acl ssl_ports port 443
+acl safe_ports port 80
+acl safe_ports port 443
+acl CONNECT method CONNECT
+
+# read clientd cidr from clients.txt
+acl clients src "/etc/squid/clients.txt"
+
+# read whitelisted domains from whitelist.txt
+acl whitelist dstdomain "/etc/squid/whitelist.txt"
+
+# deny access to anything other than ports 80 and 443
+http_access deny !safe_ports
+
+# deny CONNECT if connection is not using ssl
+http_access deny CONNECT !ssl_ports
+
+# deny acccess to cachemgr
+http_access deny manager
+
+# deny access to localhost though the proxy
+http_access deny to_localhost
+
+# allow connection from allowed clients only to the whitelisted domains
+http_access allow clients whitelist
+
+# deny everything else
+http_access deny all

modules/cloud-config-container/squid/variables-instance.tf

@@ -0,0 +1 @@
+../variables-instance.tf

modules/cloud-config-container/squid/variables.tf

@@ -0,0 +1,67 @@
+/**
+ * Copyright 2019 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+variable "cloud_config" {
+  description = "Cloud config template path. If null default will be used."
+  type        = string
+  default     = null
+}
+
+variable "config_variables" {
+  description = "Additional variables used to render the cloud-config and Squid templates."
+  type        = map(any)
+  default     = {}
+}
+
+variable "squid_config" {
+  description = "Squid configuration path, if null default will be used."
+  type        = string
+  default     = null
+}
+
+variable "file_defaults" {
+  description = "Default owner and permissions for files."
+  type = object({
+    owner       = string
+    permissions = string
+  })
+  default = {
+    owner       = "root"
+    permissions = "0644"
+  }
+}
+
+variable "files" {
+  description = "Map of extra files to create on the instance, path as key. Owner and permissions will use defaults if null."
+  type = map(object({
+    content     = string
+    owner       = string
+    permissions = string
+  }))
+  default = {}
+}
+
+variable "whitelist" {
+  description = "List of domains Squid will allow connections to"
+  type        = list(string)
+  default     = []
+}
+
+variable "clients" {
+  description = "List of CIDRs from which Squid will allow connections"
+  type        = list(string)
+  default     = []
+}