Merge pull request #442 from dovholuknf/add-zero-trust-builder

feat: continue refinment of compose builder for zero trust

README.md

@@ -96,6 +96,24 @@ This folder contains the following compose files:
   - Use `make run no-secty app-sample arm64` and `make down` to start and stop the services using this compose file.
   - Use `make pull no-secty app-sample <service(s)>` to pull all or some images for the services in this compose file.
 
+- **docker-compose-openziti.yml**
+  Contains the services needed to bring OpenZiti online, configure it, and enable consul to perform underlay-based health checking. Used in conjunction with `docker-compose-zero-trust.yml`. This compose file should be started before starting the `docker-compose-zero-trust.yml` compose file.
+
+  **Make Commands**
+
+    - Use `make openziti` and `make openziti-down` to start and stop the services using this compose file.
+    - Use `make openziti-clean` to remove all stopped containers, all volumes and all networks used by the EdgeX stack. Use this command when needing to do a fresh restart. **Note** You must _also_ run the corresponding `make zero-trust-clean` command to fully clean up.
+    - Use `make openziti-logs` to follow the logs
+
+- **docker-compose-zero-trust.yml**
+  Contains the services needed to run in zero-trust secure mode. Used in conjunction with `docker-compose-openziti.yml`. Start this compose file after starting OpenZiti. When operating in zero-trust mode, no ports are available other than the OpenZiti ports. Accessing services must be done using an OpenZiti tunneler or through using an OpenZiti SDK. The `go-mod-bootstrap` library has been upgraded to support zero-trust.
+
+  **Make Commands**
+
+  - Use `make zero-trust` and `make zero-trust-down` to start and stop the services using this compose file.
+  - Use `make zero-trust-clean` to remove all stopped containers, all volumes and all networks used by the EdgeX stack. Use this command when needing to do a fresh restart. **Note** You must _also_ run the corresponding `make openziti-clean` command to fully clean up. 
+  - Use `make openziti-logs` to follow the logs
+
 ### TAF Compose files
 
 The compose files under the `taf` subfolder are used for the automated TAF tests. These compose files are also generated from `Compose Builder` when the `make build` command is used.

compose-builder/.gitignore

@@ -2,4 +2,5 @@
 docker-compose.yml
 # staging directory is generated by the shell script gen_secure_compose_ext.sh
 gen_ext_scty/
+gen_ext_compose
 res/

compose-builder/README.md

@@ -10,7 +10,10 @@ This folder contains the `Compose Builder` which is made up of **source** compos
 
 ### Compose CLI Command
 
-The Makefile in this folder expects the `docker compose` CLI command.
+The Makefile in this folder expects the `docker compose` CLI command to be on the path or it expects you to supply
+a valid `docker compose` command by setting `DOCKER_COMPOSE`. The version of `docker compose` must be equal or greater 
+than `Docker Compose version v2.24.4`. 
+
 The old stand-alone `docker-compose` tool is no longer supported.
 See https://docs.docker.com/compose/install/ for installation details for the latest `docker compose` CLI command.
 
@@ -191,6 +194,7 @@ Standard compose variations are:
 run [options] [services]
 Runs the EdgeX services as specified by:
 Options:
+    zero-trust:       Runs with OpenZiti support for zero-trust networking
     no-secty:         Runs in Non-Secure Mode, otherwise runs in Secure Mode
     arm64:            Runs using ARM64 images
     dev:              Runs using local built images from edgex-go repo
@@ -234,6 +238,7 @@ Options:
                       Not valid in secure mode when uses with mqtt-bus
     nats-bus:         Runs with services configure for NATS Message Bus
                       The NATS Server service is also included.
+    no-cleanup:       Leaves generated files behind for debugging purposes.
 Services:
     <names...>: Runs only services listed (and their dependent services) where 'name' matches a service name in one of the compose files used
 ```
@@ -257,6 +262,7 @@ Stops all EdgeX services no matter which configuration started them
 pull [options] [services]
 Pulls the EdgeX service images as specified by:
 Options:
+    zero-trust:       Pulls images for OpenZiti, supporting zero-trust networking
     no-secty:         Pulls images for Non-Secure Mode, otherwise pull images 
                       for Secure Mode
     arm64:            Pulls ARM64 version of images
@@ -290,6 +296,7 @@ Options:
                       Pull includes NonoMQ MQTT broker when mqtt-broker or mqtt-bus are specified
                       Not valid in secure mode when uses with mqtt-bus                    
     nats-bus:         Pull includes additional services for NATS Message Bus
+    no-cleanup:       Leaves generated files behind for debugging purposes.
 
 Services:
     <names...>: Pulls only images for the service(s) listed
@@ -300,6 +307,7 @@ Services:
 gen [options]
 Generates temporary single file compose file (`docker-compose.yml`) as specified by:
 Options:
+    zero-trust:       Generates with OpenZiti support for zero-trust networking included
     no-secty:         Generates non-secure compose,
                       otherwise generates secure compose file
     arm64:            Generates compose file using ARM64 images
@@ -344,6 +352,7 @@ Options:
                       Not valid in secure mode when uses with mqtt-bus
     nats-bus:         Generates compose file with services configured for NAT Message Bus
                       The NATS Server service is also included.
+    no-cleanup:       Leaves generated files behind for debugging purposes.
 ```
 #### Clean
 
@@ -411,6 +420,7 @@ compose [options]
 Generates the EdgeX compose file as specified by options and stores them in the configured release folder. Compose files are named appropriately for release and options used to generate them.
 
 Options:
+    zero-trust:       Generates compose file with OpenZiti support for zero-trust networking included
     no-secty:         Generates non-secure compose file, otherwise generates secure compose file
     arm64:            Generates compose file using ARM64 images
     dev:              Generates using local built images from edgex-go repo
@@ -454,6 +464,7 @@ Options:
     mqtt-verbose      Enables MQTT Broker verbose logging.
     nats-bus:         Generates compose file with services configure for NATS Message Bus
                       The NATS Server service is also included.
+    no-cleanup:       Leaves generated files behind for debugging purposes.
 ```
 
 #### TAF Compose

compose-builder/add-security-zero-trust.yml

@@ -0,0 +1,97 @@
+name: edgex
+services:
+  app-rules-engine:
+    environment:
+      SERVICE_HOST: app-rules-engine.edgex.ziti
+      SERVICE_PORT: 80
+    ports: !reset null
+  core-command:
+    environment:
+      SERVICE_HOST: core-command.edgex.ziti
+      SERVICE_PORT: 80
+      CLIENTS_CORE_METADATA_HOST: "core-metadata.edgex.ziti"
+      CLIENTS_CORE_METADATA_PORT: 80
+      CLIENTS_CORE_METADATA_SECURITYOPTIONS_MODE: "zerotrust"
+    ports: !reset null
+  core-common-config-bootstrapper:
+    environment:
+      ALL_SERVICES_SERVICE_SECURITYOPTIONS_MODE: "zerotrust"
+      APP_SERVICES_CLIENTS_CORE_METADATA_HOST: core-metadata.edgex.ziti
+      APP_SERVICES_CLIENTS_CORE_METADATA_PORT: 80
+      APP_SERVICES_CLIENTS_CORE_METADATA_SECURITYOPTIONS_MODE: "zerotrust"
+      DEVICE_SERVICES_CLIENTS_CORE_METADATA_HOST: core-metadata.edgex.ziti
+      DEVICE_SERVICES_CLIENTS_CORE_METADATA_PORT: 80
+      DEVICE_SERVICES_CLIENTS_CORE_METADATA_SECURITYOPTIONS_MODE: "zerotrust"
+    ports: !reset null
+  core-data:
+    environment:
+      SERVICE_HOST: core-data.edgex.ziti
+      SERVICE_PORT: 80
+    ports: !reset null
+  core-metadata:
+    environment:
+      SERVICE_HOST: core-metadata.edgex.ziti
+      SERVICE_PORT: "80"
+    ports: !reset null
+  rules-engine:
+    environment:
+      EDGEX_CREDENTIAL_NAME: rules-engine
+      EDGEX_CREDENTIALS: /tmp/edgex/secrets/rules-engine/secrets-token.json
+      OPENZITI_CONTROLLER: openziti:1280
+    ports: !reset null
+    volumes:
+      - edgex-init:/edgex-init
+      - /tmp/edgex/secrets/rules-engine:/tmp/edgex/secrets/rules-engine:ro,z
+  support-notifications:
+    environment:
+      SERVICE_HOST: support-notifications.edgex.ziti
+      SERVICE_PORT: 80
+    ports: !reset null
+  support-scheduler:
+    environment:
+      SERVICE_HOST: support-scheduler.edgex.ziti
+      SERVICE_PORT: 80
+    ports: !reset null
+  ui:
+    command:
+      - "./edgex-ui-server"
+      - "--configDir=res/docker"
+    container_name: edgex-ui-go
+    depends_on:
+      consul:
+        condition: service_started
+      core-common-config-bootstrapper:
+        condition: service_started
+      core-metadata:
+        condition: service_started
+      security-bootstrapper:
+        condition: service_started
+    env_file:
+      - common-security.env
+      - common-sec-stage-gate.env
+    environment:
+      SERVICE_HOST: edgex-ui-go
+      CLIENTS_CORE_COMMAND_SECURITYOPTIONS_MODE: "zerotrust"
+      CLIENTS_CORE_COMMAND_HOST: "core-command.edgex.ziti"
+      CLIENTS_CORE_COMMAND_PORT: 80
+      CLIENTS_CORE_DATA_SECURITYOPTIONS_MODE: "zerotrust"
+      CLIENTS_CORE_DATA_HOST: "core-data.edgex.ziti"
+      CLIENTS_CORE_DATA_PORT: 80
+      CLIENTS_CORE_METADATA_SECURITYOPTIONS_MODE: "zerotrust"
+      CLIENTS_CORE_METADATA_HOST: "core-metadata.edgex.ziti"
+      CLIENTS_CORE_METADATA_PORT: 80
+      CLIENTS_RULES_ENGINE_SECURITYOPTIONS_MODE: "zerotrust"
+      CLIENTS_RULES_ENGINE_HOST: "rules-engine.edgex.ziti"
+      CLIENTS_RULES_ENGINE_PORT: 80
+      CLIENTS_SUPPORT_NOTIFICATIONS_SECURITYOPTIONS_MODE: "zerotrust"
+      CLIENTS_SUPPORT_NOTIFICATIONS_HOST: "support-notifications.edgex.ziti"
+      CLIENTS_SUPPORT_NOTIFICATIONS_PORT: 80
+      CLIENTS_SUPPORT_SCHEDULER_SECURITYOPTIONS_MODE: "zerotrust"
+      CLIENTS_SUPPORT_SCHEDULER_HOST: "support-scheduler.edgex.ziti"
+      CLIENTS_SUPPORT_SCHEDULER_PORT: 80
+      VAULT_ADDR: http://edgex-vault:8200
+    entrypoint:
+      - /edgex-init/ready_to_run_wait_install.sh
+    volumes:
+      - edgex-init:/edgex-init
+      - /tmp/edgex/secrets/ui:/tmp/edgex/secrets/ui:ro,z

compose-builder/add-service-secure-template.yml

@@ -25,9 +25,9 @@ services:
     environment:
       EDGEX_ADD_REGISTRY_ACL_ROLES: ${TOKEN_LIST}
 
-  security-proxy-setup:
-    environment:
-      EDGEX_ADD_PROXY_ROUTE: ${EXTRA_PROXY_ROUTE_LIST}
+  ${ZERO_TRUST}security-proxy-setup:
+  ${ZERO_TRUST}  environment:
+  ${ZERO_TRUST}    EDGEX_ADD_PROXY_ROUTE: ${EXTRA_PROXY_ROUTE_LIST}
 
   ${SERVICE_NAME}:
     entrypoint: [${SHELL_OVERRIDE} "/edgex-init/ready_to_run_wait_install.sh"]

compose-builder/gen_secure_compose_ext.sh

@@ -47,10 +47,24 @@ mkdir -p "$GEN_EXT_DIR"
 
 ADD_SERVICE_SECURE_FILE_TEMPLATE="add-service-secure-template.yml"
 
-SERVICE_EXT_COMPOSE_PATH=./"$GEN_EXT_DIR"/add-"$service_name"-secure.yml
+SERVICE_EXT_COMPOSE_PATH="./${GEN_EXT_DIR}/add-${service_name}-secure.yml"
 sed 's/${SERVICE_NAME}:/'"$service_name"':/g' "$ADD_SERVICE_SECURE_FILE_TEMPLATE" > "$SERVICE_EXT_COMPOSE_PATH"
 sed -i 's/${SERVICE_KEY}/'"$service_key"'/g' "$SERVICE_EXT_COMPOSE_PATH"
 sed -i 's,${EXECUTABLE},'"$executable"',g' "$SERVICE_EXT_COMPOSE_PATH"
+if [ "$ZERO_TRUST" = "1" ]; then
+  sed -i 's,${ZERO_TRUST},#,g' "$SERVICE_EXT_COMPOSE_PATH"
+  cat >> "$SERVICE_EXT_COMPOSE_PATH" <<HERE
+    # env_file does not override environment and these values are set in the add-* templates
+    # use a heredoc and append it to the generated file accordingly so that docker compose will
+    # reduce it down and override as expected
+    environment:
+      SERVICE_HOST: ${service_name}.edgex.ziti
+      SERVICE_PORT: 80
+    ports: !reset null
+HERE
+else
+  sed -i 's,${ZERO_TRUST},,g' "$SERVICE_EXT_COMPOSE_PATH"
+fi
 case "${service_name}" in
   device-bacnet-ip | device-bacnet-mstp | device-coap | device-gpio)
     # These services don't have dumb-init in their containers, causing an issue for the wait script, use sh instead