diff --git a/quickstart/DHCP.md b/quickstart/DHCP.md new file mode 100644 index 0000000..ed95458 --- /dev/null +++ b/quickstart/DHCP.md @@ -0,0 +1,238 @@ +# Deploying CoreDHCP on a "Real" System + + + + +The quickstart rather hastily sets up CoreDHCP with assumed network parameters, +but it would be useful to know how to configure it for an actual system. This +document serves to walk through how to configure CoreDHCP with the +[coresmd](https://github.com/OpenCHAMI/coresmd) plugins. + +## Purpose + +SMD is meant to be the source of truth for nodes/BMCs in the cluster, so the +goal of CoreDHCP + coresmd is to match MAC addresses requesting an IP address to +interfaces stored in SMD and serve the matching IP address. However, for +unknown MAC addresses to become known to SMD, they need to be added, for +example, by network discovery tools like +[Magellan](https://github.com/OpenCHAMI/magellan). To be discoverable at the +network layer, coresmd provides functionality for providing unknown MAC +addresses with temporary IP addresses so they can be discovered. Once they are +discovered and added to SMD, they can get a more "permanent" IP address from +coresmd. + +## Methodology + +Coresmd differentiates between *known* MAC addresses (handled by the `coresmd` +plugin itself) and *unknown* MAC addresses (handled by the `bootloop` plugin or +by CoreDHCP's `file` plugin depending on if IP-MAC mapping is necessary). The +general flow for a device getting a long-term IP address from scratch is as +follows: + +1. Unknown MAC gets assigned an IP with a short lease. + - This can be an available IP from a pool (`bootloop` plugin) or a fixed IP + (`file` plugin). + - If left unknown, device will continually request a new IP and get a + short-lived one until the MAC becomes known. +1. MAC with short-leased IP gets added to SMD. + - This happens outside the scope of DHCP. + - How this happens can depend on the device type: + - **BMC:** Using Magellan. **NOTE:** Modern versions of Magellan add node + interfaces to SMD if discovered via Redfish. + - **Node:** POSTing to SMD using `curl` or the Ochami CLI tool. +1. Known MAC gets assigned the IP assigned to it in SMD once the short-leased IP + address expires, but with a longer lease time. + - This happens via the `coresmd` plugin itself. + +The first step in the above can be handled by either coresmd's `bootloop` plugin +or CoreDHCP's `file` plugin, or via a combination of both. The next two sections +describe the uses for these plugins and how they work while the section after +describes how the `coresmd` plugin itself works. + +### Unknown MAC Addresses: The `file` Plugin + +This plugin is used when it *does* matter which MAC address gets which IP +address. It is paired with CoreDHCP's `lease_time` directive to set how long the +temporary IPs should last. This plugin is maintained by CoreDHCP. + +The `file` plugin is pretty simple: it hands out the IP address assigned to the +MAC address sending the DHCPDISCOVER and renews this IP once it expires. + +### Unknown MAC Addresses: The `bootloop` Plugin + +This plugin is used when it does not matter which MAC address gets which IP +address. Often, this is as a catch-all for MAC addresses not in an assignment +list, e.g. MAC addresses not caught by the `file` plugin above. + +As stated, the `bootloop` plugin is designed to assign available IPs from a pool +to unknown MAC addresses without the guarantee that specific IP addresses get +assigned to certain MAC addresses. While this plugin works with any device that +speaks DHCP, important behavioral differences are present between devices that +are able to network boot (e.g. ethernet interfaces on a node) and devices that +are not able to network boot (e.g. BMCs). The difference is how requests to +renew IP addresses are handled. When devices that can boot try to renew their +IP address, they are served an iPXE script that reboots them so they are forced +to renew their IP address. When devices that cannot boot try to renew their IP +address, their request is responded to with a DHCPNAK, which, according to [RFC +2131](https://datatracker.ietf.org/doc/html/rfc2131#section-3.2), causes the +device to reinitiate the entire DHCP handshake. + +Technically, all DHCPDISCOVERs from MAC addresses that haven't been assigned an +IP address are responded to with a DHCPOFFER with the temporary IP address and +the rebooting iPXE script. Devices that can boot execute this iPXE script while +devices that cannot do not. So, when a non-booting device tries to renew this IP +address with a DHCPREQUEST, the response is a DHCPNAK so that it will send a +DHCPDISCOVER. + +### Known MAC Addresses: The `coresmd` Plugin + +This plugin is used to assign IP addresses based on data in SMD. + +A cache in memory is maintained containing SMD Component and EthernetInterface +data which is refreshed at a configured interval. This refreshment occurs via a +separate thread (goroutine). + +When a DHCP request reaches the plugin, it checks the cache if 1) the MAC +address exists as an EthernetInterface, 2) if there is an IP address for this +interface, and 3) if there is a corresponding Component for this interface. If +all three exist, the IP address corresponding to the EthernetInterface structure +is assigned to the device. This could be a node NIC or a BMC. + +## Preparation + +### (REQUIRED) TFTP + +Since CoreDHCP does not include a TFTP server or plugin (as far as is known at +this writing), one is required that contains the following files at the TFTP +root: + +- **ipxe.efi** --- UEFI iPXE bootloader for amd64 systems +- **undionly.kpxe** --- Legacy bootloader for x86-based systems +- **reboot.ipxe** --- The reboot iPXE script which contains: + ```ipxe + #!ipxe + reboot + ``` + +### (OPTIONAL) File for `file` Plugin + +If using the `file` plugin, you will need a plaintext file that contains the +MAC-to-IP mapping. For example: + +``` +de:ca:fc:0f:fe:ee 172.16.0.101 +de:ad:be:ee:ee:ef 172.16.0.102 +``` + +## Writing a Configuration File + +The configuration file is YAML-formatted. The general format is: + +```yaml +server4: + plugins: + - plugin1: arg1 arg2 + - plugin2: arg1 arg2 + ... +``` + +... where `plugin1` and `plugin2` are plugin names in the plugin list, each +followed by space-separated arguments. + +### Part 1: Server Configuration + +The first part of this file should be plugins that configure basic server +settings, such as the IP of the DHCP server and optional DNS servers. These +settings should be configured *before* the coresmd configuration, since CoreDHCP +sends DHCP packets to be processed sequentially, *in order*, through these +plugins. + +Let's look at an example server configuration: + +```yaml +server4: + plugins: + - server_id: 172.16.0.253 + - dns: 1.1.1.1,8.8.8.8 + - router: 172.16.0.254 + - netmask: 255.255.255.0 +``` + +- **server_id:** (*REQUIRED*) This is the "identity" of the DHCP server to + distinguish it from any other servers that might be listening on the same + network. Usually this is just the IP address the server is listening on. +- **dns:** (*OPTIONAL*) A comma-separated list of DNS servers to use for names + and domains. +- **router:** (*REQUIRED*) The IP address of the network gateway for routing + packets. This can be the same as the IP address CoreDHCP is listening on if + that machine acts as a gateway. +- **netmask:** (*OPTIONAL*) The network mask used with IP addresses served by + the `file` and `bootloop` plugins, if used. This is not needed if one is + *only* using the `coresmd` plugin. + +### Part 2: CoreSMD Configuration + +The next part of the configuration file corresponds to the place where any of +the coresmd/file/bootloop plugins are configured. These need to be *below* the +server config above. + +```yaml +server4: + plugins: + ... + - coresmd: https://foobar.openchami.cluster http://172.16.0.253:8081 /root_ca/root_ca.crt 30s 1h + - lease_time: 10m + - file: /etc/coredhcp/hostsfile + - bootloop: /tmp/coredhcp.db 5m 172.16.0.156 172.16.0.200 +``` + +- **coresmd:** (*REQUIRED*) Check if MAC address in request matches any + component in SMD. Pass request through if not. + + Arguments: + - **SMD Base URI:** (*https://foobar.openchami.cluster*) Base URI for where + SMD is listening (usually behind API proxy), usually with TLS enabled. + - **Boot Script Base URI:** (*http://172.16.0.253:8081*) Base URI for where + BSS is listening to fetch boot scripts from, usually *without* TLS. This is + a separate argument because chances are that the CA certificate is not baked + into the iPXE bootloader and thus cannot perform proper certificate + validation. + - **Path to CA Certifacate:** (*/root_ca/root_ca.crt*) Path to certificate + authority certificate for validation of connections to SMD. + - **Cache Update Interval:** (*30s*) Amount of time in between cache + refreshes.[^intervals] + - **Known Device Lease Duration:** (*1h*) Amount of time a *known* device's IP + is valid for.[^intervals] +- **lease_time:** (*OPTIONAL*) Assign lease time for *unknown* nodes. This is + *required* if using the `file` or `bootloop` plugins. + + Arguments: + - **Unknown Device Lease Duration:** (*10m*) Amount of time an *unknown* + device's IP is valid for.[^intervals] +- **file:** (*OPTIONAL*) Assign specific IP addresses to specific MAC addresses + based on mapping in file. Typically, this comes right after `coresmd` since + some MACs that are unknown to SMD need to be assigned a specific IP address. + If the MAC isn't in the list, it gets passed to the "catch-all" `bootloop` + plugin below. + + Arguments: + - **Map File Path:** (*/etc/coredhcp/hostsfile*) Path to text file that maps + MAC addresses to IP addresses. +- **bootloop:** (*OPTIONAL*) Assign available IP addresses from a pool to + unknown MAC addresses. This is normally the last plugin in the file because it + is usually used as a catch-all: the MAC was not known by SMD and was not + listed in the map file. + + Arguments: + - **Storage DB Path:** (*/tmp/coredhcp.db*) Path to sqlite3 file used for + storing IP addresses that have been assigned. This file need not exist and + will be created by the plugin upon initialization (assuming permissions are + correct!). + - **Pool Start IP:** (*172.16.0.156*) Starting IP address (inclusive) of the + pool of available IP addresses to hand out. + - **Pool End IP:** (*172.16.0.200*) Ending IP address (inclusive) of the pool + of available IP addresses to hand out. + +[^intervals]: Interval strings are parsed via Go's + [time.ParseDuration](https://pkg.go.dev/time#ParseDuration) function. Check + there for valid strings. diff --git a/quickstart/README.md b/quickstart/README.md index 3fca5d1..0a519d6 100644 --- a/quickstart/README.md +++ b/quickstart/README.md @@ -12,6 +12,17 @@ This quickstart makes a few assumptions about the target operating system and is * x86_64 - Some of the containers involved are built and tested for alternative operating systems and architectures, but the solution as a whole is only tested with x86 containers * Dedicated System - The docker compose setup assumes that it can take control of several TCP ports and interact with the host network for DHCP and TFTP. It is tested on a dedicated virtual machine * Local Name Registration - The quickstart bootstraps a Certificate Authority and issues an SSL certificate with a predictable name. For access, you will need to add that name/IP to /etc/hosts on all clients or make it resolvable through your site DNS +* DHCP Network Configuration: + * Server/Gateway IP Address: __192.168.0.254__ + * Range of IPs for Unknown MAC Addresses: __192.168.150__ to __192.168.0.253__ (inclusive) + * These IPs are given to MAC addresses unknown to SMD with a short least time. + * Network Mask: __255.255.255.0__ + * DNS Servers: __1.1.1.1, 8.8.8.8__ + * Duration to renew information from SMD: __30 seconds__ + * "Long" lease time: __1 hour__ + * This duration is used in leases for devices known by SMD. + * "Short" lease time: __5 minutes__ + * This duration is used in leases for devices unknown by SMD. ## Start Here @@ -22,25 +33,20 @@ This quickstart makes a few assumptions about the target operating system and is ``` 1. Create the secrets file and choose a name for your system. We use `foobar` in our example. - __Note__ The certificates for the system use the name you provide in this file. It's not easy to change. - - __Note__ The script attempts to figure out which ip address is most likely to be your system ip. If it is unsuccessful, `LOCAL_IP=` will be empty and you'll need to update it manually - - __Note__ The full url will be https://foobar.openchami.cluster which you should set manually in /etc/hosts and point to the same ip address as `LOCAL_IP` in `.env`. - - __Note__ The `generate-configs.sh` script accepts an optional second argument that allows a custom domain to be specified. If not specified, it defaults to "openchami.cluster", which is used throughout these instructions. + - __Note__ The full url will be https://foobar.openchami.cluster which you should set manually in /etc/hosts and point to the same ip address as `LOCAL_IP` in `.env`, which should be 192.168.0.254. ```bash # Create the secrets in the .env file. Do not share them with anyone. - ./generate-configs.sh foobar - # Confirm that LOCAL_IP has a value and matches what you want the interface to OpenCHAMI to be. We do our best to guess what your primary interface is. - grep LOCAL_IP .env + ./generate-configs.sh ``` If you have problems with this step, check to make sure that the main IP address of your host is in `.env` as `LOCAL_IP`. -1. Update your /etc/hosts to point your system name to your local ip (this is important for valid certs) +1. Update your /etc/hosts to point `foobar.openchami.cluster` to 192.168.0.254 (this is important for valid certs). 1. Start the main services ```bash - docker compose -f base.yml -f postgres.yml -f jwt-security.yml -f haproxy-api-gateway.yml -f openchami-svcs.yml -f autocert.yml up -d + docker compose -f base.yml -f postgres.yml -f jwt-security.yml -f haproxy-api-gateway.yml -f openchami-svcs.yml -f autocert.yml -f tftp.yml -f coredhcp.yml up -d ``` __If this step produces an error like: `Error response from daemon: invalid IP address in add-host: ""` it means you're missing the LOCAL_IP in step 2.__ You can fix it by destroying everything, editing `.env` manually and starting over. The command to destroy is the same as the command to create, just replace `up -d` with `down --volumes` - 1. Use the running system to download your certs and create your access token(s) ```bash # Assuming you're using bash as your shell, you can use the included functions to simplify interactions with your new OpenCHAMI system. @@ -54,14 +60,6 @@ This quickstart makes a few assumptions about the target operating system and is curl --cacert cacert.pem -H "Authorization: Bearer $ACCESS_TOKEN" https://foobar.openchami.cluster/hsm/v2/State/Components # This should respond with an empty set of Components: {"Components":[]} ``` -1. Create a token that can be used by the dnsmasq-loader which reads from smd. This activates our automatic dns/dhcp system. The command automatically adds it to .env - ```bash - echo "DNSMASQ_ACCESS_TOKEN=$(gen_access_token)" >> .env - ``` -1. Use docker-compose to bring up your dnsmasq contianers. The only difference between this command and the one above is the addition of the `dnsmasq.yml` file. Docker compose needs to know about all the files to follow dependencies. - ```bash - docker compose -f base.yml -f postgres.yml -f jwt-security.yml -f haproxy-api-gateway.yml -f openchami-svcs.yml -f autocert.yml -f dnsmasq.yml up -d - ``` ## cloud-init Server Setup diff --git a/quickstart/configs/.gitignore b/quickstart/configs/.gitignore index 0bd252a..00468f4 100644 --- a/quickstart/configs/.gitignore +++ b/quickstart/configs/.gitignore @@ -1 +1,2 @@ opaal.yaml +coredhcp.yaml diff --git a/quickstart/configs/coredhcp-template.yaml b/quickstart/configs/coredhcp-template.yaml new file mode 100644 index 0000000..547da73 --- /dev/null +++ b/quickstart/configs/coredhcp-template.yaml @@ -0,0 +1,25 @@ +server4: + plugins: + # + # Base CoreDHCP config + # + + - server_id: 192.168.0.254 + - dns: 1.1.1.1 8.8.8.8 + - router: 172.16.0.254 + - netmask: 255.255.255.0 + + # + # CoreSMD config + # + + # Args: ochami_base_url boot_script_base_url ca_cert_path cache_update_interval long_lease_time + - coresmd: http://192.168.0.254:8081 /root_ca/root_ca.crt 30s 1h + + # Optionally include the file plugin here if it matters which IPs get assigned to which + # MACs. Otherwise, unknown MACs get passed to the bootloop "catch-all" plugin below. + # + #- file /etc/coredhcp/hostsfile + + # Args: storage_path short_lease_time ip_pool_start ip_pool_end + - bootloop: /tmp/coredhcp.db 5m 192.168.0.150 192.168.0.253 diff --git a/quickstart/coredhcp.yml b/quickstart/coredhcp.yml new file mode 100644 index 0000000..547c9a0 --- /dev/null +++ b/quickstart/coredhcp.yml @@ -0,0 +1,22 @@ +services: + coredhcp: + image: ghcr.io/openchami/coresmd:v0.0.5 + container_name: coredhcp + hostname: coredhcp + network_mode: host + cap_add: + - NET_ADMIN + volumes: + - ./configs/coredhcp.yaml:/etc/coredhcp/config.yaml:ro + - step-root-ca:/root_ca/:ro + command: + - "-L" + - "debug" + healthcheck: + test: pgrep coredhcp + interval: 5s + timeout: 10s + retries: 60 + depends_on: + smd: + condition: service_healthy diff --git a/quickstart/generate-configs.sh b/quickstart/generate-configs.sh index e934cc9..ba9da17 100755 --- a/quickstart/generate-configs.sh +++ b/quickstart/generate-configs.sh @@ -2,65 +2,69 @@ set -euo pipefail -if [ -f .env ] -then - echo "A config file (.env) exists. Delete to generate a new one" - exit 1 -fi - -if [ -f configs/opaal.yaml ] -then - echo "An OPAAL config (configs/opaal.yaml) exists. Delete to generate a new one" -fi - usage() { - echo "Usage: $0 system-name [system-domain]" - echo "Example: $0 foobar openchami.cluster" - echo "Example: $0 foobar" + echo "Usage: [options] $0" echo "" - echo "Generate configuration for OpenCHAMI quickstart." + echo "Generate configuration for OpenCHAMI quickstart using example" + echo "parameters." echo "" - echo "ARGUMENTS:" - echo " system-name Subdomain of system to use in certificate and config" - echo " generation. E.g. .openchami.cluster" - echo " system-domain (OPTIONAL) Domain of system to use in certificate and" - echo " config generation. Defaults to openchami.cluster" + echo "OPTIONS:" + echo " -h Print this usage message to stdout." + echo " -f Force overwriting config files." } -# Parse system name (required arg). -if [ -z "${1+x}" ] +while getopts "fh" opt; do + case "${opt}" in + f) + FORCE_OVERWRITE=true + ;; + h) + usage + exit + ;; + *) + usage >&2 + exit + ;; + esac +done +shift $((OPTIND-1)) + +if [ -f .env ] && [ -z "${FORCE_OVERWRITE+x}" ] then - usage >&2 - exit 1 + echo "A config file (.env) exists. Delete to generate a new one or -f to overwrite." + file_exists=true fi -SYSNAME="$1" -# Parse system domain (optional arg). -SYSDOMAIN="openchami.cluster" -if [ -n "${2+x}" ] +if [ -f configs/opaal.yaml ] && [ -z "${FORCE_OVERWRITE+x}" ] +then + echo "An OPAAL config (configs/opaal.yaml) exists. Delete to generate a new one or -f to overwrite." + file_exists=true +fi + +if [ -f configs/coredhcp.yaml ] && [ -z "${FORCE_OVERWRITE+x}" ] then - SYSDOMAIN="$2" + echo "A CoreDHCP config (configs/coredhcp.yaml) exists. Delete to generate a new one or -f to overwrite." + file_exists=true fi +if [ -n "${file_exists+x}" ]; then exit 1; fi + +SYSNAME=foobar +SYSDOMAIN="openchami.cluster" + +# Check for required commands if [[ ! -x $(command -v jq) ]] then echo "Command \"jq\" not found" exit 1 fi - if [[ ! -x $(command -v sed) ]] then echo "Command \"sed\" not found" exit 1 fi -get_eth0_ipv4() { - local ipv4 - local first_eth=$(ip -j addr | jq -c '.[]' | grep UP |grep -v veth | grep -v LOOPBACK |grep -v br- |grep -v NO-CARRIER | grep -E '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' | head -n 1 | jq -rc '.ifname') - ipv4=$(ip -o -4 addr show $first_eth | awk '{print $4}') - echo "${ipv4%/*}" -} - generate_random_alphanumeric() { local num_chars=${1:-32} dd bs=512 if=/dev/urandom count=1 2>/dev/null | tr -dc '[:alnum:]' | fold -w "${num_chars}" | head -n 1 @@ -73,7 +77,12 @@ generate_random_alphanumeric() { sed \ -e "s//${SYSNAME}/g" \ -e "s//${SYSDOMAIN}/g" \ - configs/opaal-template.yaml > configs/opaal.yaml +configs/opaal-template.yaml > configs/opaal.yaml + +# Generate CoreDHCP configuration from configs/coredhcp-template.yaml. +sed \ + -e "s||https://${SYSNAME}.${SYSDOMAIN}:8443|g" \ + configs/coredhcp-template.yaml > configs/coredhcp.yaml # Set the system name cat > .env <