Bare Metal Site Deployment Collection for Ansible

This collection contains roles for deploying network and server infrastructure onto bare metal. The scope of a deployment is referred to as a "site" which correlates to a /16 subnet of the class A private IPv4 network (10.site['id'].0.0/16) as well as a fully qualified domain name (site.example.com). Sites are further divided into subnets defined by a /24 mask and subdomains of the FQDN. So at company "example.com", subnet "sales" with ID 5 in site "office1" and ID 99 would equate to sales.office1.example.com and subnet 10.99.5.0/24. Services are built on top of the network while avoiding circular dependencies so that infrastructure can always be rebuilt from a complete failure. This collection is intended for use by small to mid-size MSPs, IT departments or individuals managing up to 255 physically separate IPv4 networks.

While the roles are designed to be platform-agnostic so they can be reused, different use-cases target specific platforms. As of now, those platforms are (or are planned to be):

• Switch: RouterOS (Mikrotik)
• Gateway/Router/Firewall: OpenBSD
• NAS: ZFS/Gluster (Ubuntu LTS)
• Directory: FreeIPA (Rocky Linux)
• Hypervisor/PaaS: OpenShift (OKD4)

Rough Procedure Outline

Set Up Laptop

1. Plug in ethernet adapter
2. Prioritize default gateway on Wifi (this will allow us to provision over the adapter while maintaining an internet connection).
3. Plug laptop into Switch PXE port via ethernet adapter

Bootstrap Switch

Deploy ssh key to switch and perform minimal initial configuration.

1. Write Vagrant config for bootstrap-sw VM (Debian).
2. Boot/provision bootstrap-sw VM with Mikrotik netinstall binary, firmware and initial config. VM should use the ethernet adapter as a public interface.
3. Run netinstall binary on VM
4. Boot switch with network boot option.
5. Wait for switch to provision.
6. Dump interface names to an inventory template (host_vars).
7. Rinse and repeat for each switch.

Bootstrap Gateway

Deploy ssh key to gateway and perform minimal initial configuration.

1. Write Vagrant config for bootrap-gw VM (OpenBSD).
2. Boot/provision bootstrap-gw VM with DHCP, PXE, TFTP and HTTP services.
3. Boot gateway.
4. Wait for gateway to provision.
5. Dump MAC addresses and interface names to an inventory template (host_vars).
6. Rinse and repeat for each gateway.

Write inventory files (manually)

1. Flesh out the configuration beginning with the inventory templates generated by the bootstrap playbooks. This is essentially the design process where we structure the network.

Provision Network

1. Validate inventory and fill in any missing fields (dev based on mac_addr for instance).
2. Provision switches (I tend to run WAN through the switch instead of directly to the gateway, so the switch needs to be provisioned first).
3. Establish a basic Internet connection (note that the OpenBSD gateways will not have Python yet, so this must be done with raw and script modules only).
4. Install Ansible dependencies (Python)
5. Configure network interfaces
6. Configure pf
7. Configure sshd (need to run this here because the restart ssh handler is a dependency of the next role).
8. Configure the Ansible user
9. Configure routing
10. Configure unbound as a forwarder
11. Configure nsd as an authoritative name server
12. Configure ntpd
13. Configure tftpd (for pxe)
14. Configure httpd (for pxe)
15. Configure dhcpd
16. Configure wildcard certificates via Let's Encrypt DNS verification
17. Configure relayd as a reverse proxy
18. Configure wireguard (will need additional inventory config for this)

Inventory

inventory/fqdn.yml

Ex: inventory/campus1.example.com.yml

This file defines the structure of the network and the relationship between hosts and subdomains. If you've written an Ansible inventory, it should look familiar. The site role will initialize the file but it will need to be fleshed out manually. FQDNs should be used for each host to avoid naming collisions when dealing with multiple sites.

Example

all:
  children:
    office_example_com:
      hosts:
        netinstall_routeros:
        pxe_openbsd:
      children:
        gw:
          hosts:
            gw1.office.example.com:
            gw2.office.example.com:
        net:
          vars:
            subdom: net #use subdomain net for this group
          children:
            sw:
              hosts:
                sw1.net.office.example.com:
                sw2.net.office.example.com:
                sw3.net.office.example.com:
                sw4.net.office.example.com:
                ap1.net.office.example.com:
                ap2.net.office.example.com:
        srv:
          vars:
            subdom: srv #use subdomain srv for this group
          children:
            srv:
              hosts:
                www1.srv.office.example.com:
                ftp1.srv.office.example.com:
                nas1.srv.office.example.com:
                nas2.srv.office.example.com: