Introduce edge gateway load balancers

[Didainius]

Why

We have a need (#182) to introduce edge gateway load balancing capabilities.

Details

This is a master issue for work required to introduce Edge gateway load balancer (LB) support. Because this is a complex task and multiple resources will have to be created there will be child issues for each of the planned resources.

Prerequisites

• Edge gateway up (with Load balancer enabled in Edge gateway services -> Load balancer -> Global configuration). At the current state it is manual. Once we have ability to create it this would be a feature toggle attribute.
• Existing vApp and VMs (to be used as load balancer pool members)

Considerations

• Have to check if it supports API 27.0 on all vCDs (does it depend on NSX edge gateway version?)

Proposal

The plan is to mimic Edge gateway UI in vCD. It has 6 tabs for configuration (one of them being just enable/disable which should happen on edge gateway resource once we have it). The other five are listed below. To make a working solution without manual intervention (provided edge gateway is created already) we need at least 4 of them.

• Application profiles
• Service monitoring
• Pools
• Application rules (Not a mandatory feature to have working LB)
• Virtual servers

These objects reference each other to finally make a virtual server which is the main object for load balancer.

Each of them is described well in the vCD Load Balancing documentation.

Sample HCL configuration

# Load balancer -> Service Monitoring (in Edge gateway services)
resource "vcd_lb_service_monitor" "service-monitor" {    
    org = "my-org"     # Optional
    vdc = "my-org-vdc" # Optional

    # Dependencies
    edge_gateway = ""

    name = ""
    interval = ""
    timeout = ""
    max_retries = ""
    type = ""               # Better name could be ?protocol? or "protocol_type". HTTP, HTTPS, TCP, ICMP, or UDP
    expected_response = ""  # if type="HTTP" || type="HTTPS" 
    http_method = ""        # if type="HTTP" || type="HTTPS" GET, POST, PUT, etc....
    url = ""                # if type="HTTP" || type="HTTPS"
    send = ""               # if type="HTTP" || type="HTTPS" || type="UDP"
    receive = ""            # if send != "" .Response from send. String to be matched in the response content.
    # extension seems like very natural block with key/value, but needs further evaluation if key names would alway be compatible with terraform key naming
    extension {             # Extension options listed https://docs.vmware.com/en/vCloud-Director/9.7/com.vmware.vcloud.spportal.doc/GUID-DE3EDF95-1A8E-4D2F-B7B1-1254E9420868.html
        delay   = "2"
        max-age = "10"
    }
}


#  Load balancer -> Pools (in Edge gateway services)
#resource "vcd_lb_server_pool" "pool" {
resource "vcd_lb_pool" "pool" {                             
    org = "my-org"     # Optional
    vdc = "my-org-vdc" # Optional

    # Dependencies
    edge_gateway = ""

    # Optional dependency
    monitor_id = "${vcd_lb_service_monitor.service-monitor.id}"

    name = ""
    description = ""
    algorithm = ""      # ROUND_ROBIN, IP_HASH, ...

    transparent = ""    # boolean

    # Can such list of nested blocks be easily created with iteration using HCL?
    # This nees to be friendly for population if one has resource vcd_vapp_vm with count>1
    member {
        name = ""
        ip = ""
        port = ""
        monitor_port = ""
        weight = ""
        min_connections = ""
        max_connections = ""
    }

    member {
        name = ""
        ip = ""
        port = ""
        monitor_port = ""
        weight = ""
        min_connections = ""
        max_connections = ""   
    }

}

# Load balancer -> Application Profiles (in Edge gateway services)
resource "vcd_lb_application_profile" "app-prof" {          
    org = "my-org"     # Optional
    vdc = "my-org-vdc" # Optional

    edge_gateway = ""  # Dependency

    name = ""
    type = ""                   # TCP, UDP, HTTP, HTTPS
    enable_ssl_passthrough = "" # if type=HTTPS
    http_redirect_url = ""      # string
    persistence = ""            # Source IP and , Cookie (MSRDP for type TCP)
    cookie_name = ""                        # If persistence=Cookie
    cookie_mode = ""                        # If persistence=Cookie
    persistence_expiration_time = ""        # time
    insert_x_fotwarded_for_http_header = "" # boolean
    enable_pool_side_ssl = ""               # bool

    # Only for type=HTTPS 
    virtual_server_certificates {
        server_certificate = ""
        ca_certificate     = ""
        crl = ""
    }

    # Only for enable_pool_side_ssl=true
    pool_certificates {
        server_certificate = ""
        ca_certificate     = ""
        crl = ""
    }
}



# Load balancer -> Virtual Servers (in Edge gateway services)
resource "vcd_lb_virtual_server" "lb-virtual-server" {      
    org = "my-org"     # Optional
    vdc = "my-org-vdc" # Optional

    # Dependencies
    edge_gateway = ""

    application_profile_id = "${vcd_lb_application_profile.app-prof.id}"
    pool_id = "${vcd_lb_pool.pool.id}"

    enabled = ""             # boolean
    enable_acceleration = "" # boolean
    name = ""
    description =         "" # optional
    
    ip = ""                  # must be one of the networks attached to edge gateway
    network = ""             # because of IP address we may want/need to refer exact network

    protocol = ""            # HTTP, HTTPS, TCP, UDP
    port     = ""            # port number to listen on

    connection_limit = ""
    connection_rate_limit = ""

    # Advanced settings tab (optional)
    applcations_rules = "[${vcd_lb_application_rule.app-rule1.id}]"  # list of application rule ids
}

# Optional and can be split for future
# Load balancer -> Application rules (in Edge gateway services) 
resource "vcd_lb_application_rule" "app-rule1" {             
    org = "my-org"     # Optional
    vdc = "my-org-vdc" # Optional

    # Dependencies
    edge_gateway = ""
    
    name = ""
    script = ""
}

Testing considerations

To test load balancing we'd have to actually load balance some service and do HTTP calls to it. Therefore we need to be able to setup vApp with 2 VMs and launch a simple service (like http server). One option is that Photon OS has python3 by default and it allows to spawn a server with python3 -m http.server. The important part would be to setup a customization script and spawn such configs inside test.

May be related for testing purpose: #139

References

Good read about LB options: https://docs.vmware.com/en/vCloud-Director/9.7/com.vmware.vcloud.spportal.doc/GUID-2E6696DB-5F03-411C-A7EA-358E50EBB992.html

[dataclouder]

Something worth thinking about:
given that these components can't be used stand-alone, but make sense only if used together, would it be a good idea to make this feature a single resources?

The advantage of this approach is that we can have a strict check on the needed components.

Alternatively, we can implement these four components separately, but use them through a glue component vcd_lb resource.

[Didainius]

given that these components can't be used stand-alone, but make sense only if used together, would it be a good idea to make this feature a single resources?

I tend to think that it is better to split them:

• It will be easier to handle transactions (if it was one resource, it would include multiple API calls and we could have more "failed in the middle states"). The rollbacks may become tricky. Potentially the updates as well.
• Some people (for whatever the reasons) may decide to automate only a part of the configuration. For example pools, but not load balancers (like empty vApps without VMs). This would prevent such case.

The advantage of this approach is that we can have a strict check on the needed components.

That is true. But as elements (graph nodes) depend on each other, failure of one element would stop creating it's descendants.

Alternatively, we can implement these four components separately, but use them through a glue component vcd_lb resource.

I would say vcd_lb_virtual_server does the glue work because it references application_profile, pool and applcations_rule. The only missing is service monitor but it is used in pool.

I've done a quick mockup with simply in-lining things which were referenced. This tends to become "beefy" and IMHO quite hard to comprehend (like a function with too many arguments) although I know that all the fields would not ever be used as some conflict with other.

One more thought. If people want to abstract this stuff they can easily create Terraform module and refer it as a unit in other places.

What others think?

resource "vcd_lb" "one-resource" {
  org = "my-org"     # Optional
  vdc = "my-org-vdc" # Optional

  # Dependencies
  edge_gateway = ""

  ### LB virtual server

  enabled               = "" # boolean
  enable_acceleration   = "" # boolean
  name                  = ""
  description           = "" # optional
  ip                    = "" # must be one of the networks attached to edge gateway
  network               = "" # because of IP address we may want/need to refer exact network
  protocol              = "" # HTTP, HTTPS, TCP, UDP
  port                  = "" # port number to listen on
  connection_limit      = ""
  connection_rate_limit = ""
  application_profile {
    name                               = ""
    type                               = "" # TCP, UDP, HTTP, HTTPS
    enable_ssl_passthrough             = "" # if type=HTTPS
    http_redirect_url                  = "" # string
    persistence                        = "" # Source IP and , Cookie (MSRDP for type TCP)
    cookie_name                        = "" # If persistence=Cookie
    cookie_mode                        = "" # If persistence=Cookie
    persistence_expiration_time        = "" # time
    insert_x_fotwarded_for_http_header = "" # boolean
    enable_pool_side_ssl               = "" # bool

    # Only for type=HTTPS 
    virtual_server_certificates {
      server_certificate = ""
      ca_certificate     = ""
      crl                = ""
    }

    # Only for enable_pool_side_ssl=true
    pool_certificates {
      server_certificate = ""
      ca_certificate     = ""
      crl                = ""
    }
  }
  pool {
    name        = ""
    description = ""
    algorithm   = "" # ROUND_ROBIN, IP_HASH, ...

    transparent = "" # boolean

    monitor {
      name              = ""
      interval          = ""
      timeout           = ""
      max_retries       = ""
      type              = "" # Better name could be ?protocol? or "protocol_type". HTTP, HTTPS, TCP, ICMP, or UDP
      expected_response = "" # if type="HTTP" || type="HTTPS" 
      http_method       = "" # if type="HTTP" || type="HTTPS" GET, POST, PUT, etc....
      url               = "" # if type="HTTP" || type="HTTPS"
      send              = "" # if type="HTTP" || type="HTTPS" || type="UDP"
      receive           = "" # if send != "" .Response from send. String to be matched in the response content.

      # extension seems like very natural block with key/value, but needs further evaluation if key names would alway be compatible with terraform key naming
      extension {
        delay   = "2"  # Extension options listed https://docs.vmware.com/en/vCloud-Director/9.7/com.vmware.vcloud.spportal.doc/GUID-DE3EDF95-1A8E-4D2F-B7B1-1254E9420868.html
        max-age = "10"
      }
    }

    # Can such list of nested blocks be easily created with iteration using HCL?
    # This nees to be friendly for population if one has resource vcd_vapp_vm with count>1
    member {
      name            = ""
      ip              = ""
      port            = ""
      monitor_port    = ""
      weight          = ""
      min_connections = ""
      max_connections = ""
    }

    member {
      name            = ""
      ip              = ""
      port            = ""
      monitor_port    = ""
      weight          = ""
      min_connections = ""
      max_connections = ""
    }
  }
  application_rule {
    name   = ""
    script = ""
  }
  applcation_rule {
    name   = ""
    script = ""
  }
}

[lvirbalas]

What others think?

Separate resources.

[dataclouder]

If the virtual server resource can act as glue, thus checking that the needed components are set, let's implement them as separate ones.

[Didainius]

Load balancers are release in 2.4.0