Performance Tests

Scale Tests

Test many agents interconnected

// TBD

Stress Tests

Test config speed for 20000+ interface+routes per second

In the repository tests/perf

Test many (65K+) IP addresses on loopback interface

Inspired by tests/perf

Test code

func (p *GRPCStressPlugin) runGRPCAssignIPAddressToLoopbackInterface(client configurator.ConfiguratorClient) {
        p.Log.Infof("Configuring loopback interfaces..")

        var (
                 cntIP  int
         )
        cntIP = 0
        loopbackIPs := make([]string, *numIPs)
        for i:=1; i<256; i++ {
                for j:=1; j<256; j++ {
                        for k:=1; k<256; k++ {
                                loopbackIPs[cntIP]=  strconv.Itoa(i) +"."+ strconv.Itoa(j) +"."+ strconv.Itoa(k) + ".100/24"
                                cntIP = cntIP + 1
                                if cntIP >= *numIPs {
                                      goto Exit
                                }
                        }
                }
        }
        Exit:
        memIFRed := &interfaces.Interface{
                Name:        "my_loopback",
                Type:        interfaces.Interface_SOFTWARE_LOOPBACK,
                Enabled:     true,
                PhysAddress: "08:00:27:0f:e0:4e",
                IpAddresses: loopbackIPs,
                Mtu:         9000,
                Vrf:         0,
        }

        ifaces := []*interfaces.Interface{memIFRed}

        ctx, cancel := context.WithTimeout(context.Background(), reqTimeout)
        _, err := client.Update(ctx, &configurator.UpdateRequest{
                Update: &configurator.Config{
                        VppConfig: &vpp.ConfigData{
                                Interfaces: ifaces,
                        },
                },
                FullResync: true,
        })
        if err != nil {
                log.Fatalln(err)
        }
        cancel()

        if *debug {
                p.Log.Infof("Requesting get..")

                cfg, err := client.Get(context.Background(), &configurator.GetRequest{})
                if err != nil {
                        log.Fatalln(err)
                }
                out, _ := (&jsonpb.Marshaler{Indent: "  "}).MarshalToString(cfg)
                fmt.Printf("Config:\n %+v\n", out)
        }

}

Note: goto command used on purpose - not to have complicated code

Specific tweaks to achieve stable operation while configuring 65k IP addresses:

• I used only one gRPC connection/client
• dialTimeout = time.Second * 2000 (compare with tests/perf/grpc-perf/main.go)
• reqTimeout = time.Second * 2000 (compare with tests/perf/grpc-perf/main.go)
• grpc.conf
  adjusted max-msg-size: 10000000 (in kB - default was 4096)
• Used vpp.conf - copied into /etc/vpp/vpp.conf
• I pinned the proceses to chosen processors (in tests/perf/grpc-perf/test.sh)
  taskset 0x2 vpp -c /etc/vpp/vpp.conf > "$log_vpp" 2>&1 &
  taskset 0x0c vpp-agent -etcd-config=etcd.conf -grpc-config=grpc.conf -govpp-config=govpp.conf > "$log_agent" 2>&1 &
• govpp.conf
  reply-timeout: 10000000000 (compare with plugins/govppmux/plugin_impl_govppmux.go)

Methodology for testing:
The test is run 10 times. Lowest and highest value is eliminated. The average is computed.
The modified file tests/perf/run_all.sh

#!/bin/bash
#set -x

SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

cd ${SCRIPT_DIR}/grpc-perf

source "./test.sh"

declare -a measured_values
declare -a merania

cp /dev/null /tmp/perf-report/results$j
for i in {1..10}
do
                run_test 62025
                measured_values[i]=`cat /tmp/perf-report/report.log | grep done | cut -f 4- -d" " | cut -f 2 -d: | cut -f 1 -d '"'`
                echo ${measured_values[i]} >> /tmp/perf-report/results62025
done


echo "RESULTS FOR 62025 IPs on a loopback:"
cat /tmp/perf-report/results62025 | sort -n | tr '\n' ' '
echo
sed -e "s/ms//" /tmp/perf-report/results62025 | sed -e "s/s//" | sort -n | awk 'NR == 1 { max=$1; min=$1; sum=0 } { if ($1>max) max=$1; if ($1<min) min=$1; sum+=$1;} END {printf "Min: %f\tMax: %f\n", min, max}'
sed -e "s/ms//" /tmp/perf-report/results62025 | sed -e "s/s//" | sort -n | sed -n 2,9p | awk 'NR == 1 { max=$1; min=$1; sum=0 } { if ($1>max) max=$1; if ($1<min) min=$1; sum+=$1;} END {printf "Min: %f\tMax: %f\tAverage: %f\t (NR=%d)\n", min, max, sum/NR, NR}'

Results achieved at the VM with 8GB RAM and 4 cores:
vpp-agent v2.1.0-2-g400e6c3a
VPP v19.04.1-rc0~11-gf60be1151~b52
Average time of configuring 62025 IPs on a loopback interface was 96,603737 seconds

To compare other configuration:

     1 IP     0,0167625s
   500 IPs    0,1332625s
  5500 IPs    1,732562s
 10500 IPs    4,323725s
 15500 IPs    8,14425 s
 20500 IPs   12,803213s
 25500 IPs   17,967525s
 62025 IPs   96,603737s

Note: To work flawlessly while configuring so big number of IPs, vpp-agent needs to have assigned two processors ...