test: add offline unit test for yanger and cli-pretty

In this patch we add some arbitrary system output files which contains
interface data. Such as the output from "ip link" and "ethtool". These
files are used by yanger instead of running the actual commands.

Yanger then generates a lot of YANG json data, which are merged using
jq into something that looks like the sysrepo operational data on
a running system. This data is then passed though cli-pretty and its
ouput is compared to the static expected output in the .txt files in
cli-output.

Here's a basic flow overview.
* system-output/foo.json -> yanger ->> data
* system-output/bar.json -> yanger ->> data
* data -> merge -> sysrepo-operational-data
* sysrepo-operational-data -> cli-pretty -> output.txt
* compare output.txt too cli-output/show-foobar.txt

The result of this is that if something changes in yanger OR in
cli-pretty. The corresponding system-output file needs to be updated
for the test to pass. This is indented to catch regression where the
output is unintentionally changed.

Signed-off-by: Richard Alpe <[email protected]>

board/netconf/rootfs/lib/infix/yanger

@@ -153,8 +153,11 @@ def add_ipv4_route(routes):
         out['route'].append(new)
     insert(routes, 'routes', out)
 
-def add_ip_link(ifname, iface_out):
-    cmd = ['ip', '-s', '-d', '-j', 'link', 'show', 'dev', ifname]
+def add_ip_link(ifname, iface_out, test):
+    if test:
+        cmd = ['cat', f"{test}/ip-link-show-dev-{ifname}.json"]
+    else:
+        cmd = ['ip', '-s', '-d', '-j', 'link', 'show', 'dev', ifname]
 
     data = run_json_cmd(cmd)
     if len(data) != 1:
@@ -202,8 +205,11 @@ def add_ip_link(ifname, iface_out):
     if val is not None:
         insert(iface_out, "statistics", "in-octets", str(val))
 
-def add_ip_addr(ifname, iface_out):
-    cmd = ['ip', '-j', 'addr', 'show', 'dev', ifname]
+def add_ip_addr(ifname, iface_out, test):
+    if test:
+        cmd = ['cat', f"{test}/ip-addr-show-dev-{ifname}.json"]
+    else:
+        cmd = ['ip', '-j', 'addr', 'show', 'dev', ifname]
 
     data = run_json_cmd(cmd)
     if len(data) != 1:
@@ -249,8 +255,11 @@ def add_ip_addr(ifname, iface_out):
         insert(iface_out, "ietf-ip:ipv4", "address", inet)
         insert(iface_out, "ietf-ip:ipv6", "address", inet6)
 
-def add_ethtool_groups(ifname, iface_out):
-    cmd = ['ethtool', '--json', '-S', ifname, '--all-groups']
+def add_ethtool_groups(ifname, iface_out, test):
+    if test:
+        cmd = ['cat', f"{test}/ethtool-groups-{ifname}.json"]
+    else:
+        cmd = ['ethtool', '--json', '-S', ifname, '--all-groups']
 
     data = run_json_cmd(cmd)
     if len(data) != 1:
@@ -324,11 +333,15 @@ def add_ethtool_groups(ifname, iface_out):
         if found:
             frame['in-error-oversize-frames'] = str(tot)
 
-def add_ethtool_std(ifname, iface_out):
-    cmd = ['ethtool', ifname]
+def add_ethtool_std(ifname, iface_out, test):
     keys = ['Speed', 'Duplex', 'Auto-negotiation']
     result = {}
 
+    if test:
+        cmd = ['cat', f"{test}/ethtool-{ifname}.txt"]
+    else:
+        cmd = ['ethtool', ifname]
+
     lines = run_cmd(cmd)
     for line in lines:
         line = line.strip()
@@ -363,6 +376,7 @@ if __name__ == "__main__":
     parser = argparse.ArgumentParser(description="YANG data creator")
     parser.add_argument("model", help="IETF Model")
     parser.add_argument("-p", "--param", default=None, help="Model dependant parameter")
+    parser.add_argument("-t", "--test", default=None, help="Test data base path")
     args = parser.parse_args()
 
     if (args.model == 'ietf-interfaces'):
@@ -383,10 +397,10 @@ if __name__ == "__main__":
         ifname = args.param
         iface_out = yang_data['ietf-interfaces:interfaces']['interface'][0]
 
-        add_ip_link(ifname, iface_out)
-        add_ip_addr(ifname, iface_out)
-        add_ethtool_groups(ifname, iface_out)
-        add_ethtool_std(ifname, iface_out)
+        add_ip_link(ifname, iface_out, args.test)
+        add_ip_addr(ifname, iface_out, args.test)
+        add_ethtool_groups(ifname, iface_out, args.test)
+        add_ethtool_std(ifname, iface_out, args.test)
     elif (args.model == 'ietf-routing'):
         yang_data = {
             "ietf-routing:routing": {

test/case/all-unit.yaml

@@ -1,5 +1,9 @@
 ---
 # Tests in this suite can be run on localhost without a target environment
 
+- name:  cli-output
+  suite: cli/all.yaml
+
 - name:  cli-pretty
   suite: cli_pretty/all.yaml
+

test/case/cli/all.yaml

@@ -0,0 +1,3 @@
+---
+- case: run.sh
+  name: "cli-check-output"

test/case/cli/cli-output/show-interface-br0.txt

@@ -0,0 +1,22 @@
+name                : br0
+index               : 7
+mtu                 : 1500
+operational status  : up
+auto-negotiation    : on
+duplex              : full
+speed               : 10000
+physical address    : 02:00:00:00:00:00
+ipv4 addresses      :
+ipv6 addresses      :
+in-octets           : 0
+out-octets          : 3158
+
+eth-out-frames               : 713
+eth-out-multicast-frames     : 605
+eth-out-broadcast-frames     : 69
+eth-in-frames                : 418
+eth-in-multicast-frames      : 336
+eth-in-broadcast-frames      : 46
+eth-in-error-fcs-frames      : 0
+eth-in-total-frames          : 418
+eth-in-error-oversize-frames : 0

test/case/cli/cli-output/show-interface-e0.txt

@@ -0,0 +1,22 @@
+name                : e0
+index               : 2
+mtu                 : 1500
+operational status  : up
+auto-negotiation    : on
+duplex              : full
+speed               : 10000
+physical address    : 02:00:00:00:00:00
+ipv4 addresses      :
+ipv6 addresses      :
+in-octets           : 20891
+out-octets          : 6537
+
+eth-out-frames               : 713
+eth-out-multicast-frames     : 605
+eth-out-broadcast-frames     : 69
+eth-in-frames                : 418
+eth-in-multicast-frames      : 336
+eth-in-broadcast-frames      : 46
+eth-in-error-fcs-frames      : 0
+eth-in-total-frames          : 418
+eth-in-error-oversize-frames : 0

test/case/cli/cli-output/show-interface-e1.txt

@@ -0,0 +1,22 @@
+name                : e1
+index               : 3
+mtu                 : 1500
+operational status  : up
+auto-negotiation    : on
+duplex              : full
+speed               : 10000
+physical address    : 02:00:00:00:00:01
+ipv4 addresses      :
+ipv6 addresses      :
+in-octets           : 24849
+out-octets          : 1397
+
+eth-out-frames               : 713
+eth-out-multicast-frames     : 605
+eth-out-broadcast-frames     : 69
+eth-in-frames                : 418
+eth-in-multicast-frames      : 336
+eth-in-broadcast-frames      : 46
+eth-in-error-fcs-frames      : 0
+eth-in-total-frames          : 418
+eth-in-error-oversize-frames : 0

test/case/cli/cli-output/show-interface-e2.txt

@@ -0,0 +1,22 @@
+name                : e2
+index               : 4
+mtu                 : 1500
+operational status  : up
+auto-negotiation    : on
+duplex              : full
+speed               : 10000
+physical address    : 02:00:00:00:00:02
+ipv4 addresses      :
+ipv6 addresses      :
+in-octets           : 17582
+out-octets          : 1327
+
+eth-out-frames               : 713
+eth-out-multicast-frames     : 605
+eth-out-broadcast-frames     : 69
+eth-in-frames                : 418
+eth-in-multicast-frames      : 336
+eth-in-broadcast-frames      : 46
+eth-in-error-fcs-frames      : 0
+eth-in-total-frames          : 418
+eth-in-error-oversize-frames : 0

test/case/cli/cli-output/show-interface-e3.txt

@@ -0,0 +1,22 @@
+name                : e3
+index               : 5
+mtu                 : 1500
+operational status  : up
+auto-negotiation    : on
+duplex              : full
+speed               : 10000
+physical address    : 02:00:00:00:00:03
+ipv4 addresses      :
+ipv6 addresses      :
+in-octets           : 15057
+out-octets          : 1327
+
+eth-out-frames               : 713
+eth-out-multicast-frames     : 605
+eth-out-broadcast-frames     : 69
+eth-in-frames                : 418
+eth-in-multicast-frames      : 336
+eth-in-broadcast-frames      : 46
+eth-in-error-fcs-frames      : 0
+eth-in-total-frames          : 418
+eth-in-error-oversize-frames : 0

test/case/cli/cli-output/show-interface-e4.txt

@@ -0,0 +1,22 @@
+name                : e4
+index               : 6
+mtu                 : 1500
+operational status  : up
+auto-negotiation    : on
+duplex              : full
+speed               : 10000
+physical address    : 02:00:00:00:00:04
+ipv4 addresses      :
+ipv6 addresses      :
+in-octets           : 19022
+out-octets          : 1327
+
+eth-out-frames               : 713
+eth-out-multicast-frames     : 605
+eth-out-broadcast-frames     : 69
+eth-in-frames                : 418
+eth-in-multicast-frames      : 336
+eth-in-broadcast-frames      : 46
+eth-in-error-fcs-frames      : 0
+eth-in-total-frames          : 418
+eth-in-error-oversize-frames : 0

test/case/cli/cli-output/show-interfaces.txt

@@ -0,0 +1,7 @@
+INTERFACE       PROTOCOL   STATE       DATA                                     
+br0             ethernet   UP          02:00:00:00:00:00                        
+├ e0            ethernet   UP          02:00:00:00:00:00                        
+└ e1            ethernet   UP          02:00:00:00:00:01                        
+e2              ethernet   UP          02:00:00:00:00:02                        
+e3              ethernet   UP          02:00:00:00:00:03                        
+e4              ethernet   UP          02:00:00:00:00:04                        

test/case/cli/run.sh

@@ -0,0 +1,78 @@
+#!/bin/sh
+
+SCRIPT_PATH="$(dirname "$(readlink -f "$0")")"
+ROOT_PATH="$SCRIPT_PATH/../../../"
+CLI_OUTPUT_PATH="$SCRIPT_PATH/cli-output/"
+
+CLI_PRETTY_TOOL="$ROOT_PATH/board/netconf/rootfs/lib/infix/cli-pretty"
+SR_EMULATOR_TOOL="$SCRIPT_PATH/sysrepo-emulator.sh"
+
+CLI_OUTPUT_FILE="$(mktemp)"
+
+TEST=1
+
+cleanup() {
+    rm -f "$CLI_OUTPUT_FILE"
+}
+trap cleanup EXIT
+
+ok() {
+    echo "ok $TEST - $1"
+    TEST=$((TEST + 1))
+}
+
+fail() {
+    echo "not ok $TEST - $1"
+    exit 1
+}
+
+print_update_txt() {
+    echo
+    echo "# CLI output has changed. This might not be an error if you intentionally"
+    echo "# changed something in yanger or cli-pretty. If you did, you need to update"
+    echo "# the template file."
+    echo
+    echo "# Here's how you update the CLI output templates:"
+    echo "# $SCRIPT_PATH/run.sh update"
+    echo
+    echo "# Check the result"
+    echo "# git diff"
+    echo
+    echo "# Then finish up by committing the new template"
+    echo
+}
+
+if [ ! -e "$CLI_PRETTY_TOOL" ]; then
+    echo "Error, cli-pretty tool not found"
+    exit 1
+fi
+
+if [ $# -eq 1 ] && [ $1 = "update" ]; then
+    "$SR_EMULATOR_TOOL" | "$CLI_PRETTY_TOOL" "ietf-interfaces" > "$CLI_OUTPUT_PATH/show-interfaces.txt"
+    for iface in "br0" "e0" "e1" "e2" "e3" "e4"; do
+        "$SR_EMULATOR_TOOL" | "$CLI_PRETTY_TOOL" "ietf-interfaces" -n "$iface" \
+            > "$CLI_OUTPUT_PATH/show-interface-${iface}.txt"
+    done
+    echo "All files updated. Check git diff and commit if they look OK"
+    exit 0
+fi
+
+echo "1..7"
+
+# Show interfaces
+"$SR_EMULATOR_TOOL" | "$CLI_PRETTY_TOOL" "ietf-interfaces" > "$CLI_OUTPUT_FILE"
+if ! diff -u "$CLI_OUTPUT_PATH/show-interfaces.txt" "$CLI_OUTPUT_FILE"; then
+    print_update_txt
+    fail "\"show interfaces\" output has changed"
+fi
+ok "\"show interfaces\" output looks intact"
+
+# Show detailed interfaces
+for iface in "br0" "e0" "e1" "e2" "e3" "e4"; do
+    "$SR_EMULATOR_TOOL" | "$CLI_PRETTY_TOOL" "ietf-interfaces" -n "$iface" > "$CLI_OUTPUT_FILE"
+    if ! diff -u "$CLI_OUTPUT_PATH/show-interface-${iface}.txt" "$CLI_OUTPUT_FILE"; then
+        print_update_txt
+        fail "\"show interface name $iface\" output has changed"
+    fi
+    ok "\"show interface name $iface\" output looks intact"
+done

test/case/cli/sysrepo-emulator.sh

@@ -0,0 +1,37 @@
+#!/bin/sh
+
+SCRIPT_PATH="$(dirname "$(readlink -f "$0")")"
+ROOT_PATH="$SCRIPT_PATH/../../../"
+
+YANGER_TOOL="$ROOT_PATH/board/netconf/rootfs/lib/infix/yanger"
+
+INTERFACES="br0 e0 e1 e2 e3 e4"
+
+YANGER_OUTPUT_FILE="$(mktemp)"
+
+cleanup() {
+    rm -f "$YANGER_OUTPUT_FILE"
+}
+trap cleanup EXIT
+
+if [ ! -e "$YANGER_TOOL" ]; then
+    echo "Error, yanger tool not found"
+    exit 1
+fi
+
+for iface in $INTERFACES; do
+    if ! "$YANGER_TOOL" "ietf-interfaces" \
+          -t "$SCRIPT_PATH/system-output/" \
+          -p "$iface" >> "$YANGER_OUTPUT_FILE"; then
+        echo "Error, running yanger for interface $iface" >&2
+        exit 1
+    fi
+done
+
+if ! jq -s 'reduce .[] as $item
+       ({}; .["ietf-interfaces:interfaces"].interface +=
+        $item["ietf-interfaces:interfaces"].interface)' \
+      "$YANGER_OUTPUT_FILE"; then
+    echo "Error, merging yanger output data" >&2
+    exit 1
+fi

test/case/cli/system-output/ethtool-br0.txt

@@ -0,0 +1,39 @@
+Settings for e0:
+	Supported ports: [  ]
+	Supported link modes:   10baseT/Half 10baseT/Full
+	                        100baseT/Half 100baseT/Full
+	                        1000baseT/Full
+	                        10000baseT/Full
+	                        2500baseT/Full
+	                        5000baseT/Full
+	Supported pause frame use: Symmetric
+	Supports auto-negotiation: Yes
+	Supported FEC modes: Not reported
+	Advertised link modes:  10baseT/Half 10baseT/Full
+	                        100baseT/Half 100baseT/Full
+	                        1000baseT/Full
+	                        10000baseT/Full
+	                        2500baseT/Full
+	                        5000baseT/Full
+	Advertised pause frame use: Symmetric
+	Advertised auto-negotiation: Yes
+	Advertised FEC modes: Not reported
+	Link partner advertised link modes:  10baseT/Half 10baseT/Full
+	                                     100baseT/Half 100baseT/Full
+	                                     1000baseT/Full
+	                                     10000baseT/Full
+	                                     2500baseT/Full
+	                                     5000baseT/Full
+	Link partner advertised pause frame use: Symmetric
+	Link partner advertised auto-negotiation: Yes
+	Link partner advertised FEC modes: Not reported
+	Speed: 10000Mb/s
+	Duplex: Full
+	Auto-negotiation: on
+	Port: Twisted Pair
+	PHYAD: 4
+	Transceiver: external
+	MDI-X: on (auto)
+	Supports Wake-on: g
+	Wake-on: d
+	Link detected: yes