Validation effort for PC Engines v24.05.00.01

[pietrushnic]

In this issue, I will report the progress of validation. The plan is as follows:

• Fix open-firmware-rte 3mdeb private test suite (predecessor of OSFV) so it can run tests on apu2/3/4/6 in Dasharo Certification Lab
• Retest v4.19.0.1 and publish results for apu2/3/4 (apu6 was not retested)
  • apu2
  • apu3
  • apu4
• Retest v4.19.0.1 based on iter11 cleanup and publish results (apu6 was not retested)
  • apu2
  • apu3
  • apu4
• Rebase patches using dasharo-pq to produce v24.05.00.01
• Test v24.05.00.01 and publish results
  • apu2
  • apu3
  • apu4
  • apu6

[pietrushnic]

apu2 v4.19.01 retest results were published, there seem to be at least two problems:

• SBO1.7 BIOS WP enable/disable test had to be performed manually, it requires a fix in infrastructure, although it seems that on iter11 test passed automatically - this has to be double-checked
• WOL1.1 seems to be affected by APU[2-6] WOL doesn't work pcengines/apu2-documentation#282, which means there is a need for fixes in infrastructure to make sure this test passes

[pietrushnic]

apu4 retest and and iter11 test results where added, identified issues:

• COM2 probably connected incorrectly, leading to SBO1.8 and SBO1.9 failures
• DEB1.4 fails due to issues with NFS; apu4 should be checked in Dasharo Certification Lab to confirm it is no connection different than apu2, which passes this test without issues
• all network interfaces have to be connected to pass tests NET1.1-1.3
• WOL1.1 has the same problem as in apu2 v4.19.0.1, interestingly
• GPIO1.2 fails, which may indicate an incorrect GPIO connection in the lab

It doesn't make sense to focus too much on the v4.19.0.1 retest when v4.19.0.1 cleanup provides such good results, so whatever infrastructure-related failures happened on apu4 during the retest, we leave it as is.

[pietrushnic]

Test results for iter11 on apu2 were added to the spreadsheet. apu2 has much fewer issues, it is affected by following failures:

• DEB1.4
• WOL1.1
• NET1.1-1.3

[pietrushnic]

apu2 v24.05.00.01 validatin:

• SBO1.6 UART C/D feature enable/disable test | FAIL |
  Evaluating IF condition failed: Evaluating expression '24-9-g1e4848a-dirty > 20' failed: NameError: name 'g1e4848a' is not
  defined nor importable as module
• USB1.3 reboot test | FAIL |
  No match found for 'reboot: Restarting system' in 5 minutes. Output:
  • NFS crash

[pietrushnic]

apu4 v24.05.00.01 validation:

• SBO1.6 UART C/D feature enable/disable test | FAIL |
  Resolving variable '${version.split(".")[2]}' failed: IndexError: list index out of range
• SBO1.7 BIOS WP feature enable/disable test | FAIL |
  does not contain 'Enable BIOS write protect - Currently Enabled'
• SBO3.4 Reverse order of PCI addresses - Interface Order In iPXE | FAIL |
  No match found for 'F10' in 3 minutes. Output:
• SBO1.8 Enable COM2 serial output redirection | FAIL |
  No match found for 'F10' in 3 minutes. Output:
• SBO1.9 Disable COM2 serial output redirection | SKIP |
  Serial output redirection is set to port 13541
• USB1.3 reboot test | FAIL |
  No match found for 'reboot: Restarting system' in 5 minutes. Output:
  • NFS issue
• SDD1.3 reboot test | FAIL |
  No match found for 'reboot: Restarting system' in 5 minutes. Output:
  • NFS issue
• DNI1.1 Install Debian i386 on USB storage | FAIL |
  No match found for 'Select and install software' in 1 hour 30 minutes. Output:

[            (1*installer)  2 shell  3 shell  4- log           ][ Jun 08  3:48 ]





     lqqqqqqqqqqqqqqqqqqu [!!] Install the base system tqqqqqqqqqqqqqqqqqqk
     x                                                                    x
     x               Unable to install the selected kernel                x
     x An error was returned while trying to install the kernel into the  x
     x target system.                                                     x
     x                                                                    x
     x Kernel package: 'linux-image-686-pae'.                             x
     x                                                                    x
     x Check /var/log/syslog or see virtual console 4 for the details.    x
     x                                                                    x
     x     <Go Back>                                       <Continue>     x
     x                                                                    x
     mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj

• DEB1.3 Debian from iPXE 4.16.y | FAIL |
  No match found for 'F10' in 3 minutes. Output:
• DEB1.4 50x reboots to Debian 4.14.y booted over iPXE | FAIL |
  No match found for 'F10' in 3 minutes. Output:
• NET1.2 Validate net2 interface | FAIL |
  Keyword 'iPXE dhcp' failed after retrying 3 times. The last error was: No match found for 'Configuring' in 30 seconds. Output:
  dhcp net2
  Waiting for link-up on net2................. Down (https://ipxe.org/38086101)
  iPXE>
• NET1.3 Validate net3 interface | SKIP |
  Platform doesn’t have enough ethernet interfaces available.
• ATA1.0 ATA UDMA coldboot test | FAIL |
  No match found for 'Loading Linux' in 3 minutes. Output:

Booting from Hard Disk...

• ATA1.1 ATA UDMA warmboot test | FAIL |
  No match found for 'Loading Linux' in 3 minutes. Output:

Booting from Hard Disk...

• ATA1.2 Get disk and system info | FAIL |
  No match found for 'login:' in 5 minutes. Output:

Booting from Hard Disk...

[pietrushnic]

SBO1.6 test most likely need fix because of the versioning scheme change, test fails.

[pietrushnic]

With following patch SBO1.6 pass:

diff --git a/keywords.robot b/keywords.robot
index 0c271c1ba166..d0fbdf5a916b 100644
--- a/keywords.robot
+++ b/keywords.robot
@@ -575,17 +575,18 @@ Disable UART option
     Return From Keyword If    not ${sortbootorder_support}
     ${output}=    Enter sortbootorder
     ${version}=    Get sortbootorder version    ${output}
+    ${maj_ver}=    Set Variable    ${version.split(".")[0]}
     ${patch_ver}=    Set Variable    ${version.split(".")[2]}
-    ${lines}=    IF    ${patch_ver} > ${20}
+    ${lines}=    IF    ${patch_ver} > ${20} or ${maj_ver} >= ${24}
     ...    Get Lines Containing String    ${output}    Toggle ${option_str} / GPIO
     ...    ELSE    Get Lines Containing String    ${output}    ${option_str}
-    ${option_str}=     Set Variable If    ${patch_ver} > ${20} and '${option_str}' == 'UART C'    GPIO[0..7]    ${option_str}
-    ${option_str}=     Set Variable If    ${patch_ver} > ${20} and '${option_str}' == 'UART D'    GPIO[10..17]    ${option_str}
-    ${res}=    IF    ${patch_ver} > ${20}
+    ${option_str}=     Set Variable If    (${patch_ver} > ${20} or ${maj_ver} >= ${24}) and '${option_str}' == 'UART C'    GPIO[0..7]    ${option_str}
+    ${option_str}=     Set Variable If    (${patch_ver} > ${20} or ${maj_ver} >= ${24}) and '${option_str}' == 'UART D'    GPIO[10..17]    ${option_str}
+    ${res}=    IF    ${patch_ver} > ${20} or ${maj_ver} >= ${24}
     ...    Evaluate    "${option_str} - Currently Enabled" in """${lines}"""
     ...    ELSE    Evaluate    "${option_str} - Currently Disabled" in """${lines}"""
     IF    ${res}    Telnet.Write Bare    x
-    IF    ${patch_ver} > ${20}
+    IF    ${patch_ver} > ${20} or ${maj_ver} >= ${24}
     ...    Run Keyword If    not ${res}    Verify option change    ${lines}    ${option_str}    Enabled
     ...    ELSE IF    not ${res}    Verify option change    ${lines}    ${option_str}    Disabled

[pietrushnic]

With the above patch also SBO1.8 pass on apu4

[pietrushnic]

Regarding NFS issues most of those happen when there is concurrent acces e.g. two tests mounting NFS running in parallel.

[pietrushnic]

Quite a lot of issues were fixed by switching to DTS instead of using Debian over iPXE+NFS.

Outstanding issues for apu2/3/4/6 are:

• WOL1.1, which is very likely an infrastructure issue
• WP, which is very likely an infrastructure issue

Another issue that was not addressed is that on some platforms (e.g., apu6), I cannot pass 50x Debian boot (DEB1.4) because NFS is not responding even if I try to run those tests only on one platform. It's very likely some weird issue with infrastructure.

[pietrushnic]

WOL was fixed by using direct MAC addresses of interfaces instead of retrieving them automatically. Apparently, the function for automatic retrieval had a bug.

[pietrushnic]

WP was also fixed; it happened that power off was done too fast, and sortbootorder didn't have a chance to write settings to flash. Extending the time between changing the option and power fixed the issue.