Dev jobs fail with "java.lang.IllegalArgumentException: The tools section is required"

[mathias-luedtke]

The ros_canopen dev jobs (ubuntu & debian) just started to fail inafter? the "Create collated test stats dir" section.

• https://build.ros.org/job/Mdev__ros_canopen__ubuntu_bionic_amd64/30/console
• https://build.ros.org/job/Ndev__ros_canopen__ubuntu_focal_amd64/10/console
• https://build.ros.org/job/Ndev_db__ros_canopen__debian_buster_amd64/10/console

12:52:26 ERROR: Build step failed with exception
12:52:27 java.lang.IllegalArgumentException: The tools section is required.
12:52:27 at org.jenkinsci.plugins.xunit.XUnitProcessor.(XUnitProcessor.java:141)
12:52:27 at org.jenkinsci.plugins.xunit.XUnitPublisher.perform(XUnitPublisher.java:185)
12:52:27 at jenkins.tasks.SimpleBuildStep.perform(SimpleBuildStep.java:123)
12:52:27 at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:80)
12:52:27 at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
12:52:27 at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:803)
12:52:27 at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:752)
12:52:27 at hudson.model.Build$BuildExecution.post2(Build.java:177)
12:52:27 at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:697)
12:52:27 at hudson.model.Run.execute(Run.java:1932)
12:52:27 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
12:52:27 at hudson.model.ResourceController.execute(ResourceController.java:97)
12:52:27 at hudson.model.Executor.run(Executor.java:429)
12:52:27 Build step 'Publish xUnit test result report' marked build as failure

[gavanderhoorn]

It seems additions after the "collated test stats" feature did not add additional section headings.

The error:

Build step 'Publish xUnit test result report' marked build as failure

seems to be coming from something else (ie: the mkdir -p ... is not failing).

[mathias-luedtke]

And it looks like that has affected all normally succeeding ROS1 dev jobs for at least the last ~16h. (Last successful test job: ~17h ago)

[gavanderhoorn]

Seems related: xunit 2.3.4 fails to read xunit 1.102 configs.

[clalancette]

A similar problem caused all of the overnight ROS 2 CI jobs to fail: https://ci.ros2.org/view/nightly/

[ros-discourse]

This issue has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/upgrade-to-jenkins-2-277-3-causing-job-failures-across-ros-build-farms/20013/1

[nuclearsandwich]

I think the underlying problem is the XStream updates introduced in 2.277.3

As described in the Spring and XStream updates (breaking changes!) blog post, this release updates and replaces outdated internal components. The Acegi security library used for authentication has been replaced by Spring Security (JEP-227). See the Spring Security compatibility table for the latest plugin compatibility status. A fork of the XStream library used to read and write XML files has been replaced by the upstream version of XStream (JEP-228). See the XStream compatibility table for the latest plugin compatibility status.

When reading the above paragraph of the [changelog] initially I glossed passed the "breaking changes!" after seeing the "user interface components" discussion and missed the later xstream paragraph.

According to the compatibility table xunit requires an upgrade to at least 2.4.0 to be compatible. In ros2/ci we were able to come up with a temporary workaround (ros2/ci#569) which at least appears to be running but I'm hesitant to make the same change to ros_buildfarm due to the scale of reconfiguration required as well as considering whether or not the change is even necessary or forward compatible with xunit 2.4.0. I'm in the process of setting up a staging server with the plugin updates necessary to run xunit 2.4.0 to check how things work there.

[nuclearsandwich]

The above investigation bore out and I've opened a PR for the ros-buildfarm cookbook updating the xunit plugin to 2.4.0. I'd have to do more changelog scanning to assess how risky a full upgrade to 3.x is but ros-infrastructure/cookbook-ros-buildfarm#89 should get us back to operational as long as it passes review.

[nuclearsandwich]

The above change has been deployed to build.ros2.org but was not able to resolve the issue in all cases.

[nuclearsandwich]

I fetched the XML configuration from both staging and production to compare them and found that although the creation of the devel job with the old configuration was accepted without error the affected configuration block was dropped entirely. It seems that there is no compatibility with the old config block and so I'll put an additional PR in to update the config and we'll reconfigure everything.

edit: scratch that. I had reversed the diff direction. The fully configured staging job has a <types> field and is working fine. I'm going to re-enable a job on build.ros2.org, check whether its configuration has an intact <types> field and verify that it is working with xunit 2.4.0 and if so, try restoring the types field on a job where it is missing.

[nuclearsandwich]

I've been trying to avoid simply updating the XML to avoid breaking the configuration for any ros_buildfarm users who have not yet updated to Jenkins 2.277 in light of other issues. If anyone has an earlier buildfarm and can try to change the field name from types to tools and check whether things work that would be great info to have.

[nuclearsandwich]

One thing we still don't know is whether the future versions of xunit+jenkins will "prefer" the types vs tools for the name of this XML section. Deploying the updated plugins to ci.ros2.org which is not affected by ros-infrastructure/cookbook-ros-buildfarm#90 will allow us to see what Jenkins re-serializes the xml as on save.