Build farmer handoff 2017-11-29

[clalancette]

Buildfarm health

The most pressing farming issue is the macOS nodes. Dosa is still offline (more below), and mini1 and mini2 disconnect quite a bit, particularly overnight when the router reboots. They always need manual help coming back online.

Tasks pending (roughly priority order)

• Get Dosa back online; @nuclearsandwich has the latest status on where it is
• Figure out how to run the macOS jenkins process as a service without a massive slowdown
• apt update on Ubuntu hosts (hopefully solving linux_slave_on_master will not run jobs #30)
• Investigate packet.net host issue (packet machine crash on "resource busy" #42)
• Upgrade eatable to Windows 10 (configure eatable as windows 10 machine #17)

Recurring tasks

• system and package (brew/chocolatey) updates on MacOS and Windows machines (2017-12-02: Update nodes before release #74)
  • dosa (last updated 2017-12-02)
  • mini1 (last updated 2017-12-02)
  • mini2 (last updated 2017-12-02)
  • icecube (last updated 2017-12-02)
  • portable (last updated 2017-12-02)
  • windshield (last updated 2017-12-02)

Test suite health

I fixed a couple of failing tests during the last cycle, having to do with finding ROS1 packages (this changed slightly), and having to do with the rclcpp Time changes that broke the turtlebot build. No further progress on flakey tests; the good news is that they seem to be relegated to the "repeated" builds almost every night.

[nuclearsandwich]

Get Dosa back online;

This is sorta half done. SIP is partially enabled (see ros2/ros2#409 (comment)) the build phase at an interactive terminal with this configuration is about 45 minutes. So it's not as good as with SIP enabled but it's not the 2.5 hour monstrosity wit SIP completely disabled.

[nuclearsandwich]

Figure out how to run the macOS jenkins process as a service without a massive slowdown

The next thing to try here is to add ProcessType Interactive to the plist. This should instruct the system to prioritize the build as an interactive program rather than a background task.

Dosa is currently rigged as a launchd supervised process without this tweak and the build did run considerably slower than an interactive build. We're trying some other things with Dosa today and I'll make the interactive change and see how we fare. http://ci.ros2.org/job/ci_osx/2949/

[mikaelarguedas]

Thanks @clalancette for the summary 👍

Farm + Test suite health follow-up:

Some tests started being flaky during last cycle. Mainly Python performance tests as we added more layers and thus more overhead to the Python execution model.
Some windows tests appeared to be less flaky since we refined the timer resolutions on Windows but they still miss deadline every so often so can appear failing even on regular CI.

We missed a non-trivial number of nightly jobs for various reasons (nodes disconnecting, not coming back online or build failures not being addressed for a while). We should find a way to mitigate that in the future to allow proper statistics, bug bisection and ensure users new artifacts with latest code every day.

Machine updates:

• Updating the machines is actually the highest priority this cycle as we are releasing.
• Ubuntu nodes: there's still a discrepancy among the nodes (some running trusty some running xenial). apt upgrade may not be enough to solve docker issues. Upgrading the machines to xenial would be better but may need to be delayed after the release.

[mikaelarguedas]

Closing this in favor of new handoff ticket #78