New asyncio-based execution engine (PR#196 1/3)

[jbohren]

This PR is the first set of refactored partial changes from PR #196 relating only to the execution engine and I/O handling, without the "linked" develspace support nor improved catkin clean support. It also includes the changes from #247. This is meant to be a "functional" PR and it is expected that it might need further small revisions before being merged. This also adds a Travis CI configuration that builds on Linux as well as OS X.

Main Module Changes

catkin_tools/common.py

The changes in catkin_tools.common include the following:

• Removal of the no-longer needed FakeLock
• Addition of get_recursive_build_dependants_in_workspace function which gives the packages that depend on a given package in the workspace. (This is used in the future)
• Replacement of os.Popen with subprocess.Popen
• Fixing bug where slice_to_printed_length fails if the length is larger than the length of the lookup array.
• Fixing uncaught IOError when the program receives a sigint during a wide_log message
• Adds a mkdir_p function which recursively makes directories

catkin_tools/resultspace.py

The main change to catkin_tools.resultspace is the addition of environment and env hook checksum caching. This enables faster builds by using the resultspace mechanism instead of the build_env.sh mechanism. The resultspace loading can also cache the environment, and updates the cache when a resultspace's Catkin env_hooks change in any way.

It also adds a strict flag which controls whether or not to check for a .catkin file before loading the environment.

catkin_tools/argument_parsing.py

The changes in catkin_tools.argument_parsing mainly support the new job server, which is initialized even when there isn't any support for the GNU Make jobserver, to allow for future extensions to the jobserver in other contexts (distcc, ninja, etc).

Execution Engine

The execution engine consists of the main asyncio-based Executor, Jobs, and job Stages described in #196. Jobs for different catkin build types are defined via setuptools entry_points. This improves job parallelism, but also provides a mechanism for capturing stdout and stderr independently. As such, warnings and errors are now detected and printed in a much clearer manner.

This PR improves on the CMake and Catkin jobs defined in #196 by forgoing using the build_env.sh script, and instead using the catkin_tools.resultspace-based environment loading. This not only speeds up builds, but it also fixes a bug that prevented single-pass workspace building as described in this thread.

More details on the design of the execution engine are forthcoming via a documentation PR.

[jbohren]

This has been superseded by #249.

[xqms]

Thanks for your work on separating this!

I just saw a unicode exception here:

Warnings << gazebo_ros_control:make /var/lib/jenkins/jobs/spacebot/workspace/build/_logs/gazebo_ros_control/build.make.000.log
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/home/max/install/catkin_tools/catkin_tools/execution/controllers.py", line 452, in run
    wide_log('\n'.join(lines))
  File "/home/max/install/catkin_tools/catkin_tools/common.py", line 450, in wide_log
    wide_log_fn(msg, **kwargs)
  File "/home/max/install/catkin_tools/catkin_tools/common.py", line 419, in disabled_wide_log
    log(msg, **kwargs)
  File "/home/max/install/catkin_tools/catkin_tools/common.py", line 272, in log
    print(*args, **kwargs)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2018' in position 129: ordinal not in range(128)

The unicode character 2018 is "LEFT SINGLE QUOTATION MARK", emitted by gcc on my system:

[...] warning: enumeration value ‘EFFORT’ not handled in switch