Merge remote-tracking branch 'origin/master' into v024

* origin/master: (661 commits)
  Bump version for release.
  docs: update Changelog; closes #481
  issue #481: core: preserve stderr TTY FD if one is present.
  issue #481: avoid crash if disconnect occurs during forward_modules()
  Add a few more important modules to preamble_size.py.
  .ci: add verbiage for run_batches() too.
  .ci: add README.md.
  docs: update thanks
  docs: lose "approaching stability" language, we're pretty good now
  docs: fix changelog syntax/order/"20KB"
  tests: add new compression parameter to mitogen_get_stack results
  tests: disable affinity_test on Travis :/
  issue #508: fix responder stats test due to new smaller parent.py.
  issue #508: tests: skip minify_test Py2.4/2.5 for profiler.py.
  tests: fix fallout from 36fb318
  issue #520: add AIX auth failure string to su.
  tests: move affinity_test to Ansible tests.
  core: cProfile is not available in 2.4.
  issue #505: docs: add new detail graph for one scenario.
  docs: update and re-record profile graphs in docs; closes #505
  service: fix PushFileService exception
  tests: pad out localhost-*
  service: start pool shutdown on broker shutdown.
  master: .encode() needed for Py3.
  ansible: stash PID files in CWD if requested for debugging.
  issue #508: master: minify_safe_re must be bytes for Py3.
  bench: tidy up and cpu-pin some more files.
  tests: add localhost-x100
  ansible: double the default pool size.
  ansible: raise error with correct exception type.
  issue #508: master: minify all Mitogen/ansible_mitogen sources.
  parent: PartialZlib docstrings.
  ansible: hacky parser to alow bools to be specified on command line
  parent: pre-cache bootstrap if possible.
  docs: update Changelog.
  ansible: add mitogen_ssh_compression variable.
  service: PushFileService never recorded a file as sent.
  parent: synchronize get_core_source()
  service: use correct profile aggregation name.
  SyntaxError.
  ansible: don't pin controller if <4 cores.
  tests: make soak testing work reliably on vanilla.
  docs: changelog tidyups.
  ansible: document and make affinity stuff portable to non-Linux
  ansible: fix affinity.py test failure on 2 cores.
  ansible: preheat PluginLoader caches before fork.
  tests: make mitogen_shutdown_all be run_once by default.
  docs: update Changelog.
  ansible: use Poller for WorkerProcess; closes #491.
  ansible: new multiplexer/workers configuration
  docs: update Changelog.
  docs: update Changelog.
  ansible: pin connection multiplexer to a single core
  utils: pad out reset_affinity() and integrate with detach_popen()
  utils: import reset_affinity() function.
  master: set Router.profiling if MITOGEN_PROFILING variable present.
  parent: don't kill children when profiling is active.
  ansible: hook strategy and worker processes into profiler
  profiler: import from linear2 branch
  core: tidy up existing profiling code and support MITOGEN_PROFILE_FMT
  issue #260: redundant if statement.
  ansible: ensure MuxProcess MITOGEN_PROFILING results reach disk.
  ansible/bench: make end= configurable.
  master: cache sent/forwarded module names
  Aggregate code coverage data across tox all runs
  Allow independant control of coverage erase and reporting
  Fix incorrect attempt to use coverage
  docs: update Changelog; closes #527.
  issue #527: catch new-style module tracebacks like vanilla.
  Fix DeprecationWarning in mitogen.utils.run_with_router()
  Generate coverage report even if some tests fail
  ci: fix incorrect partition/rpartition from 8a4caea
  issue #260: hide force-disconnect messages.
  issue #498: fix shutdown crash
  issue #260: avoid start_transmit()/on_transmit()/stop_transmit()
  core: ensure broker profiling output reaches disk
  master: keep is_stdlib_path() result as negative cache entry
  ci: Allow DISTROS="debian*32" variable, and KEEP=1
  Use develop mode in tox
  issue #429: fix sudo regression.
  misc: rename to scripts. tab completion!!
  core: Latch._wake improvements
  issue #498: prevent crash on double 'disconnect' signal.
  issue #413: don't double-propagate DEL_ROUTE to parent.
  issue #498: wrap Router dict mutations in a lock
  issue #429: enable en_US locale to unbreak debops test.
  issue #499: fix another mind-numbingly stupid vanilla inconsistency
  issue #497: do our best to cope with crap upstream code
  ssh: fix test to match updated log format.
  issue #429: update Changelog.
  issue #429: update Changelog.
  issue #429: teach sudo about every know i18n password string.
  issue #429: install i18n-related bits in test images.
  ssh: tidy up logs and stream names.
  tests: ensure file is closed in connection_test.
  gcloud: small updates
  tests: give ansible/gcloud/ its own requirements file.
  issue #499: another totally moronic implementation difference
  issue #499: disable new test on vanilla.
  docs: update Changelog; closes #499.
  ...

.ci/README.md

@@ -0,0 +1,44 @@
+
+# `.ci`
+
+This directory contains scripts for Travis CI and (more or less) Azure
+Pipelines, but they will also happily run on any Debian-like machine.
+
+The scripts are usually split into `_install` and `_test` steps. The `_install`
+step will damage your machine, the `_test` step will just run the tests the way
+CI runs them.
+
+There is a common library, `ci_lib.py`, which just centralized a bunch of
+random macros and also environment parsing.
+
+Some of the scripts allow you to pass extra flags through to the component
+under test, e.g. `../../.ci/ansible_tests.py -vvv` will run with verbose.
+
+Hack these scripts until your heart is content. There is no pride to be found
+here, just necessity.
+
+
+### `ci_lib.run_batches()`
+
+There are some weird looking functions to extract more paralellism from the
+build. The above function takes lists of strings, arranging for the strings in
+each list to run in order, but for the lists to run in parallel. That's great
+for doing `setup.py install` while pulling a Docker container, for example.
+
+
+### Environment Variables
+
+* `VER`: Ansible version the `_install` script should install. Default changes
+  over time.
+* `TARGET_COUNT`: number of targets for `debops_` run. Defaults to 2.
+* `DISTRO`: the `mitogen_` tests need a target Docker container distro. This
+  name comes from the Docker Hub `mitogen` user, i.e. `mitogen/$DISTRO-test`
+* `DISTROS`: the `ansible_` tests can run against multiple targets
+  simultaneously, which speeds things up. This is a space-separated list of
+  DISTRO names, but additionally, supports:
+    * `debian-py3`: when generating Ansible inventory file, set
+      `ansible_python_interpreter` to `python3`, i.e. run a test where the
+      target interpreter is Python 3.
+    * `debian*16`: generate 16 Docker containers running Debian. Also works
+      with -py3.
+

.ci/ansible_install.py

@@ -0,0 +1,21 @@
+#!/usr/bin/env python
+
+import ci_lib
+
+batches = [
+    [
+        # Must be installed separately, as PyNACL indirect requirement causes
+        # newer version to be installed if done in a single pip run.
+        'pip install "pycparser<2.19" "idna<2.7"',
+        'pip install '
+            '-r tests/requirements.txt '
+            '-r tests/ansible/requirements.txt',
+    ]
+]
+
+batches.extend(
+    ['docker pull %s' % (ci_lib.image_for_distro(distro),)]
+    for distro in ci_lib.DISTROS
+)
+
+ci_lib.run_batches(batches)

.ci/ansible_tests.py

@@ -0,0 +1,63 @@
+#!/usr/bin/env python
+# Run tests/ansible/all.yml under Ansible and Ansible-Mitogen
+
+import glob
+import os
+import sys
+
+import ci_lib
+from ci_lib import run
+
+
+TESTS_DIR = os.path.join(ci_lib.GIT_ROOT, 'tests/ansible')
+HOSTS_DIR = os.path.join(ci_lib.TMP, 'hosts')
+
+
+with ci_lib.Fold('unit_tests'):
+    os.environ['SKIP_MITOGEN'] = '1'
+    ci_lib.run('./run_tests -v')
+
+
+with ci_lib.Fold('docker_setup'):
+    containers = ci_lib.make_containers()
+    ci_lib.start_containers(containers)
+
+
+with ci_lib.Fold('job_setup'):
+    # Don't set -U as that will upgrade Paramiko to a non-2.6 compatible version.
+    run("pip install -q ansible==%s", ci_lib.ANSIBLE_VERSION)
+
+    os.chdir(TESTS_DIR)
+    os.chmod('../data/docker/mitogen__has_sudo_pubkey.key', int('0600', 7))
+
+    run("mkdir %s", HOSTS_DIR)
+    for path in glob.glob(TESTS_DIR + '/hosts/*'):
+        if not path.endswith('default.hosts'):
+            run("ln -s %s %s", path, HOSTS_DIR)
+
+    inventory_path = os.path.join(HOSTS_DIR, 'target')
+    with open(inventory_path, 'w') as fp:
+        fp.write('[test-targets]\n')
+        fp.writelines(
+            "%(name)s "
+            "ansible_host=%(hostname)s "
+            "ansible_port=%(port)s "
+            "ansible_python_interpreter=%(python_path)s "
+            "ansible_user=mitogen__has_sudo_nopw "
+            "ansible_password=has_sudo_nopw_password"
+            "\n"
+            % container
+            for container in containers
+        )
+
+    ci_lib.dump_file(inventory_path)
+
+    if not ci_lib.exists_in_path('sshpass'):
+        run("sudo apt-get update")
+        run("sudo apt-get install -y sshpass")
+
+
+with ci_lib.Fold('ansible'):
+    playbook = os.environ.get('PLAYBOOK', 'all.yml')
+    run('./run_ansible_playbook.py %s -i "%s" %s',
+        playbook, HOSTS_DIR, ' '.join(sys.argv[1:]))

.ci/azure-pipelines.yml

@@ -0,0 +1,83 @@
+# Python package
+# Create and test a Python package on multiple Python versions.
+# Add steps that analyze code, save the dist with the build record, publish to a PyPI-compatible index, and more:
+# https://docs.microsoft.com/azure/devops/pipelines/languages/python
+
+jobs:
+
+- job: 'MitogenTests'
+  pool:
+    vmImage: 'Ubuntu 16.04'
+  strategy:
+    matrix:
+      Mitogen27Debian_27:
+        python.version: '2.7'
+        MODE: mitogen
+        DISTRO: debian
+
+      MitogenPy27CentOS6_26:
+        python.version: '2.7'
+        MODE: mitogen
+        DISTRO: centos6
+
+      #Py26CentOS7:
+        #python.version: '2.7'
+        #MODE: mitogen
+        #DISTRO: centos6
+
+      Mitogen36CentOS6_26:
+        python.version: '3.6'
+        MODE: mitogen
+        DISTRO: centos6
+
+      DebOps_2460_27_27:
+        python.version: '2.7'
+        MODE: debops_common
+        VER: 2.4.6.0
+
+      DebOps_262_36_27:
+        python.version: '3.6'
+        MODE: debops_common
+        VER: 2.6.2
+
+      Ansible_2460_26:
+        python.version: '2.7'
+        MODE: ansible
+        VER: 2.4.6.0
+
+      Ansible_262_26:
+        python.version: '2.7'
+        MODE: ansible
+        VER: 2.6.2
+
+      Ansible_2460_36:
+        python.version: '3.6'
+        MODE: ansible
+        VER: 2.4.6.0
+
+      Ansible_262_36:
+        python.version: '3.6'
+        MODE: ansible
+        VER: 2.6.2
+
+      Vanilla_262_27:
+        python.version: '2.7'
+        MODE: ansible
+        VER: 2.6.2
+        DISTROS: debian
+        STRATEGY: linear
+
+  steps:
+  - task: UsePythonVersion@0
+    inputs:
+      versionSpec: '$(python.version)'
+      architecture: 'x64'
+
+  - script: .ci/prep_azure.py
+    displayName: "Install requirements."
+
+  - script: .ci/$(MODE)_install.py
+    displayName: "Install requirements."
+
+  - script: .ci/$(MODE)_tests.py
+    displayName: Run tests.