Merge remote-tracking branch 'origin/master' into v024

* origin/master: (661 commits)
  Bump version for release.
  docs: update Changelog; closes #481
  issue #481: core: preserve stderr TTY FD if one is present.
  issue #481: avoid crash if disconnect occurs during forward_modules()
  Add a few more important modules to preamble_size.py.
  .ci: add verbiage for run_batches() too.
  .ci: add README.md.
  docs: update thanks
  docs: lose "approaching stability" language, we're pretty good now
  docs: fix changelog syntax/order/"20KB"
  tests: add new compression parameter to mitogen_get_stack results
  tests: disable affinity_test on Travis :/
  issue #508: fix responder stats test due to new smaller parent.py.
  issue #508: tests: skip minify_test Py2.4/2.5 for profiler.py.
  tests: fix fallout from 36fb318adf5c56e729296c3efce84f4dd75ced4e
  issue #520: add AIX auth failure string to su.
  tests: move affinity_test to Ansible tests.
  core: cProfile is not available in 2.4.
  issue #505: docs: add new detail graph for one scenario.
  docs: update and re-record profile graphs in docs; closes #505
  service: fix PushFileService exception
  tests: pad out localhost-*
  service: start pool shutdown on broker shutdown.
  master: .encode() needed for Py3.
  ansible: stash PID files in CWD if requested for debugging.
  issue #508: master: minify_safe_re must be bytes for Py3.
  bench: tidy up and cpu-pin some more files.
  tests: add localhost-x100
  ansible: double the default pool size.
  ansible: raise error with correct exception type.
  issue #508: master: minify all Mitogen/ansible_mitogen sources.
  parent: PartialZlib docstrings.
  ansible: hacky parser to alow bools to be specified on command line
  parent: pre-cache bootstrap if possible.
  docs: update Changelog.
  ansible: add mitogen_ssh_compression variable.
  service: PushFileService never recorded a file as sent.
  parent: synchronize get_core_source()
  service: use correct profile aggregation name.
  SyntaxError.
  ansible: don't pin controller if <4 cores.
  tests: make soak testing work reliably on vanilla.
  docs: changelog tidyups.
  ansible: document and make affinity stuff portable to non-Linux
  ansible: fix affinity.py test failure on 2 cores.
  ansible: preheat PluginLoader caches before fork.
  tests: make mitogen_shutdown_all be run_once by default.
  docs: update Changelog.
  ansible: use Poller for WorkerProcess; closes #491.
  ansible: new multiplexer/workers configuration
  docs: update Changelog.
  docs: update Changelog.
  ansible: pin connection multiplexer to a single core
  utils: pad out reset_affinity() and integrate with detach_popen()
  utils: import reset_affinity() function.
  master: set Router.profiling if MITOGEN_PROFILING variable present.
  parent: don't kill children when profiling is active.
  ansible: hook strategy and worker processes into profiler
  profiler: import from linear2 branch
  core: tidy up existing profiling code and support MITOGEN_PROFILE_FMT
  issue #260: redundant if statement.
  ansible: ensure MuxProcess MITOGEN_PROFILING results reach disk.
  ansible/bench: make end= configurable.
  master: cache sent/forwarded module names
  Aggregate code coverage data across tox all runs
  Allow independant control of coverage erase and reporting
  Fix incorrect attempt to use coverage
  docs: update Changelog; closes #527.
  issue #527: catch new-style module tracebacks like vanilla.
  Fix DeprecationWarning in mitogen.utils.run_with_router()
  Generate coverage report even if some tests fail
  ci: fix incorrect partition/rpartition from 8a4caea84f
  issue #260: hide force-disconnect messages.
  issue #498: fix shutdown crash
  issue #260: avoid start_transmit()/on_transmit()/stop_transmit()
  core: ensure broker profiling output reaches disk
  master: keep is_stdlib_path() result as negative cache entry
  ci: Allow DISTROS="debian*32" variable, and KEEP=1
  Use develop mode in tox
  issue #429: fix sudo regression.
  misc: rename to scripts. tab completion!!
  core: Latch._wake improvements
  issue #498: prevent crash on double 'disconnect' signal.
  issue #413: don't double-propagate DEL_ROUTE to parent.
  issue #498: wrap Router dict mutations in a lock
  issue #429: enable en_US locale to unbreak debops test.
  issue #499: fix another mind-numbingly stupid vanilla inconsistency
  issue #497: do our best to cope with crap upstream code
  ssh: fix test to match updated log format.
  issue #429: update Changelog.
  issue #429: update Changelog.
  issue #429: teach sudo about every know i18n password string.
  issue #429: install i18n-related bits in test images.
  ssh: tidy up logs and stream names.
  tests: ensure file is closed in connection_test.
  gcloud: small updates
  tests: give ansible/gcloud/ its own requirements file.
  issue #499: another totally moronic implementation difference
  issue #499: disable new test on vanilla.
  docs: update Changelog; closes #499.
  ...

.ci/README.md

@@ -0,0 +1,44 @@
+
+# `.ci`
+
+This directory contains scripts for Travis CI and (more or less) Azure
+Pipelines, but they will also happily run on any Debian-like machine.
+
+The scripts are usually split into `_install` and `_test` steps. The `_install`
+step will damage your machine, the `_test` step will just run the tests the way
+CI runs them.
+
+There is a common library, `ci_lib.py`, which just centralized a bunch of
+random macros and also environment parsing.
+
+Some of the scripts allow you to pass extra flags through to the component
+under test, e.g. `../../.ci/ansible_tests.py -vvv` will run with verbose.
+
+Hack these scripts until your heart is content. There is no pride to be found
+here, just necessity.
+
+
+### `ci_lib.run_batches()`
+
+There are some weird looking functions to extract more paralellism from the
+build. The above function takes lists of strings, arranging for the strings in
+each list to run in order, but for the lists to run in parallel. That's great
+for doing `setup.py install` while pulling a Docker container, for example.
+
+
+### Environment Variables
+
+* `VER`: Ansible version the `_install` script should install. Default changes
+  over time.
+* `TARGET_COUNT`: number of targets for `debops_` run. Defaults to 2.
+* `DISTRO`: the `mitogen_` tests need a target Docker container distro. This
+  name comes from the Docker Hub `mitogen` user, i.e. `mitogen/$DISTRO-test`
+* `DISTROS`: the `ansible_` tests can run against multiple targets
+  simultaneously, which speeds things up. This is a space-separated list of
+  DISTRO names, but additionally, supports:
+    * `debian-py3`: when generating Ansible inventory file, set
+      `ansible_python_interpreter` to `python3`, i.e. run a test where the
+      target interpreter is Python 3.
+    * `debian*16`: generate 16 Docker containers running Debian. Also works
+      with -py3.
+

.ci/ansible_install.py

@@ -0,0 +1,21 @@
+#!/usr/bin/env python
+
+import ci_lib
+
+batches = [
+    [
+        # Must be installed separately, as PyNACL indirect requirement causes
+        # newer version to be installed if done in a single pip run.
+        'pip install "pycparser<2.19" "idna<2.7"',
+        'pip install '
+            '-r tests/requirements.txt '
+            '-r tests/ansible/requirements.txt',
+    ]
+]
+
+batches.extend(
+    ['docker pull %s' % (ci_lib.image_for_distro(distro),)]
+    for distro in ci_lib.DISTROS
+)
+
+ci_lib.run_batches(batches)

.ci/ansible_tests.py

@@ -0,0 +1,63 @@
+#!/usr/bin/env python
+# Run tests/ansible/all.yml under Ansible and Ansible-Mitogen
+
+import glob
+import os
+import sys
+
+import ci_lib
+from ci_lib import run
+
+
+TESTS_DIR = os.path.join(ci_lib.GIT_ROOT, 'tests/ansible')
+HOSTS_DIR = os.path.join(ci_lib.TMP, 'hosts')
+
+
+with ci_lib.Fold('unit_tests'):
+    os.environ['SKIP_MITOGEN'] = '1'
+    ci_lib.run('./run_tests -v')
+
+
+with ci_lib.Fold('docker_setup'):
+    containers = ci_lib.make_containers()
+    ci_lib.start_containers(containers)
+
+
+with ci_lib.Fold('job_setup'):
+    # Don't set -U as that will upgrade Paramiko to a non-2.6 compatible version.
+    run("pip install -q ansible==%s", ci_lib.ANSIBLE_VERSION)
+
+    os.chdir(TESTS_DIR)
+    os.chmod('../data/docker/mitogen__has_sudo_pubkey.key', int('0600', 7))
+
+    run("mkdir %s", HOSTS_DIR)
+    for path in glob.glob(TESTS_DIR + '/hosts/*'):
+        if not path.endswith('default.hosts'):
+            run("ln -s %s %s", path, HOSTS_DIR)
+
+    inventory_path = os.path.join(HOSTS_DIR, 'target')
+    with open(inventory_path, 'w') as fp:
+        fp.write('[test-targets]\n')
+        fp.writelines(
+            "%(name)s "
+            "ansible_host=%(hostname)s "
+            "ansible_port=%(port)s "
+            "ansible_python_interpreter=%(python_path)s "
+            "ansible_user=mitogen__has_sudo_nopw "
+            "ansible_password=has_sudo_nopw_password"
+            "\n"
+            % container
+            for container in containers
+        )
+
+    ci_lib.dump_file(inventory_path)
+
+    if not ci_lib.exists_in_path('sshpass'):
+        run("sudo apt-get update")
+        run("sudo apt-get install -y sshpass")
+
+
+with ci_lib.Fold('ansible'):
+    playbook = os.environ.get('PLAYBOOK', 'all.yml')
+    run('./run_ansible_playbook.py %s -i "%s" %s',
+        playbook, HOSTS_DIR, ' '.join(sys.argv[1:]))

.ci/azure-pipelines.yml

@@ -0,0 +1,83 @@
+# Python package
+# Create and test a Python package on multiple Python versions.
+# Add steps that analyze code, save the dist with the build record, publish to a PyPI-compatible index, and more:
+# https://docs.microsoft.com/azure/devops/pipelines/languages/python
+
+jobs:
+
+- job: 'MitogenTests'
+  pool:
+    vmImage: 'Ubuntu 16.04'
+  strategy:
+    matrix:
+      Mitogen27Debian_27:
+        python.version: '2.7'
+        MODE: mitogen
+        DISTRO: debian
+
+      MitogenPy27CentOS6_26:
+        python.version: '2.7'
+        MODE: mitogen
+        DISTRO: centos6
+
+      #Py26CentOS7:
+        #python.version: '2.7'
+        #MODE: mitogen
+        #DISTRO: centos6
+
+      Mitogen36CentOS6_26:
+        python.version: '3.6'
+        MODE: mitogen
+        DISTRO: centos6
+
+      DebOps_2460_27_27:
+        python.version: '2.7'
+        MODE: debops_common
+        VER: 2.4.6.0
+
+      DebOps_262_36_27:
+        python.version: '3.6'
+        MODE: debops_common
+        VER: 2.6.2
+
+      Ansible_2460_26:
+        python.version: '2.7'
+        MODE: ansible
+        VER: 2.4.6.0
+
+      Ansible_262_26:
+        python.version: '2.7'
+        MODE: ansible
+        VER: 2.6.2
+
+      Ansible_2460_36:
+        python.version: '3.6'
+        MODE: ansible
+        VER: 2.4.6.0
+
+      Ansible_262_36:
+        python.version: '3.6'
+        MODE: ansible
+        VER: 2.6.2
+
+      Vanilla_262_27:
+        python.version: '2.7'
+        MODE: ansible
+        VER: 2.6.2
+        DISTROS: debian
+        STRATEGY: linear
+
+  steps:
+  - task: UsePythonVersion@0
+    inputs:
+      versionSpec: '$(python.version)'
+      architecture: 'x64'
+
+  - script: .ci/prep_azure.py
+    displayName: "Install requirements."
+
+  - script: .ci/$(MODE)_install.py
+    displayName: "Install requirements."
+
+  - script: .ci/$(MODE)_tests.py
+    displayName: Run tests.

.ci/ci_lib.py

@@ -0,0 +1,222 @@
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import atexit
+import os
+import shlex
+import shutil
+import subprocess
+import sys
+import tempfile
+
+try:
+    import urlparse
+except ImportError:
+    import urllib.parse as urlparse
+
+os.chdir(
+    os.path.join(
+        os.path.dirname(__file__),
+        '..'
+    )
+)
+
+
+#
+# check_output() monkeypatch cutpasted from testlib.py
+#
+
+def subprocess__check_output(*popenargs, **kwargs):
+    # Missing from 2.6.
+    process = subprocess.Popen(stdout=subprocess.PIPE, *popenargs, **kwargs)
+    output, _ = process.communicate()
+    retcode = process.poll()
+    if retcode:
+        cmd = kwargs.get("args")
+        if cmd is None:
+            cmd = popenargs[0]
+        raise subprocess.CalledProcessError(retcode, cmd)
+    return output
+
+if not hasattr(subprocess, 'check_output'):
+    subprocess.check_output = subprocess__check_output
+
+
+# -----------------
+
+# Force stdout FD 1 to be a pipe, so tools like pip don't spam progress bars.
+
+if 'TRAVIS_HOME' in os.environ:
+    proc = subprocess.Popen(
+        args=['stdbuf', '-oL', 'cat'],
+        stdin=subprocess.PIPE
+    )
+
+    os.dup2(proc.stdin.fileno(), 1)
+    os.dup2(proc.stdin.fileno(), 2)
+
+    def cleanup_travis_junk(stdout=sys.stdout, stderr=sys.stderr, proc=proc):
+        stdout.close()
+        stderr.close()
+        proc.terminate()
+
+    atexit.register(cleanup_travis_junk)
+
+# -----------------
+
+def _argv(s, *args):
+    if args:
+        s %= args
+    return shlex.split(s)
+
+
+def run(s, *args, **kwargs):
+    argv = ['/usr/bin/time', '--'] + _argv(s, *args)
+    print('Running: %s' % (argv,))
+    ret = subprocess.check_call(argv, **kwargs)
+    print('Finished running: %s' % (argv,))
+    return ret
+
+
+def run_batches(batches):
+    combine = lambda batch: 'set -x; ' + (' && '.join(
+        '( %s; )' % (cmd,)
+        for cmd in batch
+    ))
+
+    procs = [
+        subprocess.Popen(combine(batch), shell=True)
+        for batch in batches
+    ]
+    assert [proc.wait() for proc in procs] == [0] * len(procs)
+
+
+def get_output(s, *args, **kwargs):
+    argv = _argv(s, *args)
+    print('Running: %s' % (argv,))
+    return subprocess.check_output(argv, **kwargs)
+
+
+def exists_in_path(progname):
+    return any(os.path.exists(os.path.join(dirname, progname))
+               for dirname in os.environ['PATH'].split(os.pathsep))
+
+
+class TempDir(object):
+    def __init__(self):
+        self.path = tempfile.mkdtemp(prefix='mitogen_ci_lib')
+        atexit.register(self.destroy)
+
+    def destroy(self, rmtree=shutil.rmtree):
+        rmtree(self.path)
+
+
+class Fold(object):
+    def __init__(self, name):
+        self.name = name
+
+    def __enter__(self):
+        print('travis_fold:start:%s' % (self.name))
+
+    def __exit__(self, _1, _2, _3):
+        print('')
+        print('travis_fold:end:%s' % (self.name))
+
+
+os.environ.setdefault('ANSIBLE_STRATEGY',
+    os.environ.get('STRATEGY', 'mitogen_linear'))
+ANSIBLE_VERSION = os.environ.get('VER', '2.6.2')
+GIT_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
+DISTRO = os.environ.get('DISTRO', 'debian')
+DISTROS = os.environ.get('DISTROS', 'debian centos6 centos7').split()
+TARGET_COUNT = int(os.environ.get('TARGET_COUNT', '2'))
+BASE_PORT = 2200
+TMP = TempDir().path
+
+os.environ['PYTHONDONTWRITEBYTECODE'] = 'x'
+os.environ['PYTHONPATH'] = '%s:%s' % (
+    os.environ.get('PYTHONPATH', ''),
+    GIT_ROOT
+)
+
+def get_docker_hostname():
+    url = os.environ.get('DOCKER_HOST')
+    if url in (None, 'http+docker://localunixsocket'):
+        return 'localhost'
+
+    parsed = urlparse.urlparse(url)
+    return parsed.netloc.partition(':')[0]
+
+
+def image_for_distro(distro):
+    return 'mitogen/%s-test' % (distro.partition('-')[0],)
+
+
+def make_containers():
+    docker_hostname = get_docker_hostname()
+    firstbit = lambda s: (s+'-').split('-')[0]
+    secondbit = lambda s: (s+'-').split('-')[1]
+
+    i = 1
+    lst = []
+
+    for distro in DISTROS:
+        distro, star, count = distro.partition('*')
+        if star:
+            count = int(count)
+        else:
+            count = 1
+
+        for x in range(count):
+            lst.append({
+                "distro": firstbit(distro),
+                "name": "target-%s-%s" % (distro, i),
+                "hostname": docker_hostname,
+                "port": BASE_PORT + i,
+                "python_path": (
+                    '/usr/bin/python3'
+                    if secondbit(distro) == 'py3'
+                    else '/usr/bin/python'
+                )
+            })
+            i += 1
+
+    return lst
+
+
+def start_containers(containers):
+    if os.environ.get('KEEP'):
+        return
+
+    run_batches([
+        [
+            "docker rm -f %(name)s || true" % container,
+            "docker run "
+                "--rm "
+                "--detach "
+                "--publish 0.0.0.0:%(port)s:22/tcp "
+                "--hostname=%(name)s "
+                "--name=%(name)s "
+                "mitogen/%(distro)s-test "
+            % container
+        ]
+        for container in containers
+    ])
+    return containers
+
+
+def dump_file(path):
+    print()
+    print('--- %s ---' % (path,))
+    print()
+    with open(path, 'r') as fp:
+        print(fp.read().rstrip())
+    print('---')
+    print()
+
+
+# SSH passes these through to the container when run interactively, causing
+# stdout to get messed up with libc warnings.
+os.environ.pop('LANG', None)
+os.environ.pop('LC_ALL', None)

.ci/debops_common_install.py

@@ -0,0 +1,18 @@
+#!/usr/bin/env python
+
+import ci_lib
+
+# Naturally DebOps only supports Debian.
+ci_lib.DISTROS = ['debian']
+
+ci_lib.run_batches([
+    [
+        # Must be installed separately, as PyNACL indirect requirement causes
+        # newer version to be installed if done in a single pip run.
+        'pip install "pycparser<2.19"',
+        'pip install -qqqU debops==0.7.2 ansible==%s' % ci_lib.ANSIBLE_VERSION,
+    ],
+    [
+        'docker pull %s' % (ci_lib.image_for_distro('debian'),),
+    ],
+])

.ci/debops_common_tests.py

@@ -0,0 +1,79 @@
+#!/usr/bin/env python
+
+from __future__ import print_function
+import os
+
+import ci_lib
+
+
+# DebOps only supports Debian.
+ci_lib.DISTROS = ['debian'] * ci_lib.TARGET_COUNT
+
+project_dir = os.path.join(ci_lib.TMP, 'project')
+key_file = os.path.join(
+    ci_lib.GIT_ROOT,
+    'tests/data/docker/mitogen__has_sudo_pubkey.key',
+)
+vars_path = 'ansible/inventory/group_vars/debops_all_hosts.yml'
+inventory_path = 'ansible/inventory/hosts'
+docker_hostname = ci_lib.get_docker_hostname()
+
+
+with ci_lib.Fold('docker_setup'):
+    containers = ci_lib.make_containers()
+    ci_lib.start_containers(containers)
+
+
+with ci_lib.Fold('job_setup'):
+    ci_lib.run('debops-init %s', project_dir)
+    os.chdir(project_dir)
+
+    with open('.debops.cfg', 'w') as fp:
+        fp.write(
+            "[ansible defaults]\n"
+            "strategy_plugins = %s/ansible_mitogen/plugins/strategy\n"
+            "strategy = mitogen_linear\n"
+            % (ci_lib.GIT_ROOT,)
+        )
+
+    ci_lib.run('chmod go= %s', key_file)
+    with open(vars_path, 'w') as fp:
+        fp.write(
+            "ansible_python_interpreter: /usr/bin/python2.7\n"
+            "\n"
+            "ansible_user: mitogen__has_sudo_pubkey\n"
+            "ansible_become_pass: has_sudo_pubkey_password\n"
+            "ansible_ssh_private_key_file: %s\n"
+            "\n"
+            # Speed up slow DH generation.
+            "dhparam__bits: ['128', '64']\n"
+            % (key_file,)
+        )
+
+    with open(inventory_path, 'a') as fp:
+        fp.writelines(
+            '%(name)s '
+            'ansible_host=%(hostname)s '
+            'ansible_port=%(port)d '
+            'ansible_python_interpreter=%(python_path)s '
+            '\n'
+            % container
+            for container in containers
+        )
+
+    print()
+    print(' echo --- ansible/inventory/hosts: ---')
+    ci_lib.run('cat ansible/inventory/hosts')
+    print('---')
+    print()
+
+    # Now we have real host key checking, we need to turn it off
+    os.environ['ANSIBLE_HOST_KEY_CHECKING'] = 'False'
+
+
+with ci_lib.Fold('first_run'):
+    ci_lib.run('debops common')
+
+
+with ci_lib.Fold('second_run'):
+    ci_lib.run('debops common')

.ci/mitogen_install.py

@@ -0,0 +1,15 @@
+#!/usr/bin/env python
+
+import ci_lib
+
+batches = [
+    [
+        'pip install "pycparser<2.19" "idna<2.7"',
+        'pip install -r tests/requirements.txt',
+    ],
+    [
+        'docker pull %s' % (ci_lib.image_for_distro(ci_lib.DISTRO),),
+    ]
+]
+
+ci_lib.run_batches(batches)

.ci/mitogen_py24_install.py

@@ -0,0 +1,14 @@
+#!/usr/bin/env python
+
+import ci_lib
+
+batches = [
+    [
+        'docker pull %s' % (ci_lib.image_for_distro(ci_lib.DISTRO),),
+    ],
+    [
+        'sudo tar -C / -jxvf tests/data/ubuntu-python-2.4.6.tar.bz2',
+    ]
+]
+
+ci_lib.run_batches(batches)

.ci/mitogen_py24_tests.py

@@ -0,0 +1,17 @@
+#!/usr/bin/env python
+# Mitogen tests for Python 2.4.
+
+import os
+
+import ci_lib
+
+os.environ.update({
+    'NOCOVERAGE': '1',
+    'UNIT2': '/usr/local/python2.4.6/bin/unit2',
+
+    'MITOGEN_TEST_DISTRO': ci_lib.DISTRO,
+    'MITOGEN_LOG_LEVEL': 'debug',
+    'SKIP_ANSIBLE': '1',
+})
+
+ci_lib.run('./run_tests -v')

.ci/mitogen_tests.py

@@ -0,0 +1,14 @@
+#!/usr/bin/env python
+# Run the Mitogen tests.
+
+import os
+
+import ci_lib
+
+os.environ.update({
+    'MITOGEN_TEST_DISTRO': ci_lib.DISTRO,
+    'MITOGEN_LOG_LEVEL': 'debug',
+    'SKIP_ANSIBLE': '1',
+})
+
+ci_lib.run('./run_tests -v')

.ci/prep_azure.py

@@ -0,0 +1,22 @@
+#!/usr/bin/env python
+
+import ci_lib
+
+batches = []
+batches.append([
+    'echo force-unsafe-io | sudo tee /etc/dpkg/dpkg.cfg.d/nosync',
+    'sudo add-apt-repository ppa:deadsnakes/ppa',
+    'sudo apt-get update',
+    'sudo apt-get -y install python2.6 python2.6-dev libsasl2-dev libldap2-dev',
+])
+
+batches.append([
+    'pip install -r dev_requirements.txt',
+])
+
+batches.extend(
+    ['docker pull %s' % (ci_lib.image_for_distro(distro),)]
+    for distro in ci_lib.DISTROS
+)
+
+ci_lib.run_batches(batches)

.github/ISSUE_TEMPLATE.md

@@ -1,15 +1,21 @@
 
+Please drag-drop large logs as text file attachments.
+
 Feel free to write an issue in your preferred format, however if in doubt, use
 the following checklist as a guide for what to include.
 
 * Have you tried the latest master version from Git?
+* Do you have some idea of what the underlying problem may be?
+  https://mitogen.rtfd.io/en/stable/ansible.html#common-problems has
+  instructions to help figure out the likely cause and how to gather relevant
+  logs.
 * Mention your host and target OS and versions
 * Mention your host and target Python versions
 * If reporting a performance issue, mention the number of targets and a rough
   description of your workload (lots of copies, lots of tiny file edits, etc.)
-* If reporting a crash or hang in Ansible, please rerun with -vvvv and include
-  the last 200 lines of output, along with a full copy of any traceback or
-  error text in the log. Beware "-vvvv" may include secret data! Edit as
-  necessary before posting.
+* If reporting a crash or hang in Ansible, please rerun with -vvv and include
+  200 lines of output around the point of the error, along with a full copy of
+  any traceback or error text in the log. Beware "-vvv" may include secret
+  data! Edit as necessary before posting.
 * If reporting any kind of problem with Ansible, please include the Ansible
   version along with output of "ansible-config dump --only-changed".

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,16 @@
+
+Thanks for creating a PR! Here's a quick checklist to pay attention to:
+
+* Please add an entry to docs/changelog.rst as appropriate.
+
+* Has some new parameter been added or semantics modified somehow? Please
+  ensure relevant documentation is updated in docs/ansible.rst and
+  docs/api.rst.
+
+* If it's for new functionality, is there at least a basic test in either
+  tests/ or tests/ansible/ covering it?
+
+* If it's for a new connection method, please try to stub out the
+  implementation as in tests/data/stubs/, so that construction can be tested
+  without having a working configuration.
+

.gitignore

@@ -1,6 +1,7 @@
 .coverage
 .tox
 .venv
+venvs/**
 **/.DS_Store
 *.pyc
 *.pyd

.travis.yml

@@ -1,10 +1,8 @@
 sudo: required
-addons:
-  apt:
-    update: true
 
 notifications:
   email: false
+  irc: "chat.freenode.net#mitogen-builds"
 
 language: python
 
@@ -14,19 +12,10 @@ cache:
   - /home/travis/virtualenv
 
 install:
-- pip install -r dev_requirements.txt
+- .ci/${MODE}_install.py
 
 script:
-- |
-  if [ -f "${TRAVIS_BUILD_DIR}/.travis/${MODE}_tests.sh" ]; then
-    ${TRAVIS_BUILD_DIR}/.travis/${MODE}_tests.sh;
-  else
-    ${TRAVIS_BUILD_DIR}/.travis/${MODE}_tests.py;
-  fi
-
-
-services:
-  - docker
+- .ci/${MODE}_tests.py
 
 
 # To avoid matrix explosion, just test against oldest->newest and
@@ -35,15 +24,21 @@ services:
 matrix:
   include:
     # Mitogen tests.
+    # 2.4 -> 2.4
+    - language: c
+      env: MODE=mitogen_py24 DISTRO=centos5
     # 2.7 -> 2.7
     - python: "2.7"
       env: MODE=mitogen DISTRO=debian
     # 2.7 -> 2.6
-    - python: "2.7"
-      env: MODE=mitogen DISTRO=centos6
+    #- python: "2.7"
+      #env: MODE=mitogen DISTRO=centos6
     # 2.6 -> 2.7
     - python: "2.6"
       env: MODE=mitogen DISTRO=centos7
+    # 2.6 -> 3.5
+    - python: "2.6"
+      env: MODE=mitogen DISTRO=debian-py3
     # 3.6 -> 2.6
     - python: "3.6"
       env: MODE=mitogen DISTRO=centos6
@@ -58,6 +53,10 @@ matrix:
 
     # ansible_mitogen tests.
 
+    # 2.3 -> {centos5}
+    - python: "2.6"
+      env: MODE=ansible VER=2.3.3.0 DISTROS=centos5
+
     # 2.6 -> {debian, centos6, centos7}
     - python: "2.6"
       env: MODE=ansible VER=2.4.6.0

.travis/ansible_tests.py

@@ -1,66 +0,0 @@
-#!/usr/bin/env python
-# Run tests/ansible/all.yml under Ansible and Ansible-Mitogen
-
-import os
-import sys
-
-import ci_lib
-from ci_lib import run
-
-
-BASE_PORT = 2201
-TESTS_DIR = os.path.join(ci_lib.GIT_ROOT, 'tests/ansible')
-HOSTS_DIR = os.path.join(ci_lib.TMP, 'hosts')
-
-
-with ci_lib.Fold('docker_setup'):
-    for i, distro in enumerate(ci_lib.DISTROS):
-        try:
-            run("docker rm -f target-%s", distro)
-        except: pass
-
-        run("""
-            docker run
-            --rm
-            --detach
-            --publish 0.0.0.0:%s:22/tcp
-            --hostname=target-%s
-            --name=target-%s
-            mitogen/%s-test
-        """, BASE_PORT + i, distro, distro, distro)
-
-
-with ci_lib.Fold('job_setup'):
-    os.chdir(TESTS_DIR)
-    os.chmod('../data/docker/mitogen__has_sudo_pubkey.key', int('0600', 7))
-
-    # Don't set -U as that will upgrade Paramiko to a non-2.6 compatible version.
-    run("pip install -q ansible==%s", ci_lib.ANSIBLE_VERSION)
-
-    run("mkdir %s", HOSTS_DIR)
-    run("ln -s %s/hosts/common-hosts %s", TESTS_DIR, HOSTS_DIR)
-
-    with open(os.path.join(HOSTS_DIR, 'target'), 'w') as fp:
-        fp.write('[test-targets]\n')
-        for i, distro in enumerate(ci_lib.DISTROS):
-            fp.write("target-%s "
-                     "ansible_host=%s "
-                     "ansible_port=%s "
-                     "ansible_user=mitogen__has_sudo_nopw "
-                     "ansible_password=has_sudo_nopw_password"
-                     "\n" % (
-                distro,
-                ci_lib.DOCKER_HOSTNAME,
-                BASE_PORT + i,
-            ))
-
-    # Build the binaries.
-    # run("make -C %s", TESTS_DIR)
-    if not ci_lib.exists_in_path('sshpass'):
-        run("sudo apt-get update")
-        run("sudo apt-get install -y sshpass")
-
-
-with ci_lib.Fold('ansible'):
-    run('/usr/bin/time ./run_ansible_playbook.sh all.yml -i "%s" %s',
-        HOSTS_DIR, ' '.join(sys.argv[1:]))

.travis/ci_lib.py

@@ -1,102 +0,0 @@
-
-from __future__ import absolute_import
-from __future__ import print_function
-
-import atexit
-import os
-import subprocess
-import sys
-import shlex
-import shutil
-import tempfile
-
-import os
-os.system('curl -H Metadata-Flavor:Google http://metadata.google.internal/computeMetadata/v1/instance/machine-type')
-
-#
-# check_output() monkeypatch cutpasted from testlib.py
-#
-
-def subprocess__check_output(*popenargs, **kwargs):
-    # Missing from 2.6.
-    process = subprocess.Popen(stdout=subprocess.PIPE, *popenargs, **kwargs)
-    output, _ = process.communicate()
-    retcode = process.poll()
-    if retcode:
-        cmd = kwargs.get("args")
-        if cmd is None:
-            cmd = popenargs[0]
-        raise subprocess.CalledProcessError(retcode, cmd)
-    return output
-
-if not hasattr(subprocess, 'check_output'):
-    subprocess.check_output = subprocess__check_output
-
-# -----------------
-
-def _argv(s, *args):
-    if args:
-        s %= args
-    return shlex.split(s)
-
-
-def run(s, *args, **kwargs):
-    argv = _argv(s, *args)
-    print('Running: %s' % (argv,))
-    return subprocess.check_call(argv, **kwargs)
-
-
-def get_output(s, *args, **kwargs):
-    argv = _argv(s, *args)
-    print('Running: %s' % (argv,))
-    return subprocess.check_output(argv, **kwargs)
-
-
-def exists_in_path(progname):
-    return any(os.path.exists(os.path.join(dirname, progname))
-               for dirname in os.environ['PATH'].split(os.pathsep))
-
-
-class TempDir(object):
-    def __init__(self):
-        self.path = tempfile.mkdtemp(prefix='mitogen_ci_lib')
-        atexit.register(self.destroy)
-
-    def destroy(self, rmtree=shutil.rmtree):
-        rmtree(self.path)
-
-
-class Fold(object):
-    def __init__(self, name):
-        self.name = name
-
-    def __enter__(self):
-        print('travis_fold:start:%s' % (self.name))
-
-    def __exit__(self, _1, _2, _3):
-        print('')
-        print('travis_fold:end:%s' % (self.name))
-
-
-os.environ.setdefault('ANSIBLE_STRATEGY',
-    os.environ.get('STRATEGY', 'mitogen_linear'))
-ANSIBLE_VERSION = os.environ.get('VER', '2.6.2')
-GIT_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
-DISTROS = os.environ.get('DISTROS', 'debian centos6 centos7').split()
-TMP = TempDir().path
-
-os.environ['PYTHONDONTWRITEBYTECODE'] = 'x'
-os.environ['PYTHONPATH'] = '%s:%s' % (
-    os.environ.get('PYTHONPATH', ''),
-    GIT_ROOT
-)
-
-DOCKER_HOSTNAME = subprocess.check_output([
-    sys.executable,
-    os.path.join(GIT_ROOT, 'tests/show_docker_hostname.py'),
-]).decode().strip()
-
-# SSH passes these through to the container when run interactively, causing
-# stdout to get messed up with libc warnings.
-os.environ.pop('LANG', None)
-os.environ.pop('LC_ALL', None)

.travis/debops_common_tests.sh

@@ -1,90 +0,0 @@
-#!/bin/bash -ex
-# Run some invocations of DebOps.
-
-TMPDIR="/tmp/debops-$$"
-TRAVIS_BUILD_DIR="${TRAVIS_BUILD_DIR:-`pwd`}"
-TARGET_COUNT="${TARGET_COUNT:-2}"
-ANSIBLE_VERSION="${VER:-2.6.1}"
-DISTRO=debian  # Naturally DebOps only supports Debian.
-
-export PYTHONPATH="${PYTHONPATH}:${TRAVIS_BUILD_DIR}"
-
-function on_exit()
-{
-    echo travis_fold:start:cleanup
-    [ "$KEEP" ] || {
-        rm -rf "$TMPDIR" || true
-        for i in $(seq $TARGET_COUNT)
-        do
-            docker kill target$i || true
-        done
-    }
-    echo travis_fold:end:cleanup
-}
-
-trap on_exit EXIT
-mkdir "$TMPDIR"
-
-
-echo travis_fold:start:job_setup
-pip install -qqqU debops==0.7.2 ansible==${ANSIBLE_VERSION}
-debops-init "$TMPDIR/project"
-cd "$TMPDIR/project"
-
-cat > .debops.cfg <<-EOF
-[ansible defaults]
-strategy_plugins = ${TRAVIS_BUILD_DIR}/ansible_mitogen/plugins/strategy
-strategy = mitogen_linear
-EOF
-
-chmod go= ${TRAVIS_BUILD_DIR}/tests/data/docker/mitogen__has_sudo_pubkey.key
-
-cat > ansible/inventory/group_vars/debops_all_hosts.yml <<-EOF
-ansible_python_interpreter: /usr/bin/python2.7
-
-ansible_user: mitogen__has_sudo_pubkey
-ansible_become_pass: has_sudo_pubkey_password
-ansible_ssh_private_key_file: ${TRAVIS_BUILD_DIR}/tests/data/docker/mitogen__has_sudo_pubkey.key
-
-# Speed up slow DH generation.
-dhparam__bits: ["128", "64"]
-EOF
-
-DOCKER_HOSTNAME="$(python ${TRAVIS_BUILD_DIR}/tests/show_docker_hostname.py)"
-
-for i in $(seq $TARGET_COUNT)
-do
-    port=$((2200 + $i))
-    docker run \
-        --rm \
-        --detach \
-        --publish 0.0.0.0:$port:22/tcp \
-        --name=target$i \
-        mitogen/${DISTRO}-test
-
-    echo \
-        target$i \
-        ansible_host=$DOCKER_HOSTNAME \
-        ansible_port=$port \
-        >> ansible/inventory/hosts
-done
-
-echo
-echo --- ansible/inventory/hosts: ----
-cat ansible/inventory/hosts
-echo ---
-
-# Now we have real host key checking, we need to turn it off. :)
-export ANSIBLE_HOST_KEY_CHECKING=False
-
-echo travis_fold:end:job_setup
-
-
-echo travis_fold:start:first_run
-/usr/bin/time debops common "$@"
-echo travis_fold:end:first_run
-
-
-echo travis_fold:start:second_run
-/usr/bin/time debops common "$@"
-echo travis_fold:end:second_run

.travis/mitogen_tests.sh

@@ -1,5 +0,0 @@
-#!/bin/bash -ex
-# Run the Mitogen tests.
-
-MITOGEN_TEST_DISTRO="${DISTRO:-debian}"
-MITOGEN_LOG_LEVEL=debug PYTHONPATH=. ${TRAVIS_BUILD_DIR}/run_tests -vvv

README.md

@@ -1,4 +1,13 @@
 
 # Mitogen
+
 <!-- [![Build Status](https://travis-ci.org/dw/mitogen.png?branch=master)](https://travis-ci.org/dw/mitogen}) -->
 <a href="https://mitogen.readthedocs.io/">Please see the documentation</a>.
+
+![](https://i.imgur.com/eBM6LhJ.gif)
+
+[![Total alerts](https://img.shields.io/lgtm/alerts/g/dw/mitogen.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/dw/mitogen/alerts/)
+
+[![Build Status](https://travis-ci.org/dw/mitogen.svg?branch=master)](https://travis-ci.org/dw/mitogen)
+
+[![Pipelines Status](https://dev.azure.com/dw-mitogen/Mitogen/_apis/build/status/dw.mitogen?branchName=master)](https://dev.azure.com/dw-mitogen/Mitogen/_build/latest?definitionId=1?branchName=master)

ansible_mitogen/affinity.py

@@ -0,0 +1,241 @@
+# Copyright 2017, David Wilson
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are met:
+#
+# 1. Redistributions of source code must retain the above copyright notice,
+# this list of conditions and the following disclaimer.
+#
+# 2. Redistributions in binary form must reproduce the above copyright notice,
+# this list of conditions and the following disclaimer in the documentation
+# and/or other materials provided with the distribution.
+#
+# 3. Neither the name of the copyright holder nor the names of its contributors
+# may be used to endorse or promote products derived from this software without
+# specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+# POSSIBILITY OF SUCH DAMAGE.
+
+"""
+As Mitogen separates asynchronous IO out to a broker thread, communication
+necessarily involves context switching and waking that thread. When application
+threads and the broker share a CPU, this can be almost invisibly fast - around
+25 microseconds for a full A->B->A round-trip.
+
+However when threads are scheduled on different CPUs, round-trip delays
+regularly vary wildly, and easily into milliseconds. Many contributing factors
+exist, not least scenarios like:
+
+1. A is preempted immediately after waking B, but before releasing the GIL.
+2. B wakes from IO wait only to immediately enter futex wait.
+3. A may wait 10ms or more for another timeslice, as the scheduler on its CPU
+   runs threads unrelated to its transaction (i.e. not B), wake only to release
+   its GIL, before entering IO sleep waiting for a reply from B, which cannot
+   exist yet.
+4. B wakes, acquires GIL, performs work, and sends reply to A, causing it to
+   wake. B is preempted before releasing GIL.
+5. A wakes from IO wait only to immediately enter futex wait.
+6. B may wait 10ms or more for another timeslice, wake only to release its GIL,
+   before sleeping again.
+7. A wakes, acquires GIL, finally receives reply.
+
+Per above if we are unlucky, on an even moderately busy machine it is possible
+to lose milliseconds just in scheduling delay, and the effect is compounded
+when pairs of threads in process A are communicating with pairs of threads in
+process B using the same scheme, such as when Ansible WorkerProcess is
+communicating with ContextService in the connection multiplexer. In the worst
+case it could involve 4 threads working in lockstep spread across 4 busy CPUs.
+
+Since multithreading in Python is essentially useless except for waiting on IO
+due to the presence of the GIL, at least in Ansible there is no good reason for
+threads in the same process to run on distinct CPUs - they always operate in
+lockstep due to the GIL, and are thus vulnerable to issues like above.
+
+Linux lacks any natural API to describe what we want, it only permits
+individual threads to be constrained to run on specific CPUs, and for that
+constraint to be inherited by new threads and forks of the constrained thread.
+
+This module therefore implements a CPU pinning policy for Ansible processes,
+providing methods that should be called early in any new process, either to
+rebalance which CPU it is pinned to, or in the case of subprocesses, to remove
+the pinning entirely. It is likely to require ongoing tweaking, since pinning
+necessarily involves preventing the scheduler from making load balancing
+decisions.
+"""
+
+import ctypes
+import mmap
+import multiprocessing
+import os
+import struct
+
+import mitogen.parent
+
+
+try:
+    _libc = ctypes.CDLL(None, use_errno=True)
+    _strerror = _libc.strerror
+    _strerror.restype = ctypes.c_char_p
+    _pthread_mutex_init = _libc.pthread_mutex_init
+    _pthread_mutex_lock = _libc.pthread_mutex_lock
+    _pthread_mutex_unlock = _libc.pthread_mutex_unlock
+    _sched_setaffinity = _libc.sched_setaffinity
+except (OSError, AttributeError):
+    _libc = None
+    _strerror = None
+    _pthread_mutex_init = None
+    _pthread_mutex_lock = None
+    _pthread_mutex_unlock = None
+    _sched_setaffinity = None
+
+
+class pthread_mutex_t(ctypes.Structure):
+    """
+    Wrap pthread_mutex_t to allow storing a lock in shared memory.
+    """
+    _fields_ = [
+        ('data', ctypes.c_uint8 * 512),
+    ]
+
+    def init(self):
+        if _pthread_mutex_init(self.data, 0):
+            raise Exception(_strerror(ctypes.get_errno()))
+
+    def acquire(self):
+        if _pthread_mutex_lock(self.data):
+            raise Exception(_strerror(ctypes.get_errno()))
+
+    def release(self):
+        if _pthread_mutex_unlock(self.data):
+            raise Exception(_strerror(ctypes.get_errno()))
+
+
+class State(ctypes.Structure):
+    """
+    Contents of shared memory segment. This allows :meth:`Manager.assign` to be
+    called from any child, since affinity assignment must happen from within
+    the context of the new child process.
+    """
+    _fields_ = [
+        ('lock', pthread_mutex_t),
+        ('counter', ctypes.c_uint8),
+    ]
+
+
+class Policy(object):
+    """
+    Process affinity policy.
+    """
+    def assign_controller(self):
+        """
+        Assign the Ansible top-level policy to this process.
+        """
+
+    def assign_muxprocess(self):
+        """
+        Assign the MuxProcess policy to this process.
+        """
+
+    def assign_worker(self):
+        """
+        Assign the WorkerProcess policy to this process.
+        """
+
+    def assign_subprocess(self):
+        """
+        Assign the helper subprocess policy to this process.
+        """
+
+
+class LinuxPolicy(Policy):
+    """
+    :class:`Policy` for Linux machines. The scheme here was tested on an
+    otherwise idle 16 thread machine.
+
+    - The connection multiplexer is pinned to CPU 0.
+    - The Ansible top-level (strategy) is pinned to CPU 1.
+    - WorkerProcesses are pinned sequentually to 2..N, wrapping around when no
+      more CPUs exist.
+    - Children such as SSH may be scheduled on any CPU except 0/1.
+
+    If the machine has less than 4 cores available, the top-level and workers
+    are pinned between CPU 2..N, i.e. no CPU is reserved for the top-level
+    process.
+
+    This could at least be improved by having workers pinned to independent
+    cores, before reusing the second hyperthread of an existing core.
+
+    A hook is installed that causes :meth:`reset` to run in the child of any
+    process created with :func:`mitogen.parent.detach_popen`, ensuring
+    CPU-intensive children like SSH are not forced to share the same core as
+    the (otherwise potentially very busy) parent.
+    """
+    def __init__(self):
+        self.mem = mmap.mmap(-1, 4096)
+        self.state = State.from_buffer(self.mem)
+        self.state.lock.init()
+        if self._cpu_count() < 4:
+            self._reserve_mask = 3
+            self._reserve_shift = 2
+            self._reserve_controller = True
+        else:
+            self._reserve_mask = 1
+            self._reserve_shift = 1
+            self._reserve_controller = False
+
+    def _set_affinity(self, mask):
+        mitogen.parent._preexec_hook = self._clear
+        s = struct.pack('L', mask)
+        _sched_setaffinity(os.getpid(), len(s), s)
+
+    def _cpu_count(self):
+        return multiprocessing.cpu_count()
+
+    def _balance(self):
+        self.state.lock.acquire()
+        try:
+            n = self.state.counter
+            self.state.counter += 1
+        finally:
+            self.state.lock.release()
+
+        self._set_cpu(self._reserve_shift + (
+            (n % max(1, (self._cpu_count() - self._reserve_shift)))
+        ))
+
+    def _set_cpu(self, cpu):
+        self._set_affinity(1 << cpu)
+
+    def _clear(self):
+        self._set_affinity(0xffffffff & ~self._reserve_mask)
+
+    def assign_controller(self):
+        if self._reserve_controller:
+            self._set_cpu(1)
+        else:
+            self._balance()
+
+    def assign_muxprocess(self):
+        self._set_cpu(0)
+
+    def assign_worker(self):
+        self._balance()
+
+    def assign_subprocess(self):
+        self._clear()
+
+
+if _sched_setaffinity is not None:
+    policy = LinuxPolicy()
+else:
+    policy = Policy()

ansible_mitogen/compat/simplejson/__init__.py

@@ -0,0 +1,318 @@
+r"""JSON (JavaScript Object Notation) <http://json.org> is a subset of
+JavaScript syntax (ECMA-262 3rd edition) used as a lightweight data
+interchange format.
+
+:mod:`simplejson` exposes an API familiar to users of the standard library
+:mod:`marshal` and :mod:`pickle` modules. It is the externally maintained
+version of the :mod:`json` library contained in Python 2.6, but maintains
+compatibility with Python 2.4 and Python 2.5 and (currently) has
+significant performance advantages, even without using the optional C
+extension for speedups.
+
+Encoding basic Python object hierarchies::
+
+    >>> import simplejson as json
+    >>> json.dumps(['foo', {'bar': ('baz', None, 1.0, 2)}])
+    '["foo", {"bar": ["baz", null, 1.0, 2]}]'
+    >>> print json.dumps("\"foo\bar")
+    "\"foo\bar"
+    >>> print json.dumps(u'\u1234')
+    "\u1234"
+    >>> print json.dumps('\\')
+    "\\"
+    >>> print json.dumps({"c": 0, "b": 0, "a": 0}, sort_keys=True)
+    {"a": 0, "b": 0, "c": 0}
+    >>> from StringIO import StringIO
+    >>> io = StringIO()
+    >>> json.dump(['streaming API'], io)
+    >>> io.getvalue()
+    '["streaming API"]'
+
+Compact encoding::
+
+    >>> import simplejson as json
+    >>> json.dumps([1,2,3,{'4': 5, '6': 7}], separators=(',',':'))
+    '[1,2,3,{"4":5,"6":7}]'
+
+Pretty printing::
+
+    >>> import simplejson as json
+    >>> s = json.dumps({'4': 5, '6': 7}, sort_keys=True, indent=4)
+    >>> print '\n'.join([l.rstrip() for l in  s.splitlines()])
+    {
+        "4": 5,
+        "6": 7
+    }
+
+Decoding JSON::
+
+    >>> import simplejson as json
+    >>> obj = [u'foo', {u'bar': [u'baz', None, 1.0, 2]}]
+    >>> json.loads('["foo", {"bar":["baz", null, 1.0, 2]}]') == obj
+    True
+    >>> json.loads('"\\"foo\\bar"') == u'"foo\x08ar'
+    True
+    >>> from StringIO import StringIO
+    >>> io = StringIO('["streaming API"]')
+    >>> json.load(io)[0] == 'streaming API'
+    True
+
+Specializing JSON object decoding::
+
+    >>> import simplejson as json
+    >>> def as_complex(dct):
+    ...     if '__complex__' in dct:
+    ...         return complex(dct['real'], dct['imag'])
+    ...     return dct
+    ...
+    >>> json.loads('{"__complex__": true, "real": 1, "imag": 2}',
+    ...     object_hook=as_complex)
+    (1+2j)
+    >>> import decimal
+    >>> json.loads('1.1', parse_float=decimal.Decimal) == decimal.Decimal('1.1')
+    True
+
+Specializing JSON object encoding::
+
+    >>> import simplejson as json
+    >>> def encode_complex(obj):
+    ...     if isinstance(obj, complex):
+    ...         return [obj.real, obj.imag]
+    ...     raise TypeError(repr(o) + " is not JSON serializable")
+    ...
+    >>> json.dumps(2 + 1j, default=encode_complex)
+    '[2.0, 1.0]'
+    >>> json.JSONEncoder(default=encode_complex).encode(2 + 1j)
+    '[2.0, 1.0]'
+    >>> ''.join(json.JSONEncoder(default=encode_complex).iterencode(2 + 1j))
+    '[2.0, 1.0]'
+
+
+Using simplejson.tool from the shell to validate and pretty-print::
+
+    $ echo '{"json":"obj"}' | python -m simplejson.tool
+    {
+        "json": "obj"
+    }
+    $ echo '{ 1.2:3.4}' | python -m simplejson.tool
+    Expecting property name: line 1 column 2 (char 2)
+"""
+__version__ = '2.0.9'
+__all__ = [
+    'dump', 'dumps', 'load', 'loads',
+    'JSONDecoder', 'JSONEncoder',
+]
+
+__author__ = 'Bob Ippolito <bob@redivi.com>'
+
+from decoder import JSONDecoder
+from encoder import JSONEncoder
+
+_default_encoder = JSONEncoder(
+    skipkeys=False,
+    ensure_ascii=True,
+    check_circular=True,
+    allow_nan=True,
+    indent=None,
+    separators=None,
+    encoding='utf-8',
+    default=None,
+)
+
+def dump(obj, fp, skipkeys=False, ensure_ascii=True, check_circular=True,
+        allow_nan=True, cls=None, indent=None, separators=None,
+        encoding='utf-8', default=None, **kw):
+    """Serialize ``obj`` as a JSON formatted stream to ``fp`` (a
+    ``.write()``-supporting file-like object).
+
+    If ``skipkeys`` is true then ``dict`` keys that are not basic types
+    (``str``, ``unicode``, ``int``, ``long``, ``float``, ``bool``, ``None``)
+    will be skipped instead of raising a ``TypeError``.
+
+    If ``ensure_ascii`` is false, then the some chunks written to ``fp``
+    may be ``unicode`` instances, subject to normal Python ``str`` to
+    ``unicode`` coercion rules. Unless ``fp.write()`` explicitly
+    understands ``unicode`` (as in ``codecs.getwriter()``) this is likely
+    to cause an error.
+
+    If ``check_circular`` is false, then the circular reference check
+    for container types will be skipped and a circular reference will
+    result in an ``OverflowError`` (or worse).
+
+    If ``allow_nan`` is false, then it will be a ``ValueError`` to
+    serialize out of range ``float`` values (``nan``, ``inf``, ``-inf``)
+    in strict compliance of the JSON specification, instead of using the
+    JavaScript equivalents (``NaN``, ``Infinity``, ``-Infinity``).
+
+    If ``indent`` is a non-negative integer, then JSON array elements and object
+    members will be pretty-printed with that indent level. An indent level
+    of 0 will only insert newlines. ``None`` is the most compact representation.
+
+    If ``separators`` is an ``(item_separator, dict_separator)`` tuple
+    then it will be used instead of the default ``(', ', ': ')`` separators.
+    ``(',', ':')`` is the most compact JSON representation.
+
+    ``encoding`` is the character encoding for str instances, default is UTF-8.
+
+    ``default(obj)`` is a function that should return a serializable version
+    of obj or raise TypeError. The default simply raises TypeError.
+
+    To use a custom ``JSONEncoder`` subclass (e.g. one that overrides the
+    ``.default()`` method to serialize additional types), specify it with
+    the ``cls`` kwarg.
+
+    """
+    # cached encoder
+    if (not skipkeys and ensure_ascii and
+        check_circular and allow_nan and
+        cls is None and indent is None and separators is None and
+        encoding == 'utf-8' and default is None and not kw):
+        iterable = _default_encoder.iterencode(obj)
+    else:
+        if cls is None:
+            cls = JSONEncoder
+        iterable = cls(skipkeys=skipkeys, ensure_ascii=ensure_ascii,
+            check_circular=check_circular, allow_nan=allow_nan, indent=indent,
+            separators=separators, encoding=encoding,
+            default=default, **kw).iterencode(obj)
+    # could accelerate with writelines in some versions of Python, at
+    # a debuggability cost
+    for chunk in iterable:
+        fp.write(chunk)
+
+
+def dumps(obj, skipkeys=False, ensure_ascii=True, check_circular=True,
+        allow_nan=True, cls=None, indent=None, separators=None,
+        encoding='utf-8', default=None, **kw):
+    """Serialize ``obj`` to a JSON formatted ``str``.
+
+    If ``skipkeys`` is false then ``dict`` keys that are not basic types
+    (``str``, ``unicode``, ``int``, ``long``, ``float``, ``bool``, ``None``)
+    will be skipped instead of raising a ``TypeError``.
+
+    If ``ensure_ascii`` is false, then the return value will be a
+    ``unicode`` instance subject to normal Python ``str`` to ``unicode``
+    coercion rules instead of being escaped to an ASCII ``str``.
+
+    If ``check_circular`` is false, then the circular reference check
+    for container types will be skipped and a circular reference will
+    result in an ``OverflowError`` (or worse).
+
+    If ``allow_nan`` is false, then it will be a ``ValueError`` to
+    serialize out of range ``float`` values (``nan``, ``inf``, ``-inf``) in
+    strict compliance of the JSON specification, instead of using the
+    JavaScript equivalents (``NaN``, ``Infinity``, ``-Infinity``).
+
+    If ``indent`` is a non-negative integer, then JSON array elements and
+    object members will be pretty-printed with that indent level. An indent
+    level of 0 will only insert newlines. ``None`` is the most compact
+    representation.
+
+    If ``separators`` is an ``(item_separator, dict_separator)`` tuple
+    then it will be used instead of the default ``(', ', ': ')`` separators.
+    ``(',', ':')`` is the most compact JSON representation.
+
+    ``encoding`` is the character encoding for str instances, default is UTF-8.
+
+    ``default(obj)`` is a function that should return a serializable version
+    of obj or raise TypeError. The default simply raises TypeError.
+
+    To use a custom ``JSONEncoder`` subclass (e.g. one that overrides the
+    ``.default()`` method to serialize additional types), specify it with
+    the ``cls`` kwarg.
+
+    """
+    # cached encoder
+    if (not skipkeys and ensure_ascii and
+        check_circular and allow_nan and
+        cls is None and indent is None and separators is None and
+        encoding == 'utf-8' and default is None and not kw):
+        return _default_encoder.encode(obj)
+    if cls is None:
+        cls = JSONEncoder
+    return cls(
+        skipkeys=skipkeys, ensure_ascii=ensure_ascii,
+        check_circular=check_circular, allow_nan=allow_nan, indent=indent,
+        separators=separators, encoding=encoding, default=default,
+        **kw).encode(obj)
+
+
+_default_decoder = JSONDecoder(encoding=None, object_hook=None)
+
+
+def load(fp, encoding=None, cls=None, object_hook=None, parse_float=None,
+        parse_int=None, parse_constant=None, **kw):
+    """Deserialize ``fp`` (a ``.read()``-supporting file-like object containing
+    a JSON document) to a Python object.
+
+    If the contents of ``fp`` is encoded with an ASCII based encoding other
+    than utf-8 (e.g. latin-1), then an appropriate ``encoding`` name must
+    be specified. Encodings that are not ASCII based (such as UCS-2) are
+    not allowed, and should be wrapped with
+    ``codecs.getreader(fp)(encoding)``, or simply decoded to a ``unicode``
+    object and passed to ``loads()``
+
+    ``object_hook`` is an optional function that will be called with the
+    result of any object literal decode (a ``dict``). The return value of
+    ``object_hook`` will be used instead of the ``dict``. This feature
+    can be used to implement custom decoders (e.g. JSON-RPC class hinting).
+
+    To use a custom ``JSONDecoder`` subclass, specify it with the ``cls``
+    kwarg.
+
+    """
+    return loads(fp.read(),
+        encoding=encoding, cls=cls, object_hook=object_hook,
+        parse_float=parse_float, parse_int=parse_int,
+        parse_constant=parse_constant, **kw)
+
+
+def loads(s, encoding=None, cls=None, object_hook=None, parse_float=None,
+        parse_int=None, parse_constant=None, **kw):
+    """Deserialize ``s`` (a ``str`` or ``unicode`` instance containing a JSON
+    document) to a Python object.
+
+    If ``s`` is a ``str`` instance and is encoded with an ASCII based encoding
+    other than utf-8 (e.g. latin-1) then an appropriate ``encoding`` name
+    must be specified. Encodings that are not ASCII based (such as UCS-2)
+    are not allowed and should be decoded to ``unicode`` first.
+
+    ``object_hook`` is an optional function that will be called with the
+    result of any object literal decode (a ``dict``). The return value of
+    ``object_hook`` will be used instead of the ``dict``. This feature
+    can be used to implement custom decoders (e.g. JSON-RPC class hinting).
+
+    ``parse_float``, if specified, will be called with the string
+    of every JSON float to be decoded. By default this is equivalent to
+    float(num_str). This can be used to use another datatype or parser
+    for JSON floats (e.g. decimal.Decimal).
+
+    ``parse_int``, if specified, will be called with the string
+    of every JSON int to be decoded. By default this is equivalent to
+    int(num_str). This can be used to use another datatype or parser
+    for JSON integers (e.g. float).
+
+    ``parse_constant``, if specified, will be called with one of the
+    following strings: -Infinity, Infinity, NaN, null, true, false.
+    This can be used to raise an exception if invalid JSON numbers
+    are encountered.
+
+    To use a custom ``JSONDecoder`` subclass, specify it with the ``cls``
+    kwarg.
+
+    """
+    if (cls is None and encoding is None and object_hook is None and
+            parse_int is None and parse_float is None and
+            parse_constant is None and not kw):
+        return _default_decoder.decode(s)
+    if cls is None:
+        cls = JSONDecoder
+    if object_hook is not None:
+        kw['object_hook'] = object_hook
+    if parse_float is not None:
+        kw['parse_float'] = parse_float
+    if parse_int is not None:
+        kw['parse_int'] = parse_int
+    if parse_constant is not None:
+        kw['parse_constant'] = parse_constant
+    return cls(encoding=encoding, **kw).decode(s)

ansible_mitogen/compat/simplejson/decoder.py

@@ -0,0 +1,354 @@
+"""Implementation of JSONDecoder
+"""
+import re
+import sys
+import struct
+
+from simplejson.scanner import make_scanner
+try:
+    from simplejson._speedups import scanstring as c_scanstring
+except ImportError:
+    c_scanstring = None
+
+__all__ = ['JSONDecoder']
+
+FLAGS = re.VERBOSE | re.MULTILINE | re.DOTALL
+
+def _floatconstants():
+    _BYTES = '7FF80000000000007FF0000000000000'.decode('hex')
+    if sys.byteorder != 'big':
+        _BYTES = _BYTES[:8][::-1] + _BYTES[8:][::-1]
+    nan, inf = struct.unpack('dd', _BYTES)
+    return nan, inf, -inf
+
+NaN, PosInf, NegInf = _floatconstants()
+
+
+def linecol(doc, pos):
+    lineno = doc.count('\n', 0, pos) + 1
+    if lineno == 1:
+        colno = pos
+    else:
+        colno = pos - doc.rindex('\n', 0, pos)
+    return lineno, colno
+
+
+def errmsg(msg, doc, pos, end=None):
+    # Note that this function is called from _speedups
+    lineno, colno = linecol(doc, pos)
+    if end is None:
+        #fmt = '{0}: line {1} column {2} (char {3})'
+        #return fmt.format(msg, lineno, colno, pos)
+        fmt = '%s: line %d column %d (char %d)'
+        return fmt % (msg, lineno, colno, pos)
+    endlineno, endcolno = linecol(doc, end)
+    #fmt = '{0}: line {1} column {2} - line {3} column {4} (char {5} - {6})'
+    #return fmt.format(msg, lineno, colno, endlineno, endcolno, pos, end)
+    fmt = '%s: line %d column %d - line %d column %d (char %d - %d)'
+    return fmt % (msg, lineno, colno, endlineno, endcolno, pos, end)
+
+
+_CONSTANTS = {
+    '-Infinity': NegInf,
+    'Infinity': PosInf,
+    'NaN': NaN,
+}
+
+STRINGCHUNK = re.compile(r'(.*?)(["\\\x00-\x1f])', FLAGS)
+BACKSLASH = {
+    '"': u'"', '\\': u'\\', '/': u'/',
+    'b': u'\b', 'f': u'\f', 'n': u'\n', 'r': u'\r', 't': u'\t',
+}
+
+DEFAULT_ENCODING = "utf-8"
+
+def py_scanstring(s, end, encoding=None, strict=True, _b=BACKSLASH, _m=STRINGCHUNK.match):
+    """Scan the string s for a JSON string. End is the index of the
+    character in s after the quote that started the JSON string.
+    Unescapes all valid JSON string escape sequences and raises ValueError
+    on attempt to decode an invalid string. If strict is False then literal
+    control characters are allowed in the string.
+    
+    Returns a tuple of the decoded string and the index of the character in s
+    after the end quote."""
+    if encoding is None:
+        encoding = DEFAULT_ENCODING
+    chunks = []
+    _append = chunks.append
+    begin = end - 1
+    while 1:
+        chunk = _m(s, end)
+        if chunk is None:
+            raise ValueError(
+                errmsg("Unterminated string starting at", s, begin))
+        end = chunk.end()
+        content, terminator = chunk.groups()
+        # Content is contains zero or more unescaped string characters
+        if content:
+            if not isinstance(content, unicode):
+                content = unicode(content, encoding)
+            _append(content)
+        # Terminator is the end of string, a literal control character,
+        # or a backslash denoting that an escape sequence follows
+        if terminator == '"':
+            break
+        elif terminator != '\\':
+            if strict:
+                msg = "Invalid control character %r at" % (terminator,)
+                #msg = "Invalid control character {0!r} at".format(terminator)
+                raise ValueError(errmsg(msg, s, end))
+            else:
+                _append(terminator)
+                continue
+        try:
+            esc = s[end]
+        except IndexError:
+            raise ValueError(
+                errmsg("Unterminated string starting at", s, begin))
+        # If not a unicode escape sequence, must be in the lookup table
+        if esc != 'u':
+            try:
+                char = _b[esc]
+            except KeyError:
+                msg = "Invalid \\escape: " + repr(esc)
+                raise ValueError(errmsg(msg, s, end))
+            end += 1
+        else:
+            # Unicode escape sequence
+            esc = s[end + 1:end + 5]
+            next_end = end + 5
+            if len(esc) != 4:
+                msg = "Invalid \\uXXXX escape"
+                raise ValueError(errmsg(msg, s, end))
+            uni = int(esc, 16)
+            # Check for surrogate pair on UCS-4 systems
+            if 0xd800 <= uni <= 0xdbff and sys.maxunicode > 65535:
+                msg = "Invalid \\uXXXX\\uXXXX surrogate pair"
+                if not s[end + 5:end + 7] == '\\u':
+                    raise ValueError(errmsg(msg, s, end))
+                esc2 = s[end + 7:end + 11]
+                if len(esc2) != 4:
+                    raise ValueError(errmsg(msg, s, end))
+                uni2 = int(esc2, 16)
+                uni = 0x10000 + (((uni - 0xd800) << 10) | (uni2 - 0xdc00))
+                next_end += 6
+            char = unichr(uni)
+            end = next_end
+        # Append the unescaped character
+        _append(char)
+    return u''.join(chunks), end
+
+
+# Use speedup if available
+scanstring = c_scanstring or py_scanstring
+
+WHITESPACE = re.compile(r'[ \t\n\r]*', FLAGS)
+WHITESPACE_STR = ' \t\n\r'
+
+def JSONObject((s, end), encoding, strict, scan_once, object_hook, _w=WHITESPACE.match, _ws=WHITESPACE_STR):
+    pairs = {}
+    # Use a slice to prevent IndexError from being raised, the following
+    # check will raise a more specific ValueError if the string is empty
+    nextchar = s[end:end + 1]
+    # Normally we expect nextchar == '"'
+    if nextchar != '"':
+        if nextchar in _ws:
+            end = _w(s, end).end()
+            nextchar = s[end:end + 1]
+        # Trivial empty object
+        if nextchar == '}':
+            return pairs, end + 1
+        elif nextchar != '"':
+            raise ValueError(errmsg("Expecting property name", s, end))
+    end += 1
+    while True:
+        key, end = scanstring(s, end, encoding, strict)
+
+        # To skip some function call overhead we optimize the fast paths where
+        # the JSON key separator is ": " or just ":".
+        if s[end:end + 1] != ':':
+            end = _w(s, end).end()
+            if s[end:end + 1] != ':':
+                raise ValueError(errmsg("Expecting : delimiter", s, end))
+
+        end += 1
+
+        try:
+            if s[end] in _ws:
+                end += 1
+                if s[end] in _ws:
+                    end = _w(s, end + 1).end()
+        except IndexError:
+            pass
+
+        try:
+            value, end = scan_once(s, end)
+        except StopIteration:
+            raise ValueError(errmsg("Expecting object", s, end))
+        pairs[key] = value
+
+        try:
+            nextchar = s[end]
+            if nextchar in _ws:
+                end = _w(s, end + 1).end()
+                nextchar = s[end]
+        except IndexError:
+            nextchar = ''
+        end += 1
+
+        if nextchar == '}':
+            break
+        elif nextchar != ',':
+            raise ValueError(errmsg("Expecting , delimiter", s, end - 1))
+
+        try:
+            nextchar = s[end]
+            if nextchar in _ws:
+                end += 1
+                nextchar = s[end]
+                if nextchar in _ws:
+                    end = _w(s, end + 1).end()
+                    nextchar = s[end]
+        except IndexError:
+            nextchar = ''
+
+        end += 1
+        if nextchar != '"':
+            raise ValueError(errmsg("Expecting property name", s, end - 1))
+
+    if object_hook is not None:
+        pairs = object_hook(pairs)
+    return pairs, end
+
+def JSONArray((s, end), scan_once, _w=WHITESPACE.match, _ws=WHITESPACE_STR):
+    values = []
+    nextchar = s[end:end + 1]
+    if nextchar in _ws:
+        end = _w(s, end + 1).end()
+        nextchar = s[end:end + 1]
+    # Look-ahead for trivial empty array
+    if nextchar == ']':
+        return values, end + 1
+    _append = values.append
+    while True:
+        try:
+            value, end = scan_once(s, end)
+        except StopIteration:
+            raise ValueError(errmsg("Expecting object", s, end))
+        _append(value)
+        nextchar = s[end:end + 1]
+        if nextchar in _ws:
+            end = _w(s, end + 1).end()
+            nextchar = s[end:end + 1]
+        end += 1
+        if nextchar == ']':
+            break
+        elif nextchar != ',':
+            raise ValueError(errmsg("Expecting , delimiter", s, end))
+
+        try:
+            if s[end] in _ws:
+                end += 1
+                if s[end] in _ws:
+                    end = _w(s, end + 1).end()
+        except IndexError:
+            pass
+
+    return values, end
+
+class JSONDecoder(object):
+    """Simple JSON <http://json.org> decoder
+
+    Performs the following translations in decoding by default:
+
+    +---------------+-------------------+
+    | JSON          | Python            |
+    +===============+===================+
+    | object        | dict              |
+    +---------------+-------------------+
+    | array         | list              |
+    +---------------+-------------------+
+    | string        | unicode           |
+    +---------------+-------------------+
+    | number (int)  | int, long         |
+    +---------------+-------------------+
+    | number (real) | float             |
+    +---------------+-------------------+
+    | true          | True              |
+    +---------------+-------------------+
+    | false         | False             |
+    +---------------+-------------------+
+    | null          | None              |
+    +---------------+-------------------+
+
+    It also understands ``NaN``, ``Infinity``, and ``-Infinity`` as
+    their corresponding ``float`` values, which is outside the JSON spec.
+
+    """
+
+    def __init__(self, encoding=None, object_hook=None, parse_float=None,
+            parse_int=None, parse_constant=None, strict=True):
+        """``encoding`` determines the encoding used to interpret any ``str``
+        objects decoded by this instance (utf-8 by default).  It has no
+        effect when decoding ``unicode`` objects.
+
+        Note that currently only encodings that are a superset of ASCII work,
+        strings of other encodings should be passed in as ``unicode``.
+
+        ``object_hook``, if specified, will be called with the result
+        of every JSON object decoded and its return value will be used in
+        place of the given ``dict``.  This can be used to provide custom
+        deserializations (e.g. to support JSON-RPC class hinting).
+
+        ``parse_float``, if specified, will be called with the string
+        of every JSON float to be decoded. By default this is equivalent to
+        float(num_str). This can be used to use another datatype or parser
+        for JSON floats (e.g. decimal.Decimal).
+
+        ``parse_int``, if specified, will be called with the string
+        of every JSON int to be decoded. By default this is equivalent to
+        int(num_str). This can be used to use another datatype or parser
+        for JSON integers (e.g. float).
+
+        ``parse_constant``, if specified, will be called with one of the
+        following strings: -Infinity, Infinity, NaN.
+        This can be used to raise an exception if invalid JSON numbers
+        are encountered.
+
+        """
+        self.encoding = encoding
+        self.object_hook = object_hook
+        self.parse_float = parse_float or float
+        self.parse_int = parse_int or int
+        self.parse_constant = parse_constant or _CONSTANTS.__getitem__
+        self.strict = strict
+        self.parse_object = JSONObject
+        self.parse_array = JSONArray
+        self.parse_string = scanstring
+        self.scan_once = make_scanner(self)
+
+    def decode(self, s, _w=WHITESPACE.match):
+        """Return the Python representation of ``s`` (a ``str`` or ``unicode``
+        instance containing a JSON document)
+
+        """
+        obj, end = self.raw_decode(s, idx=_w(s, 0).end())
+        end = _w(s, end).end()
+        if end != len(s):
+            raise ValueError(errmsg("Extra data", s, end, len(s)))
+        return obj
+
+    def raw_decode(self, s, idx=0):
+        """Decode a JSON document from ``s`` (a ``str`` or ``unicode`` beginning
+        with a JSON document) and return a 2-tuple of the Python
+        representation and the index in ``s`` where the document ended.
+
+        This can be used to decode a JSON document from a string that may
+        have extraneous data at the end.
+
+        """
+        try:
+            obj, end = self.scan_once(s, idx)
+        except StopIteration:
+            raise ValueError("No JSON object could be decoded")
+        return obj, end

ansible_mitogen/compat/simplejson/encoder.py

@@ -0,0 +1,440 @@
+"""Implementation of JSONEncoder
+"""
+import re
+
+try:
+    from simplejson._speedups import encode_basestring_ascii as c_encode_basestring_ascii
+except ImportError:
+    c_encode_basestring_ascii = None
+try:
+    from simplejson._speedups import make_encoder as c_make_encoder
+except ImportError:
+    c_make_encoder = None
+
+ESCAPE = re.compile(r'[\x00-\x1f\\"\b\f\n\r\t]')
+ESCAPE_ASCII = re.compile(r'([\\"]|[^\ -~])')
+HAS_UTF8 = re.compile(r'[\x80-\xff]')
+ESCAPE_DCT = {
+    '\\': '\\\\',
+    '"': '\\"',
+    '\b': '\\b',
+    '\f': '\\f',
+    '\n': '\\n',
+    '\r': '\\r',
+    '\t': '\\t',
+}
+for i in range(0x20):
+    #ESCAPE_DCT.setdefault(chr(i), '\\u{0:04x}'.format(i))
+    ESCAPE_DCT.setdefault(chr(i), '\\u%04x' % (i,))
+
+# Assume this produces an infinity on all machines (probably not guaranteed)
+INFINITY = float('1e66666')
+FLOAT_REPR = repr
+
+def encode_basestring(s):
+    """Return a JSON representation of a Python string
+
+    """
+    def replace(match):
+        return ESCAPE_DCT[match.group(0)]
+    return '"' + ESCAPE.sub(replace, s) + '"'
+
+
+def py_encode_basestring_ascii(s):
+    """Return an ASCII-only JSON representation of a Python string
+
+    """
+    if isinstance(s, str) and HAS_UTF8.search(s) is not None:
+        s = s.decode('utf-8')
+    def replace(match):
+        s = match.group(0)
+        try:
+            return ESCAPE_DCT[s]
+        except KeyError:
+            n = ord(s)
+            if n < 0x10000:
+                #return '\\u{0:04x}'.format(n)
+                return '\\u%04x' % (n,)
+            else:
+                # surrogate pair
+                n -= 0x10000
+                s1 = 0xd800 | ((n >> 10) & 0x3ff)
+                s2 = 0xdc00 | (n & 0x3ff)
+                #return '\\u{0:04x}\\u{1:04x}'.format(s1, s2)
+                return '\\u%04x\\u%04x' % (s1, s2)
+    return '"' + str(ESCAPE_ASCII.sub(replace, s)) + '"'
+
+
+encode_basestring_ascii = c_encode_basestring_ascii or py_encode_basestring_ascii
+
+class JSONEncoder(object):
+    """Extensible JSON <http://json.org> encoder for Python data structures.
+
+    Supports the following objects and types by default:
+
+    +-------------------+---------------+
+    | Python            | JSON          |
+    +===================+===============+
+    | dict              | object        |
+    +-------------------+---------------+
+    | list, tuple       | array         |
+    +-------------------+---------------+
+    | str, unicode      | string        |
+    +-------------------+---------------+
+    | int, long, float  | number        |
+    +-------------------+---------------+
+    | True              | true          |
+    +-------------------+---------------+
+    | False             | false         |
+    +-------------------+---------------+
+    | None              | null          |
+    +-------------------+---------------+
+
+    To extend this to recognize other objects, subclass and implement a
+    ``.default()`` method with another method that returns a serializable
+    object for ``o`` if possible, otherwise it should call the superclass
+    implementation (to raise ``TypeError``).
+
+    """
+    item_separator = ', '
+    key_separator = ': '
+    def __init__(self, skipkeys=False, ensure_ascii=True,
+            check_circular=True, allow_nan=True, sort_keys=False,
+            indent=None, separators=None, encoding='utf-8', default=None):
+        """Constructor for JSONEncoder, with sensible defaults.
+
+        If skipkeys is false, then it is a TypeError to attempt
+        encoding of keys that are not str, int, long, float or None.  If
+        skipkeys is True, such items are simply skipped.
+
+        If ensure_ascii is true, the output is guaranteed to be str
+        objects with all incoming unicode characters escaped.  If
+        ensure_ascii is false, the output will be unicode object.
+
+        If check_circular is true, then lists, dicts, and custom encoded
+        objects will be checked for circular references during encoding to
+        prevent an infinite recursion (which would cause an OverflowError).
+        Otherwise, no such check takes place.
+
+        If allow_nan is true, then NaN, Infinity, and -Infinity will be
+        encoded as such.  This behavior is not JSON specification compliant,
+        but is consistent with most JavaScript based encoders and decoders.
+        Otherwise, it will be a ValueError to encode such floats.
+
+        If sort_keys is true, then the output of dictionaries will be
+        sorted by key; this is useful for regression tests to ensure
+        that JSON serializations can be compared on a day-to-day basis.
+
+        If indent is a non-negative integer, then JSON array
+        elements and object members will be pretty-printed with that
+        indent level.  An indent level of 0 will only insert newlines.
+        None is the most compact representation.
+
+        If specified, separators should be a (item_separator, key_separator)
+        tuple.  The default is (', ', ': ').  To get the most compact JSON
+        representation you should specify (',', ':') to eliminate whitespace.
+
+        If specified, default is a function that gets called for objects
+        that can't otherwise be serialized.  It should return a JSON encodable
+        version of the object or raise a ``TypeError``.
+
+        If encoding is not None, then all input strings will be
+        transformed into unicode using that encoding prior to JSON-encoding.
+        The default is UTF-8.
+
+        """
+
+        self.skipkeys = skipkeys
+        self.ensure_ascii = ensure_ascii
+        self.check_circular = check_circular
+        self.allow_nan = allow_nan
+        self.sort_keys = sort_keys
+        self.indent = indent
+        if separators is not None:
+            self.item_separator, self.key_separator = separators
+        if default is not None:
+            self.default = default
+        self.encoding = encoding
+
+    def default(self, o):
+        """Implement this method in a subclass such that it returns
+        a serializable object for ``o``, or calls the base implementation
+        (to raise a ``TypeError``).
+
+        For example, to support arbitrary iterators, you could
+        implement default like this::
+
+            def default(self, o):
+                try:
+                    iterable = iter(o)
+                except TypeError:
+                    pass
+                else:
+                    return list(iterable)
+                return JSONEncoder.default(self, o)
+
+        """
+        raise TypeError(repr(o) + " is not JSON serializable")
+
+    def encode(self, o):
+        """Return a JSON string representation of a Python data structure.
+
+        >>> JSONEncoder().encode({"foo": ["bar", "baz"]})
+        '{"foo": ["bar", "baz"]}'
+
+        """
+        # This is for extremely simple cases and benchmarks.
+        if isinstance(o, basestring):
+            if isinstance(o, str):
+                _encoding = self.encoding
+                if (_encoding is not None
+                        and not (_encoding == 'utf-8')):
+                    o = o.decode(_encoding)
+            if self.ensure_ascii:
+                return encode_basestring_ascii(o)
+            else:
+                return encode_basestring(o)
+        # This doesn't pass the iterator directly to ''.join() because the
+        # exceptions aren't as detailed.  The list call should be roughly
+        # equivalent to the PySequence_Fast that ''.join() would do.
+        chunks = self.iterencode(o, _one_shot=True)
+        if not isinstance(chunks, (list, tuple)):
+            chunks = list(chunks)
+        return ''.join(chunks)
+
+    def iterencode(self, o, _one_shot=False):
+        """Encode the given object and yield each string
+        representation as available.
+
+        For example::
+
+            for chunk in JSONEncoder().iterencode(bigobject):
+                mysocket.write(chunk)
+
+        """
+        if self.check_circular:
+            markers = {}
+        else:
+            markers = None
+        if self.ensure_ascii:
+            _encoder = encode_basestring_ascii
+        else:
+            _encoder = encode_basestring
+        if self.encoding != 'utf-8':
+            def _encoder(o, _orig_encoder=_encoder, _encoding=self.encoding):
+                if isinstance(o, str):
+                    o = o.decode(_encoding)
+                return _orig_encoder(o)
+
+        def floatstr(o, allow_nan=self.allow_nan, _repr=FLOAT_REPR, _inf=INFINITY, _neginf=-INFINITY):
+            # Check for specials.  Note that this type of test is processor- and/or
+            # platform-specific, so do tests which don't depend on the internals.
+
+            if o != o:
+                text = 'NaN'
+            elif o == _inf:
+                text = 'Infinity'
+            elif o == _neginf:
+                text = '-Infinity'
+            else:
+                return _repr(o)
+
+            if not allow_nan:
+                raise ValueError(
+                    "Out of range float values are not JSON compliant: " +
+                    repr(o))
+
+            return text
+
+
+        if _one_shot and c_make_encoder is not None and not self.indent and not self.sort_keys:
+            _iterencode = c_make_encoder(
+                markers, self.default, _encoder, self.indent,
+                self.key_separator, self.item_separator, self.sort_keys,
+                self.skipkeys, self.allow_nan)
+        else:
+            _iterencode = _make_iterencode(
+                markers, self.default, _encoder, self.indent, floatstr,
+                self.key_separator, self.item_separator, self.sort_keys,
+                self.skipkeys, _one_shot)
+        return _iterencode(o, 0)
+
+def _make_iterencode(markers, _default, _encoder, _indent, _floatstr, _key_separator, _item_separator, _sort_keys, _skipkeys, _one_shot,
+        ## HACK: hand-optimized bytecode; turn globals into locals
+        False=False,
+        True=True,
+        ValueError=ValueError,
+        basestring=basestring,
+        dict=dict,
+        float=float,
+        id=id,
+        int=int,
+        isinstance=isinstance,
+        list=list,
+        long=long,
+        str=str,
+        tuple=tuple,
+    ):
+
+    def _iterencode_list(lst, _current_indent_level):
+        if not lst:
+            yield '[]'
+            return
+        if markers is not None:
+            markerid = id(lst)
+            if markerid in markers:
+                raise ValueError("Circular reference detected")
+            markers[markerid] = lst
+        buf = '['
+        if _indent is not None:
+            _current_indent_level += 1
+            newline_indent = '\n' + (' ' * (_indent * _current_indent_level))
+            separator = _item_separator + newline_indent
+            buf += newline_indent
+        else:
+            newline_indent = None
+            separator = _item_separator
+        first = True
+        for value in lst:
+            if first:
+                first = False
+            else:
+                buf = separator
+            if isinstance(value, basestring):
+                yield buf + _encoder(value)
+            elif value is None:
+                yield buf + 'null'
+            elif value is True:
+                yield buf + 'true'
+            elif value is False:
+                yield buf + 'false'
+            elif isinstance(value, (int, long)):
+                yield buf + str(value)
+            elif isinstance(value, float):
+                yield buf + _floatstr(value)
+            else:
+                yield buf
+                if isinstance(value, (list, tuple)):
+                    chunks = _iterencode_list(value, _current_indent_level)
+                elif isinstance(value, dict):
+                    chunks = _iterencode_dict(value, _current_indent_level)
+                else:
+                    chunks = _iterencode(value, _current_indent_level)
+                for chunk in chunks:
+                    yield chunk
+        if newline_indent is not None:
+            _current_indent_level -= 1
+            yield '\n' + (' ' * (_indent * _current_indent_level))
+        yield ']'
+        if markers is not None:
+            del markers[markerid]
+
+    def _iterencode_dict(dct, _current_indent_level):
+        if not dct:
+            yield '{}'
+            return
+        if markers is not None:
+            markerid = id(dct)
+            if markerid in markers:
+                raise ValueError("Circular reference detected")
+            markers[markerid] = dct
+        yield '{'
+        if _indent is not None:
+            _current_indent_level += 1
+            newline_indent = '\n' + (' ' * (_indent * _current_indent_level))
+            item_separator = _item_separator + newline_indent
+            yield newline_indent
+        else:
+            newline_indent = None
+            item_separator = _item_separator
+        first = True
+        if _sort_keys:
+            items = dct.items()
+            items.sort(key=lambda kv: kv[0])
+        else:
+            items = dct.iteritems()
+        for key, value in items:
+            if isinstance(key, basestring):
+                pass
+            # JavaScript is weakly typed for these, so it makes sense to
+            # also allow them.  Many encoders seem to do something like this.
+            elif isinstance(key, float):
+                key = _floatstr(key)
+            elif key is True:
+                key = 'true'
+            elif key is False:
+                key = 'false'
+            elif key is None:
+                key = 'null'
+            elif isinstance(key, (int, long)):
+                key = str(key)
+            elif _skipkeys:
+                continue
+            else:
+                raise TypeError("key " + repr(key) + " is not a string")
+            if first:
+                first = False
+            else:
+                yield item_separator
+            yield _encoder(key)
+            yield _key_separator
+            if isinstance(value, basestring):
+                yield _encoder(value)
+            elif value is None:
+                yield 'null'
+            elif value is True:
+                yield 'true'
+            elif value is False:
+                yield 'false'
+            elif isinstance(value, (int, long)):
+                yield str(value)
+            elif isinstance(value, float):
+                yield _floatstr(value)
+            else:
+                if isinstance(value, (list, tuple)):
+                    chunks = _iterencode_list(value, _current_indent_level)
+                elif isinstance(value, dict):
+                    chunks = _iterencode_dict(value, _current_indent_level)
+                else:
+                    chunks = _iterencode(value, _current_indent_level)
+                for chunk in chunks:
+                    yield chunk
+        if newline_indent is not None:
+            _current_indent_level -= 1
+            yield '\n' + (' ' * (_indent * _current_indent_level))
+        yield '}'
+        if markers is not None:
+            del markers[markerid]
+
+    def _iterencode(o, _current_indent_level):
+        if isinstance(o, basestring):
+            yield _encoder(o)
+        elif o is None:
+            yield 'null'
+        elif o is True:
+            yield 'true'
+        elif o is False:
+            yield 'false'
+        elif isinstance(o, (int, long)):
+            yield str(o)
+        elif isinstance(o, float):
+            yield _floatstr(o)
+        elif isinstance(o, (list, tuple)):
+            for chunk in _iterencode_list(o, _current_indent_level):
+                yield chunk
+        elif isinstance(o, dict):
+            for chunk in _iterencode_dict(o, _current_indent_level):
+                yield chunk
+        else:
+            if markers is not None:
+                markerid = id(o)
+                if markerid in markers:
+                    raise ValueError("Circular reference detected")
+                markers[markerid] = o
+            o = _default(o)
+            for chunk in _iterencode(o, _current_indent_level):
+                yield chunk
+            if markers is not None:
+                del markers[markerid]
+
+    return _iterencode

ansible_mitogen/compat/simplejson/scanner.py

@@ -0,0 +1,65 @@
+"""JSON token scanner
+"""
+import re
+try:
+    from simplejson._speedups import make_scanner as c_make_scanner
+except ImportError:
+    c_make_scanner = None
+
+__all__ = ['make_scanner']
+
+NUMBER_RE = re.compile(
+    r'(-?(?:0|[1-9]\d*))(\.\d+)?([eE][-+]?\d+)?',
+    (re.VERBOSE | re.MULTILINE | re.DOTALL))
+
+def py_make_scanner(context):
+    parse_object = context.parse_object
+    parse_array = context.parse_array
+    parse_string = context.parse_string
+    match_number = NUMBER_RE.match
+    encoding = context.encoding
+    strict = context.strict
+    parse_float = context.parse_float
+    parse_int = context.parse_int
+    parse_constant = context.parse_constant
+    object_hook = context.object_hook
+
+    def _scan_once(string, idx):
+        try:
+            nextchar = string[idx]
+        except IndexError:
+            raise StopIteration
+
+        if nextchar == '"':
+            return parse_string(string, idx + 1, encoding, strict)
+        elif nextchar == '{':
+            return parse_object((string, idx + 1), encoding, strict, _scan_once, object_hook)
+        elif nextchar == '[':
+            return parse_array((string, idx + 1), _scan_once)
+        elif nextchar == 'n' and string[idx:idx + 4] == 'null':
+            return None, idx + 4
+        elif nextchar == 't' and string[idx:idx + 4] == 'true':
+            return True, idx + 4
+        elif nextchar == 'f' and string[idx:idx + 5] == 'false':
+            return False, idx + 5
+
+        m = match_number(string, idx)
+        if m is not None:
+            integer, frac, exp = m.groups()
+            if frac or exp:
+                res = parse_float(integer + (frac or '') + (exp or ''))
+            else:
+                res = parse_int(integer)
+            return res, m.end()
+        elif nextchar == 'N' and string[idx:idx + 3] == 'NaN':
+            return parse_constant('NaN'), idx + 3
+        elif nextchar == 'I' and string[idx:idx + 8] == 'Infinity':
+            return parse_constant('Infinity'), idx + 8
+        elif nextchar == '-' and string[idx:idx + 9] == '-Infinity':
+            return parse_constant('-Infinity'), idx + 9
+        else:
+            raise StopIteration
+
+    return _scan_once
+
+make_scanner = c_make_scanner or py_make_scanner

ansible_mitogen/connection.py

@@ -29,10 +29,13 @@
 from __future__ import absolute_import
 from __future__ import unicode_literals
 
+import errno
 import logging
 import os
+import pprint
 import random
 import stat
+import sys
 import time
 
 import jinja2.runtime
@@ -41,6 +44,8 @@ import ansible.errors
 import ansible.plugins.connection
 import ansible.utils.shlex
 
+import mitogen.core
+import mitogen.fork
 import mitogen.unix
 import mitogen.utils
 
@@ -48,27 +53,42 @@ import ansible_mitogen.parsing
 import ansible_mitogen.process
 import ansible_mitogen.services
 import ansible_mitogen.target
+import ansible_mitogen.transport_config
 
 
 LOG = logging.getLogger(__name__)
 
 
-def optional_secret(value):
+def optional_int(value):
     """
-    Wrap `value` in :class:`mitogen.core.Secret` if it is not :data:`None`,
-    otherwise return :data:`None`.
+    Convert `value` to an integer if it is not :data:`None`, otherwise return
+    :data:`None`.
     """
-    if value is not None:
-        return mitogen.core.Secret(value)
+    try:
+        return int(value)
+    except (TypeError, ValueError):
+        return None
+
+
+def convert_bool(obj):
+    if isinstance(obj, bool):
+        return obj
+    if str(obj).lower() in ('no', 'false', '0'):
+        return False
+    if str(obj).lower() not in ('yes', 'true', '1'):
+        raise ansible.errors.AnsibleConnectionFailure(
+            'expected yes/no/true/false/0/1, got %r' % (obj,)
+        )
+    return True
 
 
-def parse_python_path(s):
+def default(value, default):
     """
-    Given the string set for ansible_python_interpeter, parse it using shell
-    syntax and return an appropriate argument vector.
+    Return `default` is `value` is :data:`None`, otherwise return `value`.
     """
-    if s:
-        return ansible.utils.shlex.shlex_split(s)
+    if value is None:
+        return default
+    return value
 
 
 def _connect_local(spec):
@@ -78,7 +98,7 @@ def _connect_local(spec):
     return {
         'method': 'local',
         'kwargs': {
-            'python_path': spec['python_path'],
+            'python_path': spec.python_path(),
         }
     }
 
@@ -92,21 +112,30 @@ def _connect_ssh(spec):
     else:
         check_host_keys = 'ignore'
 
+    # #334: tilde-expand private_key_file to avoid implementation difference
+    # between Python and OpenSSH.
+    private_key_file = spec.private_key_file()
+    if private_key_file is not None:
+        private_key_file = os.path.expanduser(private_key_file)
+
     return {
         'method': 'ssh',
         'kwargs': {
             'check_host_keys': check_host_keys,
-            'hostname': spec['remote_addr'],
-            'username': spec['remote_user'],
-            'password': optional_secret(spec['password']),
-            'port': spec['port'],
-            'python_path': spec['python_path'],
-            'identity_file': spec['private_key_file'],
+            'hostname': spec.remote_addr(),
+            'username': spec.remote_user(),
+            'compression': convert_bool(
+                default(spec.mitogen_ssh_compression(), True)
+            ),
+            'password': spec.password(),
+            'port': spec.port(),
+            'python_path': spec.python_path(),
+            'identity_file': private_key_file,
             'identities_only': False,
-            'ssh_path': spec['ssh_executable'],
-            'connect_timeout': spec['ansible_ssh_timeout'],
-            'ssh_args': spec['ssh_args'],
-            'ssh_debug_level': spec['mitogen_ssh_debug_level'],
+            'ssh_path': spec.ssh_executable(),
+            'connect_timeout': spec.ansible_ssh_timeout(),
+            'ssh_args': spec.ssh_args(),
+            'ssh_debug_level': spec.mitogen_ssh_debug_level(),
         }
     }
 
@@ -118,10 +147,10 @@ def _connect_docker(spec):
     return {
         'method': 'docker',
         'kwargs': {
-            'username': spec['remote_user'],
-            'container': spec['remote_addr'],
-            'python_path': spec['python_path'],
-            'connect_timeout': spec['ansible_ssh_timeout'] or spec['timeout'],
+            'username': spec.remote_user(),
+            'container': spec.remote_addr(),
+            'python_path': spec.python_path(),
+            'connect_timeout': spec.ansible_ssh_timeout() or spec.timeout(),
         }
     }
 
@@ -133,10 +162,11 @@ def _connect_kubectl(spec):
     return {
         'method': 'kubectl',
         'kwargs': {
-            'pod': spec['remote_addr'],
-            'python_path': spec['python_path'],
-            'connect_timeout': spec['ansible_ssh_timeout'] or spec['timeout'],
-            'kubectl_args': spec['extra_args'],
+            'pod': spec.remote_addr(),
+            'python_path': spec.python_path(),
+            'connect_timeout': spec.ansible_ssh_timeout() or spec.timeout(),
+            'kubectl_path': spec.mitogen_kubectl_path(),
+            'kubectl_args': spec.extra_args(),
         }
     }
 
@@ -148,10 +178,10 @@ def _connect_jail(spec):
     return {
         'method': 'jail',
         'kwargs': {
-            'username': spec['remote_user'],
-            'container': spec['remote_addr'],
-            'python_path': spec['python_path'],
-            'connect_timeout': spec['ansible_ssh_timeout'] or spec['timeout'],
+            'username': spec.remote_user(),
+            'container': spec.remote_addr(),
+            'python_path': spec.python_path(),
+            'connect_timeout': spec.ansible_ssh_timeout() or spec.timeout(),
         }
     }
 
@@ -163,9 +193,10 @@ def _connect_lxc(spec):
     return {
         'method': 'lxc',
         'kwargs': {
-            'container': spec['remote_addr'],
-            'python_path': spec['python_path'],
-            'connect_timeout': spec['ansible_ssh_timeout'] or spec['timeout'],
+            'container': spec.remote_addr(),
+            'python_path': spec.python_path(),
+            'lxc_attach_path': spec.mitogen_lxc_attach_path(),
+            'connect_timeout': spec.ansible_ssh_timeout() or spec.timeout(),
         }
     }
 
@@ -177,9 +208,10 @@ def _connect_lxd(spec):
     return {
         'method': 'lxd',
         'kwargs': {
-            'container': spec['remote_addr'],
-            'python_path': spec['python_path'],
-            'connect_timeout': spec['ansible_ssh_timeout'] or spec['timeout'],
+            'container': spec.remote_addr(),
+            'python_path': spec.python_path(),
+            'lxc_path': spec.mitogen_lxc_path(),
+            'connect_timeout': spec.ansible_ssh_timeout() or spec.timeout(),
         }
     }
 
@@ -188,24 +220,24 @@ def _connect_machinectl(spec):
     """
     Return ContextService arguments for a machinectl connection.
     """
-    return _connect_setns(dict(spec, mitogen_kind='machinectl'))
+    return _connect_setns(spec, kind='machinectl')
 
 
-def _connect_setns(spec):
+def _connect_setns(spec, kind=None):
     """
     Return ContextService arguments for a mitogen_setns connection.
     """
     return {
         'method': 'setns',
         'kwargs': {
-            'container': spec['remote_addr'],
-            'username': spec['remote_user'],
-            'python_path': spec['python_path'],
-            'kind': spec['mitogen_kind'],
-            'docker_path': spec['mitogen_docker_path'],
-            'kubectl_path': spec['mitogen_kubectl_path'],
-            'lxc_info_path': spec['mitogen_lxc_info_path'],
-            'machinectl_path': spec['mitogen_machinectl_path'],
+            'container': spec.remote_addr(),
+            'username': spec.remote_user(),
+            'python_path': spec.python_path(),
+            'kind': kind or spec.mitogen_kind(),
+            'docker_path': spec.mitogen_docker_path(),
+            'lxc_path': spec.mitogen_lxc_path(),
+            'lxc_info_path': spec.mitogen_lxc_info_path(),
+            'machinectl_path': spec.mitogen_machinectl_path(),
         }
     }
 
@@ -218,11 +250,11 @@ def _connect_su(spec):
         'method': 'su',
         'enable_lru': True,
         'kwargs': {
-            'username': spec['become_user'],
-            'password': optional_secret(spec['become_pass']),
-            'python_path': spec['python_path'],
-            'su_path': spec['become_exe'],
-            'connect_timeout': spec['timeout'],
+            'username': spec.become_user(),
+            'password': spec.become_pass(),
+            'python_path': spec.python_path(),
+            'su_path': spec.become_exe(),
+            'connect_timeout': spec.timeout(),
         }
     }
 
@@ -235,12 +267,12 @@ def _connect_sudo(spec):
         'method': 'sudo',
         'enable_lru': True,
         'kwargs': {
-            'username': spec['become_user'],
-            'password': optional_secret(spec['become_pass']),
-            'python_path': spec['python_path'],
-            'sudo_path': spec['become_exe'],
-            'connect_timeout': spec['timeout'],
-            'sudo_args': spec['sudo_args'],
+            'username': spec.become_user(),
+            'password': spec.become_pass(),
+            'python_path': spec.python_path(),
+            'sudo_path': spec.become_exe(),
+            'connect_timeout': spec.timeout(),
+            'sudo_args': spec.sudo_args(),
         }
     }
 
@@ -253,11 +285,11 @@ def _connect_doas(spec):
         'method': 'doas',
         'enable_lru': True,
         'kwargs': {
-            'username': spec['become_user'],
-            'password': optional_secret(spec['become_pass']),
-            'python_path': spec['python_path'],
-            'doas_path': spec['become_exe'],
-            'connect_timeout': spec['timeout'],
+            'username': spec.become_user(),
+            'password': spec.become_pass(),
+            'python_path': spec.python_path(),
+            'doas_path': spec.become_exe(),
+            'connect_timeout': spec.timeout(),
         }
     }
 
@@ -269,11 +301,11 @@ def _connect_mitogen_su(spec):
     return {
         'method': 'su',
         'kwargs': {
-            'username': spec['remote_user'],
-            'password': optional_secret(spec['password']),
-            'python_path': spec['python_path'],
-            'su_path': spec['become_exe'],
-            'connect_timeout': spec['timeout'],
+            'username': spec.remote_user(),
+            'password': spec.password(),
+            'python_path': spec.python_path(),
+            'su_path': spec.become_exe(),
+            'connect_timeout': spec.timeout(),
         }
     }
 
@@ -285,12 +317,12 @@ def _connect_mitogen_sudo(spec):
     return {
         'method': 'sudo',
         'kwargs': {
-            'username': spec['remote_user'],
-            'password': optional_secret(spec['password']),
-            'python_path': spec['python_path'],
-            'sudo_path': spec['become_exe'],
-            'connect_timeout': spec['timeout'],
-            'sudo_args': spec['sudo_args'],
+            'username': spec.remote_user(),
+            'password': spec.password(),
+            'python_path': spec.python_path(),
+            'sudo_path': spec.become_exe(),
+            'connect_timeout': spec.timeout(),
+            'sudo_args': spec.sudo_args(),
         }
     }
 
@@ -302,11 +334,11 @@ def _connect_mitogen_doas(spec):
     return {
         'method': 'doas',
         'kwargs': {
-            'username': spec['remote_user'],
-            'password': optional_secret(spec['password']),
-            'python_path': spec['python_path'],
-            'doas_path': spec['become_exe'],
-            'connect_timeout': spec['timeout'],
+            'username': spec.remote_user(),
+            'password': spec.password(),
+            'python_path': spec.python_path(),
+            'doas_path': spec.become_exe(),
+            'connect_timeout': spec.timeout(),
         }
     }
 
@@ -333,110 +365,40 @@ CONNECTION_METHOD = {
 }
 
 
-def config_from_play_context(transport, inventory_name, connection):
+class Broker(mitogen.master.Broker):
     """
-    Return a dict representing all important connection configuration, allowing
-    the same functions to work regardless of whether configuration came from
-    play_context (direct connection) or host vars (mitogen_via=).
+    WorkerProcess maintains at most 2 file descriptors, therefore does not need
+    the exuberant syscall expense of EpollPoller, so override it and restore
+    the poll() poller.
     """
-    return {
-        'transport': transport,
-        'inventory_name': inventory_name,
-        'remote_addr': connection._play_context.remote_addr,
-        'remote_user': connection._play_context.remote_user,
-        'become': connection._play_context.become,
-        'become_method': connection._play_context.become_method,
-        'become_user': connection._play_context.become_user,
-        'become_pass': connection._play_context.become_pass,
-        'password': connection._play_context.password,
-        'port': connection._play_context.port,
-        'python_path': parse_python_path(
-            connection.get_task_var('ansible_python_interpreter',
-                                    default='/usr/bin/python')
-        ),
-        'private_key_file': connection._play_context.private_key_file,
-        'ssh_executable': connection._play_context.ssh_executable,
-        'timeout': connection._play_context.timeout,
-        'ansible_ssh_timeout':
-            connection.get_task_var('ansible_ssh_timeout',
-                                    default=C.DEFAULT_TIMEOUT),
-        'ssh_args': [
-            mitogen.core.to_text(term)
-            for s in (
-                getattr(connection._play_context, 'ssh_args', ''),
-                getattr(connection._play_context, 'ssh_common_args', ''),
-                getattr(connection._play_context, 'ssh_extra_args', '')
-            )
-            for term in ansible.utils.shlex.shlex_split(s or '')
-        ],
-        'become_exe': connection._play_context.become_exe,
-        'sudo_args': [
-            mitogen.core.to_text(term)
-            for s in (
-                connection._play_context.sudo_flags,
-                connection._play_context.become_flags
-            )
-            for term in ansible.utils.shlex.shlex_split(s or '')
-        ],
-        'mitogen_via':
-            connection.get_task_var('mitogen_via'),
-        'mitogen_kind':
-            connection.get_task_var('mitogen_kind'),
-        'mitogen_docker_path':
-            connection.get_task_var('mitogen_docker_path'),
-        'mitogen_kubectl_path':
-            connection.get_task_var('mitogen_kubectl_path'),
-        'mitogen_lxc_info_path':
-            connection.get_task_var('mitogen_lxc_info_path'),
-        'mitogen_machinectl_path':
-            connection.get_task_var('mitogen_machinectl_path'),
-        'mitogen_ssh_debug_level':
-            connection.get_task_var('mitogen_ssh_debug_level'),
-        'extra_args':
-            connection.get_extra_args(),
-    }
-
-
-def config_from_hostvars(transport, inventory_name, connection,
-                         hostvars, become_user):
-    """
-    Override config_from_play_context() to take equivalent information from
-    host vars.
-    """
-    config = config_from_play_context(transport, inventory_name, connection)
-    hostvars = dict(hostvars)
-    return dict(config, **{
-        'remote_addr': hostvars.get('ansible_host', inventory_name),
-        'become': bool(become_user),
-        'become_user': become_user,
-        'become_pass': None,
-        'remote_user': hostvars.get('ansible_user'),  # TODO
-        'password': (hostvars.get('ansible_ssh_pass') or
-                     hostvars.get('ansible_password')),
-        'port': hostvars.get('ansible_port'),
-        'python_path': parse_python_path(hostvars.get('ansible_python_interpreter')),
-        'private_key_file': (hostvars.get('ansible_ssh_private_key_file') or
-                             hostvars.get('ansible_private_key_file')),
-        'mitogen_via': hostvars.get('mitogen_via'),
-        'mitogen_kind': hostvars.get('mitogen_kind'),
-        'mitogen_docker_path': hostvars.get('mitogen_docker_path'),
-        'mitogen_kubectl_path': hostvars.get('mitogen_kubectl_path'),
-        'mitogen_lxc_info_path': hostvars.get('mitogen_lxc_info_path'),
-        'mitogen_machinectl_path': hostvars.get('mitogen_machinctl_path'),
-    })
+    poller_class = mitogen.core.Poller
 
 
 class CallChain(mitogen.parent.CallChain):
+    """
+    Extend :class:`mitogen.parent.CallChain` to additionally cause the
+    associated :class:`Connection` to be reset if a ChannelError occurs.
+
+    This only catches failures that occur while a call is pending, it is a
+    stop-gap until a more general method is available to notice connection in
+    every situation.
+    """
     call_aborted_msg = (
         'Mitogen was disconnected from the remote environment while a call '
         'was in-progress. If you feel this is in error, please file a bug. '
         'Original error was: %s'
     )
 
+    def __init__(self, connection, context, pipelined=False):
+        super(CallChain, self).__init__(context, pipelined)
+        #: The connection to reset on CallError.
+        self._connection = connection
+
     def _rethrow(self, recv):
         try:
             return recv.get().unpickle()
         except mitogen.core.ChannelError as e:
+            self._connection.reset()
             raise ansible.errors.AnsibleConnectionFailure(
                 self.call_aborted_msg % (e,)
             )
@@ -550,11 +512,30 @@ class Connection(ansible.plugins.connection.ConnectionBase):
         self.host_vars = task_vars['hostvars']
         self.delegate_to_hostname = delegate_to_hostname
         self.loader_basedir = loader_basedir
-        self.close(new_task=True)
+        self._mitogen_reset(mode='put')
 
     def get_task_var(self, key, default=None):
-        if self._task_vars and key in self._task_vars:
-            return self._task_vars[key]
+        """
+        Fetch the value of a task variable related to connection configuration,
+        or, if delegate_to is active, fetch the same variable via HostVars for
+        the delegated-to machine.
+
+        When running with delegate_to, Ansible tasks have variables associated
+        with the original machine, not the delegated-to machine, therefore it
+        does not make sense to extract connection-related configuration for the
+        delegated-to machine from them.
+        """
+        if self._task_vars:
+            if self.delegate_to_hostname is None:
+                if key in self._task_vars:
+                    return self._task_vars[key]
+            else:
+                delegated_vars = self._task_vars['ansible_delegated_vars']
+                if self.delegate_to_hostname in delegated_vars:
+                    task_vars = delegated_vars[self.delegate_to_hostname]
+                    if key in task_vars:
+                        return task_vars[key]
+
         return default
 
     @property
@@ -566,12 +547,14 @@ class Connection(ansible.plugins.connection.ConnectionBase):
     def connected(self):
         return self.context is not None
 
-    def _config_from_via(self, via_spec):
+    def _spec_from_via(self, via_spec):
         """
         Produce a dict connection specifiction given a string `via_spec`, of
-        the form `[become_user@]inventory_hostname`.
+        the form `[[become_method:]become_user@]inventory_hostname`.
         """
         become_user, _, inventory_name = via_spec.rpartition('@')
+        become_method, _, become_user = become_user.rpartition(':')
+
         via_vars = self.host_vars[inventory_name]
         if isinstance(via_vars, jinja2.runtime.Undefined):
             raise ansible.errors.AnsibleConnectionFailure(
@@ -581,41 +564,68 @@ class Connection(ansible.plugins.connection.ConnectionBase):
                 )
             )
 
-        return config_from_hostvars(
-            transport=via_vars.get('ansible_connection', 'ssh'),
+        return ansible_mitogen.transport_config.MitogenViaSpec(
             inventory_name=inventory_name,
-            connection=self,
-            hostvars=via_vars,
+            host_vars=dict(via_vars),  # TODO: make it lazy
+            become_method=become_method or None,
             become_user=become_user or None,
         )
 
     unknown_via_msg = 'mitogen_via=%s of %s specifies an unknown hostname'
     via_cycle_msg = 'mitogen_via=%s of %s creates a cycle (%s)'
 
-    def _stack_from_config(self, config, stack=(), seen_names=()):
-        if config['inventory_name'] in seen_names:
+    def _stack_from_spec(self, spec, stack=(), seen_names=()):
+        """
+        Return a tuple of ContextService parameter dictionaries corresponding
+        to the connection described by `spec`, and any connection referenced by
+        its `mitogen_via` or `become` fields. Each element is a dict of the
+        form::
+
+            {
+                # Optional. If present and `True`, this hop is elegible for
+                # interpreter recycling.
+                "enable_lru": True,
+                # mitogen.master.Router method name.
+                "method": "ssh",
+                # mitogen.master.Router method kwargs.
+                "kwargs": {
+                    "hostname": "..."
+                }
+            }
+
+        :param ansible_mitogen.transport_config.Spec spec:
+            Connection specification.
+        :param tuple stack:
+            Stack elements from parent call (used for recursion).
+        :param tuple seen_names:
+            Inventory hostnames from parent call (cycle detection).
+        :returns:
+            Tuple `(stack, seen_names)`.
+        """
+        if spec.inventory_name() in seen_names:
             raise ansible.errors.AnsibleConnectionFailure(
                 self.via_cycle_msg % (
-                    config['mitogen_via'],
-                    config['inventory_name'],
+                    spec.mitogen_via(),
+                    spec.inventory_name(),
                     ' -> '.join(reversed(
-                        seen_names + (config['inventory_name'],)
+                        seen_names + (spec.inventory_name(),)
                     )),
                 )
             )
 
-        if config['mitogen_via']:
-            stack, seen_names = self._stack_from_config(
-                self._config_from_via(config['mitogen_via']),
+        if spec.mitogen_via():
+            stack = self._stack_from_spec(
+                self._spec_from_via(spec.mitogen_via()),
                 stack=stack,
-                seen_names=seen_names + (config['inventory_name'],)
+                seen_names=seen_names + (spec.inventory_name(),),
             )
 
-        stack += (CONNECTION_METHOD[config['transport']](config),)
-        if config['become']:
-            stack += (CONNECTION_METHOD[config['become_method']](config),)
+        stack += (CONNECTION_METHOD[spec.transport()](spec),)
+        if spec.become() and ((spec.become_user() != spec.remote_user()) or
+                              C.BECOME_ALLOW_SAME_USER):
+            stack += (CONNECTION_METHOD[spec.become_method()](spec),)
 
-        return stack, seen_names
+        return stack
 
     def _connect_broker(self):
         """
@@ -629,26 +639,6 @@ class Connection(ansible.plugins.connection.ConnectionBase):
                 broker=self.broker,
             )
 
-    def _config_from_direct_connection(self):
-        """
-        """
-        return config_from_play_context(
-            transport=self.transport,
-            inventory_name=self.inventory_hostname,
-            connection=self
-        )
-
-    def _config_from_delegate_to(self):
-        return config_from_hostvars(
-            transport=self._play_context.connection,
-            inventory_name=self.delegate_to_hostname,
-            connection=self,
-            hostvars=self.host_vars[self.delegate_to_hostname],
-            become_user=(self._play_context.become_user
-                         if self._play_context.become
-                         else None),
-        )
-
     def _build_stack(self):
         """
         Construct a list of dictionaries representing the connection
@@ -656,25 +646,35 @@ class Connection(ansible.plugins.connection.ConnectionBase):
         additionally used by the integration tests "mitogen_get_stack" action
         to fetch the would-be connection configuration.
         """
-        if self.delegate_to_hostname is not None:
-            target_config = self._config_from_delegate_to()
-        else:
-            target_config = self._config_from_direct_connection()
-
-        stack, _ = self._stack_from_config(target_config)
-        return stack
+        return self._stack_from_spec(
+            ansible_mitogen.transport_config.PlayContextSpec(
+                connection=self,
+                play_context=self._play_context,
+                transport=self.transport,
+                inventory_name=self.inventory_hostname,
+            )
+        )
 
     def _connect_stack(self, stack):
         """
         Pass `stack` to ContextService, requesting a copy of the context object
-        representing the target. If no connection exists yet, ContextService
-        will establish it before returning it or throwing an error.
+        representing the last tuple element. If no connection exists yet,
+        ContextService will recursively establish it before returning it or
+        throwing an error.
+
+        See :meth:`ansible_mitogen.services.ContextService.get` docstring for
+        description of the returned dictionary.
         """
-        dct = self.parent.call_service(
-            service_name='ansible_mitogen.services.ContextService',
-            method_name='get',
-            stack=mitogen.utils.cast(list(stack)),
-        )
+        try:
+            dct = self.parent.call_service(
+                service_name='ansible_mitogen.services.ContextService',
+                method_name='get',
+                stack=mitogen.utils.cast(list(stack)),
+            )
+        except mitogen.core.CallError:
+            LOG.warning('Connection failed; stack configuration was:\n%s',
+                        pprint.pformat(stack))
+            raise
 
         if dct['msg']:
             if dct['method_name'] in self.become_methods:
@@ -682,7 +682,7 @@ class Connection(ansible.plugins.connection.ConnectionBase):
             raise ansible.errors.AnsibleConnectionFailure(dct['msg'])
 
         self.context = dct['context']
-        self.chain = CallChain(self.context, pipelined=True)
+        self.chain = CallChain(self, self.context, pipelined=True)
         if self._play_context.become:
             self.login_context = dct['via']
         else:
@@ -691,6 +691,11 @@ class Connection(ansible.plugins.connection.ConnectionBase):
         self.init_child_result = dct['init_child_result']
 
     def get_good_temp_dir(self):
+        """
+        Return the 'good temporary directory' as discovered by
+        :func:`ansible_mitogen.target.init_child` immediately after
+        ContextService constructed the target context.
+        """
         self._connect()
         return self.init_child_result['good_temp_dir']
 
@@ -709,6 +714,20 @@ class Connection(ansible.plugins.connection.ConnectionBase):
         self.get_chain().call_no_reply(os.mkdir, self._shell.tmpdir)
         return self._shell.tmpdir
 
+    def _reset_tmp_path(self):
+        """
+        Called by _mitogen_reset(); ask the remote context to delete any
+        temporary directory created for the action. CallChain is not used here
+        to ensure exception is logged by the context on failure, since the
+        CallChain itself is about to be destructed.
+        """
+        if getattr(self._shell, 'tmpdir', None) is not None:
+            self.context.call_no_reply(
+                ansible_mitogen.target.prune_tree,
+                self._shell.tmpdir,
+            )
+            self._shell.tmpdir = None
+
     def _connect(self):
         """
         Establish a connection to the master process's UNIX listener socket,
@@ -727,38 +746,103 @@ class Connection(ansible.plugins.connection.ConnectionBase):
         stack = self._build_stack()
         self._connect_stack(stack)
 
-    def close(self, new_task=False):
+    def _mitogen_reset(self, mode):
         """
-        Arrange for the mitogen.master.Router running in the worker to
-        gracefully shut down, and wait for shutdown to complete. Safe to call
-        multiple times.
+        Forget everything we know about the connected context. This function
+        cannot be called _reset() since that name is used as a public API by
+        Ansible 2.4 wait_for_connection plug-in.
+
+        :param str mode:
+            Name of ContextService method to use to discard the context, either
+            'put' or 'reset'.
         """
-        if getattr(self._shell, 'tmpdir', None) is not None:
-            # Avoid CallChain to ensure exception is logged on failure.
-            self.context.call_no_reply(
-                ansible_mitogen.target.prune_tree,
-                self._shell.tmpdir,
-            )
-            self._shell.tmpdir = None
+        if not self.context:
+            return
 
-        if self.context:
-            self.chain.reset()
-            self.parent.call_service(
-                service_name='ansible_mitogen.services.ContextService',
-                method_name='put',
-                context=self.context
-            )
+        self._reset_tmp_path()
+        self.chain.reset()
+        self.parent.call_service(
+            service_name='ansible_mitogen.services.ContextService',
+            method_name=mode,
+            context=self.context
+        )
 
         self.context = None
         self.login_context = None
         self.init_child_result = None
         self.chain = None
-        if self.broker and not new_task:
+
+    def _shutdown_broker(self):
+        """
+        Shutdown the broker thread during :meth:`close` or :meth:`reset`.
+        """
+        if self.broker:
             self.broker.shutdown()
             self.broker.join()
             self.broker = None
             self.router = None
 
+            # #420: Ansible executes "meta" actions in the top-level process,
+            # meaning "reset_connection" will cause :class:`mitogen.core.Latch`
+            # FDs to be cached and erroneously shared by children on subsequent
+            # WorkerProcess forks. To handle that, call on_fork() to ensure any
+            # shared state is discarded.
+            # #490: only attempt to clean up when it's known that some
+            # resources exist to cleanup, otherwise later __del__ double-call
+            # to close() due to GC at random moment may obliterate an unrelated
+            # Connection's resources.
+            mitogen.fork.on_fork()
+
+    def close(self):
+        """
+        Arrange for the mitogen.master.Router running in the worker to
+        gracefully shut down, and wait for shutdown to complete. Safe to call
+        multiple times.
+        """
+        self._mitogen_reset(mode='put')
+        self._shutdown_broker()
+
+    def _reset_find_task_vars(self):
+        """
+        Monsterous hack: since "meta: reset_connection" does not run from an
+        action, we cannot capture task variables via :meth:`on_action_run`.
+        Instead walk the parent frames searching for the `all_vars` local from
+        StrategyBase._execute_meta(). If this fails, just leave task_vars
+        unset, likely causing a subtly wrong configuration to be selected.
+        """
+        frame = sys._getframe()
+        while frame and not self._task_vars:
+            self._task_vars = frame.f_locals.get('all_vars')
+            frame = frame.f_back
+
+    reset_compat_msg = (
+        'Mitogen only supports "reset_connection" on Ansible 2.5.6 or later'
+    )
+
+    def reset(self):
+        """
+        Explicitly terminate the connection to the remote host. This discards
+        any local state we hold for the connection, returns the Connection to
+        the 'disconnected' state, and informs ContextService the connection is
+        bad somehow, and should be shut down and discarded.
+        """
+        if self._task_vars is None:
+            self._reset_find_task_vars()
+
+        if self._play_context.remote_addr is None:
+            # <2.5.6 incorrectly populate PlayContext for reset_connection
+            # https://github.com/ansible/ansible/issues/27520
+            raise ansible.errors.AnsibleConnectionFailure(
+                self.reset_compat_msg
+            )
+
+        self._connect()
+        self._mitogen_reset(mode='reset')
+        self._shutdown_broker()
+
+    # Compatibility with Ansible 2.4 wait_for_connection plug-in.
+    _reset = reset
+
     def get_chain(self, use_login=False, use_fork=False):
         """
         Return the :class:`mitogen.parent.CallChain` to use for executing
@@ -774,21 +858,20 @@ class Connection(ansible.plugins.connection.ConnectionBase):
         self._connect()
         if use_login:
             return self.login_context.default_call_chain
-        if use_fork:
+        # See FORK_SUPPORTED comments in target.py.
+        if use_fork and self.init_child_result['fork_context'] is not None:
             return self.init_child_result['fork_context'].default_call_chain
         return self.chain
 
-    def create_fork_child(self):
+    def spawn_isolated_child(self):
         """
-        Fork a new child off the target context. The actual fork occurs from
-        the 'virginal fork parent', which does not any Ansible modules prior to
-        fork, to avoid conflicts resulting from custom module_utils paths.
+        Fork or launch a new child off the target context.
 
         :returns:
             mitogen.core.Context of the new child.
         """
         return self.get_chain(use_fork=True).call(
-            ansible_mitogen.target.create_fork_child
+            ansible_mitogen.target.spawn_isolated_child
         )
 
     def get_extra_args(self):
@@ -834,9 +917,9 @@ class Connection(ansible.plugins.connection.ConnectionBase):
             emulate_tty=emulate_tty,
         )
 
-        stderr += 'Shared connection to %s closed.%s' % (
-            self._play_context.remote_addr,
-            ('\r\n' if emulate_tty else '\n'),
+        stderr += b'Shared connection to %s closed.%s' % (
+            self._play_context.remote_addr.encode(),
+            (b'\r\n' if emulate_tty else b'\n'),
         )
         return rc, stdout, stderr
 
@@ -882,6 +965,11 @@ class Connection(ansible.plugins.connection.ConnectionBase):
     #: slightly more overhead, so just randomly subtract 4KiB.
     SMALL_FILE_LIMIT = mitogen.core.CHUNK_SIZE - 4096
 
+    def _throw_io_error(self, e, path):
+        if e.args[0] == errno.ENOENT:
+            s = 'file or module does not exist: ' + path
+            raise ansible.errors.AnsibleFileNotFound(s)
+
     def put_file(self, in_path, out_path):
         """
         Implement put_file() by streamily transferring the file via
@@ -892,7 +980,12 @@ class Connection(ansible.plugins.connection.ConnectionBase):
         :param str out_path:
             Remote filesystem path to write.
         """
-        st = os.stat(in_path)
+        try:
+            st = os.stat(in_path)
+        except OSError as e:
+            self._throw_io_error(e, in_path)
+            raise
+
         if not stat.S_ISREG(st.st_mode):
             raise IOError('%r is not a regular file.' % (in_path,))
 
@@ -900,17 +993,22 @@ class Connection(ansible.plugins.connection.ConnectionBase):
         # rather than introducing an extra RTT for the child to request it from
         # FileService.
         if st.st_size <= self.SMALL_FILE_LIMIT:
-            fp = open(in_path, 'rb')
             try:
-                s = fp.read(self.SMALL_FILE_LIMIT + 1)
-            finally:
-                fp.close()
+                fp = open(in_path, 'rb')
+                try:
+                    s = fp.read(self.SMALL_FILE_LIMIT + 1)
+                finally:
+                    fp.close()
+            except OSError:
+                self._throw_io_error(e, in_path)
+                raise
 
             # Ensure did not grow during read.
             if len(s) == st.st_size:
                 return self.put_data(out_path, s, mode=st.st_mode,
                                      utimes=(st.st_atime, st.st_mtime))
 
+        self._connect()
         self.parent.call_service(
             service_name='mitogen.service.FileService',
             method_name='register',

ansible_mitogen/logging.py

@@ -29,7 +29,6 @@
 from __future__ import absolute_import
 import logging
 import os
-import sys
 
 import mitogen.core
 import mitogen.utils

ansible_mitogen/mixins.py

@@ -30,7 +30,6 @@ from __future__ import absolute_import
 import logging
 import os
 import pwd
-import shutil
 import traceback
 
 try:
@@ -156,7 +155,7 @@ class ActionModuleMixin(ansible.plugins.action.ActionBase):
         """
         LOG.debug('_remote_file_exists(%r)', path)
         return self._connection.get_chain().call(
-            os.path.exists,
+            ansible_mitogen.target.file_exists,
             mitogen.utils.cast(path)
         )
 
@@ -223,7 +222,7 @@ class ActionModuleMixin(ansible.plugins.action.ActionBase):
         """
         LOG.debug('_fixup_perms2(%r, remote_user=%r, execute=%r)',
                   remote_paths, remote_user, execute)
-        if execute and self._load_name not in self.FIXUP_PERMS_RED_HERRING:
+        if execute and self._task.action not in self.FIXUP_PERMS_RED_HERRING:
             return self._remote_chmod(remote_paths, mode='u+x')
         return self.COMMAND_RESULT.copy()
 

ansible_mitogen/planner.py

@@ -46,6 +46,7 @@ from ansible.executor import module_common
 import ansible.errors
 import ansible.module_utils
 import mitogen.core
+import mitogen.select
 
 import ansible_mitogen.loaders
 import ansible_mitogen.parsing
@@ -172,7 +173,7 @@ class BinaryPlanner(Planner):
         return module_common._is_binary(self._inv.module_source)
 
     def get_push_files(self):
-        return [self._inv.module_path]
+        return [mitogen.core.to_text(self._inv.module_path)]
 
     def get_kwargs(self, **kwargs):
         return super(BinaryPlanner, self).get_kwargs(
@@ -205,15 +206,13 @@ class ScriptPlanner(BinaryPlanner):
             involved here, the vanilla implementation uses it and that use is
             exploited in common playbooks.
         """
+        key = u'ansible_%s_interpreter' % os.path.basename(path).strip()
         try:
-            key = u'ansible_%s_interpreter' % os.path.basename(path).strip()
             template = self._inv.task_vars[key]
         except KeyError:
             return path
 
-        return mitogen.utils.cast(
-            self._inv.templar.template(self._inv.task_vars[key])
-        )
+        return mitogen.utils.cast(self._inv.templar.template(template))
 
     def _get_interpreter(self):
         path, arg = ansible_mitogen.parsing.parse_hashbang(
@@ -285,7 +284,7 @@ class NewStylePlanner(ScriptPlanner):
 
     def get_push_files(self):
         return super(NewStylePlanner, self).get_push_files() + [
-            path
+            mitogen.core.to_text(path)
             for fullname, path, is_pkg in self.get_module_map()['custom']
         ]
 
@@ -416,28 +415,39 @@ def _propagate_deps(invocation, planner, context):
 
 def _invoke_async_task(invocation, planner):
     job_id = '%016x' % random.randint(0, 2**64)
-    context = invocation.connection.create_fork_child()
+    context = invocation.connection.spawn_isolated_child()
     _propagate_deps(invocation, planner, context)
-    context.call_no_reply(
-        ansible_mitogen.target.run_module_async,
-        job_id=job_id,
-        timeout_secs=invocation.timeout_secs,
-        kwargs=planner.get_kwargs(),
-    )
-
-    return {
-        'stdout': json.dumps({
-            # modules/utilities/logic/async_wrapper.py::_run_module().
-            'changed': True,
-            'started': 1,
-            'finished': 0,
-            'ansible_job_id': job_id,
-        })
-    }
 
+    with mitogen.core.Receiver(context.router) as started_recv:
+        call_recv = context.call_async(
+            ansible_mitogen.target.run_module_async,
+            job_id=job_id,
+            timeout_secs=invocation.timeout_secs,
+            started_sender=started_recv.to_sender(),
+            kwargs=planner.get_kwargs(),
+        )
 
-def _invoke_forked_task(invocation, planner):
-    context = invocation.connection.create_fork_child()
+        # Wait for run_module_async() to crash, or for AsyncRunner to indicate
+        # the job file has been written.
+        for msg in mitogen.select.Select([started_recv, call_recv]):
+            if msg.receiver is call_recv:
+                # It can only be an exception.
+                raise msg.unpickle()
+            break
+
+        return {
+            'stdout': json.dumps({
+                # modules/utilities/logic/async_wrapper.py::_run_module().
+                'changed': True,
+                'started': 1,
+                'finished': 0,
+                'ansible_job_id': job_id,
+            })
+        }
+
+
+def _invoke_isolated_task(invocation, planner):
+    context = invocation.connection.spawn_isolated_child()
     _propagate_deps(invocation, planner, context)
     try:
         return context.call(
@@ -477,7 +487,7 @@ def invoke(invocation):
     if invocation.wrap_async:
         response = _invoke_async_task(invocation, planner)
     elif planner.should_fork():
-        response = _invoke_forked_task(invocation, planner)
+        response = _invoke_isolated_task(invocation, planner)
     else:
         _propagate_deps(invocation, planner, invocation.connection.context)
         response = invocation.connection.get_chain().call(

ansible_mitogen/plugins/action/mitogen_get_stack.py

@@ -0,0 +1,54 @@
+# Copyright 2017, David Wilson
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are met:
+#
+# 1. Redistributions of source code must retain the above copyright notice,
+# this list of conditions and the following disclaimer.
+#
+# 2. Redistributions in binary form must reproduce the above copyright notice,
+# this list of conditions and the following disclaimer in the documentation
+# and/or other materials provided with the distribution.
+#
+# 3. Neither the name of the copyright holder nor the names of its contributors
+# may be used to endorse or promote products derived from this software without
+# specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+# POSSIBILITY OF SUCH DAMAGE.
+
+from __future__ import absolute_import
+from __future__ import unicode_literals
+
+"""
+Fetch the connection configuration stack that would be used to connect to a
+target, without actually connecting to it.
+"""
+
+import ansible_mitogen.connection
+
+from ansible.plugins.action import ActionBase
+
+
+class ActionModule(ActionBase):
+    def run(self, tmp=None, task_vars=None):
+        if not isinstance(self._connection,
+                          ansible_mitogen.connection.Connection):
+            return {
+                'skipped': True,
+            }
+
+        return {
+            'changed': True,
+            'result': self._connection._build_stack(),
+            '_ansible_verbose_always': True,
+        }

ansible_mitogen/plugins/connection/mitogen_docker.py

@@ -42,3 +42,10 @@ import ansible_mitogen.connection
 
 class Connection(ansible_mitogen.connection.Connection):
     transport = 'docker'
+
+    @property
+    def docker_cmd(self):
+        """
+        Ansible 2.3 synchronize module wants to know how we run Docker.
+        """
+        return 'docker'

ansible_mitogen/plugins/connection/mitogen_kubectl.py

@@ -31,7 +31,12 @@ from __future__ import absolute_import
 import os.path
 import sys
 
-import ansible.plugins.connection.kubectl
+try:
+    from ansible.plugins.connection import kubectl
+except ImportError:
+    kubectl = None
+
+from ansible.errors import AnsibleConnectionFailure
 from ansible.module_utils.six import iteritems
 
 try:
@@ -47,9 +52,19 @@ import ansible_mitogen.connection
 class Connection(ansible_mitogen.connection.Connection):
     transport = 'kubectl'
 
+    not_supported_msg = (
+        'The "mitogen_kubectl" plug-in requires a version of Ansible '
+        'that ships with the "kubectl" connection plug-in.'
+    )
+
+    def __init__(self, *args, **kwargs):
+        if kubectl is None:
+            raise AnsibleConnectionFailure(self.not_supported_msg)
+        super(Connection, self).__init__(*args, **kwargs)
+
     def get_extra_args(self):
         parameters = []
-        for key, option in iteritems(ansible.plugins.connection.kubectl.CONNECTION_OPTIONS):
+        for key, option in iteritems(kubectl.CONNECTION_OPTIONS):
             if self.get_task_var('ansible_' + key) is not None:
                 parameters += [ option, self.get_task_var('ansible_' + key) ]
 

ansible_mitogen/plugins/strategy/mitogen_host_pinned.py

@@ -0,0 +1,67 @@
+# Copyright 2017, David Wilson
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are met:
+#
+# 1. Redistributions of source code must retain the above copyright notice,
+# this list of conditions and the following disclaimer.
+#
+# 2. Redistributions in binary form must reproduce the above copyright notice,
+# this list of conditions and the following disclaimer in the documentation
+# and/or other materials provided with the distribution.
+#
+# 3. Neither the name of the copyright holder nor the names of its contributors
+# may be used to endorse or promote products derived from this software without
+# specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+# POSSIBILITY OF SUCH DAMAGE.
+
+from __future__ import absolute_import
+import os.path
+import sys
+
+#
+# This is not the real Strategy implementation module, it simply exists as a
+# proxy to the real module, which is loaded using Python's regular import
+# mechanism, to prevent Ansible's PluginLoader from making up a fake name that
+# results in ansible_mitogen plugin modules being loaded twice: once by
+# PluginLoader with a name like "ansible.plugins.strategy.mitogen", which is
+# stuffed into sys.modules even though attempting to import it will trigger an
+# ImportError, and once under its canonical name, "ansible_mitogen.strategy".
+#
+# Therefore we have a proxy module that imports it under the real name, and
+# sets up the duff PluginLoader-imported module to just contain objects from
+# the real module, so duplicate types don't exist in memory, and things like
+# debuggers and isinstance() work predictably.
+#
+
+BASE_DIR = os.path.abspath(
+    os.path.join(os.path.dirname(__file__), '../../..')
+)
+
+if BASE_DIR not in sys.path:
+    sys.path.insert(0, BASE_DIR)
+
+import ansible_mitogen.loaders
+import ansible_mitogen.strategy
+
+
+Base = ansible_mitogen.loaders.strategy_loader.get('host_pinned', class_only=True)
+
+if Base is None:
+    raise ImportError(
+        'The host_pinned strategy is only available in Ansible 2.7 or newer.'
+    )
+
+class StrategyModule(ansible_mitogen.strategy.StrategyMixin, Base):
+    pass

ansible_mitogen/process.py

@@ -50,15 +50,22 @@ import mitogen.service
 import mitogen.unix
 import mitogen.utils
 
+import ansible
 import ansible.constants as C
 import ansible_mitogen.logging
 import ansible_mitogen.services
 
 from mitogen.core import b
+import ansible_mitogen.affinity
 
 
 LOG = logging.getLogger(__name__)
 
+ANSIBLE_PKG_OVERRIDE = (
+    u"__version__ = %r\n"
+    u"__author__ = %r\n"
+)
+
 
 def clean_shutdown(sock):
     """
@@ -87,25 +94,22 @@ def getenv_int(key, default=0):
         return default
 
 
-def setup_gil():
-    """
-    Set extremely long GIL release interval to let threads naturally progress
-    through CPU-heavy sequences without forcing the wake of another thread that
-    may contend trying to run the same CPU-heavy code. For the new-style work,
-    this drops runtime ~33% and involuntary context switches by >80%,
-    essentially making threads cooperatively scheduled.
+def save_pid(name):
     """
-    try:
-        # Python 2.
-        sys.setcheckinterval(100000)
-    except AttributeError:
-        pass
+    When debugging and profiling, it is very annoying to poke through the
+    process list to discover the currently running Ansible and MuxProcess IDs,
+    especially when trying to catch an issue during early startup. So here, if
+    a magic environment variable set, stash them in hidden files in the CWD::
 
-    try:
-        # Python 3.
-        sys.setswitchinterval(10)
-    except AttributeError:
-        pass
+        alias muxpid="cat .ansible-mux.pid"
+        alias anspid="cat .ansible-controller.pid"
+
+        gdb -p $(muxpid)
+        perf top -p $(anspid)
+    """
+    if os.environ.get('MITOGEN_SAVE_PIDS'):
+        with open('.ansible-%s.pid' % (name,), 'w') as fp:
+            fp.write(str(os.getpid()))
 
 
 class MuxProcess(object):
@@ -154,13 +158,16 @@ class MuxProcess(object):
     _instance = None
 
     @classmethod
-    def start(cls):
+    def start(cls, _init_logging=True):
         """
         Arrange for the subprocess to be started, if it is not already running.
 
         The parent process picks a UNIX socket path the child will use prior to
         fork, creates a socketpair used essentially as a semaphore, then blocks
         waiting for the child to indicate the UNIX socket is ready for use.
+
+        :param bool _init_logging:
+            For testing, if :data:`False`, don't initialize logging.
         """
         if cls.worker_sock is not None:
             return
@@ -168,29 +175,34 @@ class MuxProcess(object):
         if faulthandler is not None:
             faulthandler.enable()
 
-        setup_gil()
+        mitogen.utils.setup_gil()
         cls.unix_listener_path = mitogen.unix.make_socket_path()
         cls.worker_sock, cls.child_sock = socket.socketpair()
         atexit.register(lambda: clean_shutdown(cls.worker_sock))
         mitogen.core.set_cloexec(cls.worker_sock.fileno())
         mitogen.core.set_cloexec(cls.child_sock.fileno())
 
-        if os.environ.get('MITOGEN_PROFILING'):
+        cls.profiling = os.environ.get('MITOGEN_PROFILING') is not None
+        if cls.profiling:
             mitogen.core.enable_profiling()
 
         cls.original_env = dict(os.environ)
         cls.child_pid = os.fork()
-        ansible_mitogen.logging.setup()
+        if _init_logging:
+            ansible_mitogen.logging.setup()
         if cls.child_pid:
+            save_pid('controller')
+            ansible_mitogen.affinity.policy.assign_controller()
             cls.child_sock.close()
             cls.child_sock = None
             mitogen.core.io_op(cls.worker_sock.recv, 1)
         else:
+            save_pid('mux')
+            ansible_mitogen.affinity.policy.assign_muxprocess()
             cls.worker_sock.close()
             cls.worker_sock = None
             self = cls()
             self.worker_main()
-            sys.exit()
 
     def worker_main(self):
         """
@@ -201,10 +213,19 @@ class MuxProcess(object):
         self._setup_master()
         self._setup_services()
 
-        # Let the parent know our listening socket is ready.
-        mitogen.core.io_op(self.child_sock.send, b('1'))
-        # Block until the socket is closed, which happens on parent exit.
-        mitogen.core.io_op(self.child_sock.recv, 1)
+        try:
+            # Let the parent know our listening socket is ready.
+            mitogen.core.io_op(self.child_sock.send, b('1'))
+            # Block until the socket is closed, which happens on parent exit.
+            mitogen.core.io_op(self.child_sock.recv, 1)
+        finally:
+            self.broker.shutdown()
+            self.broker.join()
+
+            # Test frameworks living somewhere higher on the stack of the
+            # original parent process may try to catch sys.exit(), so do a C
+            # level exit instead.
+            os._exit(0)
 
     def _enable_router_debug(self):
         if 'MITOGEN_ROUTER_DEBUG' in os.environ:
@@ -215,15 +236,43 @@ class MuxProcess(object):
         if secs:
             mitogen.debug.dump_to_logger(secs=secs)
 
+    def _setup_responder(self, responder):
+        """
+        Configure :class:`mitogen.master.ModuleResponder` to only permit
+        certain packages, and to generate custom responses for certain modules.
+        """
+        responder.whitelist_prefix('ansible')
+        responder.whitelist_prefix('ansible_mitogen')
+        responder.whitelist_prefix('simplejson')
+        simplejson_path = os.path.join(os.path.dirname(__file__), 'compat')
+        sys.path.insert(0, simplejson_path)
+
+        # Ansible 2.3 is compatible with Python 2.4 targets, however
+        # ansible/__init__.py is not. Instead, executor/module_common.py writes
+        # out a 2.4-compatible namespace package for unknown reasons. So we
+        # copy it here.
+        responder.add_source_override(
+            fullname='ansible',
+            path=ansible.__file__,
+            source=(ANSIBLE_PKG_OVERRIDE % (
+                ansible.__version__,
+                ansible.__author__,
+            )).encode(),
+            is_pkg=True,
+        )
+
     def _setup_master(self):
         """
         Construct a Router, Broker, and mitogen.unix listener
         """
-        self.router = mitogen.master.Router(max_message_size=4096 * 1048576)
-        self.router.responder.whitelist_prefix('ansible')
-        self.router.responder.whitelist_prefix('ansible_mitogen')
-        mitogen.core.listen(self.router.broker, 'shutdown', self.on_broker_shutdown)
-        mitogen.core.listen(self.router.broker, 'exit', self.on_broker_exit)
+        self.broker = mitogen.master.Broker(install_watcher=False)
+        self.router = mitogen.master.Router(
+            broker=self.broker,
+            max_message_size=4096 * 1048576,
+        )
+        self._setup_responder(self.router.responder)
+        mitogen.core.listen(self.broker, 'shutdown', self.on_broker_shutdown)
+        mitogen.core.listen(self.broker, 'exit', self.on_broker_exit)
         self.listener = mitogen.unix.Listener(
             router=self.router,
             path=self.unix_listener_path,
@@ -245,7 +294,7 @@ class MuxProcess(object):
                 ansible_mitogen.services.ContextService(self.router),
                 ansible_mitogen.services.ModuleDepService(self.router),
             ],
-            size=getenv_int('MITOGEN_POOL_SIZE', default=16),
+            size=getenv_int('MITOGEN_POOL_SIZE', default=32),
         )
         LOG.debug('Service pool configured: size=%d', self.pool.size)
 
@@ -256,13 +305,9 @@ class MuxProcess(object):
         then cannot clean up pending handlers, which is required for the
         threads to exit gracefully.
         """
-        self.pool.stop(join=False)
-        try:
-            os.unlink(self.listener.path)
-        except OSError as e:
-            # Prevent a shutdown race with the parent process.
-            if e.args[0] != errno.ENOENT:
-                raise
+        # In normal operation we presently kill the process because there is
+        # not yet any way to cancel connect().
+        self.pool.stop(join=self.profiling)
 
     def on_broker_exit(self):
         """
@@ -270,10 +315,9 @@ class MuxProcess(object):
         ourself. In future this should gracefully join the pool, but TERM is
         fine for now.
         """
-        if os.environ.get('MITOGEN_PROFILING'):
-            # TODO: avoid killing pool threads before they have written their
-            # .pstats. Really shouldn't be using kill() here at all, but hard
-            # to guarantee services can always be unblocked during shutdown.
-            time.sleep(1)
-
-        os.kill(os.getpid(), signal.SIGTERM)
+        if not self.profiling:
+            # In normal operation we presently kill the process because there is
+            # not yet any way to cancel connect(). When profiling, threads
+            # including the broker must shut down gracefully, otherwise pstats
+            # won't be written.
+            os.kill(os.getpid(), signal.SIGTERM)

ansible_mitogen/runner.py

@@ -26,6 +26,7 @@
 # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 # POSSIBILITY OF SUCH DAMAGE.
 
+# !mitogen: minify_safe
 
 """
 These classes implement execution for each style of Ansible module. They are
@@ -35,23 +36,36 @@ Each class in here has a corresponding Planner class in planners.py that knows
 how to build arguments for it, preseed related data, etc.
 """
 
-from __future__ import absolute_import
-from __future__ import unicode_literals
-
 import atexit
-import ctypes
-import errno
+import codecs
 import imp
-import json
-import logging
 import os
 import shlex
+import shutil
 import sys
 import tempfile
+import traceback
 import types
 
 import mitogen.core
 import ansible_mitogen.target  # TODO: circular import
+from mitogen.core import b
+from mitogen.core import bytes_partition
+from mitogen.core import str_partition
+from mitogen.core import str_rpartition
+from mitogen.core import to_text
+
+try:
+    import ctypes
+except ImportError:
+    # Python 2.4
+    ctypes = None
+
+try:
+    import json
+except ImportError:
+    # Python 2.4
+    import simplejson as json
 
 try:
     # Cannot use cStringIO as it does not support Unicode.
@@ -64,6 +78,10 @@ try:
 except ImportError:
     from pipes import quote as shlex_quote
 
+# Absolute imports for <2.5.
+logging = __import__('logging')
+
+
 # Prevent accidental import of an Ansible module from hanging on stdin read.
 import ansible.module_utils.basic
 ansible.module_utils.basic._ANSIBLE_ARGS = '{}'
@@ -72,18 +90,27 @@ ansible.module_utils.basic._ANSIBLE_ARGS = '{}'
 # resolv.conf at startup and never implicitly reload it. Cope with that via an
 # explicit call to res_init() on each task invocation. BSD-alikes export it
 # directly, Linux #defines it as "__res_init".
-libc = ctypes.CDLL(None)
 libc__res_init = None
-for symbol in 'res_init', '__res_init':
-    try:
-        libc__res_init = getattr(libc, symbol)
-    except AttributeError:
-        pass
+if ctypes:
+    libc = ctypes.CDLL(None)
+    for symbol in 'res_init', '__res_init':
+        try:
+            libc__res_init = getattr(libc, symbol)
+        except AttributeError:
+            pass
 
 iteritems = getattr(dict, 'iteritems', dict.items)
 LOG = logging.getLogger(__name__)
 
 
+if mitogen.core.PY3:
+    shlex_split = shlex.split
+else:
+    def shlex_split(s, comments=False):
+        return [mitogen.core.to_text(token)
+                for token in shlex.split(str(s), comments=comments)]
+
+
 class EnvironmentFileWatcher(object):
     """
     Usually Ansible edits to /etc/environment and ~/.pam_environment are
@@ -118,8 +145,11 @@ class EnvironmentFileWatcher(object):
 
     def _load(self):
         try:
-            with open(self.path, 'r') as fp:
+            fp = codecs.open(self.path, 'r', encoding='utf-8')
+            try:
                 return list(self._parse(fp))
+            finally:
+                fp.close()
         except IOError:
             return []
 
@@ -129,14 +159,14 @@ class EnvironmentFileWatcher(object):
         """
         for line in fp:
             # '   #export foo=some var  ' -> ['#export', 'foo=some var  ']
-            bits = shlex.split(line, comments=True)
+            bits = shlex_split(line, comments=True)
             if (not bits) or bits[0].startswith('#'):
                 continue
 
-            if bits[0] == 'export':
+            if bits[0] == u'export':
                 bits.pop(0)
 
-            key, sep, value = (' '.join(bits)).partition('=')
+            key, sep, value = str_partition(u' '.join(bits), u'=')
             if key and sep:
                 yield key, value
 
@@ -255,7 +285,7 @@ class Runner(object):
         self.service_context = service_context
         self.econtext = econtext
         self.detach = detach
-        self.args = json.loads(json_args)
+        self.args = json.loads(mitogen.core.to_text(json_args))
         self.good_temp_dir = good_temp_dir
         self.extra_env = extra_env
         self.env = env
@@ -354,6 +384,55 @@ class Runner(object):
             self.revert()
 
 
+class AtExitWrapper(object):
+    """
+    issue #397, #454: Newer Ansibles use :func:`atexit.register` to trigger
+    tmpdir cleanup when AnsibleModule.tmpdir is responsible for creating its
+    own temporary directory, however with Mitogen processes are preserved
+    across tasks, meaning cleanup must happen earlier.
+
+    Patch :func:`atexit.register`, catching :func:`shutil.rmtree` calls so they
+    can be executed on task completion, rather than on process shutdown.
+    """
+    # Wrapped in a dict to avoid instance method decoration.
+    original = {
+        'register': atexit.register
+    }
+
+    def __init__(self):
+        assert atexit.register == self.original['register'], \
+            "AtExitWrapper installed twice."
+        atexit.register = self._atexit__register
+        self.deferred = []
+
+    def revert(self):
+        """
+        Restore the original :func:`atexit.register`.
+        """
+        assert atexit.register == self._atexit__register, \
+            "AtExitWrapper not installed."
+        atexit.register = self.original['register']
+
+    def run_callbacks(self):
+        while self.deferred:
+            func, targs, kwargs = self.deferred.pop()
+            try:
+                func(*targs, **kwargs)
+            except Exception:
+                LOG.exception('While running atexit callbacks')
+
+    def _atexit__register(self, func, *targs, **kwargs):
+        """
+        Intercept :func:`atexit.register` calls, diverting any to
+        :func:`shutil.rmtree` into a private list.
+        """
+        if func == shutil.rmtree:
+            self.deferred.append((func, targs, kwargs))
+            return
+
+        self.original['register'](func, *targs, **kwargs)
+
+
 class ModuleUtilsImporter(object):
     """
     :param list module_utils:
@@ -388,7 +467,7 @@ class ModuleUtilsImporter(object):
             mod.__path__ = []
             mod.__package__ = str(fullname)
         else:
-            mod.__package__ = str(fullname.rpartition('.')[0])
+            mod.__package__ = str(str_rpartition(to_text(fullname), '.')[0])
         exec(code, mod.__dict__)
         self._loaded.add(fullname)
         return mod
@@ -404,6 +483,8 @@ class TemporaryEnvironment(object):
         self.original = dict(os.environ)
         self.env = env or {}
         for key, value in iteritems(self.env):
+            key = mitogen.core.to_text(key)
+            value = mitogen.core.to_text(value)
             if value is None:
                 os.environ.pop(key, None)
             else:
@@ -530,7 +611,7 @@ class ProgramRunner(Runner):
         Return the final argument vector used to execute the program.
         """
         return [
-            self.args['_ansible_shell_executable'],
+            self.args.get('_ansible_shell_executable', '/bin/sh'),
             '-c',
             self._get_shell_fragment(),
         ]
@@ -547,18 +628,19 @@ class ProgramRunner(Runner):
                 args=self._get_argv(),
                 emulate_tty=self.emulate_tty,
             )
-        except Exception as e:
+        except Exception:
             LOG.exception('While running %s', self._get_argv())
+            e = sys.exc_info()[1]
             return {
-                'rc': 1,
-                'stdout': '',
-                'stderr': '%s: %s' % (type(e), e),
+                u'rc': 1,
+                u'stdout': u'',
+                u'stderr': u'%s: %s' % (type(e), e),
             }
 
         return {
-            'rc': rc,
-            'stdout': mitogen.core.to_text(stdout),
-            'stderr': mitogen.core.to_text(stderr),
+            u'rc': rc,
+            u'stdout': mitogen.core.to_text(stdout),
+            u'stderr': mitogen.core.to_text(stderr),
         }
 
 
@@ -608,7 +690,7 @@ class ScriptRunner(ProgramRunner):
         self.interpreter_fragment = interpreter_fragment
         self.is_python = is_python
 
-    b_ENCODING_STRING = b'# -*- coding: utf-8 -*-'
+    b_ENCODING_STRING = b('# -*- coding: utf-8 -*-')
 
     def _get_program(self):
         return self._rewrite_source(
@@ -617,7 +699,7 @@ class ScriptRunner(ProgramRunner):
 
     def _get_argv(self):
         return [
-            self.args['_ansible_shell_executable'],
+            self.args.get('_ansible_shell_executable', '/bin/sh'),
             '-c',
             self._get_shell_fragment(),
         ]
@@ -641,13 +723,13 @@ class ScriptRunner(ProgramRunner):
         # While Ansible rewrites the #! using ansible_*_interpreter, it is
         # never actually used to execute the script, instead it is a shell
         # fragment consumed by shell/__init__.py::build_module_command().
-        new = [b'#!' + utf8(self.interpreter_fragment)]
+        new = [b('#!') + utf8(self.interpreter_fragment)]
         if self.is_python:
             new.append(self.b_ENCODING_STRING)
 
-        _, _, rest = s.partition(b'\n')
+        _, _, rest = bytes_partition(s, b('\n'))
         new.append(rest)
-        return b'\n'.join(new)
+        return b('\n').join(new)
 
 
 class NewStyleRunner(ScriptRunner):
@@ -701,6 +783,7 @@ class NewStyleRunner(ScriptRunner):
         )
         self._setup_imports()
         self._setup_excepthook()
+        self.atexit_wrapper = AtExitWrapper()
         if libc__res_init:
             libc__res_init()
 
@@ -708,6 +791,7 @@ class NewStyleRunner(ScriptRunner):
         sys.excepthook = self.original_excepthook
 
     def revert(self):
+        self.atexit_wrapper.revert()
         self._argv.revert()
         self._stdio.revert()
         self._revert_excepthook()
@@ -733,16 +817,18 @@ class NewStyleRunner(ScriptRunner):
             return self._code_by_path[self.path]
         except KeyError:
             return self._code_by_path.setdefault(self.path, compile(
-                source=self.source,
-                filename="master:" + self.path,
-                mode='exec',
-                dont_inherit=True,
+                # Py2.4 doesn't support kwargs.
+                self.source,            # source
+                "master:" + self.path,  # filename
+                'exec',                 # mode
+                0,                      # flags
+                True,                   # dont_inherit
             ))
 
     if mitogen.core.PY3:
         main_module_name = '__main__'
     else:
-        main_module_name = b'__main__'
+        main_module_name = b('__main__')
 
     def _handle_magic_exception(self, mod, exc):
         """
@@ -764,18 +850,10 @@ class NewStyleRunner(ScriptRunner):
                 exec(code, vars(mod))
             else:
                 exec('exec code in vars(mod)')
-        except Exception as e:
-            self._handle_magic_exception(mod, e)
+        except Exception:
+            self._handle_magic_exception(mod, sys.exc_info()[1])
             raise
 
-    def _run_atexit_funcs(self):
-        """
-        Newer Ansibles use atexit.register() to trigger tmpdir cleanup, when
-        AnsibleModule.tmpdir is responsible for creating its own temporary
-        directory.
-        """
-        atexit._run_exitfuncs()
-
     def _run(self):
         mod = types.ModuleType(self.main_module_name)
         mod.__package__ = None
@@ -789,24 +867,30 @@ class NewStyleRunner(ScriptRunner):
         )
 
         code = self._get_code()
-        exc = None
+        rc = 2
         try:
             try:
                 self._run_code(code, mod)
-            finally:
-                self._run_atexit_funcs()
-        except SystemExit as e:
-            exc = e
+            except SystemExit:
+                exc = sys.exc_info()[1]
+                rc = exc.args[0]
+            except Exception:
+                # This writes to stderr by default.
+                traceback.print_exc()
+                rc = 1
+
+        finally:
+            self.atexit_wrapper.run_callbacks()
 
         return {
-            'rc': exc.args[0] if exc else 2,
-            'stdout': mitogen.core.to_text(sys.stdout.getvalue()),
-            'stderr': mitogen.core.to_text(sys.stderr.getvalue()),
+            u'rc': rc,
+            u'stdout': mitogen.core.to_text(sys.stdout.getvalue()),
+            u'stderr': mitogen.core.to_text(sys.stderr.getvalue()),
         }
 
 
 class JsonArgsRunner(ScriptRunner):
-    JSON_ARGS = b'<<INCLUDE_ANSIBLE_MODULE_JSON_ARGS>>'
+    JSON_ARGS = b('<<INCLUDE_ANSIBLE_MODULE_JSON_ARGS>>')
 
     def _get_args_contents(self):
         return json.dumps(self.args).encode()

ansible_mitogen/services.py

@@ -26,6 +26,8 @@
 # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 # POSSIBILITY OF SUCH DAMAGE.
 
+# !mitogen: minify_safe
+
 """
 Classes in this file define Mitogen 'services' that run (initially) within the
 connection multiplexer process that is forked off the top-level controller
@@ -79,14 +81,35 @@ else:
 
 
 def _get_candidate_temp_dirs():
-    options = ansible.constants.config.get_plugin_options('shell', 'sh')
+    try:
+        # >=2.5
+        options = ansible.constants.config.get_plugin_options('shell', 'sh')
+        remote_tmp = options.get('remote_tmp') or ansible.constants.DEFAULT_REMOTE_TMP
+        system_tmpdirs = options.get('system_tmpdirs', ('/var/tmp', '/tmp'))
+    except AttributeError:
+        # 2.3
+        remote_tmp = ansible.constants.DEFAULT_REMOTE_TMP
+        system_tmpdirs = ('/var/tmp', '/tmp')
+
+    return mitogen.utils.cast([remote_tmp] + list(system_tmpdirs))
 
-    # Pre 2.5 this came from ansible.constants.
-    remote_tmp = (options.get('remote_tmp') or
-                  ansible.constants.DEFAULT_REMOTE_TMP)
-    dirs = list(options.get('system_tmpdirs', ('/var/tmp', '/tmp')))
-    dirs.insert(0, remote_tmp)
-    return mitogen.utils.cast(dirs)
+
+def key_from_dict(**kwargs):
+    """
+    Return a unique string representation of a dict as quickly as possible.
+    Used to generated deduplication keys from a request.
+    """
+    out = []
+    stack = [kwargs]
+    while stack:
+        obj = stack.pop()
+        if isinstance(obj, dict):
+            stack.extend(sorted(obj.items()))
+        elif isinstance(obj, (list, tuple)):
+            stack.extend(obj)
+        else:
+            out.append(str(obj))
+    return ''.join(out)
 
 
 class Error(Exception):
@@ -113,7 +136,7 @@ class ContextService(mitogen.service.Service):
         super(ContextService, self).__init__(*args, **kwargs)
         self._lock = threading.Lock()
         #: Records the :meth:`get` result dict for successful calls, returned
-        #: for identical subsequent calls. Keyed by :meth:`key_from_kwargs`.
+        #: for identical subsequent calls. Keyed by :meth:`key_from_dict`.
         self._response_by_key = {}
         #: List of :class:`mitogen.core.Latch` awaiting the result for a
         #: particular key.
@@ -126,8 +149,27 @@ class ContextService(mitogen.service.Service):
         #: :attr:`max_interpreters` is reached, the most recently used context
         #: is destroyed to make room for any additional context.
         self._lru_by_via = {}
-        #: :meth:`key_from_kwargs` result by Context.
+        #: :func:`key_from_dict` result by Context.
         self._key_by_context = {}
+        #: Mapping of Context -> parent Context
+        self._via_by_context = {}
+
+    @mitogen.service.expose(mitogen.service.AllowParents())
+    @mitogen.service.arg_spec({
+        'context': mitogen.core.Context
+    })
+    def reset(self, context):
+        """
+        Return a reference, forcing close and discard of the underlying
+        connection. Used for 'meta: reset_connection' or when some other error
+        is detected.
+        """
+        LOG.debug('%r.reset(%r)', self, context)
+        self._lock.acquire()
+        try:
+            self._shutdown_unlocked(context)
+        finally:
+            self._lock.release()
 
     @mitogen.service.expose(mitogen.service.AllowParents())
     @mitogen.service.arg_spec({
@@ -149,29 +191,13 @@ class ContextService(mitogen.service.Service):
         finally:
             self._lock.release()
 
-    def key_from_kwargs(self, **kwargs):
-        """
-        Generate a deduplication key from the request.
-        """
-        out = []
-        stack = [kwargs]
-        while stack:
-            obj = stack.pop()
-            if isinstance(obj, dict):
-                stack.extend(sorted(obj.items()))
-            elif isinstance(obj, (list, tuple)):
-                stack.extend(obj)
-            else:
-                out.append(str(obj))
-        return ''.join(out)
-
     def _produce_response(self, key, response):
         """
         Reply to every waiting request matching a configuration key with a
         response dictionary, deleting the list of waiters when done.
 
         :param str key:
-            Result of :meth:`key_from_kwargs`
+            Result of :meth:`key_from_dict`
         :param dict response:
             Response dictionary
         :returns:
@@ -187,6 +213,19 @@ class ContextService(mitogen.service.Service):
             self._lock.release()
         return count
 
+    def _forget_context_unlocked(self, context):
+        key = self._key_by_context.get(context)
+        if key is None:
+            LOG.debug('%r: attempt to forget unknown %r', self, context)
+            return
+
+        self._response_by_key.pop(key, None)
+        self._latches_by_key.pop(key, None)
+        self._key_by_context.pop(context, None)
+        self._refs_by_context.pop(context, None)
+        self._via_by_context.pop(context, None)
+        self._lru_by_via.pop(context, None)
+
     def _shutdown_unlocked(self, context, lru=None, new_context=None):
         """
         Arrange for `context` to be shut down, and optionally add `new_context`
@@ -194,15 +233,15 @@ class ContextService(mitogen.service.Service):
         """
         LOG.info('%r._shutdown_unlocked(): shutting down %r', self, context)
         context.shutdown()
-
-        key = self._key_by_context[context]
-        del self._response_by_key[key]
-        del self._refs_by_context[context]
-        del self._key_by_context[context]
-        if lru and context in lru:
-            lru.remove(context)
-        if new_context:
-            lru.append(new_context)
+        via = self._via_by_context.get(context)
+        if via:
+            lru = self._lru_by_via.get(via)
+            if lru:
+                if context in lru:
+                    lru.remove(context)
+                if new_context:
+                    lru.append(new_context)
+        self._forget_context_unlocked(context)
 
     def _update_lru_unlocked(self, new_context, spec, via):
         """
@@ -210,6 +249,8 @@ class ContextService(mitogen.service.Service):
         by `kwargs`, destroying the most recently created context if the list
         is full. Finally add `new_context` to the list.
         """
+        self._via_by_context[new_context] = via
+
         lru = self._lru_by_via.setdefault(via, [])
         if len(lru) < self.max_interpreters:
             lru.append(new_context)
@@ -232,6 +273,23 @@ class ContextService(mitogen.service.Service):
         finally:
             self._lock.release()
 
+    @mitogen.service.expose(mitogen.service.AllowParents())
+    def dump(self):
+        """
+        For testing, return a list of dicts describing every currently
+        connected context.
+        """
+        return [
+            {
+                'context_name': context.name,
+                'via': getattr(self._via_by_context.get(context),
+                               'name', None),
+                'refs': self._refs_by_context.get(context),
+            }
+            for context, key in sorted(self._key_by_context.items(),
+                                       key=lambda c_k: c_k[0].context_id)
+        ]
+
     @mitogen.service.expose(mitogen.service.AllowParents())
     def shutdown_all(self):
         """
@@ -241,30 +299,19 @@ class ContextService(mitogen.service.Service):
         try:
             for context in list(self._key_by_context):
                 self._shutdown_unlocked(context)
-            self._lru_by_via = {}
         finally:
             self._lock.release()
 
-    def _on_stream_disconnect(self, stream):
+    def _on_context_disconnect(self, context):
         """
-        Respond to Stream disconnection by deleting any record of contexts
-        reached via that stream. This method runs in the Broker thread and must
-        not to block.
+        Respond to Context disconnect event by deleting any record of the no
+        longer reachable context.  This method runs in the Broker thread and
+        must not to block.
         """
-        # TODO: there is a race between creation of a context and disconnection
-        # of its related stream. An error reply should be sent to any message
-        # in _latches_by_key below.
         self._lock.acquire()
         try:
-            for context, key in list(self._key_by_context.items()):
-                if context.context_id in stream.routes:
-                    LOG.info('Dropping %r due to disconnect of %r',
-                             context, stream)
-                    self._response_by_key.pop(key, None)
-                    self._latches_by_key.pop(key, None)
-                    self._refs_by_context.pop(context, None)
-                    self._lru_by_via.pop(context, None)
-                    self._refs_by_context.pop(context, None)
+            LOG.info('%r: Forgetting %r due to stream disconnect', self, context)
+            self._forget_context_unlocked(context)
         finally:
             self._lock.release()
 
@@ -330,13 +377,10 @@ class ContextService(mitogen.service.Service):
         context = method(via=via, unidirectional=True, **spec['kwargs'])
         if via and spec.get('enable_lru'):
             self._update_lru(context, spec, via)
-        else:
-            # For directly connected contexts, listen to the associated
-            # Stream's disconnect event and use it to invalidate dependent
-            # Contexts.
-            stream = self.router.stream_by_id(context.context_id)
-            mitogen.core.listen(stream, 'disconnect',
-                                lambda: self._on_stream_disconnect(stream))
+
+        # Forget the context when its disconnect event fires.
+        mitogen.core.listen(context, 'disconnect',
+            lambda: self._on_context_disconnect(context))
 
         self._send_module_forwards(context)
         init_child_result = context.call(
@@ -360,7 +404,7 @@ class ContextService(mitogen.service.Service):
 
     def _wait_or_start(self, spec, via=None):
         latch = mitogen.core.Latch()
-        key = self.key_from_kwargs(via=via, **spec)
+        key = key_from_dict(via=via, **spec)
         self._lock.acquire()
         try:
             response = self._response_by_key.get(key)
@@ -453,14 +497,16 @@ class ModuleDepService(mitogen.service.Service):
 
     def _get_builtin_names(self, builtin_path, resolved):
         return [
-            fullname
+            mitogen.core.to_text(fullname)
             for fullname, path, is_pkg in resolved
             if os.path.abspath(path).startswith(builtin_path)
         ]
 
     def _get_custom_tups(self, builtin_path, resolved):
         return [
-            (fullname, path, is_pkg)
+            (mitogen.core.to_text(fullname),
+             mitogen.core.to_text(path),
+             is_pkg)
             for fullname, path, is_pkg in resolved
             if not os.path.abspath(path).startswith(builtin_path)
         ]

ansible_mitogen/strategy.py

@@ -28,11 +28,45 @@
 
 from __future__ import absolute_import
 import os
+import signal
+import threading
 
+import mitogen.core
+import ansible_mitogen.affinity
 import ansible_mitogen.loaders
 import ansible_mitogen.mixins
 import ansible_mitogen.process
 
+import ansible.executor.process.worker
+
+
+def _patch_awx_callback():
+    """
+    issue #400: AWX loads a display callback that suffers from thread-safety
+    issues. Detect the presence of older AWX versions and patch the bug.
+    """
+    # AWX uses sitecustomize.py to force-load this package. If it exists, we're
+    # running under AWX.
+    try:
+        from awx_display_callback.events import EventContext
+        from awx_display_callback.events import event_context
+    except ImportError:
+        return
+
+    if hasattr(EventContext(), '_local'):
+        # Patched version.
+        return
+
+    def patch_add_local(self, **kwargs):
+        tls = vars(self._local)
+        ctx = tls.setdefault('_ctx', {})
+        ctx.update(kwargs)
+
+    EventContext._local = threading.local()
+    EventContext.add_local = patch_add_local
+
+_patch_awx_callback()
+
 
 def wrap_action_loader__get(name, *args, **kwargs):
     """
@@ -46,7 +80,6 @@ def wrap_action_loader__get(name, *args, **kwargs):
     """
     klass = action_loader__get(name, class_only=True)
     if klass:
-        wrapped_name = 'MitogenActionModule_' + name
         bases = (ansible_mitogen.mixins.ActionModuleMixin, klass)
         adorned_klass = type(str(name), bases, {})
         if kwargs.get('class_only'):
@@ -65,6 +98,22 @@ def wrap_connection_loader__get(name, *args, **kwargs):
     return connection_loader__get(name, *args, **kwargs)
 
 
+def wrap_worker__run(*args, **kwargs):
+    """
+    While the strategy is active, rewrite connection_loader.get() calls for
+    some transports into requests for a compatible Mitogen transport.
+    """
+    # Ignore parent's attempts to murder us when we still need to write
+    # profiling output.
+    if mitogen.core._profile_hook.__name__ != '_profile_hook':
+        signal.signal(signal.SIGTERM, signal.SIG_IGN)
+
+    ansible_mitogen.affinity.policy.assign_worker()
+    return mitogen.core._profile_hook('WorkerProcess',
+        lambda: worker__run(*args, **kwargs)
+    )
+
+
 class StrategyMixin(object):
     """
     This mix-in enhances any built-in strategy by arranging for various Mitogen
@@ -139,22 +188,56 @@ class StrategyMixin(object):
         connection_loader__get = ansible_mitogen.loaders.connection_loader.get
         ansible_mitogen.loaders.connection_loader.get = wrap_connection_loader__get
 
+        global worker__run
+        worker__run = ansible.executor.process.worker.WorkerProcess.run
+        ansible.executor.process.worker.WorkerProcess.run = wrap_worker__run
+
     def _remove_wrappers(self):
         """
         Uninstall the PluginLoader monkey patches.
         """
         ansible_mitogen.loaders.action_loader.get = action_loader__get
         ansible_mitogen.loaders.connection_loader.get = connection_loader__get
+        ansible.executor.process.worker.WorkerProcess.run = worker__run
 
-    def _add_connection_plugin_path(self):
+    def _add_plugin_paths(self):
         """
-        Add the mitogen connection plug-in directory to the ModuleLoader path,
-        avoiding the need for manual configuration.
+        Add the Mitogen plug-in directories to the ModuleLoader path, avoiding
+        the need for manual configuration.
         """
         base_dir = os.path.join(os.path.dirname(__file__), 'plugins')
         ansible_mitogen.loaders.connection_loader.add_directory(
             os.path.join(base_dir, 'connection')
         )
+        ansible_mitogen.loaders.action_loader.add_directory(
+            os.path.join(base_dir, 'action')
+        )
+
+    def _queue_task(self, host, task, task_vars, play_context):
+        """
+        Many PluginLoader caches are defective as they are only populated in
+        the ephemeral WorkerProcess. Touch each plug-in path before forking to
+        ensure all workers receive a hot cache.
+        """
+        ansible_mitogen.loaders.module_loader.find_plugin(
+            name=task.action,
+            mod_type='',
+        )
+        ansible_mitogen.loaders.connection_loader.get(
+            name=play_context.connection,
+            class_only=True,
+        )
+        ansible_mitogen.loaders.action_loader.get(
+            name=task.action,
+            class_only=True,
+        )
+
+        return super(StrategyMixin, self)._queue_task(
+            host=host,
+            task=task,
+            task_vars=task_vars,
+            play_context=play_context,
+        )
 
     def run(self, iterator, play_context, result=0):
         """
@@ -162,9 +245,12 @@ class StrategyMixin(object):
         the strategy's real run() method.
         """
         ansible_mitogen.process.MuxProcess.start()
-        self._add_connection_plugin_path()
+        run = super(StrategyMixin, self).run
+        self._add_plugin_paths()
         self._install_wrappers()
         try:
-            return super(StrategyMixin, self).run(iterator, play_context)
+            return mitogen.core._profile_hook('Strategy',
+                lambda: run(iterator, play_context)
+            )
         finally:
             self._remove_wrappers()

ansible_mitogen/target.py

@@ -26,24 +26,19 @@
 # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 # POSSIBILITY OF SUCH DAMAGE.
 
+# !mitogen: minify_safe
+
 """
 Helper functions intended to be executed on the target. These are entrypoints
 for file transfer, module execution and sundry bits like changing file modes.
 """
 
-from __future__ import absolute_import
-from __future__ import unicode_literals
-
 import errno
-import functools
 import grp
-import json
-import logging
 import operator
 import os
 import pwd
 import re
-import resource
 import signal
 import stat
 import subprocess
@@ -52,10 +47,32 @@ import tempfile
 import traceback
 import types
 
+# Absolute imports for <2.5.
+logging = __import__('logging')
+
 import mitogen.core
 import mitogen.fork
 import mitogen.parent
 import mitogen.service
+from mitogen.core import b
+
+try:
+    import json
+except ImportError:
+    import simplejson as json
+
+try:
+    reduce
+except NameError:
+    # Python 3.x.
+    from functools import reduce
+
+try:
+    BaseException
+except NameError:
+    # Python 2.4
+    BaseException = Exception
+
 
 # Ansible since PR #41749 inserts "import __main__" into
 # ansible.module_utils.basic. Mitogen's importer will refuse such an import, so
@@ -71,16 +88,23 @@ import ansible_mitogen.runner
 LOG = logging.getLogger(__name__)
 
 MAKE_TEMP_FAILED_MSG = (
-    "Unable to find a useable temporary directory. This likely means no\n"
-    "system-supplied TMP directory can be written to, or all directories\n"
-    "were mounted on 'noexec' filesystems.\n"
-    "\n"
-    "The following paths were tried:\n"
-    "    %(namelist)s\n"
-    "\n"
-    "Please check '-vvv' output for a log of individual path errors."
+    u"Unable to find a useable temporary directory. This likely means no\n"
+    u"system-supplied TMP directory can be written to, or all directories\n"
+    u"were mounted on 'noexec' filesystems.\n"
+    u"\n"
+    u"The following paths were tried:\n"
+    u"    %(namelist)s\n"
+    u"\n"
+    u"Please check '-vvv' output for a log of individual path errors."
 )
 
+# Python 2.4/2.5 cannot support fork+threads whatsoever, it doesn't even fix up
+# interpreter state. So 2.4/2.5 interpreters start .local() contexts for
+# isolation instead. Since we don't have any crazy memory sharing problems to
+# avoid, there is no virginal fork parent either. The child is started directly
+# from the login/become process. In future this will be default everywhere,
+# fork is brainwrong from the stone age.
+FORK_SUPPORTED = sys.version_info >= (2, 6)
 
 #: Initialized to an econtext.parent.Context pointing at a pristine fork of
 #: the target Python interpreter before it executes any code or imports.
@@ -91,20 +115,46 @@ _fork_parent = None
 good_temp_dir = None
 
 
-# issue #362: subprocess.Popen(close_fds=True) aka. AnsibleModule.run_command()
-# loops the entire SC_OPEN_MAX space. CentOS>5 ships with 1,048,576 FDs by
-# default, resulting in huge (>500ms) runtime waste running many commands.
-# Therefore if we are a child, cap the range to something reasonable.
-rlimit = resource.getrlimit(resource.RLIMIT_NOFILE)
-if (rlimit[0] > 512 or rlimit[1] > 512) and not mitogen.is_master:
-    resource.setrlimit(resource.RLIMIT_NOFILE, (512, 512))
-    subprocess.MAXFD = 512  # Python <3.x
-del rlimit
+def subprocess__Popen__close_fds(self, but):
+    """
+    issue #362, #435: subprocess.Popen(close_fds=True) aka.
+    AnsibleModule.run_command() loops the entire FD space on Python<3.2.
+    CentOS>5 ships with 1,048,576 FDs by default, resulting in huge (>500ms)
+    latency starting children. Therefore replace Popen._close_fds on Linux with
+    a version that is O(fds) rather than O(_SC_OPEN_MAX).
+    """
+    try:
+        names = os.listdir(u'/proc/self/fd')
+    except OSError:
+        # May fail if acting on a container that does not have /proc mounted.
+        self._original_close_fds(but)
+        return
+
+    for name in names:
+        if not name.isdigit():
+            continue
+
+        fd = int(name, 10)
+        if fd > 2 and fd != but:
+            try:
+                os.close(fd)
+            except OSError:
+                pass
+
+
+if (
+    sys.platform.startswith(u'linux') and
+    sys.version < u'3.0' and
+    hasattr(subprocess.Popen, u'_close_fds') and
+    not mitogen.is_master
+):
+    subprocess.Popen._original_close_fds = subprocess.Popen._close_fds
+    subprocess.Popen._close_fds = subprocess__Popen__close_fds
 
 
 def get_small_file(context, path):
     """
-    Basic in-memory caching module fetcher. This generates an one roundtrip for
+    Basic in-memory caching module fetcher. This generates one roundtrip for
     every previously unseen file, so it is only a temporary solution.
 
     :param context:
@@ -117,7 +167,7 @@ def get_small_file(context, path):
         Bytestring file data.
     """
     pool = mitogen.service.get_or_create_pool(router=context.router)
-    service = pool.get_service('mitogen.service.PushFileService')
+    service = pool.get_service(u'mitogen.service.PushFileService')
     return service.get(path)
 
 
@@ -127,8 +177,8 @@ def transfer_file(context, in_path, out_path, sync=False, set_owner=False):
     controller.
 
     :param mitogen.core.Context context:
-        Reference to the context hosting the FileService that will be used to
-        fetch the file.
+        Reference to the context hosting the FileService that will transmit the
+        file.
     :param bytes in_path:
         FileService registered name of the input file.
     :param bytes out_path:
@@ -159,9 +209,10 @@ def transfer_file(context, in_path, out_path, sync=False, set_owner=False):
             if not ok:
                 raise IOError('transfer of %r was interrupted.' % (in_path,))
 
-            os.fchmod(fp.fileno(), metadata['mode'])
+            set_file_mode(tmp_path, metadata['mode'], fd=fp.fileno())
             if set_owner:
-                set_fd_owner(fp.fileno(), metadata['owner'], metadata['group'])
+                set_file_owner(tmp_path, metadata['owner'], metadata['group'],
+                               fd=fp.fileno())
         finally:
             fp.close()
 
@@ -184,7 +235,8 @@ def prune_tree(path):
     try:
         os.unlink(path)
         return
-    except OSError as e:
+    except OSError:
+        e = sys.exc_info()[1]
         if not (os.path.isdir(path) and
                 e.args[0] in (errno.EPERM, errno.EISDIR)):
             LOG.error('prune_tree(%r): %s', path, e)
@@ -194,7 +246,8 @@ def prune_tree(path):
         # Ensure write access for readonly directories. Ignore error in case
         # path is on a weird filesystem (e.g. vfat).
         os.chmod(path, int('0700', 8))
-    except OSError as e:
+    except OSError:
+        e = sys.exc_info()[1]
         LOG.warning('prune_tree(%r): %s', path, e)
 
     try:
@@ -202,7 +255,8 @@ def prune_tree(path):
             if name not in ('.', '..'):
                 prune_tree(os.path.join(path, name))
         os.rmdir(path)
-    except OSError as e:
+    except OSError:
+        e = sys.exc_info()[1]
         LOG.error('prune_tree(%r): %s', path, e)
 
 
@@ -223,7 +277,8 @@ def is_good_temp_dir(path):
     if not os.path.exists(path):
         try:
             os.makedirs(path, mode=int('0700', 8))
-        except OSError as e:
+        except OSError:
+            e = sys.exc_info()[1]
             LOG.debug('temp dir %r unusable: did not exist and attempting '
                       'to create it failed: %s', path, e)
             return False
@@ -233,24 +288,26 @@ def is_good_temp_dir(path):
             prefix='ansible_mitogen_is_good_temp_dir',
             dir=path,
         )
-    except (OSError, IOError) as e:
+    except (OSError, IOError):
+        e = sys.exc_info()[1]
         LOG.debug('temp dir %r unusable: %s', path, e)
         return False
 
     try:
         try:
             os.chmod(tmp.name, int('0700', 8))
-        except OSError as e:
-            LOG.debug('temp dir %r unusable: %s: chmod failed: %s',
-                      path, e)
+        except OSError:
+            e = sys.exc_info()[1]
+            LOG.debug('temp dir %r unusable: chmod failed: %s', path, e)
             return False
 
         try:
             # access(.., X_OK) is sufficient to detect noexec.
             if not os.access(tmp.name, os.X_OK):
                 raise OSError('filesystem appears to be mounted noexec')
-        except OSError as e:
-            LOG.debug('temp dir %r unusable: %s: %s', path, e)
+        except OSError:
+            e = sys.exc_info()[1]
+            LOG.debug('temp dir %r unusable: %s', path, e)
             return False
     finally:
         tmp.close()
@@ -305,8 +362,9 @@ def init_child(econtext, log_level, candidate_temp_dirs):
         Dict like::
 
             {
-                'fork_context': mitogen.core.Context.
-                'home_dir': str.
+                'fork_context': mitogen.core.Context or None,
+                'good_temp_dir': ...
+                'home_dir': str
             }
 
         Where `fork_context` refers to the newly forked 'fork parent' context
@@ -320,28 +378,36 @@ def init_child(econtext, log_level, candidate_temp_dirs):
     logging.getLogger('ansible_mitogen').setLevel(log_level)
 
     global _fork_parent
-    mitogen.parent.upgrade_router(econtext)
-    _fork_parent = econtext.router.fork()
+    if FORK_SUPPORTED:
+        mitogen.parent.upgrade_router(econtext)
+        _fork_parent = econtext.router.fork()
 
     global good_temp_dir
     good_temp_dir = find_good_temp_dir(candidate_temp_dirs)
 
     return {
-        'fork_context': _fork_parent,
-        'home_dir': mitogen.core.to_text(os.path.expanduser('~')),
-        'good_temp_dir': good_temp_dir,
+        u'fork_context': _fork_parent,
+        u'home_dir': mitogen.core.to_text(os.path.expanduser('~')),
+        u'good_temp_dir': good_temp_dir,
     }
 
 
 @mitogen.core.takes_econtext
-def create_fork_child(econtext):
+def spawn_isolated_child(econtext):
     """
     For helper functions executed in the fork parent context, arrange for
     the context's router to be upgraded as necessary and for a new child to be
     prepared.
+
+    The actual fork occurs from the 'virginal fork parent', which does not have
+    any Ansible modules loaded prior to fork, to avoid conflicts resulting from
+    custom module_utils paths.
     """
     mitogen.parent.upgrade_router(econtext)
-    context = econtext.router.fork()
+    if FORK_SUPPORTED:
+        context = econtext.router.fork()
+    else:
+        context = econtext.router.local()
     LOG.debug('create_fork_child() -> %r', context)
     return context
 
@@ -355,7 +421,7 @@ def run_module(kwargs):
     """
     runner_name = kwargs.pop('runner_name')
     klass = getattr(ansible_mitogen.runner, runner_name)
-    impl = klass(**kwargs)
+    impl = klass(**mitogen.core.Kwargs(kwargs))
     return impl.run()
 
 
@@ -366,9 +432,10 @@ def _get_async_dir():
 
 
 class AsyncRunner(object):
-    def __init__(self, job_id, timeout_secs, econtext, kwargs):
+    def __init__(self, job_id, timeout_secs, started_sender, econtext, kwargs):
         self.job_id = job_id
         self.timeout_secs = timeout_secs
+        self.started_sender = started_sender
         self.econtext = econtext
         self.kwargs = kwargs
         self._timed_out = False
@@ -388,8 +455,11 @@ class AsyncRunner(object):
         dct.setdefault('ansible_job_id', self.job_id)
         dct.setdefault('data', '')
 
-        with open(self.path + '.tmp', 'w') as fp:
+        fp = open(self.path + '.tmp', 'w')
+        try:
             fp.write(json.dumps(dct))
+        finally:
+            fp.close()
         os.rename(self.path + '.tmp', self.path)
 
     def _on_sigalrm(self, signum, frame):
@@ -448,6 +518,7 @@ class AsyncRunner(object):
             'finished': 0,
             'pid': os.getpid()
         })
+        self.started_sender.send(True)
 
         if self.timeout_secs > 0:
             self._install_alarm()
@@ -483,13 +554,26 @@ class AsyncRunner(object):
 
 
 @mitogen.core.takes_econtext
-def run_module_async(kwargs, job_id, timeout_secs, econtext):
+def run_module_async(kwargs, job_id, timeout_secs, started_sender, econtext):
     """
     Execute a module with its run status and result written to a file,
     terminating on the process on completion. This function must run in a child
     forked using :func:`create_fork_child`.
-    """
-    arunner = AsyncRunner(job_id, timeout_secs, econtext, kwargs)
+
+    @param mitogen.core.Sender started_sender:
+        A sender that will receive :data:`True` once the job has reached a
+        point where its initial job file has been written. This is required to
+        avoid a race where an overly eager controller can check for a task
+        before it has reached that point in execution, which is possible at
+        least on Python 2.4, where forking is not available for async tasks.
+    """
+    arunner = AsyncRunner(
+        job_id,
+        timeout_secs,
+        started_sender,
+        econtext,
+        kwargs
+    )
     arunner.run()
 
 
@@ -541,8 +625,8 @@ def exec_args(args, in_data='', chdir=None, shell=None, emulate_tty=False):
     stdout, stderr = proc.communicate(in_data)
 
     if emulate_tty:
-        stdout = stdout.replace(b'\n', b'\r\n')
-    return proc.returncode, stdout, stderr or ''
+        stdout = stdout.replace(b('\n'), b('\r\n'))
+    return proc.returncode, stdout, stderr or b('')
 
 
 def exec_command(cmd, in_data='', chdir=None, shell=None, emulate_tty=False):
@@ -574,7 +658,7 @@ def read_path(path):
     return open(path, 'rb').read()
 
 
-def set_fd_owner(fd, owner, group=None):
+def set_file_owner(path, owner, group=None, fd=None):
     if owner:
         uid = pwd.getpwnam(owner).pw_uid
     else:
@@ -585,7 +669,11 @@ def set_fd_owner(fd, owner, group=None):
     else:
         gid = os.getegid()
 
-    os.fchown(fd, (uid, gid))
+    if fd is not None and hasattr(os, 'fchown'):
+        os.fchown(fd, (uid, gid))
+    else:
+        # Python<2.6
+        os.chown(path, (uid, gid))
 
 
 def write_path(path, s, owner=None, group=None, mode=None,
@@ -603,9 +691,9 @@ def write_path(path, s, owner=None, group=None, mode=None,
     try:
         try:
             if mode:
-                os.fchmod(fp.fileno(), mode)
+                set_file_mode(tmp_path, mode, fd=fp.fileno())
             if owner or group:
-                set_fd_owner(fp.fileno(), owner, group)
+                set_file_owner(tmp_path, owner, group, fd=fp.fileno())
             fp.write(s)
         finally:
             fp.close()
@@ -645,14 +733,14 @@ def apply_mode_spec(spec, mode):
     Given a symbolic file mode change specification in the style of chmod(1)
     `spec`, apply changes in the specification to the numeric file mode `mode`.
     """
-    for clause in spec.split(','):
+    for clause in mitogen.core.to_text(spec).split(','):
         match = CHMOD_CLAUSE_PAT.match(clause)
         who, op, perms = match.groups()
         for ch in who or 'a':
             mask = CHMOD_MASKS[ch]
             bits = CHMOD_BITS[ch]
             cur_perm_bits = mode & mask
-            new_perm_bits = functools.reduce(operator.or_, (bits[p] for p in perms), 0)
+            new_perm_bits = reduce(operator.or_, (bits[p] for p in perms), 0)
             mode &= ~mask
             if op == '=':
                 mode |= new_perm_bits
@@ -663,15 +751,30 @@ def apply_mode_spec(spec, mode):
     return mode
 
 
-def set_file_mode(path, spec):
+def set_file_mode(path, spec, fd=None):
     """
     Update the permissions of a file using the same syntax as chmod(1).
     """
-    mode = os.stat(path).st_mode
-
-    if spec.isdigit():
+    if isinstance(spec, int):
+        new_mode = spec
+    elif not mitogen.core.PY3 and isinstance(spec, long):
+        new_mode = spec
+    elif spec.isdigit():
         new_mode = int(spec, 8)
     else:
+        mode = os.stat(path).st_mode
         new_mode = apply_mode_spec(spec, mode)
 
-    os.chmod(path, new_mode)
+    if fd is not None and hasattr(os, 'fchmod'):
+        os.fchmod(fd, new_mode)
+    else:
+        os.chmod(path, new_mode)
+
+
+def file_exists(path):
+    """
+    Return :data:`True` if `path` exists. This is a wrapper function over
+    :func:`os.path.exists`, since its implementation module varies across
+    Python versions.
+    """
+    return os.path.exists(path)

ansible_mitogen/transport_config.py

@@ -0,0 +1,593 @@
+# Copyright 2017, David Wilson
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are met:
+#
+# 1. Redistributions of source code must retain the above copyright notice,
+# this list of conditions and the following disclaimer.
+#
+# 2. Redistributions in binary form must reproduce the above copyright notice,
+# this list of conditions and the following disclaimer in the documentation
+# and/or other materials provided with the distribution.
+#
+# 3. Neither the name of the copyright holder nor the names of its contributors
+# may be used to endorse or promote products derived from this software without
+# specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+# POSSIBILITY OF SUCH DAMAGE.
+
+from __future__ import absolute_import
+from __future__ import unicode_literals
+
+"""
+Mitogen extends Ansible's target configuration mechanism in several ways that
+require some care:
+
+* Per-task configurables in Ansible like ansible_python_interpreter are
+  connection-layer configurables in Mitogen. They must be extracted during each
+  task execution to form the complete connection-layer configuration.
+
+* Mitogen has extra configurables not supported by Ansible at all, such as
+  mitogen_ssh_debug_level. These are extracted the same way as
+  ansible_python_interpreter.
+
+* Mitogen allows connections to be delegated to other machines. Ansible has no
+  internal framework for this, and so Mitogen must figure out a delegated
+  connection configuration all on its own. It cannot reuse much of the Ansible
+  machinery for building a connection configuration, as that machinery is
+  deeply spread out and hard-wired to expect Ansible's usual mode of operation.
+
+For normal and delegate_to connections, Ansible's PlayContext is reused where
+possible to maximize compatibility, but for proxy hops, configurations are
+built up using the HostVars magic class to call VariableManager.get_vars()
+behind the scenes on our behalf. Where Ansible has multiple sources of a
+configuration item, for example, ansible_ssh_extra_args, Mitogen must (ideally
+perfectly) reproduce how Ansible arrives at its value, without using mechanisms
+that are hard-wired or change across Ansible versions.
+
+That is what this file is for. It exports two spec classes, one that takes all
+information from PlayContext, and another that takes (almost) all information
+from HostVars.
+"""
+
+import abc
+import os
+import ansible.utils.shlex
+import ansible.constants as C
+
+from ansible.module_utils.six import with_metaclass
+
+
+import mitogen.core
+
+
+def parse_python_path(s):
+    """
+    Given the string set for ansible_python_interpeter, parse it using shell
+    syntax and return an appropriate argument vector.
+    """
+    if s:
+        return ansible.utils.shlex.shlex_split(s)
+
+
+def optional_secret(value):
+    """
+    Wrap `value` in :class:`mitogen.core.Secret` if it is not :data:`None`,
+    otherwise return :data:`None`.
+    """
+    if value is not None:
+        return mitogen.core.Secret(value)
+
+
+def first_true(it, default=None):
+    """
+    Return the first truthy element from `it`.
+    """
+    for elem in it:
+        if elem:
+            return elem
+    return default
+
+
+class Spec(with_metaclass(abc.ABCMeta, object)):
+    """
+    A source for variables that comprise a connection configuration.
+    """
+
+    @abc.abstractmethod
+    def transport(self):
+        """
+        The name of the Ansible plug-in implementing the connection.
+        """
+
+    @abc.abstractmethod
+    def inventory_name(self):
+        """
+        The name of the target being connected to as it appears in Ansible's
+        inventory.
+        """
+
+    @abc.abstractmethod
+    def remote_addr(self):
+        """
+        The network address of the target, or for container and other special
+        targets, some other unique identifier.
+        """
+
+    @abc.abstractmethod
+    def remote_user(self):
+        """
+        The username of the login account on the target.
+        """
+
+    @abc.abstractmethod
+    def password(self):
+        """
+        The password of the login account on the target.
+        """
+
+    @abc.abstractmethod
+    def become(self):
+        """
+        :data:`True` if privilege escalation should be active.
+        """
+
+    @abc.abstractmethod
+    def become_method(self):
+        """
+        The name of the Ansible become method to use.
+        """
+
+    @abc.abstractmethod
+    def become_user(self):
+        """
+        The username of the target account for become.
+        """
+
+    @abc.abstractmethod
+    def become_pass(self):
+        """
+        The password of the target account for become.
+        """
+
+    @abc.abstractmethod
+    def port(self):
+        """
+        The port of the login service on the target machine.
+        """
+
+    @abc.abstractmethod
+    def python_path(self):
+        """
+        Path to the Python interpreter on the target machine.
+        """
+
+    @abc.abstractmethod
+    def private_key_file(self):
+        """
+        Path to the SSH private key file to use to login.
+        """
+
+    @abc.abstractmethod
+    def ssh_executable(self):
+        """
+        Path to the SSH executable.
+        """
+
+    @abc.abstractmethod
+    def timeout(self):
+        """
+        The generic timeout for all connections.
+        """
+
+    @abc.abstractmethod
+    def ansible_ssh_timeout(self):
+        """
+        The SSH-specific timeout for a connection.
+        """
+
+    @abc.abstractmethod
+    def ssh_args(self):
+        """
+        The list of additional arguments that should be included in an SSH
+        invocation.
+        """
+
+    @abc.abstractmethod
+    def become_exe(self):
+        """
+        The path to the executable implementing the become method on the remote
+        machine.
+        """
+
+    @abc.abstractmethod
+    def sudo_args(self):
+        """
+        The list of additional arguments that should be included in a become
+        invocation.
+        """
+        # TODO: split out into sudo_args/become_args.
+
+    @abc.abstractmethod
+    def mitogen_via(self):
+        """
+        The value of the mitogen_via= variable for this connection. Indicates
+        the connection should be established via an intermediary.
+        """
+
+    @abc.abstractmethod
+    def mitogen_kind(self):
+        """
+        The type of container to use with the "setns" transport.
+        """
+
+    @abc.abstractmethod
+    def mitogen_docker_path(self):
+        """
+        The path to the "docker" program for the 'docker' transport.
+        """
+
+    @abc.abstractmethod
+    def mitogen_kubectl_path(self):
+        """
+        The path to the "kubectl" program for the 'docker' transport.
+        """
+
+    @abc.abstractmethod
+    def mitogen_lxc_path(self):
+        """
+        The path to the "lxc" program for the 'lxd' transport.
+        """
+
+    @abc.abstractmethod
+    def mitogen_lxc_attach_path(self):
+        """
+        The path to the "lxc-attach" program for the 'lxc' transport.
+        """
+
+    @abc.abstractmethod
+    def mitogen_lxc_info_path(self):
+        """
+        The path to the "lxc-info" program for the 'lxc' transport.
+        """
+
+    @abc.abstractmethod
+    def mitogen_machinectl_path(self):
+        """
+        The path to the "machinectl" program for the 'setns' transport.
+        """
+
+    @abc.abstractmethod
+    def mitogen_ssh_debug_level(self):
+        """
+        The SSH debug level.
+        """
+
+    @abc.abstractmethod
+    def mitogen_ssh_compression(self):
+        """
+        Whether SSH compression is enabled.
+        """
+
+    @abc.abstractmethod
+    def extra_args(self):
+        """
+        Connection-specific arguments.
+        """
+
+
+class PlayContextSpec(Spec):
+    """
+    PlayContextSpec takes almost all its information as-is from Ansible's
+    PlayContext. It is used for normal connections and delegate_to connections,
+    and should always be accurate.
+    """
+    def __init__(self, connection, play_context, transport, inventory_name):
+        self._connection = connection
+        self._play_context = play_context
+        self._transport = transport
+        self._inventory_name = inventory_name
+
+    def transport(self):
+        return self._transport
+
+    def inventory_name(self):
+        return self._inventory_name
+
+    def remote_addr(self):
+        return self._play_context.remote_addr
+
+    def remote_user(self):
+        return self._play_context.remote_user
+
+    def become(self):
+        return self._play_context.become
+
+    def become_method(self):
+        return self._play_context.become_method
+
+    def become_user(self):
+        return self._play_context.become_user
+
+    def become_pass(self):
+        return optional_secret(self._play_context.become_pass)
+
+    def password(self):
+        return optional_secret(self._play_context.password)
+
+    def port(self):
+        return self._play_context.port
+
+    def python_path(self):
+        return parse_python_path(
+            self._connection.get_task_var('ansible_python_interpreter')
+        )
+
+    def private_key_file(self):
+        return self._play_context.private_key_file
+
+    def ssh_executable(self):
+        return self._play_context.ssh_executable
+
+    def timeout(self):
+        return self._play_context.timeout
+
+    def ansible_ssh_timeout(self):
+        return (
+            self._connection.get_task_var('ansible_timeout') or
+            self._connection.get_task_var('ansible_ssh_timeout') or
+            self.timeout()
+        )
+
+    def ssh_args(self):
+        return [
+            mitogen.core.to_text(term)
+            for s in (
+                getattr(self._play_context, 'ssh_args', ''),
+                getattr(self._play_context, 'ssh_common_args', ''),
+                getattr(self._play_context, 'ssh_extra_args', '')
+            )
+            for term in ansible.utils.shlex.shlex_split(s or '')
+        ]
+
+    def become_exe(self):
+        return self._play_context.become_exe
+
+    def sudo_args(self):
+        return [
+            mitogen.core.to_text(term)
+            for term in ansible.utils.shlex.shlex_split(
+                first_true((
+                    self._play_context.become_flags,
+                    self._play_context.sudo_flags,
+                    # Ansible 2.3.
+                    getattr(C, 'DEFAULT_BECOME_FLAGS', ''),
+                    getattr(C, 'DEFAULT_SUDO_FLAGS', '')
+                ), default='')
+            )
+        ]
+
+    def mitogen_via(self):
+        return self._connection.get_task_var('mitogen_via')
+
+    def mitogen_kind(self):
+        return self._connection.get_task_var('mitogen_kind')
+
+    def mitogen_docker_path(self):
+        return self._connection.get_task_var('mitogen_docker_path')
+
+    def mitogen_kubectl_path(self):
+        return self._connection.get_task_var('mitogen_kubectl_path')
+
+    def mitogen_lxc_path(self):
+        return self._connection.get_task_var('mitogen_lxc_path')
+
+    def mitogen_lxc_attach_path(self):
+        return self._connection.get_task_var('mitogen_lxc_attach_path')
+
+    def mitogen_lxc_info_path(self):
+        return self._connection.get_task_var('mitogen_lxc_info_path')
+
+    def mitogen_machinectl_path(self):
+        return self._connection.get_task_var('mitogen_machinectl_path')
+
+    def mitogen_ssh_debug_level(self):
+        return self._connection.get_task_var('mitogen_ssh_debug_level')
+
+    def mitogen_ssh_compression(self):
+        return self._connection.get_task_var('mitogen_ssh_compression')
+
+    def extra_args(self):
+        return self._connection.get_extra_args()
+
+
+class MitogenViaSpec(Spec):
+    """
+    MitogenViaSpec takes most of its information from the HostVars of the
+    running task. HostVars is a lightweight wrapper around VariableManager, so
+    it is better to say that VariableManager.get_vars() is the ultimate source
+    of MitogenViaSpec's information.
+
+    Due to this, mitogen_via= hosts must have all their configuration
+    information represented as host and group variables. We cannot use any
+    per-task configuration, as all that data belongs to the real target host.
+
+    Ansible uses all kinds of strange historical logic for calculating
+    variables, including making their precedence configurable. MitogenViaSpec
+    must ultimately reimplement all of that logic. It is likely that if you are
+    having a configruation problem with connection delegation, the answer to
+    your problem lies in the method implementations below!
+    """
+    def __init__(self, inventory_name, host_vars,
+                 become_method, become_user):
+        self._inventory_name = inventory_name
+        self._host_vars = host_vars
+        self._become_method = become_method
+        self._become_user = become_user
+
+    def transport(self):
+        return (
+            self._host_vars.get('ansible_connection') or
+            C.DEFAULT_TRANSPORT
+        )
+
+    def inventory_name(self):
+        return self._inventory_name
+
+    def remote_addr(self):
+        return (
+            self._host_vars.get('ansible_host') or
+            self._inventory_name
+        )
+
+    def remote_user(self):
+        return (
+            self._host_vars.get('ansible_user') or
+            self._host_vars.get('ansible_ssh_user') or
+            C.DEFAULT_REMOTE_USER
+        )
+
+    def become(self):
+        return bool(self._become_user)
+
+    def become_method(self):
+        return self._become_method or C.DEFAULT_BECOME_METHOD
+
+    def become_user(self):
+        return self._become_user
+
+    def become_pass(self):
+        return optional_secret(
+            # TODO: Might have to come from PlayContext.
+            self._host_vars.get('ansible_become_password') or
+            self._host_vars.get('ansible_become_pass')
+        )
+
+    def password(self):
+        return optional_secret(
+            # TODO: Might have to come from PlayContext.
+            self._host_vars.get('ansible_ssh_pass') or
+            self._host_vars.get('ansible_password')
+        )
+
+    def port(self):
+        return (
+            self._host_vars.get('ansible_port') or
+            C.DEFAULT_REMOTE_PORT
+        )
+
+    def python_path(self):
+        return parse_python_path(
+            self._host_vars.get('ansible_python_interpreter')
+            # This variable has no default for remote hosts. For local hosts it
+            # is sys.executable.
+        )
+
+    def private_key_file(self):
+        # TODO: must come from PlayContext too.
+        return (
+            self._host_vars.get('ansible_ssh_private_key_file') or
+            self._host_vars.get('ansible_private_key_file') or
+            C.DEFAULT_PRIVATE_KEY_FILE
+        )
+
+    def ssh_executable(self):
+        return (
+            self._host_vars.get('ansible_ssh_executable') or
+            C.ANSIBLE_SSH_EXECUTABLE
+        )
+
+    def timeout(self):
+        # TODO: must come from PlayContext too.
+        return C.DEFAULT_TIMEOUT
+
+    def ansible_ssh_timeout(self):
+        return (
+            self._host_vars.get('ansible_timeout') or
+            self._host_vars.get('ansible_ssh_timeout') or
+            self.timeout()
+        )
+
+    def ssh_args(self):
+        return [
+            mitogen.core.to_text(term)
+            for s in (
+                (
+                    self._host_vars.get('ansible_ssh_args') or
+                    getattr(C, 'ANSIBLE_SSH_ARGS', None) or
+                    os.environ.get('ANSIBLE_SSH_ARGS')
+                    # TODO: ini entry. older versions.
+                ),
+                (
+                    self._host_vars.get('ansible_ssh_common_args') or
+                    os.environ.get('ANSIBLE_SSH_COMMON_ARGS')
+                    # TODO: ini entry.
+                ),
+                (
+                    self._host_vars.get('ansible_ssh_extra_args') or
+                    os.environ.get('ANSIBLE_SSH_EXTRA_ARGS')
+                    # TODO: ini entry.
+                ),
+            )
+            for term in ansible.utils.shlex.shlex_split(s)
+            if s
+        ]
+
+    def become_exe(self):
+        return (
+            self._host_vars.get('ansible_become_exe') or
+            C.DEFAULT_BECOME_EXE
+        )
+
+    def sudo_args(self):
+        return [
+            mitogen.core.to_text(term)
+            for s in (
+                self._host_vars.get('ansible_sudo_flags') or '',
+                self._host_vars.get('ansible_become_flags') or '',
+            )
+            for term in ansible.utils.shlex.shlex_split(s)
+        ]
+
+    def mitogen_via(self):
+        return self._host_vars.get('mitogen_via')
+
+    def mitogen_kind(self):
+        return self._host_vars.get('mitogen_kind')
+
+    def mitogen_docker_path(self):
+        return self._host_vars.get('mitogen_docker_path')
+
+    def mitogen_kubectl_path(self):
+        return self._host_vars.get('mitogen_kubectl_path')
+
+    def mitogen_lxc_path(self):
+        return self.host_vars.get('mitogen_lxc_path')
+
+    def mitogen_lxc_attach_path(self):
+        return self._host_vars.get('mitogen_lxc_attach_path')
+
+    def mitogen_lxc_info_path(self):
+        return self._host_vars.get('mitogen_lxc_info_path')
+
+    def mitogen_machinectl_path(self):
+        return self._host_vars.get('mitogen_machinectl_path')
+
+    def mitogen_ssh_debug_level(self):
+        return self._host_vars.get('mitogen_ssh_debug_level')
+
+    def mitogen_ssh_compression(self):
+        return self._host_vars.get('mitogen_ssh_compression')
+
+    def extra_args(self):
+        return []  # TODO

dev_requirements.txt

@@ -1,17 +1,11 @@
--r docs/docs-requirements.txt
-ansible==2.6.1
-coverage==4.5.1
-Django==1.6.11  # Last version supporting 2.6.
-mock==2.0.0
-pytz==2018.5
-paramiko==2.3.2  # Last 2.6-compat version.
-pytest-catchlog==1.2.2
-pytest==3.1.2
-PyYAML==3.11; python_version < '2.7'
-PyYAML==3.12; python_version >= '2.7'
-timeoutcontext==1.2.0
-unittest2==1.1.0
-# Fix InsecurePlatformWarning while creating py26 tox environment
-# https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
-urllib3[secure]; python_version < '2.7.9'
-google-api-python-client==1.6.5
+# This file is no longer used by CI jobs, it's mostly for interactive use.
+# Instead CI jobs grab the relevant sub-requirement.
+
+# mitogen_tests
+-r tests/requirements.txt
+
+# ansible_tests
+-r tests/ansible/requirements.txt
+
+# readthedocs
+-r docs/requirements.txt

docs/Makefile

@@ -2,7 +2,7 @@
 #
 
 default:
-	sphinx-autobuild -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
+	sphinx-build . build/html/
 
 # You can set these variables from the command line.
 SPHINXOPTS    =

docs/_static/style.css

@@ -12,6 +12,17 @@ div.body li {
 }
 
 
+/*
+ * Undo the hyphens: auto in Sphinx basic.css.
+ */
+div.body p, div.body dd, div.body li, div.body blockquote {
+    -moz-hyphens: inherit;
+    -ms-hyphens: inherit;
+    -webkit-hyphens: inherit;
+    hyphens: inherit;
+}
+
+
 
 /*
  * Setting :width; on an image causes Sphinx to turn the image into a link, so
@@ -27,6 +38,12 @@ div.body li {
     width: 150px;
 }
 
+.mitogen-right-200 {
+    float: right;
+    padding-left: 8px;
+    width: 200px;
+}
+
 .mitogen-right-225 {
     float: right;
     padding-left: 8px;
@@ -50,3 +67,10 @@ div.body li {
     padding-left: 8px;
     width: 350px;
 }
+
+.mitogen-logo-wrap {
+    shape-margin: 8px;
+    shape-outside: polygon(
+        100% 0, 50% 10%, 24% 24%, 0% 50%, 24% 75%, 50% 90%, 100% 100%
+    );
+}

docs/ansible.rst

@@ -1,19 +1,17 @@
 
-.. image:: images/ansible/ansible_mitogen.svg
-    :class: mitogen-right-225
-
-
 Mitogen for Ansible
 ===================
 
+.. image:: images/ansible/ansible_mitogen.svg
+    :class: mitogen-right-200 mitogen-logo-wrap
+
 An extension to `Ansible`_ is included that implements connections over
 Mitogen, replacing embedded shell invocations with pure-Python equivalents
 invoked via highly efficient remote procedure calls to persistent interpreters
 tunnelled over SSH. No changes are required to target hosts.
 
-The extension is approaching stability and real-world usage is encouraged. `Bug
-reports`_ are welcome: Ansible is huge, and only wide testing will ensure
-soundness.
+The extension is stable and real-world use is encouraged. `Bug reports`_ are
+welcome: Ansible is huge, and only wide testing will ensure soundness.
 
 .. _Ansible: https://www.ansible.com/
 
@@ -58,7 +56,7 @@ write files.
 Installation
 ------------
 
-1. Thoroughly review :ref:`noteworthy_differences` and :ref:`changelog`.
+1. Thoroughly review :ref:`noteworthy_differences` and :ref:`known_issues`.
 2. Download and extract |mitogen_url|.
 3. Modify ``ansible.cfg``:
 
@@ -70,8 +68,9 @@ Installation
 
    The ``strategy`` key is optional. If omitted, the
    ``ANSIBLE_STRATEGY=mitogen_linear`` environment variable can be set on a
-   per-run basis. Like ``mitogen_linear``, the ``mitogen_free`` strategy exists
-   to mimic the ``free`` strategy.
+   per-run basis. Like ``mitogen_linear``, the ``mitogen_free`` and
+   ``mitogen_host_pinned`` strategies exists to mimic the ``free`` and
+   ``host_pinned`` strategies.
 
 4. If targets have a restrictive ``sudoers`` file, add a rule like:
 
@@ -179,7 +178,7 @@ Noteworthy Differences
   practice, and light web searches failed to reveal many examples of them.
 
 * Ansible permits up to ``forks`` connections to be setup in parallel, whereas
-  in Mitogen this is handled by a fixed-size thread pool. Up to 16 connections
+  in Mitogen this is handled by a fixed-size thread pool. Up to 32 connections
   may be established in parallel by default, this can be modified by setting
   the ``MITOGEN_POOL_SIZE`` environment variable.
 
@@ -430,7 +429,7 @@ Temporary Files
 
 Temporary file handling in Ansible is tricky, and the precise behaviour varies
 across major versions. A variety of temporary files and directories are
-created, depending on the operating mode:
+created, depending on the operating mode.
 
 In the best case when pipelining is enabled and no temporary uploads are
 required, for each task Ansible will create one directory below a
@@ -769,10 +768,10 @@ Connect to classic LXC containers, like `lxc
 connection delegation is supported, and ``lxc-attach`` is always used rather
 than the LXC Python bindings, as is usual with ``lxc``.
 
-The ``lxc-attach`` command must be available on the host machine.
-
 * ``ansible_python_interpreter``
 * ``ansible_host``: Name of LXC container (default: inventory hostname).
+* ``mitogen_lxc_attach_path``: path to ``lxc-attach`` command if not available
+    on the system path.
 
 
 .. _method-lxd:
@@ -787,6 +786,8 @@ the host machine.
 
 * ``ansible_python_interpreter``
 * ``ansible_host``: Name of LXC container (default: inventory hostname).
+* ``mitogen_lxc_path``: path to ``lxc`` command if not available on the system
+  path.
 
 
 .. _machinectl:
@@ -899,6 +900,10 @@ except connection delegation is supported.
 * ``ssh_args``, ``ssh_common_args``, ``ssh_extra_args``
 * ``mitogen_ssh_debug_level``: integer between `0..3` indicating the SSH client
   debug level. Ansible must also be run with '-vvv' to view the output.
+* ``mitogen_ssh_compression``: :data:`True` to enable SSH compression,
+  otherwise :data:`False`. This will change to off by default in a future
+  release. If you are targetting many hosts on a fast network, please consider
+  disabling SSH compression.
 
 
 Debugging
@@ -919,6 +924,194 @@ logging is necessary. File-based logging can be enabled by setting
 enabled, one file per context will be created on the local machine and every
 target machine, as ``/tmp/mitogen.<pid>.log``.
 
+
+Common Problems
+~~~~~~~~~~~~~~~
+
+The most common bug reports fall into the following categories, so it is worth
+checking whether you can categorize a problem using the tools provided before
+reporting it:
+
+**Missed/Incorrect Configuration Variables**
+    In some cases Ansible may support a configuration variable that Mitogen
+    does not yet support, or Mitogen supports, but the support is broken. For
+    example, Mitogen may pick the wrong username or SSH parameters.
+
+    To detect this, use the special ``mitogen_get_stack`` action described
+    below to verify the settings Mitogen has chosen for the connection make
+    sense.
+
+**Process Environment Differences**
+    Mitogen's process model differs significantly to Ansible's in many places.
+    In the past, bugs have been reported because Ansible plug-ins modify an
+    environment variable after Mitogen processes are started.
+
+    If your task's failure may relate to the process environment in some way,
+    for example, ``SSH_AUTH_SOCK``, ``LC_ALL`` or ``PATH``, then an environment
+    difference may explain it. Environment differences are always considered
+    bugs in the extension, and are very easy to repair, so even if you find a
+    workaround, please report them to avoid someone else encountering the same
+    problem.
+
+**Variable Expansion Differences**
+    To avoid many classes of bugs, Mitogen avoids shell wherever possible.
+    Ansible however is traditionally built on shell, and it is often difficult
+    to tell just how many times a configuration parameter will pass through
+    shell expansion and quoting, and in what context before it is used.
+
+    Due to this, in some circumstances Mitogen may parse some expanded
+    variables differently, for example, in the wrong user account. Careful
+    review of ``-vvv`` and ``mitogen_ssh_debug_level`` logs can reveal this.
+    For example in the past, Mitogen used a different method of expanding
+    ``~/.ssh/id_rsa``, causing authentication to fail when ``ansible-playbook``
+    was run via ``sudo -E``.
+
+**External Tool Integration Differences**
+    Mitogen reimplements any aspect of Ansible that involves integrating with
+    SSH, sudo, Docker, or related tools. For this reason, sometimes its support
+    for those tools differs or is less mature than in Ansible.
+
+    In the past Mitogen has had bug reports due to failing to recognize a
+    particular variation of a login or password prompt on an exotic or
+    non-English operating system, or confusing a login banner for a password
+    prompt. Careful review of ``-vvv`` logs help identify these cases, as
+    Mitogen logs all strings it receives during connection, and how it
+    interprets them.
+
+
+.. _mitogen-get-stack:
+
+The `mitogen_get_stack` Action
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+When a Mitogen strategy is loaded, a special ``mitogen_get_stack`` action is
+available that returns a concise description of the connection configuration as
+extracted from Ansible and passed to the core library. Using it, you can learn
+whether a problem lies in the Ansible extension or deeper in library code.
+
+The action may be used in a playbook as ``mitogen_get_stack:`` just like a
+regular module, or directly from the command-line::
+
+    $ ANSIBLE_STRATEGY=mitogen_linear ansible -m mitogen_get_stack -b -k k3
+    SSH password:
+    k3 | SUCCESS => {
+        "changed": true,
+        "result": [
+            {
+                "kwargs": {
+                    "check_host_keys": "enforce",
+                    "connect_timeout": 10,
+                    "hostname": "k3",
+                    "identities_only": false,
+                    "identity_file": null,
+                    "password": "mysecretpassword",
+                    "port": null,
+                    "python_path": null,
+                    "ssh_args": [
+                        "-C",
+                        "-o",
+                        "ControlMaster=auto",
+                        "-o",
+                        "ControlPersist=60s"
+                    ],
+                    "ssh_debug_level": null,
+                    "ssh_path": "ssh",
+                    "username": null
+                },
+                "method": "ssh"
+            },
+            {
+                "enable_lru": true,
+                "kwargs": {
+                    "connect_timeout": 10,
+                    "password": null,
+                    "python_path": null,
+                    "sudo_args": [
+                        "-H",
+                        "-S",
+                        "-n"
+                    ],
+                    "sudo_path": null,
+                    "username": "root"
+                },
+                "method": "sudo"
+            }
+        ]
+    }
+
+Each object in the list represents a single 'hop' in the connection, from
+nearest to furthest. Unlike in Ansible, the core library treats ``become``
+steps and SSH steps identically, so they are represented distinctly in the
+output.
+
+The presence of ``null`` means no explicit value was extracted from Ansible,
+and either the Mitogen library or SSH will choose a value for the parameter. In
+the example above, Mitogen will choose ``/usr/bin/python`` for ``python_path``,
+and SSH will choose ``22`` for ``port``, or whatever ``Port`` it parses from
+``~/.ssh/config``. Note the presence of ``null`` may indicate the extension
+failed to extract the correct value.
+
+When using ``mitogen_get_stack`` to diagnose a problem, pay special attention
+to ensuring the invocation exactly matches the problematic task. For example,
+if the failing task has ``delegate_to:`` or ``become:`` enabled, the
+``mitogen_get_stack`` invocation must include those statements in order for the
+output to be accurate.
+
+If a playbook cannot start at all, you may need to temporarily use
+``gather_facts: no`` to allow the first task to proceed. This action does not
+create connections, so if it is the first task, it is still possible to review
+its output.
+
+
+The `mitogen_ssh_debug_level` Variable
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Mitogen has support for capturing SSH diagnostic logs, and integrating them
+into the regular debug log output produced when ``-vvv`` is active. This
+provides a single audit trail of every component active during SSH
+authentication.
+
+Particularly for authentication failures, setting this variable to 3, in
+combination with ``-vvv``, allows review of every parameter passed to SSH, and
+review of every action SSH attempted during authentication.
+
+For example, this method can be used to ascertain whether SSH attempted agent
+authentication, or what private key files it was able to access and which it tried.
+
+
+Post-authentication Bootstrap Failure
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If logging indicates Mitogen was able to authenticate, but some error occurred
+after authentication preventing the Python bootstrap from completing, it can be
+immensely useful to temporarily replace ``ansible_python_interpreter`` with a
+wrapper that runs Python under ``strace``::
+
+    $ ssh badbox
+
+    badbox$ cat > strace-python.sh
+    #!/bin/sh
+    strace -o /tmp/strace-python.$$ -ff -s 100 python "$@"
+    ^D
+
+    badbox$ chmod +x strace-python.sh
+    badbox$ logout
+
+    $ ansible-playbook site.yml \
+        -e ansible_python_interpreter=./strace-python.sh \
+        -l badbox
+
+This will produce a potentially large number of log files under ``/tmp/``. The
+lowest-numbered traced PID is generally the main Python interpreter. The most
+intricate bootstrap steps happen there, any error should be visible near the
+end of the trace.
+
+It is also possible the first stage bootstrap failed. That is usually the next
+lowest-numbered PID and tends to be the smallest file. Even if you can't
+ascertain the problem with your configuration from these logs, including them
+in a bug report can save days of detective effort.
+
+
 .. _diagnosing-hangs:
 
 Diagnosing Hangs
@@ -944,6 +1137,25 @@ cases `faulthandler <https://faulthandler.readthedocs.io/>`_ may be used:
    of the stacks, along with a description of the last task executing prior to
    the hang.
 
+It is possible the hang occurred in a process on a target. If ``strace`` is
+available, look for the host name not listed in Ansible output as reporting a
+result for the most recent task, log into it, and use ``strace -ff -p <pid>``
+on each process whose name begins with ``mitogen:``::
+
+    $ strace -ff -p 29858
+    strace: Process 29858 attached with 3 threads
+    [pid 29864] futex(0x55ea9be52f60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff <unfinished ...>
+    [pid 29860] restart_syscall(<... resuming interrupted poll ...> <unfinished ...>
+    [pid 29858] futex(0x55ea9be52f60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff
+    ^C
+
+    $ 
+
+This shows one thread waiting on IO (``poll``) and two more waiting on the same
+lock. It is taken from a real example of a deadlock due to a forking bug.
+Please include any such information for all processes that you are able to
+collect in any bug report.
+
 
 Getting Help
 ~~~~~~~~~~~~
@@ -955,35 +1167,103 @@ FreeNode IRC network.
 Sample Profiles
 ---------------
 
-Local VM connection
-~~~~~~~~~~~~~~~~~~~
+The summaries below may be reproduced using data and scripts maintained in the
+`pcaps branch <https://github.com/dw/mitogen/tree/pcaps/>`_. Traces were
+recorded using Ansible 2.5.14.
+
+
+Trivial Loop: Local Host
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+This demonstrates Mitogen vs. SSH pipelining to the local machine running
+`bench/loop-100-items.yml
+<https://github.com/dw/mitogen/blob/master/tests/ansible/bench/loop-100-items.yml>`_,
+executing a simple command 100 times. Most Ansible controller overhead is
+isolated, characterizing just module executor and connection layer performance.
+Mitogen requires **63x less bandwidth and 5.9x less time**.
+
+.. image:: images/ansible/pcaps/loop-100-items-local.svg
+
+Unlike in SSH pipelining where payloads are sent as a single compressed block,
+by default Mitogen enables SSH compression for its uncompressed RPC data. In
+many-host scenarios it may be desirable to disable compression. This has
+negligible impact on footprint, since program code is separately compressed and
+sent only once. Compression also benefits SSH pipelining, but the presence of
+large precompressed per-task payloads may present a more significant CPU burden
+during many-host runs.
+
+.. image:: images/ansible/pcaps/loop-100-items-local-detail.svg
 
-This demonstrates Mitogen vs. connection pipelining to a local VM executing
-``bench/loop-100-items.yml``, which simply executes ``hostname`` 100 times.
-Mitogen requires **43x less bandwidth and 6.5x less time**.
+In a detailed trace, improved interaction with the host machine is visible. In
+this playbook because no forks were required to start SSH clients from the
+worker process executing the loop, the worker's memory was never marked
+read-only, thus avoiding a major hidden performance problem - the page fault
+rate is more than halved.
+
+
+File Transfer: UK to France
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+`This playbook
+<https://github.com/dw/mitogen/blob/master/tests/ansible/regression/issue_140__thread_pileup.yml>`_
+was used to compare file transfer performance over a ~26 ms link. It uses the
+``with_filetree`` loop syntax to copy a directory of 1,000 0-byte files to the
+target.
+
+.. raw:: html
+
+    <style>
+        .nojunk td,
+        .nojunk th { padding: 4px; font-size: 90%; text-align: right !important; }
+
+        table.docutils col {
+            width: auto !important;
+        }
+    </style>
+
+.. csv-table::
+    :header: , Secs, CPU Secs, Sent, Received, Roundtrips
+    :class: nojunk
+    :align: right
+
+    Mitogen, 98.54, 43.04, "815 KiB", "447 KiB", 3.79
+    SSH Pipelining, "1,483.54", 329.37, "99,539 KiB", "6,870 KiB", 57.01
+
+*Roundtrips* is the approximate number of network roundtrips required to
+describe the runtime that was consumed. Due to Mitogen's built-in file transfer
+support, continuous reinitialization of an external `scp`/`sftp` client is
+avoided, permitting large ``with_filetree`` copies to become practical without
+any special casing within the playbook or the Ansible implementation.
+
+
+DebOps: UK to India
+~~~~~~~~~~~~~~~~~~~
 
-.. image:: images/ansible/run_hostname_100_times_mito.svg
-.. image:: images/ansible/run_hostname_100_times_plain.svg
+This is an all-green run of 246 tasks from the `DebOps
+<https://docs.debops.org/en/master/>`_ 0.7.2 `common.yml
+<https://github.com/debops/debops-playbooks/blob/master/playbooks/common.yml>`_
+playbook over a ~370 ms link between the UK and India. The playbook touches a
+wide variety of modules, many featuring unavoidable waits for slow computation
+on the target.
 
+More tasks of a wider variety are featured than previously, placing strain on
+Mitogen's module loading and in-memory caching. By running over a long-distance
+connection, it highlights behaviour of the connection layer in the presence of
+high latency.
 
-Kathmandu to Paris
-~~~~~~~~~~~~~~~~~~
+Mitogen requires **14.5x less bandwidth and 4x less time**.
 
-This is a full Django application playbook over a ~180ms link between Kathmandu
-and Paris. Aside from large pauses where the host performs useful work, the
-high latency of this link means Mitogen only manages a 1.7x speedup.
+.. image:: images/ansible/pcaps/debops-uk-india.svg
 
-Many early roundtrips are due to inefficiencies in Mitogen's importer that will
-be fixed over time, however the majority, comprising at least 10 seconds, are
-due to idling while the host's previous result and next command are in-flight
-on the network.
 
-The initial extension lays groundwork for exciting structural changes to the
-execution model: a future version will tackle latency head-on by delegating
-some control flow to the target host, melding the performance and scalability
-benefits of pull-based operation with the management simplicity of push-based
-operation.
+Django App: UK to India
+~~~~~~~~~~~~~~~~~~~~~~~
 
-.. image:: images/ansible/costapp.png
+This short playbook features only 23 steps executed over the same ~370 ms link
+as previously, with many steps running unavoidably expensive tasks like
+building C++ code, and compiling static web site assets.
 
+Despite the small margin for optimization, Mitogen still manages **6.2x less
+bandwidth and 1.8x less time**.
 
+.. image:: images/ansible/pcaps/costapp-uk-india.svg

docs/changelog.rst

@@ -15,6 +15,528 @@ Release Notes
     </style>
 
 
+.. _known_issues:
+
+Known Issues
+------------
+
+Mitogen For Ansible
+~~~~~~~~~~~~~~~~~~~
+
+* The Ansible 2.7 `reboot
+  <https://docs.ansible.com/ansible/latest/modules/reboot_module.html>`_ module
+  may require a ``pre_reboot_delay`` on systemd hosts, as insufficient time
+  exists for the reboot command's exit status to be reported before necessary
+  processes are torn down.
+
+* On OS X when a SSH password is specified and the default connection type of
+  ``smart`` is used, Ansible may select the Paramiko plug-in rather than
+  Mitogen. If you specify a password on OS X, ensure ``connection: ssh``
+  appears in your playbook, ``ansible.cfg``, or as ``-c ssh`` on the
+  command-line.
+
+* The ``raw`` action executes as a regular Mitogen connection, which requires
+  Python on the target, precluding its use for installing Python. This will be
+  addressed in a future 0.2 release. For now, simply mix Mitogen and vanilla
+  Ansible strategies in your playbook:
+
+  .. code-block:: yaml
+
+    - hosts: web-servers
+      strategy: linear
+      tasks:
+      - name: Install Python if necessary.
+        raw: test -e /usr/bin/python || apt install -y python-minimal
+
+    - hosts: web-servers
+      strategy: mitogen_linear
+      roles:
+      - nginx
+      - initech_app
+      - y2k_fix
+
+.. * When running with ``-vvv``, log messages will be printed to the console
+     *after* the Ansible run completes, as connection multiplexer shutdown only
+     begins after Ansible exits. This is due to a lack of suitable shutdown hook
+     in Ansible, and is fairly harmless, albeit cosmetically annoying. A future
+     release may include a solution.
+
+.. * Configurations will break that rely on the `hashbang argument splitting
+     behaviour <https://github.com/ansible/ansible/issues/15635>`_ of the
+     ``ansible_python_interpreter`` setting, contrary to the Ansible
+     documentation. This will be addressed in a future 0.2 release.
+
+* Performance does not scale linearly with target count. This requires
+  significant additional work, as major bottlenecks exist in the surrounding
+  Ansible code. Performance-related bug reports for any scenario remain
+  welcome with open arms.
+
+* Performance on Python 3 is significantly worse than on Python 2. While this
+  has not yet been investigated, at least some of the regression appears to be
+  part of the core library, and should therefore be straightforward to fix as
+  part of 0.2.x.
+
+* *Module Replacer* style Ansible modules are not supported.
+
+* Actions are single-threaded for each `(host, user account)` combination,
+  including actions that execute on the local machine. Playbooks may experience
+  slowdown compared to vanilla Ansible if they employ long-running
+  ``local_action`` or ``delegate_to`` tasks delegating many target hosts to a
+  single machine and user account.
+
+* Connection Delegation remains in preview and has bugs around how it infers
+  connections. Connection establishment will remain single-threaded for the 0.2
+  series, however connection inference bugs will be addressed in a future 0.2
+  release.
+
+* Connection Delegation does not support automatic tunnelling of SSH-dependent
+  actions, such as the ``synchronize`` module. This will be addressed in the
+  0.3 series.
+
+
+Core Library
+~~~~~~~~~~~~
+
+* Serialization is still based on :mod:`pickle`. While there is high confidence
+  remote code execution is impossible in Mitogen's configuration, an untrusted
+  context may at least trigger disproportionately high memory usage injecting
+  small messages (*"billion laughs attack"*). Replacement is an important
+  future priority, but not critical for an initial release.
+
+* Child processes are not reliably reaped, leading to a pileup of zombie
+  processes when a program makes many short-lived connections in a single
+  invocation. This does not impact Mitogen for Ansible, however it limits the
+  usefulness of the core library. A future 0.2 release will address it.
+
+* Some races remain around :class:`mitogen.core.Broker <Broker>` destruction,
+  disconnection and corresponding file descriptor closure. These are only
+  problematic in situations where child process reaping is also problematic.
+
+* The `fakessh` component does not shut down correctly and requires flow
+  control added to the design. While minimal fixes are possible, due to the
+  absence of flow control the original design is functionally incomplete.
+
+* The multi-threaded :ref:`service` remains in a state of design flux and
+  should be considered obsolete, despite heavy use in Mitogen for Ansible. A
+  future replacement may be integrated more tightly with, or entirely replace
+  the RPC dispatcher on the main thread.
+
+* Documentation is in a state of disrepair. This will be improved over the 0.2
+  series.
+
+
+v0.2.4 (2018-??-??)
+-------------------
+
+Mitogen for Ansible
+~~~~~~~~~~~~~~~~~~~
+
+This release includes a huge variety of important fixes and new optimizations.
+It is 35% faster than 0.2.3 on a synthetic 64 target run that places heavy load
+on the connection multiplexer.
+
+Enhancements
+^^^^^^^^^^^^
+
+* `#76 <https://github.com/dw/mitogen/issues/76>`_,
+  `#351 <https://github.com/dw/mitogen/issues/351>`_,
+  `#352 <https://github.com/dw/mitogen/issues/352>`_: disconnect propagation
+  has improved, allowing Ansible to cancel waits for responses from abruptly
+  disconnected targets. This ensures a task will reliably fail rather than
+  hang, for example on network failure or EC2 instance maintenance.
+
+* `#369 <https://github.com/dw/mitogen/issues/369>`_,
+  `#407 <https://github.com/dw/mitogen/issues/407>`_: :meth:`Connection.reset`
+  is implemented, allowing `meta: reset_connection
+  <https://docs.ansible.com/ansible/latest/modules/meta_module.html>`_ to shut
+  down the remote interpreter as documented, and improving support for the
+  `reboot
+  <https://docs.ansible.com/ansible/latest/modules/reboot_module.html>`_
+  module.
+
+* `09aa27a6 <https://github.com/dw/mitogen/commit/09aa27a6>`_: the
+  ``mitogen_host_pinned`` strategy wraps the ``host_pinned`` strategy
+  introduced in Ansible 2.7.
+
+* `#477 <https://github.com/dw/mitogen/issues/477>`_: Python 2.4 is fully
+  supported by the core library and tested automatically, in any parent/child
+  combination of 2.4, 2.6, 2.7 and 3.6 interpreters.
+
+* `#477 <https://github.com/dw/mitogen/issues/477>`_: Ansible 2.3 is fully
+  supported and tested automatically. In combination with the core library
+  Python 2.4 support, this allows Red Hat Enterprise Linux 5 targets to be
+  managed with Mitogen. The ``simplejson`` package need not be installed on
+  such targets, as is usually required by Ansible.
+
+* `#412 <https://github.com/dw/mitogen/issues/412>`_: to simplify diagnosing
+  connection configuration problems, Mitogen ships a ``mitogen_get_stack``
+  action that is automatically added to the action plug-in path. See
+  :ref:`mitogen-get-stack` for more information.
+
+* `152effc2 <https://github.com/dw/mitogen/commit/152effc2>`_,
+  `bd4b04ae <https://github.com/dw/mitogen/commit/bd4b04ae>`_: a CPU affinity
+  policy was added for Linux controllers, reducing latency and SMP overhead on
+  hot paths exercised for every task. This yielded a 19% speedup in a 64-target
+  job composed of many short tasks, and should easily be visible as a runtime
+  improvement in many-host runs.
+
+* `2b44d598 <https://github.com/dw/mitogen/commit/2b44d598>`_: work around a
+  defective caching mechanism by pre-heating it before spawning workers. This
+  saves 40% runtime on a synthetic repetitive task.
+
+* `0979422a <https://github.com/dw/mitogen/commit/0979422a>`_: an expensive
+  dependency scanning step was redundantly invoked for every task,
+  bottlenecking the connection multiplexer.
+
+* `eaa990a97 <https://github.com/dw/mitogen/commit/eaa990a97>`_: a new
+  ``mitogen_ssh_compression`` variable is supported, allowing Mitogen's default
+  SSH compression to be disabled. SSH compression is a large contributor to CPU
+  usage in many-target runs, and severely limits file transfer. On a `"shell:
+  hostname"` task repeated 500 times, Mitogen requires around 800 bytes per
+  task with compression, rising to 3 KiB without. File transfer throughput
+  rises from ~25MiB/s when enabled to ~200MiB/s when disabled.
+
+* `#260 <https://github.com/dw/mitogen/issues/260>`_,
+  `a18a083c <https://github.com/dw/mitogen/commit/a18a083c>`_: brokers no
+  longer wait for readiness indication to transmit, and instead assume
+  transmission will succeed. As this is usually true, one loop iteration and
+  two poller reconfigurations are avoided, yielding a significant reduction in
+  interprocess round-trip latency.
+
+* `#415 <https://github.com/dw/mitogen/issues/415>`_,
+  `#491 <https://github.com/dw/mitogen/issues/491>`_,
+  `#493 <https://github.com/dw/mitogen/issues/493>`_: the interface employed
+  for in-process queues changed from `kqueue
+  <https://www.freebsd.org/cgi/man.cgi?query=kqueue&sektion=2>`_ / `epoll
+  <http://man7.org/linux/man-pages/man7/epoll.7.html>`_ to `poll()
+  <http://man7.org/linux/man-pages/man2/poll.2.html>`_, which requires no setup
+  or teardown, yielding a 38% latency reduction for inter-thread communication.
+
+
+Fixes
+^^^^^
+
+* `#251 <https://github.com/dw/mitogen/issues/251>`_,
+  `#359 <https://github.com/dw/mitogen/issues/359>`_,
+  `#396 <https://github.com/dw/mitogen/issues/396>`_,
+  `#401 <https://github.com/dw/mitogen/issues/401>`_,
+  `#404 <https://github.com/dw/mitogen/issues/404>`_,
+  `#412 <https://github.com/dw/mitogen/issues/412>`_,
+  `#434 <https://github.com/dw/mitogen/issues/434>`_,
+  `#436 <https://github.com/dw/mitogen/issues/436>`_,
+  `#465 <https://github.com/dw/mitogen/issues/465>`_: connection delegation and
+  ``delegate_to:`` handling suffered a major regression in 0.2.3. The 0.2.2
+  behaviour has been restored, and further work has been made to improve the
+  compatibility of connection delegation's configuration building methods.
+
+* `#323 <https://github.com/dw/mitogen/issues/323>`_,
+  `#333 <https://github.com/dw/mitogen/issues/333>`_: work around a Windows
+  Subsystem for Linux bug that caused tracebacks to appear during shutdown.
+
+* `#334 <https://github.com/dw/mitogen/issues/334>`_: the SSH method
+  tilde-expands private key paths using Ansible's logic. Previously the path
+  was passed unmodified to SSH, which expanded it using :func:`pwd.getpwnam`.
+  This differs from :func:`os.path.expanduser`, which uses the ``HOME``
+  environment variable if it is set, causing behaviour to diverge when Ansible
+  was invoked across user accounts via ``sudo``.
+
+* `#364 <https://github.com/dw/mitogen/issues/364>`_: file transfers from
+  controllers running Python 2.7.2 or earlier could be interrupted due to a
+  forking bug in the :mod:`tempfile` module.
+
+* `#370 <https://github.com/dw/mitogen/issues/370>`_: the Ansible
+  `reboot <https://docs.ansible.com/ansible/latest/modules/reboot_module.html>`_
+  module is supported.
+
+* `#373 <https://github.com/dw/mitogen/issues/373>`_: the LXC and LXD methods
+  print a useful hint on failure, as no useful error is normally logged to the
+  console by these tools.
+
+* `#374 <https://github.com/dw/mitogen/issues/374>`_,
+  `#391 <https://github.com/dw/mitogen/issues/391>`_: file transfer and module
+  execution from 2.x controllers to 3.x targets was broken due to a regression
+  caused by refactoring, and compounded by `#426
+  <https://github.com/dw/mitogen/issues/426>`_.
+
+* `#400 <https://github.com/dw/mitogen/issues/400>`_: work around a threading
+  bug in the AWX display callback when running with high verbosity setting.
+
+* `#409 <https://github.com/dw/mitogen/issues/409>`_: the setns method was
+  silently broken due to missing tests. Basic coverage was added to prevent a
+  recurrence.
+
+* `#409 <https://github.com/dw/mitogen/issues/409>`_: the LXC and LXD methods
+  support ``mitogen_lxc_path`` and ``mitogen_lxc_attach_path`` variables to
+  control the location of third pary utilities.
+
+* `#410 <https://github.com/dw/mitogen/issues/410>`_: the sudo method supports
+  the SELinux ``--type`` and ``--role`` options.
+
+* `#420 <https://github.com/dw/mitogen/issues/420>`_: if a :class:`Connection`
+  was constructed in the Ansible top-level process, for example while executing
+  ``meta: reset_connection``, resources could become undesirably shared in
+  subsequent children.
+
+* `#426 <https://github.com/dw/mitogen/issues/426>`_: an oversight while
+  porting to Python 3 meant no automated 2->3 tests were running. A significant
+  number of 2->3 bugs were fixed, mostly in the form of Unicode/bytes
+  mismatches.
+
+* `#429 <https://github.com/dw/mitogen/issues/429>`_: the ``sudo`` method can
+  now recognize internationalized password prompts.
+
+* `#362 <https://github.com/dw/mitogen/issues/362>`_,
+  `#435 <https://github.com/dw/mitogen/issues/435>`_: the previous fix for slow
+  Python 2.x subprocess creation on Red Hat caused newly spawned children to
+  have a reduced open files limit. A more intrusive fix has been added to
+  directly address the problem without modifying the subprocess environment.
+
+* `#397 <https://github.com/dw/mitogen/issues/397>`_,
+  `#454 <https://github.com/dw/mitogen/issues/454>`_: the previous approach to
+  handling modern Ansible temporary file cleanup was too aggressive, and could
+  trigger early finalization of Cython-based extension modules, leading to
+  segmentation faults.
+
+* `#499 <https://github.com/dw/mitogen/issues/499>`_: the ``allow_same_user``
+  Ansible configuration setting is respected.
+
+* `#527 <https://github.com/dw/mitogen/issues/527>`_: crashes in modules are
+  trapped and reported in a manner that matches Ansible. In particular, a
+  module crash no longer leads to an exception that may crash the corresponding
+  action plug-in.
+
+* `dc1d4251 <https://github.com/dw/mitogen/commit/dc1d4251>`_: the
+  ``synchronize`` module could fail with the Docker transport due to a missing
+  attribute.
+
+* `599da068 <https://github.com/dw/mitogen/commit/599da068>`_: fix a race
+  when starting async tasks, where it was possible for the controller to
+  observe no status file on disk before the task had a chance to write one.
+
+* `2c7af9f04 <https://github.com/dw/mitogen/commit/2c7af9f04>`_: Ansible
+  modules were repeatedly re-transferred. The bug was hidden by the previously
+  mandatorily enabled SSH compression.
+
+
+Core Library
+~~~~~~~~~~~~
+
+* `#76 <https://github.com/dw/mitogen/issues/76>`_: routing records the
+  destination context IDs ever received on each stream, and when disconnection
+  occurs, propagates :data:`mitogen.core.DEL_ROUTE` messages towards every
+  stream that ever communicated with the disappearing peer, rather than simply
+  towards parents. Conversations between nodes anywhere in the tree receive
+  :data:`mitogen.core.DEL_ROUTE` when either participant disconnects, allowing
+  receivers to wake with :class:`mitogen.core.ChannelError`, even when one
+  participant is not a parent of the other.
+
+* `#109 <https://github.com/dw/mitogen/issues/109>`_,
+  `57504ba6 <https://github.com/dw/mitogen/commit/57504ba6>`_: newer Python 3
+  releases explicitly populate :data:`sys.meta_path` with importer internals,
+  causing Mitogen to install itself at the end of the importer chain rather
+  than the front.
+
+* `#310 <https://github.com/dw/mitogen/issues/310>`_: support has returned for
+  trying to figure out the real source of non-module objects installed in
+  :data:`sys.modules`, so they can be imported. This is needed to handle syntax
+  sugar used by packages like :mod:`plumbum`.
+
+* `#349 <https://github.com/dw/mitogen/issues/349>`_: an incorrect format
+  string could cause large stack traces when attempting to import built-in
+  modules on Python 3.
+
+* `#387 <https://github.com/dw/mitogen/issues/387>`_,
+  `#413 <https://github.com/dw/mitogen/issues/413>`_: dead messages include an
+  optional reason in their body. This is used to cause
+  :class:`mitogen.core.ChannelError` to report far more useful diagnostics at
+  the point the error occurs that previously would have been buried in debug
+  log output from an unrelated context.
+
+* `#408 <https://github.com/dw/mitogen/issues/408>`_: a variety of fixes were
+  made to restore Python 2.4 compatibility.
+
+* `#399 <https://github.com/dw/mitogen/issues/399>`_,
+  `#437 <https://github.com/dw/mitogen/issues/437>`_: ignore a
+  :class:`DeprecationWarning` to avoid failure of the ``su`` method on Python
+  3.7.
+
+* `#405 <https://github.com/dw/mitogen/issues/405>`_: if an oversized message
+  is rejected, and it has a ``reply_to`` set, a dead message is returned to the
+  sender. This ensures function calls exceeding the configured maximum size
+  crash rather than hang.
+
+* `#406 <https://github.com/dw/mitogen/issues/406>`_:
+  :class:`mitogen.core.Broker` did not call :meth:`mitogen.core.Poller.close`
+  during shutdown, leaking the underlying poller FD in masters and parents.
+
+* `#406 <https://github.com/dw/mitogen/issues/406>`_: connections could leak
+  FDs when a child process failed to start.
+
+* `#288 <https://github.com/dw/mitogen/issues/288>`_,
+  `#406 <https://github.com/dw/mitogen/issues/406>`_,
+  `#417 <https://github.com/dw/mitogen/issues/417>`_: connections could leave
+  FD wrapper objects that had not been closed lying around to be closed during
+  garbage collection, causing reused FD numbers to be closed at random moments.
+
+* `#411 <https://github.com/dw/mitogen/issues/411>`_: the SSH method typed
+  "``y``" rather than the requisite "``yes``" when `check_host_keys="accept"`
+  was configured. This would lead to connection timeouts due to the hung
+  response.
+
+* `#414 <https://github.com/dw/mitogen/issues/414>`_,
+  `#425 <https://github.com/dw/mitogen/issues/425>`_: avoid deadlock of forked
+  children by reinitializing the :mod:`mitogen.service` pool lock.
+
+* `#416 <https://github.com/dw/mitogen/issues/416>`_: around 1.4KiB of memory
+  was leaked on every RPC, due to a list of strong references keeping alive any
+  handler ever registered for disconnect notification.
+
+* `#418 <https://github.com/dw/mitogen/issues/418>`_: the
+  :func:`mitogen.parent.iter_read` helper would leak poller FDs, because
+  execution of its :keyword:`finally` block was delayed on Python 3. Now
+  callers explicitly close the generator when finished.
+
+* `#422 <https://github.com/dw/mitogen/issues/422>`_: the fork method could
+  fail to start if :data:`sys.stdout` was opened in block buffered mode, and
+  buffered data was pending in the parent prior to fork.
+
+* `#438 <https://github.com/dw/mitogen/issues/438>`_: a descriptive error is
+  logged when stream corruption is detected.
+
+* `#439 <https://github.com/dw/mitogen/issues/439>`_: descriptive errors are
+  raised when attempting to invoke unsupported function types.
+
+* `#444 <https://github.com/dw/mitogen/issues/444>`_: messages regarding
+  unforwardable extension module are no longer logged as errors.
+
+* `#445 <https://github.com/dw/mitogen/issues/445>`_: service pools unregister
+  the :data:`mitogen.core.CALL_SERVICE` handle at shutdown, ensuring any
+  outstanding messages are either processed by the pool as it shuts down, or
+  have dead messages sent in reply to them, preventing peer contexts from
+  hanging due to a forgotten buffered message.
+
+* `#446 <https://github.com/dw/mitogen/issues/446>`_: given thread A calling
+  :meth:`mitogen.core.Receiver.close`, and thread B, C, and D sleeping in
+  :meth:`mitogen.core.Receiver.get`, previously only one sleeping thread would
+  be woken with :class:`mitogen.core.ChannelError` when the receiver was
+  closed. Now all threads are woken per the docstring.
+
+* `#447 <https://github.com/dw/mitogen/issues/447>`_: duplicate attempts to
+  invoke :meth:`mitogen.core.Router.add_handler` cause an error to be raised,
+  ensuring accidental re-registration of service pools are reported correctly.
+
+* `#448 <https://github.com/dw/mitogen/issues/448>`_: the import hook
+  implementation now raises :class:`ModuleNotFoundError` instead of
+  :class:`ImportError` in Python 3.6 and above, to cope with an upcoming
+  version of the :mod:`subprocess` module requiring this new subclass to be
+  raised.
+
+* `#453 <https://github.com/dw/mitogen/issues/453>`_: the loggers used in
+  children for standard IO redirection have propagation disabled, preventing
+  accidental reconfiguration of the :mod:`logging` package in a child from
+  setting up a feedback loop.
+
+* `#456 <https://github.com/dw/mitogen/issues/456>`_: a descriptive error is
+  logged when :meth:`mitogen.core.Broker.defer` is called after the broker has
+  shut down, preventing new messages being enqueued that will never be sent,
+  and subsequently producing a program hang.
+
+* `#459 <https://github.com/dw/mitogen/issues/459>`_: the beginnings of a
+  :meth:`mitogen.master.Router.get_stats` call has been added. The initial
+  statistics cover the module loader only.
+
+* `#462 <https://github.com/dw/mitogen/issues/462>`_: Mitogen could fail to
+  open a PTY on broken Linux systems due to a bad interaction between the glibc
+  :func:`grantpt` function and an incorrectly mounted ``/dev/pts`` filesystem.
+  Since correct group ownership is not required in most scenarios, when this
+  problem is detected, the PTY is allocated and opened directly by the library.
+
+* `#479 <https://github.com/dw/mitogen/issues/479>`_: Mitogen could fail to
+  import :mod:`__main__` on Python 3.4 and newer due to a breaking change in
+  the :mod:`pkgutil` API. The program's main script is now handled specially.
+
+* `#481 <https://github.com/dw/mitogen/issues/481>`_: the version of `sudo`
+  that shipped with CentOS 5 replaced itself with the program to be executed,
+  and therefore did not hold any child PTY open on our behalf. The child
+  context is updated to preserve any PTY FD in order to avoid the kernel
+  sending `SIGHUP` early during startup.
+
+* `#523 <https://github.com/dw/mitogen/issues/523>`_: the test suite didn't
+  generate a code coverage report if any test failed.
+
+* `#524 <https://github.com/dw/mitogen/issues/524>`_: Python 3.6+ emitted a
+  :class:`DeprecationWarning` for :func:`mitogen.utils.run_with_router`.
+
+* `#529 <https://github.com/dw/mitogen/issues/529>`_: Code coverage of the
+  test suite was not measured across all Python versions.
+
+* `16ca111e <https://github.com/dw/mitogen/commit/16ca111e>`_: handle OpenSSH
+  7.5 permission denied prompts when ``~/.ssh/config`` rewrites are present.
+
+* `9ec360c2 <https://github.com/dw/mitogen/commit/9ec360c2>`_: a new
+  :meth:`mitogen.core.Broker.defer_sync` utility function is provided.
+
+* `f20e0bba <https://github.com/dw/mitogen/commit/f20e0bba>`_:
+  :meth:`mitogen.service.FileService.register_prefix` permits granting
+  unprivileged access to whole filesystem subtrees, rather than single files at
+  a time.
+
+* `8f85ee03 <https://github.com/dw/mitogen/commit/8f85ee03>`_:
+  :meth:`mitogen.core.Router.myself` returns a :class:`mitogen.core.Context`
+  referring to the current process.
+
+* `824c7931 <https://github.com/dw/mitogen/commit/824c7931>`_: exceptions
+  raised by the import hook were updated to include probable reasons for
+  a failure.
+
+* `57b652ed <https://github.com/dw/mitogen/commit/57b652ed>`_: a stray import
+  meant an extra roundtrip and ~4KiB of data was wasted for any context that
+  imported :mod:`mitogen.parent`.
+
+
+Thanks!
+~~~~~~~
+
+Mitogen would not be possible without the support of users. A huge thanks for
+bug reports, testing, features and fixes in this release contributed by
+`Alex Willmer <https://github.com/moreati>`_,
+`Andreas Krüger <https://github.com/woopstar>`_,
+`Anton Stroganov <https://github.com/Aeon>`_,
+`Berend De Schouwer <https://github.com/berenddeschouwer>`_,
+`Brian Candler <https://github.com/candlerb>`_,
+`dsgnr <https://github.com/dsgnr>`_,
+`Duane Zamrok <https://github.com/dewthefifth>`_,
+`Eric Chang <https://github.com/changchichung>`_,
+`Gerben Meijer <https://github.com/infernix>`_,
+`Guy Knights <https://github.com/knightsg>`_,
+`Jesse London <https://github.com/jesteria>`_,
+`Jiří Vávra <https://github.com/Houbovo>`_,
+`Johan Beisser <https://github.com/jbeisser>`_,
+`Jonathan Rosser <https://github.com/jrosser>`_,
+`Josh Smift <https://github.com/jbscare>`_,
+`Kevin Carter <https://github.com/cloudnull>`_,
+`Mehdi <https://github.com/mehdisat7>`_,
+`Michael DeHaan <https://github.com/mpdehaan>`_,
+`Michal Medvecky <https://github.com/michalmedvecky>`_,
+`Mohammed Naser <https://github.com/mnaser/>`_,
+`Peter V. Saveliev <https://github.com/svinota/>`_,
+`Pieter Avonts <https://github.com/pieteravonts/>`_,
+`Ross Williams <https://github.com/overhacked/>`_,
+`Sergey <https://github.com/LuckySB/>`_,
+`Stéphane <https://github.com/sboisson/>`_,
+`Strahinja Kustudic <https://github.com/kustodian>`_,
+`Tom Parker-Shemilt <https://github.com/palfrey/>`_,
+`Younès HAFRI <https://github.com/yhafri>`_,
+`@killua-eu <https://github.com/killua-eu>`_,
+`@myssa91 <https://github.com/myssa91>`_,
+`@ohmer1 <https://github.com/ohmer1>`_,
+`@s3c70r <https://github.com/s3c70r/>`_,
+`@syntonym <https://github.com/syntonym/>`_,
+`@trim777 <https://github.com/trim777/>`_,
+`@whky <https://github.com/whky/>`_, and
+`@yodatak <https://github.com/yodatak/>`_.
+
+
 v0.2.3 (2018-10-23)
 -------------------
 
@@ -217,7 +739,7 @@ Thanks!
 ~~~~~~~
 
 Mitogen would not be possible without the support of users. A huge thanks for
-bug reports, features and fixes in this release contributed by
+bug reports, testing, features and fixes in this release contributed by
 `Alex Russu <https://github.com/alexrussu>`_,
 `Alex Willmer <https://github.com/moreati>`_,
 `atoom <https://github.com/atoom>`_,
@@ -398,69 +920,6 @@ Mitogen for Ansible
 * Built-in file transfer compatible with connection delegation.
 
 
-**Known Issues**
-
-* The ``raw`` action executes as a regular Mitogen connection, which requires
-  Python on the target, precluding its use for installing Python. This will be
-  addressed in a future 0.2 release. For now, simply mix Mitogen and vanilla
-  Ansible strategies in your playbook:
-
-  .. code-block:: yaml
-
-    - hosts: web-servers
-      strategy: linear
-      tasks:
-      - name: Install Python if necessary.
-        raw: test -e /usr/bin/python || apt install -y python-minimal
-
-    - hosts: web-servers
-      strategy: mitogen_linear
-      roles:
-      - nginx
-      - initech_app
-      - y2k_fix
-
-.. * When running with ``-vvv``, log messages will be printed to the console
-     *after* the Ansible run completes, as connection multiplexer shutdown only
-     begins after Ansible exits. This is due to a lack of suitable shutdown hook
-     in Ansible, and is fairly harmless, albeit cosmetically annoying. A future
-     release may include a solution.
-
-.. * Configurations will break that rely on the `hashbang argument splitting
-     behaviour <https://github.com/ansible/ansible/issues/15635>`_ of the
-     ``ansible_python_interpreter`` setting, contrary to the Ansible
-     documentation. This will be addressed in a future 0.2 release.
-
-* The Ansible 2.7 ``reboot`` module is not yet supported.
-
-* Performance does not scale linearly with target count. This requires
-  significant additional work, as major bottlenecks exist in the surrounding
-  Ansible code. Performance-related bug reports for any scenario remain
-  welcome with open arms.
-
-* Performance on Python 3 is significantly worse than on Python 2. While this
-  has not yet been investigated, at least some of the regression appears to be
-  part of the core library, and should therefore be straightforward to fix as
-  part of 0.2.x.
-
-* *Module Replacer* style Ansible modules are not supported.
-
-* Actions are single-threaded for each `(host, user account)` combination,
-  including actions that execute on the local machine. Playbooks may experience
-  slowdown compared to vanilla Ansible if they employ long-running
-  ``local_action`` or ``delegate_to`` tasks delegating many target hosts to a
-  single machine and user account.
-
-* Connection Delegation remains in preview and has bugs around how it infers
-  connections. Connection establishment will remain single-threaded for the 0.2
-  series, however connection inference bugs will be addressed in a future 0.2
-  release.
-
-* Connection Delegation does not support automatic tunnelling of SSH-dependent
-  actions, such as the ``synchronize`` module. This will be addressed in the
-  0.3 series.
-
-
 Core Library
 ~~~~~~~~~~~~
 
@@ -473,33 +932,3 @@ Core Library
   Windows Subsystem for Linux explicitly supported.
 
 * Automatic tests covering Python 2.6, 2.7 and 3.6 on Linux only.
-
-
-**Known Issues**
-
-* Serialization is still based on :mod:`pickle`. While there is high confidence
-  remote code execution is impossible in Mitogen's configuration, an untrusted
-  context may at least trigger disproportionately high memory usage injecting
-  small messages (*"billion laughs attack"*). Replacement is an important
-  future priority, but not critical for an initial release.
-
-* Child processes are not reliably reaped, leading to a pileup of zombie
-  processes when a program makes many short-lived connections in a single
-  invocation. This does not impact Mitogen for Ansible, however it limits the
-  usefulness of the core library. A future 0.2 release will address it.
-
-* Some races remain around :class:`mitogen.core.Broker <Broker>` destruction,
-  disconnection and corresponding file descriptor closure. These are only
-  problematic in situations where child process reaping is also problematic.
-
-* The `fakessh` component does not shut down correctly and requires flow
-  control added to the design. While minimal fixes are possible, due to the
-  absence of flow control the original design is functionally incomplete.
-
-* The multi-threaded :ref:`service` remains in a state of design flux and
-  should be considered obsolete, despite heavy use in Mitogen for Ansible. A
-  future replacement may be integrated more tightly with, or entirely replace
-  the RPC dispatcher on the main thread.
-
-* Documentation is in a state of disrepair. This will be improved over the 0.2
-  series.

docs/examples.rst

@@ -137,6 +137,9 @@ We could instead express the above using Mitogen:
 
 ::
 
+    import shutil, os, subprocess
+    import mitogen
+
     def run(*args):
         return subprocess.check_call(args)
 
@@ -144,22 +147,24 @@ We could instead express the above using Mitogen:
         with open(path, 'rb') as fp:
             return s in fp.read()
 
-    device = '/dev/sdb1'
-    mount_point = '/media/Media Volume'
+    @mitogen.main()
+    def main(router):
+        device = '/dev/sdb1'
+        mount_point = '/media/Media Volume'
 
-    bastion = router.ssh(hostname='bastion')
-    bastion_sudo = router.sudo(via=bastion)
+        bastion = router.ssh(hostname='bastion')
+        bastion_sudo = router.sudo(via=bastion)
 
-    if PROD:
-        fileserver = router.ssh(hostname='fileserver', via=bastion)
-        if fileserver.call(file_contains, device, '/proc/mounts'):
-            print('{} already mounted!'.format(device))
-            fileserver.call(run, 'umount', device)
-        fileserver.call(shutil.rmtree, mount_point)
-        fileserver.call(os.mkdir, mount_point, 0777)
-        fileserver.call(run, 'mount', device, mount_point)
+        if PROD:
+            fileserver = router.ssh(hostname='fileserver', via=bastion)
+            if fileserver.call(file_contains, device, '/proc/mounts'):
+                print('{} already mounted!'.format(device))
+                fileserver.call(run, 'umount', device)
+            fileserver.call(shutil.rmtree, mount_point)
+            fileserver.call(os.mkdir, mount_point, 0777)
+            fileserver.call(run, 'mount', device, mount_point)
 
-    bastion_sudo.call(run, 'touch', '/var/run/start_backup')
+        bastion_sudo.call(run, 'touch', '/var/run/start_backup')
 
 * In which context must the ``PROD`` variable be defined?
 * On which machine is each step executed?
@@ -185,9 +190,9 @@ nested.py:
 .. code-block:: python
 
     import os
-    import mitogen.utils
+    import mitogen
 
-    @mitogen.utils.run_with_router
+    @mitogen.main()
     def main(router):
         mitogen.utils.log_to_file()
 

docs/getting_started.rst

@@ -248,7 +248,7 @@ Running User Functions
 ----------------------
 
 So far we have used the interactive interpreter to call some standard library
-functions, but if since source code typed at the interpreter cannot be
+functions, but since the source code typed at the interpreter cannot be
 recovered, Mitogen is unable to execute functions defined in this way.
 
 We must therefore continue by writing our code as a script::

docs/howitworks.rst

@@ -16,17 +16,17 @@ The UNIX First Stage
 
 To allow delivery of the bootstrap compressed using :py:mod:`zlib`, it is
 necessary for something on the remote to be prepared to decompress the payload
-and feed it to a Python interpreter. Since we would like to avoid writing an
-error-prone shell fragment to implement this, and since we must avoid writing
-to the remote machine's disk in case it is read-only, the Python process
-started on the remote machine by Mitogen immediately forks in order to
+and feed it to a Python interpreter [#f1]_. Since we would like to avoid
+writing an error-prone shell fragment to implement this, and since we must
+avoid writing to the remote machine's disk in case it is read-only, the Python
+process started on the remote machine by Mitogen immediately forks in order to
 implement the decompression.
 
 
 Python Command Line
 ###################
 
-The Python command line sent to the host is a :mod:`zlib`-compressed [#f1]_ and
+The Python command line sent to the host is a :mod:`zlib`-compressed [#f2]_ and
 base64-encoded copy of the :py:meth:`mitogen.master.Stream._first_stage`
 function, which has been carefully optimized to reduce its size. Prior to
 compression and encoding, ``CONTEXT_NAME`` is replaced with the desired context
@@ -65,10 +65,10 @@ allowing reading by the first stage of exactly the required bytes.
 Configuring argv[0]
 ###################
 
-Forking provides us with an excellent opportunity for tidying up the eventual
-Python interpreter, in particular, restarting it using a fresh command-line to
-get rid of the large base64-encoded first stage parameter, and to replace
-**argv[0]** with something descriptive.
+Forking provides an excellent opportunity to tidy up the eventual Python
+interpreter, in particular, restarting it using a fresh command-line to get rid
+of the large base64-encoded first stage parameter, and to replace **argv[0]**
+with something descriptive.
 
 After configuring its ``stdin`` to point to the read end of the pipe, the
 parent half of the fork re-executes Python, with **argv[0]** taken from the
@@ -273,6 +273,10 @@ parent and child. Integers use big endian in their encoded form.
       - Size
       - Description
 
+    * - `magic`
+      - 2
+      - Integer 0x4d49 (``MI``), used to detect stream corruption.
+
     * - `dst_id`
       - 4
       - Integer target context ID. :py:class:`Router` delivers messages
@@ -577,6 +581,28 @@ When ``sudo:node22a:webapp`` wants to send a message to
     :class: mitogen-full-width
 
 
+Disconnect Propagation
+######################
+
+To ensure timely shutdown when a failure occurs, where some context is awaiting
+a response from another context that has become disconnected,
+:class:`mitogen.core.Router` additionally records the destination context ID of
+every message received on a particular stream.
+
+When ``DEL_ROUTE`` is generated locally or received on some other stream,
+:class:`mitogen.parent.RouteMonitor` uses this to find every stream that ever
+communicated with the route that is about to go away, and forwards the message
+to each found.
+
+The recipient ``DEL_ROUTE`` handler in turn uses the message to find any
+:class:`mitogen.core.Context` in the local process corresponding to the
+disappearing route, and if found, fires a ``disconnected`` event on it.
+
+Any interested party, such as :class:`mitogen.core.Receiver`, may subscribe to
+the event and use it to abort any threads that were asleep waiting for a reply
+that will never arrive.
+
+
 .. _source-verification:
 
 Source Verification
@@ -998,7 +1024,13 @@ receive items in the order they are requested, as they become available.
 
 .. rubric:: Footnotes
 
-.. [#f1] Compression may seem redundant, however it is basically free and reducing IO
+.. [#f1] Although some connection methods such as SSH support compression, and
+   Mitogen enables SSH compression by default, there are circumstances where
+   disabling SSH compression is desirable, and many scenarios for future
+   connection methods where transport-layer compression is not supported at
+   all. 
+
+.. [#f2] Compression may seem redundant, however it is basically free and reducing IO
    is always a good idea. The 33% / 200 byte saving may mean the presence or
    absence of an additional frame on the network, or in real world terms after
    accounting for SSH overhead, around a 2% reduced chance of a stall during

docs/images/ansible/pcaps/.gitattributes

@@ -0,0 +1,3 @@
+**pcap** filter=lfs diff=lfs merge=lfs -text
+run_hostname_100_times_mito.pcap.gz filter=lfs diff=lfs merge=lfs -text
+run_hostname_100_times_vanilla.pcap.gz filter=lfs diff=lfs merge=lfs -text

docs/images/ansible/pcaps/stroke-width.py

@@ -0,0 +1,16 @@
+
+import sys
+# Add viewBox attr to SVGs lacking it, so IE scales properly.
+
+import lxml.etree
+import glob
+
+
+for name in sys.argv[1:]:  # glob.glob('*/*.svg'): #+ glob.glob('images/ansible/*.svg'):
+    doc = lxml.etree.parse(open(name))
+    svg = doc.getroot()
+    for elem in svg.cssselect('[stroke-width]'):
+        if elem.attrib['stroke-width'] < '2':
+            elem.attrib['stroke-width'] = '2'
+
+    open(name, 'w').write(lxml.etree.tostring(svg, xml_declaration=True, encoding='UTF-8'))

docs/images/ansible/pcaps/svg-boxify.py

@@ -0,0 +1,13 @@
+
+# Add viewBox attr to SVGs lacking it, so IE scales properly.
+
+import lxml.etree
+import glob
+
+
+for name in glob.glob('images/*.svg') + glob.glob('images/ansible/*.svg'):
+    doc = lxml.etree.parse(open(name))
+    svg = doc.getroot()
+    if 'viewBox' not in svg.attrib:
+        svg.attrib['viewBox'] = '0 0 %(width)s %(height)s' % svg.attrib
+        open(name, 'w').write(lxml.etree.tostring(svg, xml_declaration=True, encoding='UTF-8'))

docs/index.rst

@@ -2,8 +2,6 @@
 Mitogen
 =======
 
-Mitogen is a Python library for writing distributed self-replicating programs.
-
 .. raw:: html
 
     <style>
@@ -13,7 +11,9 @@ Mitogen is a Python library for writing distributed self-replicating programs.
     </style>
 
 .. image:: images/mitogen.svg
-    :class: mitogen-right-225
+    :class: mitogen-right-200 mitogen-logo-wrap
+
+Mitogen is a Python library for writing distributed self-replicating programs.
 
 There is no requirement for installing packages, copying files around, writing
 shell snippets, upfront configuration, or providing any secondary link to a
@@ -351,6 +351,7 @@ usual into the slave process.
         os.system('tar zxvf my_app.tar.gz')
 
 
+    @mitogen.main()
     def main(broker):
         if len(sys.argv) != 2:
             print(__doc__)
@@ -359,10 +360,6 @@ usual into the slave process.
         context = mitogen.ssh.connect(broker, sys.argv[1])
         context.call(install_app)
 
-    if __name__ == '__main__' and mitogen.is_master:
-        import mitogen.utils
-        mitogen.utils.run_with_broker(main)
-
 
 Event-driven IO
 ###############
@@ -398,12 +395,12 @@ a large fleet of machines, or to alert the parent of unexpected state changes.
 Compatibility
 #############
 
-Mitogen is syntax-compatible with **Python 2.4** released November 2004, making
-it suitable for managing a fleet of potentially ancient corporate hardware,
-such as Red Hat Enterprise Linux 5, released in 2007.
+Mitogen is compatible with **Python 2.4** released November 2004, making it
+suitable for managing a fleet of potentially ancient corporate hardware, such
+as Red Hat Enterprise Linux 5, released in 2007.
 
 Every combination of Python 3.x/2.x parent and child should be possible,
-however at present only Python 2.6, 2.7 and 3.6 are tested automatically.
+however at present only Python 2.4, 2.6, 2.7 and 3.6 are tested automatically.
 
 
 Zero Dependencies

docs/internals.rst

@@ -15,6 +15,20 @@ Constants
 .. autodata:: CHUNK_SIZE
 
 
+Poller Classes
+==============
+
+.. currentmodule:: mitogen.core
+.. autoclass:: Poller
+   :members:
+
+.. currentmodule:: mitogen.parent
+.. autoclass:: EpollPoller
+
+.. currentmodule:: mitogen.parent
+.. autoclass:: KqueuePoller
+
+
 Latch Class
 ===========
 
@@ -32,171 +46,42 @@ PidfulStreamHandler Class
 
 
 Side Class
-----------
+==========
 
 .. currentmodule:: mitogen.core
-
-.. class:: Side (stream, fd, keep_alive=True)
-
-    Represent a single side of a :py:class:`BasicStream`. This exists to allow
-    streams implemented using unidirectional (e.g. UNIX pipe) and bidirectional
-    (e.g. UNIX socket) file descriptors to operate identically.
-
-    :param mitogen.core.Stream stream:
-        The stream this side is associated with.
-
-    :param int fd:
-        Underlying file descriptor.
-
-    :param bool keep_alive:
-        Value for :py:attr:`keep_alive`
-
-    During construction, the file descriptor has its :py:data:`os.O_NONBLOCK`
-    flag enabled using :py:func:`fcntl.fcntl`.
-
-    .. attribute:: stream
-
-        The :py:class:`Stream` for which this is a read or write side.
-
-    .. attribute:: fd
-
-        Integer file descriptor to perform IO on, or :data:`None` if
-        :py:meth:`close` has been called.
-
-    .. attribute:: keep_alive
-
-        If :data:`True`, causes presence of this side in :py:class:`Broker`'s
-        active reader set to defer shutdown until the side is disconnected.
-
-    .. method:: fileno
-
-        Return :py:attr:`fd` if it is not :data:`None`, otherwise raise
-        :py:class:`StreamError`. This method is implemented so that
-        :py:class:`Side` can be used directly by :py:func:`select.select`.
-
-    .. method:: close
-
-        Call :py:func:`os.close` on :py:attr:`fd` if it is not :data:`None`,
-        then set it to :data:`None`.
-
-    .. method:: read (n=CHUNK_SIZE)
-
-        Read up to `n` bytes from the file descriptor, wrapping the underlying
-        :py:func:`os.read` call with :py:func:`io_op` to trap common
-        disconnection conditions.
-
-        :py:meth:`read` always behaves as if it is reading from a regular UNIX
-        file; socket, pipe, and TTY disconnection errors are masked and result
-        in a 0-sized read just like a regular file.
-
-        :returns:
-            Bytes read, or the empty to string to indicate disconnection was
-            detected.
-
-    .. method:: write (s)
-
-        Write as much of the bytes from `s` as possible to the file descriptor,
-        wrapping the underlying :py:func:`os.write` call with :py:func:`io_op`
-        to trap common disconnection connditions.
-
-        :returns:
-            Number of bytes written, or :data:`None` if disconnection was
-            detected.
+.. autoclass:: Side
+   :members:
 
 
 Stream Classes
---------------
+==============
 
 .. currentmodule:: mitogen.core
-
-.. class:: BasicStream
-
-    .. attribute:: receive_side
-
-        A :py:class:`Side` representing the stream's receive file descriptor.
-
-    .. attribute:: transmit_side
-
-        A :py:class:`Side` representing the stream's transmit file descriptor.
-
-    .. method:: on_disconnect (broker)
-
-        Called by :py:class:`Broker` to force disconnect the stream. The base
-        implementation simply closes :py:attr:`receive_side` and
-        :py:attr:`transmit_side` and unregisters the stream from the broker.
-
-    .. method:: on_receive (broker)
-
-        Called by :py:class:`Broker` when the stream's :py:attr:`receive_side` has
-        been marked readable using :py:meth:`Broker.start_receive` and the
-        broker has detected the associated file descriptor is ready for
-        reading.
-
-        Subclasses must implement this method if
-        :py:meth:`Broker.start_receive` is ever called on them, and the method
-        must call :py:meth:`on_disconect` if reading produces an empty string.
-
-    .. method:: on_transmit (broker)
-
-        Called by :py:class:`Broker` when the stream's :py:attr:`transmit_side`
-        has been marked writeable using :py:meth:`Broker._start_transmit` and
-        the broker has detected the associated file descriptor is ready for
-        writing.
-
-        Subclasses must implement this method if
-        :py:meth:`Broker._start_transmit` is ever called on them.
-
-    .. method:: on_shutdown (broker)
-
-        Called by :py:meth:`Broker.shutdown` to allow the stream time to
-        gracefully shutdown. The base implementation simply called
-        :py:meth:`on_disconnect`.
+.. autoclass:: BasicStream
+   :members:
 
 .. autoclass:: Stream
    :members:
 
-   .. method:: pending_bytes ()
-
-        Returns the number of bytes queued for transmission on this stream.
-        This can be used to limit the amount of data buffered in RAM by an
-        otherwise unlimited consumer.
-
-        For an accurate result, this method should be called from the Broker
-        thread, using a wrapper like:
-
-        ::
-
-            def get_pending_bytes(self, stream):
-                latch = mitogen.core.Latch()
-                self.broker.defer(
-                    lambda: latch.put(stream.pending_bytes())
-                )
-                return latch.get()
-
-
 .. currentmodule:: mitogen.fork
-
 .. autoclass:: Stream
    :members:
 
 .. currentmodule:: mitogen.parent
-
 .. autoclass:: Stream
    :members:
 
 .. currentmodule:: mitogen.ssh
-
 .. autoclass:: Stream
    :members:
 
 .. currentmodule:: mitogen.sudo
-
 .. autoclass:: Stream
    :members:
 
 
 Other Stream Subclasses
------------------------
+=======================
 
 .. currentmodule:: mitogen.core
 
@@ -208,10 +93,11 @@ Other Stream Subclasses
 
 
 Poller Class
-------------
+============
 
 .. currentmodule:: mitogen.core
 .. autoclass:: Poller
+    :members:
 
 .. currentmodule:: mitogen.parent
 .. autoclass:: KqueuePoller
@@ -221,7 +107,7 @@ Poller Class
 
 
 Importer Class
---------------
+==============
 
 .. currentmodule:: mitogen.core
 .. autoclass:: Importer
@@ -229,15 +115,23 @@ Importer Class
 
 
 Responder Class
----------------
+===============
 
 .. currentmodule:: mitogen.master
 .. autoclass:: ModuleResponder
    :members:
 
 
+RouteMonitor Class
+==================
+
+.. currentmodule:: mitogen.parent
+.. autoclass:: RouteMonitor
+   :members:
+
+
 Forwarder Class
----------------
+===============
 
 .. currentmodule:: mitogen.parent
 .. autoclass:: ModuleForwarder
@@ -245,67 +139,19 @@ Forwarder Class
 
 
 ExternalContext Class
----------------------
+=====================
 
 .. currentmodule:: mitogen.core
+.. autoclass:: ExternalContext
+    :members:
 
-.. class:: ExternalContext
-
-    External context implementation.
-
-    .. attribute:: broker
-
-        The :py:class:`mitogen.core.Broker` instance.
-
-    .. attribute:: context
-
-            The :py:class:`mitogen.core.Context` instance.
-
-    .. attribute:: channel
-
-            The :py:class:`mitogen.core.Channel` over which
-            :py:data:`CALL_FUNCTION` requests are received.
-
-    .. attribute:: stdout_log
-
-        The :py:class:`mitogen.core.IoLogger` connected to ``stdout``.
-
-    .. attribute:: importer
-
-        The :py:class:`mitogen.core.Importer` instance.
-
-    .. attribute:: stdout_log
-
-        The :py:class:`IoLogger` connected to ``stdout``.
-
-    .. attribute:: stderr_log
-
-        The :py:class:`IoLogger` connected to ``stderr``.
-
-    .. method:: _dispatch_calls
-
-        Implementation for the main thread in every child context.
 
 mitogen.master
 ==============
 
-.. currentmodule:: mitogen.master
-
-.. class:: ProcessMonitor
-
-    Install a :py:data:`signal.SIGCHLD` handler that generates callbacks when a
-    specific child process has exitted.
-
-    .. method:: add (pid, callback)
-
-        Add a callback function to be notified of the exit status of a process.
-
-        :param int pid:
-            Process ID to be notified of.
-
-        :param callback:
-            Function invoked as `callback(status)`, where `status` is the raw
-            exit status of the child process.
+.. currentmodule:: mitogen.parent
+.. autoclass:: ProcessMonitor
+    :members:
 
 
 Blocking I/O Functions

docs/requirements.txt

@@ -1,4 +1,3 @@
 Sphinx==1.7.1
-sphinx-autobuild==0.6.0 # Last version to support Python 2.6
 sphinxcontrib-programoutput==0.11
 alabaster==0.7.10

docs/shame.rst

@@ -1,90 +0,0 @@
-
-Importer Wall Of Shame
-----------------------
-
-The following modules and packages violate protocol or best practice in some way:
-
-* They run magic during ``__init.py__`` that makes life hard for Mitogen.
-  Executing code during module import is always bad, and Mitogen is a concrete
-  benchmark for why it's bad.
-
-* They install crap in :py:data:`sys.modules` that completely ignore or
-  partially implement the protocols laid out in PEP-302.
-
-* They "vendor" a third party package, either incompletely, using hacks visible
-  through the runtime's standard interfaces, or with ancient versions of code
-  that in turn mess with :py:data:`sys.modules` in some horrible way.
-
-Bugs will probably be filed for these in time, but it does not address the huge
-installed base of existing old software versions, so hacks are needed anyway.
-
-
-``pbr``
-=======
-
-It claims to use ``pkg_resources`` to read version information
-(``_get_version_from_pkg_metadata()``), which would result in PEP-302 being
-reused and everything just working wonderfully, but instead it actually does
-direct filesystem access.
-
-**What could it do instead?**
-
-* ``pkg_resources.resource_stream()``
-
-**What Mitogen is forced to do**
-
-When it sees ``pbr`` being loaded, it smodges the process environment with a
-``PBR_VERSION`` variable to override any attempt to auto-detect the version.
-This will probably break code I haven't seen yet.
-
-
-``pkg_resources``
-=================
-
-Anything that imports ``pkg_resources`` will eventually cause ``pkg_resources``
-to try and import and scan ``__main__`` for its ``__requires__`` attribute
-(``pkg_resources/__init__.py::_build_master()``). This breaks any app that is
-not expecting its ``__main__`` to suddenly be sucked over a network and
-injected into a remote process, like py.test.
-
-A future version of Mitogen might have a more general hack that doesn't import
-the master's ``__main__`` as ``__main__`` in the slave, avoiding all kinds of
-issues like these.
-
-**What could it do instead?**
-
-* Explicit is better than implicit: wait until the magical behaviour is
-  explicitly requested (i.e. an API call).
-
-* Use ``get("__main__")`` on :py:data:`sys.modules` rather than ``import``, but
-  this method isn't general enough, it only really helps tools like Mitogen.
-
-**What Mitogen is forced to do**
-
-Examine the stack during every attempt to import ``__main__`` and check if the
-requestee module is named ``pkg_resources``, if so then refuse the import.
-
-
-``six``
-=======
-
-The ``six`` module makes some effort to conform to PEP-302, but it is missing
-several critical pieces, e.g. the ``__loader__`` attribute. This not only
-breaks the Python standard library tooling (such as the :py:mod:`inspect`
-module), but also Mitogen. Newer versions of ``six`` improve things somewhat,
-but there are still outstanding issues preventing Mitogen from working with
-``six``.
-
-This package is sufficiently popular that it must eventually be supported. See
-`here for an example issue`_.
-
-.. _here for an example issue: https://github.com/dw/mitogen/issues/31
-
-**What could it do instead?**
-
-* Any custom hacks installed into :py:data:`sys.modules` should support the
-  protocols laid out in PEP-302.
-
-**What Mitogen is forced to do**
-
-Vendored versions of ``six`` currently don't work at all.

docs/toc.rst

@@ -14,7 +14,6 @@ Table Of Contents
    api
    examples
    internals
-   shame
 
 .. toctree::
    :hidden:

examples/mitogen-fuse.py

@@ -20,7 +20,6 @@ import mitogen.master
 import mitogen.utils
 
 import __main__
-import posix
 import os
 
 

examples/ping_pong.py

@@ -0,0 +1,46 @@
+# Wire up a ping/pong counting loop between 2 subprocesses.
+
+from __future__ import print_function
+import mitogen.core
+import mitogen.select
+
+
+@mitogen.core.takes_router
+def ping_pong(control_sender, router):
+    with mitogen.core.Receiver(router) as recv:
+        # Tell caller how to communicate with us.
+        control_sender.send(recv.to_sender())
+
+        # Wait for caller to tell us how to talk back:
+        data_sender = recv.get().unpickle()
+
+        n = 0
+        while (n + 1) < 30:
+            n = recv.get().unpickle()
+            print('the number is currently', n)
+            data_sender.send(n + 1)
+
+
+@mitogen.main()
+def main(router):
+    # Create a receiver for control messages.
+    with mitogen.core.Receiver(router) as recv:
+        # Start ping_pong() in child 1 and fetch its sender.
+        c1 = router.local()
+        c1_call = c1.call_async(ping_pong, recv.to_sender())
+        c1_sender = recv.get().unpickle()
+
+        # Start ping_pong() in child 2 and fetch its sender.
+        c2 = router.local()
+        c2_call = c2.call_async(ping_pong, recv.to_sender())
+        c2_sender = recv.get().unpickle()
+
+        # Tell the children about each others' senders.
+        c1_sender.send(c2_sender)
+        c2_sender.send(c1_sender)
+
+    # Start the loop.
+    c1_sender.send(0)
+
+    # Wait for both functions to return.
+    mitogen.select.Select.all([c1_call, c2_call])

examples/select_loop.py

@@ -0,0 +1,103 @@
+
+#
+# This demonstrates using a nested select.Select() to simultaneously watch for
+# in-progress events generated by a bunch of function calls, and the completion
+# of those function calls.
+#
+# We start 5 children and run a function in each of them in parallel. The
+# function writes the numbers 1..5 to a Sender before returning. The master
+# reads the numbers from each child as they are generated, and exits the loop
+# when the last function returns.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import time
+
+import mitogen
+import mitogen.select
+
+
+def count_to(sender, n, wait=0.333):
+    for x in range(n):
+        sender.send(x)
+        time.sleep(wait)
+
+
+@mitogen.main()
+def main(router):
+    # Start 5 subprocesses and give them made up names.
+    contexts = {
+        'host%d' % (i,): router.local()
+        for i in range(5)
+    }
+
+    # Used later to recover hostname. A future Mitogen will provide a better
+    # way to get app data references back out of its IO primitives, for now you
+    # need to do it manually.
+    hostname_by_context_id = {
+        context.context_id: hostname
+        for hostname, context in contexts.items()
+    }
+
+    # I am a select that holds the receivers that will receive the function
+    # call results. Selects are one-shot by default, which means each receiver
+    # is removed from them as a result arrives. Therefore it means the last
+    # function has completed when bool(calls_sel) is False.
+    calls_sel = mitogen.select.Select()
+
+    # I receive the numbers as they are counted.
+    status_recv = mitogen.core.Receiver(router)
+
+    # Start the function calls
+    for hostname, context in contexts.items():
+        calls_sel.add(
+            context.call_async(
+                count_to,
+                sender=status_recv.to_sender(),
+                n=5,
+                wait=0.333
+            )
+        )
+
+    # Create a select subscribed to the function call result Select, and to the
+    # number-counting receiver. Any message arriving on any child of this
+    # Select will wake it up -- be it a message arriving on the status
+    # receiver, or any message arriving on any of the function call result
+    # receivers.
+
+    # Once last call is completed, calls_sel will be empty since it's
+    # oneshot=True (the default), causing __bool__ to be False
+    both_sel = mitogen.select.Select([status_recv, calls_sel], oneshot=False)
+
+    # Internally selects store a strong reference from Receiver->Select that
+    # will keep the Select alive as long as the receiver is alive. If a
+    # receiver or select otherwise 'outlives' some parent select, attempting to
+    # re-add it to a new select will raise an error. In all cases it's
+    # desirable to call Select.close(). This can be done as a context manager.
+    with calls_sel, both_sel:
+        while calls_sel:
+            try:
+                msg = both_sel.get(timeout=60.0)
+            except mitogen.core.TimeoutError:
+                print("No update in 60 seconds, something's broke")
+                break
+
+            hostname = hostname_by_context_id[msg.src_id]
+
+            if msg.receiver is status_recv:   # https://mitogen.readthedocs.io/en/stable/api.html#mitogen.core.Message.receiver
+                # handle a status update
+                print('Got status update from %s: %s' % (hostname, msg.unpickle()))
+            elif msg.receiver is calls_sel:  # subselect
+                # handle a function call result.
+                try:
+                    assert None == msg.unpickle()
+                    print('Task succeeded on %s' % (hostname,))
+                except mitogen.core.CallError as e:
+                    print('Task failed on host %s: %s' % (hostname, e))
+
+        if calls_sel:
+            print('Some tasks did not complete.')
+        else:
+            print('All tasks completed.')

examples/service/client.py

@@ -1,6 +1,4 @@
 
-import socket
-
 import mitogen.master
 import mitogen.unix
 import mitogen.service

examples/service/server.py

@@ -3,8 +3,6 @@
 # hopefully lose those hard-coded magic numbers somehow), but meanwhile this is
 # a taster of how it looks today.
 
-import time
-
 import mitogen
 import mitogen.service
 import mitogen.unix

examples/the_basics.py

@@ -0,0 +1,295 @@
+
+#
+# This program is a stand-in for good intro docs. It just documents various
+# basics of using Mitogen.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import hashlib
+import io
+import os
+import spwd
+
+import mitogen.core
+import mitogen.master
+import mitogen.service
+import mitogen.utils
+
+
+
+def get_file_contents(path):
+    """
+    Get the contents of a file.
+    """
+    with open(path, 'rb') as fp:
+        # mitogen.core.Blob() is a bytes subclass with a repr() that returns a
+        # summary of the blob, rather than the raw blob data. This makes
+        # logging output *much* nicer. Unlike most custom types, blobs can be
+        # serialized.
+        return mitogen.core.Blob(fp.read())
+
+
+def put_file_contents(path, s):
+    """
+    Write the contents of a file.
+    """
+    with open(path, 'wb') as fp:
+        fp.write(s)
+
+
+def streamy_download_file(context, path):
+    """
+    Fetch a file from the FileService hosted by `context`.
+    """
+    bio = io.BytesIO()
+
+    # FileService.get() is not actually an exposed service method, it's just a
+    # classmethod that wraps up the complicated dance of implementing the
+    # transfer.
+    ok, metadata = mitogen.service.FileService.get(context, path, bio)
+
+    return {
+        'success': ok,
+        'metadata': metadata,
+        'size': len(bio.getvalue()),
+    }
+
+
+def get_password_hash(username):
+    """
+    Fetch a user's password hash.
+    """
+    try:
+        h = spwd.getspnam(username)
+    except KeyError:
+        return None
+
+    # mitogen.core.Secret() is a Unicode subclass with a repr() that hides the
+    # secret data. This keeps secret stuff out of logs. Like blobs, secrets can
+    # also be serialized.
+    return mitogen.core.Secret(h)
+
+
+def md5sum(path):
+    """
+    Return the MD5 checksum for a file.
+    """
+    return hashlib.md5(get_file_contents(path)).hexdigest()
+
+
+
+def work_on_machine(context):
+    """
+    Do stuff to a remote context.
+    """
+    print("Created context. Context ID is", context.context_id)
+
+    # You don't need to understand any/all of this, but it's helpful to grok
+    # the whole chain:
+
+    # - Context.call() is a light wrapper around .call_async(), the wrapper
+    #   simply blocks the caller until a reply arrives.
+    # - .call_async() serializes the call signature into a message and passes
+    #   it to .send_async()
+    # - .send_async() creates a mitogen.core.Receiver() on the local router.
+    #   The receiver constructor uses Router.add_handle() to allocate a
+    #   'reply_to' handle and install a callback function that wakes the
+    #   receiver when a reply message arrives.
+    # - .send_async() puts the reply handle in Message.reply_to field and
+    #   passes it to .send()
+    # - Context.send() stamps the destination context ID into the
+    #   Message.dst_id field and passes it to Router.route()
+    # - Router.route() uses Broker.defer() to schedule _async_route(msg)
+    #   on the Broker thread.
+    # [broker thread]
+    # - The broker thread wakes and calls _async_route(msg)
+    # - Router._async_route() notices 'dst_id' is for a remote context and
+    #   looks up the stream on which messages for dst_id should be sent (may be
+    #   direct connection or not), and calls Stream.send()
+    # - Stream.send() packs the message into a bytestring, appends it to
+    #   Stream._output_buf, and calls Broker.start_transmit()
+    # - Broker finishes work, reenters IO loop. IO loop wakes due to writeable
+    #   stream.
+    # - Stream.on_transmit() writes the full/partial buffer to SSH, calls
+    #   stop_transmit() to mark the stream unwriteable once _output_buf is
+    #   empty.
+    # - Broker IO loop sleeps, no readers/writers.
+    # - Broker wakes due to SSH stream readable.
+    # - Stream.on_receive() called, reads the reply message, converts it to a
+    #   Message and passes it to Router._async_route().
+    # - Router._async_route() notices message is for local context, looks up
+    #   target handle in the .add_handle() registry.
+    # - Receiver._on_receive() called, appends message to receiver queue.
+    # [main thread]
+    # - Receiver.get() used to block the original Context.call() wakes and pops
+    #   the message from the queue.
+    # - Message data (pickled return value) is deserialized and returned to the
+    #   caller.
+    print("It's running on the local machine. Its PID is",
+          context.call(os.getpid))
+
+    # Now let's call a function defined in this module. On receiving the
+    # function call request, the child attempts to import __main__, which is
+    # initially missing, causing the importer in the child to request it from
+    # its parent. That causes _this script_ to be sent as the module source
+    # over the wire.
+    print("Calling md5sum(/etc/passwd) in the child:",
+          context.call(md5sum, '/etc/passwd'))
+
+    # Now let's "transfer" a file. The simplest way to do this is calling a
+    # function that returns the file data, which is totally fine for small
+    # files.
+    print("Download /etc/passwd via function call: %d bytes" % (
+        len(context.call(get_file_contents, '/etc/passwd'))
+    ))
+
+    # And using function calls, in the other direction:
+    print("Upload /tmp/blah via function call: %s" % (
+        context.call(put_file_contents, '/tmp/blah', b'blah!'),
+    ))
+
+    # Now lets transfer what might be a big files. The problem with big files
+    # is that they may not fit in RAM. This uses mitogen.services.FileService
+    # to implement streamy file transfer instead. The sender must have a
+    # 'service pool' running that will host FileService. First let's do the
+    # 'upload' direction, where the master hosts FileService.
+
+    # Steals the 'Router' reference from the context object. In a real app the
+    # pool would be constructed once at startup, this is just demo code.
+    file_service = mitogen.service.FileService(context.router)
+
+    # Start the pool.
+    pool = mitogen.service.Pool(context.router, services=[file_service])
+
+    # Grant access to a file on the local disk from unprivileged contexts.
+    # .register() is also exposed as a service method -- you can call it on a
+    # child context from any more privileged context.
+    file_service.register('/etc/passwd')
+
+    # Now call our wrapper function that knows how to handle the transfer. In a
+    # real app, this wrapper might also set ownership/modes or do any other
+    # app-specific stuff relating to the file that was transferred.
+    print("Streamy upload /etc/passwd: remote result: %s" % (
+        context.call(
+            streamy_download_file,
+            # To avoid hard-wiring streamy_download_file(), we want to pass it
+            # a Context object that hosts the file service it should request
+            # files from. Router.myself() returns a Context referring to this
+            # process.
+            context=router.myself(),
+            path='/etc/passwd',
+        ),
+    ))
+
+    # Shut down the pool now we're done with it, else app will hang at exit.
+    # Once again, this should only happen once at app startup/exit, not for
+    # every file transfer!
+    pool.stop(join=True)
+
+    # Now let's do the same thing but in reverse: we use FileService on the
+    # remote download a file. This uses context.call_service(), which invokes a
+    # special code path that causes auto-initialization of a thread pool in the
+    # target, and auto-construction of the target service, but only if the
+    # service call was made by a more privileged context. We could write a
+    # helper function that runs in the remote to do all that by hand, but the
+    # library handles it for us.
+
+    # Make the file accessible. A future FileService could avoid the need for
+    # this for privileged contexts.
+    context.call_service(
+        service_name=mitogen.service.FileService,
+        method_name='register',
+        path='/etc/passwd'
+    )
+
+    # Now we can use our streamy_download_file() function in reverse -- running
+    # it from this process and having it fetch from the remote process:
+    print("Streamy download /etc/passwd: result: %s" % (
+        streamy_download_file(context, '/etc/passwd'),
+    ))
+
+
+def main():
+    # Setup logging. Mitogen produces a LOT of logging. Over the course of the
+    # stable series, Mitogen's loggers will be carved up so more selective /
+    # user-friendly logging is possible. mitogen.log_to_file() just sets up
+    # something basic, defaulting to INFO level, but you can override from the
+    # command-line by passing MITOGEN_LOG_LEVEL=debug or MITOGEN_LOG_LEVEL=io.
+    # IO logging is sometimes useful for hangs, but it is often creates more
+    # confusion than it solves.
+    mitogen.utils.log_to_file()
+
+    # Construct the Broker thread. It manages an async IO loop listening for
+    # reads from any active connection, or wakes from any non-Broker thread.
+    # Because Mitogen uses a background worker thread, it is extremely
+    # important to pay attention to the use of UNIX fork in your code --
+    # forking entails making a snapshot of the state of all locks in the
+    # program, including those in the logging module, and thus can create code
+    # that appears to work for a long time, before deadlocking randomly.
+    # Forking in a Mitogen app requires significant upfront planning!
+    broker = mitogen.master.Broker()
+
+    # Construct a Router. This accepts messages (mitogen.core.Message) and
+    # either dispatches locally addressed messages to local handlers (added via
+    # Router.add_handle()) on the broker thread, or forwards the message
+    # towards the target context.
+
+    # The router also acts as an uglyish God object for creating new
+    # connections. This was a design mistake, really those methods should be
+    # directly imported from e.g. 'mitogen.ssh'.
+    router = mitogen.master.Router(broker)
+
+    # Router can act like a context manager. It simply ensures
+    # Broker.shutdown() is called on exception / exit. That prevents the app
+    # hanging due to a forgotten background thread. For throwaway scripts,
+    # there are also decorator versions "@mitogen.main()" and
+    # "@mitogen.utils.with_router" that do the same thing with less typing.
+    with router:
+        # Now let's construct a context. The '.local()' constructor just creates
+        # the context as a subprocess, the simplest possible case.
+        child = router.local()
+        print("Created a context:", child)
+        print()
+
+        # This demonstrates the standard IO redirection. We call the print
+        # function in the remote context, that should cause a log message to be
+        # emitted. Any subprocesses started by the remote also get the same
+        # treatment, so it's very easy to spot otherwise discarded errors/etc.
+        # from remote tools.
+        child.call(print, "Hello from child.")
+
+        # Context objects make it semi-convenient to treat the local machine the
+        # same as a remote machine.
+        work_on_machine(child)
+
+        # Now let's construct a proxied context. We'll simply use the .local()
+        # constructor again, but construct it via 'child'. In effect we are
+        # constructing a sub-sub-process. Instead of .local() here, we could
+        # have used .sudo() or .ssh() or anything else.
+        subchild = router.local(via=child)
+        print()
+        print()
+        print()
+        print("Created a context as a child of another context:", subchild)
+
+        # Do everything again with the new child.
+        work_on_machine(subchild)
+
+        # We can selectively shut down individual children if we want:
+        subchild.shutdown(wait=True)
+
+        # Or we can simply fall off the end of the scope, effectively calling
+        # Broker.shutdown(), which causes all children to die as part of
+        # shutdown.
+
+
+# The child module importer detects the execution guard below and removes any
+# code appearing after it, and refuses to execute "__main__" if it is absent.
+# This is necessary to prevent a common problem where people try to call
+# functions defined in __main__ without first wrapping it up to be importable
+# as a module, which previously hung the target, or caused bizarre recursive
+# script runs.
+if __name__ == '__main__':
+    main()

mitogen/__init__.py

@@ -26,6 +26,8 @@
 # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 # POSSIBILITY OF SUCH DAMAGE.
 
+# !mitogen: minify_safe
+
 """
 On the Mitogen master, this is imported from ``mitogen/__init__.py`` as would
 be expected. On the slave, it is built dynamically during startup.
@@ -33,7 +35,7 @@ be expected. On the slave, it is built dynamically during startup.
 
 
 #: Library version as a tuple.
-__version__ = (0, 2, 3)
+__version__ = (0, 2, 4)
 
 
 #: This is :data:`False` in slave contexts. Previously it was used to prevent
@@ -57,7 +59,12 @@ parent_id = None
 parent_ids = []
 
 
-def main(log_level='INFO', profiling=False):
+import os
+_default_profiling = os.environ.get('MITOGEN_PROFILING') is not None
+del os
+
+
+def main(log_level='INFO', profiling=_default_profiling):
     """
     Convenience decorator primarily useful for writing discardable test
     scripts.
@@ -106,7 +113,7 @@ def main(log_level='INFO', profiling=False):
             mitogen.master.Router.profiling = profiling
         utils.log_to_file(level=log_level)
         return mitogen.core._profile_hook(
-            'main',
+            'app.main',
             utils.run_with_router,
             func,
         )

mitogen/compat/functools.py

@@ -1,288 +0,0 @@
-# encoding: utf-8
-"""Selected backports from Python stdlib functools module
-"""
-# Written by Nick Coghlan <ncoghlan at gmail.com>,
-# Raymond Hettinger <python at rcn.com>,
-# and Łukasz Langa <lukasz at langa.pl>.
-#   Copyright (C) 2006-2013 Python Software Foundation.
-
-__all__ = [
-    'update_wrapper', 'wraps', 'WRAPPER_ASSIGNMENTS', 'WRAPPER_UPDATES',
-    'lru_cache',
-]
-
-from threading import RLock
-
-
-################################################################################
-### update_wrapper() and wraps() decorator
-################################################################################
-
-# update_wrapper() and wraps() are tools to help write
-# wrapper functions that can handle naive introspection
-
-WRAPPER_ASSIGNMENTS = ('__module__', '__name__', '__qualname__', '__doc__',
-                       '__annotations__')
-WRAPPER_UPDATES = ('__dict__',)
-def update_wrapper(wrapper,
-                   wrapped,
-                   assigned = WRAPPER_ASSIGNMENTS,
-                   updated = WRAPPER_UPDATES):
-    """Update a wrapper function to look like the wrapped function
-       wrapper is the function to be updated
-       wrapped is the original function
-       assigned is a tuple naming the attributes assigned directly
-       from the wrapped function to the wrapper function (defaults to
-       functools.WRAPPER_ASSIGNMENTS)
-       updated is a tuple naming the attributes of the wrapper that
-       are updated with the corresponding attribute from the wrapped
-       function (defaults to functools.WRAPPER_UPDATES)
-    """
-    for attr in assigned:
-        try:
-            value = getattr(wrapped, attr)
-        except AttributeError:
-            pass
-        else:
-            setattr(wrapper, attr, value)
-    for attr in updated:
-        getattr(wrapper, attr).update(getattr(wrapped, attr, {}))
-    # Issue #17482: set __wrapped__ last so we don't inadvertently copy it
-    # from the wrapped function when updating __dict__
-    wrapper.__wrapped__ = wrapped
-    # Return the wrapper so this can be used as a decorator via partial()
-    return wrapper
-
-def wraps(wrapped,
-          assigned = WRAPPER_ASSIGNMENTS,
-          updated = WRAPPER_UPDATES):
-    """Decorator factory to apply update_wrapper() to a wrapper function
-       Returns a decorator that invokes update_wrapper() with the decorated
-       function as the wrapper argument and the arguments to wraps() as the
-       remaining arguments. Default arguments are as for update_wrapper().
-       This is a convenience function to simplify applying partial() to
-       update_wrapper().
-    """
-    return partial(update_wrapper, wrapped=wrapped,
-                   assigned=assigned, updated=updated)
-
-
-################################################################################
-### partial() argument application
-################################################################################
-
-# Purely functional, no descriptor behaviour
-def partial(func, *args, **keywords):
-    """New function with partial application of the given arguments
-    and keywords.
-    """
-    if hasattr(func, 'func'):
-        args = func.args + args
-        tmpkw = func.keywords.copy()
-        tmpkw.update(keywords)
-        keywords = tmpkw
-        del tmpkw
-        func = func.func
-
-    def newfunc(*fargs, **fkeywords):
-        newkeywords = keywords.copy()
-        newkeywords.update(fkeywords)
-        return func(*(args + fargs), **newkeywords)
-    newfunc.func = func
-    newfunc.args = args
-    newfunc.keywords = keywords
-    return newfunc
-
-
-################################################################################
-### LRU Cache function decorator
-################################################################################
-
-class _HashedSeq(list):
-    """ This class guarantees that hash() will be called no more than once
-        per element.  This is important because the lru_cache() will hash
-        the key multiple times on a cache miss.
-    """
-
-    __slots__ = 'hashvalue'
-
-    def __init__(self, tup, hash=hash):
-        self[:] = tup
-        self.hashvalue = hash(tup)
-
-    def __hash__(self):
-        return self.hashvalue
-
-def _make_key(args, kwds, typed,
-             kwd_mark = (object(),),
-             fasttypes = set([int, str, frozenset, type(None)]),
-             sorted=sorted, tuple=tuple, type=type, len=len):
-    """Make a cache key from optionally typed positional and keyword arguments
-    The key is constructed in a way that is flat as possible rather than
-    as a nested structure that would take more memory.
-    If there is only a single argument and its data type is known to cache
-    its hash value, then that argument is returned without a wrapper.  This
-    saves space and improves lookup speed.
-    """
-    key = args
-    if kwds:
-        sorted_items = sorted(kwds.items())
-        key += kwd_mark
-        for item in sorted_items:
-            key += item
-    if typed:
-        key += tuple(type(v) for v in args)
-        if kwds:
-            key += tuple(type(v) for k, v in sorted_items)
-    elif len(key) == 1 and type(key[0]) in fasttypes:
-        return key[0]
-    return _HashedSeq(key)
-
-def lru_cache(maxsize=128, typed=False):
-    """Least-recently-used cache decorator.
-    If *maxsize* is set to None, the LRU features are disabled and the cache
-    can grow without bound.
-    If *typed* is True, arguments of different types will be cached separately.
-    For example, f(3.0) and f(3) will be treated as distinct calls with
-    distinct results.
-    Arguments to the cached function must be hashable.
-    View the cache statistics named tuple (hits, misses, maxsize, currsize)
-    with f.cache_info().  Clear the cache and statistics with f.cache_clear().
-    Access the underlying function with f.__wrapped__.
-    See:  http://en.wikipedia.org/wiki/Cache_algorithms#Least_Recently_Used
-    """
-
-    # Users should only access the lru_cache through its public API:
-    #       cache_info, cache_clear, and f.__wrapped__
-    # The internals of the lru_cache are encapsulated for thread safety and
-    # to allow the implementation to change (including a possible C version).
-
-    # Early detection of an erroneous call to @lru_cache without any arguments
-    # resulting in the inner function being passed to maxsize instead of an
-    # integer or None.
-    if maxsize is not None and not isinstance(maxsize, int):
-        raise TypeError('Expected maxsize to be an integer or None')
-
-    def decorating_function(user_function):
-        wrapper = _lru_cache_wrapper(user_function, maxsize, typed)
-        return update_wrapper(wrapper, user_function)
-
-    return decorating_function
-
-def _lru_cache_wrapper(user_function, maxsize, typed):
-    # Constants shared by all lru cache instances:
-    sentinel = object()          # unique object used to signal cache misses
-    make_key = _make_key         # build a key from the function arguments
-    PREV, NEXT, KEY, RESULT = 0, 1, 2, 3   # names for the link fields
-
-    cache = {}
-    cache_get = cache.get    # bound method to lookup a key or return None
-    lock = RLock()           # because linkedlist updates aren't threadsafe
-    root = []                # root of the circular doubly linked list
-    root[:] = [root, root, None, None]     # initialize by pointing to self
-    hits_misses_full_root = [0, 0, False, root]
-    HITS,MISSES,FULL,ROOT = 0, 1, 2, 3
-
-    if maxsize == 0:
-
-        def wrapper(*args, **kwds):
-            # No caching -- just a statistics update after a successful call
-            result = user_function(*args, **kwds)
-            hits_misses_full_root[MISSES] += 1
-            return result
-
-    elif maxsize is None:
-
-        def wrapper(*args, **kwds):
-            # Simple caching without ordering or size limit
-            key = make_key(args, kwds, typed)
-            result = cache_get(key, sentinel)
-            if result is not sentinel:
-                hits_misses_full_root[HITS] += 1
-                return result
-            result = user_function(*args, **kwds)
-            cache[key] = result
-            hits_misses_full_root[MISSES] += 1
-            return result
-
-    else:
-
-        def wrapper(*args, **kwds):
-            # Size limited caching that tracks accesses by recency
-            key = make_key(args, kwds, typed)
-            lock.acquire()
-            try:
-                link = cache_get(key)
-                if link is not None:
-                    # Move the link to the front of the circular queue
-                    root = hits_misses_full_root[ROOT]
-                    link_prev, link_next, _key, result = link
-                    link_prev[NEXT] = link_next
-                    link_next[PREV] = link_prev
-                    last = root[PREV]
-                    last[NEXT] = root[PREV] = link
-                    link[PREV] = last
-                    link[NEXT] = root
-                    hits_misses_full_root[HITS] += 1
-                    return result
-            finally:
-                lock.release()
-            result = user_function(*args, **kwds)
-            lock.acquire()
-            try:
-                if key in cache:
-                    # Getting here means that this same key was added to the
-                    # cache while the lock was released.  Since the link
-                    # update is already done, we need only return the
-                    # computed result and update the count of misses.
-                    pass
-                elif hits_misses_full_root[FULL]:
-                    # Use the old root to store the new key and result.
-                    oldroot = root = hits_misses_full_root[ROOT]
-                    oldroot[KEY] = key
-                    oldroot[RESULT] = result
-                    # Empty the oldest link and make it the new root.
-                    # Keep a reference to the old key and old result to
-                    # prevent their ref counts from going to zero during the
-                    # update. That will prevent potentially arbitrary object
-                    # clean-up code (i.e. __del__) from running while we're
-                    # still adjusting the links.
-                    root = hits_misses_full_root[ROOT] = oldroot[NEXT]
-                    oldkey = root[KEY]
-                    oldresult = root[RESULT]
-                    root[KEY] = root[RESULT] = None
-                    # Now update the cache dictionary.
-                    del cache[oldkey]
-                    # Save the potentially reentrant cache[key] assignment
-                    # for last, after the root and links have been put in
-                    # a consistent state.
-                    cache[key] = oldroot
-                else:
-                    # Put result in a new link at the front of the queue.
-                    root = hits_misses_full_root[ROOT]
-                    last = root[PREV]
-                    link = [last, root, key, result]
-                    last[NEXT] = root[PREV] = cache[key] = link
-                    # Use the __len__() method instead of the len() function
-                    # which could potentially be wrapped in an lru_cache itself.
-                    hits_misses_full_root[FULL] = (cache.__len__() >= maxsize)
-                hits_misses_full_root[MISSES]
-            finally:
-                lock.release()
-            return result
-
-    def cache_clear():
-        """Clear the cache and cache statistics"""
-        lock.acquire()
-        try:
-            cache.clear()
-            root = hits_misses_full_root[ROOT]
-            root[:] = [root, root, None, None]
-            hits_misses_full[HITS] = 0
-            hits_misses_full[MISSES] = 0
-            hits_misses_full[FULL] = False
-        finally:
-            lock.release()
-
-    wrapper.cache_clear = cache_clear
-    return wrapper

mitogen/compat/pkgutil.py

@@ -1,5 +1,7 @@
 """Utilities to support packages."""
 
+# !mitogen: minify_safe
+
 # NOTE: This module must remain compatible with Python 2.3, as it is shared
 # by setuptools for distribution with Python 2.3 and up.
 

mitogen/compat/tokenize.py

@@ -22,6 +22,8 @@ are the same, except instead of generating tokens, tokeneater is a callback
 function to which the 5 fields described above are passed as 5 arguments,
 each time a new token is found."""
 
+# !mitogen: minify_safe
+
 __author__ = 'Ka-Ping Yee <ping@lfw.org>'
 __credits__ = ('GvR, ESR, Tim Peters, Thomas Wouters, Fred Drake, '
                'Skip Montanaro, Raymond Hettinger')
@@ -392,8 +394,11 @@ def generate_tokens(readline):
                    (initial == '.' and token != '.'):      # ordinary number
                     yield (NUMBER, token, spos, epos, line)
                 elif initial in '\r\n':
-                    yield (NL if parenlev > 0 else NEWLINE,
-                           token, spos, epos, line)
+                    if parenlev > 0:
+                        n = NL
+                    else:
+                        n = NEWLINE
+                    yield (n, token, spos, epos, line)
                 elif initial == '#':
                     assert not token.endswith("\n")
                     yield (COMMENT, token, spos, epos, line)

mitogen/debug.py

@@ -26,6 +26,8 @@
 # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 # POSSIBILITY OF SUCH DAMAGE.
 
+# !mitogen: minify_safe
+
 """
 Basic signal handler for dumping thread stacks.
 """
@@ -99,10 +101,6 @@ def get_router_info():
     }
 
 
-def get_router_info(router):
-    pass
-
-
 def get_stream_info(router_id):
     router = get_routers().get(router_id)
     return {

mitogen/doas.py

@@ -26,8 +26,9 @@
 # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 # POSSIBILITY OF SUCH DAMAGE.
 
+# !mitogen: minify_safe
+
 import logging
-import os
 
 import mitogen.core
 import mitogen.parent
@@ -45,10 +46,6 @@ class Stream(mitogen.parent.Stream):
     create_child = staticmethod(mitogen.parent.hybrid_tty_create_child)
     child_is_immediate_subprocess = False
 
-    #: Once connected, points to the corresponding DiagLogStream, allowing it
-    #: to be disconnected at the same time this stream is being torn down.
-    tty_stream = None
-
     username = 'root'
     password = None
     doas_path = 'doas'
@@ -71,13 +68,8 @@ class Stream(mitogen.parent.Stream):
         if incorrect_prompts is not None:
             self.incorrect_prompts = map(str.lower, incorrect_prompts)
 
-    def connect(self):
-        super(Stream, self).connect()
-        self.name = u'doas.' + mitogen.core.to_text(self.username)
-
-    def on_disconnect(self, broker):
-        self.tty_stream.on_disconnect(broker)
-        super(Stream, self).on_disconnect(broker)
+    def _get_name(self):
+        return u'doas.' + mitogen.core.to_text(self.username)
 
     def get_boot_command(self):
         bits = [self.doas_path, '-u', self.username, '--']
@@ -88,15 +80,8 @@ class Stream(mitogen.parent.Stream):
     password_incorrect_msg = 'doas password is incorrect'
     password_required_msg = 'doas password is required'
 
-    def _connect_bootstrap(self, extra_fd):
-        self.tty_stream = mitogen.parent.DiagLogStream(extra_fd, self)
-
+    def _connect_input_loop(self, it):
         password_sent = False
-        it = mitogen.parent.iter_read(
-            fds=[self.receive_side.fd, extra_fd],
-            deadline=self.connect_deadline,
-        )
-
         for buf in it:
             LOG.debug('%r: received %r', self, buf)
             if buf.endswith(self.EC0_MARKER):
@@ -111,8 +96,18 @@ class Stream(mitogen.parent.Stream):
                 if password_sent:
                     raise PasswordError(self.password_incorrect_msg)
                 LOG.debug('sending password')
-                self.tty_stream.transmit_side.write(
+                self.diag_stream.transmit_side.write(
                     mitogen.core.to_text(self.password + '\n').encode('utf-8')
                 )
                 password_sent = True
         raise mitogen.core.StreamError('bootstrap failed')
+
+    def _connect_bootstrap(self):
+        it = mitogen.parent.iter_read(
+            fds=[self.receive_side.fd, self.diag_stream.receive_side.fd],
+            deadline=self.connect_deadline,
+        )
+        try:
+            self._connect_input_loop(it)
+        finally:
+            it.close()

mitogen/docker.py

@@ -26,6 +26,8 @@
 # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 # POSSIBILITY OF SUCH DAMAGE.
 
+# !mitogen: minify_safe
+
 import logging
 
 import mitogen.core
@@ -62,9 +64,8 @@ class Stream(mitogen.parent.Stream):
         if username:
             self.username = username
 
-    def connect(self):
-        super(Stream, self).connect()
-        self.name = u'docker.' + (self.container or self.image)
+    def _get_name(self):
+        return u'docker.' + (self.container or self.image)
 
     def get_boot_command(self):
         args = ['--interactive']

mitogen/fakessh.py

@@ -26,6 +26,8 @@
 # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 # POSSIBILITY OF SUCH DAMAGE.
 
+# !mitogen: minify_safe
+
 """
 :mod:`mitogen.fakessh` is a stream implementation that starts a subprocess with
 its environment modified such that ``PATH`` searches for `ssh` return a Mitogen

mitogen/fork.py

@@ -26,6 +26,8 @@
 # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 # POSSIBILITY OF SUCH DAMAGE.
 
+# !mitogen: minify_safe
+
 import logging
 import os
 import random
@@ -39,6 +41,18 @@ import mitogen.parent
 
 LOG = logging.getLogger('mitogen')
 
+# Python 2.4/2.5 cannot support fork+threads whatsoever, it doesn't even fix up
+# interpreter state. So 2.4/2.5 interpreters start .local() contexts for
+# isolation instead. Since we don't have any crazy memory sharing problems to
+# avoid, there is no virginal fork parent either. The child is started directly
+# from the login/become process. In future this will be default everywhere,
+# fork is brainwrong from the stone age.
+FORK_SUPPORTED = sys.version_info >= (2, 6)
+
+
+class Error(mitogen.core.StreamError):
+    pass
+
 
 def fixup_prngs():
     """
@@ -85,6 +99,10 @@ def on_fork():
     mitogen.core.Latch._on_fork()
     mitogen.core.Side._on_fork()
 
+    mitogen__service = sys.modules.get('mitogen.service')
+    if mitogen__service:
+        mitogen__service._pool_lock = threading.Lock()
+
 
 def handle_child_crash():
     """
@@ -109,9 +127,19 @@ class Stream(mitogen.parent.Stream):
     #: User-supplied function for cleaning up child process state.
     on_fork = None
 
+    python_version_msg = (
+        "The mitogen.fork method is not supported on Python versions "
+        "prior to 2.6, since those versions made no attempt to repair "
+        "critical interpreter state following a fork. Please use the "
+        "local() method instead."
+    )
+
     def construct(self, old_router, max_message_size, on_fork=None,
                   debug=False, profiling=False, unidirectional=False,
                   on_start=None):
+        if not FORK_SUPPORTED:
+            raise Error(self.python_version_msg)
+
         # fork method only supports a tiny subset of options.
         super(Stream, self).construct(max_message_size=max_message_size,
                                       debug=debug, profiling=profiling,
@@ -180,14 +208,15 @@ class Stream(mitogen.parent.Stream):
             config['on_start'] = self.on_start
 
         try:
-            mitogen.core.ExternalContext(config).main()
-        except Exception:
-            # TODO: report exception somehow.
-            os._exit(72)
+            try:
+                mitogen.core.ExternalContext(config).main()
+            except Exception:
+                # TODO: report exception somehow.
+                os._exit(72)
         finally:
             # Don't trigger atexit handlers, they were copied from the parent.
             os._exit(0)
 
-    def _connect_bootstrap(self, extra_fd):
+    def _connect_bootstrap(self):
         # None required.
         pass

mitogen/jail.py

@@ -26,6 +26,8 @@
 # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 # POSSIBILITY OF SUCH DAMAGE.
 
+# !mitogen: minify_safe
+
 import logging
 
 import mitogen.core
@@ -52,9 +54,8 @@ class Stream(mitogen.parent.Stream):
         if jexec_path:
             self.jexec_path = jexec_path
 
-    def connect(self):
-        super(Stream, self).connect()
-        self.name = u'jail.' + self.container
+    def _get_name(self):
+        return u'jail.' + self.container
 
     def get_boot_command(self):
         bits = [self.jexec_path]

mitogen/kubectl.py

@@ -1,5 +1,4 @@
-# coding: utf-8
-# Copyright 2018, Yannig Perré
+# Copyright 2018, Yannig Perre
 #
 # Redistribution and use in source and binary forms, with or without
 # modification, are permitted provided that the following conditions are met:
@@ -27,6 +26,8 @@
 # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 # POSSIBILITY OF SUCH DAMAGE.
 
+# !mitogen: minify_safe
+
 import logging
 
 import mitogen.core
@@ -56,9 +57,8 @@ class Stream(mitogen.parent.Stream):
             self.kubectl_path = kubectl_path
         self.kubectl_args = kubectl_args or []
 
-    def connect(self):
-        super(Stream, self).connect()
-        self.name = u'kubectl.%s%s' % (self.pod, self.kubectl_args)
+    def _get_name(self):
+        return u'kubectl.%s%s' % (self.pod, self.kubectl_args)
 
     def get_boot_command(self):
         bits = [self.kubectl_path] + self.kubectl_args + ['exec', '-it', self.pod]

mitogen/lxc.py

@@ -26,6 +26,8 @@
 # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 # POSSIBILITY OF SUCH DAMAGE.
 
+# !mitogen: minify_safe
+
 import logging
 
 import mitogen.core
@@ -48,15 +50,20 @@ class Stream(mitogen.parent.Stream):
     container = None
     lxc_attach_path = 'lxc-attach'
 
+    eof_error_hint = (
+        'Note: many versions of LXC do not report program execution failure '
+        'meaningfully. Please check the host logs (/var/log) for more '
+        'information.'
+    )
+
     def construct(self, container, lxc_attach_path=None, **kwargs):
         super(Stream, self).construct(**kwargs)
         self.container = container
         if lxc_attach_path:
             self.lxc_attach_path = lxc_attach_path
 
-    def connect(self):
-        super(Stream, self).connect()
-        self.name = u'lxc.' + self.container
+    def _get_name(self):
+        return u'lxc.' + self.container
 
     def get_boot_command(self):
         bits = [

mitogen/lxd.py

@@ -26,6 +26,8 @@
 # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 # POSSIBILITY OF SUCH DAMAGE.
 
+# !mitogen: minify_safe
+
 import logging
 
 import mitogen.core
@@ -49,15 +51,20 @@ class Stream(mitogen.parent.Stream):
     lxc_path = 'lxc'
     python_path = 'python'
 
+    eof_error_hint = (
+        'Note: many versions of LXC do not report program execution failure '
+        'meaningfully. Please check the host logs (/var/log) for more '
+        'information.'
+    )
+
     def construct(self, container, lxc_path=None, **kwargs):
         super(Stream, self).construct(**kwargs)
         self.container = container
         if lxc_path:
             self.lxc_path = lxc_path
 
-    def connect(self):
-        super(Stream, self).connect()
-        self.name = u'lxd.' + self.container
+    def _get_name(self):
+        return u'lxd.' + self.container
 
     def get_boot_command(self):
         bits = [

mitogen/master.py

@@ -26,6 +26,8 @@
 # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 # POSSIBILITY OF SUCH DAMAGE.
 
+# !mitogen: minify_safe
+
 """
 This module implements functionality required by master processes, such as
 starting new contexts via SSH. Its size is also restricted, since it must
@@ -43,6 +45,7 @@ import pkgutil
 import re
 import string
 import sys
+import time
 import threading
 import types
 import zlib
@@ -58,13 +61,25 @@ import mitogen.minify
 import mitogen.parent
 
 from mitogen.core import b
-from mitogen.core import to_text
-from mitogen.core import LOG
 from mitogen.core import IOLOG
+from mitogen.core import LOG
+from mitogen.core import str_partition
+from mitogen.core import str_rpartition
+from mitogen.core import to_text
 
 imap = getattr(itertools, 'imap', map)
 izip = getattr(itertools, 'izip', zip)
 
+try:
+    any
+except NameError:
+    from mitogen.core import any
+
+try:
+    next
+except NameError:
+    from mitogen.core import next
+
 
 RLOG = logging.getLogger('mitogen.ctx')
 
@@ -127,7 +142,7 @@ def get_child_modules(path):
     return [to_text(name) for _, name, _ in it]
 
 
-def get_core_source():
+def _get_core_source():
     """
     Master version of parent.get_core_source().
     """
@@ -137,31 +152,30 @@ def get_core_source():
 
 if mitogen.is_master:
     # TODO: find a less surprising way of installing this.
-    mitogen.parent.get_core_source = get_core_source
+    mitogen.parent._get_core_source = _get_core_source
 
 
 LOAD_CONST = dis.opname.index('LOAD_CONST')
 IMPORT_NAME = dis.opname.index('IMPORT_NAME')
 
 
+def _getarg(nextb, c):
+    if c >= dis.HAVE_ARGUMENT:
+        return nextb() | (nextb() << 8)
+
+
 if sys.version_info < (3, 0):
     def iter_opcodes(co):
         # Yield `(op, oparg)` tuples from the code object `co`.
         ordit = imap(ord, co.co_code)
         nextb = ordit.next
-        return ((c, (None
-                     if c < dis.HAVE_ARGUMENT else
-                     (nextb() | (nextb() << 8))))
-                    for c in ordit)
+        return ((c, _getarg(nextb, c)) for c in ordit)
 elif sys.version_info < (3, 6):
     def iter_opcodes(co):
         # Yield `(op, oparg)` tuples from the code object `co`.
         ordit = iter(co.co_code)
         nextb = ordit.__next__
-        return ((c, (None
-                     if c < dis.HAVE_ARGUMENT else
-                     (nextb() | (nextb() << 8))))
-                    for c in ordit)
+        return ((c, _getarg(nextb, c)) for c in ordit)
 else:
     def iter_opcodes(co):
         # Yield `(op, oparg)` tuples from the code object `co`.
@@ -172,9 +186,10 @@ else:
 
 
 def scan_code_imports(co):
-    """Given a code object `co`, scan its bytecode yielding any
-    ``IMPORT_NAME`` and associated prior ``LOAD_CONST`` instructions
-    representing an `Import` statement or `ImportFrom` statement.
+    """
+    Given a code object `co`, scan its bytecode yielding any ``IMPORT_NAME``
+    and associated prior ``LOAD_CONST`` instructions representing an `Import`
+    statement or `ImportFrom` statement.
 
     :return:
         Generator producing `(level, modname, namelist)` tuples, where:
@@ -188,6 +203,7 @@ def scan_code_imports(co):
     """
     opit = iter_opcodes(co)
     opit, opit2, opit3 = itertools.tee(opit, 3)
+
     try:
         next(opit2)
         next(opit3)
@@ -195,14 +211,22 @@ def scan_code_imports(co):
     except StopIteration:
         return
 
-    for oparg1, oparg2, (op3, arg3) in izip(opit, opit2, opit3):
-        if op3 == IMPORT_NAME:
-            op2, arg2 = oparg2
-            op1, arg1 = oparg1
-            if op1 == op2 == LOAD_CONST:
-                yield (co.co_consts[arg1],
-                       co.co_names[arg3],
-                       co.co_consts[arg2] or ())
+    if sys.version_info >= (2, 5):
+        for oparg1, oparg2, (op3, arg3) in izip(opit, opit2, opit3):
+            if op3 == IMPORT_NAME:
+                op2, arg2 = oparg2
+                op1, arg1 = oparg1
+                if op1 == op2 == LOAD_CONST:
+                    yield (co.co_consts[arg1],
+                           co.co_names[arg3],
+                           co.co_consts[arg2] or ())
+    else:
+        # Python 2.4 did not yet have 'level', so stack format differs.
+        for oparg1, (op2, arg2) in izip(opit, opit2):
+            if op2 == IMPORT_NAME:
+                op1, arg1 = oparg1
+                if op1 == LOAD_CONST:
+                    yield (-1, co.co_names[arg2], co.co_consts[arg1] or ())
 
 
 class ThreadWatcher(object):
@@ -324,17 +348,32 @@ class LogForwarder(object):
             self._cache[msg.src_id] = logger = logging.getLogger(name)
 
         name, level_s, s = msg.data.decode('latin1').split('\x00', 2)
-        logger.log(int(level_s), '%s: %s', name, s, extra={
-            'mitogen_message': s,
-            'mitogen_context': self._router.context_by_id(msg.src_id),
-            'mitogen_name': name,
-        })
+
+        # See logging.Handler.makeRecord()
+        record = logging.LogRecord(
+            name=logger.name,
+            level=int(level_s),
+            pathname='(unknown file)',
+            lineno=0,
+            msg=('%s: %s' % (name, s)),
+            args=(),
+            exc_info=None,
+        )
+        record.mitogen_message = s
+        record.mitogen_context = self._router.context_by_id(msg.src_id)
+        record.mitogen_name = name
+        logger.handle(record)
 
     def __repr__(self):
         return 'LogForwarder(%r)' % (self._router,)
 
 
 class ModuleFinder(object):
+    """
+    Given the name of a loaded module, make a best-effort attempt at finding
+    related modules likely needed by a child context requesting the original
+    module.
+    """
     def __init__(self):
         #: Import machinery is expensive, keep :py:meth`:get_module_source`
         #: results around.
@@ -372,9 +411,37 @@ class ModuleFinder(object):
         if os.path.exists(path) and self._looks_like_script(path):
             return path
 
+    def _get_main_module_defective_python_3x(self, fullname):
+        """
+        Recent versions of Python 3.x introduced an incomplete notion of
+        importer specs, and in doing so created permanent asymmetry in the
+        :mod:`pkgutil` interface handling for the `__main__` module. Therefore
+        we must handle `__main__` specially.
+        """
+        if fullname != '__main__':
+            return None
+
+        mod = sys.modules.get(fullname)
+        if not mod:
+            return None
+
+        path = getattr(mod, '__file__', None)
+        if not (os.path.exists(path) and self._looks_like_script(path)):
+            return None
+
+        fp = open(path, 'rb')
+        try:
+            source = fp.read()
+        finally:
+            fp.close()
+
+        return path, source, False
+
     def _get_module_via_pkgutil(self, fullname):
-        """Attempt to fetch source code via pkgutil. In an ideal world, this
-        would be the only required implementation of get_module()."""
+        """
+        Attempt to fetch source code via pkgutil. In an ideal world, this would
+        be the only required implementation of get_module().
+        """
         try:
             # Pre-'import spec' this returned None, in Python3.6 it raises
             # ImportError.
@@ -448,8 +515,70 @@ class ModuleFinder(object):
 
         return path, source, is_pkg
 
-    get_module_methods = [_get_module_via_pkgutil,
-                          _get_module_via_sys_modules]
+    def _get_module_via_parent_enumeration(self, fullname):
+        """
+        Attempt to fetch source code by examining the module's (hopefully less
+        insane) parent package. Required for older versions of
+        ansible.compat.six and plumbum.colors.
+        """
+        if fullname not in sys.modules:
+            # Don't attempt this unless a module really exists in sys.modules,
+            # else we could return junk.
+            return
+
+        pkgname, _, modname = str_rpartition(to_text(fullname), u'.')
+        pkg = sys.modules.get(pkgname)
+        if pkg is None or not hasattr(pkg, '__file__'):
+            return
+
+        pkg_path = os.path.dirname(pkg.__file__)
+        try:
+            fp, path, ext = imp.find_module(modname, [pkg_path])
+            try:
+                path = self._py_filename(path)
+                if not path:
+                    fp.close()
+                    return
+
+                source = fp.read()
+            finally:
+                if fp:
+                    fp.close()
+
+            if isinstance(source, mitogen.core.UnicodeType):
+                # get_source() returns "string" according to PEP-302, which was
+                # reinterpreted for Python 3 to mean a Unicode string.
+                source = source.encode('utf-8')
+            return path, source, False
+        except ImportError:
+            e = sys.exc_info()[1]
+            LOG.debug('imp.find_module(%r, %r) -> %s', modname, [pkg_path], e)
+
+    def add_source_override(self, fullname, path, source, is_pkg):
+        """
+        Explicitly install a source cache entry, preventing usual lookup
+        methods from being used.
+
+        Beware the value of `path` is critical when `is_pkg` is specified,
+        since it directs where submodules are searched for.
+
+        :param str fullname:
+            Name of the module to override.
+        :param str path:
+            Module's path as it will appear in the cache.
+        :param bytes source:
+            Module source code as a bytestring.
+        :param bool is_pkg:
+            :data:`True` if the module is a package.
+        """
+        self._found_cache[fullname] = (path, source, is_pkg)
+
+    get_module_methods = [
+        _get_main_module_defective_python_3x,
+        _get_module_via_pkgutil,
+        _get_module_via_sys_modules,
+        _get_module_via_parent_enumeration,
+    ]
 
     def get_module_source(self, fullname):
         """Given the name of a loaded module `fullname`, attempt to find its
@@ -466,6 +595,7 @@ class ModuleFinder(object):
         for method in self.get_module_methods:
             tup = method(self, fullname)
             if tup:
+                #LOG.debug('%r returned %r', method, tup)
                 break
         else:
             tup = None, None, None
@@ -477,7 +607,8 @@ class ModuleFinder(object):
     def resolve_relpath(self, fullname, level):
         """Given an ImportFrom AST node, guess the prefix that should be tacked
         on to an alias name to produce a canonical name. `fullname` is the name
-        of the module in which the ImportFrom appears."""
+        of the module in which the ImportFrom appears.
+        """
         mod = sys.modules.get(fullname, None)
         if hasattr(mod, '__path__'):
             fullname += '.__init__'
@@ -494,12 +625,12 @@ class ModuleFinder(object):
 
     def generate_parent_names(self, fullname):
         while '.' in fullname:
-            fullname, _, _ = fullname.rpartition('.')
+            fullname, _, _ = str_rpartition(to_text(fullname), u'.')
             yield fullname
 
     def find_related_imports(self, fullname):
         """
-        Return a list of non-stdlb modules that are directly imported by
+        Return a list of non-stdlib modules that are directly imported by
         `fullname`, plus their parents.
 
         The list is determined by retrieving the source code of
@@ -537,7 +668,7 @@ class ModuleFinder(object):
 
         return self._related_cache.setdefault(fullname, sorted(
             set(
-                name
+                mitogen.core.to_text(name)
                 for name in maybe_names
                 if sys.modules.get(name) is not None
                 and not is_stdlib_name(name)
@@ -550,8 +681,8 @@ class ModuleFinder(object):
         Return a list of non-stdlib modules that are imported directly or
         indirectly by `fullname`, plus their parents.
 
-        This method is like :py:meth:`on_disconect`, but it also recursively
-        searches any modules which are imported by `fullname`.
+        This method is like :py:meth:`find_related_imports`, but also
+        recursively searches any modules which are imported by `fullname`.
 
         :param fullname: Fully qualified name of an _already imported_ module
             for which source code can be retrieved
@@ -563,7 +694,7 @@ class ModuleFinder(object):
         while stack:
             name = stack.pop(0)
             names = self.find_related_imports(name)
-            stack.extend(set(names).difference(found, stack))
+            stack.extend(set(names).difference(set(found).union(stack)))
             found.update(names)
 
         found.discard(fullname)
@@ -577,6 +708,23 @@ class ModuleResponder(object):
         self._cache = {}  # fullname -> pickled
         self.blacklist = []
         self.whitelist = ['']
+
+        #: Context -> set([fullname, ..])
+        self._forwarded_by_context = {}
+
+        #: Number of GET_MODULE messages received.
+        self.get_module_count = 0
+        #: Total time spent in uncached GET_MODULE.
+        self.get_module_secs = 0.0
+        #: Total time spent minifying modules.
+        self.minify_secs = 0.0
+        #: Number of successful LOAD_MODULE messages sent.
+        self.good_load_module_count = 0
+        #: Total bytes in successful LOAD_MODULE payloads.
+        self.good_load_module_size = 0
+        #: Number of negative LOAD_MODULE messages sent.
+        self.bad_load_module_count = 0
+
         router.add_handler(
             fn=self._on_get_module,
             handle=mitogen.core.GET_MODULE,
@@ -585,6 +733,12 @@ class ModuleResponder(object):
     def __repr__(self):
         return 'ModuleResponder(%r)' % (self._router,)
 
+    def add_source_override(self, fullname, path, source, is_pkg):
+        """
+        See :meth:`ModuleFinder.add_source_override.
+        """
+        self._finder.add_source_override(fullname, path, source, is_pkg)
+
     MAIN_RE = re.compile(b(r'^if\s+__name__\s*==\s*.__main__.\s*:'), re.M)
     main_guard_msg = (
         "A child context attempted to import __main__, however the main "
@@ -620,20 +774,39 @@ class ModuleResponder(object):
     def _make_negative_response(self, fullname):
         return (fullname, None, None, None, ())
 
-    def _build_tuple(self, fullname):
-        if mitogen.core.is_blacklisted_import(self, fullname):
-            raise ImportError('blacklisted')
+    minify_safe_re = re.compile(b(r'\s+#\s*!mitogen:\s*minify_safe'))
 
+    def _build_tuple(self, fullname):
         if fullname in self._cache:
             return self._cache[fullname]
 
+        if mitogen.core.is_blacklisted_import(self, fullname):
+            raise ImportError('blacklisted')
+
         path, source, is_pkg = self._finder.get_module_source(fullname)
+        if path and is_stdlib_path(path):
+            # Prevent loading of 2.x<->3.x stdlib modules! This costs one
+            # RTT per hit, so a client-side solution is also required.
+            LOG.debug('%r: refusing to serve stdlib module %r',
+                      self, fullname)
+            tup = self._make_negative_response(fullname)
+            self._cache[fullname] = tup
+            return tup
+
         if source is None:
-            LOG.error('_build_tuple(%r): could not locate source', fullname)
+            # TODO: make this .warning() or similar again once importer has its
+            # own logging category.
+            LOG.debug('_build_tuple(%r): could not locate source', fullname)
             tup = self._make_negative_response(fullname)
             self._cache[fullname] = tup
             return tup
 
+        if self.minify_safe_re.search(source):
+            # If the module contains a magic marker, it's safe to minify.
+            t0 = time.time()
+            source = mitogen.minify.minimize_source(source).encode('utf-8')
+            self.minify_secs += time.time() - t0
+
         if is_pkg:
             pkg_present = get_child_modules(path)
             LOG.debug('_build_tuple(%r, %r) -> %r',
@@ -662,38 +835,40 @@ class ModuleResponder(object):
 
     def _send_load_module(self, stream, fullname):
         if fullname not in stream.sent_modules:
-            LOG.debug('_send_load_module(%r, %r)', stream, fullname)
-            self._router._async_route(
-                mitogen.core.Message.pickled(
-                    self._build_tuple(fullname),
-                    dst_id=stream.remote_id,
-                    handle=mitogen.core.LOAD_MODULE,
-                )
+            tup = self._build_tuple(fullname)
+            msg = mitogen.core.Message.pickled(
+                tup,
+                dst_id=stream.remote_id,
+                handle=mitogen.core.LOAD_MODULE,
             )
+            LOG.debug('%s: sending module %s (%.2f KiB)',
+                      stream.name, fullname, len(msg.data) / 1024.0)
+            self._router._async_route(msg)
             stream.sent_modules.add(fullname)
+            if tup[2] is not None:
+                self.good_load_module_count += 1
+                self.good_load_module_size += len(msg.data)
+            else:
+                self.bad_load_module_count += 1
 
     def _send_module_load_failed(self, stream, fullname):
+        self.bad_load_module_count += 1
         stream.send(
             mitogen.core.Message.pickled(
-                (fullname, None, None, None, ()),
+                self._make_negative_response(fullname),
                 dst_id=stream.remote_id,
                 handle=mitogen.core.LOAD_MODULE,
             )
         )
 
     def _send_module_and_related(self, stream, fullname):
+        if fullname in stream.sent_modules:
+            return
+
         try:
             tup = self._build_tuple(fullname)
-            if tup[2] and is_stdlib_path(tup[2]):
-                # Prevent loading of 2.x<->3.x stdlib modules! This costs one
-                # RTT per hit, so a client-side solution is also required.
-                LOG.warning('%r: refusing to serve stdlib module %r',
-                            self, fullname)
-                self._send_module_load_failed(stream, fullname)
-                return
-
             for name in tup[4]:  # related
-                parent, _, _ = name.partition('.')
+                parent, _, _ = str_partition(name, '.')
                 if parent != fullname and parent not in stream.sent_modules:
                     # Parent hasn't been sent, so don't load submodule yet.
                     continue
@@ -709,13 +884,18 @@ class ModuleResponder(object):
             return
 
         LOG.debug('%r._on_get_module(%r)', self, msg.data)
+        self.get_module_count += 1
         stream = self._router.stream_by_id(msg.src_id)
         fullname = msg.data.decode()
         if fullname in stream.sent_modules:
             LOG.warning('_on_get_module(): dup request for %r from %r',
                         fullname, stream)
 
-        self._send_module_and_related(stream, fullname)
+        t0 = time.time()
+        try:
+            self._send_module_and_related(stream, fullname)
+        finally:
+            self.get_module_secs += time.time() - t0
 
     def _send_forward_module(self, stream, context, fullname):
         if stream.remote_id != context.context_id:
@@ -728,26 +908,59 @@ class ModuleResponder(object):
             )
 
     def _forward_one_module(self, context, fullname):
+        forwarded = self._forwarded_by_context.get(context)
+        if forwarded is None:
+            forwarded = set()
+            self._forwarded_by_context[context] = forwarded
+
+        if fullname in forwarded:
+            return
+
         path = []
         while fullname:
             path.append(fullname)
-            fullname, _, _ = fullname.rpartition('.')
+            fullname, _, _ = str_rpartition(fullname, u'.')
+
+        stream = self._router.stream_by_id(context.context_id)
+        if stream is None:
+            LOG.debug('%r: dropping forward of %s to no longer existent '
+                      '%r', self, path[0], context)
+            return
 
         for fullname in reversed(path):
-            stream = self._router.stream_by_id(context.context_id)
             self._send_module_and_related(stream, fullname)
             self._send_forward_module(stream, context, fullname)
 
     def _forward_modules(self, context, fullnames):
         IOLOG.debug('%r._forward_modules(%r, %r)', self, context, fullnames)
         for fullname in fullnames:
-            self._forward_one_module(context, fullname)
+            self._forward_one_module(context, mitogen.core.to_text(fullname))
 
     def forward_modules(self, context, fullnames):
         self._router.broker.defer(self._forward_modules, context, fullnames)
 
 
 class Broker(mitogen.core.Broker):
+    """
+    .. note::
+
+        You may construct as many brokers as desired, and use the same broker
+        for multiple routers, however usually only one broker need exist.
+        Multiple brokers may be useful when dealing with sets of children with
+        differing lifetimes. For example, a subscription service where
+        non-payment results in termination for one customer.
+
+    :param bool install_watcher:
+        If :data:`True`, an additional thread is started to monitor the
+        lifetime of the main thread, triggering :meth:`shutdown`
+        automatically in case the user forgets to call it, or their code
+        crashed.
+
+        You should not rely on this functionality in your program, it is only
+        intended as a fail-safe and to simplify the API for new users. In
+        particular, alternative Python implementations may not be able to
+        support watching the main thread.
+    """
     shutdown_timeout = 5.0
     _watcher = None
     poller_class = mitogen.parent.PREFERRED_POLLER
@@ -767,8 +980,43 @@ class Broker(mitogen.core.Broker):
 
 
 class Router(mitogen.parent.Router):
+    """
+    Extend :class:`mitogen.core.Router` with functionality useful to masters,
+    and child contexts who later become masters. Currently when this class is
+    required, the target context's router is upgraded at runtime.
+
+    .. note::
+
+        You may construct as many routers as desired, and use the same broker
+        for multiple routers, however usually only one broker and router need
+        exist. Multiple routers may be useful when dealing with separate trust
+        domains, for example, manipulating infrastructure belonging to separate
+        customers or projects.
+
+    :param mitogen.master.Broker broker:
+        Broker to use. If not specified, a private :class:`Broker` is created.
+
+    :param int max_message_size:
+        Override the maximum message size this router is willing to receive or
+        transmit. Any value set here is automatically inherited by any children
+        created by the router.
+
+        This has a liberal default of 128 MiB, but may be set much lower.
+        Beware that setting it below 64KiB may encourage unexpected failures as
+        parents and children can no longer route large Python modules that may
+        be required by your application.
+    """
+
     broker_class = Broker
-    profiling = False
+
+    #: When :data:`True`, cause the broker thread and any subsequent broker and
+    #: main threads existing in any child to write
+    #: ``/tmp/mitogen.stats.<pid>.<thread_name>.log`` containing a
+    #: :mod:`cProfile` dump on graceful exit. Must be set prior to construction
+    #: of any :class:`Broker`, e.g. via::
+    #:
+    #:      mitogen.master.Router.profiling = True
+    profiling = os.environ.get('MITOGEN_PROFILING') is not None
 
     def __init__(self, broker=None, max_message_size=None):
         if broker is None:
@@ -789,7 +1037,67 @@ class Router(mitogen.parent.Router):
             persist=True,
         )
 
+    def _on_broker_exit(self):
+        super(Router, self)._on_broker_exit()
+        dct = self.get_stats()
+        dct['self'] = self
+        dct['minify_ms'] = 1000 * dct['minify_secs']
+        dct['get_module_ms'] = 1000 * dct['get_module_secs']
+        dct['good_load_module_size_kb'] = dct['good_load_module_size'] / 1024.0
+        dct['good_load_module_size_avg'] = (
+            (
+                dct['good_load_module_size'] /
+                (float(dct['good_load_module_count']) or 1.0)
+            ) / 1024.0
+        )
+
+        LOG.debug(
+            '%(self)r: stats: '
+                '%(get_module_count)d module requests in '
+                '%(get_module_ms)d ms, '
+                '%(good_load_module_count)d sent '
+                '(%(minify_ms)d ms minify time), '
+                '%(bad_load_module_count)d negative responses. '
+                'Sent %(good_load_module_size_kb).01f kb total, '
+                '%(good_load_module_size_avg).01f kb avg.'
+            % dct
+        )
+
+    def get_stats(self):
+        """
+        Return performance data for the module responder.
+
+        :returns:
+
+            Dict containing keys:
+
+            * `get_module_count`: Integer count of
+              :data:`mitogen.core.GET_MODULE` messages received.
+            * `get_module_secs`: Floating point total seconds spent servicing
+              :data:`mitogen.core.GET_MODULE` requests.
+            * `good_load_module_count`: Integer count of successful
+              :data:`mitogen.core.LOAD_MODULE` messages sent.
+            * `good_load_module_size`: Integer total bytes sent in
+              :data:`mitogen.core.LOAD_MODULE` message payloads.
+            * `bad_load_module_count`: Integer count of negative
+              :data:`mitogen.core.LOAD_MODULE` messages sent.
+            * `minify_secs`: CPU seconds spent minifying modules marked
+               minify-safe.
+        """
+        return {
+            'get_module_count': self.responder.get_module_count,
+            'get_module_secs': self.responder.get_module_secs,
+            'good_load_module_count': self.responder.good_load_module_count,
+            'good_load_module_size': self.responder.good_load_module_size,
+            'bad_load_module_count': self.responder.bad_load_module_count,
+            'minify_secs': self.responder.minify_secs,
+        }
+
     def enable_debug(self):
+        """
+        Cause this context and any descendant child contexts to write debug
+        logs to ``/tmp/mitogen.<pid>.log``.
+        """
         mitogen.core.enable_debug_logging()
         self.debug = True
 
@@ -824,6 +1132,12 @@ class IdAllocator(object):
     BLOCK_SIZE = 1000
 
     def allocate(self):
+        """
+        Arrange for a unique context ID to be allocated and associated with a
+        route leading to the active context. In masters, the ID is generated
+        directly, in children it is forwarded to the master via a
+        :data:`mitogen.core.ALLOCATE_ID` message.
+        """
         self.lock.acquire()
         try:
             id_ = self.next_id
@@ -849,8 +1163,6 @@ class IdAllocator(object):
 
         id_, last_id = self.allocate_block()
         requestee = self.router.context_by_id(msg.src_id)
-        allocated = self.router.context_by_id(id_, msg.src_id)
-
         LOG.debug('%r: allocating [%r..%r) to %r',
                   self, id_, last_id, requestee)
         msg.reply((id_, last_id))

mitogen/minify.py

@@ -26,6 +26,8 @@
 # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 # POSSIBILITY OF SUCH DAMAGE.
 
+# !mitogen: minify_safe
+
 import sys
 
 try:
@@ -40,13 +42,7 @@ if sys.version_info < (2, 7, 11):
 else:
     import tokenize
 
-try:
-    from functools import lru_cache
-except ImportError:
-    from mitogen.compat.functools import lru_cache
-
 
-@lru_cache()
 def minimize_source(source):
     """Remove comments and docstrings from Python `source`, preserving line
     numbers and syntax of empty blocks.

mitogen/profiler.py

@@ -0,0 +1,166 @@
+# Copyright 2017, David Wilson
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are met:
+#
+# 1. Redistributions of source code must retain the above copyright notice,
+# this list of conditions and the following disclaimer.
+#
+# 2. Redistributions in binary form must reproduce the above copyright notice,
+# this list of conditions and the following disclaimer in the documentation
+# and/or other materials provided with the distribution.
+#
+# 3. Neither the name of the copyright holder nor the names of its contributors
+# may be used to endorse or promote products derived from this software without
+# specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+# POSSIBILITY OF SUCH DAMAGE.
+
+# !mitogen: minify_safe
+
+"""mitogen.profiler
+    Record and report cProfile statistics from a run. Creates one aggregated
+    output file, one aggregate containing only workers, and one for the
+    top-level process.
+
+Usage:
+    mitogen.profiler record <dest_path> <tool> [args ..]
+    mitogen.profiler report <dest_path> [sort_mode]
+    mitogen.profiler stat <sort_mode> <tool> [args ..]
+
+Mode:
+    record: Record a trace.
+    report: Report on a previously recorded trace.
+    stat: Record and report in a single step.
+
+Where:
+    dest_path: Filesystem prefix to write .pstats files to.
+    sort_mode: Sorting mode; defaults to "cumulative". See:
+        https://docs.python.org/2/library/profile.html#pstats.Stats.sort_stats
+
+Example:
+    mitogen.profiler record /tmp/mypatch ansible-playbook foo.yml
+    mitogen.profiler dump /tmp/mypatch-worker.pstats
+"""
+
+from __future__ import print_function
+import os
+import pstats
+import cProfile
+import shutil
+import subprocess
+import sys
+import tempfile
+import time
+
+import mitogen.core
+
+
+def try_merge(stats, path):
+    try:
+        stats.add(path)
+        return True
+    except Exception as e:
+        print('Failed. Race? Will retry. %s' % (e,))
+        return False
+
+
+def merge_stats(outpath, inpaths):
+    first, rest = inpaths[0], inpaths[1:]
+    for x in range(5):
+        try:
+            stats = pstats.Stats(first)
+        except EOFError:
+            time.sleep(0.2)
+            continue
+
+        print("Writing %r..." % (outpath,))
+        for path in rest:
+            #print("Merging %r into %r.." % (os.path.basename(path), outpath))
+            for x in range(5):
+                if try_merge(stats, path):
+                    break
+                time.sleep(0.2)
+
+    stats.dump_stats(outpath)
+
+
+def generate_stats(outpath, tmpdir):
+    print('Generating stats..')
+    all_paths = []
+    paths_by_ident = {}
+
+    for name in os.listdir(tmpdir):
+        if name.endswith('-dump.pstats'):
+            ident, _, pid = name.partition('-')
+            path = os.path.join(tmpdir, name)
+            all_paths.append(path)
+            paths_by_ident.setdefault(ident, []).append(path)
+
+    merge_stats('%s-all.pstat' % (outpath,), all_paths)
+    for ident, paths in paths_by_ident.items():
+        merge_stats('%s-%s.pstat' % (outpath, ident), paths)
+
+
+def do_record(tmpdir, path, *args):
+    env = os.environ.copy()
+    fmt = '%(identity)s-%(pid)s.%(now)s-dump.%(ext)s'
+    env['MITOGEN_PROFILING'] = '1'
+    env['MITOGEN_PROFILE_FMT'] = os.path.join(tmpdir, fmt)
+    rc = subprocess.call(args, env=env)
+    generate_stats(path, tmpdir)
+    return rc
+
+
+def do_report(tmpdir, path, sort='cumulative'):
+    stats = pstats.Stats(path).sort_stats(sort)
+    stats.print_stats(100)
+
+
+def do_stat(tmpdir, sort, *args):
+    valid_sorts = pstats.Stats.sort_arg_dict_default
+    if sort not in valid_sorts:
+        sys.stderr.write('Invalid sort %r, must be one of %s\n' %
+                         (sort, ', '.join(sorted(valid_sorts))))
+        sys.exit(1)
+
+    outfile = os.path.join(tmpdir, 'combined')
+    do_record(tmpdir, outfile, *args)
+    aggs = ('app.main', 'mitogen.broker', 'mitogen.child_main',
+            'mitogen.service.pool', 'Strategy', 'WorkerProcess',
+            'all')
+    for agg in aggs:
+        path = '%s-%s.pstat' % (outfile, agg)
+        if os.path.exists(path):
+            print()
+            print()
+            print('------ Aggregation %r ------' % (agg,))
+            print()
+            do_report(tmpdir, path, sort)
+            print()
+
+
+def main():
+    if len(sys.argv) < 2 or sys.argv[1] not in ('record', 'report', 'stat'):
+        sys.stderr.write(__doc__)
+        sys.exit(1)
+
+    func = globals()['do_' + sys.argv[1]]
+    tmpdir = tempfile.mkdtemp(prefix='mitogen.profiler')
+    try:
+        sys.exit(func(tmpdir, *sys.argv[2:]) or 0)
+    finally:
+        shutil.rmtree(tmpdir)
+
+if __name__ == '__main__':
+    main()

mitogen/select.py

@@ -26,6 +26,8 @@
 # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 # POSSIBILITY OF SUCH DAMAGE.
 
+# !mitogen: minify_safe
+
 import mitogen.core
 
 
@@ -34,11 +36,57 @@ class Error(mitogen.core.Error):
 
 
 class Select(object):
-    notify = None
+    """
+    Support scatter/gather asynchronous calls and waiting on multiple
+    receivers, channels, and sub-Selects. Accepts a sequence of
+    :class:`mitogen.core.Receiver` or :class:`mitogen.select.Select` instances
+    and returns the first value posted to any receiver or select.
 
-    @classmethod
-    def all(cls, receivers):
-        return list(msg.unpickle() for msg in cls(receivers))
+    If `oneshot` is :data:`True`, then remove each receiver as it yields a
+    result; since :meth:`__iter__` terminates once the final receiver is
+    removed, this makes it convenient to respond to calls made in parallel:
+
+    .. code-block:: python
+
+        total = 0
+        recvs = [c.call_async(long_running_operation) for c in contexts]
+
+        for msg in mitogen.select.Select(recvs):
+            print('Got %s from %s' % (msg, msg.receiver))
+            total += msg.unpickle()
+
+        # Iteration ends when last Receiver yields a result.
+        print('Received total %s from %s receivers' % (total, len(recvs)))
+
+    :class:`Select` may drive a long-running scheduler:
+
+    .. code-block:: python
+
+        with mitogen.select.Select(oneshot=False) as select:
+            while running():
+                for msg in select:
+                    process_result(msg.receiver.context, msg.unpickle())
+                for context, workfunc in get_new_work():
+                    select.add(context.call_async(workfunc))
+
+    :class:`Select` may be nested:
+
+    .. code-block:: python
+
+        subselects = [
+            mitogen.select.Select(get_some_work()),
+            mitogen.select.Select(get_some_work()),
+            mitogen.select.Select([
+                mitogen.select.Select(get_some_work()),
+                mitogen.select.Select(get_some_work())
+            ])
+        ]
+
+        for msg in mitogen.select.Select(selects):
+            print(msg.unpickle())
+    """
+
+    notify = None
 
     def __init__(self, receivers=(), oneshot=True):
         self._receivers = []
@@ -47,14 +95,50 @@ class Select(object):
         for recv in receivers:
             self.add(recv)
 
+    @classmethod
+    def all(cls, receivers):
+        """
+        Take an iterable of receivers and retrieve a :class:`Message` from
+        each, returning the result of calling `msg.unpickle()` on each in turn.
+        Results are returned in the order they arrived.
+
+        This is sugar for handling batch :meth:`Context.call_async
+        <mitogen.parent.Context.call_async>` invocations:
+
+        .. code-block:: python
+
+            print('Total disk usage: %.02fMiB' % (sum(
+                mitogen.select.Select.all(
+                    context.call_async(get_disk_usage)
+                    for context in contexts
+                ) / 1048576.0
+            ),))
+
+        However, unlike in a naive comprehension such as:
+
+        .. code-block:: python
+
+            recvs = [c.call_async(get_disk_usage) for c in contexts]
+            sum(recv.get().unpickle() for recv in recvs)
+
+        Result processing happens in the order results arrive, rather than the
+        order requests were issued, so :meth:`all` should always be faster.
+        """
+        return list(msg.unpickle() for msg in cls(receivers))
+
     def _put(self, value):
         self._latch.put(value)
         if self.notify:
             self.notify(self)
 
     def __bool__(self):
+        """
+        Return :data:`True` if any receivers are registered with this select.
+        """
         return bool(self._receivers)
 
+    __nonzero__ = __bool__
+
     def __enter__(self):
         return self
 
@@ -62,6 +146,11 @@ class Select(object):
         self.close()
 
     def __iter__(self):
+        """
+        Yield the result of :meth:`get` until no receivers remain in the
+        select, either because `oneshot` is :data:`True`, or each receiver was
+        explicitly removed via :meth:`remove`.
+        """
         while self._receivers:
             yield self.get()
 
@@ -80,6 +169,14 @@ class Select(object):
     owned_msg = 'Cannot add: Receiver is already owned by another Select'
 
     def add(self, recv):
+        """
+        Add the :class:`mitogen.core.Receiver` or :class:`Select` `recv` to the
+        select.
+
+        :raises mitogen.select.Error:
+            An attempt was made to add a :class:`Select` to which this select
+            is indirectly a member of.
+        """
         if isinstance(recv, Select):
             recv._check_no_loop(self)
 
@@ -95,6 +192,12 @@ class Select(object):
     not_present_msg = 'Instance is not a member of this Select'
 
     def remove(self, recv):
+        """
+        Remove the :class:`mitogen.core.Receiver` or :class:`Select` `recv`
+        from the select. Note that if the receiver has notified prior to
+        :meth:`remove`, it will still be returned by a subsequent :meth:`get`.
+        This may change in a future version.
+        """
         try:
             if recv.notify != self._put:
                 raise ValueError
@@ -104,16 +207,59 @@ class Select(object):
             raise Error(self.not_present_msg)
 
     def close(self):
+        """
+        Remove the select's notifier function from each registered receiver,
+        mark the associated latch as closed, and cause any thread currently
+        sleeping in :meth:`get` to be woken with
+        :class:`mitogen.core.LatchError`.
+
+        This is necessary to prevent memory leaks in long-running receivers. It
+        is called automatically when the Python :keyword:`with` statement is
+        used.
+        """
         for recv in self._receivers[:]:
             self.remove(recv)
         self._latch.close()
 
     def empty(self):
+        """
+        Return :data:`True` if calling :meth:`get` would block.
+
+        As with :class:`Queue.Queue`, :data:`True` may be returned even though
+        a subsequent call to :meth:`get` will succeed, since a message may be
+        posted at any moment between :meth:`empty` and :meth:`get`.
+
+        :meth:`empty` may return :data:`False` even when :meth:`get` would
+        block if another thread has drained a receiver added to this select.
+        This can be avoided by only consuming each receiver from a single
+        thread.
+        """
         return self._latch.empty()
 
     empty_msg = 'Cannot get(), Select instance is empty'
 
     def get(self, timeout=None, block=True):
+        """
+        Fetch the next available value from any receiver, or raise
+        :class:`mitogen.core.TimeoutError` if no value is available within
+        `timeout` seconds.
+
+        On success, the message's :attr:`receiver
+        <mitogen.core.Message.receiver>` attribute is set to the receiver.
+
+        :param float timeout:
+            Timeout in seconds.
+        :param bool block:
+            If :data:`False`, immediately raise
+            :class:`mitogen.core.TimeoutError` if the select is empty.
+        :return:
+            :class:`mitogen.core.Message`
+        :raises mitogen.core.TimeoutError:
+            Timeout was reached.
+        :raises mitogen.core.LatchError:
+            :meth:`close` has been called, and the underlying latch is no
+            longer valid.
+        """
         if not self._receivers:
             raise Error(self.empty_msg)
 

mitogen/service.py

@@ -26,6 +26,8 @@
 # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 # POSSIBILITY OF SUCH DAMAGE.
 
+# !mitogen: minify_safe
+
 import grp
 import os
 import os.path
@@ -40,6 +42,16 @@ import mitogen.core
 import mitogen.select
 from mitogen.core import b
 from mitogen.core import LOG
+from mitogen.core import str_rpartition
+
+try:
+    all
+except NameError:
+    def all(it):
+        for elem in it:
+            if not elem:
+                return False
+        return True
 
 
 DEFAULT_POOL_SIZE = 16
@@ -52,9 +64,13 @@ _pool_lock = threading.Lock()
 if mitogen.core.PY3:
     def func_code(func):
         return func.__code__
+    def func_name(func):
+        return func.__name__
 else:
     def func_code(func):
         return func.func_code
+    def func_name(func):
+        return func.func_name
 
 
 @mitogen.core.takes_router
@@ -64,7 +80,12 @@ def get_or_create_pool(size=None, router=None):
     _pool_lock.acquire()
     try:
         if _pool_pid != os.getpid():
-            _pool = Pool(router, [], size=size or DEFAULT_POOL_SIZE)
+            _pool = Pool(router, [], size=size or DEFAULT_POOL_SIZE,
+                         overwrite=True)
+            # In case of Broker shutdown crash, Pool can cause 'zombie'
+            # processes.
+            mitogen.core.listen(router.broker, 'shutdown',
+                                lambda: _pool.stop(join=False))
             _pool_pid = os.getpid()
         return _pool
     finally:
@@ -187,7 +208,7 @@ class Activator(object):
     )
 
     def activate(self, pool, service_name, msg):
-        mod_name, _, class_name = service_name.rpartition('.')
+        mod_name, _, class_name = str_rpartition(service_name, '.')
         if msg and not self.is_permitted(mod_name, class_name, msg):
             raise mitogen.core.CallError(self.not_active_msg, service_name)
 
@@ -244,7 +265,7 @@ class Invoker(object):
             if no_reply:
                 LOG.exception('While calling no-reply method %s.%s',
                               type(self.service).__name__,
-                              method.func_name)
+                              func_name(method))
             else:
                 raise
 
@@ -395,11 +416,13 @@ class Service(object):
         Called when a message arrives on any of :attr:`select`'s registered
         receivers.
         """
+        pass
 
     def on_shutdown(self):
         """
         Called by Pool.shutdown() once the last worker thread has exitted.
         """
+        pass
 
 
 class Pool(object):
@@ -426,12 +449,13 @@ class Pool(object):
     """
     activator_class = Activator
 
-    def __init__(self, router, services, size=1):
+    def __init__(self, router, services, size=1, overwrite=False):
         self.router = router
         self._activator = self.activator_class()
         self._receiver = mitogen.core.Receiver(
             router=router,
             handle=mitogen.core.CALL_SERVICE,
+            overwrite=overwrite,
         )
 
         self._select = mitogen.select.Select(oneshot=False)
@@ -449,7 +473,7 @@ class Pool(object):
             thread = threading.Thread(
                 name=name,
                 target=mitogen.core._profile_hook,
-                args=(name, self._worker_main),
+                args=('mitogen.service.pool', self._worker_main),
             )
             thread.start()
             self._threads.append(thread)
@@ -473,6 +497,7 @@ class Pool(object):
 
     def stop(self, join=True):
         self.closed = True
+        self._receiver.close()
         self._select.close()
         if join:
             self.join()
@@ -533,7 +558,7 @@ class Pool(object):
                 msg = self._select.get()
             except (mitogen.core.ChannelError, mitogen.core.LatchError):
                 e = sys.exc_info()[1]
-                LOG.info('%r: channel or latch closed, exitting: %s', self, e)
+                LOG.debug('%r: channel or latch closed, exitting: %s', self, e)
                 return
 
             func = self._func_by_recv[msg.receiver]
@@ -547,7 +572,7 @@ class Pool(object):
             self._worker_run()
         except Exception:
             th = threading.currentThread()
-            LOG.exception('%r: worker %r crashed', self, th.name)
+            LOG.exception('%r: worker %r crashed', self, th.getName())
             raise
 
     def __repr__(self):
@@ -555,7 +580,7 @@ class Pool(object):
         return 'mitogen.service.Pool(%#x, size=%d, th=%r)' % (
             id(self),
             len(self._threads),
-            th.name,
+            th.getName(),
         )
 
 
@@ -588,6 +613,7 @@ class PushFileService(Service):
         self._sent_by_stream = {}
 
     def get(self, path):
+        assert isinstance(path, mitogen.core.UnicodeType)
         self._lock.acquire()
         try:
             if path in self._cache:
@@ -614,7 +640,7 @@ class PushFileService(Service):
                 path=path,
                 context=context
             ).close()
-        else:
+        elif path not in sent:
             child.call_service_async(
                 service_name=self.name(),
                 method_name='store_and_forward',
@@ -622,6 +648,7 @@ class PushFileService(Service):
                 data=self._cache[path],
                 context=context
             ).close()
+            sent.add(path)
 
     @expose(policy=AllowParents())
     @arg_spec({
@@ -635,7 +662,7 @@ class PushFileService(Service):
         with a set of small files and Python modules.
         """
         for path in paths:
-            self.propagate_to(context, path)
+            self.propagate_to(context, mitogen.core.to_text(path))
         self.router.responder.forward_modules(context, modules)
 
     @expose(policy=AllowParents())
@@ -664,7 +691,7 @@ class PushFileService(Service):
     @expose(policy=AllowParents())
     @no_reply()
     @arg_spec({
-        'path': mitogen.core.FsPathTypes,
+        'path': mitogen.core.UnicodeType,
         'data': mitogen.core.Blob,
         'context': mitogen.core.Context,
     })
@@ -688,7 +715,7 @@ class PushFileService(Service):
         if path not in self._cache:
             LOG.error('%r: %r is not in local cache', self, path)
             return
-        self._forward(path, context)
+        self._forward(context, path)
 
 
 class FileService(Service):
@@ -743,7 +770,7 @@ class FileService(Service):
            proceed normally, without the associated thread needing to be
            forcefully killed.
     """
-    unregistered_msg = 'Path is not registered with FileService.'
+    unregistered_msg = 'Path %r is not registered with FileService.'
     context_mismatch_msg = 'sender= kwarg context must match requestee context'
 
     #: Burst size. With 1MiB and 10ms RTT max throughput is 100MiB/sec, which
@@ -752,8 +779,10 @@ class FileService(Service):
 
     def __init__(self, router):
         super(FileService, self).__init__(router)
-        #: Mapping of registered path -> file size.
-        self._metadata_by_path = {}
+        #: Set of registered paths.
+        self._paths = set()
+        #: Set of registered directory prefixes.
+        self._prefixes = set()
         #: Mapping of Stream->FileStreamState.
         self._state_by_stream = {}
 
@@ -770,26 +799,43 @@ class FileService(Service):
     def register(self, path):
         """
         Authorize a path for access by children. Repeat calls with the same
-        path is harmless.
+        path has no effect.
 
         :param str path:
             File path.
         """
-        if path in self._metadata_by_path:
-            return
+        if path not in self._paths:
+            LOG.debug('%r: registering %r', self, path)
+            self._paths.add(path)
+
+    @expose(policy=AllowParents())
+    @arg_spec({
+        'path': mitogen.core.FsPathTypes,
+    })
+    def register_prefix(self, path):
+        """
+        Authorize a path and any subpaths for access by children. Repeat calls
+        with the same path has no effect.
+
+        :param str path:
+            File path.
+        """
+        if path not in self._prefixes:
+            LOG.debug('%r: registering prefix %r', self, path)
+            self._prefixes.add(path)
 
+    def _generate_stat(self, path):
         st = os.stat(path)
         if not stat.S_ISREG(st.st_mode):
             raise IOError('%r is not a regular file.' % (path,))
 
-        LOG.debug('%r: registering %r', self, path)
-        self._metadata_by_path[path] = {
-            'size': st.st_size,
-            'mode': st.st_mode,
-            'owner': self._name_or_none(pwd.getpwuid, 0, 'pw_name'),
-            'group': self._name_or_none(grp.getgrgid, 0, 'gr_name'),
-            'mtime': st.st_mtime,
-            'atime': st.st_atime,
+        return {
+            u'size': st.st_size,
+            u'mode': st.st_mode,
+            u'owner': self._name_or_none(pwd.getpwuid, 0, 'pw_name'),
+            u'group': self._name_or_none(grp.getgrgid, 0, 'gr_name'),
+            u'mtime': float(st.st_mtime),  # Python 2.4 uses int.
+            u'atime': float(st.st_atime),  # Python 2.4 uses int.
         }
 
     def on_shutdown(self):
@@ -841,6 +887,24 @@ class FileService(Service):
                 fp.close()
                 state.jobs.pop(0)
 
+    def _prefix_is_authorized(self, path):
+        """
+        Return the set of all possible directory prefixes for `path`.
+        :func:`os.path.abspath` is used to ensure the path is absolute.
+
+        :param str path:
+            The path.
+        :returns: Set of prefixes.
+        """
+        path = os.path.abspath(path)
+        while True:
+            if path in self._prefixes:
+                return True
+            if path == '/':
+                break
+            path = os.path.dirname(path)
+        return False
+
     @expose(policy=AllowAny())
     @no_reply()
     @arg_spec({
@@ -867,26 +931,33 @@ class FileService(Service):
         :raises Error:
             Unregistered path, or Sender did not match requestee context.
         """
-        if path not in self._metadata_by_path:
-            raise Error(self.unregistered_msg)
+        if path not in self._paths and not self._prefix_is_authorized(path):
+            msg.reply(mitogen.core.CallError(
+                Error(self.unregistered_msg % (path,))
+            ))
+            return
+
         if msg.src_id != sender.context.context_id:
-            raise Error(self.context_mismatch_msg)
+            msg.reply(mitogen.core.CallError(
+                Error(self.context_mismatch_msg)
+            ))
+            return
 
         LOG.debug('Serving %r', path)
+
+        # Response must arrive first so requestee can begin receive loop,
+        # otherwise first ack won't arrive until all pending chunks were
+        # delivered. In that case max BDP would always be 128KiB, aka. max
+        # ~10Mbit/sec over a 100ms link.
         try:
             fp = open(path, 'rb', self.IO_SIZE)
+            msg.reply(self._generate_stat(path))
         except IOError:
             msg.reply(mitogen.core.CallError(
                 sys.exc_info()[1]
             ))
             return
 
-        # Response must arrive first so requestee can begin receive loop,
-        # otherwise first ack won't arrive until all pending chunks were
-        # delivered. In that case max BDP would always be 128KiB, aka. max
-        # ~10Mbit/sec over a 100ms link.
-        msg.reply(self._metadata_by_path[path])
-
         stream = self.router.stream_by_id(sender.context.context_id)
         state = self._state_by_stream.setdefault(stream, FileStreamState())
         state.lock.acquire()
@@ -934,8 +1005,12 @@ class FileService(Service):
         :param bytes out_path:
             Name of the output path on the local disk.
         :returns:
-            :data:`True` on success, or :data:`False` if the transfer was
-            interrupted and the output should be discarded.
+            Tuple of (`ok`, `metadata`), where `ok` is :data:`True` on success,
+            or :data:`False` if the transfer was interrupted and the output
+            should be discarded.
+
+            `metadata` is a dictionary of file metadata as documented in
+            :meth:`fetch`.
         """
         LOG.debug('get_file(): fetching %r from %r', path, context)
         t0 = time.time()
@@ -947,6 +1022,7 @@ class FileService(Service):
             sender=recv.to_sender(),
         )
 
+        received_bytes = 0
         for chunk in recv:
             s = chunk.unpickle()
             LOG.debug('get_file(%r): received %d bytes', path, len(s))
@@ -956,11 +1032,19 @@ class FileService(Service):
                 size=len(s),
             ).close()
             out_fp.write(s)
+            received_bytes += len(s)
 
-        ok = out_fp.tell() == metadata['size']
-        if not ok:
+        ok = received_bytes == metadata['size']
+        if received_bytes < metadata['size']:
             LOG.error('get_file(%r): receiver was closed early, controller '
-                      'is likely shutting down.', path)
+                      'may be shutting down, or the file was truncated '
+                      'during transfer. Expected %d bytes, received %d.',
+                      path, metadata['size'], received_bytes)
+        elif received_bytes > metadata['size']:
+            LOG.error('get_file(%r): the file appears to have grown '
+                      'while transfer was in progress. Expected %d '
+                      'bytes, received %d.',
+                      path, metadata['size'], received_bytes)
 
         LOG.debug('target.get_file(): fetched %d bytes of %r from %r in %dms',
                   metadata['size'], path, context, 1000 * (time.time() - t0))

mitogen/setns.py

@@ -26,6 +26,8 @@
 # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 # POSSIBILITY OF SUCH DAMAGE.
 
+# !mitogen: minify_safe
+
 import ctypes
 import grp
 import logging
@@ -69,7 +71,7 @@ def _run_command(args):
 
     output, _ = proc.communicate()
     if not proc.returncode:
-        return output
+        return output.decode('utf-8', 'replace')
 
     raise Error("%s exitted with status %d: %s",
                 mitogen.parent.Argv(args), proc.returncode, output)
@@ -223,11 +225,14 @@ class Stream(mitogen.parent.Stream):
     def create_child(self, args):
         return mitogen.parent.create_child(args, preexec_fn=self.preexec_fn)
 
+    def _get_name(self):
+        return u'setns.' + self.container
+
     def connect(self):
+        self.name = self._get_name()
         attr, func = self.GET_LEADER_BY_KIND[self.kind]
         tool_path = getattr(self, attr)
         self.leader_pid = func(tool_path, self.container)
         LOG.debug('Leader PID for %s container %r: %d',
                   self.kind, self.container, self.leader_pid)
         super(Stream, self).connect()
-        self.name = u'setns.' + self.container

mitogen/ssh.py

@@ -26,12 +26,14 @@
 # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 # POSSIBILITY OF SUCH DAMAGE.
 
+# !mitogen: minify_safe
+
 """
 Functionality to allow establishing new slave contexts over an SSH connection.
 """
 
 import logging
-import time
+import re
 
 try:
     from shlex import quote as shlex_quote
@@ -40,16 +42,28 @@ except ImportError:
 
 import mitogen.parent
 from mitogen.core import b
+from mitogen.core import bytes_partition
+
+try:
+    any
+except NameError:
+    from mitogen.core import any
 
 
 LOG = logging.getLogger('mitogen')
 
 # sshpass uses 'assword' because it doesn't lowercase the input.
 PASSWORD_PROMPT = b('password')
-PERMDENIED_PROMPT = b('permission denied')
 HOSTKEY_REQ_PROMPT = b('are you sure you want to continue connecting (yes/no)?')
 HOSTKEY_FAIL = b('host key verification failed.')
 
+# [user@host: ] permission denied
+PERMDENIED_RE = re.compile(
+    ('(?:[^@]+@[^:]+: )?'  # Absent in OpenSSH <7.5
+     'Permission denied').encode(),
+    re.I
+)
+
 
 DEBUG_PREFIXES = (b('debug1:'), b('debug2:'), b('debug3:'))
 
@@ -85,18 +99,19 @@ def filter_debug(stream, it):
                     # interesting token from above or the bootstrap
                     # ('password', 'MITO000\n').
                     break
-                elif buf.startswith(DEBUG_PREFIXES):
+                elif any(buf.startswith(p) for p in DEBUG_PREFIXES):
                     state = 'in_debug'
                 else:
                     state = 'in_plain'
             elif state == 'in_debug':
                 if b('\n') not in buf:
                     break
-                line, _, buf = buf.partition(b('\n'))
-                LOG.debug('%r: %s', stream, line.rstrip())
+                line, _, buf = bytes_partition(buf, b('\n'))
+                LOG.debug('%s: %s', stream.name,
+                          mitogen.core.to_text(line.rstrip()))
                 state = 'start_of_line'
             elif state == 'in_plain':
-                line, nl, buf = buf.partition(b('\n'))
+                line, nl, buf = bytes_partition(buf, b('\n'))
                 yield line + nl, not (nl or buf)
                 if nl:
                     state = 'start_of_line'
@@ -120,10 +135,6 @@ class Stream(mitogen.parent.Stream):
     #: Number of -v invocations to pass on command line.
     ssh_debug_level = 0
 
-    #: If batch_mode=False, points to the corresponding DiagLogStream, allowing
-    #: it to be disconnected at the same time this stream is being torn down.
-    tty_stream = None
-
     #: The path to the SSH binary.
     ssh_path = 'ssh'
 
@@ -188,11 +199,6 @@ class Stream(mitogen.parent.Stream):
                 'stderr_pipe': True,
             }
 
-    def on_disconnect(self, broker):
-        if self.tty_stream is not None:
-            self.tty_stream.on_disconnect(broker)
-        super(Stream, self).on_disconnect(broker)
-
     def get_boot_command(self):
         bits = [self.ssh_path]
         if self.ssh_debug_level:
@@ -235,11 +241,11 @@ class Stream(mitogen.parent.Stream):
         base = super(Stream, self).get_boot_command()
         return bits + [shlex_quote(s).strip() for s in base]
 
-    def connect(self):
-        super(Stream, self).connect()
-        self.name = u'ssh.' + mitogen.core.to_text(self.hostname)
+    def _get_name(self):
+        s = u'ssh.' + mitogen.core.to_text(self.hostname)
         if self.port:
-            self.name += u':%s' % (self.port,)
+            s += u':%s' % (self.port,)
+        return s
 
     auth_incorrect_msg = 'SSH authentication is incorrect'
     password_incorrect_msg = 'SSH password is incorrect'
@@ -257,8 +263,8 @@ class Stream(mitogen.parent.Stream):
 
     def _host_key_prompt(self):
         if self.check_host_keys == 'accept':
-            LOG.debug('%r: accepting host key', self)
-            self.tty_stream.transmit_side.write(b('y\n'))
+            LOG.debug('%s: accepting host key', self.name)
+            self.diag_stream.transmit_side.write(b('yes\n'))
             return
 
         # _host_key_prompt() should never be reached with ignore or enforce
@@ -266,22 +272,10 @@ class Stream(mitogen.parent.Stream):
         # with ours.
         raise HostKeyError(self.hostkey_config_msg)
 
-    def _ec0_received(self):
-        if self.tty_stream is not None:
-            self._router.broker.start_receive(self.tty_stream)
-        return super(Stream, self)._ec0_received()
-
-    def _connect_bootstrap(self, extra_fd):
-        fds = [self.receive_side.fd]
-        if extra_fd is not None:
-            self.tty_stream = mitogen.parent.DiagLogStream(extra_fd, self)
-            fds.append(extra_fd)
-
-        it = mitogen.parent.iter_read(fds=fds, deadline=self.connect_deadline)
-
+    def _connect_input_loop(self, it):
         password_sent = False
         for buf, partial in filter_debug(self, it):
-            LOG.debug('%r: received %r', self, buf)
+            LOG.debug('%s: stdout: %s', self.name, buf.rstrip())
             if buf.endswith(self.EC0_MARKER):
                 self._ec0_received()
                 return
@@ -289,11 +283,7 @@ class Stream(mitogen.parent.Stream):
                 self._host_key_prompt()
             elif HOSTKEY_FAIL in buf.lower():
                 raise HostKeyError(self.hostkey_failed_msg)
-            elif buf.lower().startswith((
-                    PERMDENIED_PROMPT,
-                    b("%s@%s: " % (self.username, self.hostname))
-                        + PERMDENIED_PROMPT,
-                )):
+            elif PERMDENIED_RE.match(buf):
                 # issue #271: work around conflict with user shell reporting
                 # 'permission denied' e.g. during chdir($HOME) by only matching
                 # it at the start of the line.
@@ -307,10 +297,21 @@ class Stream(mitogen.parent.Stream):
             elif partial and PASSWORD_PROMPT in buf.lower():
                 if self.password is None:
                     raise PasswordError(self.password_required_msg)
-                LOG.debug('%r: sending password', self)
-                self.tty_stream.transmit_side.write(
+                LOG.debug('%s: sending password', self.name)
+                self.diag_stream.transmit_side.write(
                     (self.password + '\n').encode()
                 )
                 password_sent = True
 
         raise mitogen.core.StreamError('bootstrap failed')
+
+    def _connect_bootstrap(self):
+        fds = [self.receive_side.fd]
+        if self.diag_stream is not None:
+            fds.append(self.diag_stream.receive_side.fd)
+
+        it = mitogen.parent.iter_read(fds=fds, deadline=self.connect_deadline)
+        try:
+            self._connect_input_loop(it)
+        finally:
+            it.close()

mitogen/su.py

@@ -26,13 +26,19 @@
 # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 # POSSIBILITY OF SUCH DAMAGE.
 
+# !mitogen: minify_safe
+
 import logging
-import os
 
 import mitogen.core
 import mitogen.parent
 from mitogen.core import b
 
+try:
+    any
+except NameError:
+    from mitogen.core import any
+
 
 LOG = logging.getLogger(__name__)
 
@@ -60,6 +66,7 @@ class Stream(mitogen.parent.Stream):
         b('su: sorry'),                    # BSD
         b('su: authentication failure'),   # Linux
         b('su: incorrect password'),       # CentOS 6
+        b('authentication is denied'),     # AIX
     )
 
     def construct(self, username=None, password=None, su_path=None,
@@ -76,12 +83,8 @@ class Stream(mitogen.parent.Stream):
         if incorrect_prompts is not None:
             self.incorrect_prompts = map(str.lower, incorrect_prompts)
 
-    def connect(self):
-        super(Stream, self).connect()
-        self.name = u'su.' + mitogen.core.to_text(self.username)
-
-    def on_disconnect(self, broker):
-        super(Stream, self).on_disconnect(broker)
+    def _get_name(self):
+        return u'su.' + mitogen.core.to_text(self.username)
 
     def get_boot_command(self):
         argv = mitogen.parent.Argv(super(Stream, self).get_boot_command())
@@ -90,12 +93,8 @@ class Stream(mitogen.parent.Stream):
     password_incorrect_msg = 'su password is incorrect'
     password_required_msg = 'su password is required'
 
-    def _connect_bootstrap(self, extra_fd):
+    def _connect_input_loop(self, it):
         password_sent = False
-        it = mitogen.parent.iter_read(
-            fds=[self.receive_side.fd],
-            deadline=self.connect_deadline,
-        )
 
         for buf in it:
             LOG.debug('%r: received %r', self, buf)
@@ -115,4 +114,15 @@ class Stream(mitogen.parent.Stream):
                     mitogen.core.to_text(self.password + '\n').encode('utf-8')
                 )
                 password_sent = True
+
         raise mitogen.core.StreamError('bootstrap failed')
+
+    def _connect_bootstrap(self):
+        it = mitogen.parent.iter_read(
+            fds=[self.receive_side.fd],
+            deadline=self.connect_deadline,
+        )
+        try:
+            self._connect_input_loop(it)
+        finally:
+            it.close()

mitogen/sudo.py

@@ -26,10 +26,12 @@
 # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 # POSSIBILITY OF SUCH DAMAGE.
 
+# !mitogen: minify_safe
+
+import base64
 import logging
 import optparse
-import os
-import time
+import re
 
 import mitogen.core
 import mitogen.parent
@@ -37,6 +39,73 @@ from mitogen.core import b
 
 
 LOG = logging.getLogger(__name__)
+
+# These are base64-encoded UTF-8 as our existing minifier/module server
+# struggles with Unicode Python source in some (forgotten) circumstances.
+PASSWORD_PROMPTS = [
+    'cGFzc3dvcmQ=',                                              # english
+    'bG96aW5rYQ==',                                              # sr@latin.po
+    '44OR44K544Ov44O844OJ',                                      # ja.po
+    '4Kaq4Ka+4Ka44KaT4Kef4Ka+4Kaw4KeN4Kah',                      # bn.po
+    '2YPZhNmF2Kkg2KfZhNiz2LE=',                                  # ar.po
+    'cGFzYWhpdHph',                                              # eu.po
+    '0L/QsNGA0L7Qu9GM',                                          # uk.po
+    'cGFyb29s',                                                  # et.po
+    'c2FsYXNhbmE=',                                              # fi.po
+    '4Kiq4Ki+4Ki44Ki14Kiw4Kih',                                  # pa.po
+    'Y29udHJhc2lnbm8=',                                          # ia.po
+    'Zm9jYWwgZmFpcmU=',                                          # ga.po
+    '16HXodee15Q=',                                              # he.po
+    '4Kqq4Kq+4Kq44Kq14Kqw4KuN4Kqh',                              # gu.po
+    '0L/QsNGA0L7Qu9Cw',                                          # bg.po
+    '4Kyq4K2N4Kyw4Kys4K2H4Ky2IOCsuOCsmeCtjeCsleCth+CspA==',      # or.po
+    '4K6V4K6f4K614K+B4K6a4K+N4K6a4K+K4K6y4K+N',                  # ta.po
+    'cGFzc3dvcnQ=',                                              # de.po
+    '7JWU7Zi4',                                                  # ko.po
+    '0LvQvtC30LjQvdC60LA=',                                      # sr.po
+    'beG6rXQga2jhuql1',                                          # vi.po
+    'c2VuaGE=',                                                  # pt_BR.po
+    'cGFzc3dvcmQ=',                                              # it.po
+    'aGVzbG8=',                                                  # cs.po
+    '5a+G56K877ya',                                              # zh_TW.po
+    'aGVzbG8=',                                                  # sk.po
+    '4LC44LCC4LCV4LGH4LCk4LCq4LCm4LCu4LGB',                      # te.po
+    '0L/QsNGA0L7Qu9GM',                                          # kk.po
+    'aGFzxYJv',                                                  # pl.po
+    'Y29udHJhc2VueWE=',                                          # ca.po
+    'Y29udHJhc2XDsWE=',                                          # es.po
+    '4LSF4LSf4LSv4LS+4LSz4LS14LS+4LSV4LWN4LSV4LWN',              # ml.po
+    'c2VuaGE=',                                                  # pt.po
+    '5a+G56CB77ya',                                              # zh_CN.po
+    '4KSX4KWB4KSq4KWN4KSk4KS24KSs4KWN4KSm',                      # mr.po
+    'bMO2c2Vub3Jk',                                              # sv.po
+    '4YOe4YOQ4YOg4YOd4YOa4YOY',                                  # ka.po
+    '4KS24KSs4KWN4KSm4KSV4KWC4KSf',                              # hi.po
+    'YWRnYW5nc2tvZGU=',                                          # da.po
+    '4La74LeE4LeD4LeK4La04Lav4La6',                              # si.po
+    'cGFzc29yZA==',                                              # nb.po
+    'd2FjaHR3b29yZA==',                                          # nl.po
+    '4Kaq4Ka+4Ka44KaT4Kef4Ka+4Kaw4KeN4Kah',                      # bn_IN.po
+    'cGFyb2xh',                                                  # tr.po
+    '4LKX4LOB4LKq4LON4LKk4LKq4LKm',                              # kn.po
+    'c2FuZGk=',                                                  # id.po
+    '0L/QsNGA0L7Qu9GM',                                          # ru.po
+    'amVsc3rDsw==',                                              # hu.po
+    'bW90IGRlIHBhc3Nl',                                          # fr.po
+    'aXBoYXNpd2VkaQ==',                                          # zu.po
+    '4Z6W4Z624Z6A4Z+S4Z6Z4Z6f4Z6Y4Z+S4Z6E4Z624Z6P4Z+LwqDhn5Y=',  # km.po
+    '4KaX4KeB4Kaq4KeN4Kak4Ka24Kas4KeN4Kam',                      # as.po
+]
+
+
+PASSWORD_PROMPT_RE = re.compile(
+    u'|'.join(
+        base64.b64decode(s).decode('utf-8')
+        for s in PASSWORD_PROMPTS
+    )
+)
+
+
 PASSWORD_PROMPT = b('password')
 SUDO_OPTIONS = [
     #(False, 'bool', '--askpass', '-A')
@@ -55,7 +124,10 @@ SUDO_OPTIONS = [
     #(False, 'bool', '--list', '-l')
     #(False, 'bool', '--preserve-groups', '-P')
     #(False, 'str', '--prompt', '-p')
-    #(False, 'str', '--role', '-r')
+
+    # SELinux options. Passed through as-is.
+    (False, 'str', '--role', '-r'),
+    (False, 'str', '--type', '-t'),
 
     # These options are supplied by default by Ansible, but are ignored, as
     # sudo always runs under a TTY with Mitogen.
@@ -63,9 +135,8 @@ SUDO_OPTIONS = [
     (True, 'bool', '--non-interactive', '-n'),
 
     #(False, 'str', '--shell', '-s')
-    #(False, 'str', '--type', '-t')
     #(False, 'str', '--other-user', '-U')
-    #(False, 'str', '--user', '-u')
+    (False, 'str', '--user', '-u'),
     #(False, 'bool', '--version', '-V')
     #(False, 'bool', '--validate', '-v')
 ]
@@ -103,14 +174,17 @@ class PasswordError(mitogen.core.StreamError):
     pass
 
 
+def option(default, *args):
+    for arg in args:
+        if arg is not None:
+            return arg
+    return default
+
+
 class Stream(mitogen.parent.Stream):
     create_child = staticmethod(mitogen.parent.hybrid_tty_create_child)
     child_is_immediate_subprocess = False
 
-    #: Once connected, points to the corresponding DiagLogStream, allowing it to
-    #: be disconnected at the same time this stream is being torn down.
-    tty_stream = None
-
     sudo_path = 'sudo'
     username = 'root'
     password = None
@@ -118,32 +192,27 @@ class Stream(mitogen.parent.Stream):
     set_home = False
     login = False
 
+    selinux_role = None
+    selinux_type = None
+
     def construct(self, username=None, sudo_path=None, password=None,
                   preserve_env=None, set_home=None, sudo_args=None,
-                  login=None, **kwargs):
+                  login=None, selinux_role=None, selinux_type=None, **kwargs):
         super(Stream, self).construct(**kwargs)
         opts = parse_sudo_flags(sudo_args or [])
 
-        if username is not None:
-            self.username = username
-        if sudo_path is not None:
-            self.sudo_path = sudo_path
-        if password is not None:
-            self.password = password
-        if (preserve_env or opts.preserve_env) is not None:
-            self.preserve_env = preserve_env or opts.preserve_env
-        if (set_home or opts.set_home) is not None:
-            self.set_home = set_home or opts.set_home
-        if (login or opts.login) is not None:
-            self.login = True
-
-    def connect(self):
-        super(Stream, self).connect()
-        self.name = u'sudo.' + mitogen.core.to_text(self.username)
-
-    def on_disconnect(self, broker):
-        self.tty_stream.on_disconnect(broker)
-        super(Stream, self).on_disconnect(broker)
+        self.username = option(self.username, username, opts.user)
+        self.sudo_path = option(self.sudo_path, sudo_path)
+        self.password = password or None
+        self.preserve_env = option(self.preserve_env,
+            preserve_env, opts.preserve_env)
+        self.set_home = option(self.set_home, set_home, opts.set_home)
+        self.login = option(self.login, login, opts.login)
+        self.selinux_role = option(self.selinux_role, selinux_role, opts.role)
+        self.selinux_type = option(self.selinux_type, selinux_type, opts.type)
+
+    def _get_name(self):
+        return u'sudo.' + mitogen.core.to_text(self.username)
 
     def get_boot_command(self):
         # Note: sudo did not introduce long-format option processing until July
@@ -156,35 +225,53 @@ class Stream(mitogen.parent.Stream):
             bits += ['-H']
         if self.login:
             bits += ['-i']
+        if self.selinux_role:
+            bits += ['-r', self.selinux_role]
+        if self.selinux_type:
+            bits += ['-t', self.selinux_type]
 
-        bits = bits + super(Stream, self).get_boot_command()
+        bits = bits + ['--'] + super(Stream, self).get_boot_command()
         LOG.debug('sudo command line: %r', bits)
         return bits
 
     password_incorrect_msg = 'sudo password is incorrect'
     password_required_msg = 'sudo password is required'
 
-    def _connect_bootstrap(self, extra_fd):
-        self.tty_stream = mitogen.parent.DiagLogStream(extra_fd, self)
-
+    def _connect_input_loop(self, it):
         password_sent = False
-        it = mitogen.parent.iter_read(
-            fds=[self.receive_side.fd, extra_fd],
-            deadline=self.connect_deadline,
-        )
 
         for buf in it:
-            LOG.debug('%r: received %r', self, buf)
+            LOG.debug('%s: received %r', self.name, buf)
             if buf.endswith(self.EC0_MARKER):
                 self._ec0_received()
                 return
-            elif PASSWORD_PROMPT in buf.lower():
+
+            match = PASSWORD_PROMPT_RE.search(buf.decode('utf-8').lower())
+            if match is not None:
+                LOG.debug('%s: matched password prompt %r',
+                          self.name, match.group(0))
                 if self.password is None:
                     raise PasswordError(self.password_required_msg)
                 if password_sent:
                     raise PasswordError(self.password_incorrect_msg)
-                self.tty_stream.transmit_side.write(
-                    mitogen.core.to_text(self.password + '\n').encode('utf-8')
+                self.diag_stream.transmit_side.write(
+                    (mitogen.core.to_text(self.password) + '\n').encode('utf-8')
                 )
                 password_sent = True
+
         raise mitogen.core.StreamError('bootstrap failed')
+
+    def _connect_bootstrap(self):
+        fds = [self.receive_side.fd]
+        if self.diag_stream is not None:
+            fds.append(self.diag_stream.receive_side.fd)
+
+        it = mitogen.parent.iter_read(
+            fds=fds,
+            deadline=self.connect_deadline,
+        )
+
+        try:
+            self._connect_input_loop(it)
+        finally:
+            it.close()

mitogen/unix.py

@@ -26,6 +26,8 @@
 # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 # POSSIBILITY OF SUCH DAMAGE.
 
+# !mitogen: minify_safe
+
 """
 Permit connection of additional contexts that may act with the authority of
 this context. For now, the UNIX socket is always mode 0600, i.e. can only be
@@ -49,20 +51,30 @@ from mitogen.core import LOG
 def is_path_dead(path):
     s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
     try:
-        s.connect(path)
-    except socket.error:
-        e = sys.exc_info()[1]
-        return e[0] in (errno.ECONNREFUSED, errno.ENOENT)
+        try:
+            s.connect(path)
+        except socket.error:
+            e = sys.exc_info()[1]
+            return e.args[0] in (errno.ECONNREFUSED, errno.ENOENT)
+    finally:
+        s.close()
     return False
 
 
 def make_socket_path():
-    return tempfile.mktemp(prefix='mitogen_unix_')
+    return tempfile.mktemp(prefix='mitogen_unix_', suffix='.sock')
 
 
 class Listener(mitogen.core.BasicStream):
     keep_alive = True
 
+    def __repr__(self):
+        return '%s.%s(%r)' % (
+            __name__,
+            self.__class__.__name__,
+            self.path,
+        )
+
     def __init__(self, router, path=None, backlog=100):
         self._router = router
         self.path = path or make_socket_path()
@@ -78,11 +90,26 @@ class Listener(mitogen.core.BasicStream):
         self.receive_side = mitogen.core.Side(self, self._sock.fileno())
         router.broker.start_receive(self)
 
+    def _unlink_socket(self):
+        try:
+            os.unlink(self.path)
+        except OSError:
+            e = sys.exc_info()[1]
+            # Prevent a shutdown race with the parent process.
+            if e.args[0] != errno.ENOENT:
+                raise
+
+    def on_shutdown(self, broker):
+        broker.stop_receive(self)
+        self._unlink_socket()
+        self._sock.close()
+        self.receive_side.closed = True
+
     def _accept_client(self, sock):
         sock.setblocking(True)
         try:
             pid, = struct.unpack('>L', sock.recv(4))
-        except socket.error:
+        except (struct.error, socket.error):
             LOG.error('%r: failed to read remote identity: %s',
                       self, sys.exc_info()[1])
             return
@@ -102,6 +129,7 @@ class Listener(mitogen.core.BasicStream):
                       self, pid, sys.exc_info()[1])
             return
 
+        LOG.debug('%r: accepted %r', self, stream)
         stream.accept(sock.fileno(), sock.fileno())
         self._router.register(context, stream)
 

mitogen/utils.py

@@ -26,6 +26,8 @@
 # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 # POSSIBILITY OF SUCH DAMAGE.
 
+# !mitogen: minify_safe
+
 import datetime
 import logging
 import os
@@ -34,6 +36,7 @@ import sys
 import mitogen
 import mitogen.core
 import mitogen.master
+import mitogen.parent
 
 
 LOG = logging.getLogger('mitogen')
@@ -45,7 +48,33 @@ else:
     iteritems = dict.iteritems
 
 
+def setup_gil():
+    """
+    Set extremely long GIL release interval to let threads naturally progress
+    through CPU-heavy sequences without forcing the wake of another thread that
+    may contend trying to run the same CPU-heavy code. For the new-style
+    Ansible work, this drops runtime ~33% and involuntary context switches by
+    >80%, essentially making threads cooperatively scheduled.
+    """
+    try:
+        # Python 2.
+        sys.setcheckinterval(100000)
+    except AttributeError:
+        pass
+
+    try:
+        # Python 3.
+        sys.setswitchinterval(10)
+    except AttributeError:
+        pass
+
+
 def disable_site_packages():
+    """
+    Remove all entries mentioning ``site-packages`` or ``Extras`` from
+    :attr:sys.path. Used primarily for testing on OS X within a virtualenv,
+    where OS X bundles some ancient version of the :mod:`six` module.
+    """
     for entry in sys.path[:]:
         if 'site-packages' in entry or 'Extras' in entry:
             sys.path.remove(entry)
@@ -57,7 +86,9 @@ def _formatTime(record, datefmt=None):
 
 
 def log_get_formatter():
-    datefmt = '%H:%M:%S.%f'
+    datefmt = '%H:%M:%S'
+    if sys.version_info > (2, 6):
+        datefmt += '.%f'
     fmt = '%(asctime)s %(levelname).1s %(name)s: %(message)s'
     formatter = logging.Formatter(fmt, datefmt)
     formatter.formatTime = _formatTime
@@ -65,6 +96,26 @@ def log_get_formatter():
 
 
 def log_to_file(path=None, io=False, level='INFO'):
+    """
+    Install a new :class:`logging.Handler` writing applications logs to the
+    filesystem. Useful when debugging slave IO problems.
+
+    Parameters to this function may be overridden at runtime using environment
+    variables. See :ref:`logging-env-vars`.
+
+    :param str path:
+        If not :data:`None`, a filesystem path to write logs to. Otherwise,
+        logs are written to :data:`sys.stderr`.
+
+    :param bool io:
+        If :data:`True`, include extremely verbose IO logs in the output.
+        Useful for debugging hangs, less useful for debugging application code.
+
+    :param str level:
+        Name of the :mod:`logging` package constant that is the minimum level
+        to log at. Useful levels are ``DEBUG``, ``INFO``, ``WARNING``, and
+        ``ERROR``.
+    """
     log = logging.getLogger('')
     if path:
         fp = open(path, 'w', 1)
@@ -94,6 +145,14 @@ def log_to_file(path=None, io=False, level='INFO'):
 
 
 def run_with_router(func, *args, **kwargs):
+    """
+    Arrange for `func(router, *args, **kwargs)` to run with a temporary
+    :class:`mitogen.master.Router`, ensuring the Router and Broker are
+    correctly shut down during normal or exceptional return.
+
+    :returns:
+        `func`'s return value.
+    """
     broker = mitogen.master.Broker()
     router = mitogen.master.Router(broker)
     try:
@@ -104,6 +163,17 @@ def run_with_router(func, *args, **kwargs):
 
 
 def with_router(func):
+    """
+    Decorator version of :func:`run_with_router`. Example:
+
+    .. code-block:: python
+
+        @with_router
+        def do_stuff(router, arg):
+            pass
+
+        do_stuff(blah, 123)
+    """
     def wrapper(*args, **kwargs):
         return run_with_router(func, *args, **kwargs)
     if mitogen.core.PY3:
@@ -122,7 +192,27 @@ PASSTHROUGH = (
     mitogen.core.Secret,
 )
 
+
 def cast(obj):
+    """
+    Many tools love to subclass built-in types in order to implement useful
+    functionality, such as annotating the safety of a Unicode string, or adding
+    additional methods to a dict. However, cPickle loves to preserve those
+    subtypes during serialization, resulting in CallError during :meth:`call
+    <mitogen.parent.Context.call>` in the target when it tries to deserialize
+    the data.
+
+    This function walks the object graph `obj`, producing a copy with any
+    custom sub-types removed. The functionality is not default since the
+    resulting walk may be computationally expensive given a large enough graph.
+
+    See :ref:`serialization-rules` for a list of supported types.
+
+    :param obj:
+        Object to undecorate.
+    :returns:
+        Undecorated object.
+    """
     if isinstance(obj, dict):
         return dict((cast(k), cast(v)) for k, v in iteritems(obj))
     if isinstance(obj, (list, tuple)):

preamble_size.py

@@ -16,6 +16,9 @@ import mitogen.service
 import mitogen.ssh
 import mitogen.sudo
 
+import ansible_mitogen.runner
+import ansible_mitogen.target
+
 router = mitogen.master.Router()
 context = mitogen.parent.Context(router, 0)
 stream = mitogen.ssh.Stream(router, 0, max_message_size=0, hostname='foo')
@@ -31,7 +34,7 @@ if '--dump' in sys.argv:
 
 
 print(
-    '               '
+    '                           '
     ' '
     '  Original   '
     '  '
@@ -42,6 +45,9 @@ print(
 
 for mod in (
         mitogen.parent,
+        mitogen.fork,
+        ansible_mitogen.target,
+        ansible_mitogen.runner,
         mitogen.ssh,
         mitogen.sudo,
         mitogen.select,
@@ -56,7 +62,7 @@ for mod in (
     compressed = zlib.compress(minimized, 9)
     compressed_size = len(compressed)
     print(
-        '%-15s'
+        '%-25s'
         ' '
         '%5i %4.1fKiB'
         '  '

run_tests

@@ -1,23 +1,55 @@
-#/usr/bin/env bash
+#!/usr/bin/env bash
+
+# From https://unix.stackexchange.com/a/432145
+# Return the maximum of one or more integer arguments
+max() {
+    local max number
+
+    max="$1"
+
+    for number in "${@:2}"; do
+        if ((number > max)); then
+        max="$number"
+        fi
+    done
+
+    printf '%d\n' "$max"
+}
 
 echo '----- ulimits -----'
 ulimit -a
 echo '-------------------'
 echo
 
-set -o errexit
+# Don't use errexit, so coverage report is still generated when tests fail
 set -o pipefail
 
-UNIT2="$(which unit2)"
+NOCOVERAGE="${NOCOVERAGE:-}"
+NOCOVERAGE_ERASE="${NOCOVERAGE_ERASE:-$NOCOVERAGE}"
+NOCOVERAGE_REPORT="${NOCOVERAGE_REPORT:-$NOCOVERAGE}"
+
+if [ ! "$UNIT2" ]; then
+    UNIT2="$(which unit2)"
+fi
 
-coverage erase
+if [ ! "$NOCOVERAGE_ERASE" ]; then
+    coverage erase
+fi
 
 # First run overwites coverage output.
 [ "$SKIP_MITOGEN" ] || {
-    coverage run "${UNIT2}" discover \
-                            --start-directory "tests" \
-                            --pattern '*_test.py' \
-                            "$@"
+    if [ ! "$NOCOVERAGE" ]; then
+        coverage run -a "${UNIT2}" discover \
+            --start-directory "tests" \
+            --pattern '*_test.py' \
+            "$@"
+    else
+        "${UNIT2}" discover \
+            --start-directory "tests" \
+            --pattern '*_test.py' \
+            "$@"
+    fi
+    MITOGEN_TEST_STATUS=$?
 }
 
 # Second run appends. This is since 'discover' treats subdirs as packages and
@@ -27,11 +59,24 @@ coverage erase
 # mess of Git history.
 [ "$SKIP_ANSIBLE" ] || {
     export PYTHONPATH=`pwd`/tests:$PYTHONPATH
-    coverage run -a "${UNIT2}" discover \
-                            --start-directory "tests/ansible" \
-                            --pattern '*_test.py' \
-                            "$@"
+    if [ ! "$NOCOVERAGE" ]; then
+        coverage run -a "${UNIT2}" discover \
+            --start-directory "tests/ansible" \
+            --pattern '*_test.py' \
+            "$@"
+    else
+        "${UNIT2}" discover \
+            --start-directory "tests/ansible" \
+            --pattern '*_test.py' \
+            "$@"
+    fi
+    ANSIBLE_TEST_STATUS=$?
 }
 
-coverage html
-echo coverage report is at "file://$(pwd)/htmlcov/index.html"
+if [ ! "$NOCOVERAGE_REPORT" ]; then
+    coverage html
+    echo "coverage report is at file://$(pwd)/htmlcov/index.html"
+fi
+
+# Exit with a non-zero status if any test run did so
+exit "$(max $MITOGEN_TEST_STATUS $ANSIBLE_TEST_STATUS)"

scripts/pogrep.py

@@ -0,0 +1,40 @@
+
+# issue #429: tool for extracting keys out of message catalogs and turning them
+# into the big gob of base64 as used in mitogen/sudo.py
+#
+# Usage:
+#   - apt-get source libpam0g
+#   - cd */po/
+#   - python ~/pogrep.py "Password: "
+
+import sys
+import shlex
+import glob
+
+
+last_word = None
+
+for path in glob.glob('*.po'):
+    for line in open(path):
+        bits = shlex.split(line, comments=True)
+        if not bits:
+            continue
+
+        word = bits[0]
+        if len(bits) < 2 or not word:
+            continue
+
+        rest = bits[1]
+        if not rest:
+            continue
+
+        if last_word == 'msgid' and word == 'msgstr':
+            if last_rest == sys.argv[1]:
+                thing = rest.rstrip(': ').decode('utf-8').lower().encode('utf-8').encode('base64').rstrip()
+                print '    %-60s # %s' % (repr(thing)+',', path)
+
+        last_word = word
+        last_rest = rest
+
+#ag -A 1 'msgid "Password: "'|less | grep msgstr | grep -v '""'|cut -d'"' -f2|cut -d'"' -f1| tr -d :
+

setup.py

@@ -50,7 +50,6 @@ setup(
     packages = find_packages(exclude=['tests', 'examples']),
     zip_safe = False,
     classifiers = [
-        'Development Status :: 3 - Alpha',
         'Environment :: Console',
         'Intended Audience :: System Administrators',
         'License :: OSI Approved :: BSD License',