Merge remote-tracking branch 'origin/stream-refactor'

* origin/stream-refactor:
  [stream-refactor] Py3.x test fixes
  [stream-refactor] mark py24 as allow-fail
  [stream-refactor] Debian Docker container image initctl
  [stream-refactor] replace cutpaste with Stream.accept() in mitogen.unix
  [stream-refactor] fix flake8 errors
  [stream-refactor] fix testlib assertion format string
  [stream-refactor] make mitogen-fuse work on Linux
  [stream-refactor] repair preamble_size.py again
  [stream-refactor] don't abort Connection until all buffers are empty
  Normalize docstring formatting
  [stream-refactor] fix LogHandler.uncork() race
  [stream-refactor] BufferedWriter must disconenct Stream, not Protocol
  [stream-refactor] statically link doas binary using musl
  [stream-refactor] stop writing to /tmp/foo in fd_check.py.
  [stream-refactor] yet another 2.4 issue in create_child_test
  [stream-refactor] fix Py2.4 failure by implementing missing Timer method
  [stream-refactor] allow up to 30 seconds to connect in unix_test
  [stream-refactor] mark setns module as requiring Python >2.4
  [stream-refactor] another 2.4 fix for create_child_test
  .travis.yml: Add reverse shell spawn for Travis too
  core: better Side attribute docstrings
  [stream-refactor] remove one more getuser() usage
  [stream-refactor] allow doas_test to succeed on CentOS
  Pin idna==2.7 when running on Python<2.7.
  [stream-refactor] Py2.4 compat fix for iter_split_test.
  [stream-refactor] add descriptive task names to _container_prep
  [stream-refactor] 3.x socket.send() requires bytes
  [stream-refactor] fix 2.4 syntax error.
  [stream-refactor] avoid os.wait3() for Py2.4.
  Allow specifying -vvv to debops_tests.
  [stream-refactor] send MITO002 earlier
  module_finder: pass raw file to compile()
  [stream-refactor] merge stdout+stderr when reporting EofError
  [stream-refactor] fix crash in detach() / during async/multiple_items_loop.yml
  [stream-refactor] fix crash in runner/forking_active.yml
  [stream-refactor] replace old detach_popen() reference
  ansible: fixturize creation of MuxProcess
  unix: ensure mitogen.context_id is reset when client disconnects
  [stream-refactor] make syntax 2.4 compatible
  [stream-refactor] make trusty our Travis dist.
  [stream-refactor] fix su_test failure (issue #363)
  [stream-refactor] more readable log string format
  [stream-refactor] dont doubly log last partial line
  [stream-refactor] import fd_check.py used by create_child_test
  [stream-refactor] port mitogen.buildah, added to master since work began
  [stream-refactor] fix unix.Listener construction
  [stream-refactor] fix crash when no stderr present.
  [stream-refactor] fix Process constructor invocation
  Add tests/ansible/.*.pid to gitignore (for ansible_mitogen/process.py)
  Add extra/ to gitignore
  import release-notes script.
  [stream-refactor] repaired rest of create_child_test.
  [stream-refactor] rename Process attrs, fix up more create_child_test
  [stream-refactor] import incomplete create_child_test
  issue #482: tests: check for zombie process after test.
  issue #363: add test.
  tests: clean up old-style SSH exception catch
  issue #271: add mitogen__permdenied user to Docker image.
  ssh: fix issue #271 regression due to refactor, add test.
  Refactor Stream, introduce quasi-asynchronous connect, much more
  core: teach iter_split() to break on callback returning False.
  issue #507: log fatal errors to syslog.
  testlib: have LogCapturer.raw() return unicode on 2.x.
  core/master: docstring, repr, and debug log message cleanups
  parent: remove unused Timer parameter.
  tests: jail_test fixes.
  parent: docstring improvements, cfmakeraw() regression.
  core: introduce Protocol, DelimitedProtocol and BufferedWriter.
  core: introduce mitogen.core.pipe()
  tests/bench: import ssh-roundtrip.py.
  tests: note location of related tests.
  tests: add real test for doas.
  tests: install OpenBSD doas port in Debian image.
  tests: add setns_test that works if password localhost sudo works.
  Import minimal jail_test.
  core: move message encoding to Message.pack(), add+refactor tests.
  master: expect forwarded logs to be in UTF-8.
  tests: add some UTF-8 to ssh_login_banner to encourage breakage.
  core: bootstrap FD management improvements
  core: pending timers should keep broker alive.
  core: more succinct iter_split().
  core: replace UTF8_CODEC with encodings.utf_8.encode() function.
  docs: remove bytearray from supported types list.
  core: docstring style cleanups, dead code.
  testlib: disable lsof warnings due to Docker crap
  parent: discard cancelled events in TimerList.get_timeout().
  core: split out iter_split() for use in parent.py.
  parent: various style cleanups, remove unused function.
  issue #170: add TimerList docstrings.
  core: eliminate some quadratric behaviour from IoLogger
  issue #170: update Changelog; closes #170.
  issue #170: add timers to internals.rst.
  issue #170: implement timers.

.ci/debops_common_tests.py

@@ -3,6 +3,7 @@
 from __future__ import print_function
 import os
 import shutil
+import sys
 
 import ci_lib
 
@@ -68,8 +69,8 @@ with ci_lib.Fold('job_setup'):
 
 
 with ci_lib.Fold('first_run'):
-    ci_lib.run('debops common')
+    ci_lib.run('debops common %s', ' '.join(sys.argv[1:]))
 
 
 with ci_lib.Fold('second_run'):
-    ci_lib.run('debops common')
+    ci_lib.run('debops common %s', ' '.join(sys.argv[1:]))

.gitignore

@@ -9,6 +9,8 @@ venvs/**
 MANIFEST
 build/
 dist/
+extra/
+tests/ansible/.*.pid
 docs/_build/
 htmlcov/
 *.egg-info

.travis.yml

@@ -1,4 +1,5 @@
 sudo: required
+dist: trusty
 
 notifications:
   email: false
@@ -20,6 +21,7 @@ install:
 - .ci/${MODE}_install.py
 
 script:
+- .ci/spawn_reverse_shell.py
 - .ci/${MODE}_tests.py
 
 
@@ -27,6 +29,11 @@ script:
 # newest->oldest in various configuartions.
 
 matrix:
+  allow_failures:
+    # Python 2.4 tests are still unreliable
+    - language: c
+      env: MODE=mitogen_py24 DISTRO=centos5
+
   include:
     # Mitogen tests.
     # 2.4 -> 2.4

ansible_mitogen/affinity.py

@@ -177,9 +177,9 @@ class FixedPolicy(Policy):
     cores, before reusing the second hyperthread of an existing core.
 
     A hook is installed that causes :meth:`reset` to run in the child of any
-    process created with :func:`mitogen.parent.detach_popen`, ensuring
+    process created with :func:`mitogen.parent.popen`, ensuring CPU-intensive
-    CPU-intensive children like SSH are not forced to share the same core as
+    children like SSH are not forced to share the same core as the (otherwise
-    the (otherwise potentially very busy) parent.
+    potentially very busy) parent.
     """
     def __init__(self, cpu_count=None):
         #: For tests.

ansible_mitogen/module_finder.py

@@ -57,7 +57,7 @@ def get_code(module):
     """
     Compile and return a Module's code object.
     """
-    fp = open(module.path)
+    fp = open(module.path, 'rb')
     try:
         return compile(fp.read(), str(module.name), 'exec')
     finally:

ansible_mitogen/process.py

@@ -79,8 +79,15 @@ def clean_shutdown(sock):
     MuxProcess, debug logs may appear on the user's terminal *after* the prompt
     has been printed.
     """
+    try:
         sock.shutdown(socket.SHUT_WR)
+    except socket.error:
+        # Already closed. This is possible when tests are running.
+        LOG.debug('clean_shutdown: ignoring duplicate call')
+        return
+
     sock.recv(1)
+    sock.close()
 
 
 def getenv_int(key, default=0):
@@ -154,8 +161,15 @@ class MuxProcess(object):
     #: forked WorkerProcesses to contact the MuxProcess
     unix_listener_path = None
 
-    #: Singleton.
+    @classmethod
-    _instance = None
+    def _reset(cls):
+        """
+        Used to clean up in unit tests.
+        """
+        assert cls.worker_sock is not None
+        cls.worker_sock.close()
+        cls.worker_sock = None
+        os.waitpid(cls.worker_pid, 0)
 
     @classmethod
     def start(cls, _init_logging=True):
@@ -178,7 +192,7 @@ class MuxProcess(object):
         mitogen.utils.setup_gil()
         cls.unix_listener_path = mitogen.unix.make_socket_path()
         cls.worker_sock, cls.child_sock = socket.socketpair()
-        atexit.register(lambda: clean_shutdown(cls.worker_sock))
+        atexit.register(clean_shutdown, cls.worker_sock)
         mitogen.core.set_cloexec(cls.worker_sock.fileno())
         mitogen.core.set_cloexec(cls.child_sock.fileno())
 
@@ -189,8 +203,8 @@ class MuxProcess(object):
             ansible_mitogen.logging.setup()
 
         cls.original_env = dict(os.environ)
-        cls.child_pid = os.fork()
+        cls.worker_pid = os.fork()
-        if cls.child_pid:
+        if cls.worker_pid:
             save_pid('controller')
             ansible_mitogen.logging.set_process_name('top')
             ansible_mitogen.affinity.policy.assign_controller()
@@ -308,7 +322,7 @@ class MuxProcess(object):
         self._setup_responder(self.router.responder)
         mitogen.core.listen(self.broker, 'shutdown', self.on_broker_shutdown)
         mitogen.core.listen(self.broker, 'exit', self.on_broker_exit)
-        self.listener = mitogen.unix.Listener(
+        self.listener = mitogen.unix.Listener.build_stream(
             router=self.router,
             path=self.unix_listener_path,
             backlog=C.DEFAULT_FORKS,

docs/api.rst

@@ -383,6 +383,9 @@ Connection Methods
     the root PID of a running Docker, LXC, LXD, or systemd-nspawn
     container.
 
+    The setns method depends on the built-in :mod:`ctypes` module, and thus
+    does not support Python 2.4.
+
     A program is required only to find the root PID, after which management
     of the child Python interpreter is handled directly.
 

docs/changelog.rst

@@ -102,6 +102,14 @@ Fixes
   potential influx of 2.8-related bug reports.
 
 
+Core Library
+~~~~~~~~~~~~
+
+* `#170 <https://github.com/dw/mitogen/issues/170>`_: to better support child
+  process management and a future asynchronous connect implementation, a
+  :class:`mitogen.parent.TimerList` API is available.
+
+
 Thanks!
 ~~~~~~~
 

docs/getting_started.rst

@@ -341,15 +341,13 @@ The following built-in types may be used as parameters or return values in
 remote procedure calls:
 
 * :class:`bool`
-* :class:`bytearray`
+* :func:`bytes` (:class:`str` on Python 2.x)
-* :func:`bytes`
 * :class:`dict`
 * :class:`int`
 * :func:`list`
 * :class:`long`
-* :class:`str`
 * :func:`tuple`
-* :func:`unicode`
+* :func:`unicode` (:class:`str` on Python 3.x)
 
 User-defined types may not be used, except for:
 

docs/internals.rst

@@ -15,46 +15,49 @@ Constants
 .. autodata:: CHUNK_SIZE
 
 
-Poller Classes
+Pollers
-==============
+=======
 
 .. currentmodule:: mitogen.core
 .. autoclass:: Poller
     :members:
 
+.. currentmodule:: mitogen.parent
+.. autoclass:: KqueuePoller
+
 .. currentmodule:: mitogen.parent
 .. autoclass:: EpollPoller
 
 .. currentmodule:: mitogen.parent
-.. autoclass:: KqueuePoller
+.. autoclass:: PollPoller
 
 
-Latch Class
+Latch
-===========
+=====
 
 .. currentmodule:: mitogen.core
 .. autoclass:: Latch
    :members:
 
 
-PidfulStreamHandler Class
+PidfulStreamHandler
-=========================
+===================
 
 .. currentmodule:: mitogen.core
 .. autoclass:: PidfulStreamHandler
    :members:
 
 
-Side Class
+Side
-==========
+====
 
 .. currentmodule:: mitogen.core
 .. autoclass:: Side
    :members:
 
 
-Stream Classes
+Stream
-==============
+======
 
 .. currentmodule:: mitogen.core
 .. autoclass:: BasicStream
@@ -79,83 +82,84 @@ Stream Classes
 .. autoclass:: Stream
    :members:
 
-
-Other Stream Subclasses
-=======================
-
 .. currentmodule:: mitogen.core
-
 .. autoclass:: IoLogger
    :members:
 
+.. currentmodule:: mitogen.core
 .. autoclass:: Waker
    :members:
 
 
-Poller Class
+Importer
-============
+========
 
 .. currentmodule:: mitogen.core
-.. autoclass:: Poller
+.. autoclass:: Importer
    :members:
 
-.. currentmodule:: mitogen.parent
-.. autoclass:: KqueuePoller
 
-.. currentmodule:: mitogen.parent
+ModuleResponder
-.. autoclass:: EpollPoller
+===============
+
+.. currentmodule:: mitogen.master
+.. autoclass:: ModuleResponder
+   :members:
 
 
-Importer Class
+RouteMonitor
-==============
+============
 
-.. currentmodule:: mitogen.core
+.. currentmodule:: mitogen.parent
-.. autoclass:: Importer
+.. autoclass:: RouteMonitor
    :members:
 
 
-Responder Class
+TimerList
-===============
+=========
 
-.. currentmodule:: mitogen.master
+.. currentmodule:: mitogen.parent
-.. autoclass:: ModuleResponder
+.. autoclass:: TimerList
    :members:
 
 
-RouteMonitor Class
+Timer
-==================
+=====
 
 .. currentmodule:: mitogen.parent
-.. autoclass:: RouteMonitor
+.. autoclass:: Timer
    :members:
 
 
-Forwarder Class
+Forwarder
-===============
+=========
 
 .. currentmodule:: mitogen.parent
 .. autoclass:: ModuleForwarder
    :members:
 
 
-ExternalContext Class
+ExternalContext
-=====================
+===============
 
 .. currentmodule:: mitogen.core
 .. autoclass:: ExternalContext
     :members:
 
 
-mitogen.master
+Process
-==============
+=======
 
 .. currentmodule:: mitogen.parent
-.. autoclass:: ProcessMonitor
+.. autoclass:: Process
     :members:
 
 
-Blocking I/O Functions
+Helpers
-======================
+=======
+
+Blocking I/O
+------------
 
 These functions exist to support the blocking phase of setting up a new
 context. They will eventually be replaced with asynchronous equivalents.
@@ -167,8 +171,8 @@ context. They will eventually be replaced with asynchronous equivalents.
 .. autofunction:: write_all
 
 
-Subprocess Creation Functions
+Subprocess Functions
-=============================
+------------
 
 .. currentmodule:: mitogen.parent
 .. autofunction:: create_child
@@ -176,8 +180,8 @@ Subprocess Creation Functions
 .. autofunction:: tty_create_child
 
 
-Helper Functions
+Helpers
-================
+-------
 
 .. currentmodule:: mitogen.core
 .. autofunction:: to_text

examples/mitogen-fuse.py

@@ -241,9 +241,13 @@ def main(router):
         print('usage: %s <host> <mountpoint>' % sys.argv[0])
         sys.exit(1)
 
-    blerp = fuse.FUSE(
+    kwargs = {}
+    if sys.platform == 'darwin':
+        kwargs['volname'] = '%s (Mitogen)' % (sys.argv[1],)
+
+    f = fuse.FUSE(
         operations=Operations(sys.argv[1]),
         mountpoint=sys.argv[2],
         foreground=True,
-        volname='%s (Mitogen)' % (sys.argv[1],),
+        **kwargs
     )

mitogen/__init__.py

@@ -111,10 +111,10 @@ def main(log_level='INFO', profiling=_default_profiling):
         if profiling:
             mitogen.core.enable_profiling()
             mitogen.master.Router.profiling = profiling
-        utils.log_to_file(level=log_level)
+        mitogen.utils.log_to_file(level=log_level)
         return mitogen.core._profile_hook(
             'app.main',
-            utils.run_with_router,
+            mitogen.utils.run_with_router,
             func,
         )
     return wrapper

mitogen/buildah.py

@@ -37,37 +37,37 @@ import mitogen.parent
 LOG = logging.getLogger(__name__)
 
 
-class Stream(mitogen.parent.Stream):
+class Options(mitogen.parent.Options):
-    child_is_immediate_subprocess = False
-
     container = None
     username = None
     buildah_path = 'buildah'
 
-    # TODO: better way of capturing errors such as "No such container."
+    def __init__(self, container=None, buildah_path=None, username=None,
-    create_child_args = {
-        'merge_stdio': True
-    }
-
-    def construct(self, container=None,
-                  buildah_path=None, username=None,
                  **kwargs):
-        assert container or image
+        super(Options, self).__init__(**kwargs)
-        super(Stream, self).construct(**kwargs)
+        assert container is not None
-        if container:
         self.container = container
         if buildah_path:
             self.buildah_path = buildah_path
         if username:
             self.username = username
 
+
+class Connection(mitogen.parent.Connection):
+    options_class = Options
+    child_is_immediate_subprocess = False
+
+    # TODO: better way of capturing errors such as "No such container."
+    create_child_args = {
+        'merge_stdio': True
+    }
+
     def _get_name(self):
-        return u'buildah.' + self.container
+        return u'buildah.' + self.options.container
 
     def get_boot_command(self):
-        args = []
+        args = [self.options.buildah_path, 'run']
-        if self.username:
+        if self.options.username:
-            args += ['--user=' + self.username]
+            args += ['--user=' + self.options.username]
-        bits = [self.buildah_path, 'run'] + args + ['--', self.container]
+        args += ['--', self.options.container]
-
+        return args + super(Connection, self).get_boot_command()
-        return bits + super(Stream, self).get_boot_command()

mitogen/debug.py

@@ -230,7 +230,7 @@ class ContextDebugger(object):
     def _handle_debug_msg(self, msg):
         try:
             method, args, kwargs = msg.unpickle()
-            msg.reply(getattr(cls, method)(*args, **kwargs))
+            msg.reply(getattr(self, method)(*args, **kwargs))
         except Exception:
             e = sys.exc_info()[1]
             msg.reply(mitogen.core.CallError(e))

mitogen/doas.py

@@ -29,6 +29,7 @@
 # !mitogen: minify_safe
 
 import logging
+import re
 
 import mitogen.core
 import mitogen.parent
@@ -37,77 +38,106 @@ from mitogen.core import b
 
 LOG = logging.getLogger(__name__)
 
+password_incorrect_msg = 'doas password is incorrect'
+password_required_msg = 'doas password is required'
+
 
 class PasswordError(mitogen.core.StreamError):
     pass
 
 
-class Stream(mitogen.parent.Stream):
+class Options(mitogen.parent.Options):
-    create_child = staticmethod(mitogen.parent.hybrid_tty_create_child)
+    username = u'root'
-    child_is_immediate_subprocess = False
-
-    username = 'root'
     password = None
     doas_path = 'doas'
-    password_prompt = b('Password:')
+    password_prompt = u'Password:'
     incorrect_prompts = (
-        b('doas: authentication failed'),
+        u'doas: authentication failed',  # slicer69/doas
+        u'doas: Authorization failed',   # openbsd/src
     )
 
-    def construct(self, username=None, password=None, doas_path=None,
+    def __init__(self, username=None, password=None, doas_path=None,
                  password_prompt=None, incorrect_prompts=None, **kwargs):
-        super(Stream, self).construct(**kwargs)
+        super(Options, self).__init__(**kwargs)
         if username is not None:
-            self.username = username
+            self.username = mitogen.core.to_text(username)
         if password is not None:
-            self.password = password
+            self.password = mitogen.core.to_text(password)
         if doas_path is not None:
             self.doas_path = doas_path
         if password_prompt is not None:
-            self.password_prompt = password_prompt.lower()
+            self.password_prompt = mitogen.core.to_text(password_prompt)
         if incorrect_prompts is not None:
-            self.incorrect_prompts = map(str.lower, incorrect_prompts)
+            self.incorrect_prompts = [
+                mitogen.core.to_text(p)
+                for p in incorrect_prompts
+            ]
 
-    def _get_name(self):
-        return u'doas.' + mitogen.core.to_text(self.username)
 
-    def get_boot_command(self):
+class BootstrapProtocol(mitogen.parent.RegexProtocol):
-        bits = [self.doas_path, '-u', self.username, '--']
+    password_sent = False
-        bits = bits + super(Stream, self).get_boot_command()
-        LOG.debug('doas command line: %r', bits)
-        return bits
 
-    password_incorrect_msg = 'doas password is incorrect'
+    def setup_patterns(self, conn):
-    password_required_msg = 'doas password is required'
+        prompt_pattern = re.compile(
+            re.escape(conn.options.password_prompt).encode('utf-8'),
+            re.I
+        )
+        incorrect_prompt_pattern = re.compile(
+            u'|'.join(
+                re.escape(s)
+                for s in conn.options.incorrect_prompts
+            ).encode('utf-8'),
+            re.I
+        )
 
-    def _connect_input_loop(self, it):
+        self.PATTERNS = [
-        password_sent = False
+            (incorrect_prompt_pattern, type(self)._on_incorrect_password),
-        for buf in it:
+        ]
-            LOG.debug('%r: received %r', self, buf)
+        self.PARTIAL_PATTERNS = [
-            if buf.endswith(self.EC0_MARKER):
+            (prompt_pattern, type(self)._on_password_prompt),
-                self._ec0_received()
+        ]
+
+    def _on_incorrect_password(self, line, match):
+        if self.password_sent:
+            self.stream.conn._fail_connection(
+                PasswordError(password_incorrect_msg)
+            )
+
+    def _on_password_prompt(self, line, match):
+        if self.stream.conn.options.password is None:
+            self.stream.conn._fail_connection(
+                PasswordError(password_required_msg)
+            )
             return
-            if any(s in buf.lower() for s in self.incorrect_prompts):
+
-                if password_sent:
+        if self.password_sent:
-                    raise PasswordError(self.password_incorrect_msg)
+            self.stream.conn._fail_connection(
-            elif self.password_prompt in buf.lower():
+                PasswordError(password_incorrect_msg)
-                if self.password is None:
-                    raise PasswordError(self.password_required_msg)
-                if password_sent:
-                    raise PasswordError(self.password_incorrect_msg)
-                LOG.debug('sending password')
-                self.diag_stream.transmit_side.write(
-                    mitogen.core.to_text(self.password + '\n').encode('utf-8')
             )
-                password_sent = True
+            return
-        raise mitogen.core.StreamError('bootstrap failed')
 
-    def _connect_bootstrap(self):
+        LOG.debug('sending password')
-        it = mitogen.parent.iter_read(
+        self.stream.transmit_side.write(
-            fds=[self.receive_side.fd, self.diag_stream.receive_side.fd],
+            (self.stream.conn.options.password + '\n').encode('utf-8')
-            deadline=self.connect_deadline,
         )
-        try:
+        self.password_sent = True
-            self._connect_input_loop(it)
+
-        finally:
+
-            it.close()
+class Connection(mitogen.parent.Connection):
+    options_class = Options
+    diag_protocol_class = BootstrapProtocol
+
+    create_child = staticmethod(mitogen.parent.hybrid_tty_create_child)
+    child_is_immediate_subprocess = False
+
+    def _get_name(self):
+        return u'doas.' + self.options.username
+
+    def stderr_stream_factory(self):
+        stream = super(Connection, self).stderr_stream_factory()
+        stream.protocol.setup_patterns(self)
+        return stream
+
+    def get_boot_command(self):
+        bits = [self.options.doas_path, '-u', self.options.username, '--']
+        return bits + super(Connection, self).get_boot_command()

mitogen/docker.py

@@ -37,45 +37,47 @@ import mitogen.parent
 LOG = logging.getLogger(__name__)
 
 
-class Stream(mitogen.parent.Stream):
+class Options(mitogen.parent.Options):
-    child_is_immediate_subprocess = False
-
     container = None
     image = None
     username = None
-    docker_path = 'docker'
+    docker_path = u'docker'
-
-    # TODO: better way of capturing errors such as "No such container."
-    create_child_args = {
-        'merge_stdio': True
-    }
 
-    def construct(self, container=None, image=None,
+    def __init__(self, container=None, image=None, docker_path=None,
-                  docker_path=None, username=None,
+                 username=None, **kwargs):
-                  **kwargs):
+        super(Options, self).__init__(**kwargs)
         assert container or image
-        super(Stream, self).construct(**kwargs)
         if container:
-            self.container = container
+            self.container = mitogen.core.to_text(container)
         if image:
-            self.image = image
+            self.image = mitogen.core.to_text(image)
         if docker_path:
-            self.docker_path = docker_path
+            self.docker_path = mitogen.core.to_text(docker_path)
         if username:
-            self.username = username
+            self.username = mitogen.core.to_text(username)
+
+
+class Connection(mitogen.parent.Connection):
+    options_class = Options
+    child_is_immediate_subprocess = False
+
+    # TODO: better way of capturing errors such as "No such container."
+    create_child_args = {
+        'merge_stdio': True
+    }
 
     def _get_name(self):
-        return u'docker.' + (self.container or self.image)
+        return u'docker.' + (self.options.container or self.options.image)
 
     def get_boot_command(self):
         args = ['--interactive']
-        if self.username:
+        if self.options.username:
-            args += ['--user=' + self.username]
+            args += ['--user=' + self.options.username]
 
-        bits = [self.docker_path]
+        bits = [self.options.docker_path]
-        if self.container:
+        if self.options.container:
-            bits += ['exec'] + args + [self.container]
+            bits += ['exec'] + args + [self.options.container]
-        elif self.image:
+        elif self.options.image:
-            bits += ['run'] + args + ['--rm', self.image]
+            bits += ['run'] + args + ['--rm', self.options.image]
 
-        return bits + super(Stream, self).get_boot_command()
+        return bits + super(Connection, self).get_boot_command()

mitogen/fakessh.py

@@ -117,14 +117,12 @@ SSH_GETOPTS = (
 _mitogen = None
 
 
-class IoPump(mitogen.core.BasicStream):
+class IoPump(mitogen.core.Protocol):
     _output_buf = ''
     _closed = False
 
-    def __init__(self, broker, stdin_fd, stdout_fd):
+    def __init__(self, broker):
         self._broker = broker
-        self.receive_side = mitogen.core.Side(self, stdout_fd)
-        self.transmit_side = mitogen.core.Side(self, stdin_fd)
 
     def write(self, s):
         self._output_buf += s
@@ -134,13 +132,13 @@ class IoPump(mitogen.core.BasicStream):
         self._closed = True
         # If local process hasn't exitted yet, ensure its write buffer is
         # drained before lazily triggering disconnect in on_transmit.
-        if self.transmit_side.fd is not None:
+        if self.transmit_side.fp.fileno() is not None:
             self._broker._start_transmit(self)
 
-    def on_shutdown(self, broker):
+    def on_shutdown(self, stream, broker):
         self.close()
 
-    def on_transmit(self, broker):
+    def on_transmit(self, stream, broker):
         written = self.transmit_side.write(self._output_buf)
         IOLOG.debug('%r.on_transmit() -> len %r', self, written)
         if written is None:
@@ -153,8 +151,8 @@ class IoPump(mitogen.core.BasicStream):
             if self._closed:
                 self.on_disconnect(broker)
 
-    def on_receive(self, broker):
+    def on_receive(self, stream, broker):
-        s = self.receive_side.read()
+        s = stream.receive_side.read()
         IOLOG.debug('%r.on_receive() -> len %r', self, len(s))
         if s:
             mitogen.core.fire(self, 'receive', s)
@@ -163,8 +161,8 @@ class IoPump(mitogen.core.BasicStream):
 
     def __repr__(self):
         return 'IoPump(%r, %r)' % (
-            self.receive_side.fd,
+            self.receive_side.fp.fileno(),
-            self.transmit_side.fd,
+            self.transmit_side.fp.fileno(),
         )
 
 
@@ -173,14 +171,15 @@ class Process(object):
     Manages the lifetime and pipe connections of the SSH command running in the
     slave.
     """
-    def __init__(self, router, stdin_fd, stdout_fd, proc=None):
+    def __init__(self, router, stdin, stdout, proc=None):
         self.router = router
-        self.stdin_fd = stdin_fd
+        self.stdin = stdin
-        self.stdout_fd = stdout_fd
+        self.stdout = stdout
         self.proc = proc
         self.control_handle = router.add_handler(self._on_control)
         self.stdin_handle = router.add_handler(self._on_stdin)
-        self.pump = IoPump(router.broker, stdin_fd, stdout_fd)
+        self.pump = IoPump.build_stream(router.broker)
+        self.pump.accept(stdin, stdout)
         self.stdin = None
         self.control = None
         self.wake_event = threading.Event()
@@ -193,7 +192,7 @@ class Process(object):
             pmon.add(proc.pid, self._on_proc_exit)
 
     def __repr__(self):
-        return 'Process(%r, %r)' % (self.stdin_fd, self.stdout_fd)
+        return 'Process(%r, %r)' % (self.stdin, self.stdout)
 
     def _on_proc_exit(self, status):
         LOG.debug('%r._on_proc_exit(%r)', self, status)
@@ -202,12 +201,12 @@ class Process(object):
     def _on_stdin(self, msg):
         if msg.is_dead:
             IOLOG.debug('%r._on_stdin() -> %r', self, data)
-            self.pump.close()
+            self.pump.protocol.close()
             return
 
         data = msg.unpickle()
         IOLOG.debug('%r._on_stdin() -> len %d', self, len(data))
-        self.pump.write(data)
+        self.pump.protocol.write(data)
 
     def _on_control(self, msg):
         if not msg.is_dead:
@@ -279,13 +278,7 @@ def _start_slave(src_id, cmdline, router):
         stdout=subprocess.PIPE,
     )
 
-    process = Process(
+    process = Process(router, proc.stdin, proc.stdout, proc)
-        router,
-        proc.stdin.fileno(),
-        proc.stdout.fileno(),
-        proc,
-    )
-
     return process.control_handle, process.stdin_handle
 
 
@@ -361,7 +354,9 @@ def _fakessh_main(dest_context_id, econtext):
     LOG.debug('_fakessh_main: received control_handle=%r, stdin_handle=%r',
               control_handle, stdin_handle)
 
-    process = Process(econtext.router, 1, 0)
+    process = Process(econtext.router,
+                      stdin=os.fdopen(1, 'w+b', 0),
+                      stdout=os.fdopen(0, 'r+b', 0))
     process.start_master(
         stdin=mitogen.core.Sender(dest, stdin_handle),
         control=mitogen.core.Sender(dest, control_handle),
@@ -427,7 +422,7 @@ def run(dest, router, args, deadline=None, econtext=None):
 
     stream = mitogen.core.Stream(router, context_id)
     stream.name = u'fakessh'
-    stream.accept(sock1.fileno(), sock1.fileno())
+    stream.accept(sock1, sock1)
     router.register(fakessh, stream)
 
     # Held in socket buffer until process is booted.

mitogen/fork.py

@@ -28,6 +28,7 @@
 
 # !mitogen: minify_safe
 
+import errno
 import logging
 import os
 import random
@@ -37,6 +38,7 @@ import traceback
 
 import mitogen.core
 import mitogen.parent
+from mitogen.core import b
 
 
 LOG = logging.getLogger('mitogen')
@@ -119,32 +121,45 @@ def handle_child_crash():
     os._exit(1)
 
 
-class Stream(mitogen.parent.Stream):
+class Process(mitogen.parent.Process):
-    child_is_immediate_subprocess = True
+    def poll(self):
-
+        try:
+            pid, status = os.waitpid(self.pid, os.WNOHANG)
+        except OSError:
+            e = sys.exc_info()[1]
+            if e.args[0] == errno.ECHILD:
+                LOG.warn('%r: waitpid(%r) produced ECHILD', self, self.pid)
+                return
+            raise
+
+        if not pid:
+            return
+        if os.WIFEXITED(status):
+            return os.WEXITSTATUS(status)
+        elif os.WIFSIGNALED(status):
+            return -os.WTERMSIG(status)
+        elif os.WIFSTOPPED(status):
+            return -os.WSTOPSIG(status)
+
+
+class Options(mitogen.parent.Options):
     #: Reference to the importer, if any, recovered from the parent.
     importer = None
 
     #: User-supplied function for cleaning up child process state.
     on_fork = None
 
-    python_version_msg = (
+    def __init__(self, old_router, max_message_size, on_fork=None, debug=False,
-        "The mitogen.fork method is not supported on Python versions "
+                 profiling=False, unidirectional=False, on_start=None,
-        "prior to 2.6, since those versions made no attempt to repair "
+                 name=None):
-        "critical interpreter state following a fork. Please use the "
-        "local() method instead."
-    )
-
-    def construct(self, old_router, max_message_size, on_fork=None,
-                  debug=False, profiling=False, unidirectional=False,
-                  on_start=None):
         if not FORK_SUPPORTED:
             raise Error(self.python_version_msg)
 
         # fork method only supports a tiny subset of options.
-        super(Stream, self).construct(max_message_size=max_message_size,
+        super(Options, self).__init__(
-                                      debug=debug, profiling=profiling,
+            max_message_size=max_message_size, debug=debug,
-                                      unidirectional=False)
+            profiling=profiling, unidirectional=unidirectional, name=name,
+        )
         self.on_fork = on_fork
         self.on_start = on_start
 
@@ -152,17 +167,26 @@ class Stream(mitogen.parent.Stream):
         if isinstance(responder, mitogen.parent.ModuleForwarder):
             self.importer = responder.importer
 
+
+class Connection(mitogen.parent.Connection):
+    options_class = Options
+    child_is_immediate_subprocess = True
+
+    python_version_msg = (
+        "The mitogen.fork method is not supported on Python versions "
+        "prior to 2.6, since those versions made no attempt to repair "
+        "critical interpreter state following a fork. Please use the "
+        "local() method instead."
+    )
+
     name_prefix = u'fork'
 
     def start_child(self):
         parentfp, childfp = mitogen.parent.create_socketpair()
-        self.pid = os.fork()
+        pid = os.fork()
-        if self.pid:
+        if pid:
             childfp.close()
-            # Decouple the socket from the lifetime of the Python socket object.
+            return Process(pid, stdin=parentfp, stdout=parentfp)
-            fd = os.dup(parentfp.fileno())
-            parentfp.close()
-            return self.pid, fd, None
         else:
             parentfp.close()
             self._wrap_child_main(childfp)
@@ -173,12 +197,24 @@ class Stream(mitogen.parent.Stream):
         except BaseException:
             handle_child_crash()
 
+    def get_econtext_config(self):
+        config = super(Connection, self).get_econtext_config()
+        config['core_src_fd'] = None
+        config['importer'] = self.options.importer
+        config['send_ec2'] = False
+        config['setup_package'] = False
+        if self.options.on_start:
+            config['on_start'] = self.options.on_start
+        return config
+
     def _child_main(self, childfp):
         on_fork()
-        if self.on_fork:
+        if self.options.on_fork:
-            self.on_fork()
+            self.options.on_fork()
         mitogen.core.set_block(childfp.fileno())
 
+        childfp.send(b('MITO002\n'))
+
         # Expected by the ExternalContext.main().
         os.dup2(childfp.fileno(), 1)
         os.dup2(childfp.fileno(), 100)
@@ -201,23 +237,12 @@ class Stream(mitogen.parent.Stream):
         if childfp.fileno() not in (0, 1, 100):
             childfp.close()
 
-        config = self.get_econtext_config()
-        config['core_src_fd'] = None
-        config['importer'] = self.importer
-        config['setup_package'] = False
-        if self.on_start:
-            config['on_start'] = self.on_start
-
         try:
             try:
-                mitogen.core.ExternalContext(config).main()
+                mitogen.core.ExternalContext(self.get_econtext_config()).main()
             except Exception:
                 # TODO: report exception somehow.
                 os._exit(72)
         finally:
             # Don't trigger atexit handlers, they were copied from the parent.
             os._exit(0)
-
-    def _connect_bootstrap(self):
-        # None required.
-        pass

mitogen/jail.py

@@ -37,29 +37,34 @@ import mitogen.parent
 LOG = logging.getLogger(__name__)
 
 
-class Stream(mitogen.parent.Stream):
+class Options(mitogen.parent.Options):
-    child_is_immediate_subprocess = False
-    create_child_args = {
-        'merge_stdio': True
-    }
-
     container = None
     username = None
-    jexec_path = '/usr/sbin/jexec'
+    jexec_path = u'/usr/sbin/jexec'
 
-    def construct(self, container, jexec_path=None, username=None, **kwargs):
+    def __init__(self, container, jexec_path=None, username=None, **kwargs):
-        super(Stream, self).construct(**kwargs)
+        super(Options, self).__init__(**kwargs)
-        self.container = container
+        self.container = mitogen.core.to_text(container)
-        self.username = username
+        if username:
+            self.username = mitogen.core.to_text(username)
         if jexec_path:
             self.jexec_path = jexec_path
 
+
+class Connection(mitogen.parent.Connection):
+    options_class = Options
+
+    child_is_immediate_subprocess = False
+    create_child_args = {
+        'merge_stdio': True
+    }
+
     def _get_name(self):
-        return u'jail.' + self.container
+        return u'jail.' + self.options.container
 
     def get_boot_command(self):
-        bits = [self.jexec_path]
+        bits = [self.options.jexec_path]
-        if self.username:
+        if self.options.username:
-            bits += ['-U', self.username]
+            bits += ['-U', self.options.username]
-        bits += [self.container]
+        bits += [self.options.container]
-        return bits + super(Stream, self).get_boot_command()
+        return bits + super(Connection, self).get_boot_command()

mitogen/kubectl.py

@@ -37,29 +37,36 @@ import mitogen.parent
 LOG = logging.getLogger(__name__)
 
 
-class Stream(mitogen.parent.Stream):
+class Options(mitogen.parent.Options):
-    child_is_immediate_subprocess = True
-
     pod = None
     kubectl_path = 'kubectl'
     kubectl_args = None
 
-    # TODO: better way of capturing errors such as "No such container."
+    def __init__(self, pod, kubectl_path=None, kubectl_args=None, **kwargs):
-    create_child_args = {
+        super(Options, self).__init__(**kwargs)
-        'merge_stdio': True
-    }
-
-    def construct(self, pod, kubectl_path=None, kubectl_args=None, **kwargs):
-        super(Stream, self).construct(**kwargs)
         assert pod
         self.pod = pod
         if kubectl_path:
             self.kubectl_path = kubectl_path
         self.kubectl_args = kubectl_args or []
 
+
+class Connection(mitogen.parent.Connection):
+    options_class = Options
+    child_is_immediate_subprocess = True
+
+    # TODO: better way of capturing errors such as "No such container."
+    create_child_args = {
+        'merge_stdio': True
+    }
+
     def _get_name(self):
-        return u'kubectl.%s%s' % (self.pod, self.kubectl_args)
+        return u'kubectl.%s%s' % (self.options.pod, self.options.kubectl_args)
 
     def get_boot_command(self):
-        bits = [self.kubectl_path] + self.kubectl_args + ['exec', '-it', self.pod]
+        bits = [
-        return bits + ["--"] + super(Stream, self).get_boot_command()
+            self.options.kubectl_path
+        ] + self.options.kubectl_args + [
+            'exec', '-it', self.options.pod
+        ]
+        return bits + ["--"] + super(Connection, self).get_boot_command()

mitogen/lxc.py

@@ -37,7 +37,20 @@ import mitogen.parent
 LOG = logging.getLogger(__name__)
 
 
-class Stream(mitogen.parent.Stream):
+class Options(mitogen.parent.Options):
+    container = None
+    lxc_attach_path = 'lxc-attach'
+
+    def __init__(self, container, lxc_attach_path=None, **kwargs):
+        super(Options, self).__init__(**kwargs)
+        self.container = container
+        if lxc_attach_path:
+            self.lxc_attach_path = lxc_attach_path
+
+
+class Connection(mitogen.parent.Connection):
+    options_class = Options
+
     child_is_immediate_subprocess = False
     create_child_args = {
         # If lxc-attach finds any of stdin, stdout, stderr connected to a TTY,
@@ -47,29 +60,20 @@ class Stream(mitogen.parent.Stream):
         'merge_stdio': True
     }
 
-    container = None
-    lxc_attach_path = 'lxc-attach'
-
     eof_error_hint = (
         'Note: many versions of LXC do not report program execution failure '
         'meaningfully. Please check the host logs (/var/log) for more '
         'information.'
     )
 
-    def construct(self, container, lxc_attach_path=None, **kwargs):
-        super(Stream, self).construct(**kwargs)
-        self.container = container
-        if lxc_attach_path:
-            self.lxc_attach_path = lxc_attach_path
-
     def _get_name(self):
-        return u'lxc.' + self.container
+        return u'lxc.' + self.options.container
 
     def get_boot_command(self):
         bits = [
-            self.lxc_attach_path,
+            self.options.lxc_attach_path,
             '--clear-env',
-            '--name', self.container,
+            '--name', self.options.container,
             '--',
         ]
-        return bits + super(Stream, self).get_boot_command()
+        return bits + super(Connection, self).get_boot_command()

mitogen/lxd.py

@@ -37,7 +37,21 @@ import mitogen.parent
 LOG = logging.getLogger(__name__)
 
 
-class Stream(mitogen.parent.Stream):
+class Options(mitogen.parent.Options):
+    container = None
+    lxc_path = 'lxc'
+    python_path = 'python'
+
+    def __init__(self, container, lxc_path=None, **kwargs):
+        super(Options, self).__init__(**kwargs)
+        self.container = container
+        if lxc_path:
+            self.lxc_path = lxc_path
+
+
+class Connection(mitogen.parent.Connection):
+    options_class = Options
+
     child_is_immediate_subprocess = False
     create_child_args = {
         # If lxc finds any of stdin, stdout, stderr connected to a TTY, to
@@ -47,31 +61,21 @@ class Stream(mitogen.parent.Stream):
         'merge_stdio': True
     }
 
-    container = None
-    lxc_path = 'lxc'
-    python_path = 'python'
-
     eof_error_hint = (
         'Note: many versions of LXC do not report program execution failure '
         'meaningfully. Please check the host logs (/var/log) for more '
         'information.'
     )
 
-    def construct(self, container, lxc_path=None, **kwargs):
-        super(Stream, self).construct(**kwargs)
-        self.container = container
-        if lxc_path:
-            self.lxc_path = lxc_path
-
     def _get_name(self):
-        return u'lxd.' + self.container
+        return u'lxd.' + self.options.container
 
     def get_boot_command(self):
         bits = [
-            self.lxc_path,
+            self.options.lxc_path,
             'exec',
             '--mode=noninteractive',
-            self.container,
+            self.options.container,
             '--',
         ]
-        return bits + super(Stream, self).get_boot_command()
+        return bits + super(Connection, self).get_boot_command()

mitogen/master.py

@@ -91,7 +91,8 @@ RLOG = logging.getLogger('mitogen.ctx')
 
 
 def _stdlib_paths():
-    """Return a set of paths from which Python imports the standard library.
+    """
+    Return a set of paths from which Python imports the standard library.
     """
     attr_candidates = [
         'prefix',
@@ -111,8 +112,8 @@ def _stdlib_paths():
 
 
 def is_stdlib_name(modname):
-    """Return :data:`True` if `modname` appears to come from the standard
+    """
-    library.
+    Return :data:`True` if `modname` appears to come from the standard library.
     """
     if imp.is_builtin(modname) != 0:
         return True
@@ -139,7 +140,8 @@ def is_stdlib_path(path):
 
 
 def get_child_modules(path):
-    """Return the suffixes of submodules directly neated beneath of the package
+    """
+    Return the suffixes of submodules directly neated beneath of the package
     directory at `path`.
 
     :param str path:
@@ -301,8 +303,10 @@ class ThreadWatcher(object):
 
     @classmethod
     def _reset(cls):
-        """If we have forked since the watch dictionaries were initialized, all
+        """
-        that has is garbage, so clear it."""
+        If we have forked since the watch dictionaries were initialized, all
+        that has is garbage, so clear it.
+        """
         if os.getpid() != cls._cls_pid:
             cls._cls_pid = os.getpid()
             cls._cls_instances_by_target.clear()
@@ -394,7 +398,7 @@ class LogForwarder(object):
             name = '%s.%s' % (RLOG.name, context.name)
             self._cache[msg.src_id] = logger = logging.getLogger(name)
 
-        name, level_s, s = msg.data.decode('latin1').split('\x00', 2)
+        name, level_s, s = msg.data.decode('utf-8', 'replace').split('\x00', 2)
 
         # See logging.Handler.makeRecord()
         record = logging.LogRecord(
@@ -531,14 +535,15 @@ class SysModulesMethod(FinderMethod):
             return
 
         if not isinstance(module, types.ModuleType):
-            LOG.debug('sys.modules[%r] absent or not a regular module',
+            LOG.debug('%r: sys.modules[%r] absent or not a regular module',
-                      fullname)
+                      self, fullname)
             return
 
         path = _py_filename(getattr(module, '__file__', ''))
         if not path:
             return
 
+        LOG.debug('%r: sys.modules[%r]: found %s', self, fullname, path)
         is_pkg = hasattr(module, '__path__')
         try:
             source = inspect.getsource(module)
@@ -667,7 +672,8 @@ class ModuleFinder(object):
     ]
 
     def get_module_source(self, fullname):
-        """Given the name of a loaded module `fullname`, attempt to find its
+        """
+        Given the name of a loaded module `fullname`, attempt to find its
         source code.
 
         :returns:
@@ -691,9 +697,10 @@ class ModuleFinder(object):
         return tup
 
     def resolve_relpath(self, fullname, level):
-        """Given an ImportFrom AST node, guess the prefix that should be tacked
+        """
-        on to an alias name to produce a canonical name. `fullname` is the name
+        Given an ImportFrom AST node, guess the prefix that should be tacked on
-        of the module in which the ImportFrom appears.
+        to an alias name to produce a canonical name. `fullname` is the name of
+        the module in which the ImportFrom appears.
         """
         mod = sys.modules.get(fullname, None)
         if hasattr(mod, '__path__'):
@@ -817,7 +824,7 @@ class ModuleResponder(object):
         )
 
     def __repr__(self):
-        return 'ModuleResponder(%r)' % (self._router,)
+        return 'ModuleResponder'
 
     def add_source_override(self, fullname, path, source, is_pkg):
         """
@@ -844,9 +851,11 @@ class ModuleResponder(object):
         self.blacklist.append(fullname)
 
     def neutralize_main(self, path, src):
-        """Given the source for the __main__ module, try to find where it
+        """
-        begins conditional execution based on a "if __name__ == '__main__'"
+        Given the source for the __main__ module, try to find where it begins
-        guard, and remove any code after that point."""
+        conditional execution based on a "if __name__ == '__main__'" guard, and
+        remove any code after that point.
+        """
         match = self.MAIN_RE.search(src)
         if match:
             return src[:match.start()]
@@ -920,17 +929,17 @@ class ModuleResponder(object):
         return tup
 
     def _send_load_module(self, stream, fullname):
-        if fullname not in stream.sent_modules:
+        if fullname not in stream.protocol.sent_modules:
             tup = self._build_tuple(fullname)
             msg = mitogen.core.Message.pickled(
                 tup,
-                dst_id=stream.remote_id,
+                dst_id=stream.protocol.remote_id,
                 handle=mitogen.core.LOAD_MODULE,
             )
-            LOG.debug('%s: sending module %s (%.2f KiB)',
+            LOG.debug('%s: sending %s (%.2f KiB) to %s',
-                      stream.name, fullname, len(msg.data) / 1024.0)
+                      self, fullname, len(msg.data) / 1024.0, stream.name)
             self._router._async_route(msg)
-            stream.sent_modules.add(fullname)
+            stream.protocol.sent_modules.add(fullname)
             if tup[2] is not None:
                 self.good_load_module_count += 1
                 self.good_load_module_size += len(msg.data)
@@ -939,23 +948,23 @@ class ModuleResponder(object):
 
     def _send_module_load_failed(self, stream, fullname):
         self.bad_load_module_count += 1
-        stream.send(
+        stream.protocol.send(
             mitogen.core.Message.pickled(
                 self._make_negative_response(fullname),
-                dst_id=stream.remote_id,
+                dst_id=stream.protocol.remote_id,
                 handle=mitogen.core.LOAD_MODULE,
             )
         )
 
     def _send_module_and_related(self, stream, fullname):
-        if fullname in stream.sent_modules:
+        if fullname in stream.protocol.sent_modules:
             return
 
         try:
             tup = self._build_tuple(fullname)
             for name in tup[4]:  # related
                 parent, _, _ = str_partition(name, '.')
-                if parent != fullname and parent not in stream.sent_modules:
+                if parent != fullname and parent not in stream.protocol.sent_modules:
                     # Parent hasn't been sent, so don't load submodule yet.
                     continue
 
@@ -976,7 +985,7 @@ class ModuleResponder(object):
         fullname = msg.data.decode()
         LOG.debug('%s requested module %s', stream.name, fullname)
         self.get_module_count += 1
-        if fullname in stream.sent_modules:
+        if fullname in stream.protocol.sent_modules:
             LOG.warning('_on_get_module(): dup request for %r from %r',
                         fullname, stream)
 
@@ -987,12 +996,12 @@ class ModuleResponder(object):
             self.get_module_secs += time.time() - t0
 
     def _send_forward_module(self, stream, context, fullname):
-        if stream.remote_id != context.context_id:
+        if stream.protocol.remote_id != context.context_id:
-            stream.send(
+            stream.protocol._send(
                 mitogen.core.Message(
                     data=b('%s\x00%s' % (context.context_id, fullname)),
                     handle=mitogen.core.FORWARD_MODULE,
-                    dst_id=stream.remote_id,
+                    dst_id=stream.protocol.remote_id,
                 )
             )
 
@@ -1061,6 +1070,7 @@ class Broker(mitogen.core.Broker):
                 on_join=self.shutdown,
             )
         super(Broker, self).__init__()
+        self.timers = mitogen.parent.TimerList()
 
     def shutdown(self):
         super(Broker, self).shutdown()

mitogen/minify.py

@@ -44,7 +44,8 @@ else:
 
 
 def minimize_source(source):
-    """Remove comments and docstrings from Python `source`, preserving line
+    """
+    Remove comments and docstrings from Python `source`, preserving line
     numbers and syntax of empty blocks.
 
     :param str source:
@@ -62,7 +63,8 @@ def minimize_source(source):
 
 
 def strip_comments(tokens):
-    """Drop comment tokens from a `tokenize` stream.
+    """
+    Drop comment tokens from a `tokenize` stream.
 
     Comments on lines 1-2 are kept, to preserve hashbang and encoding.
     Trailing whitespace is remove from all lines.
@@ -84,7 +86,8 @@ def strip_comments(tokens):
 
 
 def strip_docstrings(tokens):
-    """Replace docstring tokens with NL tokens in a `tokenize` stream.
+    """
+    Replace docstring tokens with NL tokens in a `tokenize` stream.
 
     Any STRING token not part of an expression is deemed a docstring.
     Indented docstrings are not yet recognised.
@@ -119,7 +122,8 @@ def strip_docstrings(tokens):
 
 
 def reindent(tokens, indent=' '):
-    """Replace existing indentation in a token steam, with `indent`.
+    """
+    Replace existing indentation in a token steam, with `indent`.
     """
     old_levels = []
     old_level = 0

mitogen/profiler.py

@@ -28,7 +28,8 @@
 
 # !mitogen: minify_safe
 
-"""mitogen.profiler
+"""
+mitogen.profiler
     Record and report cProfile statistics from a run. Creates one aggregated
     output file, one aggregate containing only workers, and one for the
     top-level process.
@@ -152,7 +153,7 @@ def do_stat(tmpdir, sort, *args):
 
 def main():
     if len(sys.argv) < 2 or sys.argv[1] not in ('record', 'report', 'stat'):
-        sys.stderr.write(__doc__)
+        sys.stderr.write(__doc__.lstrip())
         sys.exit(1)
 
     func = globals()['do_' + sys.argv[1]]

mitogen/service.py

@@ -485,7 +485,6 @@ class Pool(object):
             )
             thread.start()
             self._threads.append(thread)
-
         LOG.debug('%r: initialized', self)
 
     def _py_24_25_compat(self):
@@ -658,7 +657,7 @@ class PushFileService(Service):
 
     def _forward(self, context, path):
         stream = self.router.stream_by_id(context.context_id)
-        child = mitogen.core.Context(self.router, stream.remote_id)
+        child = mitogen.core.Context(self.router, stream.protocol.remote_id)
         sent = self._sent_by_stream.setdefault(stream, set())
         if path in sent:
             if child.context_id != context.context_id:
@@ -891,7 +890,7 @@ class FileService(Service):
     # The IO loop pumps 128KiB chunks. An ideal message is a multiple of this,
     # odd-sized messages waste one tiny write() per message on the trailer.
     # Therefore subtract 10 bytes pickle overhead + 24 bytes header.
-    IO_SIZE = mitogen.core.CHUNK_SIZE - (mitogen.core.Stream.HEADER_LEN + (
+    IO_SIZE = mitogen.core.CHUNK_SIZE - (mitogen.core.Message.HEADER_LEN + (
         len(
             mitogen.core.Message.pickled(
                 mitogen.core.Blob(b(' ') * mitogen.core.CHUNK_SIZE)

mitogen/setns.py

@@ -116,9 +116,15 @@ def get_machinectl_pid(path, name):
     raise Error("could not find PID from machinectl output.\n%s", output)
 
 
-class Stream(mitogen.parent.Stream):
+GET_LEADER_BY_KIND = {
-    child_is_immediate_subprocess = False
+    'docker': ('docker_path', get_docker_pid),
+    'lxc': ('lxc_info_path', get_lxc_pid),
+    'lxd': ('lxc_path', get_lxd_pid),
+    'machinectl': ('machinectl_path', get_machinectl_pid),
+}
 
+
+class Options(mitogen.parent.Options):
     container = None
     username = 'root'
     kind = None
@@ -128,24 +134,17 @@ class Stream(mitogen.parent.Stream):
     lxc_info_path = 'lxc-info'
     machinectl_path = 'machinectl'
 
-    GET_LEADER_BY_KIND = {
+    def __init__(self, container, kind, username=None, docker_path=None,
-        'docker': ('docker_path', get_docker_pid),
-        'lxc': ('lxc_info_path', get_lxc_pid),
-        'lxd': ('lxc_path', get_lxd_pid),
-        'machinectl': ('machinectl_path', get_machinectl_pid),
-    }
-
-    def construct(self, container, kind, username=None, docker_path=None,
                  lxc_path=None, lxc_info_path=None, machinectl_path=None,
                  **kwargs):
-        super(Stream, self).construct(**kwargs)
+        super(Options, self).__init__(**kwargs)
-        if kind not in self.GET_LEADER_BY_KIND:
+        if kind not in GET_LEADER_BY_KIND:
             raise Error('unsupported container kind: %r', kind)
 
-        self.container = container
+        self.container = mitogen.core.to_text(container)
         self.kind = kind
         if username:
-            self.username = username
+            self.username = mitogen.core.to_text(username)
         if docker_path:
             self.docker_path = docker_path
         if lxc_path:
@@ -155,6 +154,11 @@ class Stream(mitogen.parent.Stream):
         if machinectl_path:
             self.machinectl_path = machinectl_path
 
+
+class Connection(mitogen.parent.Connection):
+    options_class = Options
+    child_is_immediate_subprocess = False
+
     # Order matters. https://github.com/karelzak/util-linux/commit/854d0fe/
     NS_ORDER = ('ipc', 'uts', 'net', 'pid', 'mnt', 'user')
 
@@ -189,15 +193,15 @@ class Stream(mitogen.parent.Stream):
         try:
             os.setgroups([grent.gr_gid
                           for grent in grp.getgrall()
-                          if self.username in grent.gr_mem])
+                          if self.options.username in grent.gr_mem])
-            pwent = pwd.getpwnam(self.username)
+            pwent = pwd.getpwnam(self.options.username)
             os.setreuid(pwent.pw_uid, pwent.pw_uid)
             # shadow-4.4/libmisc/setupenv.c. Not done: MAIL, PATH
             os.environ.update({
                 'HOME': pwent.pw_dir,
                 'SHELL': pwent.pw_shell or '/bin/sh',
-                'LOGNAME': self.username,
+                'LOGNAME': self.options.username,
-                'USER': self.username,
+                'USER': self.options.username,
             })
             if ((os.path.exists(pwent.pw_dir) and
                  os.access(pwent.pw_dir, os.X_OK))):
@@ -217,7 +221,7 @@ class Stream(mitogen.parent.Stream):
         # namespaces, meaning starting new threads in the exec'd program will
         # fail. The solution is forking, so inject a /bin/sh call to achieve
         # this.
-        argv = super(Stream, self).get_boot_command()
+        argv = super(Connection, self).get_boot_command()
         # bash will exec() if a single command was specified and the shell has
         # nothing left to do, so "; exit $?" gives bash a reason to live.
         return ['/bin/sh', '-c', '%s; exit $?' % (mitogen.parent.Argv(argv),)]
@@ -226,13 +230,12 @@ class Stream(mitogen.parent.Stream):
         return mitogen.parent.create_child(args, preexec_fn=self.preexec_fn)
 
     def _get_name(self):
-        return u'setns.' + self.container
+        return u'setns.' + self.options.container
 
-    def connect(self):
+    def connect(self, **kwargs):
-        self.name = self._get_name()
+        attr, func = GET_LEADER_BY_KIND[self.options.kind]
-        attr, func = self.GET_LEADER_BY_KIND[self.kind]
+        tool_path = getattr(self.options, attr)
-        tool_path = getattr(self, attr)
+        self.leader_pid = func(tool_path, self.options.container)
-        self.leader_pid = func(tool_path, self.container)
         LOG.debug('Leader PID for %s container %r: %d',
-                  self.kind, self.container, self.leader_pid)
+                  self.options.kind, self.options.container, self.leader_pid)
-        super(Stream, self).connect()
+        return super(Connection, self).connect(**kwargs)

mitogen/ssh.py

@@ -29,7 +29,7 @@
 # !mitogen: minify_safe
 
 """
-Functionality to allow establishing new slave contexts over an SSH connection.
+Construct new children via the OpenSSH client.
 """
 
 import logging
@@ -52,82 +52,122 @@ except NameError:
 
 LOG = logging.getLogger('mitogen')
 
+auth_incorrect_msg = 'SSH authentication is incorrect'
+password_incorrect_msg = 'SSH password is incorrect'
+password_required_msg = 'SSH password was requested, but none specified'
+hostkey_config_msg = (
+    'SSH requested permission to accept unknown host key, but '
+    'check_host_keys=ignore. This is likely due to ssh_args=  '
+    'conflicting with check_host_keys=. Please correct your '
+    'configuration.'
+)
+hostkey_failed_msg = (
+    'Host key checking is enabled, and SSH reported an unrecognized or '
+    'mismatching host key.'
+)
+
 # sshpass uses 'assword' because it doesn't lowercase the input.
-PASSWORD_PROMPT = b('password')
+PASSWORD_PROMPT_PATTERN = re.compile(
-HOSTKEY_REQ_PROMPT = b('are you sure you want to continue connecting (yes/no)?')
+    b('password'),
-HOSTKEY_FAIL = b('host key verification failed.')
+    re.I
+)
+
+HOSTKEY_REQ_PATTERN = re.compile(
+    b(r'are you sure you want to continue connecting \(yes/no\)\?'),
+    re.I
+)
+
+HOSTKEY_FAIL_PATTERN = re.compile(
+    b(r'host key verification failed\.'),
+    re.I
+)
 
 # [user@host: ] permission denied
-PERMDENIED_RE = re.compile(
+# issue #271: work around conflict with user shell reporting 'permission
-    ('(?:[^@]+@[^:]+: )?'  # Absent in OpenSSH <7.5
+# denied' e.g. during chdir($HOME) by only matching it at the start of the
-     'Permission denied').encode(),
+# line.
+PERMDENIED_PATTERN = re.compile(
+    b('^(?:[^@]+@[^:]+: )?'  # Absent in OpenSSH <7.5
+      'Permission denied'),
     re.I
 )
 
+DEBUG_PATTERN = re.compile(b('^debug[123]:'))
+
 
-DEBUG_PREFIXES = (b('debug1:'), b('debug2:'), b('debug3:'))
+class PasswordError(mitogen.core.StreamError):
+    pass
 
 
-def filter_debug(stream, it):
+class HostKeyError(mitogen.core.StreamError):
-    """
+    pass
-    Read line chunks from it, either yielding them directly, or building up and
-    logging individual lines if they look like SSH debug output.
 
-    This contains the mess of dealing with both line-oriented input, and partial
-    lines such as the password prompt.
 
-    Yields `(line, partial)` tuples, where `line` is the line, `partial` is
+class SetupProtocol(mitogen.parent.RegexProtocol):
-    :data:`True` if no terminating newline character was present and no more
+    """
-    data exists in the read buffer. Consuming code can use this to unreliably
+    This protocol is attached to stderr of the SSH client. It responds to
-    detect the presence of an interactive prompt.
+    various interactive prompts as required.
     """
-    # The `partial` test is unreliable, but is only problematic when verbosity
+    password_sent = False
-    # is enabled: it's possible for a combination of SSH banner, password
+
-    # prompt, verbose output, timing and OS buffering specifics to create a
+    def _on_host_key_request(self, line, match):
-    # situation where an otherwise newline-terminated line appears to not be
+        if self.stream.conn.options.check_host_keys == 'accept':
-    # terminated, due to a partial read(). If something is broken when
+            LOG.debug('%s: accepting host key', self.stream.name)
-    # ssh_debug_level>0, this is the first place to look.
+            self.stream.transmit_side.write(b('yes\n'))
-    state = 'start_of_line'
+            return
-    buf = b('')
+
-    for chunk in it:
+        # _host_key_prompt() should never be reached with ignore or enforce
-        buf += chunk
+        # mode, SSH should have handled that. User's ssh_args= is conflicting
-        while buf:
+        # with ours.
-            if state == 'start_of_line':
+        self.stream.conn._fail_connection(HostKeyError(hostkey_config_msg))
-                if len(buf) < 8:
+
-                    # short read near buffer limit, block awaiting at least 8
+    def _on_host_key_failed(self, line, match):
-                    # bytes so we can discern a debug line, or the minimum
+        self.stream.conn._fail_connection(HostKeyError(hostkey_failed_msg))
-                    # interesting token from above or the bootstrap
+
-                    # ('password', 'MITO000\n').
+    def _on_permission_denied(self, line, match):
-                    break
+        if self.stream.conn.options.password is not None and \
-                elif any(buf.startswith(p) for p in DEBUG_PREFIXES):
+                self.password_sent:
-                    state = 'in_debug'
+            self.stream.conn._fail_connection(
+                PasswordError(password_incorrect_msg)
+            )
+        elif PASSWORD_PROMPT_PATTERN.search(line) and \
+                self.stream.conn.options.password is None:
+            # Permission denied (password,pubkey)
+            self.stream.conn._fail_connection(
+                PasswordError(password_required_msg)
+            )
         else:
-                    state = 'in_plain'
+            self.stream.conn._fail_connection(
-            elif state == 'in_debug':
+                PasswordError(auth_incorrect_msg)
-                if b('\n') not in buf:
+            )
-                    break
-                line, _, buf = bytes_partition(buf, b('\n'))
-                LOG.debug('%s: %s', stream.name,
-                          mitogen.core.to_text(line.rstrip()))
-                state = 'start_of_line'
-            elif state == 'in_plain':
-                line, nl, buf = bytes_partition(buf, b('\n'))
-                yield line + nl, not (nl or buf)
-                if nl:
-                    state = 'start_of_line'
 
+    def _on_password_prompt(self, line, match):
+        LOG.debug('%s: (password prompt): %s', self.stream.name, line)
+        if self.stream.conn.options.password is None:
+            self.stream.conn._fail(PasswordError(password_required_msg))
 
-class PasswordError(mitogen.core.StreamError):
+        self.stream.transmit_side.write(
-    pass
+            (self.stream.conn.options.password + '\n').encode('utf-8')
+        )
+        self.password_sent = True
 
+    def _on_debug_line(self, line, match):
+        text = mitogen.core.to_text(line.rstrip())
+        LOG.debug('%s: %s', self.stream.name, text)
 
-class HostKeyError(mitogen.core.StreamError):
+    PATTERNS = [
-    pass
+        (DEBUG_PATTERN, _on_debug_line),
+        (HOSTKEY_FAIL_PATTERN, _on_host_key_failed),
+        (PERMDENIED_PATTERN, _on_permission_denied),
+    ]
 
+    PARTIAL_PATTERNS = [
+        (PASSWORD_PROMPT_PATTERN, _on_password_prompt),
+        (HOSTKEY_REQ_PATTERN, _on_host_key_request),
+    ]
 
-class Stream(mitogen.parent.Stream):
-    child_is_immediate_subprocess = False
 
+class Options(mitogen.parent.Options):
     #: Default to whatever is available as 'python' on the remote machine,
     #: overriding sys.executable use.
     python_path = 'python'
@@ -141,19 +181,19 @@ class Stream(mitogen.parent.Stream):
     hostname = None
     username = None
     port = None
-
     identity_file = None
     password = None
     ssh_args = None
 
     check_host_keys_msg = 'check_host_keys= must be set to accept, enforce or ignore'
 
-    def construct(self, hostname, username=None, ssh_path=None, port=None,
+    def __init__(self, hostname, username=None, ssh_path=None, port=None,
                  check_host_keys='enforce', password=None, identity_file=None,
                  compression=True, ssh_args=None, keepalive_enabled=True,
                  keepalive_count=3, keepalive_interval=15,
                  identities_only=True, ssh_debug_level=None, **kwargs):
-        super(Stream, self).construct(**kwargs)
+        super(Options, self).__init__(**kwargs)
+
         if check_host_keys not in ('accept', 'enforce', 'ignore'):
             raise ValueError(self.check_host_keys_msg)
 
@@ -175,143 +215,81 @@ class Stream(mitogen.parent.Stream):
         if ssh_debug_level:
             self.ssh_debug_level = ssh_debug_level
 
-        self._init_create_child()
+
+class Connection(mitogen.parent.Connection):
+    options_class = Options
+    diag_protocol_class = SetupProtocol
+
+    child_is_immediate_subprocess = False
+
+    def _get_name(self):
+        s = u'ssh.' + mitogen.core.to_text(self.options.hostname)
+        if self.options.port and self.options.port != 22:
+            s += u':%s' % (self.options.port,)
+        return s
 
     def _requires_pty(self):
         """
-        Return :data:`True` if the configuration requires a PTY to be
+        Return :data:`True` if a PTY to is required for this configuration,
-        allocated. This is only true if we must interactively accept host keys,
+        because it must interactively accept host keys or type a password.
-        or type a password.
         """
-        return (self.check_host_keys == 'accept' or
+        return (
-                self.password is not None)
+            self.options.check_host_keys == 'accept' or
+            self.options.password is not None
+        )
 
-    def _init_create_child(self):
+    def create_child(self, **kwargs):
         """
-        Initialize the base class :attr:`create_child` and
+        Avoid PTY use when possible to avoid a scaling limitation.
-        :attr:`create_child_args` according to whether we need a PTY or not.
         """
         if self._requires_pty():
-            self.create_child = mitogen.parent.hybrid_tty_create_child
+            return mitogen.parent.hybrid_tty_create_child(**kwargs)
         else:
-            self.create_child = mitogen.parent.create_child
+            return mitogen.parent.create_child(stderr_pipe=True, **kwargs)
-            self.create_child_args = {
-                'stderr_pipe': True,
-            }
 
     def get_boot_command(self):
-        bits = [self.ssh_path]
+        bits = [self.options.ssh_path]
-        if self.ssh_debug_level:
+        if self.options.ssh_debug_level:
-            bits += ['-' + ('v' * min(3, self.ssh_debug_level))]
+            bits += ['-' + ('v' * min(3, self.options.ssh_debug_level))]
         else:
             # issue #307: suppress any login banner, as it may contain the
             # password prompt, and there is no robust way to tell the
             # difference.
             bits += ['-o', 'LogLevel ERROR']
-        if self.username:
+        if self.options.username:
-            bits += ['-l', self.username]
+            bits += ['-l', self.options.username]
-        if self.port is not None:
+        if self.options.port is not None:
-            bits += ['-p', str(self.port)]
+            bits += ['-p', str(self.options.port)]
-        if self.identities_only and (self.identity_file or self.password):
+        if self.options.identities_only and (self.options.identity_file or
+                                             self.options.password):
             bits += ['-o', 'IdentitiesOnly yes']
-        if self.identity_file:
+        if self.options.identity_file:
-            bits += ['-i', self.identity_file]
+            bits += ['-i', self.options.identity_file]
-        if self.compression:
+        if self.options.compression:
             bits += ['-o', 'Compression yes']
-        if self.keepalive_enabled:
+        if self.options.keepalive_enabled:
             bits += [
-                '-o', 'ServerAliveInterval %s' % (self.keepalive_interval,),
+                '-o', 'ServerAliveInterval %s' % (
-                '-o', 'ServerAliveCountMax %s' % (self.keepalive_count,),
+                    self.options.keepalive_interval,
+                ),
+                '-o', 'ServerAliveCountMax %s' % (
+                    self.options.keepalive_count,
+                ),
             ]
         if not self._requires_pty():
             bits += ['-o', 'BatchMode yes']
-        if self.check_host_keys == 'enforce':
+        if self.options.check_host_keys == 'enforce':
             bits += ['-o', 'StrictHostKeyChecking yes']
-        if self.check_host_keys == 'accept':
+        if self.options.check_host_keys == 'accept':
             bits += ['-o', 'StrictHostKeyChecking ask']
-        elif self.check_host_keys == 'ignore':
+        elif self.options.check_host_keys == 'ignore':
             bits += [
                 '-o', 'StrictHostKeyChecking no',
                 '-o', 'UserKnownHostsFile /dev/null',
                 '-o', 'GlobalKnownHostsFile /dev/null',
             ]
-        if self.ssh_args:
+        if self.options.ssh_args:
-            bits += self.ssh_args
+            bits += self.options.ssh_args
-        bits.append(self.hostname)
+        bits.append(self.options.hostname)
-        base = super(Stream, self).get_boot_command()
+        base = super(Connection, self).get_boot_command()
         return bits + [shlex_quote(s).strip() for s in base]
-
-    def _get_name(self):
-        s = u'ssh.' + mitogen.core.to_text(self.hostname)
-        if self.port:
-            s += u':%s' % (self.port,)
-        return s
-
-    auth_incorrect_msg = 'SSH authentication is incorrect'
-    password_incorrect_msg = 'SSH password is incorrect'
-    password_required_msg = 'SSH password was requested, but none specified'
-    hostkey_config_msg = (
-        'SSH requested permission to accept unknown host key, but '
-        'check_host_keys=ignore. This is likely due to ssh_args=  '
-        'conflicting with check_host_keys=. Please correct your '
-        'configuration.'
-    )
-    hostkey_failed_msg = (
-        'Host key checking is enabled, and SSH reported an unrecognized or '
-        'mismatching host key.'
-    )
-
-    def _host_key_prompt(self):
-        if self.check_host_keys == 'accept':
-            LOG.debug('%s: accepting host key', self.name)
-            self.diag_stream.transmit_side.write(b('yes\n'))
-            return
-
-        # _host_key_prompt() should never be reached with ignore or enforce
-        # mode, SSH should have handled that. User's ssh_args= is conflicting
-        # with ours.
-        raise HostKeyError(self.hostkey_config_msg)
-
-    def _connect_input_loop(self, it):
-        password_sent = False
-        for buf, partial in filter_debug(self, it):
-            LOG.debug('%s: stdout: %s', self.name, buf.rstrip())
-            if buf.endswith(self.EC0_MARKER):
-                self._ec0_received()
-                return
-            elif HOSTKEY_REQ_PROMPT in buf.lower():
-                self._host_key_prompt()
-            elif HOSTKEY_FAIL in buf.lower():
-                raise HostKeyError(self.hostkey_failed_msg)
-            elif PERMDENIED_RE.match(buf):
-                # issue #271: work around conflict with user shell reporting
-                # 'permission denied' e.g. during chdir($HOME) by only matching
-                # it at the start of the line.
-                if self.password is not None and password_sent:
-                    raise PasswordError(self.password_incorrect_msg)
-                elif PASSWORD_PROMPT in buf and self.password is None:
-                    # Permission denied (password,pubkey)
-                    raise PasswordError(self.password_required_msg)
-                else:
-                    raise PasswordError(self.auth_incorrect_msg)
-            elif partial and PASSWORD_PROMPT in buf.lower():
-                if self.password is None:
-                    raise PasswordError(self.password_required_msg)
-                LOG.debug('%s: sending password', self.name)
-                self.diag_stream.transmit_side.write(
-                    (self.password + '\n').encode()
-                )
-                password_sent = True
-
-        raise mitogen.core.StreamError('bootstrap failed')
-
-    def _connect_bootstrap(self):
-        fds = [self.receive_side.fd]
-        if self.diag_stream is not None:
-            fds.append(self.diag_stream.receive_side.fd)
-
-        it = mitogen.parent.iter_read(fds=fds, deadline=self.connect_deadline)
-        try:
-            self._connect_input_loop(it)
-        finally:
-            it.close()

mitogen/su.py

@@ -29,6 +29,7 @@
 # !mitogen: minify_safe
 
 import logging
+import re
 
 import mitogen.core
 import mitogen.parent
@@ -42,87 +43,119 @@ except NameError:
 
 LOG = logging.getLogger(__name__)
 
+password_incorrect_msg = 'su password is incorrect'
+password_required_msg = 'su password is required'
+
 
 class PasswordError(mitogen.core.StreamError):
     pass
 
 
-class Stream(mitogen.parent.Stream):
+class SetupBootstrapProtocol(mitogen.parent.BootstrapProtocol):
-    # TODO: BSD su cannot handle stdin being a socketpair, but it does let the
+    password_sent = False
-    # child inherit fds from the parent. So we can still pass a socketpair in
+
-    # for hybrid_tty_create_child(), there just needs to be either a shell
+    def setup_patterns(self, conn):
-    # snippet or bootstrap support for fixing things up afterwards.
+        """
-    create_child = staticmethod(mitogen.parent.tty_create_child)
+        su options cause the regexes used to vary. This is a mess, requires
-    child_is_immediate_subprocess = False
+        reworking.
+        """
+        incorrect_pattern = re.compile(
+            mitogen.core.b('|').join(
+                re.escape(s.encode('utf-8'))
+                for s in conn.options.incorrect_prompts
+            ),
+            re.I
+        )
+        prompt_pattern = re.compile(
+            re.escape(
+                conn.options.password_prompt.encode('utf-8')
+            ),
+            re.I
+        )
+
+        self.PATTERNS = mitogen.parent.BootstrapProtocol.PATTERNS + [
+            (incorrect_pattern, type(self)._on_password_incorrect),
+        ]
+        self.PARTIAL_PATTERNS = mitogen.parent.BootstrapProtocol.PARTIAL_PATTERNS + [
+            (prompt_pattern, type(self)._on_password_prompt),
+        ]
 
-    #: Once connected, points to the corresponding DiagLogStream, allowing it to
+    def _on_password_prompt(self, line, match):
-    #: be disconnected at the same time this stream is being torn down.
+        LOG.debug('%r: (password prompt): %r',
+            self.stream.name, line.decode('utf-8', 'replace'))
+
+        if self.stream.conn.options.password is None:
+            self.stream.conn._fail_connection(
+                PasswordError(password_required_msg)
+            )
+            return
+
+        if self.password_sent:
+            self.stream.conn._fail_connection(
+                PasswordError(password_incorrect_msg)
+            )
+            return
+
+        self.stream.transmit_side.write(
+            (self.stream.conn.options.password + '\n').encode('utf-8')
+        )
+        self.password_sent = True
 
-    username = 'root'
+    def _on_password_incorrect(self, line, match):
+        self.stream.conn._fail_connection(
+            PasswordError(password_incorrect_msg)
+        )
+
+
+class Options(mitogen.parent.Options):
+    username = u'root'
     password = None
     su_path = 'su'
-    password_prompt = b('password:')
+    password_prompt = u'password:'
     incorrect_prompts = (
-        b('su: sorry'),                    # BSD
+        u'su: sorry',                   # BSD
-        b('su: authentication failure'),   # Linux
+        u'su: authentication failure',  # Linux
-        b('su: incorrect password'),       # CentOS 6
+        u'su: incorrect password',      # CentOS 6
-        b('authentication is denied'),     # AIX
+        u'authentication is denied',    # AIX
     )
 
-    def construct(self, username=None, password=None, su_path=None,
+    def __init__(self, username=None, password=None, su_path=None,
                  password_prompt=None, incorrect_prompts=None, **kwargs):
-        super(Stream, self).construct(**kwargs)
+        super(Options, self).__init__(**kwargs)
         if username is not None:
-            self.username = username
+            self.username = mitogen.core.to_text(username)
         if password is not None:
-            self.password = password
+            self.password = mitogen.core.to_text(password)
         if su_path is not None:
             self.su_path = su_path
         if password_prompt is not None:
-            self.password_prompt = password_prompt.lower()
+            self.password_prompt = password_prompt
         if incorrect_prompts is not None:
-            self.incorrect_prompts = map(str.lower, incorrect_prompts)
+            self.incorrect_prompts = [
+                mitogen.core.to_text(p)
+                for p in incorrect_prompts
+            ]
 
-    def _get_name(self):
-        return u'su.' + mitogen.core.to_text(self.username)
 
-    def get_boot_command(self):
+class Connection(mitogen.parent.Connection):
-        argv = mitogen.parent.Argv(super(Stream, self).get_boot_command())
+    options_class = Options
-        return [self.su_path, self.username, '-c', str(argv)]
+    stream_protocol_class = SetupBootstrapProtocol
 
-    password_incorrect_msg = 'su password is incorrect'
+    # TODO: BSD su cannot handle stdin being a socketpair, but it does let the
-    password_required_msg = 'su password is required'
+    # child inherit fds from the parent. So we can still pass a socketpair in
-
+    # for hybrid_tty_create_child(), there just needs to be either a shell
-    def _connect_input_loop(self, it):
+    # snippet or bootstrap support for fixing things up afterwards.
-        password_sent = False
+    create_child = staticmethod(mitogen.parent.tty_create_child)
+    child_is_immediate_subprocess = False
 
-        for buf in it:
+    def _get_name(self):
-            LOG.debug('%r: received %r', self, buf)
+        return u'su.' + self.options.username
-            if buf.endswith(self.EC0_MARKER):
-                self._ec0_received()
-                return
-            if any(s in buf.lower() for s in self.incorrect_prompts):
-                if password_sent:
-                    raise PasswordError(self.password_incorrect_msg)
-            elif self.password_prompt in buf.lower():
-                if self.password is None:
-                    raise PasswordError(self.password_required_msg)
-                if password_sent:
-                    raise PasswordError(self.password_incorrect_msg)
-                LOG.debug('sending password')
-                self.transmit_side.write(
-                    mitogen.core.to_text(self.password + '\n').encode('utf-8')
-                )
-                password_sent = True
 
-        raise mitogen.core.StreamError('bootstrap failed')
+    def stream_factory(self):
+        stream = super(Connection, self).stream_factory()
+        stream.protocol.setup_patterns(self)
+        return stream
 
-    def _connect_bootstrap(self):
+    def get_boot_command(self):
-        it = mitogen.parent.iter_read(
+        argv = mitogen.parent.Argv(super(Connection, self).get_boot_command())
-            fds=[self.receive_side.fd],
+        return [self.options.su_path, self.options.username, '-c', str(argv)]
-            deadline=self.connect_deadline,
-        )
-        try:
-            self._connect_input_loop(it)
-        finally:
-            it.close()

mitogen/sudo.py

@@ -40,6 +40,9 @@ from mitogen.core import b
 
 LOG = logging.getLogger(__name__)
 
+password_incorrect_msg = 'sudo password is incorrect'
+password_required_msg = 'sudo password is required'
+
 # These are base64-encoded UTF-8 as our existing minifier/module server
 # struggles with Unicode Python source in some (forgotten) circumstances.
 PASSWORD_PROMPTS = [
@@ -99,14 +102,13 @@ PASSWORD_PROMPTS = [
 
 
 PASSWORD_PROMPT_RE = re.compile(
-    u'|'.join(
+    mitogen.core.b('|').join(
-        base64.b64decode(s).decode('utf-8')
+        base64.b64decode(s)
         for s in PASSWORD_PROMPTS
+    ),
+    re.I
 )
-)
-
 
-PASSWORD_PROMPT = b('password')
 SUDO_OPTIONS = [
     #(False, 'bool', '--askpass', '-A')
     #(False, 'str', '--auth-type', '-a')
@@ -181,10 +183,7 @@ def option(default, *args):
     return default
 
 
-class Stream(mitogen.parent.Stream):
+class Options(mitogen.parent.Options):
-    create_child = staticmethod(mitogen.parent.hybrid_tty_create_child)
-    child_is_immediate_subprocess = False
-
     sudo_path = 'sudo'
     username = 'root'
     password = None
@@ -195,15 +194,16 @@ class Stream(mitogen.parent.Stream):
     selinux_role = None
     selinux_type = None
 
-    def construct(self, username=None, sudo_path=None, password=None,
+    def __init__(self, username=None, sudo_path=None, password=None,
                  preserve_env=None, set_home=None, sudo_args=None,
                  login=None, selinux_role=None, selinux_type=None, **kwargs):
-        super(Stream, self).construct(**kwargs)
+        super(Options, self).__init__(**kwargs)
         opts = parse_sudo_flags(sudo_args or [])
 
         self.username = option(self.username, username, opts.user)
         self.sudo_path = option(self.sudo_path, sudo_path)
-        self.password = password or None
+        if password:
+            self.password = mitogen.core.to_text(password)
         self.preserve_env = option(self.preserve_env,
             preserve_env, opts.preserve_env)
         self.set_home = option(self.set_home, set_home, opts.set_home)
@@ -211,67 +211,59 @@ class Stream(mitogen.parent.Stream):
         self.selinux_role = option(self.selinux_role, selinux_role, opts.role)
         self.selinux_type = option(self.selinux_type, selinux_type, opts.type)
 
-    def _get_name(self):
-        return u'sudo.' + mitogen.core.to_text(self.username)
 
-    def get_boot_command(self):
+class SetupProtocol(mitogen.parent.RegexProtocol):
-        # Note: sudo did not introduce long-format option processing until July
+    password_sent = False
-        # 2013, so even though we parse long-format options, supply short-form
-        # to the sudo command.
-        bits = [self.sudo_path, '-u', self.username]
-        if self.preserve_env:
-            bits += ['-E']
-        if self.set_home:
-            bits += ['-H']
-        if self.login:
-            bits += ['-i']
-        if self.selinux_role:
-            bits += ['-r', self.selinux_role]
-        if self.selinux_type:
-            bits += ['-t', self.selinux_type]
-
-        bits = bits + ['--'] + super(Stream, self).get_boot_command()
-        LOG.debug('sudo command line: %r', bits)
-        return bits
 
-    password_incorrect_msg = 'sudo password is incorrect'
+    def _on_password_prompt(self, line, match):
-    password_required_msg = 'sudo password is required'
+        LOG.debug('%s: (password prompt): %s',
+            self.stream.name, line.decode('utf-8', 'replace'))
 
-    def _connect_input_loop(self, it):
+        if self.stream.conn.options.password is None:
-        password_sent = False
+            self.stream.conn._fail_connection(
+                PasswordError(password_required_msg)
+            )
+            return
 
-        for buf in it:
+        if self.password_sent:
-            LOG.debug('%s: received %r', self.name, buf)
+            self.stream.conn._fail_connection(
-            if buf.endswith(self.EC0_MARKER):
+                PasswordError(password_incorrect_msg)
-                self._ec0_received()
+            )
             return
 
-            match = PASSWORD_PROMPT_RE.search(buf.decode('utf-8').lower())
+        self.stream.transmit_side.write(
-            if match is not None:
+            (self.stream.conn.options.password + '\n').encode('utf-8')
-                LOG.debug('%s: matched password prompt %r',
-                          self.name, match.group(0))
-                if self.password is None:
-                    raise PasswordError(self.password_required_msg)
-                if password_sent:
-                    raise PasswordError(self.password_incorrect_msg)
-                self.diag_stream.transmit_side.write(
-                    (mitogen.core.to_text(self.password) + '\n').encode('utf-8')
         )
-                password_sent = True
+        self.password_sent = True
 
-        raise mitogen.core.StreamError('bootstrap failed')
+    PARTIAL_PATTERNS = [
+        (PASSWORD_PROMPT_RE, _on_password_prompt),
+    ]
 
-    def _connect_bootstrap(self):
-        fds = [self.receive_side.fd]
-        if self.diag_stream is not None:
-            fds.append(self.diag_stream.receive_side.fd)
 
-        it = mitogen.parent.iter_read(
+class Connection(mitogen.parent.Connection):
-            fds=fds,
+    diag_protocol_class = SetupProtocol
-            deadline=self.connect_deadline,
+    options_class = Options
-        )
+    create_child = staticmethod(mitogen.parent.hybrid_tty_create_child)
+    child_is_immediate_subprocess = False
+
+    def _get_name(self):
+        return u'sudo.' + mitogen.core.to_text(self.options.username)
+
+    def get_boot_command(self):
+        # Note: sudo did not introduce long-format option processing until July
+        # 2013, so even though we parse long-format options, supply short-form
+        # to the sudo command.
+        bits = [self.options.sudo_path, '-u', self.options.username]
+        if self.options.preserve_env:
+            bits += ['-E']
+        if self.options.set_home:
+            bits += ['-H']
+        if self.options.login:
+            bits += ['-i']
+        if self.options.selinux_role:
+            bits += ['-r', self.options.selinux_role]
+        if self.options.selinux_type:
+            bits += ['-t', self.options.selinux_type]
 
-        try:
+        return bits + ['--'] + super(Connection, self).get_boot_command()
-            self._connect_input_loop(it)
-        finally:
-            it.close()

mitogen/unix.py

@@ -65,9 +65,38 @@ def make_socket_path():
     return tempfile.mktemp(prefix='mitogen_unix_', suffix='.sock')
 
 
-class Listener(mitogen.core.BasicStream):
+class ListenerStream(mitogen.core.Stream):
+    def on_receive(self, broker):
+        sock, _ = self.receive_side.fp.accept()
+        try:
+            self.protocol.on_accept_client(sock)
+        except:
+            sock.close()
+            raise
+
+
+class Listener(mitogen.core.Protocol):
+    stream_class = ListenerStream
     keep_alive = True
 
+    @classmethod
+    def build_stream(cls, router, path=None, backlog=100):
+        if not path:
+            path = make_socket_path()
+        sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+        if os.path.exists(path) and is_path_dead(path):
+            LOG.debug('%r: deleting stale %r', cls.__name__, path)
+            os.unlink(path)
+
+        sock.bind(path)
+        os.chmod(path, int('0600', 8))
+        sock.listen(backlog)
+
+        stream = super(Listener, cls).build_stream(router, path)
+        stream.accept(sock, sock)
+        router.broker.start_receive(stream)
+        return stream
+
     def __repr__(self):
         return '%s.%s(%r)' % (
             __name__,
@@ -75,20 +104,9 @@ class Listener(mitogen.core.BasicStream):
             self.path,
         )
 
-    def __init__(self, router, path=None, backlog=100):
+    def __init__(self, router, path):
         self._router = router
-        self.path = path or make_socket_path()
+        self.path = path
-        self._sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
-
-        if os.path.exists(self.path) and is_path_dead(self.path):
-            LOG.debug('%r: deleting stale %r', self, self.path)
-            os.unlink(self.path)
-
-        self._sock.bind(self.path)
-        os.chmod(self.path, int('0600', 8))
-        self._sock.listen(backlog)
-        self.receive_side = mitogen.core.Side(self, self._sock.fileno())
-        router.broker.start_receive(self)
 
     def _unlink_socket(self):
         try:
@@ -100,12 +118,11 @@ class Listener(mitogen.core.BasicStream):
                 raise
 
     def on_shutdown(self, broker):
-        broker.stop_receive(self)
+        broker.stop_receive(self.stream)
         self._unlink_socket()
-        self._sock.close()
+        self.stream.receive_side.close()
-        self.receive_side.closed = True
 
-    def _accept_client(self, sock):
+    def on_accept_client(self, sock):
         sock.setblocking(True)
         try:
             pid, = struct.unpack('>L', sock.recv(4))
@@ -115,12 +132,6 @@ class Listener(mitogen.core.BasicStream):
             return
 
         context_id = self._router.id_allocator.allocate()
-        context = mitogen.parent.Context(self._router, context_id)
-        stream = mitogen.core.Stream(self._router, context_id)
-        stream.name = u'unix_client.%d' % (pid,)
-        stream.auth_id = mitogen.context_id
-        stream.is_privileged = True
-
         try:
             sock.send(struct.pack('>LLL', context_id, mitogen.context_id,
                                   os.getpid()))
@@ -129,21 +140,20 @@ class Listener(mitogen.core.BasicStream):
                       self, pid, sys.exc_info()[1])
             return
 
+        context = mitogen.parent.Context(self._router, context_id)
+        stream = mitogen.core.MitogenProtocol.build_stream(
+            router=self._router,
+            remote_id=context_id,
+        )
+        stream.name = u'unix_client.%d' % (pid,)
+        stream.protocol.auth_id = mitogen.context_id
+        stream.protocol.is_privileged = True
+        stream.accept(sock, sock)
         LOG.debug('%r: accepted %r', self, stream)
-        stream.accept(sock.fileno(), sock.fileno())
         self._router.register(context, stream)
 
-    def on_receive(self, broker):
-        sock, _ = self._sock.accept()
-        try:
-            self._accept_client(sock)
-        finally:
-            sock.close()
-
 
-def connect(path, broker=None):
+def _connect(path, broker, sock):
-    LOG.debug('unix.connect(path=%r)', path)
-    sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
     sock.connect(path)
     sock.send(struct.pack('>L', os.getpid()))
     mitogen.context_id, remote_id, pid = struct.unpack('>LLL', sock.recv(12))
@@ -154,15 +164,35 @@ def connect(path, broker=None):
               mitogen.context_id, remote_id)
 
     router = mitogen.master.Router(broker=broker)
-    stream = mitogen.core.Stream(router, remote_id)
+    stream = mitogen.core.MitogenProtocol.build_stream(router, remote_id)
-    stream.accept(sock.fileno(), sock.fileno())
+    stream.accept(sock, sock)
     stream.name = u'unix_listener.%d' % (pid,)
 
+    mitogen.core.listen(stream, 'disconnect', _cleanup)
+    mitogen.core.listen(router.broker, 'shutdown',
+        lambda: router.disconnect_stream(stream))
+
     context = mitogen.parent.Context(router, remote_id)
     router.register(context, stream)
+    return router, context
 
-    mitogen.core.listen(router.broker, 'shutdown',
-                        lambda: router.disconnect_stream(stream))
 
+def connect(path, broker=None):
+    LOG.debug('unix.connect(path=%r)', path)
+    sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+    try:
+        return _connect(path, broker, sock)
+    except:
         sock.close()
-    return router, context
+        raise
+
+
+def _cleanup():
+    """
+    Reset mitogen.context_id and friends when our connection to the parent is
+    lost. Per comments on #91, these globals need to move to the Router so
+    fix-ups like this become unnecessary.
+    """
+    mitogen.context_id = 0
+    mitogen.parent_id = None
+    mitogen.parent_ids = []

preamble_size.py

@@ -19,15 +19,17 @@ import mitogen.sudo
 
 router = mitogen.master.Router()
 context = mitogen.parent.Context(router, 0)
-stream = mitogen.ssh.Stream(router, 0, max_message_size=0, hostname='foo')
+options = mitogen.ssh.Options(max_message_size=0, hostname='foo')
+conn = mitogen.ssh.Connection(options, router)
+conn.context = context
 
-print('SSH command size: %s' % (len(' '.join(stream.get_boot_command())),))
+print('SSH command size: %s' % (len(' '.join(conn.get_boot_command())),))
 print('Preamble size: %s (%.2fKiB)' % (
-    len(stream.get_preamble()),
+    len(conn.get_preamble()),
-    len(stream.get_preamble()) / 1024.0,
+    len(conn.get_preamble()) / 1024.0,
 ))
 if '--dump' in sys.argv:
-    print(zlib.decompress(stream.get_preamble()))
+    print(zlib.decompress(conn.get_preamble()))
     exit()
 
 

scripts/release-notes.py

@@ -0,0 +1,48 @@
+# coding=UTF-8
+
+# Generate the fragment used to make email release announcements
+# usage: release-notes.py 0.2.6
+
+import os
+import sys
+import urllib
+import lxml.html
+
+import subprocess
+
+
+response = urllib.urlopen('https://mitogen.networkgenomics.com/changelog.html')
+tree = lxml.html.parse(response)
+
+prefix = 'v' + sys.argv[1].replace('.', '-')
+
+for elem in tree.getroot().cssselect('div.section[id]'):
+    if elem.attrib['id'].startswith(prefix):
+        break
+else:
+    print('cant find')
+
+
+
+for child in tree.getroot().cssselect('body > *'):
+    child.getparent().remove(child)
+
+body, = tree.getroot().cssselect('body')
+body.append(elem)
+
+proc = subprocess.Popen(
+    args=['w3m', '-T', 'text/html', '-dump', '-cols', '72'],
+    stdin=subprocess.PIPE,
+    stdout=subprocess.PIPE,
+)
+
+stdout, _ = proc.communicate(input=(lxml.html.tostring(tree)))
+stdout = stdout.decode('UTF-8')
+stdout = stdout.translate({
+    ord(u'¶'): None,
+    ord(u'•'): ord(u'*'),
+    ord(u'’'): ord(u"'"),
+    ord(u'“'): ord(u'"'),
+    ord(u'”'): ord(u'"'),
+})
+print(stdout)

tests/ansible/tests/affinity_test.py

@@ -199,10 +199,10 @@ class LinuxPolicyTest(testlib.TestCase):
             self.policy._set_cpu(3)
             my_cpu = self._get_cpus()
 
-            pid = mitogen.parent.detach_popen(
+            proc = mitogen.parent.popen(
                 args=['cp', '/proc/self/status', tf.name]
             )
-            os.waitpid(pid, 0)
+            proc.wait()
 
             his_cpu = self._get_cpus(tf.name)
             self.assertNotEquals(my_cpu, his_cpu)

tests/ansible/tests/connection_test.py

@@ -13,42 +13,29 @@ import ansible.errors
 import ansible.playbook.play_context
 
 import mitogen.core
+import mitogen.utils
+
 import ansible_mitogen.connection
 import ansible_mitogen.plugins.connection.mitogen_local
 import ansible_mitogen.process
-import testlib
-
 
-LOGGER_NAME = ansible_mitogen.target.LOG.name
+import testlib
 
 
-# TODO: fixtureize
+class MuxProcessMixin(object):
-import mitogen.utils
+    @classmethod
-mitogen.utils.log_to_file()
+    def setUpClass(cls):
+        #mitogen.utils.log_to_file()
         ansible_mitogen.process.MuxProcess.start(_init_logging=False)
+        super(MuxProcessMixin, cls).setUpClass()
 
-
+    @classmethod
-class OptionalIntTest(unittest2.TestCase):
+    def tearDownClass(cls):
-    func = staticmethod(ansible_mitogen.connection.optional_int)
+        super(MuxProcessMixin, cls).tearDownClass()
-
+        ansible_mitogen.process.MuxProcess._reset()
-    def test_already_int(self):
-        self.assertEquals(0, self.func(0))
-        self.assertEquals(1, self.func(1))
-        self.assertEquals(-1, self.func(-1))
-
-    def test_is_string(self):
-        self.assertEquals(0, self.func("0"))
-        self.assertEquals(1, self.func("1"))
-        self.assertEquals(-1, self.func("-1"))
-
-    def test_is_none(self):
-        self.assertEquals(None, self.func(None))
-
-    def test_is_junk(self):
-        self.assertEquals(None, self.func({1:2}))
 
 
-class ConnectionMixin(object):
+class ConnectionMixin(MuxProcessMixin):
     klass = ansible_mitogen.plugins.connection.mitogen_local.Connection
 
     def make_connection(self):
@@ -70,6 +57,26 @@ class ConnectionMixin(object):
         super(ConnectionMixin, self).tearDown()
 
 
+class OptionalIntTest(unittest2.TestCase):
+    func = staticmethod(ansible_mitogen.connection.optional_int)
+
+    def test_already_int(self):
+        self.assertEquals(0, self.func(0))
+        self.assertEquals(1, self.func(1))
+        self.assertEquals(-1, self.func(-1))
+
+    def test_is_string(self):
+        self.assertEquals(0, self.func("0"))
+        self.assertEquals(1, self.func("1"))
+        self.assertEquals(-1, self.func("-1"))
+
+    def test_is_none(self):
+        self.assertEquals(None, self.func(None))
+
+    def test_is_junk(self):
+        self.assertEquals(None, self.func({1:2}))
+
+
 class PutDataTest(ConnectionMixin, unittest2.TestCase):
     def test_out_path(self):
         path = tempfile.mktemp(prefix='mitotest')

tests/bench/ssh-roundtrip.py

@@ -0,0 +1,35 @@
+"""
+Measure latency of SSH RPC.
+"""
+
+import sys
+import time
+
+import mitogen
+import mitogen.utils
+import ansible_mitogen.affinity
+
+mitogen.utils.setup_gil()
+ansible_mitogen.affinity.policy.assign_worker()
+
+try:
+    xrange
+except NameError:
+    xrange = range
+
+def do_nothing():
+    pass
+
+@mitogen.main()
+def main(router):
+    f = router.ssh(hostname=sys.argv[1])
+    f.call(do_nothing)
+    t0 = time.time()
+    end = time.time() + 5.0
+    i = 0
+    while time.time() < end:
+        f.call(do_nothing)
+        i += 1
+        t1 = time.time()
+
+    print('++', float(1e3 * (t1 - t0) / (1.0+i)), 'ms')

tests/broker_test.py

@@ -1,4 +1,5 @@
 
+import time
 import threading
 
 import mock

tests/buildah_test.py

@@ -21,7 +21,7 @@ class ConstructorTest(testlib.RouterMixin, testlib.TestCase):
         self.assertEquals(argv[1], 'run')
         self.assertEquals(argv[2], '--')
         self.assertEquals(argv[3], 'container_name')
-        self.assertEquals(argv[4], stream.python_path)
+        self.assertEquals(argv[4], stream.conn.options.python_path)
 
 
 if __name__ == '__main__':

tests/context_test.py

@@ -8,40 +8,7 @@ from mitogen.core import b
 import testlib
 
 
-class EvilObject(object):
+class PickleTest(testlib.RouterMixin, testlib.TestCase):
-    pass
-
-
-def roundtrip(v):
-    msg = mitogen.core.Message.pickled(v)
-    return mitogen.core.Message(data=msg.data).unpickle()
-
-
-class EvilObjectTest(testlib.TestCase):
-    def test_deserialization_fails(self):
-        msg = mitogen.core.Message.pickled(EvilObject())
-        e = self.assertRaises(mitogen.core.StreamError,
-            lambda: msg.unpickle()
-        )
-
-
-class BlobTest(testlib.TestCase):
-    klass = mitogen.core.Blob
-
-    # Python 3 pickle protocol 2 does weird stuff depending on whether an empty
-    # or nonempty bytes is being serialized. For non-empty, it yields a
-    # _codecs.encode() call. For empty, it yields a bytes() call.
-
-    def test_nonempty_bytes(self):
-        v = mitogen.core.Blob(b('dave'))
-        self.assertEquals(b('dave'), roundtrip(v))
-
-    def test_empty_bytes(self):
-        v = mitogen.core.Blob(b(''))
-        self.assertEquals(b(''), roundtrip(v))
-
-
-class ContextTest(testlib.RouterMixin, testlib.TestCase):
     klass = mitogen.core.Context
 
     # Ensure Context can be round-tripped by regular pickle in addition to

tests/create_child_test.py

@@ -0,0 +1,284 @@
+
+import fcntl
+import os
+import stat
+import sys
+import time
+import tempfile
+
+import mock
+import unittest2
+
+import mitogen.parent
+from mitogen.core import b
+
+import testlib
+
+
+def run_fd_check(func, fd, mode, on_start=None):
+    tf = tempfile.NamedTemporaryFile()
+    args = [
+        sys.executable,
+        testlib.data_path('fd_check.py'),
+        tf.name,
+        str(fd),
+        mode,
+    ]
+
+    proc = func(args=args)
+    os = None
+    if on_start:
+        os = on_start(proc)
+    proc.proc.wait()
+    try:
+        return proc, eval(tf.read()), os
+    finally:
+        tf.close()
+
+
+def close_proc(proc):
+    proc.stdin.close()
+    proc.stdout.close()
+    if proc.stderr:
+        prco.stderr.close()
+
+
+def wait_read(fp, n):
+    poller = mitogen.core.Poller()
+    try:
+        poller.start_receive(fp.fileno())
+        for _ in poller.poll():
+            return os.read(fp.fileno(), n)
+        assert False
+    finally:
+        poller.close()
+
+
+class StdinSockMixin(object):
+    def test_stdin(self):
+        proc, info, _ = run_fd_check(self.func, 0, 'read',
+            lambda proc: proc.stdin.send(b('TEST')))
+        st = os.fstat(proc.stdin.fileno())
+        self.assertTrue(stat.S_ISSOCK(st.st_mode))
+        self.assertEquals(st.st_dev, info['st_dev'])
+        self.assertEquals(st.st_mode, info['st_mode'])
+        flags = fcntl.fcntl(proc.stdin.fileno(), fcntl.F_GETFL)
+        self.assertTrue(flags & os.O_RDWR)
+        self.assertTrue(info['buf'], 'TEST')
+        self.assertTrue(info['flags'] & os.O_RDWR)
+
+
+class StdoutSockMixin(object):
+    def test_stdout(self):
+        proc, info, buf = run_fd_check(self.func, 1, 'write',
+            lambda proc: wait_read(proc.stdout, 4))
+        st = os.fstat(proc.stdout.fileno())
+        self.assertTrue(stat.S_ISSOCK(st.st_mode))
+        self.assertEquals(st.st_dev, info['st_dev'])
+        self.assertEquals(st.st_mode, info['st_mode'])
+        flags = fcntl.fcntl(proc.stdout.fileno(), fcntl.F_GETFL)
+        self.assertTrue(flags & os.O_RDWR)
+        self.assertTrue(buf, 'TEST')
+        self.assertTrue(info['flags'] & os.O_RDWR)
+
+
+class CreateChildTest(StdinSockMixin, StdoutSockMixin, testlib.TestCase):
+    func = staticmethod(mitogen.parent.create_child)
+
+    def test_stderr(self):
+        proc, info, _ = run_fd_check(self.func, 2, 'write')
+        st = os.fstat(sys.stderr.fileno())
+        self.assertEquals(st.st_dev, info['st_dev'])
+        self.assertEquals(st.st_mode, info['st_mode'])
+        self.assertEquals(st.st_ino, info['st_ino'])
+
+
+class CreateChildMergedTest(StdinSockMixin, StdoutSockMixin,
+                            testlib.TestCase):
+    def func(self, *args, **kwargs):
+        kwargs['merge_stdio'] = True
+        return mitogen.parent.create_child(*args, **kwargs)
+
+    def test_stderr(self):
+        proc, info, buf = run_fd_check(self.func, 2, 'write',
+            lambda proc: wait_read(proc.stdout, 4))
+        self.assertEquals(None, proc.stderr)
+        st = os.fstat(proc.stdout.fileno())
+        self.assertTrue(stat.S_ISSOCK(st.st_mode))
+        self.assertEquals(st.st_dev, info['st_dev'])
+        self.assertEquals(st.st_mode, info['st_mode'])
+        flags = fcntl.fcntl(proc.stdout.fileno(), fcntl.F_GETFL)
+        self.assertTrue(flags & os.O_RDWR)
+        self.assertTrue(buf, 'TEST')
+        self.assertTrue(info['flags'] & os.O_RDWR)
+
+
+class CreateChildStderrPipeTest(StdinSockMixin, StdoutSockMixin,
+                                testlib.TestCase):
+    def func(self, *args, **kwargs):
+        kwargs['stderr_pipe'] = True
+        return mitogen.parent.create_child(*args, **kwargs)
+
+    def test_stderr(self):
+        proc, info, buf = run_fd_check(self.func, 2, 'write',
+            lambda proc: wait_read(proc.stderr, 4))
+        st = os.fstat(proc.stderr.fileno())
+        self.assertTrue(stat.S_ISFIFO(st.st_mode))
+        self.assertEquals(st.st_dev, info['st_dev'])
+        self.assertEquals(st.st_mode, info['st_mode'])
+        flags = fcntl.fcntl(proc.stderr.fileno(), fcntl.F_GETFL)
+        self.assertFalse(flags & os.O_WRONLY)
+        self.assertFalse(flags & os.O_RDWR)
+        self.assertTrue(buf, 'TEST')
+        self.assertTrue(info['flags'] & os.O_WRONLY)
+
+
+class TtyCreateChildTest(testlib.TestCase):
+    func = staticmethod(mitogen.parent.tty_create_child)
+
+    def test_stdin(self):
+        proc, info, _ = run_fd_check(self.func, 0, 'read',
+            lambda proc: proc.stdin.write(b('TEST')))
+        st = os.fstat(proc.stdin.fileno())
+        self.assertTrue(stat.S_ISCHR(st.st_mode))
+        self.assertTrue(stat.S_ISCHR(info['st_mode']))
+
+        self.assertTrue(isinstance(info['ttyname'],
+                        mitogen.core.UnicodeType))
+        os.ttyname(proc.stdin.fileno())  # crashes if not TTY
+
+        flags = fcntl.fcntl(proc.stdin.fileno(), fcntl.F_GETFL)
+        self.assertTrue(flags & os.O_RDWR)
+        self.assertTrue(info['flags'] & os.O_RDWR)
+
+        self.assertNotEquals(st.st_dev, info['st_dev'])
+        self.assertTrue(info['buf'], 'TEST')
+
+    def test_stdout(self):
+        proc, info, buf = run_fd_check(self.func, 1, 'write',
+            lambda proc: wait_read(proc.stdout, 4))
+
+        st = os.fstat(proc.stdout.fileno())
+        self.assertTrue(stat.S_ISCHR(st.st_mode))
+        self.assertTrue(stat.S_ISCHR(info['st_mode']))
+
+        self.assertTrue(isinstance(info['ttyname'],
+                        mitogen.core.UnicodeType))
+        os.ttyname(proc.stdout.fileno()) # crashes if wrong
+
+        flags = fcntl.fcntl(proc.stdout.fileno(), fcntl.F_GETFL)
+        self.assertTrue(flags & os.O_RDWR)
+        self.assertTrue(info['flags'] & os.O_RDWR)
+
+        self.assertNotEquals(st.st_dev, info['st_dev'])
+        self.assertTrue(flags & os.O_RDWR)
+        self.assertTrue(buf, 'TEST')
+
+    def test_stderr(self):
+        proc, info, buf = run_fd_check(self.func, 2, 'write',
+            lambda proc: wait_read(proc.stdout, 4))
+
+        st = os.fstat(proc.stdout.fileno())
+        self.assertTrue(stat.S_ISCHR(st.st_mode))
+        self.assertTrue(stat.S_ISCHR(info['st_mode']))
+
+        self.assertTrue(isinstance(info['ttyname'],
+                        mitogen.core.UnicodeType))
+        os.ttyname(proc.stdin.fileno())  # crashes if not TTY
+
+        flags = fcntl.fcntl(proc.stdout.fileno(), fcntl.F_GETFL)
+        self.assertTrue(flags & os.O_RDWR)
+        self.assertTrue(info['flags'] & os.O_RDWR)
+
+        self.assertNotEquals(st.st_dev, info['st_dev'])
+        self.assertTrue(flags & os.O_RDWR)
+        self.assertTrue(buf, 'TEST')
+
+    def test_dev_tty_open_succeeds(self):
+        # In the early days of UNIX, a process that lacked a controlling TTY
+        # would acquire one simply by opening an existing TTY. Linux and OS X
+        # continue to follow this behaviour, however at least FreeBSD moved to
+        # requiring an explicit ioctl(). Linux supports it, but we don't yet
+        # use it there and anyway the behaviour will never change, so no point
+        # in fixing things that aren't broken. Below we test that
+        # getpass-loving apps like sudo and ssh get our slave PTY when they
+        # attempt to open /dev/tty, which is what they both do on attempting to
+        # read a password.
+        tf = tempfile.NamedTemporaryFile()
+        try:
+            proc = self.func([
+                'bash', '-c', 'exec 2>%s; echo hi > /dev/tty' % (tf.name,)
+            ])
+            deadline = time.time() + 5.0
+            self.assertEquals(mitogen.core.b('hi\n'), wait_read(proc.stdout, 3))
+            waited_pid, status = os.waitpid(proc.pid, 0)
+            self.assertEquals(proc.pid, waited_pid)
+            self.assertEquals(0, status)
+            self.assertEquals(mitogen.core.b(''), tf.read())
+            proc.stdout.close()
+        finally:
+            tf.close()
+
+
+class StderrDiagTtyMixin(object):
+    def test_stderr(self):
+        proc, info, buf = run_fd_check(self.func, 2, 'write',
+            lambda proc: wait_read(proc.stderr, 4))
+
+        st = os.fstat(proc.stderr.fileno())
+        self.assertTrue(stat.S_ISCHR(st.st_mode))
+        self.assertTrue(stat.S_ISCHR(info['st_mode']))
+
+        self.assertTrue(isinstance(info['ttyname'],
+                        mitogen.core.UnicodeType))
+        os.ttyname(proc.stderr.fileno())  # crashes if wrong
+
+        flags = fcntl.fcntl(proc.stderr.fileno(), fcntl.F_GETFL)
+        self.assertTrue(flags & os.O_RDWR)
+        self.assertTrue(info['flags'] & os.O_RDWR)
+
+        self.assertNotEquals(st.st_dev, info['st_dev'])
+        self.assertTrue(flags & os.O_RDWR)
+        self.assertTrue(buf, 'TEST')
+
+
+class HybridTtyCreateChildTest(StdinSockMixin, StdoutSockMixin,
+                               StderrDiagTtyMixin, testlib.TestCase):
+    func = staticmethod(mitogen.parent.hybrid_tty_create_child)
+
+
+
+if 0:
+    # issue #410
+    class SelinuxHybridTtyCreateChildTest(StderrDiagTtyMixin, testlib.TestCase):
+        func = staticmethod(mitogen.parent.selinux_hybrid_tty_create_child)
+
+        def test_stdin(self):
+            proc, info, buf = run_fd_check(self.func, 0, 'read',
+                lambda proc: proc.transmit_side.write('TEST'))
+            st = os.fstat(proc.transmit_side.fd)
+            self.assertTrue(stat.S_ISFIFO(st.st_mode))
+            self.assertEquals(st.st_dev, info['st_dev'])
+            self.assertEquals(st.st_mode, info['st_mode'])
+            flags = fcntl.fcntl(proc.transmit_side.fd, fcntl.F_GETFL)
+            self.assertTrue(flags & os.O_WRONLY)
+            self.assertTrue(buf, 'TEST')
+            self.assertFalse(info['flags'] & os.O_WRONLY)
+            self.assertFalse(info['flags'] & os.O_RDWR)
+
+        def test_stdout(self):
+            proc, info, buf = run_fd_check(self.func, 1, 'write',
+                lambda proc: wait_read(proc.receive_side, 4))
+            st = os.fstat(proc.receive_side.fd)
+            self.assertTrue(stat.S_ISFIFO(st.st_mode))
+            self.assertEquals(st.st_dev, info['st_dev'])
+            self.assertEquals(st.st_mode, info['st_mode'])
+            flags = fcntl.fcntl(proc.receive_side.fd, fcntl.F_GETFL)
+            self.assertFalse(flags & os.O_WRONLY)
+            self.assertFalse(flags & os.O_RDWR)
+            self.assertTrue(info['flags'] & os.O_WRONLY)
+            self.assertTrue(buf, 'TEST')
+
+
+if __name__ == '__main__':
+    unittest2.main()

tests/data/docker/README.md

@@ -0,0 +1,7 @@
+
+
+# doas-debian.tar.gz
+
+A dynamically linked copy of the OpenBSD ``doas`` tool for Debian, port is from
+https://github.com/multiplexd/doas (the slicer69 port is broken, it reads the
+password from stdin).

tests/data/docker/mitogen__permdenied.profile

@@ -0,0 +1,4 @@
+
+mkdir -p bad
+chmod 0 bad
+cd bad

tests/data/docker/ssh_login_banner.txt

@@ -19,3 +19,5 @@ incidents to law enforcement officials.
 **************************************************************
 NOTE: This system is connected to DOMAIN.COM,
 please use your password.
+
+ستتم محاكمة المعتدين. هذا يختبر التدويل

tests/data/fd_check.py

@@ -0,0 +1,46 @@
+#!/usr/bin/env python
+
+import fcntl
+import os
+import sys
+
+
+def ttyname(fd):
+    try:
+        t = os.ttyname(fd)
+        if hasattr(t, 'decode'):
+            t = t.decode()
+        return t
+    except OSError:
+        return None
+
+
+def controlling_tty():
+    try:
+        fp = open('/dev/tty')
+        try:
+            return ttyname(fp.fileno())
+        finally:
+            fp.close()
+    except (IOError, OSError):
+        return None
+
+
+fd = int(sys.argv[2])
+st = os.fstat(fd)
+
+if sys.argv[3] == 'write':
+    os.write(fd, u'TEST'.encode())
+    buf = u''
+else:
+    buf = os.read(fd, 4).decode()
+
+open(sys.argv[1], 'w').write(repr({
+    'buf': buf,
+    'flags': fcntl.fcntl(fd, fcntl.F_GETFL),
+    'st_mode': st.st_mode,
+    'st_dev': st.st_dev,
+    'st_ino': st.st_ino,
+    'ttyname': ttyname(fd),
+    'controlling_tty': controlling_tty(),
+}))

tests/data/iter_read_generator.py

@@ -1,13 +0,0 @@
-#!/usr/bin/env python
-# I produce text every 100ms, for testing mitogen.core.iter_read()
-
-import sys
-import time
-
-
-i = 0
-while True:
-    i += 1
-    sys.stdout.write(str(i))
-    sys.stdout.flush()
-    time.sleep(0.1)

tests/data/stubs/stub-jexec.py

@@ -0,0 +1,15 @@
+#!/usr/bin/env python
+
+import json
+import os
+import subprocess
+import sys
+
+os.environ['ORIGINAL_ARGV'] = json.dumps(sys.argv)
+os.environ['THIS_IS_STUB_JEXEC'] = '1'
+
+# This must be a child process and not exec() since Mitogen replaces its stderr
+# descriptor, causing the last user of the slave PTY to close it, resulting in
+# the master side indicating EIO.
+subprocess.call(sys.argv[sys.argv.index('somejail') + 1:])
+os._exit(0)

tests/data/stubs/stub-su.py

@@ -4,6 +4,16 @@ import json
 import os
 import subprocess
 import sys
+import time
+
+# #363: old input loop would fail to spot auth failure because of scheduling
+# vs. su calling write() twice.
+if 'DO_SLOW_AUTH_FAILURE' in os.environ:
+    os.write(2, u'su: '.encode())
+    time.sleep(0.5)
+    os.write(2, u'incorrect password\n'.encode())
+    os._exit(1)
+
 
 os.environ['ORIGINAL_ARGV'] = json.dumps(sys.argv)
 os.environ['THIS_IS_STUB_SU'] = '1'

tests/data/write_all_consumer.py

@@ -1,9 +0,0 @@
-#!/usr/bin/env python
-# I consume 65535 bytes every 10ms, for testing mitogen.core.write_all()
-
-import os
-import time
-
-while True:
-    os.read(0, 65535)
-    time.sleep(0.01)

tests/doas_test.py

@@ -2,6 +2,7 @@
 import os
 
 import mitogen
+import mitogen.doas
 import mitogen.parent
 
 import unittest2
@@ -27,5 +28,38 @@ class ConstructorTest(testlib.RouterMixin, testlib.TestCase):
         self.assertEquals('1', context.call(os.getenv, 'THIS_IS_STUB_DOAS'))
 
 
+class DoasTest(testlib.DockerMixin, testlib.TestCase):
+    # Only mitogen/debian-test has doas.
+    mitogen_test_distro = 'debian'
+
+    def test_password_required(self):
+        ssh = self.docker_ssh(
+            username='mitogen__has_sudo',
+            password='has_sudo_password',
+        )
+        e = self.assertRaises(mitogen.core.StreamError,
+            lambda: self.router.doas(via=ssh)
+        )
+        self.assertTrue(mitogen.doas.password_required_msg in str(e))
+
+    def test_password_incorrect(self):
+        ssh = self.docker_ssh(
+            username='mitogen__has_sudo',
+            password='has_sudo_password',
+        )
+        e = self.assertRaises(mitogen.core.StreamError,
+            lambda: self.router.doas(via=ssh, password='x')
+        )
+        self.assertTrue(mitogen.doas.password_incorrect_msg in str(e))
+
+    def test_password_okay(self):
+        ssh = self.docker_ssh(
+            username='mitogen__has_sudo',
+            password='has_sudo_password',
+        )
+        context = self.router.su(via=ssh, password='rootpassword')
+        self.assertEquals(0, context.call(os.getuid))
+
+
 if __name__ == '__main__':
     unittest2.main()

tests/docker_test.py

@@ -21,7 +21,7 @@ class ConstructorTest(testlib.RouterMixin, testlib.TestCase):
         self.assertEquals(argv[1], 'exec')
         self.assertEquals(argv[2], '--interactive')
         self.assertEquals(argv[3], 'container_name')
-        self.assertEquals(argv[4], stream.python_path)
+        self.assertEquals(argv[4], stream.conn.options.python_path)
 
 
 if __name__ == '__main__':

tests/first_stage_test.py

@@ -19,8 +19,10 @@ class CommandLineTest(testlib.RouterMixin, testlib.TestCase):
     #   * 3.x starting 2.7
 
     def test_valid_syntax(self):
-        stream = mitogen.parent.Stream(self.router, 0, max_message_size=123)
+        options = mitogen.parent.Options(max_message_size=123)
-        args = stream.get_boot_command()
+        conn = mitogen.parent.Connection(options, self.router)
+        conn.context = mitogen.core.Context(None, 123)
+        args = conn.get_boot_command()
 
         # Executing the boot command will print "EC0" and expect to read from
         # stdin, which will fail because it's pointing at /dev/null, causing
@@ -38,7 +40,8 @@ class CommandLineTest(testlib.RouterMixin, testlib.TestCase):
             )
             stdout, stderr = proc.communicate()
             self.assertEquals(0, proc.returncode)
-            self.assertEquals(mitogen.parent.Stream.EC0_MARKER, stdout)
+            self.assertEquals(stdout,
+                mitogen.parent.BootstrapProtocol.EC0_MARKER+b('\n'))
             self.assertIn(b("Error -5 while decompressing data"), stderr)
         finally:
             fp.close()

tests/image_prep/README.md

@@ -11,10 +11,13 @@ code, the OS X config just has the user accounts.
 See ../README.md for a (mostly) description of the accounts created.
 
 
+
 ## Building the containers
 
 ``./build_docker_images.sh``
 
+Requires Ansible 2.3.x.x in order to target CentOS 5
+
 
 ## Preparing an OS X box
 

tests/image_prep/_container_setup.yml

@@ -46,65 +46,98 @@
     - when: ansible_virtualization_type != "docker"
       meta: end_play
 
-    - apt:
+    - name: Ensure requisite Debian packages are installed
+      apt:
         name: "{{packages.common + packages[distro][ver]}}"
         state: installed
         update_cache: true
       when: distro == "Debian"
 
-    - yum:
+    - name: Ensure requisite Red Hat packaed are installed
+      yum:
         name: "{{packages.common + packages[distro][ver]}}"
         state: installed
         update_cache: true
       when: distro == "CentOS"
 
-    - command: apt-get clean
+    - name: Clean up apt cache
+      command: apt-get clean
       when: distro == "Debian"
 
-    - command: yum clean all
+    - name: Clean up apt package lists
-      when: distro == "CentOS"
+      shell: rm -rf {{item}}/*
-
-    - shell: rm -rf {{item}}/*
       with_items:
       - /var/cache/apt
       - /var/lib/apt/lists
+      when: distro == "Debian"
 
-    - copy:
+    - name: Clean up yum cache
+      command: yum clean all
+      when: distro == "CentOS"
+
+    - name: Enable UTF-8 locale on Debian
+      copy:
         dest: /etc/locale.gen
         content: |
           en_US.UTF-8 UTF-8
           fr_FR.UTF-8 UTF-8
       when: distro == "Debian"
 
-    - shell: locale-gen
+    - name: Generate UTF-8 locale on Debian
+      shell: locale-gen
       when: distro == "Debian"
 
-    # Vanilla Ansible needs simplejson on CentOS 5.
+    - name: Install prebuilt 'doas' binary
-    - shell: mkdir -p /usr/lib/python2.4/site-packages/simplejson/
+      unarchive:
+        dest: /
+        src: ../data/docker/doas-debian.tar.gz
+
+    - name: Make prebuilt 'doas' binary executable
+      file:
+        path: /usr/local/bin/doas
+        mode: 'u=rwxs,go=rx'
+        owner: root
+        group: root
+
+    - name: Install doas.conf
+      copy:
+        dest: /etc/doas.conf
+        content: |
+          permit :mitogen__group
+          permit :root
+
+    - name: Vanilla Ansible needs simplejson on CentOS 5.
+      shell: mkdir -p /usr/lib/python2.4/site-packages/simplejson/
       when: distro == "CentOS" and ver == "5"
 
-    - synchronize:
+    - name: Vanilla Ansible needs simplejson on CentOS 5.
+      synchronize:
         dest: /usr/lib/python2.4/site-packages/simplejson/
         src: ../../ansible_mitogen/compat/simplejson/
       when: distro == "CentOS" and ver == "5"
 
-    - user:
+    - name: Set root user password and shell
+      user:
         name: root
         password: "{{ 'rootpassword' | password_hash('sha256') }}"
         shell: /bin/bash
 
-    - file:
+    - name: Ensure /var/run/sshd exists
+      file:
         path: /var/run/sshd
         state: directory
 
-    - command: ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key
+    - name: Generate SSH host key
+      command: ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key
       args:
         creates: /etc/ssh/ssh_host_rsa_key
 
-    - group:
+    - name: Ensure correct sudo group exists
+      group:
         name: "{{sudo_group[distro]}}"
 
-    - copy:
+    - name: Ensure /etc/sentinel exists
+      copy:
         dest: /etc/sentinel
         content: |
           i-am-mitogen-test-docker-image
@@ -119,7 +152,8 @@
         path: /etc/sudoers.d
         mode: 'u=rwx,go='
 
-    - blockinfile:
+    - name: Install test-related sudo rules
+      blockinfile:
         path: /etc/sudoers
         block: |
           # https://www.toofishes.net/blog/trouble-sudoers-or-last-entry-wins/
@@ -131,31 +165,36 @@
           Defaults>mitogen__require_tty requiretty
           Defaults>mitogen__require_tty_pw_required requiretty,targetpw
 
-    # Prevent permission denied errors.
+    - name: Prevent permission denied errors.
-    - file:
+      file:
         path: /etc/sudoers.d/README
         state: absent
 
-    - lineinfile:
+    - name: Install CentOS wheel sudo rule
+      lineinfile:
         path: /etc/sudoers
         line: "%wheel ALL=(ALL) ALL"
       when: distro == "CentOS"
 
-    - lineinfile:
+    - name: Enable SSH banner
+      lineinfile:
         path: /etc/ssh/sshd_config
         line: Banner /etc/ssh/banner.txt
 
-    - lineinfile:
+    - name: Allow remote SSH root login
+      lineinfile:
         path: /etc/ssh/sshd_config
         line: PermitRootLogin yes
         regexp: '.*PermitRootLogin.*'
 
-    - lineinfile:
+    - name: Allow remote SSH root login
+      lineinfile:
         path: /etc/pam.d/sshd
         regexp: '.*session.*required.*pam_loginuid.so'
         line: session optional pam_loginuid.so
 
-    - copy:
+    - name: Install convenience script for running an straced Python
+      copy:
         mode: 'u+rwx,go=rx'
         dest: /usr/local/bin/pywrap
         content: |

tests/image_prep/_user_accounts.yml

@@ -20,6 +20,7 @@
       - readonly_homedir
       - require_tty
       - require_tty_pw_required
+      - permdenied
       - slow_user
       - webapp
       - sudo1
@@ -98,6 +99,14 @@
       - bashrc
       - profile
 
+    - name: "Login throws permission denied errors (issue #271)"
+      copy:
+        dest: ~mitogen__permdenied/.{{item}}
+        src: ../data/docker/mitogen__permdenied.profile
+      with_items:
+      - bashrc
+      - profile
+
     - name: Install pubkey for mitogen__has_sudo_pubkey
       block:
         - file:

tests/image_prep/build_docker_images.py

@@ -26,7 +26,13 @@ label_by_id = {}
 
 for base_image, label in [
         ('astj/centos5-vault', 'centos5'),  # Python 2.4.3
-        ('debian:stretch', 'debian'),       # Python 2.7.13, 3.5.3
+        # Debian containers later than debuerreotype/debuerreotype#48 no longer
+        # ship a stub 'initctl', causing (apparently) the Ansible service
+        # module run at the end of DebOps to trigger a full stop/start of SSHd.
+        # When SSHd is killed, Docker responds by destroying the container.
+        # Proper solution is to include a full /bin/init; Docker --init doesn't
+        # help. In the meantime, just use a fixed older version.
+        ('debian:stretch-20181112', 'debian'),       # Python 2.7.13, 3.5.3
         ('centos:6', 'centos6'),            # Python 2.6.6
         ('centos:7', 'centos7')             # Python 2.7.5
     ]:

tests/iter_split_test.py

@@ -0,0 +1,66 @@
+
+import mock
+import unittest2
+
+import mitogen.core
+
+import testlib
+
+try:
+    next
+except NameError:
+    def next(it):
+        return it.next()
+
+
+class IterSplitTest(unittest2.TestCase):
+    func = staticmethod(mitogen.core.iter_split)
+
+    def test_empty_buffer(self):
+        lst = []
+        trailer, cont = self.func(buf='', delim='\n', func=lst.append)
+        self.assertTrue(cont)
+        self.assertEquals('', trailer)
+        self.assertEquals([], lst)
+
+    def test_empty_line(self):
+        lst = []
+        trailer, cont = self.func(buf='\n', delim='\n', func=lst.append)
+        self.assertTrue(cont)
+        self.assertEquals('', trailer)
+        self.assertEquals([''], lst)
+
+    def test_one_line(self):
+        buf = 'xxxx\n'
+        lst = []
+        trailer, cont = self.func(buf=buf, delim='\n', func=lst.append)
+        self.assertTrue(cont)
+        self.assertEquals('', trailer)
+        self.assertEquals(lst, ['xxxx'])
+
+    def test_one_incomplete(self):
+        buf = 'xxxx\nyy'
+        lst = []
+        trailer, cont = self.func(buf=buf, delim='\n', func=lst.append)
+        self.assertTrue(cont)
+        self.assertEquals('yy', trailer)
+        self.assertEquals(lst, ['xxxx'])
+
+    def test_returns_false_immediately(self):
+        buf = 'xxxx\nyy'
+        func = lambda buf: False
+        trailer, cont = self.func(buf=buf, delim='\n', func=func)
+        self.assertFalse(cont)
+        self.assertEquals('yy', trailer)
+
+    def test_returns_false_second_call(self):
+        buf = 'xxxx\nyy\nzz'
+        it = iter([True, False])
+        func = lambda buf: next(it)
+        trailer, cont = self.func(buf=buf, delim='\n', func=func)
+        self.assertFalse(cont)
+        self.assertEquals('zz', trailer)
+
+
+if __name__ == '__main__':
+    unittest2.main()

tests/jail_test.py

@@ -0,0 +1,33 @@
+
+import os
+
+import mitogen
+import mitogen.parent
+
+import unittest2
+
+import testlib
+
+
+class ConstructorTest(testlib.RouterMixin, testlib.TestCase):
+    jexec_path = testlib.data_path('stubs/stub-jexec.py')
+
+    def test_okay(self):
+        context = self.router.jail(
+            jexec_path=self.jexec_path,
+            container='somejail',
+        )
+        stream = self.router.stream_by_id(context.context_id)
+
+        argv = eval(context.call(os.getenv, 'ORIGINAL_ARGV'))
+        self.assertEquals(argv[:4], [
+            self.jexec_path,
+            'somejail',
+            stream.conn.options.python_path,
+            '-c',
+        ])
+        self.assertEquals('1', context.call(os.getenv, 'THIS_IS_STUB_JEXEC'))
+
+
+if __name__ == '__main__':
+    unittest2.main()

tests/lxc_test.py

@@ -38,7 +38,7 @@ class ConstructorTest(testlib.RouterMixin, testlib.TestCase):
                 lxc_attach_path='true',
             )
         )
-        self.assertTrue(str(e).endswith(mitogen.lxc.Stream.eof_error_hint))
+        self.assertTrue(str(e).endswith(mitogen.lxc.Connection.eof_error_hint))
 
 
 if __name__ == '__main__':

tests/lxd_test.py

@@ -30,7 +30,7 @@ class ConstructorTest(testlib.RouterMixin, testlib.TestCase):
                 lxc_path='true',
             )
         )
-        self.assertTrue(str(e).endswith(mitogen.lxd.Stream.eof_error_hint))
+        self.assertTrue(str(e).endswith(mitogen.lxd.Connection.eof_error_hint))
 
 
 if __name__ == '__main__':

tests/message_test.py

@@ -0,0 +1,545 @@
+
+import sys
+import struct
+
+import mock
+import unittest2
+
+import mitogen.core
+import mitogen.master
+import testlib
+
+from mitogen.core import b
+
+
+class ConstructorTest(testlib.TestCase):
+    klass = mitogen.core.Message
+
+    def test_dst_id_default(self):
+        self.assertEquals(self.klass().dst_id, None)
+
+    def test_dst_id_explicit(self):
+        self.assertEquals(self.klass(dst_id=1111).dst_id, 1111)
+
+    @mock.patch('mitogen.context_id', 1234)
+    def test_src_id_default(self):
+        self.assertEquals(self.klass().src_id, 1234)
+
+    def test_src_id_explicit(self):
+        self.assertEquals(self.klass(src_id=4321).src_id, 4321)
+
+    @mock.patch('mitogen.context_id', 5555)
+    def test_auth_id_default(self):
+        self.assertEquals(self.klass().auth_id, 5555)
+
+    def test_auth_id_explicit(self):
+        self.assertEquals(self.klass(auth_id=2222).auth_id, 2222)
+
+    def test_handle_default(self):
+        self.assertEquals(self.klass().handle, None)
+
+    def test_handle_explicit(self):
+        self.assertEquals(self.klass(handle=1234).handle, 1234)
+
+    def test_reply_to_default(self):
+        self.assertEquals(self.klass().reply_to, None)
+
+    def test_reply_to_explicit(self):
+        self.assertEquals(self.klass(reply_to=8888).reply_to, 8888)
+
+    def test_data_default(self):
+        m = self.klass()
+        self.assertEquals(m.data, b(''))
+        self.assertTrue(isinstance(m.data, mitogen.core.BytesType))
+
+    def test_data_explicit(self):
+        m = self.klass(data=b('asdf'))
+        self.assertEquals(m.data, b('asdf'))
+        self.assertTrue(isinstance(m.data, mitogen.core.BytesType))
+
+    def test_data_hates_unicode(self):
+        self.assertRaises(Exception,
+            lambda: self.klass(data=u'asdf'))
+
+
+class PackTest(testlib.TestCase):
+    klass = mitogen.core.Message
+
+    def test_header_format_sanity(self):
+        self.assertEquals(self.klass.HEADER_LEN,
+                          struct.calcsize(self.klass.HEADER_FMT))
+
+    def test_header_length_correct(self):
+        s = self.klass(dst_id=123, handle=123).pack()
+        self.assertEquals(len(s), self.klass.HEADER_LEN)
+
+    def test_magic(self):
+        s = self.klass(dst_id=123, handle=123).pack()
+        magic, = struct.unpack('>h', s[:2])
+        self.assertEquals(self.klass.HEADER_MAGIC, magic)
+
+    def test_dst_id(self):
+        s = self.klass(dst_id=123, handle=123).pack()
+        dst_id, = struct.unpack('>L', s[2:6])
+        self.assertEquals(123, dst_id)
+
+    def test_src_id(self):
+        s = self.klass(src_id=5432, dst_id=123, handle=123).pack()
+        src_id, = struct.unpack('>L', s[6:10])
+        self.assertEquals(5432, src_id)
+
+    def test_auth_id(self):
+        s = self.klass(auth_id=1919, src_id=5432, dst_id=123, handle=123).pack()
+        auth_id, = struct.unpack('>L', s[10:14])
+        self.assertEquals(1919, auth_id)
+
+    def test_handle(self):
+        s = self.klass(dst_id=123, handle=9999).pack()
+        handle, = struct.unpack('>L', s[14:18])
+        self.assertEquals(9999, handle)
+
+    def test_reply_to(self):
+        s = self.klass(dst_id=1231, handle=7777, reply_to=9132).pack()
+        reply_to, = struct.unpack('>L', s[18:22])
+        self.assertEquals(9132, reply_to)
+
+    def test_data_length_empty(self):
+        s = self.klass(dst_id=1231, handle=7777).pack()
+        data_length, = struct.unpack('>L', s[22:26])
+        self.assertEquals(0, data_length)
+
+    def test_data_length_present(self):
+        s = self.klass(dst_id=1231, handle=7777, data=b('hello')).pack()
+        data_length, = struct.unpack('>L', s[22:26])
+        self.assertEquals(5, data_length)
+
+    def test_data_empty(self):
+        s = self.klass(dst_id=1231, handle=7777).pack()
+        data = s[26:]
+        self.assertEquals(b(''), data)
+
+    def test_data_present(self):
+        s = self.klass(dst_id=11, handle=77, data=b('hello')).pack()
+        data = s[26:]
+        self.assertEquals(b('hello'), data)
+
+
+class IsDeadTest(testlib.TestCase):
+    klass = mitogen.core.Message
+
+    def test_is_dead(self):
+        msg = self.klass(reply_to=mitogen.core.IS_DEAD)
+        self.assertTrue(msg.is_dead)
+
+    def test_is_not_dead(self):
+        msg = self.klass(reply_to=5555)
+        self.assertFalse(msg.is_dead)
+
+
+class DeadTest(testlib.TestCase):
+    klass = mitogen.core.Message
+
+    def test_no_reason(self):
+        msg = self.klass.dead()
+        self.assertEquals(msg.reply_to, mitogen.core.IS_DEAD)
+        self.assertTrue(msg.is_dead)
+        self.assertEquals(msg.data, b(''))
+
+    def test_with_reason(self):
+        msg = self.klass.dead(reason=u'oh no')
+        self.assertEquals(msg.reply_to, mitogen.core.IS_DEAD)
+        self.assertTrue(msg.is_dead)
+        self.assertEquals(msg.data, b('oh no'))
+
+
+class EvilObject(object):
+    pass
+
+
+class PickledTest(testlib.TestCase):
+    # getting_started.html#rpc-serialization-rules
+    klass = mitogen.core.Message
+
+    def roundtrip(self, v, router=None):
+        msg = self.klass.pickled(v)
+        msg2 = self.klass(data=msg.data)
+        msg2.router = router
+        return msg2.unpickle()
+
+    def test_bool(self):
+        for b in True, False:
+            self.assertEquals(b, self.roundtrip(b))
+
+    @unittest2.skipIf(condition=sys.version_info < (2, 6),
+                      reason='bytearray missing on <2.6')
+    def test_bytearray(self):
+        ba = bytearray(b('123'))
+        self.assertRaises(mitogen.core.StreamError,
+            lambda: self.roundtrip(ba)
+        )
+
+    def test_bytes(self):
+        by = b('123')
+        self.assertEquals(by, self.roundtrip(by))
+
+    def test_dict(self):
+        d = {1: 2, u'a': 3, b('b'): 4, 'c': {}}
+        roundtrip = self.roundtrip(d)
+        self.assertEquals(d, roundtrip)
+        self.assertTrue(isinstance(roundtrip, dict))
+        for k in d:
+            self.assertTrue(isinstance(roundtrip[k], type(d[k])))
+
+    def test_int(self):
+        self.assertEquals(123, self.klass.pickled(123).unpickle())
+
+    def test_list(self):
+        l = [1, u'b', b('c')]
+        roundtrip = self.roundtrip(l)
+        self.assertTrue(isinstance(roundtrip, list))
+        self.assertEquals(l, roundtrip)
+        for k in range(len(l)):
+            self.assertTrue(isinstance(roundtrip[k], type(l[k])))
+
+    @unittest2.skipIf(condition=sys.version_info > (3, 0),
+                      reason='long missing in >3.x')
+    def test_long(self):
+        l = long(0xffffffffffff)
+        roundtrip = self.roundtrip(l)
+        self.assertEquals(l, roundtrip)
+        self.assertTrue(isinstance(roundtrip, long))
+
+    def test_tuple(self):
+        l = (1, u'b', b('c'))
+        roundtrip = self.roundtrip(l)
+        self.assertEquals(l, roundtrip)
+        self.assertTrue(isinstance(roundtrip, tuple))
+        for k in range(len(l)):
+            self.assertTrue(isinstance(roundtrip[k], type(l[k])))
+
+    def test_unicode(self):
+        u = u'abcd'
+        roundtrip = self.roundtrip(u)
+        self.assertEquals(u, roundtrip)
+        self.assertTrue(isinstance(roundtrip, mitogen.core.UnicodeType))
+
+    #### custom types. see also: types_test.py, call_error_test.py
+
+    # Python 3 pickle protocol 2 does weird stuff depending on whether an empty
+    # or nonempty bytes is being serialized. For non-empty, it yields a
+    # _codecs.encode() call. For empty, it yields a bytes() call.
+
+    def test_blob_nonempty(self):
+        v = mitogen.core.Blob(b('dave'))
+        roundtrip = self.roundtrip(v)
+        self.assertTrue(isinstance(roundtrip, mitogen.core.Blob))
+        self.assertEquals(b('dave'), roundtrip)
+
+    def test_blob_empty(self):
+        v = mitogen.core.Blob(b(''))
+        roundtrip = self.roundtrip(v)
+        self.assertTrue(isinstance(roundtrip, mitogen.core.Blob))
+        self.assertEquals(b(''), v)
+
+    def test_secret_nonempty(self):
+        s = mitogen.core.Secret(u'dave')
+        roundtrip = self.roundtrip(s)
+        self.assertTrue(isinstance(roundtrip, mitogen.core.Secret))
+        self.assertEquals(u'dave', roundtrip)
+
+    def test_secret_empty(self):
+        s = mitogen.core.Secret(u'')
+        roundtrip = self.roundtrip(s)
+        self.assertTrue(isinstance(roundtrip, mitogen.core.Secret))
+        self.assertEquals(u'', roundtrip)
+
+    def test_call_error(self):
+        ce = mitogen.core.CallError('nope')
+        ce2 = self.assertRaises(mitogen.core.CallError,
+            lambda: self.roundtrip(ce))
+        self.assertEquals(ce.args[0], ce2.args[0])
+
+    def test_context(self):
+        router = mitogen.master.Router()
+        try:
+            c = router.context_by_id(1234)
+            roundtrip = self.roundtrip(c)
+            self.assertTrue(isinstance(roundtrip, mitogen.core.Context))
+            self.assertEquals(c.context_id, 1234)
+        finally:
+            router.broker.shutdown()
+            router.broker.join()
+
+    def test_sender(self):
+        router = mitogen.master.Router()
+        try:
+            recv = mitogen.core.Receiver(router)
+            sender = recv.to_sender()
+            roundtrip = self.roundtrip(sender, router=router)
+            self.assertTrue(isinstance(roundtrip, mitogen.core.Sender))
+            self.assertEquals(roundtrip.context.context_id, mitogen.context_id)
+            self.assertEquals(roundtrip.dst_handle, sender.dst_handle)
+        finally:
+            router.broker.shutdown()
+            router.broker.join()
+
+    ####
+
+    def test_custom_object_deserialization_fails(self):
+        self.assertRaises(mitogen.core.StreamError,
+            lambda: self.roundtrip(EvilObject())
+        )
+
+
+class ReplyTest(testlib.TestCase):
+    # getting_started.html#rpc-serialization-rules
+    klass = mitogen.core.Message
+
+    def test_reply_calls_router_route(self):
+        msg = self.klass(src_id=1234, reply_to=9191)
+        router = mock.Mock()
+        msg.reply(123, router=router)
+        self.assertEquals(1, router.route.call_count)
+
+    def test_reply_pickles_object(self):
+        msg = self.klass(src_id=1234, reply_to=9191)
+        router = mock.Mock()
+        msg.reply(123, router=router)
+        _, (reply,), _ = router.route.mock_calls[0]
+        self.assertEquals(reply.dst_id, 1234)
+        self.assertEquals(reply.unpickle(), 123)
+
+    def test_reply_uses_preformatted_message(self):
+        msg = self.klass(src_id=1234, reply_to=9191)
+        router = mock.Mock()
+        my_reply = mitogen.core.Message.pickled(4444)
+        msg.reply(my_reply, router=router)
+        _, (reply,), _ = router.route.mock_calls[0]
+        self.assertTrue(my_reply is reply)
+        self.assertEquals(reply.dst_id, 1234)
+        self.assertEquals(reply.unpickle(), 4444)
+
+    def test_reply_sets_dst_id(self):
+        msg = self.klass(src_id=1234, reply_to=9191)
+        router = mock.Mock()
+        msg.reply(123, router=router)
+        _, (reply,), _ = router.route.mock_calls[0]
+        self.assertEquals(reply.dst_id, 1234)
+
+    def test_reply_sets_handle(self):
+        msg = self.klass(src_id=1234, reply_to=9191)
+        router = mock.Mock()
+        msg.reply(123, router=router)
+        _, (reply,), _ = router.route.mock_calls[0]
+        self.assertEquals(reply.handle, 9191)
+
+
+class UnpickleTest(testlib.TestCase):
+    # mostly done by PickleTest, just check behaviour of parameters
+    klass = mitogen.core.Message
+
+    def test_throw(self):
+        ce = mitogen.core.CallError('nope')
+        m = self.klass.pickled(ce)
+        ce2 = self.assertRaises(mitogen.core.CallError,
+            lambda: m.unpickle())
+        self.assertEquals(ce.args[0], ce2.args[0])
+
+    def test_no_throw(self):
+        ce = mitogen.core.CallError('nope')
+        m = self.klass.pickled(ce)
+        ce2 = m.unpickle(throw=False)
+        self.assertEquals(ce.args[0], ce2.args[0])
+
+    def test_throw_dead(self):
+        m = self.klass.pickled('derp', reply_to=mitogen.core.IS_DEAD)
+        self.assertRaises(mitogen.core.ChannelError,
+            lambda: m.unpickle())
+
+    def test_no_throw_dead(self):
+        m = self.klass.pickled('derp', reply_to=mitogen.core.IS_DEAD)
+        self.assertEquals('derp', m.unpickle(throw_dead=False))
+
+
+class UnpickleCompatTest(testlib.TestCase):
+    # try weird variations of pickles from different Python versions.
+    klass = mitogen.core.Message
+
+    def check(self, value, encoded, **kwargs):
+        if isinstance(encoded, mitogen.core.UnicodeType):
+            encoded = encoded.encode('latin1')
+        m = self.klass(data=encoded)
+        m.router = mitogen.master.Router()
+        try:
+            return m.unpickle(**kwargs)
+        finally:
+            m.router.broker.shutdown()
+            m.router.broker.join()
+
+    def test_py24_bytes(self):
+        self.check('test',
+           ('\x80\x02U\x04testq\x00.'))
+
+    def test_py24_unicode(self):
+        self.check(u'test',
+           ('\x80\x02X\x04\x00\x00\x00testq\x00.'))
+
+    def test_py24_int(self):
+        self.check(123,
+           ('\x80\x02K{.'))
+
+    def test_py24_long(self):
+        self.check(17592186044415,
+           ('\x80\x02\x8a\x06\xff\xff\xff\xff\xff\x0f.'))
+
+    def test_py24_dict(self):
+        self.check({},
+           ('\x80\x02}q\x00.'))
+
+    def test_py24_tuple(self):
+        self.check((1, 2, u'b'),
+           ('\x80\x02K\x01K\x02X\x01\x00\x00\x00bq\x00\x87q\x01.'))
+
+    def test_py24_bool(self):
+        self.check(True,
+           ('\x80\x02\x88.'))
+
+    def test_py24_list(self):
+        self.check([1, 2, u'b'],
+           ('\x80\x02]q\x00(K\x01K\x02X\x01\x00\x00\x00bq\x01e.'))
+
+    def test_py24_blob(self):
+        self.check(mitogen.core.mitogen.core.Blob(b('bigblob')),
+           ('\x80\x02cmitogen.core\nBlob\nq\x00U\x07bigblobq\x01\x85q\x02Rq\x03.'))
+
+    def test_py24_secret(self):
+        self.check(mitogen.core.Secret(u'mypassword'),
+           ('\x80\x02cmitogen.core\nSecret\nq\x00X\n\x00\x00\x00mypasswordq\x01\x85q\x02Rq\x03.'))
+
+    def test_py24_call_error(self):
+        self.check(mitogen.core.CallError('big error'),
+           ('\x80\x02cmitogen.core\n_unpickle_call_error\nq\x00X\t\x00\x00\x00big errorq\x01\x85q\x02R.'), throw=False)
+
+    def test_py24_context(self):
+        self.check(mitogen.core.Context(1234, None),
+           ('\x80\x02cmitogen.core\n_unpickle_context\nq\x00M\xd2\x04N\x86q\x01Rq\x02.'))
+
+    def test_py24_sender(self):
+        self.check(mitogen.core.Sender(mitogen.core.Context(55555, None), 4444),
+           ('\x80\x02cmitogen.core\n_unpickle_sender\nq\x00M\x03\xd9M\\\x11\x86q\x01Rq\x02.'))
+
+    def test_py27_bytes(self):
+        self.check(b('test'),
+           ('\x80\x02U\x04testq\x01.'))
+
+    def test_py27_unicode(self):
+        self.check(u'test',
+           ('\x80\x02X\x04\x00\x00\x00testq\x01.'))
+
+    def test_py27_int(self):
+        self.check(123,
+           ('\x80\x02K{.'))
+
+    def test_py27_long(self):
+        self.check(17592186044415,
+           ('\x80\x02\x8a\x06\xff\xff\xff\xff\xff\x0f.'))
+
+    def test_py27_dict(self):
+        self.check({},
+           ('\x80\x02}q\x01.'))
+
+    def test_py27_tuple(self):
+        self.check((1, 2, u'b'),
+           ('\x80\x02K\x01K\x02X\x01\x00\x00\x00b\x87q\x01.'))
+
+    def test_py27_bool(self):
+        self.check(True,
+           ('\x80\x02\x88.'))
+
+    def test_py27_list(self):
+        self.check([1, 2, u'b'],
+           ('\x80\x02]q\x01(K\x01K\x02X\x01\x00\x00\x00be.'))
+
+    def test_py27_blob(self):
+        self.check(mitogen.core.mitogen.core.Blob(b('bigblob')),
+           ('\x80\x02cmitogen.core\nBlob\nq\x01U\x07bigblob\x85Rq\x02.'))
+
+    def test_py27_secret(self):
+        self.check(mitogen.core.Secret(u'mypassword'),
+           ('\x80\x02cmitogen.core\nSecret\nq\x01X\n\x00\x00\x00mypassword\x85Rq\x02.'))
+
+    def test_py27_call_error(self):
+        self.check(mitogen.core.CallError(u'big error',),
+           ('\x80\x02cmitogen.core\n_unpickle_call_error\nq\x01X\t\x00\x00\x00big errorq\x02\x85Rq\x03.'), throw=False)
+
+    def test_py27_context(self):
+        self.check(mitogen.core.Context(1234, None),
+           ('\x80\x02cmitogen.core\n_unpickle_context\nq\x01M\xd2\x04N\x86Rq\x02.'))
+
+    def test_py27_sender(self):
+        self.check(mitogen.core.Sender(mitogen.core.Context(55555, None), 4444),
+           ('\x80\x02cmitogen.core\n_unpickle_sender\nq\x01M\x03\xd9M\\\x11\x86Rq\x02.'))
+
+    def test_py36_bytes(self):
+        self.check(b('test'),
+           ('\x80\x02c_codecs\nencode\nq\x00X\x04\x00\x00\x00testq\x01X\x06\x00\x00\x00latin1q\x02\x86q\x03Rq\x04.'))
+
+    def test_py36_unicode(self):
+        self.check('test',
+           ('\x80\x02X\x04\x00\x00\x00testq\x00.'))
+
+    def test_py36_int(self):
+        self.check(123,
+           ('\x80\x02K{.'))
+
+    def test_py36_long(self):
+        self.check(17592186044415,
+           ('\x80\x02\x8a\x06\xff\xff\xff\xff\xff\x0f.'))
+
+    def test_py36_dict(self):
+        self.check({},
+           ('\x80\x02}q\x00.'))
+
+    def test_py36_tuple(self):
+        self.check((1, 2, u'b'),
+           ('\x80\x02K\x01K\x02X\x01\x00\x00\x00bq\x00\x87q\x01.'))
+
+    def test_py36_bool(self):
+        self.check(True,
+           ('\x80\x02\x88.'))
+
+    def test_py36_list(self):
+        self.check([1, 2, u'b'],
+           ('\x80\x02]q\x00(K\x01K\x02X\x01\x00\x00\x00bq\x01e.'))
+
+    def test_py36_blob(self):
+        self.check(mitogen.core.mitogen.core.Blob(b('bigblob')),
+           ('\x80\x02cmitogen.core\nBlob\nq\x00c_codecs\nencode\nq\x01X\x07\x00\x00\x00bigblobq\x02X\x06\x00\x00\x00latin1q\x03\x86q\x04Rq\x05\x85q\x06Rq\x07.'))
+
+    def test_py36_secret(self):
+        self.check(mitogen.core.Secret('mypassword'),
+           ('\x80\x02cmitogen.core\nSecret\nq\x00X\n\x00\x00\x00mypasswordq\x01\x85q\x02Rq\x03.'))
+
+    def test_py36_call_error(self):
+        self.check(mitogen.core.CallError('big error'),
+           ('\x80\x02cmitogen.core\n_unpickle_call_error\nq\x00X\t\x00\x00\x00big errorq\x01\x85q\x02Rq\x03.'), throw=False)
+
+    def test_py36_context(self):
+        self.check(mitogen.core.Context(1234, None),
+           ('\x80\x02cmitogen.core\n_unpickle_context\nq\x00M\xd2\x04N\x86q\x01Rq\x02.'))
+
+    def test_py36_sender(self):
+        self.check(mitogen.core.Sender(mitogen.core.Context(55555, None), 4444),
+           ('\x80\x02cmitogen.core\n_unpickle_sender\nq\x00M\x03\xd9M\\\x11\x86q\x01Rq\x02.'))
+
+
+class ReprTest(testlib.TestCase):
+    klass = mitogen.core.Message
+
+    def test_repr(self):
+        # doesn't crash
+        repr(self.klass.pickled('test'))
+
+
+if __name__ == '__main__':
+    unittest2.main()

tests/mitogen_protocol_test.py

@@ -0,0 +1,34 @@
+
+import unittest2
+import mock
+
+import mitogen.core
+
+import testlib
+
+
+class ReceiveOneTest(testlib.TestCase):
+    klass = mitogen.core.MitogenProtocol
+
+    def test_corruption(self):
+        broker = mock.Mock()
+        router = mock.Mock()
+        stream = mock.Mock()
+
+        protocol = self.klass(router, 1)
+        protocol.stream = stream
+
+        junk = mitogen.core.b('x') * mitogen.core.Message.HEADER_LEN
+
+        capture = testlib.LogCapturer()
+        capture.start()
+        protocol.on_receive(broker, junk)
+        capture.stop()
+
+        self.assertEquals(1, stream.on_disconnect.call_count)
+        expect = self.klass.corrupt_msg % (stream.name, junk)
+        self.assertTrue(expect in capture.raw())
+
+
+if __name__ == '__main__':
+    unittest2.main()

tests/parent_test.py

@@ -14,6 +14,11 @@ from testlib import Popen__terminate
 
 import mitogen.parent
 
+try:
+    file
+except NameError:
+    from io import FileIO as file
+
 
 def wait_for_child(pid, timeout=1.0):
     deadline = time.time() + timeout
@@ -49,7 +54,7 @@ def wait_for_empty_output_queue(sync_recv, context):
     while True:
         # Now wait for the RPC to exit the output queue.
         stream = router.stream_by_id(context.context_id)
-        if broker.defer_sync(lambda: stream.pending_bytes()) == 0:
+        if broker.defer_sync(lambda: stream.protocol.pending_bytes()) == 0:
             return
         time.sleep(0.1)
 
@@ -69,35 +74,17 @@ class GetDefaultRemoteNameTest(testlib.TestCase):
         self.assertEquals("ECORP_Administrator@box:123", self.func())
 
 
-class WstatusToStrTest(testlib.TestCase):
+class ReturncodeToStrTest(testlib.TestCase):
-    func = staticmethod(mitogen.parent.wstatus_to_str)
+    func = staticmethod(mitogen.parent.returncode_to_str)
 
     def test_return_zero(self):
-        pid = os.fork()
+        self.assertEquals(self.func(0), 'exited with return code 0')
-        if not pid:
-            os._exit(0)
-        (pid, status), _ = mitogen.core.io_op(os.waitpid, pid, 0)
-        self.assertEquals(self.func(status),
-                          'exited with return code 0')
 
     def test_return_one(self):
-        pid = os.fork()
+        self.assertEquals(self.func(1), 'exited with return code 1')
-        if not pid:
-            os._exit(1)
-        (pid, status), _ = mitogen.core.io_op(os.waitpid, pid, 0)
-        self.assertEquals(
-            self.func(status),
-            'exited with return code 1'
-        )
 
     def test_sigkill(self):
-        pid = os.fork()
+        self.assertEquals(self.func(-signal.SIGKILL),
-        if not pid:
-            time.sleep(600)
-        os.kill(pid, signal.SIGKILL)
-        (pid, status), _ = mitogen.core.io_op(os.waitpid, pid, 0)
-        self.assertEquals(
-            self.func(status),
             'exited due to signal %s (SIGKILL)' % (int(signal.SIGKILL),)
         )
 
@@ -107,20 +94,20 @@ class WstatusToStrTest(testlib.TestCase):
 class ReapChildTest(testlib.RouterMixin, testlib.TestCase):
     def test_connect_timeout(self):
         # Ensure the child process is reaped if the connection times out.
-        stream = mitogen.parent.Stream(
+        options = mitogen.parent.Options(
-            router=self.router,
-            remote_id=1234,
             old_router=self.router,
             max_message_size=self.router.max_message_size,
             python_path=testlib.data_path('python_never_responds.py'),
             connect_timeout=0.5,
         )
+
+        conn = mitogen.parent.Connection(options, router=self.router)
         self.assertRaises(mitogen.core.TimeoutError,
-            lambda: stream.connect()
+            lambda: conn.connect(context=mitogen.core.Context(None, 1234))
         )
-        wait_for_child(stream.pid)
+        wait_for_child(conn.proc.pid)
         e = self.assertRaises(OSError,
-            lambda: os.kill(stream.pid, 0)
+            lambda: os.kill(conn.proc.pid, 0)
         )
         self.assertEquals(e.args[0], errno.ESRCH)
 
@@ -133,7 +120,7 @@ class StreamErrorTest(testlib.RouterMixin, testlib.TestCase):
                 connect_timeout=3,
             )
         )
-        prefix = "EOF on stream; last 300 bytes received: "
+        prefix = mitogen.parent.Connection.eof_error_msg
         self.assertTrue(e.args[0].startswith(prefix))
 
     def test_via_eof(self):
@@ -142,12 +129,12 @@ class StreamErrorTest(testlib.RouterMixin, testlib.TestCase):
         e = self.assertRaises(mitogen.core.StreamError,
             lambda: self.router.local(
                 via=local,
-                python_path='true',
+                python_path='echo',
                 connect_timeout=3,
             )
         )
-        s = "EOF on stream; last 300 bytes received: "
+        expect = mitogen.parent.Connection.eof_error_msg
-        self.assertTrue(s in e.args[0])
+        self.assertTrue(expect in e.args[0])
 
     def test_direct_enoent(self):
         e = self.assertRaises(mitogen.core.StreamError,
@@ -185,11 +172,15 @@ class OpenPtyTest(testlib.TestCase):
     func = staticmethod(mitogen.parent.openpty)
 
     def test_pty_returned(self):
-        master_fd, slave_fd = self.func()
+        master_fp, slave_fp = self.func()
-        self.assertTrue(isinstance(master_fd, int))
+        try:
-        self.assertTrue(isinstance(slave_fd, int))
+            self.assertTrue(master_fp.isatty())
-        os.close(master_fd)
+            self.assertTrue(isinstance(master_fp, file))
-        os.close(slave_fd)
+            self.assertTrue(slave_fp.isatty())
+            self.assertTrue(isinstance(slave_fp, file))
+        finally:
+            master_fp.close()
+            slave_fp.close()
 
     @mock.patch('os.openpty')
     def test_max_reached(self, openpty):
@@ -204,20 +195,20 @@ class OpenPtyTest(testlib.TestCase):
     @mock.patch('os.openpty')
     def test_broken_linux_fallback(self, openpty):
         openpty.side_effect = OSError(errno.EPERM)
-        master_fd, slave_fd = self.func()
+        master_fp, slave_fp = self.func()
         try:
-            st = os.fstat(master_fd)
+            st = os.fstat(master_fp.fileno())
             self.assertEquals(5, os.major(st.st_rdev))
-            flags = fcntl.fcntl(master_fd, fcntl.F_GETFL)
+            flags = fcntl.fcntl(master_fp.fileno(), fcntl.F_GETFL)
             self.assertTrue(flags & os.O_RDWR)
 
-            st = os.fstat(slave_fd)
+            st = os.fstat(slave_fp.fileno())
             self.assertEquals(136, os.major(st.st_rdev))
-            flags = fcntl.fcntl(slave_fd, fcntl.F_GETFL)
+            flags = fcntl.fcntl(slave_fp.fileno(), fcntl.F_GETFL)
             self.assertTrue(flags & os.O_RDWR)
         finally:
-            os.close(master_fd)
+            master_fp.close()
-            os.close(slave_fd)
+            slave_fp.close()
 
 
 class TtyCreateChildTest(testlib.TestCase):
@@ -235,123 +226,20 @@ class TtyCreateChildTest(testlib.TestCase):
         # read a password.
         tf = tempfile.NamedTemporaryFile()
         try:
-            pid, fd, _ = self.func([
+            proc = self.func([
                 'bash', '-c', 'exec 2>%s; echo hi > /dev/tty' % (tf.name,)
             ])
             deadline = time.time() + 5.0
-            for line in mitogen.parent.iter_read([fd], deadline):
+            mitogen.core.set_block(proc.stdin.fileno())
-                self.assertEquals(mitogen.core.b('hi\n'), line)
+            # read(3) below due to https://bugs.python.org/issue37696
-                break
+            self.assertEquals(mitogen.core.b('hi\n'), proc.stdin.read(3))
-            waited_pid, status = os.waitpid(pid, 0)
+            waited_pid, status = os.waitpid(proc.pid, 0)
-            self.assertEquals(pid, waited_pid)
+            self.assertEquals(proc.pid, waited_pid)
             self.assertEquals(0, status)
             self.assertEquals(mitogen.core.b(''), tf.read())
-            os.close(fd)
-        finally:
-            tf.close()
-
-
-class IterReadTest(testlib.TestCase):
-    func = staticmethod(mitogen.parent.iter_read)
-
-    def make_proc(self):
-        # I produce text every 100ms.
-        args = [testlib.data_path('iter_read_generator.py')]
-        proc = subprocess.Popen(args, stdout=subprocess.PIPE)
-        mitogen.core.set_nonblock(proc.stdout.fileno())
-        return proc
-
-    def test_no_deadline(self):
-        proc = self.make_proc()
-        try:
-            reader = self.func([proc.stdout.fileno()])
-            for i, chunk in enumerate(reader):
-                self.assertEqual(1+i, int(chunk))
-                if i > 2:
-                    break
-        finally:
-            Popen__terminate(proc)
-            proc.stdout.close()
-
-    def test_deadline_exceeded_before_call(self):
-        proc = self.make_proc()
-        reader = self.func([proc.stdout.fileno()], 0)
-        try:
-            got = []
-            try:
-                for chunk in reader:
-                    got.append(chunk)
-                assert 0, 'TimeoutError not raised'
-            except mitogen.core.TimeoutError:
-                self.assertEqual(len(got), 0)
-        finally:
-            Popen__terminate(proc)
             proc.stdout.close()
-
-    def test_deadline_exceeded_during_call(self):
-        proc = self.make_proc()
-        deadline = time.time() + 0.4
-
-        reader = self.func([proc.stdout.fileno()], deadline)
-        try:
-            got = []
-            try:
-                for chunk in reader:
-                    if time.time() > (deadline + 1.0):
-                        assert 0, 'TimeoutError not raised'
-                    got.append(chunk)
-            except mitogen.core.TimeoutError:
-                # Give a little wiggle room in case of imperfect scheduling.
-                # Ideal number should be 9.
-                self.assertLess(deadline, time.time())
-                self.assertLess(1, len(got))
-                self.assertLess(len(got), 20)
         finally:
-            Popen__terminate(proc)
+            tf.close()
-            proc.stdout.close()
-
-
-class WriteAllTest(testlib.TestCase):
-    func = staticmethod(mitogen.parent.write_all)
-
-    def make_proc(self):
-        args = [testlib.data_path('write_all_consumer.py')]
-        proc = subprocess.Popen(args, stdin=subprocess.PIPE)
-        mitogen.core.set_nonblock(proc.stdin.fileno())
-        return proc
-
-    ten_ms_chunk = (mitogen.core.b('x') * 65535)
-
-    def test_no_deadline(self):
-        proc = self.make_proc()
-        try:
-            self.func(proc.stdin.fileno(), self.ten_ms_chunk)
-        finally:
-            Popen__terminate(proc)
-            proc.stdin.close()
-
-    def test_deadline_exceeded_before_call(self):
-        proc = self.make_proc()
-        try:
-            self.assertRaises(mitogen.core.TimeoutError, (
-                lambda: self.func(proc.stdin.fileno(), self.ten_ms_chunk, 0)
-            ))
-        finally:
-            Popen__terminate(proc)
-            proc.stdin.close()
-
-    def test_deadline_exceeded_during_call(self):
-        proc = self.make_proc()
-        try:
-            deadline = time.time() + 0.1   # 100ms deadline
-            self.assertRaises(mitogen.core.TimeoutError, (
-                lambda: self.func(proc.stdin.fileno(),
-                                  self.ten_ms_chunk * 100,  # 1s of data
-                                  deadline)
-            ))
-        finally:
-            Popen__terminate(proc)
-            proc.stdin.close()
 
 
 class DisconnectTest(testlib.RouterMixin, testlib.TestCase):
@@ -394,7 +282,7 @@ class DisconnectTest(testlib.RouterMixin, testlib.TestCase):
         c2 = self.router.local()
 
         # Let c1 call functions in c2.
-        self.router.stream_by_id(c1.context_id).auth_id = mitogen.context_id
+        self.router.stream_by_id(c1.context_id).protocol.auth_id = mitogen.context_id
         c1.call(mitogen.parent.upgrade_router)
 
         sync_recv = mitogen.core.Receiver(self.router)
@@ -412,14 +300,14 @@ class DisconnectTest(testlib.RouterMixin, testlib.TestCase):
     def test_far_sibling_disconnected(self):
         # God mode: child of child notices child of child of parent has
         # disconnected.
-        c1 = self.router.local()
+        c1 = self.router.local(name='c1')
-        c11 = self.router.local(via=c1)
+        c11 = self.router.local(name='c11', via=c1)
 
-        c2 = self.router.local()
+        c2 = self.router.local(name='c2')
-        c22 = self.router.local(via=c2)
+        c22 = self.router.local(name='c22', via=c2)
 
         # Let c1 call functions in c2.
-        self.router.stream_by_id(c1.context_id).auth_id = mitogen.context_id
+        self.router.stream_by_id(c1.context_id).protocol.auth_id = mitogen.context_id
         c11.call(mitogen.parent.upgrade_router)
 
         sync_recv = mitogen.core.Receiver(self.router)

tests/poller_test.py

@@ -42,8 +42,8 @@ class SockMixin(object):
         self.l2_sock, self.r2_sock = socket.socketpair()
         self.l2 = self.l2_sock.fileno()
         self.r2 = self.r2_sock.fileno()
-        for fd in self.l1, self.r1, self.l2, self.r2:
+        for fp in self.l1, self.r1, self.l2, self.r2:
-            mitogen.core.set_nonblock(fd)
+            mitogen.core.set_nonblock(fp)
 
     def fill(self, fd):
         """Make `fd` unwriteable."""
@@ -354,17 +354,17 @@ class FileClosedMixin(PollerMixin, SockMixin):
 class TtyHangupMixin(PollerMixin):
     def test_tty_hangup_detected(self):
         # bug in initial select.poll() implementation failed to detect POLLHUP.
-        master_fd, slave_fd = mitogen.parent.openpty()
+        master_fp, slave_fp = mitogen.parent.openpty()
         try:
-            self.p.start_receive(master_fd)
+            self.p.start_receive(master_fp.fileno())
             self.assertEquals([], list(self.p.poll(0)))
-            os.close(slave_fd)
+            slave_fp.close()
-            slave_fd = None
+            slave_fp = None
-            self.assertEquals([master_fd], list(self.p.poll(0)))
+            self.assertEquals([master_fp.fileno()], list(self.p.poll(0)))
         finally:
-            if slave_fd is not None:
+            if slave_fp is not None:
-                os.close(slave_fd)
+                slave_fp.close()
-            os.close(master_fd)
+            master_fp.close()
 
 
 class DistinctDataMixin(PollerMixin, SockMixin):

tests/requirements.txt

@@ -13,3 +13,5 @@ unittest2==1.1.0
 # Fix InsecurePlatformWarning while creating py26 tox environment
 # https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
 urllib3[secure]; python_version < '2.7.9'
+# Last idna compatible with Python 2.6 was idna 2.7.
+idna==2.7; python_version < '2.7'

tests/responder_test.py

@@ -105,7 +105,7 @@ class BrokenModulesTest(testlib.TestCase):
         # unavailable. Should never happen in the real world.
 
         stream = mock.Mock()
-        stream.sent_modules = set()
+        stream.protocol.sent_modules = set()
         router = mock.Mock()
         router.stream_by_id = lambda n: stream
 
@@ -143,7 +143,7 @@ class BrokenModulesTest(testlib.TestCase):
         import six_brokenpkg
 
         stream = mock.Mock()
-        stream.sent_modules = set()
+        stream.protocol.sent_modules = set()
         router = mock.Mock()
         router.stream_by_id = lambda n: stream
 

tests/router_test.py

@@ -171,7 +171,7 @@ class CrashTest(testlib.BrokerMixin, testlib.TestCase):
         self.assertTrue(sem.get().is_dead)
 
         # Ensure it was logged.
-        expect = '_broker_main() crashed'
+        expect = 'broker crashed'
         self.assertTrue(expect in log.stop())
 
         self.broker.join()
@@ -364,8 +364,8 @@ class UnidirectionalTest(testlib.RouterMixin, testlib.TestCase):
         # treated like a parent.
         l1 = self.router.local()
         l1s = self.router.stream_by_id(l1.context_id)
-        l1s.auth_id = mitogen.context_id
+        l1s.protocol.auth_id = mitogen.context_id
-        l1s.is_privileged = True
+        l1s.protocol.is_privileged = True
 
         l2 = self.router.local()
         e = self.assertRaises(mitogen.core.CallError,
@@ -378,12 +378,21 @@ class UnidirectionalTest(testlib.RouterMixin, testlib.TestCase):
 class EgressIdsTest(testlib.RouterMixin, testlib.TestCase):
     def test_egress_ids_populated(self):
         # Ensure Stream.egress_ids is populated on message reception.
-        c1 = self.router.local()
+        c1 = self.router.local(name='c1')
-        stream = self.router.stream_by_id(c1.context_id)
+        c2 = self.router.local(name='c2')
-        self.assertEquals(set(), stream.egress_ids)
 
-        c1.call(time.sleep, 0)
+        c1s = self.router.stream_by_id(c1.context_id)
-        self.assertEquals(set([mitogen.context_id]), stream.egress_ids)
+        try:
+            c1.call(ping_context, c2)
+        except mitogen.core.CallError:
+            # Fails because siblings cant call funcs in each other, but this
+            # causes messages to be sent.
+            pass
+
+        self.assertEquals(c1s.protocol.egress_ids, set([
+            mitogen.context_id,
+            c2.context_id,
+        ]))
 
 
 if __name__ == '__main__':

tests/service_test.py

@@ -44,8 +44,8 @@ class ActivationTest(testlib.RouterMixin, testlib.TestCase):
         self.assertTrue(isinstance(id_, int))
 
     def test_sibling_cannot_activate_framework(self):
-        l1 = self.router.local()
+        l1 = self.router.local(name='l1')
-        l2 = self.router.local()
+        l2 = self.router.local(name='l2')
         exc = self.assertRaises(mitogen.core.CallError,
             lambda: l2.call(call_service_in, l1, MyService2.name(), 'get_id'))
         self.assertTrue(mitogen.core.Router.refused_msg in exc.args[0])

tests/setns_test.py

@@ -0,0 +1,46 @@
+
+import os
+import socket
+import sys
+
+import mitogen
+import mitogen.parent
+
+import unittest2
+
+import testlib
+
+
+class DockerTest(testlib.DockerMixin, testlib.TestCase):
+    def test_okay(self):
+        # Magic calls must happen as root.
+        try:
+            root = self.router.sudo()
+        except mitogen.core.StreamError:
+            raise unittest2.SkipTest("requires sudo to localhost root")
+
+        via_ssh = self.docker_ssh(
+            username='mitogen__has_sudo',
+            password='has_sudo_password',
+        )
+
+        via_setns = self.router.setns(
+            kind='docker',
+            container=self.dockerized_ssh.container_name,
+            via=root,
+        )
+
+        self.assertEquals(
+            via_ssh.call(socket.gethostname),
+            via_setns.call(socket.gethostname),
+        )
+
+
+DockerTest = unittest2.skipIf(
+    condition=sys.version_info < (2, 5),
+    reason="mitogen.setns unsupported on Python <2.4"
+)(DockerTest)
+
+
+if __name__ == '__main__':
+    unittest2.main()

tests/ssh_test.py

@@ -42,8 +42,6 @@ class ConstructorTest(testlib.RouterMixin, testlib.TestCase):
 
 
 class SshTest(testlib.DockerMixin, testlib.TestCase):
-    stream_class = mitogen.ssh.Stream
-
     def test_debug_decoding(self):
         # ensure filter_debug_logs() decodes the logged string.
         capture = testlib.LogCapturer()
@@ -60,6 +58,14 @@ class SshTest(testlib.DockerMixin, testlib.TestCase):
         expect = "%s: debug1: Reading configuration data" % (context.name,)
         self.assertTrue(expect in s)
 
+    def test_bash_permission_denied(self):
+        # issue #271: only match Permission Denied at start of line.
+        context = self.docker_ssh(
+            username='mitogen__permdenied',
+            password='permdenied_password',
+            ssh_debug_level=3,
+        )
+
     def test_stream_name(self):
         context = self.docker_ssh(
             username='mitogen__has_sudo',
@@ -85,27 +91,21 @@ class SshTest(testlib.DockerMixin, testlib.TestCase):
         self.assertEquals(name, sudo.name)
 
     def test_password_required(self):
-        try:
+        e = self.assertRaises(mitogen.ssh.PasswordError,
-            context = self.docker_ssh(
+            lambda: self.docker_ssh(
                 username='mitogen__has_sudo',
             )
-            assert 0, 'exception not thrown'
+        )
-        except mitogen.ssh.PasswordError:
+        self.assertEqual(e.args[0], mitogen.ssh.password_required_msg)
-            e = sys.exc_info()[1]
-
-        self.assertEqual(e.args[0], self.stream_class.password_required_msg)
 
     def test_password_incorrect(self):
-        try:
+        e = self.assertRaises(mitogen.ssh.PasswordError,
-            context = self.docker_ssh(
+            lambda: self.docker_ssh(
                 username='mitogen__has_sudo',
                 password='badpw',
             )
-            assert 0, 'exception not thrown'
+        )
-        except mitogen.ssh.PasswordError:
+        self.assertEqual(e.args[0], mitogen.ssh.password_incorrect_msg)
-            e = sys.exc_info()[1]
-
-        self.assertEqual(e.args[0], self.stream_class.password_incorrect_msg)
 
     def test_password_specified(self):
         context = self.docker_ssh(
@@ -119,15 +119,12 @@ class SshTest(testlib.DockerMixin, testlib.TestCase):
         )
 
     def test_pubkey_required(self):
-        try:
+        e = self.assertRaises(mitogen.ssh.PasswordError,
-            context = self.docker_ssh(
+            lambda: self.docker_ssh(
                 username='mitogen__has_sudo_pubkey',
             )
-            assert 0, 'exception not thrown'
+        )
-        except mitogen.ssh.PasswordError:
+        self.assertEqual(e.args[0], mitogen.ssh.password_required_msg)
-            e = sys.exc_info()[1]
-
-        self.assertEqual(e.args[0], self.stream_class.password_required_msg)
 
     def test_pubkey_specified(self):
         context = self.docker_ssh(
@@ -150,7 +147,7 @@ class SshTest(testlib.DockerMixin, testlib.TestCase):
                     check_host_keys='enforce',
                 )
             )
-            self.assertEquals(e.args[0], mitogen.ssh.Stream.hostkey_failed_msg)
+            self.assertEquals(e.args[0], mitogen.ssh.hostkey_failed_msg)
         finally:
             fp.close()
 
@@ -184,8 +181,6 @@ class SshTest(testlib.DockerMixin, testlib.TestCase):
 class BannerTest(testlib.DockerMixin, testlib.TestCase):
     # Verify the ability to disambiguate random spam appearing in the SSHd's
     # login banner from a legitimate password prompt.
-    stream_class = mitogen.ssh.Stream
-
     def test_verbose_enabled(self):
         context = self.docker_ssh(
             username='mitogen__has_sudo',
@@ -210,8 +205,6 @@ class StubPermissionDeniedTest(StubSshMixin, testlib.TestCase):
 
 
 class StubCheckHostKeysTest(StubSshMixin, testlib.TestCase):
-    stream_class = mitogen.ssh.Stream
-
     def test_check_host_keys_accept(self):
         # required=true, host_key_checking=accept
         context = self.stub_ssh(STUBSSH_MODE='ask', check_host_keys='accept')

tests/stream_test.py

@@ -1,33 +0,0 @@
-
-import unittest2
-import mock
-
-import mitogen.core
-
-import testlib
-
-
-class ReceiveOneTest(testlib.TestCase):
-    klass = mitogen.core.Stream
-
-    def test_corruption(self):
-        broker = mock.Mock()
-        router = mock.Mock()
-
-        stream = self.klass(router, 1)
-        junk = mitogen.core.b('x') * stream.HEADER_LEN
-        stream._input_buf = [junk]
-        stream._input_buf_len = len(junk)
-
-        capture = testlib.LogCapturer()
-        capture.start()
-        ret = stream._receive_one(broker)
-        #self.assertEquals(1, broker.stop_receive.mock_calls)
-        capture.stop()
-
-        self.assertFalse(ret)
-        self.assertTrue((self.klass.corrupt_msg % (junk,)) in capture.raw())
-
-
-if __name__ == '__main__':
-    unittest2.main()

tests/su_test.py

@@ -2,8 +2,7 @@
 import os
 
 import mitogen
-import mitogen.lxd
+import mitogen.su
-import mitogen.parent
 
 import unittest2
 
@@ -11,22 +10,64 @@ import testlib
 
 
 class ConstructorTest(testlib.RouterMixin, testlib.TestCase):
-    su_path = testlib.data_path('stubs/stub-su.py')
+    stub_su_path = testlib.data_path('stubs/stub-su.py')
 
     def run_su(self, **kwargs):
         context = self.router.su(
-            su_path=self.su_path,
+            su_path=self.stub_su_path,
             **kwargs
         )
         argv = eval(context.call(os.getenv, 'ORIGINAL_ARGV'))
         return context, argv
 
-
     def test_basic(self):
         context, argv = self.run_su()
         self.assertEquals(argv[1], 'root')
         self.assertEquals(argv[2], '-c')
 
 
+class SuTest(testlib.DockerMixin, testlib.TestCase):
+    stub_su_path = testlib.data_path('stubs/stub-su.py')
+
+    def test_slow_auth_failure(self):
+        # #363: old input loop would fail to spot auth failure because of
+        # scheduling vs. su calling write() twice.
+        os.environ['DO_SLOW_AUTH_FAILURE'] = '1'
+        try:
+            self.assertRaises(mitogen.su.PasswordError,
+                lambda: self.router.su(su_path=self.stub_su_path)
+            )
+        finally:
+            del os.environ['DO_SLOW_AUTH_FAILURE']
+
+    def test_password_required(self):
+        ssh = self.docker_ssh(
+            username='mitogen__has_sudo',
+            password='has_sudo_password',
+        )
+        e = self.assertRaises(mitogen.core.StreamError,
+            lambda: self.router.su(via=ssh)
+        )
+        self.assertTrue(mitogen.su.password_required_msg in str(e))
+
+    def test_password_incorrect(self):
+        ssh = self.docker_ssh(
+            username='mitogen__has_sudo',
+            password='has_sudo_password',
+        )
+        e = self.assertRaises(mitogen.core.StreamError,
+            lambda: self.router.su(via=ssh, password='x')
+        )
+        self.assertTrue(mitogen.su.password_incorrect_msg in str(e))
+
+    def test_password_okay(self):
+        ssh = self.docker_ssh(
+            username='mitogen__has_sudo',
+            password='has_sudo_password',
+        )
+        context = self.router.su(via=ssh, password='rootpassword')
+        self.assertEquals(0, context.call(os.getuid))
+
+
 if __name__ == '__main__':
     unittest2.main()

tests/sudo_test.py

@@ -2,8 +2,7 @@
 import os
 
 import mitogen
-import mitogen.lxd
+import mitogen.sudo
-import mitogen.parent
 
 import unittest2
 
@@ -79,7 +78,7 @@ class NonEnglishPromptTest(testlib.DockerMixin, testlib.TestCase):
         e = self.assertRaises(mitogen.core.StreamError,
             lambda: self.router.sudo(via=ssh)
         )
-        self.assertTrue(mitogen.sudo.Stream.password_required_msg in str(e))
+        self.assertTrue(mitogen.sudo.password_required_msg in str(e))
 
     def test_password_incorrect(self):
         ssh = self.docker_ssh(
@@ -91,7 +90,7 @@ class NonEnglishPromptTest(testlib.DockerMixin, testlib.TestCase):
         e = self.assertRaises(mitogen.core.StreamError,
             lambda: self.router.sudo(via=ssh, password='x')
         )
-        self.assertTrue(mitogen.sudo.Stream.password_incorrect_msg in str(e))
+        self.assertTrue(mitogen.sudo.password_incorrect_msg in str(e))
 
     def test_password_okay(self):
         ssh = self.docker_ssh(
@@ -103,7 +102,7 @@ class NonEnglishPromptTest(testlib.DockerMixin, testlib.TestCase):
         e = self.assertRaises(mitogen.core.StreamError,
             lambda: self.router.sudo(via=ssh, password='rootpassword')
         )
-        self.assertTrue(mitogen.sudo.Stream.password_incorrect_msg in str(e))
+        self.assertTrue(mitogen.sudo.password_incorrect_msg in str(e))
 
 
 if __name__ == '__main__':

tests/testlib.py

@@ -283,7 +283,11 @@ class LogCapturer(object):
         self.logger.level = logging.DEBUG
 
     def raw(self):
-        return self.sio.getvalue()
+        s = self.sio.getvalue()
+        # Python 2.x logging package hard-wires UTF-8 output.
+        if isinstance(s, mitogen.core.BytesType):
+            s = s.decode('utf-8')
+        return s
 
     def msgs(self):
         return self.handler.msgs
@@ -327,17 +331,36 @@ class TestCase(unittest2.TestCase):
 
         for name in counts:
             assert counts[name] == 1, \
-                'Found %d copies of thread %r running after tests.' % (name,)
+                'Found %d copies of thread %r running after tests.' % (
+                    counts[name], name
+                )
 
     def _teardown_check_fds(self):
         mitogen.core.Latch._on_fork()
         if get_fd_count() != self._fd_count_before:
-            import os; os.system('lsof -p %s' % (os.getpid(),))
+            import os; os.system('lsof -w -p %s' % (os.getpid(),))
             assert 0, "%s leaked FDs. Count before: %s, after: %s" % (
                 self, self._fd_count_before, get_fd_count(),
             )
 
+    def _teardown_check_zombies(self):
+        try:
+            pid, status = os.waitpid(0, os.WNOHANG)
+        except OSError:
+            return  # ECHILD
+
+        if pid:
+            assert 0, "%s failed to reap subprocess %d (status %d)." % (
+                self, pid, status
+            )
+
+        print()
+        print('Children of unit test process:')
+        os.system('ps uww --ppid ' + str(os.getpid()))
+        assert 0, "%s leaked still-running subprocesses." % (self,)
+
     def tearDown(self):
+        self._teardown_check_zombies()
         self._teardown_check_threads()
         self._teardown_check_fds()
         super(TestCase, self).tearDown()

tests/timer_test.py

@@ -0,0 +1,189 @@
+
+import time
+
+import mock
+import unittest2
+
+import mitogen.core
+import mitogen.parent
+
+import testlib
+
+
+class TimerListMixin(object):
+    klass = mitogen.parent.TimerList
+
+    def setUp(self):
+        self.list = self.klass()
+
+
+class GetTimeoutTest(TimerListMixin, testlib.TestCase):
+    def test_empty(self):
+        self.assertEquals(None, self.list.get_timeout())
+
+    def test_one_event(self):
+        self.list.schedule(2, lambda: None)
+        self.list._now = lambda: 1
+        self.assertEquals(1, self.list.get_timeout())
+
+    def test_two_events_same_moment(self):
+        self.list.schedule(2, lambda: None)
+        self.list.schedule(2, lambda: None)
+        self.list._now = lambda: 1
+        self.assertEquals(1, self.list.get_timeout())
+
+    def test_two_events(self):
+        self.list.schedule(2, lambda: None)
+        self.list.schedule(3, lambda: None)
+        self.list._now = lambda: 1
+        self.assertEquals(1, self.list.get_timeout())
+
+    def test_two_events_expired(self):
+        self.list.schedule(2, lambda: None)
+        self.list.schedule(3, lambda: None)
+        self.list._now = lambda: 3
+        self.assertEquals(0, self.list.get_timeout())
+
+    def test_two_events_in_past(self):
+        self.list.schedule(2, lambda: None)
+        self.list.schedule(3, lambda: None)
+        self.list._now = lambda: 30
+        self.assertEquals(0, self.list.get_timeout())
+
+    def test_two_events_in_past(self):
+        self.list.schedule(2, lambda: None)
+        self.list.schedule(3, lambda: None)
+        self.list._now = lambda: 30
+        self.assertEquals(0, self.list.get_timeout())
+
+    def test_one_cancelled(self):
+        t1 = self.list.schedule(2, lambda: None)
+        t2 = self.list.schedule(3, lambda: None)
+        self.list._now = lambda: 0
+        t1.cancel()
+        self.assertEquals(3, self.list.get_timeout())
+
+    def test_two_cancelled(self):
+        t1 = self.list.schedule(2, lambda: None)
+        t2 = self.list.schedule(3, lambda: None)
+        self.list._now = lambda: 0
+        t1.cancel()
+        t2.cancel()
+        self.assertEquals(None, self.list.get_timeout())
+
+
+class ScheduleTest(TimerListMixin, testlib.TestCase):
+    def test_in_past(self):
+        self.list._now = lambda: 30
+        timer = self.list.schedule(29, lambda: None)
+        self.assertEquals(29, timer.when)
+        self.assertEquals(0, self.list.get_timeout())
+
+    def test_in_future(self):
+        self.list._now = lambda: 30
+        timer = self.list.schedule(31, lambda: None)
+        self.assertEquals(31, timer.when)
+        self.assertEquals(1, self.list.get_timeout())
+
+    def test_same_moment(self):
+        self.list._now = lambda: 30
+        timer = self.list.schedule(31, lambda: None)
+        timer2 = self.list.schedule(31, lambda: None)
+        self.assertEquals(31, timer.when)
+        self.assertEquals(31, timer2.when)
+        self.assertTrue(timer is not timer2)
+        self.assertEquals(1, self.list.get_timeout())
+
+
+class ExpireTest(TimerListMixin, testlib.TestCase):
+    def test_in_past(self):
+        timer = self.list.schedule(29, mock.Mock())
+        self.list._now = lambda: 30
+        self.list.expire()
+        self.assertEquals(1, len(timer.func.mock_calls))
+
+    def test_in_future(self):
+        timer = self.list.schedule(29, mock.Mock())
+        self.list._now = lambda: 28
+        self.list.expire()
+        self.assertEquals(0, len(timer.func.mock_calls))
+
+    def test_same_moment(self):
+        timer = self.list.schedule(29, mock.Mock())
+        timer2 = self.list.schedule(29, mock.Mock())
+        self.list._now = lambda: 29
+        self.list.expire()
+        self.assertEquals(1, len(timer.func.mock_calls))
+        self.assertEquals(1, len(timer2.func.mock_calls))
+
+    def test_cancelled(self):
+        self.list._now = lambda: 29
+        timer = self.list.schedule(29, mock.Mock())
+        timer.cancel()
+        self.assertEquals(None, self.list.get_timeout())
+        self.list._now = lambda: 29
+        self.list.expire()
+        self.assertEquals(0, len(timer.func.mock_calls))
+        self.assertEquals(None, self.list.get_timeout())
+
+
+class CancelTest(TimerListMixin, testlib.TestCase):
+    def test_single_cancel(self):
+        self.list._now = lambda: 29
+        timer = self.list.schedule(29, mock.Mock())
+        timer.cancel()
+        self.list.expire()
+        self.assertEquals(0, len(timer.func.mock_calls))
+
+    def test_double_cancel(self):
+        self.list._now = lambda: 29
+        timer = self.list.schedule(29, mock.Mock())
+        timer.cancel()
+        timer.cancel()
+        self.list.expire()
+        self.assertEquals(0, len(timer.func.mock_calls))
+
+
+@mitogen.core.takes_econtext
+def do_timer_test_econtext(econtext):
+    do_timer_test(econtext.broker)
+
+
+def do_timer_test(broker):
+    now = time.time()
+    latch = mitogen.core.Latch()
+    broker.defer(lambda:
+        broker.timers.schedule(
+            now + 0.250,
+            lambda: latch.put('hi'),
+        )
+    )
+
+    assert 'hi' == latch.get()
+    assert time.time() > (now + 0.250)
+
+
+class BrokerTimerTest(testlib.TestCase):
+    klass = mitogen.master.Broker
+
+    def test_call_later(self):
+        broker = self.klass()
+        try:
+            do_timer_test(broker)
+        finally:
+            broker.shutdown()
+            broker.join()
+
+    def test_child_upgrade(self):
+        router = mitogen.master.Router()
+        try:
+            c = router.local()
+            c.call(mitogen.parent.upgrade_router)
+            c.call(do_timer_test_econtext)
+        finally:
+            router.broker.shutdown()
+            router.broker.join()
+
+
+if __name__ == '__main__':
+    unittest2.main()

tests/types_test.py

@@ -16,6 +16,11 @@ from mitogen.core import b
 import testlib
 
 
+####
+#### see also message_test.py / PickledTest
+####
+
+
 class BlobTest(testlib.TestCase):
     klass = mitogen.core.Blob
 

tests/unix_test.py

@@ -67,12 +67,12 @@ class ListenerTest(testlib.RouterMixin, testlib.TestCase):
     klass = mitogen.unix.Listener
 
     def test_constructor_basic(self):
-        listener = self.klass(router=self.router)
+        listener = self.klass.build_stream(router=self.router)
         capture = testlib.LogCapturer()
         capture.start()
         try:
-            self.assertFalse(mitogen.unix.is_path_dead(listener.path))
+            self.assertFalse(mitogen.unix.is_path_dead(listener.protocol.path))
-            os.unlink(listener.path)
+            os.unlink(listener.protocol.path)
             # ensure we catch 0 byte read error log message
             self.broker.shutdown()
             self.broker.join()
@@ -86,22 +86,25 @@ class ClientTest(testlib.TestCase):
 
     def _try_connect(self, path):
         # give server a chance to setup listener
-        for x in range(10):
+        timeout = time.time() + 30.0
+        while True:
             try:
                 return mitogen.unix.connect(path)
             except socket.error:
-                if x == 9:
+                if time.time() > timeout:
                     raise
                 time.sleep(0.1)
 
     def _test_simple_client(self, path):
         router, context = self._try_connect(path)
+        try:
             self.assertEquals(0, context.context_id)
             self.assertEquals(1, mitogen.context_id)
             self.assertEquals(0, mitogen.parent_id)
             resp = context.call_service(service_name=MyService, method_name='ping')
             self.assertEquals(mitogen.context_id, resp['src_id'])
             self.assertEquals(0, resp['auth_id'])
+        finally:
             router.broker.shutdown()
             router.broker.join()
             os.unlink(path)
@@ -112,7 +115,7 @@ class ClientTest(testlib.TestCase):
         latch = mitogen.core.Latch()
         try:
             try:
-                listener = cls.klass(path=path, router=router)
+                listener = cls.klass.build_stream(path=path, router=router)
                 pool = mitogen.service.Pool(router=router, services=[
                     MyService(latch=latch, router=router),
                 ])