mitogen/docs/internals.rst

Internal API Reference
**********************


mitogen.core
============


Latch Class
-----------

.. currentmodule:: mitogen.core

.. class:: Latch ()

    A latch is a :py:class:`Queue.Queue`-like object that supports mutation and
    waiting from multiple threads, however unlike :py:class:`Queue.Queue`,
    waiting threads always remain interruptible, so CTRL+C always succeeds, and
    waits where a timeout is set experience no wake up latency. These
    properties are not possible in combination using the built-in threading
    primitives available in Python 2.x.

    Latches implement queues using the UNIX self-pipe trick, and a per-thread
    :py:func:`socket.socketpair` that is lazily created the first time any
    latch attempts to sleep on a thread, and dynamically associated with the
    waiting Latch only for duration of the wait.

    See :ref:`waking-sleeping-threads` for further discussion.

    .. method:: empty ()

        Return :py:data:`True` if calling :py:meth:`get` would block.

        As with :py:class:`Queue.Queue`, :py:data:`True` may be returned even
        though a subsequent call to :py:meth:`get` will succeed, since a
        message may be posted at any moment between :py:meth:`empty` and
        :py:meth:`get`.

        As with :py:class:`Queue.Queue`, :py:data:`False` may be returned even
        though a subsequent call to :py:meth:`get` will block, since another
        waiting thread may be woken at any moment between :py:meth:`empty` and
        :py:meth:`get`.

    .. method:: get (timeout=None, block=True)

        Return the next object enqueued on this latch, or sleep waiting for
        one.

        :param float timeout:
            If not :py:data:`None`, specifies a timeout in seconds.

        :param bool block:
            If :py:data:`False`, immediately raise
            :py:class:`mitogen.core.TimeoutError` if the latch is empty.

        :raises mitogen.core.LatchError:
            :py:meth:`close` has been called, and the object is no longer valid.

        :raises mitogen.core.TimeoutError:
            Timeout was reached.

        :returns:
            The de-queued object.

    .. method:: put (obj)

        Enqueue an object on this latch, waking the first thread that is asleep
        waiting for a result, if one exists.

        :raises mitogen.core.LatchError:
            :py:meth:`close` has been called, and the object is no longer valid.

    .. method:: close ()

        Mark the latch as closed, and cause every sleeping thread to be woken,
        with :py:class:`mitogen.core.LatchError` raised in each thread.


Side Class
----------

.. currentmodule:: mitogen.core

.. class:: Side (stream, fd, keep_alive=True)

    Represent a single side of a :py:class:`BasicStream`. This exists to allow
    streams implemented using unidirectional (e.g. UNIX pipe) and bidirectional
    (e.g. UNIX socket) file descriptors to operate identically.

    :param mitogen.core.Stream stream:
        The stream this side is associated with.

    :param int fd:
        Underlying file descriptor.

    :param bool keep_alive:
        Value for :py:attr:`keep_alive`

    During construction, the file descriptor has its :py:data:`os.O_NONBLOCK`
    flag enabled using :py:func:`fcntl.fcntl`.

    .. attribute:: stream

        The :py:class:`Stream` for which this is a read or write side.

    .. attribute:: fd

        Integer file descriptor to perform IO on, or ``None`` if
        :py:meth:`close` has been called.

    .. attribute:: keep_alive

        If ``True``, causes presence of this side in :py:class:`Broker`'s
        active reader set to defer shutdown until the side is disconnected.

    .. method:: fileno

        Return :py:attr:`fd` if it is not ``None``, otherwise raise
        :py:class:`StreamError`. This method is implemented so that
        :py:class:`Side` can be used directly by :py:func:`select.select`.

    .. method:: close

        Call :py:func:`os.close` on :py:attr:`fd` if it is not ``None``, then
        set it to ``None``.

    .. method:: read (n=CHUNK_SIZE)

        Read up to `n` bytes from the file descriptor, wrapping the underlying
        :py:func:`os.read` call with :py:func:`io_op` to trap common
        disconnection conditions.

        :py:meth:`read` always behaves as if it is reading from a regular UNIX
        file; socket, pipe, and TTY disconnection errors are masked and result
        in a 0-sized read just like a regular file.

        :returns:
            Bytes read, or the empty to string to indicate disconnection was
            detected.

    .. method:: write (s)

        Write as much of the bytes from `s` as possible to the file descriptor,
        wrapping the underlying :py:func:`os.write` call with :py:func:`io_op`
        to trap common disconnection connditions.

        :py:meth:`read` always behaves as if it is writing to a regular UNIX
        file; socket, pipe, and TTY disconnection errors are masked and result
        in a 0-sized write.

        :returns:
            Number of bytes written, or ``None`` if disconnection was detected.


Stream Classes
--------------

.. currentmodule:: mitogen.core

.. class:: BasicStream

    .. attribute:: receive_side

        A :py:class:`Side` representing the stream's receive file descriptor.

    .. attribute:: transmit_side

        A :py:class:`Side` representing the stream's transmit file descriptor.

    .. method:: on_disconnect (broker)

        Called by :py:class:`Broker` to force disconnect the stream. The base
        implementation simply closes :py:attr:`receive_side` and
        :py:attr:`transmit_side` and unregisters the stream from the broker.

    .. method:: on_receive (broker)

        Called by :py:class:`Broker` when the stream's :py:attr:`receive_side` has
        been marked readable using :py:meth:`Broker.start_receive` and the
        broker has detected the associated file descriptor is ready for
        reading.

        Subclasses must implement this method if
        :py:meth:`Broker.start_receive` is ever called on them, and the method
        must call :py:meth:`on_disconect` if reading produces an empty string.

    .. method:: on_transmit (broker)

        Called by :py:class:`Broker` when the stream's :py:attr:`transmit_side`
        has been marked writeable using :py:meth:`Broker._start_transmit` and
        the broker has detected the associated file descriptor is ready for
        writing.

        Subclasses must implement this method if
        :py:meth:`Broker._start_transmit` is ever called on them.

    .. method:: on_shutdown (broker)

        Called by :py:meth:`Broker.shutdown` to allow the stream time to
        gracefully shutdown. The base implementation simply called
        :py:meth:`on_disconnect`.

.. autoclass:: Stream
   :members:

.. currentmodule:: mitogen.fork

.. autoclass:: Stream
   :members:

.. currentmodule:: mitogen.parent

.. autoclass:: Stream
   :members:

.. currentmodule:: mitogen.ssh

.. autoclass:: Stream
   :members:

.. currentmodule:: mitogen.sudo

.. autoclass:: Stream
   :members:


Other Stream Subclasses
-----------------------

.. currentmodule:: mitogen.core

.. autoclass:: IoLogger
   :members:

.. autoclass:: Waker
   :members:


Importer Class
--------------

.. currentmodule:: mitogen.core
.. autoclass:: Importer
   :members:


Responder Class
---------------

.. currentmodule:: mitogen.master
.. autoclass:: ModuleResponder
   :members:


Forwarder Class
---------------

.. currentmodule:: mitogen.parent
.. autoclass:: ModuleForwarder
   :members:


ExternalContext Class
---------------------

.. currentmodule:: mitogen.core

.. class:: ExternalContext

    External context implementation.

    .. attribute:: broker

        The :py:class:`mitogen.core.Broker` instance.

    .. attribute:: context

            The :py:class:`mitogen.core.Context` instance.

    .. attribute:: channel

            The :py:class:`mitogen.core.Channel` over which
            :py:data:`CALL_FUNCTION` requests are received.

    .. attribute:: stdout_log

        The :py:class:`mitogen.core.IoLogger` connected to ``stdout``.

    .. attribute:: importer

        The :py:class:`mitogen.core.Importer` instance.

    .. attribute:: stdout_log

        The :py:class:`IoLogger` connected to ``stdout``.

    .. attribute:: stderr_log

        The :py:class:`IoLogger` connected to ``stderr``.

    .. method:: _dispatch_calls

        Implementation for the main thread in every child context.

mitogen.master
==============

.. currentmodule:: mitogen.master

.. class:: ProcessMonitor

    Install a :py:data:`signal.SIGCHLD` handler that generates callbacks when a
    specific child process has exitted.

    .. method:: add (pid, callback)

        Add a callback function to be notified of the exit status of a process.

        :param int pid:
            Process ID to be notified of.

        :param callback:
            Function invoked as `callback(status)`, where `status` is the raw
            exit status of the child process.


Blocking I/O Functions
----------------------

These functions exist to support the blocking phase of setting up a new
context. They will eventually be replaced with asynchronous equivalents.


.. currentmodule:: mitogen.master

.. function:: iter_read(fd, deadline=None)

    Return a generator that arranges for up to 4096-byte chunks to be read at a
    time from the file descriptor `fd` until the generator is destroyed.

    :param fd:
        File descriptor to read from.

    :param deadline:
        If not ``None``, an absolute UNIX timestamp after which timeout should
        occur.

    :raises mitogen.core.TimeoutError:
        Attempt to read beyond deadline.

    :raises mitogen.core.StreamError:
        Attempt to read past end of file.


.. currentmodule:: mitogen.master

.. function:: write_all (fd, s, deadline=None)

    Arrange for all of bytestring `s` to be written to the file descriptor
    `fd`.

    :param int fd:
        File descriptor to write to.

    :param bytes s:
        Bytestring to write to file descriptor.

    :param float deadline:
        If not ``None``, an absolute UNIX timestamp after which timeout should
        occur.

    :raises mitogen.core.TimeoutError:
        Bytestring could not be written entirely before deadline was exceeded.

    :raises mitogen.core.StreamError:
        File descriptor was disconnected before write could complete.


Helper Functions
----------------

.. currentmodule:: mitogen.core

.. function:: io_op (func, \*args)

    Wrap a function that may raise :py:class:`OSError`, trapping common error
    codes relating to disconnection events in various subsystems:

    * When performing IO against a TTY, disconnection of the remote end is
      signalled by :py:data:`errno.EIO`.
    * When performing IO against a socket, disconnection of the remote end is
      signalled by :py:data:`errno.ECONNRESET`.
    * When performing IO against a pipe, disconnection of the remote end is
      signalled by :py:data:`errno.EPIPE`.

    :returns:
        Tuple of `(return_value, disconnected)`, where `return_value` is the
        return value of `func(\*args)`, and `disconnected` is ``True`` if
        disconnection was detected, otherwise ``False``.


.. currentmodule:: mitogen.master

.. function:: create_child (\*args)

    Create a child process whose stdin/stdout is connected to a socket,
    returning `(pid, socket_obj)`.


.. currentmodule:: mitogen.master

.. function:: tty_create_child (\*args)

    Return a file descriptor connected to the master end of a pseudo-terminal,
    whose slave end is connected to stdin/stdout/stderr of a new child process.
    The child is created such that the pseudo-terminal becomes its controlling
    TTY, ensuring access to /dev/tty returns a new file descriptor open on the
    slave end.

    :param list args:
        :py:func:`os.execl` argument list.

    :returns:
        `(pid, fd)`


.. currentmodule:: mitogen.master

.. function:: get_child_modules (path)

    Return the suffixes of submodules directly neated beneath of the package
    directory at `path`.

    :param str path:
        Path to the module's source code on disk, or some PEP-302-recognized
        equivalent. Usually this is the module's ``__file__`` attribute, but
        is specified explicitly to avoid loading the module.

    :return:
        List of submodule name suffixes.


.. currentmodule:: mitogen.parent

.. autofunction:: minimize_source (source)

    Remove comments and docstrings from Python `source`, preserving line
    numbers and syntax of empty blocks.

    :param str source:
        The source to minimize.

    :returns str:
        The minimized source.