You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
465 lines
13 KiB
ReStructuredText
465 lines
13 KiB
ReStructuredText
|
|
Internal API Reference
|
|
**********************
|
|
|
|
|
|
mitogen.core
|
|
============
|
|
|
|
|
|
Latch Class
|
|
-----------
|
|
|
|
.. currentmodule:: mitogen.core
|
|
|
|
.. class:: Latch ()
|
|
|
|
A latch is a :py:class:`Queue.Queue`-like object that supports mutation and
|
|
waiting from multiple threads, however unlike :py:class:`Queue.Queue`,
|
|
waiting threads always remain interruptible, so CTRL+C always succeeds, and
|
|
waits where a timeout is set experience no wake up latency. These
|
|
properties are not possible in combination using the built-in threading
|
|
primitives available in Python 2.x.
|
|
|
|
Latches implement queues using the UNIX self-pipe trick, and a per-thread
|
|
:py:func:`socket.socketpair` that is lazily created the first time any
|
|
latch attempts to sleep on a thread, and dynamically associated with the
|
|
waiting Latch only for duration of the wait.
|
|
|
|
See :ref:`waking-sleeping-threads` for further discussion.
|
|
|
|
.. method:: empty ()
|
|
|
|
Return :py:data:`True` if calling :py:meth:`get` would block.
|
|
|
|
As with :py:class:`Queue.Queue`, :py:data:`True` may be returned even
|
|
though a subsequent call to :py:meth:`get` will succeed, since a
|
|
message may be posted at any moment between :py:meth:`empty` and
|
|
:py:meth:`get`.
|
|
|
|
As with :py:class:`Queue.Queue`, :py:data:`False` may be returned even
|
|
though a subsequent call to :py:meth:`get` will block, since another
|
|
waiting thread may be woken at any moment between :py:meth:`empty` and
|
|
:py:meth:`get`.
|
|
|
|
.. method:: get (timeout=None, block=True)
|
|
|
|
Return the next object enqueued on this latch, or sleep waiting for
|
|
one.
|
|
|
|
:param float timeout:
|
|
If not :py:data:`None`, specifies a timeout in seconds.
|
|
|
|
:param bool block:
|
|
If :py:data:`False`, immediately raise
|
|
:py:class:`mitogen.core.TimeoutError` if the latch is empty.
|
|
|
|
:raises mitogen.core.LatchError:
|
|
:py:meth:`close` has been called, and the object is no longer valid.
|
|
|
|
:raises mitogen.core.TimeoutError:
|
|
Timeout was reached.
|
|
|
|
:returns:
|
|
The de-queued object.
|
|
|
|
.. method:: put (obj)
|
|
|
|
Enqueue an object on this latch, waking the first thread that is asleep
|
|
waiting for a result, if one exists.
|
|
|
|
:raises mitogen.core.LatchError:
|
|
:py:meth:`close` has been called, and the object is no longer valid.
|
|
|
|
.. method:: close ()
|
|
|
|
Mark the latch as closed, and cause every sleeping thread to be woken,
|
|
with :py:class:`mitogen.core.LatchError` raised in each thread.
|
|
|
|
|
|
Side Class
|
|
----------
|
|
|
|
.. currentmodule:: mitogen.core
|
|
|
|
.. class:: Side (stream, fd, keep_alive=True)
|
|
|
|
Represent a single side of a :py:class:`BasicStream`. This exists to allow
|
|
streams implemented using unidirectional (e.g. UNIX pipe) and bidirectional
|
|
(e.g. UNIX socket) file descriptors to operate identically.
|
|
|
|
:param mitogen.core.Stream stream:
|
|
The stream this side is associated with.
|
|
|
|
:param int fd:
|
|
Underlying file descriptor.
|
|
|
|
:param bool keep_alive:
|
|
Value for :py:attr:`keep_alive`
|
|
|
|
During construction, the file descriptor has its :py:data:`os.O_NONBLOCK`
|
|
flag enabled using :py:func:`fcntl.fcntl`.
|
|
|
|
.. attribute:: stream
|
|
|
|
The :py:class:`Stream` for which this is a read or write side.
|
|
|
|
.. attribute:: fd
|
|
|
|
Integer file descriptor to perform IO on, or ``None`` if
|
|
:py:meth:`close` has been called.
|
|
|
|
.. attribute:: keep_alive
|
|
|
|
If ``True``, causes presence of this side in :py:class:`Broker`'s
|
|
active reader set to defer shutdown until the side is disconnected.
|
|
|
|
.. method:: fileno
|
|
|
|
Return :py:attr:`fd` if it is not ``None``, otherwise raise
|
|
:py:class:`StreamError`. This method is implemented so that
|
|
:py:class:`Side` can be used directly by :py:func:`select.select`.
|
|
|
|
.. method:: close
|
|
|
|
Call :py:func:`os.close` on :py:attr:`fd` if it is not ``None``, then
|
|
set it to ``None``.
|
|
|
|
.. method:: read (n=CHUNK_SIZE)
|
|
|
|
Read up to `n` bytes from the file descriptor, wrapping the underlying
|
|
:py:func:`os.read` call with :py:func:`io_op` to trap common
|
|
disconnection conditions.
|
|
|
|
:py:meth:`read` always behaves as if it is reading from a regular UNIX
|
|
file; socket, pipe, and TTY disconnection errors are masked and result
|
|
in a 0-sized read just like a regular file.
|
|
|
|
:returns:
|
|
Bytes read, or the empty to string to indicate disconnection was
|
|
detected.
|
|
|
|
.. method:: write (s)
|
|
|
|
Write as much of the bytes from `s` as possible to the file descriptor,
|
|
wrapping the underlying :py:func:`os.write` call with :py:func:`io_op`
|
|
to trap common disconnection connditions.
|
|
|
|
:py:meth:`read` always behaves as if it is writing to a regular UNIX
|
|
file; socket, pipe, and TTY disconnection errors are masked and result
|
|
in a 0-sized write.
|
|
|
|
:returns:
|
|
Number of bytes written, or ``None`` if disconnection was detected.
|
|
|
|
|
|
Stream Classes
|
|
--------------
|
|
|
|
.. currentmodule:: mitogen.core
|
|
|
|
.. class:: BasicStream
|
|
|
|
.. attribute:: receive_side
|
|
|
|
A :py:class:`Side` representing the stream's receive file descriptor.
|
|
|
|
.. attribute:: transmit_side
|
|
|
|
A :py:class:`Side` representing the stream's transmit file descriptor.
|
|
|
|
.. method:: on_disconnect (broker)
|
|
|
|
Called by :py:class:`Broker` to force disconnect the stream. The base
|
|
implementation simply closes :py:attr:`receive_side` and
|
|
:py:attr:`transmit_side` and unregisters the stream from the broker.
|
|
|
|
.. method:: on_receive (broker)
|
|
|
|
Called by :py:class:`Broker` when the stream's :py:attr:`receive_side` has
|
|
been marked readable using :py:meth:`Broker.start_receive` and the
|
|
broker has detected the associated file descriptor is ready for
|
|
reading.
|
|
|
|
Subclasses must implement this method if
|
|
:py:meth:`Broker.start_receive` is ever called on them, and the method
|
|
must call :py:meth:`on_disconect` if reading produces an empty string.
|
|
|
|
.. method:: on_transmit (broker)
|
|
|
|
Called by :py:class:`Broker` when the stream's :py:attr:`transmit_side`
|
|
has been marked writeable using :py:meth:`Broker._start_transmit` and
|
|
the broker has detected the associated file descriptor is ready for
|
|
writing.
|
|
|
|
Subclasses must implement this method if
|
|
:py:meth:`Broker._start_transmit` is ever called on them.
|
|
|
|
.. method:: on_shutdown (broker)
|
|
|
|
Called by :py:meth:`Broker.shutdown` to allow the stream time to
|
|
gracefully shutdown. The base implementation simply called
|
|
:py:meth:`on_disconnect`.
|
|
|
|
.. autoclass:: Stream
|
|
:members:
|
|
|
|
.. method:: pending_bytes ()
|
|
|
|
Returns the number of bytes queued for transmission on this stream.
|
|
This can be used to limit the amount of data buffered in RAM by an
|
|
otherwise unlimited consumer.
|
|
|
|
For an accurate result, this method should be called from the Broker
|
|
thread, using a wrapper like:
|
|
|
|
::
|
|
|
|
def get_pending_bytes(self, stream):
|
|
latch = mitogen.core.Latch()
|
|
self.broker.defer(
|
|
lambda: latch.put(stream.pending_bytes())
|
|
)
|
|
return latch.get()
|
|
|
|
|
|
.. currentmodule:: mitogen.fork
|
|
|
|
.. autoclass:: Stream
|
|
:members:
|
|
|
|
.. currentmodule:: mitogen.parent
|
|
|
|
.. autoclass:: Stream
|
|
:members:
|
|
|
|
.. currentmodule:: mitogen.ssh
|
|
|
|
.. autoclass:: Stream
|
|
:members:
|
|
|
|
.. currentmodule:: mitogen.sudo
|
|
|
|
.. autoclass:: Stream
|
|
:members:
|
|
|
|
|
|
Other Stream Subclasses
|
|
-----------------------
|
|
|
|
.. currentmodule:: mitogen.core
|
|
|
|
.. autoclass:: IoLogger
|
|
:members:
|
|
|
|
.. autoclass:: Waker
|
|
:members:
|
|
|
|
|
|
Importer Class
|
|
--------------
|
|
|
|
.. currentmodule:: mitogen.core
|
|
.. autoclass:: Importer
|
|
:members:
|
|
|
|
|
|
Responder Class
|
|
---------------
|
|
|
|
.. currentmodule:: mitogen.master
|
|
.. autoclass:: ModuleResponder
|
|
:members:
|
|
|
|
|
|
Forwarder Class
|
|
---------------
|
|
|
|
.. currentmodule:: mitogen.parent
|
|
.. autoclass:: ModuleForwarder
|
|
:members:
|
|
|
|
|
|
ExternalContext Class
|
|
---------------------
|
|
|
|
.. currentmodule:: mitogen.core
|
|
|
|
.. class:: ExternalContext
|
|
|
|
External context implementation.
|
|
|
|
.. attribute:: broker
|
|
|
|
The :py:class:`mitogen.core.Broker` instance.
|
|
|
|
.. attribute:: context
|
|
|
|
The :py:class:`mitogen.core.Context` instance.
|
|
|
|
.. attribute:: channel
|
|
|
|
The :py:class:`mitogen.core.Channel` over which
|
|
:py:data:`CALL_FUNCTION` requests are received.
|
|
|
|
.. attribute:: stdout_log
|
|
|
|
The :py:class:`mitogen.core.IoLogger` connected to ``stdout``.
|
|
|
|
.. attribute:: importer
|
|
|
|
The :py:class:`mitogen.core.Importer` instance.
|
|
|
|
.. attribute:: stdout_log
|
|
|
|
The :py:class:`IoLogger` connected to ``stdout``.
|
|
|
|
.. attribute:: stderr_log
|
|
|
|
The :py:class:`IoLogger` connected to ``stderr``.
|
|
|
|
.. method:: _dispatch_calls
|
|
|
|
Implementation for the main thread in every child context.
|
|
|
|
mitogen.master
|
|
==============
|
|
|
|
.. currentmodule:: mitogen.master
|
|
|
|
.. class:: ProcessMonitor
|
|
|
|
Install a :py:data:`signal.SIGCHLD` handler that generates callbacks when a
|
|
specific child process has exitted.
|
|
|
|
.. method:: add (pid, callback)
|
|
|
|
Add a callback function to be notified of the exit status of a process.
|
|
|
|
:param int pid:
|
|
Process ID to be notified of.
|
|
|
|
:param callback:
|
|
Function invoked as `callback(status)`, where `status` is the raw
|
|
exit status of the child process.
|
|
|
|
|
|
Blocking I/O Functions
|
|
----------------------
|
|
|
|
These functions exist to support the blocking phase of setting up a new
|
|
context. They will eventually be replaced with asynchronous equivalents.
|
|
|
|
|
|
.. currentmodule:: mitogen.master
|
|
|
|
.. function:: iter_read(fd, deadline=None)
|
|
|
|
Return a generator that arranges for up to 4096-byte chunks to be read at a
|
|
time from the file descriptor `fd` until the generator is destroyed.
|
|
|
|
:param fd:
|
|
File descriptor to read from.
|
|
|
|
:param deadline:
|
|
If not ``None``, an absolute UNIX timestamp after which timeout should
|
|
occur.
|
|
|
|
:raises mitogen.core.TimeoutError:
|
|
Attempt to read beyond deadline.
|
|
|
|
:raises mitogen.core.StreamError:
|
|
Attempt to read past end of file.
|
|
|
|
|
|
.. currentmodule:: mitogen.master
|
|
|
|
.. function:: write_all (fd, s, deadline=None)
|
|
|
|
Arrange for all of bytestring `s` to be written to the file descriptor
|
|
`fd`.
|
|
|
|
:param int fd:
|
|
File descriptor to write to.
|
|
|
|
:param bytes s:
|
|
Bytestring to write to file descriptor.
|
|
|
|
:param float deadline:
|
|
If not ``None``, an absolute UNIX timestamp after which timeout should
|
|
occur.
|
|
|
|
:raises mitogen.core.TimeoutError:
|
|
Bytestring could not be written entirely before deadline was exceeded.
|
|
|
|
:raises mitogen.core.StreamError:
|
|
File descriptor was disconnected before write could complete.
|
|
|
|
|
|
Helper Functions
|
|
----------------
|
|
|
|
.. currentmodule:: mitogen.core
|
|
|
|
.. function:: io_op (func, \*args)
|
|
|
|
Wrap a function that may raise :py:class:`OSError`, trapping common error
|
|
codes relating to disconnection events in various subsystems:
|
|
|
|
* When performing IO against a TTY, disconnection of the remote end is
|
|
signalled by :py:data:`errno.EIO`.
|
|
* When performing IO against a socket, disconnection of the remote end is
|
|
signalled by :py:data:`errno.ECONNRESET`.
|
|
* When performing IO against a pipe, disconnection of the remote end is
|
|
signalled by :py:data:`errno.EPIPE`.
|
|
|
|
:returns:
|
|
Tuple of `(return_value, disconnected)`, where `return_value` is the
|
|
return value of `func(\*args)`, and `disconnected` is ``True`` if
|
|
disconnection was detected, otherwise ``False``.
|
|
|
|
|
|
.. currentmodule:: mitogen.parent
|
|
|
|
.. autofunction:: create_child
|
|
|
|
|
|
.. currentmodule:: mitogen.parent
|
|
|
|
.. autofunction:: tty_create_child
|
|
|
|
|
|
.. currentmodule:: mitogen.parent
|
|
|
|
.. autofunction:: hybrid_tty_create_child
|
|
|
|
|
|
.. currentmodule:: mitogen.master
|
|
|
|
.. function:: get_child_modules (path)
|
|
|
|
Return the suffixes of submodules directly neated beneath of the package
|
|
directory at `path`.
|
|
|
|
:param str path:
|
|
Path to the module's source code on disk, or some PEP-302-recognized
|
|
equivalent. Usually this is the module's ``__file__`` attribute, but
|
|
is specified explicitly to avoid loading the module.
|
|
|
|
:return:
|
|
List of submodule name suffixes.
|
|
|
|
|
|
.. currentmodule:: mitogen.parent
|
|
|
|
.. autofunction:: minimize_source (source)
|
|
|
|
Remove comments and docstrings from Python `source`, preserving line
|
|
numbers and syntax of empty blocks.
|
|
|
|
:param str source:
|
|
The source to minimize.
|
|
|
|
:returns str:
|
|
The minimized source.
|