diff --git a/docs/api.rst b/docs/api.rst index 43ae8ecb..20389180 100644 --- a/docs/api.rst +++ b/docs/api.rst @@ -28,11 +28,7 @@ mitogen Package mitogen.core ------------ -.. module:: mitogen.core - -This module implements most package functionality, but remains separate from -non-essential code in order to reduce its size, since it is also serves as the -bootstrap implementation sent to every new slave context. +.. automodule:: mitogen.core .. currentmodule:: mitogen.core .. decorator:: takes_econtext @@ -63,102 +59,25 @@ bootstrap implementation sent to every new slave context. mitogen.master -------------- -.. module:: mitogen.master +.. automodule:: mitogen.master + -This module implements functionality required by master processes, such as -starting new contexts via SSH. Its size is also restricted, since it must -be sent to any context that will be used to establish additional child -contexts. +mitogen.parent +-------------- + +.. automodule:: mitogen.parent mitogen.fakessh --------------- -.. module:: mitogen.fakessh - -fakessh is a stream implementation that starts a local subprocess with its -environment modified such that ``PATH`` searches for `ssh` return an mitogen -implementation of the SSH command. When invoked, this tool arranges for the -command line supplied by the calling program to be executed in a context -already established by the master process, reusing the master's (possibly -proxied) connection to that context. - -This allows tools like `rsync` and `scp` to transparently reuse the connections -and tunnels already established by the host program to connect to a target -machine, without wasteful redundant SSH connection setup, 3-way handshakes, or -firewall hopping configurations, and enables these tools to be used in -impossible scenarios, such as over `sudo` with ``requiretty`` enabled. - -The fake `ssh` command source is written to a temporary file on disk, and -consists of a copy of the :py:mod:`mitogen.core` source code (just like any -other child context), with a line appended to cause it to connect back to the -host process over an FD it inherits. As there is no reliance on an existing -filesystem file, it is possible for child contexts to use fakessh. - -As a consequence of connecting back through an inherited FD, only one SSH -invocation is possible, which is fine for tools like `rsync`, however in future -this restriction will be lifted. - -Sequence: - - 1. ``fakessh`` Context and Stream created by parent context. The stream's - buffer has a :py:func:`_fakessh_main` :py:data:`CALL_FUNCTION - ` enqueued. - 2. Target program (`rsync/scp/sftp`) invoked, which internally executes - `ssh` from ``PATH``. - 3. :py:mod:`mitogen.core` bootstrap begins, recovers the stream FD - inherited via the target program, established itself as the fakessh - context. - 4. :py:func:`_fakessh_main` :py:data:`CALL_FUNCTION - ` is read by fakessh context, - - a. sets up :py:class:`IoPump` for stdio, registers - stdin_handle for local context. - b. Enqueues :py:data:`CALL_FUNCTION ` for - :py:func:`_start_slave` invoked in target context, - - i. the program from the `ssh` command line is started - ii. sets up :py:class:`IoPump` for `ssh` command line process's - stdio pipes - iii. returns `(control_handle, stdin_handle)` to - :py:func:`_fakessh_main` - - 5. :py:func:`_fakessh_main` receives control/stdin handles from from - :py:func:`_start_slave`, - - a. registers remote's stdin_handle with local :py:class:`IoPump`. - b. sends `("start", local_stdin_handle)` to remote's control_handle - c. registers local :py:class:`IoPump` with - :py:class:`mitogen.core.Broker`. - d. loops waiting for `local stdout closed && remote stdout closed` - - 6. :py:func:`_start_slave` control channel receives `("start", stdin_handle)`, - - a. registers remote's stdin_handle with local :py:class:`IoPump` - b. registers local :py:class:`IoPump` with - :py:class:`mitogen.core.Broker`. - c. loops waiting for `local stdout closed && remote stdout closed` - +.. image:: images/fakessh.png + :align: right +.. automodule:: mitogen.fakessh .. currentmodule:: mitogen.fakessh -.. function:: run (dest, router, args, daedline=None, econtext=None) - - Run the command specified by the argument vector `args` such that ``PATH`` - searches for SSH by the command will cause its attempt to use SSH to - execute a remote program to be redirected to use mitogen to execute that - program using the context `dest` instead. - - :param mitogen.core.Context dest: - The destination context to execute the SSH command line in. - - :param mitogen.core.Router router: +.. autofunction:: run (dest, router, args, daedline=None, econtext=None) - :param list[str] args: - Command line arguments for local program, e.g. - ``['rsync', '/tmp', 'remote:/tmp']`` - - :returns: - Exit status of the child process. Message Class @@ -168,6 +87,11 @@ Message Class .. class:: Message + Messages are the fundamental unit of communication, comprising the fields + from in the :ref:`stream-protocol` header, an optional reference to the + receiving :class:`mitogen.core.Router` for ingress messages, and helper + methods for deserialization and generating replies. + .. attribute:: router The :py:class:`mitogen.core.Router` responsible for routing the @@ -181,16 +105,36 @@ Message Class .. attribute:: dst_id + Integer target context ID. :py:class:`mitogen.core.Router` delivers + messages locally when their :attr:`dst_id` matches + :data:`mitogen.context_id`, otherwise they are routed up or downstream. + .. attribute:: src_id + Integer source context ID. Used as the target of replies if any are + generated. + .. attribute:: auth_id + The context ID under whose authority the message is acting. See + :py:ref:`source-verification`. + .. attribute:: handle + Integer target handle in the destination context. This is one of the + :py:ref:`standard-handles`, or a dynamically generated handle used to + receive a one-time reply, such as the return value of a function call. + .. attribute:: reply_to + Integer target handle to direct any reply to this message. Used to + receive a one-time reply, such as the return value of a function call. + :data:`IS_DEAD` has a special meaning when it appears in this field. + .. attribute:: data + Message data, which may be raw or pickled. + .. attribute:: is_dead :data:`True` if :attr:`reply_to` is set to the magic value diff --git a/docs/howitworks.rst b/docs/howitworks.rst index dde63c14..0b0ae42d 100644 --- a/docs/howitworks.rst +++ b/docs/howitworks.rst @@ -258,7 +258,7 @@ Stream Protocol .. currentmodule:: mitogen.core Once connected, a basic framing protocol is used to communicate between -parent and child: +parent and child. Integers use big endian in their encoded form. .. list-table:: :header-rows: 1 @@ -342,23 +342,6 @@ Masters listen on the following handles: million parent contexts to be created and destroyed before the associated Router must be recreated. -.. _IS_DEAD: -.. currentmodule:: mitogen.core -.. data:: IS_DEAD - - Special value used to signal disconnection or the inability to route a - message, when it appears in the `reply_to` field. Usually causes - :class:`mitogen.core.ChannelError` to be raised when it is received. - - It indicates the sender did not know how to process the message, or wishes - no further messages to be delivered to it. It is used when: - - * a remote receiver is disconnected or explicitly closed. - * a related message could not be delivered due to no route existing for it. - * a router is being torn down, as a sentinel value to notify - :py:meth:`mitogen.core.Router.add_handler` callbacks to clean up. - - Children listen on the following handles: .. _LOAD_MODULE: @@ -478,6 +461,24 @@ Non-master parents also listen on the following handles: ensuring they are cached and deduplicated at each hop in the chain leading to the target context. +Special values for the `reply_to` field: + +.. _IS_DEAD: +.. currentmodule:: mitogen.core +.. data:: IS_DEAD + + Special value used to signal disconnection or the inability to route a + message, when it appears in the `reply_to` field. Usually causes + :class:`mitogen.core.ChannelError` to be raised when it is received. + + It indicates the sender did not know how to process the message, or wishes + no further messages to be delivered to it. It is used when: + + * a remote receiver is disconnected or explicitly closed. + * a related message could not be delivered due to no route existing for it. + * a router is being torn down, as a sentinel value to notify + :py:meth:`mitogen.core.Router.add_handler` callbacks to clean up. + Additional handles are created to receive the result of every function call triggered by :py:meth:`call_async() `. diff --git a/mitogen/__init__.py b/mitogen/__init__.py index 02be6897..96164d9d 100644 --- a/mitogen/__init__.py +++ b/mitogen/__init__.py @@ -36,33 +36,19 @@ be expected. On the slave, it is built dynamically during startup. __version__ = (0, 0, 2) -#: This is ``False`` in slave contexts. It is used in single-file Python -#: programs to avoid reexecuting the program's :py:func:`main` function in the -#: slave. For example: -#: -#: .. code-block:: python -#: -#: def do_work(): -#: os.system('hostname') -#: -#: def main(broker): -#: context = mitogen.master.connect(broker) -#: context.call(do_work) # Causes slave to import __main__. -#: -#: if __name__ == '__main__' and mitogen.is_master: -#: import mitogen.utils -#: mitogen.utils.run_with_broker(main) -#: +#: This is :data:`False` in slave contexts. Previously it was used to prevent +#: re-execution of :mod:`__main__` in single file programs, however that now +#: happens automatically. is_master = True -#: This is ``0`` in a master, otherwise it is a master-generated ID unique to +#: This is `0` in a master, otherwise it is the master-assigned ID unique to #: the slave context used for message routing. context_id = 0 -#: This is ``None`` in a master, otherwise it is the master-generated ID unique -#: to the slave's parent context. +#: This is :data:`None` in a master, otherwise it is the master-assigned ID +#: unique to the slave's parent context. parent_id = None @@ -76,8 +62,8 @@ def main(log_level='INFO', profiling=False): Convenience decorator primarily useful for writing discardable test scripts. - In the master process, when `func` is defined in the ``__main__`` module, - arranges for `func(router)` to be invoked immediately, with + In the master process, when `func` is defined in the :mod:`__main__` + module, arranges for `func(router)` to be invoked immediately, with :py:class:`mitogen.master.Router` construction and destruction handled just as in :py:func:`mitogen.utils.run_with_router`. In slaves, this function does nothing. diff --git a/mitogen/core.py b/mitogen/core.py index 2a5049f2..4fe2292f 100644 --- a/mitogen/core.py +++ b/mitogen/core.py @@ -26,6 +26,12 @@ # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE # POSSIBILITY OF SUCH DAMAGE. +""" +This module implements most package functionality, but remains separate from +non-essential code in order to reduce its size, since it is also serves as the +bootstrap implementation sent to every new slave context. +""" + import collections import errno import fcntl diff --git a/mitogen/fakessh.py b/mitogen/fakessh.py index 27cb4acd..d3950b64 100644 --- a/mitogen/fakessh.py +++ b/mitogen/fakessh.py @@ -26,6 +26,70 @@ # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE # POSSIBILITY OF SUCH DAMAGE. +""" +:mod:`mitogen.fakessh` is a stream implementation that starts a subprocess with +its environment modified such that ``PATH`` searches for `ssh` return a Mitogen +implementation of SSH. When invoked, this implementation arranges for the +command line supplied by the caller to be executed in a remote context, reusing +the parent context's (possibly proxied) connection to that remote context. + +This allows tools like `rsync` and `scp` to transparently reuse the connections +and tunnels already established by the host program to connect to a target +machine, without wasteful redundant SSH connection setup, 3-way handshakes, or +firewall hopping configurations, and enables these tools to be used in +impossible scenarios, such as over `sudo` with ``requiretty`` enabled. + +The fake `ssh` command source is written to a temporary file on disk, and +consists of a copy of the :py:mod:`mitogen.core` source code (just like any +other child context), with a line appended to cause it to connect back to the +host process over an FD it inherits. As there is no reliance on an existing +filesystem file, it is possible for child contexts to use fakessh. + +As a consequence of connecting back through an inherited FD, only one SSH +invocation is possible, which is fine for tools like `rsync`, however in future +this restriction will be lifted. + +Sequence: + + 1. ``fakessh`` Context and Stream created by parent context. The stream's + buffer has a :py:func:`_fakessh_main` :py:data:`CALL_FUNCTION + ` enqueued. + 2. Target program (`rsync/scp/sftp`) invoked, which internally executes + `ssh` from ``PATH``. + 3. :py:mod:`mitogen.core` bootstrap begins, recovers the stream FD + inherited via the target program, established itself as the fakessh + context. + 4. :py:func:`_fakessh_main` :py:data:`CALL_FUNCTION + ` is read by fakessh context, + + a. sets up :py:class:`IoPump` for stdio, registers + stdin_handle for local context. + b. Enqueues :py:data:`CALL_FUNCTION ` for + :py:func:`_start_slave` invoked in target context, + + i. the program from the `ssh` command line is started + ii. sets up :py:class:`IoPump` for `ssh` command line process's + stdio pipes + iii. returns `(control_handle, stdin_handle)` to + :py:func:`_fakessh_main` + + 5. :py:func:`_fakessh_main` receives control/stdin handles from from + :py:func:`_start_slave`, + + a. registers remote's stdin_handle with local :py:class:`IoPump`. + b. sends `("start", local_stdin_handle)` to remote's control_handle + c. registers local :py:class:`IoPump` with + :py:class:`mitogen.core.Broker`. + d. loops waiting for `local stdout closed && remote stdout closed` + + 6. :py:func:`_start_slave` control channel receives `("start", stdin_handle)`, + + a. registers remote's stdin_handle with local :py:class:`IoPump` + b. registers local :py:class:`IoPump` with + :py:class:`mitogen.core.Broker`. + c. loops waiting for `local stdout closed && remote stdout closed` +""" + import getopt import inspect import os @@ -330,6 +394,26 @@ def _get_econtext_config(context, sock2): @mitogen.core.takes_econtext @mitogen.core.takes_router def run(dest, router, args, deadline=None, econtext=None): + """ + Run the command specified by `args` such that ``PATH`` searches for SSH by + the command will cause its attempt to use SSH to execute a remote program + to be redirected to use mitogen to execute that program using the context + `dest` instead. + + :param list args: + Argument vector. + :param mitogen.core.Context dest: + The destination context to execute the SSH command line in. + + :param mitogen.core.Router router: + + :param list[str] args: + Command line arguments for local program, e.g. + ``['rsync', '/tmp', 'remote:/tmp']`` + + :returns: + Exit status of the child process. + """ if econtext is not None: mitogen.parent.upgrade_router(econtext) diff --git a/mitogen/master.py b/mitogen/master.py index e604fc66..9a5b7e83 100644 --- a/mitogen/master.py +++ b/mitogen/master.py @@ -26,6 +26,13 @@ # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE # POSSIBILITY OF SUCH DAMAGE. +""" +This module implements functionality required by master processes, such as +starting new contexts via SSH. Its size is also restricted, since it must +be sent to any context that will be used to establish additional child +contexts. +""" + import dis import imp import inspect diff --git a/mitogen/parent.py b/mitogen/parent.py index 76a91b24..41505934 100644 --- a/mitogen/parent.py +++ b/mitogen/parent.py @@ -26,6 +26,12 @@ # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE # POSSIBILITY OF SUCH DAMAGE. +""" +This module defines functionality common to master and parent processes. It is +sent to any child context that is due to become a parent, due to recursive +connection. +""" + import errno import fcntl import getpass