archive
/
mitogen
mirror of https://github.com/mitogen-hq/mitogen.git
1
0
Fork 0
Code Releases Activity

docs: tidy ups

Browse Source
pull/255/head
David Wilson 6 years ago
parent 61365236ad
commit d2714752ee
7 changed files with 166 additions and 132 deletions
Show Stats Download Patch File Download Diff File

128
docs/api.rst
Unescape Escape View File

@ -28,11 +28,7 @@ mitogen Package
mitogen.core
------------
.. module:: mitogen.core
This module implements most package functionality, but remains separate from
non-essential code in order to reduce its size, since it is also serves as the
bootstrap implementation sent to every new slave context.
.. automodule:: mitogen.core
.. currentmodule:: mitogen.core
.. decorator:: takes_econtext
@ -63,102 +59,25 @@ bootstrap implementation sent to every new slave context.
mitogen.master
--------------
.. module:: mitogen.master
.. automodule:: mitogen.master
This module implements functionality required by master processes, such as
starting new contexts via SSH. Its size is also restricted, since it must
be sent to any context that will be used to establish additional child
contexts.
mitogen.parent
--------------
.. automodule:: mitogen.parent
mitogen.fakessh
---------------
.. module:: mitogen.fakessh
fakessh is a stream implementation that starts a local subprocess with its
environment modified such that ``PATH`` searches for `ssh` return an mitogen
implementation of the SSH command. When invoked, this tool arranges for the
command line supplied by the calling program to be executed in a context
already established by the master process, reusing the master's (possibly
proxied) connection to that context.
This allows tools like `rsync` and `scp` to transparently reuse the connections
and tunnels already established by the host program to connect to a target
machine, without wasteful redundant SSH connection setup, 3-way handshakes, or
firewall hopping configurations, and enables these tools to be used in
impossible scenarios, such as over `sudo` with ``requiretty`` enabled.
The fake `ssh` command source is written to a temporary file on disk, and
consists of a copy of the :py:mod:`mitogen.core` source code (just like any
other child context), with a line appended to cause it to connect back to the
host process over an FD it inherits. As there is no reliance on an existing
filesystem file, it is possible for child contexts to use fakessh.
As a consequence of connecting back through an inherited FD, only one SSH
invocation is possible, which is fine for tools like `rsync`, however in future
this restriction will be lifted.
Sequence:
1. ``fakessh`` Context and Stream created by parent context. The stream's
buffer has a :py:func:`_fakessh_main` :py:data:`CALL_FUNCTION
<mitogen.core.CALL_FUNCTION>` enqueued.
2. Target program (`rsync/scp/sftp`) invoked, which internally executes
`ssh` from ``PATH``.
3. :py:mod:`mitogen.core` bootstrap begins, recovers the stream FD
inherited via the target program, established itself as the fakessh
context.
4. :py:func:`_fakessh_main` :py:data:`CALL_FUNCTION
<mitogen.core.CALL_FUNCTION>` is read by fakessh context,
a. sets up :py:class:`IoPump` for stdio, registers
stdin_handle for local context.
b. Enqueues :py:data:`CALL_FUNCTION <mitogen.core.CALL_FUNCTION>` for
:py:func:`_start_slave` invoked in target context,
i. the program from the `ssh` command line is started
ii. sets up :py:class:`IoPump` for `ssh` command line process's
stdio pipes
iii. returns `(control_handle, stdin_handle)` to
:py:func:`_fakessh_main`
5. :py:func:`_fakessh_main` receives control/stdin handles from from
:py:func:`_start_slave`,
a. registers remote's stdin_handle with local :py:class:`IoPump`.
b. sends `("start", local_stdin_handle)` to remote's control_handle
c. registers local :py:class:`IoPump` with
:py:class:`mitogen.core.Broker`.
d. loops waiting for `local stdout closed && remote stdout closed`
6. :py:func:`_start_slave` control channel receives `("start", stdin_handle)`,
a. registers remote's stdin_handle with local :py:class:`IoPump`
b. registers local :py:class:`IoPump` with
:py:class:`mitogen.core.Broker`.
c. loops waiting for `local stdout closed && remote stdout closed`
.. image:: images/fakessh.png
:align: right
.. automodule:: mitogen.fakessh
.. currentmodule:: mitogen.fakessh
.. function:: run (dest, router, args, daedline=None, econtext=None)
Run the command specified by the argument vector `args` such that ``PATH``
searches for SSH by the command will cause its attempt to use SSH to
execute a remote program to be redirected to use mitogen to execute that
program using the context `dest` instead.
:param mitogen.core.Context dest:
The destination context to execute the SSH command line in.
:param mitogen.core.Router router:
.. autofunction:: run (dest, router, args, daedline=None, econtext=None)
:param list[str] args:
Command line arguments for local program, e.g.
``['rsync', '/tmp', 'remote:/tmp']``
:returns:
Exit status of the child process.
Message Class
@ -168,6 +87,11 @@ Message Class
.. class:: Message
Messages are the fundamental unit of communication, comprising the fields
from in the :ref:`stream-protocol` header, an optional reference to the
receiving :class:`mitogen.core.Router` for ingress messages, and helper
methods for deserialization and generating replies.
.. attribute:: router
The :py:class:`mitogen.core.Router` responsible for routing the
@ -181,16 +105,36 @@ Message Class
.. attribute:: dst_id
Integer target context ID. :py:class:`mitogen.core.Router` delivers
messages locally when their :attr:`dst_id` matches
:data:`mitogen.context_id`, otherwise they are routed up or downstream.
.. attribute:: src_id
Integer source context ID. Used as the target of replies if any are
generated.
.. attribute:: auth_id
The context ID under whose authority the message is acting. See
:py:ref:`source-verification`.
.. attribute:: handle
Integer target handle in the destination context. This is one of the
:py:ref:`standard-handles`, or a dynamically generated handle used to
receive a one-time reply, such as the return value of a function call.
.. attribute:: reply_to
Integer target handle to direct any reply to this message. Used to
receive a one-time reply, such as the return value of a function call.
:data:`IS_DEAD` has a special meaning when it appears in this field.
.. attribute:: data
Message data, which may be raw or pickled.
.. attribute:: is_dead
:data:`True` if :attr:`reply_to` is set to the magic value

37
docs/howitworks.rst
Unescape Escape View File

@ -258,7 +258,7 @@ Stream Protocol
.. currentmodule:: mitogen.core
Once connected, a basic framing protocol is used to communicate between
parent and child:
parent and child. Integers use big endian in their encoded form.
.. list-table::
:header-rows: 1
@ -342,23 +342,6 @@ Masters listen on the following handles:
million parent contexts to be created and destroyed before the associated
Router must be recreated.
.. _IS_DEAD:
.. currentmodule:: mitogen.core
.. data:: IS_DEAD
Special value used to signal disconnection or the inability to route a
message, when it appears in the `reply_to` field. Usually causes
:class:`mitogen.core.ChannelError` to be raised when it is received.
It indicates the sender did not know how to process the message, or wishes
no further messages to be delivered to it. It is used when:
* a remote receiver is disconnected or explicitly closed.
* a related message could not be delivered due to no route existing for it.
* a router is being torn down, as a sentinel value to notify
:py:meth:`mitogen.core.Router.add_handler` callbacks to clean up.
Children listen on the following handles:
.. _LOAD_MODULE:
@ -478,6 +461,24 @@ Non-master parents also listen on the following handles:
ensuring they are cached and deduplicated at each hop in the chain leading
to the target context.
Special values for the `reply_to` field:
.. _IS_DEAD:
.. currentmodule:: mitogen.core
.. data:: IS_DEAD
Special value used to signal disconnection or the inability to route a
message, when it appears in the `reply_to` field. Usually causes
:class:`mitogen.core.ChannelError` to be raised when it is received.
It indicates the sender did not know how to process the message, or wishes
no further messages to be delivered to it. It is used when:
* a remote receiver is disconnected or explicitly closed.
* a related message could not be delivered due to no route existing for it.
* a router is being torn down, as a sentinel value to notify
:py:meth:`mitogen.core.Router.add_handler` callbacks to clean up.
Additional handles are created to receive the result of every function call
triggered by :py:meth:`call_async() <mitogen.parent.Context.call_async>`.

30
mitogen/__init__.py
Unescape Escape View File

@ -36,33 +36,19 @@ be expected. On the slave, it is built dynamically during startup.
__version__ = (0, 0, 2)
#: This is ``False`` in slave contexts. It is used in single-file Python
#: programs to avoid reexecuting the program's :py:func:`main` function in the
#: slave. For example:
#:
#: .. code-block:: python
#:
#: def do_work():
#: os.system('hostname')
#:
#: def main(broker):
#: context = mitogen.master.connect(broker)
#: context.call(do_work) # Causes slave to import __main__.
#:
#: if __name__ == '__main__' and mitogen.is_master:
#: import mitogen.utils
#: mitogen.utils.run_with_broker(main)
#:
#: This is :data:`False` in slave contexts. Previously it was used to prevent
#: re-execution of :mod:`__main__` in single file programs, however that now
#: happens automatically.
is_master = True
#: This is ``0`` in a master, otherwise it is a master-generated ID unique to
#: This is `0` in a master, otherwise it is the master-assigned ID unique to
#: the slave context used for message routing.
context_id = 0
#: This is ``None`` in a master, otherwise it is the master-generated ID unique
#: to the slave's parent context.
#: This is :data:`None` in a master, otherwise it is the master-assigned ID
#: unique to the slave's parent context.
parent_id = None
@ -76,8 +62,8 @@ def main(log_level='INFO', profiling=False):
Convenience decorator primarily useful for writing discardable test
scripts.
In the master process, when `func` is defined in the ``__main__`` module,
arranges for `func(router)` to be invoked immediately, with
In the master process, when `func` is defined in the :mod:`__main__`
module, arranges for `func(router)` to be invoked immediately, with
:py:class:`mitogen.master.Router` construction and destruction handled just
as in :py:func:`mitogen.utils.run_with_router`. In slaves, this function
does nothing.

6
mitogen/core.py
Unescape Escape View File

@ -26,6 +26,12 @@
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
"""
This module implements most package functionality, but remains separate from
non-essential code in order to reduce its size, since it is also serves as the
bootstrap implementation sent to every new slave context.
"""
import collections
import errno
import fcntl

84
mitogen/fakessh.py
Unescape Escape View File

@ -26,6 +26,70 @@
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
"""
:mod:`mitogen.fakessh` is a stream implementation that starts a subprocess with
its environment modified such that ``PATH`` searches for `ssh` return a Mitogen
implementation of SSH. When invoked, this implementation arranges for the
command line supplied by the caller to be executed in a remote context, reusing
the parent context's (possibly proxied) connection to that remote context.
This allows tools like `rsync` and `scp` to transparently reuse the connections
and tunnels already established by the host program to connect to a target
machine, without wasteful redundant SSH connection setup, 3-way handshakes, or
firewall hopping configurations, and enables these tools to be used in
impossible scenarios, such as over `sudo` with ``requiretty`` enabled.
The fake `ssh` command source is written to a temporary file on disk, and
consists of a copy of the :py:mod:`mitogen.core` source code (just like any
other child context), with a line appended to cause it to connect back to the
host process over an FD it inherits. As there is no reliance on an existing
filesystem file, it is possible for child contexts to use fakessh.
As a consequence of connecting back through an inherited FD, only one SSH
invocation is possible, which is fine for tools like `rsync`, however in future
this restriction will be lifted.
Sequence:
1. ``fakessh`` Context and Stream created by parent context. The stream's
buffer has a :py:func:`_fakessh_main` :py:data:`CALL_FUNCTION
<mitogen.core.CALL_FUNCTION>` enqueued.
2. Target program (`rsync/scp/sftp`) invoked, which internally executes
`ssh` from ``PATH``.
3. :py:mod:`mitogen.core` bootstrap begins, recovers the stream FD
inherited via the target program, established itself as the fakessh
context.
4. :py:func:`_fakessh_main` :py:data:`CALL_FUNCTION
<mitogen.core.CALL_FUNCTION>` is read by fakessh context,
a. sets up :py:class:`IoPump` for stdio, registers
stdin_handle for local context.
b. Enqueues :py:data:`CALL_FUNCTION <mitogen.core.CALL_FUNCTION>` for
:py:func:`_start_slave` invoked in target context,
i. the program from the `ssh` command line is started
ii. sets up :py:class:`IoPump` for `ssh` command line process's
stdio pipes
iii. returns `(control_handle, stdin_handle)` to
:py:func:`_fakessh_main`
5. :py:func:`_fakessh_main` receives control/stdin handles from from
:py:func:`_start_slave`,
a. registers remote's stdin_handle with local :py:class:`IoPump`.
b. sends `("start", local_stdin_handle)` to remote's control_handle
c. registers local :py:class:`IoPump` with
:py:class:`mitogen.core.Broker`.
d. loops waiting for `local stdout closed && remote stdout closed`
6. :py:func:`_start_slave` control channel receives `("start", stdin_handle)`,
a. registers remote's stdin_handle with local :py:class:`IoPump`
b. registers local :py:class:`IoPump` with
:py:class:`mitogen.core.Broker`.
c. loops waiting for `local stdout closed && remote stdout closed`
"""
import getopt
import inspect
import os
@ -330,6 +394,26 @@ def _get_econtext_config(context, sock2):
@mitogen.core.takes_econtext
@mitogen.core.takes_router
def run(dest, router, args, deadline=None, econtext=None):
"""
Run the command specified by `args` such that ``PATH`` searches for SSH by
the command will cause its attempt to use SSH to execute a remote program
to be redirected to use mitogen to execute that program using the context
`dest` instead.
:param list args:
Argument vector.
:param mitogen.core.Context dest:
The destination context to execute the SSH command line in.
:param mitogen.core.Router router:
:param list[str] args:
Command line arguments for local program, e.g.
``['rsync', '/tmp', 'remote:/tmp']``
:returns:
Exit status of the child process.
"""
if econtext is not None:
mitogen.parent.upgrade_router(econtext)

7
mitogen/master.py
Unescape Escape View File

@ -26,6 +26,13 @@
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
"""
This module implements functionality required by master processes, such as
starting new contexts via SSH. Its size is also restricted, since it must
be sent to any context that will be used to establish additional child
contexts.
"""
import dis
import imp
import inspect

6
mitogen/parent.py
Unescape Escape View File

@ -26,6 +26,12 @@
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
"""
This module defines functionality common to master and parent processes. It is
sent to any child context that is due to become a parent, due to recursive
connection.
"""
import errno
import fcntl
import getpass

Write Preview
Loading…
Cancel
Save