docs: fix pickler docs, begin relabelling master/slave->parent/child

pull/45/head
David Wilson 8 years ago
parent abf1aa33dd
commit a71c9fd128

@ -54,7 +54,7 @@ pipe, and the child half writes the string ``EC0\n``, then begins reading the
writing the decompressed result to the write-end of the UNIX pipe. writing the decompressed result to the write-end of the UNIX pipe.
To allow recovery of ``stdin`` for reuse by the bootstrapped process for To allow recovery of ``stdin`` for reuse by the bootstrapped process for
master<->slave communication, it is necessary for the first stage to avoid parent<->child communication, it is necessary for the first stage to avoid
closing ``stdin`` or reading from it until until EOF. Therefore, the master closing ``stdin`` or reading from it until until EOF. Therefore, the master
sends the :py:mod:`zlib`-compressed payload prefixed with an integer size, sends the :py:mod:`zlib`-compressed payload prefixed with an integer size,
allowing reading by the first stage of exactly the required bytes. allowing reading by the first stage of exactly the required bytes.
@ -100,10 +100,10 @@ Preserving The `mitogen.core` Source
#################################### ####################################
One final trick is implemented in the first stage: after bootstrapping the new One final trick is implemented in the first stage: after bootstrapping the new
slave, it writes a duplicate copy of the :py:mod:`mitogen.core` source it just child, it writes a duplicate copy of the :py:mod:`mitogen.core` source it just
used to bootstrap it back into another pipe connected to the slave. The slave's used to bootstrap it back into another pipe connected to the child. The child's
module importer cache is initialized with a copy of the source, so that module importer cache is initialized with a copy of the source, so that
subsequent bootstraps of slave-of-slaves do not require the source to be subsequent bootstraps of children-of-children do not require the source to be
fetched from the master a second time. fetched from the master a second time.
@ -129,7 +129,7 @@ Generating A Synthetic `mitogen` Package
Since the bootstrap consists of the :py:mod:`mitogen.core` source code, and Since the bootstrap consists of the :py:mod:`mitogen.core` source code, and
this code is loaded by Python by way of its main script (``__main__`` module), this code is loaded by Python by way of its main script (``__main__`` module),
initially the module layout in the slave will be incorrect. initially the module layout in the child will be incorrect.
The first step taken after bootstrap is to rearrange :py:data:`sys.modules` slightly The first step taken after bootstrap is to rearrange :py:data:`sys.modules` slightly
so that :py:mod:`mitogen.core` appears in the correct location, and all so that :py:mod:`mitogen.core` appears in the correct location, and all
@ -158,12 +158,12 @@ it. Therefore the bootstrap must call :py:func:`os.wait` soon after startup.
Setup Logging Setup Logging
############# #############
The slave's :py:mod:`logging` package root logger is configured to have the The child's :py:mod:`logging` package root logger is configured to have the
same log level as the root logger in the master, and same log level as the root logger in the master, and
:py:class:`mitogen.core.LogHandler` is installed to forward logs to the master :py:class:`mitogen.core.LogHandler` is installed to forward logs to the master
context's :py:data:`FORWARD_LOG <mitogen.core.FORWARD_LOG>` handle. context's :py:data:`FORWARD_LOG <mitogen.core.FORWARD_LOG>` handle.
The log level is copied into the slave to avoid generating a potentially large The log level is copied into the child to avoid generating a potentially large
amount of network IO forwarding logs that will simply be filtered away once amount of network IO forwarding logs that will simply be filtered away once
they reach the master. they reach the master.
@ -185,8 +185,8 @@ end is added to the IO multiplexer, and whose write end is used to overwrite
the handles inherited during process creation. the handles inherited during process creation.
Even without IO redirection, something must replace ``stdin`` and ``stdout``, Even without IO redirection, something must replace ``stdin`` and ``stdout``,
otherwise it is possible for the stream used for communication between the otherwise it is possible for the stream used for communication between parent
master and slave to be accidentally corrupted by subprocesses run by user code. and child to be accidentally corrupted by subprocesses run by user code.
The inherited ``stdin`` is replaced by a file descriptor pointing to The inherited ``stdin`` is replaced by a file descriptor pointing to
``/dev/null``. ``/dev/null``.
@ -198,7 +198,7 @@ active, so that ``print`` statements and suchlike promptly appear in the logs.
Function Call Dispatch Function Call Dispatch
###################### ######################
After all initialization is complete, the slave's main thread sits in a loop After all initialization is complete, the child's main thread sits in a loop
reading from a :py:class:`Channel <mitogen.core.Channel>` connected to the reading from a :py:class:`Channel <mitogen.core.Channel>` connected to the
:py:data:`CALL_FUNCTION <mitogen.core.CALL_FUNCTION>` handle. This handle is :py:data:`CALL_FUNCTION <mitogen.core.CALL_FUNCTION>` handle. This handle is
written to by written to by
@ -211,14 +211,14 @@ Shutdown
When the master signals the :py:data:`CALL_FUNCTION When the master signals the :py:data:`CALL_FUNCTION
<mitogen.core.CALL_FUNCTION>` :py:class:`Channel <mitogen.core.Channel>` is <mitogen.core.CALL_FUNCTION>` :py:class:`Channel <mitogen.core.Channel>` is
closed, the slave calls :py:meth:`shutdown() <mitogen.core.Broker.shutdown>` closed, the child calls :py:meth:`shutdown() <mitogen.core.Broker.shutdown>`
followed by :py:meth:`wait() <mitogen.core.Broker.wait>` on its own broker, followed by :py:meth:`wait() <mitogen.core.Broker.wait>` on its own broker,
triggering graceful shutdown. triggering graceful shutdown.
During shutdown, the master will wait a few seconds for slaves to disconnect During shutdown, the master will wait a few seconds for children to disconnect
gracefully before force disconnecting them, while the slaves will use that time gracefully before force disconnecting them, while the children will use that
to call :py:meth:`socket.shutdown(SHUT_WR) <socket.socket.shutdown>` on their time to call :py:meth:`socket.shutdown(SHUT_WR) <socket.socket.shutdown>` on
:py:class:`IoLogger <mitogen.core.IoLogger>` socket's write ends before their :py:class:`IoLogger <mitogen.core.IoLogger>` socket's write ends before
draining any remaining data buffered on the read ends. draining any remaining data buffered on the read ends.
An alternative approach is to wait until the socket is completely closed, with An alternative approach is to wait until the socket is completely closed, with
@ -239,7 +239,7 @@ Stream Protocol
--------------- ---------------
Once connected, a basic framing protocol is used to communicate between Once connected, a basic framing protocol is used to communicate between
master and slave: parent and child:
+--------------------+------+------------------------------------------------------+ +--------------------+------+------------------------------------------------------+
| Field | Size | Description | | Field | Size | Description |
@ -273,13 +273,13 @@ Masters listen on the following handles:
.. data:: mitogen.core.ALLOCATE_ID .. data:: mitogen.core.ALLOCATE_ID
Replies to any message sent to it with a newly allocated unique context ID, Replies to any message sent to it with a newly allocated unique context ID,
to allow slaves to safely start their own contexts. In future this is to allow children to safely start their own contexts. In future this is
likely to be replaced by 32-bit context IDs and random allocation, with an likely to be replaced by 32-bit context IDs and random allocation, with an
improved ``ADD_ROUTE`` message sent upstream rather than downstream that improved ``ADD_ROUTE`` message sent upstream rather than downstream that
generates NACKs if any ancestor already knows the ID. generates NACKs if any ancestor already knows the ID.
Slaves listen on the following handles: Children listen on the following handles:
.. data:: mitogen.core.CALL_FUNCTION .. data:: mitogen.core.CALL_FUNCTION
@ -290,9 +290,9 @@ Slaves listen on the following handles:
`class_name.func_name(\*args, \**kwargs)`. `class_name.func_name(\*args, \**kwargs)`.
When this channel is closed (by way of sending ``_DEAD`` to it), the When this channel is closed (by way of sending ``_DEAD`` to it), the
slave's main thread begins graceful shutdown of its own `Broker` and child's main thread begins graceful shutdown of its own `Broker` and
`Router`. Each slave is responsible for sending ``_DEAD`` to each of its `Router`. Each child is responsible for sending ``_DEAD`` to each of its
directly connected slaves in response to the master sending ``_DEAD`` to directly connected children in response to the master sending ``_DEAD`` to
it, and arranging for the connection to its parent context to be closed it, and arranging for the connection to its parent context to be closed
shortly thereafter. shortly thereafter.
@ -310,7 +310,7 @@ Slaves listen on the following handles:
Given a chain `master -> ssh1 -> sudo1`, no `ADD_ROUTE` message is Given a chain `master -> ssh1 -> sudo1`, no `ADD_ROUTE` message is
necessary, since :py:class:`mitogen.core.Router` in the `ssh` context can necessary, since :py:class:`mitogen.core.Router` in the `ssh` context can
arrange to update its routes while setting up the new slave during arrange to update its routes while setting up the new child during
`proxy_connect()`. `proxy_connect()`.
However, given a chain like `master -> ssh1 -> sudo1 -> ssh2 -> sudo2`, However, given a chain like `master -> ssh1 -> sudo1 -> ssh2 -> sudo2`,
@ -319,8 +319,8 @@ Slaves listen on the following handles:
establishment. establishment.
Slaves that have ever been used to create a descendent child context also Children that have ever been used to create a descendent child also listen on
listen on the following handles: the following handles:
.. data:: mitogen.core.GET_MODULE .. data:: mitogen.core.GET_MODULE
@ -328,7 +328,7 @@ listen on the following handles:
(:py:class:`mitogen.master.ModuleForwarder`) serves responses using (:py:class:`mitogen.master.ModuleForwarder`) serves responses using
:py:class:`mitogen.core.Importer`'s cache before forwarding the request to :py:class:`mitogen.core.Importer`'s cache before forwarding the request to
its parent context. The response is cached by each context in turn before its parent context. The response is cached by each context in turn before
being forwarded on to the slave context that originally made the request. being forwarded on to the child context that originally made the request.
In this way, the master need never re-send a module it has already sent to In this way, the master need never re-send a module it has already sent to
a direct descendant. a direct descendant.
@ -352,17 +352,16 @@ still registered with :py:meth:`add_handler()
Use of Pickle Use of Pickle
############# #############
The current implementation uses the Python :py:mod:`cPickle` module, with The current implementation uses the Python :py:mod:`cPickle` module, with a
mitigations to prevent untrusted slaves from triggering code excution in the restrictive class whitelist to prevent triggering undesirable code execution.
master. The primary reason for using :py:mod:`cPickle` is that it is The primary reason for using :py:mod:`cPickle` is that it is computationally
computationally efficient, and avoids including a potentially large body of efficient, and avoids including a potentially large body of serialization code
serialization code in the bootstrap. in the bootstrap.
The pickler active in slave contexts will instantiate any class, however in the The pickler will instantiate only built-in types and one of 3 constructor
master it is initially restricted to only permitting functions, to support unpickling :py:class:`CallError
:py:class:`CallError <mitogen.core.CallError>` and :py:data:`_DEAD <mitogen.core.CallError>`, :py:data:`_DEAD <mitogen.core._DEAD>`, and
<mitogen.core._DEAD>`. While not recommended, it is possible to register more :py:class:`Context <mitogen.core.Context>`.
using :py:meth:`mitogen.master.LocalStream.allow_class`.
The choice of Pickle is one area to be revisited later. All accounts suggest it The choice of Pickle is one area to be revisited later. All accounts suggest it
cannot be used securely, however few of those accounts appear to be expert, and cannot be used securely, however few of those accounts appear to be expert, and
@ -440,12 +439,12 @@ a new child context, but that is okay for now, since child contexts cannot
currently allocate new context IDs anyway. currently allocate new context IDs anyway.
Differences Between Master And Slave Brokers Differences Between Master And Child Brokers
############################################ ############################################
The main difference between :py:class:`mitogen.core.Broker` and The main difference between :py:class:`mitogen.core.Broker` and
:py:class:`mitogen.master.Broker` is that when the stream connection to the :py:class:`mitogen.master.Broker` is that when the stream connection to the
parent is lost in a slave, the broker will trigger its own shutdown. parent is lost in a child, the broker will trigger its own shutdown.
The Module Importer The Module Importer
@ -512,7 +511,6 @@ mechanism is not portable to non-UNIX operating systems, and does not work in
every case, for example when Python blocks signals during a variety of every case, for example when Python blocks signals during a variety of
:py:mod:`threading` package operations. :py:mod:`threading` package operations.
At some point it is likely Mitogen will be extended to support starting slaves At some point it is likely Mitogen will be extended to support children running
running on Windows. When that happens, it would be nice if the process model on on Windows. When that happens, it would be nice if the process model on Windows
Windows and UNIX did not differ, and in fact the code used on both were and UNIX did not differ, and in fact the code used on both were identical.
identical.

Loading…
Cancel
Save