@ -1,6 +1,6 @@
How eco ntext Works
How Mitog en Works
==================
=================
Some effort is required to accomplish the seemingly magical feat of
Some effort is required to accomplish the seemingly magical feat of
bootstrapping a remote Python process without any software installed on the
bootstrapping a remote Python process without any software installed on the
@ -17,7 +17,7 @@ necessary for something on the remote to be prepared to decompress the payload
and feed it to a Python interpreter. Since we would like to avoid writing an
and feed it to a Python interpreter. Since we would like to avoid writing an
error-prone shell fragment to implement this, and since we must avoid writing
error-prone shell fragment to implement this, and since we must avoid writing
to the remote machine's disk in case it is read-only, the Python process
to the remote machine's disk in case it is read-only, the Python process
started on the remote machine by `` econtext ``
started on the remote machine by Mitogen immediately forks in order to
implement the decompression.
implement the decompression.
@ -25,7 +25,7 @@ Python Command Line
###################
###################
The Python command line sent to the host is a base64-encoded copy of the
The Python command line sent to the host is a base64-encoded copy of the
:py:meth: ` eco ntext .master.LocalStream._first_stage` function, which has been:py:meth: ` mitog en.master.LocalStream._first_stage` function, which has been
carefully optimized to reduce its size. Prior to base64 encoding,
carefully optimized to reduce its size. Prior to base64 encoding,
`` CONTEXT_NAME `` is replaced with the desired context name in the function's`` CONTEXT_NAME `` is replaced with the desired context name in the function's
source code.
source code.
@ -50,8 +50,8 @@ process.
After fork, the parent half overwrites its `` stdin `` with the read end of the
After fork, the parent half overwrites its `` stdin `` with the read end of the
pipe, and the child half writes the string `` EC0\n `` , then begins reading the
pipe, and the child half writes the string `` EC0\n `` , then begins reading the
:py:mod: `zlib` -compressed payload supplied on `` stdin `` by the econtext master,:py:mod: `zlib` -compressed payload supplied on `` stdin `` by the master, and
and writing the decompressed result to the write-end of the UNIX pipe.writing the decompressed result to the write-end of the UNIX pipe.
To allow recovery of `` stdin `` for reuse by the bootstrapped process for
To allow recovery of `` stdin `` for reuse by the bootstrapped process for
master<->slave communication, it is necessary for the first stage to avoid
master<->slave communication, it is necessary for the first stage to avoid
@ -83,25 +83,25 @@ Bootstrap Preparation
Now we have the mechanism in place to send a :py:mod: `zlib` -compressed script
Now we have the mechanism in place to send a :py:mod: `zlib` -compressed script
to the remote Python interpreter, it is time to choose what to send.
to the remote Python interpreter, it is time to choose what to send.
The script sent is simply the source code for :py:mod: ` eco ntext .core`, with a
The script sent is simply the source code for :py:mod: ` mitog en.core`, with a
single line suffixed to trigger execution of the
single line suffixed to trigger execution of the
:py:meth: ` eco ntext .core.ExternalContext.main` function. The encoded arguments:py:meth: ` mitog en.core.ExternalContext.main` function. The encoded arguments
to the main function include some additional details, such as the logging package
to the main function include some additional details, such as the logging package
level that was active in the parent process, and a random secret key that may
level that was active in the parent process, and a random secret key that may
later be used to generate HMAC signatures over the data frames that will be
later be used to generate HMAC signatures over the data frames that will be
exchanged after bootstrap.
exchanged after bootstrap.
After the script source code is prepared, it is passed through
After the script source code is prepared, it is passed through
:py:func: ` eco ntext .master.minimize_source` to strip it of docstrings and:py:func: ` mitog en.master.minimize_source` to strip it of docstrings and
comments, while preserving line numbers. This reduces the compressed payload
comments, while preserving line numbers. This reduces the compressed payload
by around 20%.
by around 20%.
Preserving The ` eco ntext .core`
Preserving The ` mitog en.core`
#####################################
####################################
One final trick is implemented in the first stage: after bootstrapping the new
One final trick is implemented in the first stage: after bootstrapping the new
slave, it writes a duplicate copy of the :py:mod: ` eco ntext .core` source it just
slave, it writes a duplicate copy of the :py:mod: ` mitog en.core` source it just
used to bootstrap it back into another pipe connected to the slave. The slave's
used to bootstrap it back into another pipe connected to the slave. The slave's
module importer cache is initialized with a copy of the source, so that
module importer cache is initialized with a copy of the source, so that
subsequent bootstraps of slave-of-slaves do not require the source to be
subsequent bootstraps of slave-of-slaves do not require the source to be
@ -122,26 +122,26 @@ to receive messages.
ExternalContext.main()
ExternalContext.main()
----------------------
----------------------
.. automethod :: eco ntext .core.ExternalContext.main.. automethod :: mitog en.core.ExternalContext.main
Generating A Synthetic ` eco ntext `
Generating A Synthetic ` mitog en`
#########################################
########################################
Since the bootstrap consists of the :py:mod: ` eco ntext .core` source code, and
Since the bootstrap consists of the :py:mod: ` mitog en.core` source code, and
this code is loaded by Python by way of its main script (`` __main__ `` module),
this code is loaded by Python by way of its main script (`` __main__ `` module),
initially the module layout in the slave will be incorrect.
initially the module layout in the slave will be incorrect.
The first step taken after bootstrap is to rearrange :py:data: `sys.modules` slightly
The first step taken after bootstrap is to rearrange :py:data: `sys.modules` slightly
so that :py:mod: ` eco ntext .core` appears in the correct location, and all
so that :py:mod: ` mitog en.core` appears in the correct location, and all
classes defined in that module have their `` __module__ `` attribute fixed up
classes defined in that module have their `` __module__ `` attribute fixed up
such that :py:mod: `cPickle` correctly serializes instance module names.
such that :py:mod: `cPickle` correctly serializes instance module names.
Once a synthetic :py:mod: ` eco ntext ` package and :py:mod: ` eco ntext .core` module
Once a synthetic :py:mod: ` mitog en` package and :py:mod: ` mitog en.core` module
have been generated, the bootstrap **deletes** `sys.modules['__main__']`
have been generated, the bootstrap **deletes** `sys.modules['__main__']`
that any attempt to import it (by :py:mod: `cPickle` ) will cause the import to
that any attempt to import it (by :py:mod: `cPickle` ) will cause the import to
be satisfied by fetching the econtext master's actual `` __main__ `` module. This
be satisfied by fetching the master's actual `` __main__ `` module. This is
is necessary to allow master programs to be written as a self-contained Pythonnecessary to allow master programs to be written as a self-contained Python
script.
script.
@ -161,8 +161,8 @@ Setup Logging
The slave's :py:mod: `logging` package root logger is configured to have the
The slave's :py:mod: `logging` package root logger is configured to have the
same log level as the root logger in the master, and
same log level as the root logger in the master, and
:py:class: ` eco ntext .core.LogHandler` is installed to forward logs to the master:py:class: ` mitog en.core.LogHandler` is installed to forward logs to the master
context's :py:data: `FORWARD_LOG < eco ntext .core.FORWARD_LOG>` handle.
context's :py:data: `FORWARD_LOG < mitog en.core.FORWARD_LOG>` handle.
The log level is copied into the slave to avoid generating a potentially large
The log level is copied into the slave to avoid generating a potentially large
amount of network IO forwarding logs that will simply be filtered away once
amount of network IO forwarding logs that will simply be filtered away once
@ -172,7 +172,7 @@ they reach the master.
The Module Importer
The Module Importer
###################
###################
An instance of :py:class: ` eco ntext .core.Importer` is installed in
An instance of :py:class: ` mitog en.core.Importer` is installed in
:py:data: `sys.meta_path` , where Python's `` import `` statement will execute it:py:data: `sys.meta_path` , where Python's `` import `` statement will execute it
before attempting to find a module locally.
before attempting to find a module locally.
@ -180,7 +180,7 @@ before attempting to find a module locally.
Standard IO Redirection
Standard IO Redirection
#######################
#######################
Two instances of :py:class: ` eco ntext .core.IoLogger` are created, one for
Two instances of :py:class: ` mitog en.core.IoLogger` are created, one for
`` stdout `` and one for `` stderr `` . This class creates a UNIX pipe whose read`` stdout `` and one for `` stderr `` . This class creates a UNIX pipe whose read
end is added to the IO multiplexer, and whose write end is used to overwrite
end is added to the IO multiplexer, and whose write end is used to overwrite
the handles inherited during process creation.
the handles inherited during process creation.
@ -200,26 +200,26 @@ Function Call Dispatch
######################
######################
After all initialization is complete, the slave's main thread sits in a loop
After all initialization is complete, the slave's main thread sits in a loop
reading from a :py:class: `Channel < eco ntext .core.Channel>` connected to the
reading from a :py:class: `Channel < mitog en.core.Channel>` connected to the
:py:data: `CALL_FUNCTION < eco ntext .core.CALL_FUNCTION>` handle. This handle is:py:data: `CALL_FUNCTION < mitog en.core.CALL_FUNCTION>` handle. This handle is
written to by
written to by
:py:meth: `call_with_deadline() < eco ntext .master.Context.call_with_deadline>`:py:meth: `call_with_deadline() < mitog en.master.Context.call_with_deadline>`
and :py:meth: `call() < eco ntext .master.Context.call>`.
and :py:meth: `call() < mitog en.master.Context.call>`.
Shutdown
Shutdown
########
########
When the master signals the :py:data:`CALL_FUNCTION
When the master signals the :py:data:`CALL_FUNCTION
< eco ntext .core.CALL_FUNCTION>` :py:class:` co ntext .core.Channel>` is< mitog en.core.CALL_FUNCTION>` :py:class:` mitog en.core.Channel>` is
closed, the slave calls :py:meth: `shutdown() < eco ntext .core.Broker.shutdown>`
closed, the slave calls :py:meth: `shutdown() < mitog en.core.Broker.shutdown>`
followed by :py:meth: `wait() < eco ntext .core.Broker.wait>` on its own broker,
followed by :py:meth: `wait() < mitog en.core.Broker.wait>` on its own broker,
triggering graceful shutdown.
triggering graceful shutdown.
During shutdown, the master will wait a few seconds for slaves to disconnect
During shutdown, the master will wait a few seconds for slaves to disconnect
gracefully before force disconnecting them, while the slaves will use that time
gracefully before force disconnecting them, while the slaves will use that time
to call :py:meth: `socket.shutdown(SHUT_WR) <socket.socket.shutdown>` on their
to call :py:meth: `socket.shutdown(SHUT_WR) <socket.socket.shutdown>` on their
:py:class: `IoLogger < eco ntext .core.IoLogger>` socket's write ends before:py:class: `IoLogger < mitog en.core.IoLogger>` socket's write ends before
draining any remaining data buffered on the read ends.
draining any remaining data buffered on the read ends.
An alternative approach is to wait until the socket is completely closed, with
An alternative approach is to wait until the socket is completely closed, with
@ -260,12 +260,12 @@ master and slave:
Masters listen on the following handles:
Masters listen on the following handles:
.. data :: eco ntext .core.FORWARD_LOG.. data :: mitog en.core.FORWARD_LOG
Receives `(logger_name, level, msg)`
Receives `(logger_name, level, msg)`
master's `` co ntext .ctx.<context_name>`` logger.
master's `` mitog en.ctx.<context_name>`` logger.
.. data :: eco ntext .core.GET_MODULE.. data :: mitog en.core.GET_MODULE
Receives `(reply_to, fullname)`
Receives `(reply_to, fullname)`
module named `` fullname `` , and writes the source along with some metadata
module named `` fullname `` , and writes the source along with some metadata
@ -274,11 +274,11 @@ Masters listen on the following handles:
Slaves listen on the following handles:
Slaves listen on the following handles:
.. data :: eco ntext .core.CALL_FUNCTION.. data :: mitog en.core.CALL_FUNCTION
Receives `(with_context, mod_name, class_name, func_name, args, kwargs)`
Receives `(with_context, mod_name, class_name, func_name, args, kwargs)`
5-tuples from
5-tuples from
:py:meth: `call_with_deadline() < eco ntext .master.Context.call_with_deadline>`,
:py:meth: `call_with_deadline() < mitog en.master.Context.call_with_deadline>`,
imports `` mod_name `` , then attempts to execute
imports `` mod_name `` , then attempts to execute
`class_name.func_name(\*args, \**kwargs)`
`class_name.func_name(\*args, \**kwargs)`
@ -289,7 +289,7 @@ Slaves listen on the following handles:
it, and arranging for the connection to its parent context to be closed
it, and arranging for the connection to its parent context to be closed
shortly thereafter.
shortly thereafter.
.. data :: eco ntext .core.ADD_ROUTE.. data :: mitog en.core.ADD_ROUTE
Receives `(target_id, via_id)`
Receives `(target_id, via_id)`
arriving at this context on any Stream should be forwarded on the stream
arriving at this context on any Stream should be forwarded on the stream
@ -302,7 +302,7 @@ Slaves listen on the following handles:
established.
established.
Given a chain `master -> ssh1 -> sudo1` `ADD_ROUTE`
Given a chain `master -> ssh1 -> sudo1` `ADD_ROUTE`
necessary, since :py:class: ` eco ntext .core.Router` in the `ssh`
necessary, since :py:class: ` mitog en.core.Router` in the `ssh`
arrange to update its routes while setting up the new slave during
arrange to update its routes while setting up the new slave during
`proxy_connect()`
`proxy_connect()`
@ -315,11 +315,11 @@ Slaves listen on the following handles:
Slaves that have ever been used to create a descendent child context also
Slaves that have ever been used to create a descendent child context also
listen on the following handles:
listen on the following handles:
.. data :: eco ntext .core.GET_MODULE.. data :: mitog en.core.GET_MODULE
As with master's `` GET_MODULE `` , except this implementation
As with master's `` GET_MODULE `` , except this implementation
(:py:class: ` eco ntext .master.ModuleForwarder`) serves responses using
(:py:class: ` mitog en.master.ModuleForwarder`) serves responses using
:py:class: ` eco ntext .core.Importer`'s cache before forwarding the request to
:py:class: ` mitog en.core.Importer`'s cache before forwarding the request to
its parent context. The response is cached by each context in turn before
its parent context. The response is cached by each context in turn before
being forwarded on to the slave context that originally made the request.
being forwarded on to the slave context that originally made the request.
In this way, the master need never re-send a module it has already sent to
In this way, the master need never re-send a module it has already sent to
@ -327,19 +327,19 @@ listen on the following handles:
Additional handles are created to receive the result of every function call
Additional handles are created to receive the result of every function call
triggered by :py:meth: `call_with_deadline() < eco ntext .master.Context.call_with_deadline>`.
triggered by :py:meth: `call_with_deadline() < mitog en.master.Context.call_with_deadline>`.
Sentinel Value
Sentinel Value
##############
##############
.. autodata :: eco ntext .core._DEAD.. autodata :: mitog en.core._DEAD
The special value :py:data: ` eco ntext .core._DEAD` is used to signal
The special value :py:data: ` mitog en.core._DEAD` is used to signal
disconnection or closure of the remote end. It is used internally by
disconnection or closure of the remote end. It is used internally by
:py:class: `Channel < eco ntext .core.Channel>` and also passed to any function:py:class: `Channel < mitog en.core.Channel>` and also passed to any function
still registered with :py:meth:`add_handler()
still registered with :py:meth:`add_handler()
< eco ntext .core.Router.add_handler>` during Broker shutdown.< mitog en.core.Router.add_handler>` during Broker shutdown.
Use of Pickle
Use of Pickle
@ -353,9 +353,9 @@ serialization code in the bootstrap.
The pickler active in slave contexts will instantiate any class, however in the
The pickler active in slave contexts will instantiate any class, however in the
master it is initially restricted to only permitting
master it is initially restricted to only permitting
:py:class: `CallError < eco ntext .core.CallError>` and :py:data:`_DEAD:py:class: `CallError < mitog en.core.CallError>` and :py:data:`_DEAD
< eco ntext .core._DEAD>`. While not recommended, it is possible to register more< mitog en.core._DEAD>`. While not recommended, it is possible to register more
using :py:meth: ` eco ntext .master.LocalStream.allow_class`.
using :py:meth: ` mitog en.master.LocalStream.allow_class`.
The choice of Pickle is one area to be revisited later. All accounts suggest it
The choice of Pickle is one area to be revisited later. All accounts suggest it
cannot be used securely, however few of those accounts appear to be expert, and
cannot be used securely, however few of those accounts appear to be expert, and
@ -379,7 +379,7 @@ off-the-shelf implementations are for the most part entirely inappropriate. For
example, a minimal copy of Twisted weighs in at around 440KiB and is composed
example, a minimal copy of Twisted weighs in at around 440KiB and is composed
of approximately 115 files. Even if we could arrange for an entire Python
of approximately 115 files. Even if we could arrange for an entire Python
package to be transferred during bootstrap, this minimal configuration is
package to be transferred during bootstrap, this minimal configuration is
massive in comparison to eco ntext 's solution, multiplies quickly in the
massive in comparison to Mitog en's solution, multiplies quickly in the
presence of many machines, and would require manually splitting up the parts of
presence of many machines, and would require manually splitting up the parts of
Twisted that we would like to use.
Twisted that we would like to use.
@ -391,7 +391,7 @@ Routing assumes it is impossible to construct a tree such that one of a
context's parents will not know the ID of a target the context is attempting to
context's parents will not know the ID of a target the context is attempting to
communicate with.
communicate with.
When :py:class: ` eco ntext .core.Router` receives a message, it checks the IDs
When :py:class: ` mitog en.core.Router` receives a message, it checks the IDs
associated with its directly connected streams for a potential route. If any
associated with its directly connected streams for a potential route. If any
stream matches, either because it directly connects to the target ID, or
stream matches, either because it directly connects to the target ID, or
because the master sent an `` ADD_ROUTE `` message associating it, then the
because the master sent an `` ADD_ROUTE `` message associating it, then the
@ -436,15 +436,15 @@ currently allocate new context IDs anyway.
Differences Between Master And Slave Brokers
Differences Between Master And Slave Brokers
############################################
############################################
The main difference between :py:class: ` eco ntext .core.Broker` and
The main difference between :py:class: ` mitog en.core.Broker` and
:py:class: ` eco ntext .master.Broker` is that when the stream connection to the:py:class: ` mitog en.master.Broker` is that when the stream connection to the
parent is lost in a slave, the broker will trigger its own shutdown.
parent is lost in a slave, the broker will trigger its own shutdown.
The Module Importer
The Module Importer
-------------------
-------------------
:py:class: ` eco ntext .core.Importer` is still a work in progress, as there:py:class: ` mitog en.core.Importer` is still a work in progress, as there
are a variety of approaches to implementing it, and the present implementation
are a variety of approaches to implementing it, and the present implementation
is not pefectly efficient in every case.
is not pefectly efficient in every case.
@ -452,7 +452,7 @@ It operates by intercepting ``import`` statements via `sys.meta_path`, asking
Python if it can satisfy the import by itself, and if not, indicating to Python
Python if it can satisfy the import by itself, and if not, indicating to Python
that it is capable of loading the module.
that it is capable of loading the module.
In :py:meth: `load_module() < eco ntext .core.Importer.load_module>` an RPC is
In :py:meth: `load_module() < mitog en.core.Importer.load_module>` an RPC is
started to the parent context, requesting the module source code. Once the
started to the parent context, requesting the module source code. Once the
source is fetched, the method builds a new module object using the best
source is fetched, the method builds a new module object using the best
practice documented in PEP-302.
practice documented in PEP-302.
@ -480,7 +480,7 @@ pointless network roundtrips. Therefore in addition to the
child modules known to exist.
child modules known to exist.
Before indicating it can satisfy an import request,
Before indicating it can satisfy an import request,
:py:class: ` eco ntext .core.Importer` first checks to see if the module belongs to:py:class: ` mitog en.core.Importer` first checks to see if the module belongs to
a package it has previously imported, and if so, ignores the request if the
a package it has previously imported, and if so, ignores the request if the
module does not appear in the enumeration of child modules belonging to the
module does not appear in the enumeration of child modules belonging to the
package.
package.
@ -505,7 +505,7 @@ mechanism is not portable to non-UNIX operating systems, and does not work in
every case, for example when Python blocks signals during a variety of
every case, for example when Python blocks signals during a variety of
:py:mod: `threading` package operations.:py:mod: `threading` package operations.
At some point it is likely eco ntext will be extended to support starting slaves
At some point it is likely Mitog en will be extended to support starting slaves
running on Windows. When that happens, it would be nice if the process model on
running on Windows. When that happens, it would be nice if the process model on
Windows and UNIX did not differ, and in fact the code used on both were
Windows and UNIX did not differ, and in fact the code used on both were
identical.
identical.
David Wilson
8 years ago