From 1bc8681cb40c98f3f0862f7514f22a5527b4f2ed Mon Sep 17 00:00:00 2001 From: David Wilson Date: Sun, 14 Aug 2016 16:33:09 +0100 Subject: [PATCH] More doc updates. --- docs/history.rst | 20 +++++++-------- docs/howitworks.rst | 60 +++++++++++++++++++++++++++++++------------- docs/index.rst | 4 +-- econtext/__init__.py | 5 ---- econtext/core.py | 3 +-- 5 files changed, 56 insertions(+), 36 deletions(-) diff --git a/docs/history.rst b/docs/history.rst index d6e14557..1f1c21e3 100644 --- a/docs/history.rst +++ b/docs/history.rst @@ -9,27 +9,27 @@ History The first version of econtext was written in late 2006 for use in an infrastructure management program, however at the time I lacked the pragmatism necessary for pushing my little design from concept to a working -implementation. I tired of it when I couldn't find a way to combine every +implementation. I tired of it when no way could be found to combine every communication style (*blocking execute function, asynchronous execute function, proxy slave-of-slave context*) into one neat abstraction. That unification -still has not happened, but I'm no longer as burdened by the problem. +still has not happened, but I'm no longer as burdened by it. Every few years I would pick through the source code, especially after periods -of working commercially with some contemporary management systems, none of -which in the meantime had anything close to as neat an approach to running +of working commercially with some contemporary infrastructure management +systems, none of which had anything close to as neat an approach to running Python code on remote machines, and suffered from shockingly beginner-level bugs such as failing to report SSH diagnostic messages. And every few years I'd put that code down again, especially since moving to an OS X laptop where :py:func:`select.poll` was not available, the struggle to get -back on top of the project seemed more hassle than it was worth. +back on top seemed more hassle than it was worth. That changed in 2016 during a quiet evening at home with a clear head and nothing better to do, after a full day of exposure to Ansible's intensely unbearable tendency to make running a 50 line Python script across a 1Gbit/sec -LAN feel like I were configuring a host on Mars. Poking through what Ansible -was doing, I was shocked to discover it writing temporary files everywhere, and -uploading a 56KiB zip file apparently for every playbook step. +LAN feel like I were configuring a host on Mars. Poking through Ansible, I was +shocked to discover it writing temporary files everywhere, and uploading a +56KiB zip file apparently for every playbook step. .. figure:: _static/wtf.gif @@ -42,8 +42,8 @@ write the IO and log forwarders, rewrite the module importer, move from :py:func:`select.poll` to :py:func:`select.select`, and even refactor the special cases out of the main loop. -So there you have it. As of writing :py:mod:`econtext.core` consists of 550 -source lines, and those 550 lines have taken me almost a decade to write. I +So there you have it. As of writing :py:mod:`econtext.core` consists of 528 +source lines, and those 528 lines have taken me almost a decade to write. I have long had a preference for avoiding infrastructure work commercially, not least for the inescapable depression induced by considering the wasted effort across the world caused by universally horrific tooling. This is my small diff --git a/docs/howitworks.rst b/docs/howitworks.rst index ed0550f6..5f1601d4 100644 --- a/docs/howitworks.rst +++ b/docs/howitworks.rst @@ -195,6 +195,26 @@ written to by and :py:meth:`call() `. +Shutdown +######## + +When the master signals the :py:data:`CALL_FUNCTION +` :py:class:`Channel ` is +closed, the slave calls :py:meth:`shutdown() ` +followed by :py:meth:`wait() ` on its own broker, +triggering graceful shutdown. + +During shutdown, the master will wait a few seconds for slaves to disconnect +gracefully before force disconnecting them, while the slaves will use that time +to call :py:meth:`socket.shutdown(SHUT_WR) ` on their +:py:class:`IoLogger ` socket's write ends before +draining any remaining data buffered on the read ends. + +If the main thread (responsible for function call dispatch) fails to trigger +shutdown (because some user function is hanging), then the eventual force +disconnection by the master will cause the IO multiplexer thread to enter +shutdown by itself. + Stream Protocol --------------- @@ -239,17 +259,22 @@ Slaves listen on the following handles: imports ``mod_name``, then attempts to execute `class_name.func_name(\*args, \**kwargs)`. -.. data:: econtext.core.SHUTDOWN - - Triggers :py:meth:`econtext.core.Broker.shutdown` remotely, causing the - slave to drain its :py:class:`IoLoggers ` and - output stream buffer before disconnecting from the master and terminating - the process. - Additional handles are created to receive the result of every function call triggered by :py:meth:`call_with_deadline() `. +Sentinel Value +############## + +.. autodata:: econtext.core._DEAD + +The special value :py:data:`econtext.core._DEAD` is used to signal +disconnection or closure of the remote end. It is used internally by +:py:class:`Channel ` and also passed to any function +still registered with :py:meth:`add_handle_cb() +` during Broker shutdown. + + Use of Pickle ############# @@ -261,13 +286,14 @@ serialization code in the bootstrap. The pickler active in slave contexts will instantiate any class, however in the master it is initially restricted to only permitting -:py:class:`econtext.core.CallError`. While not recommended, it is possible to -register more using :py:meth:`econtext.master.LocalStream.allow_class`. +:py:class:`CallError ` and :py:data:`_DEAD +`. While not recommended, it is possible to register more +using :py:meth:`econtext.master.LocalStream.allow_class`. -The choice of pickle is one area to be revisited later. All accounts suggest it -cannot be used securely, however few of those accounts appear to be expert -opinions, and none mention any additional attacks that would not be prevented -by using a restrictive class whitelist. +The choice of Pickle is one area to be revisited later. All accounts suggest it +cannot be used securely, however few of those accounts appear to be expert, and +none mention any additional attacks that would not be prevented by using a +restrictive class whitelist. Use of HMAC @@ -348,15 +374,15 @@ package. Child Module Enumeration ######################## -Package children are enumerated using the :py:mod:`pkgutil` module. +Package children are enumerated using :py:func:`pkgutil.iter_modules`. Use Of Threads -------------- -The package mandatorily runs the IO multiplexer in a thread. This is so the -multiplexer always retains control flow in order to shut down gracefully, say, -if the user's code has hung and the master context has disconnected. +The package always runs the IO multiplexer in a thread. This is so the +multiplexer retains control flow in order to shut down gracefully, say, if the +user's code has hung and the master context has disconnected. While it is possible for the IO multiplexer to recover control of a hung function call on UNIX using for example :py:mod:`signal.SIGALRM `, this diff --git a/docs/index.rst b/docs/index.rst index d34d9d52..83d68d4f 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -18,8 +18,8 @@ Introduction ------------ The Python ``econtext`` package implements external *execution contexts*: an -execution context is somewhere you can run Python code external to your main -process, even on a remote machine. +execution context is somewhere you can run Python functions external to your +main process, even on a remote machine. There is **no requirement for installing packages, copying files around, writing shell scripts, upfront configuration, or providing any secondary link diff --git a/econtext/__init__.py b/econtext/__init__.py index 576bb09d..a3d7315d 100644 --- a/econtext/__init__.py +++ b/econtext/__init__.py @@ -1,9 +1,6 @@ """ On the econtext master, this is imported from ``econtext/__init__.py`` as would be expected. On the slave, it is built dynamically during startup. - -As a convenience, the econtext package exports all of the functions and -variables from :py:mod:`econtext.core`. """ #: This is ``True`` in slave contexts. It is used in single-file Python @@ -24,5 +21,3 @@ variables from :py:mod:`econtext.core`. #: econtext.utils.run_with_broker(main) #: slave = False - -from econtext.core import * # NOQA diff --git a/econtext/core.py b/econtext/core.py index ce8cf78f..6d3aae1b 100644 --- a/econtext/core.py +++ b/econtext/core.py @@ -71,6 +71,7 @@ class Dead(object): return '' +#: Sentinel value used to represent Channel disconnection. _DEAD = Dead() @@ -656,8 +657,6 @@ class ExternalContext(object): sys.modules['econtext'] = econtext sys.modules['econtext.core'] = econtext.core - exec 'from econtext.core import *' in vars(econtext) - for klass in vars(econtext.core).itervalues(): if hasattr(klass, '__module__'): klass.__module__ = 'econtext.core'