When a context's Stream is disconnected, now any reply_to handlers
waiting for that specific context will be cancelled, rather than hanging
until all pending handelrs are cancelled during Broker is torn down.
This is groundwork for a bunch of things, including moving connect() to
the Broker thread
This propagates messages up the tree until they encounter a node that
knows how to forward them back downstream. As a consequence for invalid
context IDs, a 'no route' message will always be logged by the master.
Now there is a single handle namespace in each context, indpendent of
the source of the message. Update module forwarder etc. to cope with
that.
This is to support slave contexts communicating without the master's
intercession.
By the the time Waker.wake() writes to the pipe, any shared state that
would trigger action on the Broker thread has already been updated, so
reading subsequent bytes caused by later Waker.wake() can't miss
anything.
This avoids spewing tons of logs messages every time there is a storm of
user-generated log messages.
* Header now contains (src, dst) context IDs for routing.
* econtext.context_id now contains current process' context ID.
* Now do 16kb-sized reads rather than 4kb.
* econtext package is uniformly imported in econtext/core.py in slave
and master.
* Introduce econtext.core.Message() to centralize pickling policy, and
various function interfaces, may rip it out again later.
* Teach slave/first stage to preserve the copy of econtext.core sent to
it, so that it can be used for subsequent slave-of-slave bootstraps.
* Disconnect Stream from Context, and teach Context to send messages via
Router. In this way the Context class works identically for slaves
directly connected via a Stream, or those for whom other slaves are
acting as proxies.
* Implement Router, which knows a list of contexts reachable via a
Stream. Move context registry out of Broker and into Router.
* Move _invoke crap out of stream and into Context.
* Try to avoid pickling on the Broker thread wherever possible.
* Delete connection-specific fields from Context, they live on the
associated Stream subclass now instead.
* Merge alloc_handle() and add_handle_cb() into add_handler().
* s/enqueue/send/
* Add a hacky guard to prevent send_await() deadlock from Broker thread.
* Temporarily break shutdown logic: graceful shutdown is broken since
Broker doesn't know about which contexts exist any more.
* Handle EIO in iter_read() too. Also need to support ECONNRESET in here.
* Make iter_read() show last 100 bytes on failure.
* econtext.master.connect() is now econtext.master.Router.connect(),
move most of the context/stream construction cutpaste into a single
function, and Stream.construct().
* Stop using sys.executable, since it is the empty string when Python
has been started with a custom argv[0]. Hard-wire python2.7 for now.
* Streams now have names, which are used as the default name for the
associated Context during construction. That way Stream<->Context
association is still fairly obviously and Stream.repr() prints
something nice.
Stop using cPickle on the broker thread where it is not known whether
the pickle data would cause the import machinery to be invoked, which
currently relies on blocking calls. Huge mess but it works.
This is due to:
context.call(some.module.func, another.module.func)
We stringify ("some.module", "func"), but the reference to
another.module.func is passed into the pickle machinery, and there's no
way to generically stringify all function references in user data for
reification on the main thread, without doing something like this
instead.
David Wilson