|
|
@ -13,7 +13,7 @@ Liability Waiver
|
|
|
|
.. image:: images/radiation.png
|
|
|
|
.. image:: images/radiation.png
|
|
|
|
:align: right
|
|
|
|
:align: right
|
|
|
|
|
|
|
|
|
|
|
|
Before proceeding, it is crucial you understand what you're involving yourself
|
|
|
|
Before proceeding, it is critical you understand what you're involving yourself
|
|
|
|
and possibly your team with:
|
|
|
|
and possibly your team with:
|
|
|
|
|
|
|
|
|
|
|
|
* Constructing the most fundamental class, :py:class:`Broker
|
|
|
|
* Constructing the most fundamental class, :py:class:`Broker
|
|
|
@ -22,21 +22,21 @@ and possibly your team with:
|
|
|
|
not suffer from.
|
|
|
|
not suffer from.
|
|
|
|
|
|
|
|
|
|
|
|
While every effort is made to hide this complexity, you should expect
|
|
|
|
While every effort is made to hide this complexity, you should expect
|
|
|
|
threading-related encounters during development. See :ref:`troubleshooting`
|
|
|
|
threading-related encounters during development, and crucially after your
|
|
|
|
for more information.
|
|
|
|
program is in production. See :ref:`troubleshooting` for more information.
|
|
|
|
|
|
|
|
|
|
|
|
* While high-level abstractions are provided, you must understand how Mitogen
|
|
|
|
* While high-level abstractions are provided, you must understand how Mitogen
|
|
|
|
works before depending on it. Mitogen interacts with many aspects of the
|
|
|
|
works before depending on it. Mitogen interacts with many aspects of the
|
|
|
|
operating system, network, SSH, sudo, sockets, TTYs, shell, Python runtime,
|
|
|
|
operating system, threading, SSH, sudo, sockets, TTYs, shell, Python runtime,
|
|
|
|
and timing and ordering uncertainty introduced through interaction with the
|
|
|
|
and timing and ordering uncertainty introduced through interaction with the
|
|
|
|
network and OS scheduling.
|
|
|
|
network, GIL and OS scheduling.
|
|
|
|
|
|
|
|
|
|
|
|
Knowledge of this domain is typically gained through painful years of failed
|
|
|
|
Knowledge of this domain is typically attained through painful years of
|
|
|
|
attempts hacking system-level programs, and learning through continual
|
|
|
|
failed attempts hacking system-level programs, and learning through continual
|
|
|
|
suffering how to debug the messes left behind. If you feel you lack resources
|
|
|
|
suffering how to debug the atrocities left behind. If you feel you lack
|
|
|
|
to diagnose problems independently, Mitogen is not appropriate, prefer a
|
|
|
|
resources or willpower to diagnose problems independently, Mitogen is not
|
|
|
|
higher level solution instead. Bug reports failing this expectation risk
|
|
|
|
appropriate, prefer a higher level solution instead. Bug reports failing this
|
|
|
|
unfavourable treatment.
|
|
|
|
expectation risk uncharitable treatment.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Broker And Router
|
|
|
|
Broker And Router
|
|
|
@ -51,12 +51,11 @@ children from a private thread, while in children, it is additionally
|
|
|
|
responsible for ensuring robust destruction if communication with the master
|
|
|
|
responsible for ensuring robust destruction if communication with the master
|
|
|
|
is lost.
|
|
|
|
is lost.
|
|
|
|
|
|
|
|
|
|
|
|
:py:class:`Router` is responsible for receiving messages and either dispatching
|
|
|
|
:py:class:`Router` is responsible for receiving messages and dispatching them
|
|
|
|
them to a callback from the broker thread (registered by
|
|
|
|
to a callback from the broker thread (registered by :py:meth:`add_handler()
|
|
|
|
:py:meth:`add_handler() <mitogen.core.Router.add_handler>`), or forwarding them
|
|
|
|
<mitogen.core.Router.add_handler>`), or forwarding them to a :py:class:`Stream
|
|
|
|
to a :py:class:`Stream <mitogen.core.Stream>`. See :ref:`routing` for an
|
|
|
|
<mitogen.core.Stream>`. See :ref:`routing` for an in-depth description.
|
|
|
|
in-depth description. :py:class:`Router` also doubles as the entry point to
|
|
|
|
:py:class:`Router` also doubles as the entry point to Mitogen's public API:
|
|
|
|
Mitogen's public API:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
.. code-block:: python
|
|
|
|
.. code-block:: python
|
|
|
|
|
|
|
|
|
|
|
@ -87,16 +86,95 @@ If your program cannot live beneath :py:func:`mitogen.utils.run_with_router` on
|
|
|
|
the stack, you must must arrange for :py:meth:`Broker.shutdown` to be called
|
|
|
|
the stack, you must must arrange for :py:meth:`Broker.shutdown` to be called
|
|
|
|
anywhere the main thread may exit.
|
|
|
|
anywhere the main thread may exit.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
.. note::
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
You may construct as many routers and brokers in a process as desired, and
|
|
|
|
|
|
|
|
use the same broker for multiple routers, however in the usual case only
|
|
|
|
|
|
|
|
one broker and router need exist.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
It may be useful to construct multiple routers when a service is dealing
|
|
|
|
|
|
|
|
with separate trust domains, for example, manipulating infrastructure
|
|
|
|
|
|
|
|
belonging to separate customers or separate projects.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
It may be useful to construct multiple brokers when a service is dealing
|
|
|
|
|
|
|
|
with sets of children with differing lifetimes. For example, a subscription
|
|
|
|
|
|
|
|
service where non-payment results in termination for one customer.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Enable Logging
|
|
|
|
|
|
|
|
--------------
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Mitogen makes heavy use of the :py:mod:`logging` package, both for child
|
|
|
|
|
|
|
|
``stdio`` redirection, and soft errors and warnings that may be generated.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
You should always configure the :py:mod:`logging` package in any program that
|
|
|
|
|
|
|
|
integrates Mitogen. If your program does not otherwise use the
|
|
|
|
|
|
|
|
:py:mod:`logging` package, a basic configuration can be performed by calling
|
|
|
|
|
|
|
|
:py:func:`mitogen.utils.log_to_file`:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
.. code-block:: python
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
>>> import mitogen.utils
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
# Errors, warnings, and child stdio will be written to stderr.
|
|
|
|
|
|
|
|
>>> mitogen.utils.log_to_file()
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Additionally, if your program has :py:const:`logging.DEBUG` as the default
|
|
|
|
|
|
|
|
logging level, you may wish to update its configuration to restrict the
|
|
|
|
|
|
|
|
``mitogen`` logger to :py:const:`logging.INFO`, otherwise vast amounts of
|
|
|
|
|
|
|
|
useless output will be generated by default.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Creating A Context
|
|
|
|
Creating A Context
|
|
|
|
------------------
|
|
|
|
------------------
|
|
|
|
|
|
|
|
|
|
|
|
Contexts simply refer to external Python programs over which your program has
|
|
|
|
Contexts are simply external Python programs over which your program has
|
|
|
|
control. They can be created as subprocesses on the local machine, in another
|
|
|
|
control. They can be created as subprocesses on the local machine, in another
|
|
|
|
user account via ``sudo``, on a remote machine via ``ssh``, and in any
|
|
|
|
user account via `sudo`, on a remote machine via `ssh`, or any recursive
|
|
|
|
recursive combination of the above.
|
|
|
|
combination of the above.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Now a :py:class:`Router` exists, our first :py:class:`contexts <Context>` can
|
|
|
|
|
|
|
|
be created. To demonstrate basic functionality, we will start with some
|
|
|
|
|
|
|
|
:py:meth:`local() <Router.local>` contexts created as subprocesses:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
.. code-block:: python
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
>>> local = router.local()
|
|
|
|
|
|
|
|
>>> local_with_name = router.local(remote_name='i-have-a-name')
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Examination of the system process list with the ``pstree`` utility reveals the
|
|
|
|
|
|
|
|
resulting process hierarchy::
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| | \-+= 27660 dmw python
|
|
|
|
|
|
|
|
| | |--- 27661 dmw mitogen:dmw@Eldil.local:27660
|
|
|
|
|
|
|
|
| | \--- 27663 dmw mitogen:i-have-a-name
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Both contexts are visible as subprocesses of the interactive Python
|
|
|
|
|
|
|
|
interpreter, with their ``argv[0]`` including a description of their identity.
|
|
|
|
|
|
|
|
To aid systems administrators in identifying errant software running on their
|
|
|
|
|
|
|
|
machines, the default `remote_name` includes the location of the program that
|
|
|
|
|
|
|
|
started the context, however as shown, this can be overridden.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
.. note::
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Presently contexts are constructed in a blocking manner on the thread that
|
|
|
|
|
|
|
|
invoked the :ref:`context factory <context-factories>`. In a future
|
|
|
|
|
|
|
|
release, the factory will instead return immediately, and construction will
|
|
|
|
|
|
|
|
happen asynchronously on the broker thread.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Calling A Function
|
|
|
|
|
|
|
|
------------------
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Now that we have some contexts created, it is time to execute some code in
|
|
|
|
|
|
|
|
them.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Recursion
|
|
|
|
|
|
|
|
---------
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Let's try something a little more complex:
|
|
|
|
|
|
|
|
|
|
|
|
Now a :py:class:`Router` exists, our first :py:class:`Context` can be created.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
.. _serialization-rules:
|
|
|
|
.. _serialization-rules:
|
|
|
|