mitogen/docs/getting_started.rst

Getting Started
===============

.. warning::

    This section is incomplete.


Liability Waiver
----------------

Before proceeding, it is critical you understand what you're involving yourself
and possibly your team and its successors with:

.. image:: images/pandora.jpg
    :align: right

* Constructing the most fundamental class, :py:class:`Broker
  <mitogen.master.Broker>`, causes a new thread to be spawned, exposing a huge
  class of difficult to analyse behaviours that Python software generally does
  not suffer from.

  While every effort is made to hide this complexity, you should expect
  threading-related encounters during development, and crucially, years after
  your program reached production. See :ref:`troubleshooting` for more
  information.

* While high-level abstractions are provided, they are only a convenience, you
  must still understand :ref:`how Mitogen works <howitworks>` before depending
  on it. Mitogen interacts with many aspects of the operating system,
  threading, SSH, sudo, sockets, TTYs, shell, Python runtime, and timing and
  ordering uncertainty introduced through interaction with the network, GIL and
  OS scheduling.

  Knowledge of this domain is typically attained through painful years of
  failed attempts hacking system-level programs, and learning through continual
  suffering how to debug the atrocities left behind. If you feel you lack
  resources or willpower to diagnose problems independently, Mitogen is not
  appropriate, prefer a higher level solution instead.


Broker And Router
-----------------

.. image:: images/layout.png
.. currentmodule:: mitogen.core

Execution starts when your program constructs a :py:class:`Broker` and
associated :py:class:`Router`. The broker is responsible for multiplexing IO to
children from a private thread, while in children, it is additionally
responsible for ensuring robust destruction if communication with the master
is lost.

:py:class:`Router` is responsible for receiving messages and dispatching them
to a callback from the broker thread (registered by :py:meth:`add_handler()
<mitogen.core.Router.add_handler>`), or forwarding them to a :py:class:`Stream
<mitogen.core.Stream>`. See :ref:`routing` for an in-depth description.
:py:class:`Router` also doubles as the entry point to Mitogen's public API::

    >>> import mitogen.master

    >>> broker = mitogen.master.Broker()
    >>> router = mitogen.master.Router(broker)

    >>> try:
    ...     # Your code here.
    ...     pass
    ... finally:
    ...     broker.shutdown()

As Python will not stop if threads still exist after the main thread exits,
:py:meth:`Broker.shutdown` must be called reliably at exit. Helpers are
provided by :py:mod:`mitogen.utils` to ensure :py:class:`Broker` is reliably
destroyed::

    def do_mitogen_stuff(router):
        # Your code here.

    mitogen.utils.run_with_router(do_mitogen_stuff)

If your program cannot live beneath :py:func:`mitogen.utils.run_with_router` on
the stack, you must arrange for :py:meth:`Broker.shutdown` to be called
anywhere the main thread may exit.


Enable Logging
--------------

Mitogen makes heavy use of the :py:mod:`logging` package, both for child
``stdio`` redirection, and soft errors and warnings that may be generated.

You should always configure the :py:mod:`logging` package in any program that
integrates Mitogen. If your program does not otherwise use the
:py:mod:`logging` package, a basic configuration can be performed by calling
:py:func:`mitogen.utils.log_to_file`::

    >>> import mitogen.utils

    # Errors, warnings, and child stdio will be written to stderr.
    >>> mitogen.utils.log_to_file()

Additionally, if your program has :py:const:`logging.DEBUG` as the default
logging level, you may wish to update its configuration to restrict the
``mitogen`` logger to :py:const:`logging.INFO`, otherwise vast amounts of
output will be generated by default.


.. _logging-env-vars:

Logging Environment Variables
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

``MITOGEN_LOG_LEVEL``
    Overrides the :py:mod:`logging` package log level set by any call to
    :py:func:`mitogen.utils.log_to_file`. Defaults to ``INFO``.

``MITOGEN_LOG_USEC``
    If present, forces microsecond-level timestamps for any call to
    :py:func:`mitogen.utils.log_to_file`.

``MITOGEN_LOG_IO``
    If present, forces IO logging for any call to
    :py:func:`mitogen.utils.log_to_file`. IO logging produces extremely verbose
    logs of any IO interaction, which is useful when debugging deadlocks.


Creating A Context
------------------

Contexts are simply external Python programs over which your program has
control, and can execute code within. They can be created as subprocesses on
the local machine, in another user account via `sudo`, on a remote machine via
`ssh`, or any recursive combination of the above.

Now a :py:class:`Router` exists, our first :py:class:`contexts <Context>` can
be created. To demonstrate basic functionality, we will start with some
:py:meth:`local() <Router.local>` contexts created as subprocesses::

    >>> local = router.local()
    >>> local_with_name = router.local(remote_name='i-have-a-name')

Examination of the system process list with the ``pstree`` utility reveals the
resulting process hierarchy::

    | |   \-+= 27660 dmw python
    | |     |--- 27661 dmw mitogen:dmw@Eldil.local:27660
    | |     \--- 27663 dmw mitogen:i-have-a-name

Both contexts are visible as subprocesses of the interactive Python
interpreter, with their ``argv[0]`` including a description of their identity.
To aid systems administrators in identifying errant software running on their
machines, the default `remote_name` includes the location of the program that
started the context, however as shown, this can be overridden.

.. note::

    Presently contexts are constructed in a blocking manner on the thread that
    invoked the :ref:`context factory <context-factories>`. In a future
    release, the factory will instead return immediately, and construction will
    happen asynchronously on the broker thread.


Calling A Function
------------------

.. currentmodule:: mitogen.master

Now that some contexts exist, it is time to execute code in them. Any regular
function, static method, or class method reachable directly from module scope
may be used, including built-in functions such as :func:`time.time`.

The :py:meth:`Context.call` method is used to execute a function and block the
caller until the return value is available or an exception is raised::

    >>> import time
    >>> import os

    >>> # Returns the current time.
    >>> print 'Time in remote context:', local.call(time.time)

    >>> try:
    ...     # Raises OSError.
    ...     local.call(os.chdir, '/nonexistent')
    ... except mitogen.core.CallError, e:
    ...     print 'Call failed:', str(e)

It is a simple wrapper around the more flexible :meth:`Context.call_async`,
which immediately returns a :class:`Receiver <mitogen.core.Receiver>` wired up
to receive the return value instead. A receiver may simply be discarded, kept
around indefinitely without ever reading its result, or used to wait on the
results from several calls. Here :meth:`get() <mitogen.core.Receiver.get>`
is called to block the thread until the result arrives::

    >>> call = local.call_async(time.time)
    >>> msg = call.get()
    >>> print msg.unpickle()
    1507292737.75547


Running User Functions
----------------------

So far we have used the interactive interpreter to call some standard library
functions, but if since source code typed at the interpreter cannot be
recovered, Mitogen is unable to execute functions defined in this way.

We must therefore continue by writing our code as a script::

    # first-script.py
    import mitogen.utils

    def my_first_function():
        print 'Hello from remote context!'
        return 123

    def main(router):
        local = router.local()
        print local.call(my_first_function)

    if __name__ == '__main__':
        mitogen.utils.log_to_file(main)
        mitogen.utils.run_with_router(main)

Let's try running it:

.. code-block:: bash

    $ python first-script.py
    19:11:32 I mitogen.ctx.local.32466: stdout: Hello from remote context!
    123


Waiting On Multiple Calls
-------------------------

Using :meth:`Context.call_async` it is possible to start multiple function
calls then sleep waiting for responses as they are available. This makes it
trivial to run tasks in parallel across processes (including remote processes)
without the need for writing asynchronous code::

    hostnames = ['host1', 'host2', 'host3', 'host4']
    contexts = [router.ssh(hostname=hn) for hn in hostnames]
    calls = [context.call(my_func) for context in contexts]

    for recv, (msg, data) in mitogen.master.Select(calls):
        print 'Reply from %s: %s' % (recv.context, data)



Recovering Mitogen Object References In Children
------------------------------------------------

::

    @mitogen.core.takes_econtext
    def func1(a, b, econtext):
        ...

    @mitogen.core.takes_router
    def func2(a, b, router):
        ...



Recursion
---------

Let's try something a little more complex:


.. _serialization-rules:

RPC Serialization Rules
-----------------------

The following built-in types may be used as parameters or return values in
remote procedure calls:

* :class:`bool`
* :class:`bytearray`
* :func:`bytes`
* :class:`dict`
* :class:`int`
* :func:`list`
* :class:`long`
* :class:`str`
* :func:`tuple`
* :func:`unicode`

User-defined types may not be used, except for:

* :py:class:`mitogen.core.CallError`
* :py:class:`mitogen.core.Context`
* :py:class:`mitogen.core._DEAD`

Subclasses of built-in types must be undecorated using
:py:func:`mitogen.utils.cast`.


.. _troubleshooting:

Troubleshooting
---------------

.. warning::

    This section is incomplete.

A typical example is a hang due to your application's main thread exitting
perhaps due to an unhandled exception, without first arranging for any
:py:class:`Broker <mitogen.master.Broker>` to be shut down gracefully.

Another example would be your main thread hanging indefinitely because a bug
in Mitogen fails to notice an event (such as RPC completion) your thread is
waiting for will never complete. Solving this kind of hang is a work in
progress.

router.enable_debug()