mitogen/docs/getting_started.rst

Getting Started
===============

.. warning::

    This section is incomplete.


Liability Waiver
----------------

.. image:: images/pandora.jpg
    :align: right

Before proceeding, it is critical you understand what you're involving yourself
and possibly your team and its successors with:

* Constructing the most fundamental class, :py:class:`Broker
  <mitogen.master.Broker>`, causes a new thread to be spawned, exposing a huge
  class of difficult to analyse behaviours that Python software generally does
  not suffer from.

  While every effort is made to hide this complexity, you should expect
  threading-related encounters during development, and crucially, years after
  your program reached production. See :ref:`troubleshooting` for more
  information.

* While high-level abstractions are provided, you must understand how Mitogen
  works before depending on it. Mitogen interacts with many aspects of the
  operating system, threading, SSH, sudo, sockets, TTYs, shell, Python runtime,
  and timing and ordering uncertainty introduced through interaction with the
  network, GIL and OS scheduling.

  Knowledge of this domain is typically attained through painful years of
  failed attempts hacking system-level programs, and learning through continual
  suffering how to debug the atrocities left behind. If you feel you lack
  resources or willpower to diagnose problems independently, Mitogen is not
  appropriate, prefer a higher level solution instead. Bug reports failing this
  expectation risk uncharitable treatment.


Broker And Router
-----------------

.. image:: images/layout.png
.. currentmodule:: mitogen.master

Execution starts when your program constructs a :py:class:`Broker` and
associated :py:class:`Router`. The broker is responsible for multiplexing IO to
children from a private thread, while in children, it is additionally
responsible for ensuring robust destruction if communication with the master
is lost.

:py:class:`Router` is responsible for receiving messages and dispatching them
to a callback from the broker thread (registered by :py:meth:`add_handler()
<mitogen.core.Router.add_handler>`), or forwarding them to a :py:class:`Stream
<mitogen.core.Stream>`. See :ref:`routing` for an in-depth description.
:py:class:`Router` also doubles as the entry point to Mitogen's public API:

.. code-block:: python

    >>> import mitogen.master

    >>> broker = mitogen.master.Broker()
    >>> router = mitogen.master.Router(broker)

    >>> try:
    ...     # Your code here.
    ...     pass
    ... finally:
    ...     broker.shutdown()

As Python will not stop if threads still exist after the main thread exits,
:py:meth:`Broker.shutdown` must be called reliably at exit. Helpers are
provided by :py:mod:`mitogen.utils` to ensure :py:class:`Broker` is reliably
destroyed:

.. code-block:: python

    def do_mitogen_stuff(router):
        # Your code here.

    mitogen.utils.run_with_router(do_mitogen_stuff)

If your program cannot live beneath :py:func:`mitogen.utils.run_with_router` on
the stack, you must must arrange for :py:meth:`Broker.shutdown` to be called
anywhere the main thread may exit.

.. note::

    You may construct as many routers and brokers in a process as desired, and
    use the same broker for multiple routers, however in the usual case only
    one broker and router need exist.

    It may be useful to construct multiple routers when a service is dealing
    with separate trust domains, for example, manipulating infrastructure
    belonging to separate customers or separate projects.

    It may be useful to construct multiple brokers when a service is dealing
    with sets of children with differing lifetimes. For example, a subscription
    service where non-payment results in termination for one customer.


Enable Logging
--------------

Mitogen makes heavy use of the :py:mod:`logging` package, both for child
``stdio`` redirection, and soft errors and warnings that may be generated.

You should always configure the :py:mod:`logging` package in any program that
integrates Mitogen. If your program does not otherwise use the
:py:mod:`logging` package, a basic configuration can be performed by calling
:py:func:`mitogen.utils.log_to_file`:

.. code-block:: python

    >>> import mitogen.utils

    # Errors, warnings, and child stdio will be written to stderr.
    >>> mitogen.utils.log_to_file()

Additionally, if your program has :py:const:`logging.DEBUG` as the default
logging level, you may wish to update its configuration to restrict the
``mitogen`` logger to :py:const:`logging.INFO`, otherwise vast amounts of
output will be generated by default.


Creating A Context
------------------

Contexts are simply external Python programs over which your program has
control, and can execute code within. They can be created as subprocesses on
the local machine, in another user account via `sudo`, on a remote machine via
`ssh`, or any recursive combination of the above.

Now a :py:class:`Router` exists, our first :py:class:`contexts <Context>` can
be created. To demonstrate basic functionality, we will start with some
:py:meth:`local() <Router.local>` contexts created as subprocesses:

.. code-block:: python

    >>> local = router.local()
    >>> local_with_name = router.local(remote_name='i-have-a-name')

Examination of the system process list with the ``pstree`` utility reveals the
resulting process hierarchy::

    | |   \-+= 27660 dmw python
    | |     |--- 27661 dmw mitogen:dmw@Eldil.local:27660
    | |     \--- 27663 dmw mitogen:i-have-a-name

Both contexts are visible as subprocesses of the interactive Python
interpreter, with their ``argv[0]`` including a description of their identity.
To aid systems administrators in identifying errant software running on their
machines, the default `remote_name` includes the location of the program that
started the context, however as shown, this can be overridden.

.. note::

    Presently contexts are constructed in a blocking manner on the thread that
    invoked the :ref:`context factory <context-factories>`. In a future
    release, the factory will instead return immediately, and construction will
    happen asynchronously on the broker thread.


Calling A Function
------------------

Now that we have some contexts created, it is time to execute some code in
them.


Recursion
---------

Let's try something a little more complex:


.. _serialization-rules:

RPC Serialization Rules
-----------------------

The following built-in types may be used as parameters or return values in
remote procedure calls:

* :class:`bool`
* :class:`bytearray`
* :func:`bytes`
* :class:`dict`
* :class:`int`
* :func:`list`
* :class:`long`
* :class:`str`
* :func:`tuple`
* :func:`unicode`

User-defined types may not be used, except for:

* :py:class:`mitogen.core.CallError`
* :py:class:`mitogen.core.Context`
* :py:class:`mitogen.core._DEAD`


.. _troubleshooting:

Troubleshooting
---------------

.. warning::

    This section is incomplete.

A typical example is a hang due to your application's main thread exitting
perhaps due to an unhandled exception, without first arranging for any
:py:class:`Broker <mitogen.master.Broker>` to be shut down gracefully.

Another example would be your main thread hanging indefinitely because a bug
in Mitogen fails to notice an event (such as RPC completion) your thread is
waiting for will never complete. Solving this kind of hang is a work in
progress.

router.enable_debug()