Getting Started =============== .. warning:: This section is incomplete. Liability Waiver ---------------- .. image:: images/pandora.jpg :align: right Before proceeding, it is critical you understand what you're involving yourself and possibly your team and its successors with: * Constructing the most fundamental class, :py:class:`Broker `, causes a new thread to be spawned, exposing a huge class of difficult to analyse behaviours that Python software generally does not suffer from. While every effort is made to hide this complexity, you should expect threading-related encounters during development, and crucially, years after your program reached production. See :ref:`troubleshooting` for more information. * While high-level abstractions are provided, you must understand how Mitogen works before depending on it. Mitogen interacts with many aspects of the operating system, threading, SSH, sudo, sockets, TTYs, shell, Python runtime, and timing and ordering uncertainty introduced through interaction with the network, GIL and OS scheduling. Knowledge of this domain is typically attained through painful years of failed attempts hacking system-level programs, and learning through continual suffering how to debug the atrocities left behind. If you feel you lack resources or willpower to diagnose problems independently, Mitogen is not appropriate, prefer a higher level solution instead. Bug reports failing this expectation risk uncharitable treatment. Broker And Router ----------------- .. image:: images/layout.png .. currentmodule:: mitogen.master Execution starts when your program constructs a :py:class:`Broker` and associated :py:class:`Router`. The broker is responsible for multiplexing IO to children from a private thread, while in children, it is additionally responsible for ensuring robust destruction if communication with the master is lost. :py:class:`Router` is responsible for receiving messages and dispatching them to a callback from the broker thread (registered by :py:meth:`add_handler() `), or forwarding them to a :py:class:`Stream `. See :ref:`routing` for an in-depth description. :py:class:`Router` also doubles as the entry point to Mitogen's public API: .. code-block:: python >>> import mitogen.master >>> broker = mitogen.master.Broker() >>> router = mitogen.master.Router(broker) >>> try: ... # Your code here. ... pass ... finally: ... broker.shutdown() As Python will not stop if threads still exist after the main thread exits, :py:meth:`Broker.shutdown` must be called reliably at exit. Helpers are provided by :py:mod:`mitogen.utils` to ensure :py:class:`Broker` is reliably destroyed: .. code-block:: python def do_mitogen_stuff(router): # Your code here. mitogen.utils.run_with_router(do_mitogen_stuff) If your program cannot live beneath :py:func:`mitogen.utils.run_with_router` on the stack, you must must arrange for :py:meth:`Broker.shutdown` to be called anywhere the main thread may exit. .. note:: You may construct as many routers and brokers in a process as desired, and use the same broker for multiple routers, however in the usual case only one broker and router need exist. It may be useful to construct multiple routers when a service is dealing with separate trust domains, for example, manipulating infrastructure belonging to separate customers or separate projects. It may be useful to construct multiple brokers when a service is dealing with sets of children with differing lifetimes. For example, a subscription service where non-payment results in termination for one customer. Enable Logging -------------- Mitogen makes heavy use of the :py:mod:`logging` package, both for child ``stdio`` redirection, and soft errors and warnings that may be generated. You should always configure the :py:mod:`logging` package in any program that integrates Mitogen. If your program does not otherwise use the :py:mod:`logging` package, a basic configuration can be performed by calling :py:func:`mitogen.utils.log_to_file`: .. code-block:: python >>> import mitogen.utils # Errors, warnings, and child stdio will be written to stderr. >>> mitogen.utils.log_to_file() Additionally, if your program has :py:const:`logging.DEBUG` as the default logging level, you may wish to update its configuration to restrict the ``mitogen`` logger to :py:const:`logging.INFO`, otherwise vast amounts of output will be generated by default. Creating A Context ------------------ Contexts are simply external Python programs over which your program has control. They can be created as subprocesses on the local machine, in another user account via `sudo`, on a remote machine via `ssh`, or any recursive combination of the above. Now a :py:class:`Router` exists, our first :py:class:`contexts ` can be created. To demonstrate basic functionality, we will start with some :py:meth:`local() ` contexts created as subprocesses: .. code-block:: python >>> local = router.local() >>> local_with_name = router.local(remote_name='i-have-a-name') Examination of the system process list with the ``pstree`` utility reveals the resulting process hierarchy:: | | \-+= 27660 dmw python | | |--- 27661 dmw mitogen:dmw@Eldil.local:27660 | | \--- 27663 dmw mitogen:i-have-a-name Both contexts are visible as subprocesses of the interactive Python interpreter, with their ``argv[0]`` including a description of their identity. To aid systems administrators in identifying errant software running on their machines, the default `remote_name` includes the location of the program that started the context, however as shown, this can be overridden. .. note:: Presently contexts are constructed in a blocking manner on the thread that invoked the :ref:`context factory `. In a future release, the factory will instead return immediately, and construction will happen asynchronously on the broker thread. Calling A Function ------------------ Now that we have some contexts created, it is time to execute some code in them. Recursion --------- Let's try something a little more complex: .. _serialization-rules: RPC Serialization Rules ----------------------- The following built-in types may be used as parameters or return values in remote procedure calls: * bool * bytearray * bytes * dict * int * list * long * str * tuple * unicode User-defined types may not be used, except for: * :py:class:`mitogen.core.CallError` * :py:class:`mitogen.core.Context` * :py:class:`mitogen.core._DEAD` .. _troubleshooting: Troubleshooting --------------- .. warning:: This section is incomplete. A typical example is a hang due to your application's main thread exitting perhaps due to an unhandled exception, without first arranging for any :py:class:`Broker ` to be shut down gracefully. Another example would be your main thread hanging indefinitely because a bug in Mitogen fails to notice an event (such as RPC completion) your thread is waiting for will never complete. Solving this kind of hang is a work in progress. router.enable_debug()