diff --git a/docs/api.rst b/docs/api.rst index 67c61dee..008ae247 100644 --- a/docs/api.rst +++ b/docs/api.rst @@ -35,9 +35,9 @@ mitogen.core Decorator that marks a function or class method to automatically receive a kwarg named `econtext`, referencing the - :py:class:`mitogen.core.ExternalContext` active in the context in which the + :class:`mitogen.core.ExternalContext` active in the context in which the function is being invoked in. The decorator is only meaningful when the - function is invoked via :py:data:`CALL_FUNCTION + function is invoked via :data:`CALL_FUNCTION `. When the function is invoked directly, `econtext` must still be passed to @@ -47,10 +47,10 @@ mitogen.core .. decorator:: takes_router Decorator that marks a function or class method to automatically receive a - kwarg named `router`, referencing the :py:class:`mitogen.core.Router` + kwarg named `router`, referencing the :class:`mitogen.core.Router` active in the context in which the function is being invoked in. The decorator is only meaningful when the function is invoked via - :py:data:`CALL_FUNCTION `. + :data:`CALL_FUNCTION `. When the function is invoked directly, `router` must still be passed to it explicitly. @@ -94,18 +94,18 @@ Message Class .. attribute:: router - The :py:class:`mitogen.core.Router` responsible for routing the - message. This is :py:data:`None` for locally originated messages. + The :class:`mitogen.core.Router` responsible for routing the + message. This is :data:`None` for locally originated messages. .. attribute:: receiver - The :py:class:`mitogen.core.Receiver` over which the message was last - received. Part of the :py:class:`mitogen.select.Select` interface. - Defaults to :py:data:`None`. + The :class:`mitogen.core.Receiver` over which the message was last + received. Part of the :class:`mitogen.select.Select` interface. + Defaults to :data:`None`. .. attribute:: dst_id - Integer target context ID. :py:class:`mitogen.core.Router` delivers + Integer target context ID. :class:`mitogen.core.Router` delivers messages locally when their :attr:`dst_id` matches :data:`mitogen.context_id`, otherwise they are routed up or downstream. @@ -117,12 +117,12 @@ Message Class .. attribute:: auth_id The context ID under whose authority the message is acting. See - :py:ref:`source-verification`. + :ref:`source-verification`. .. attribute:: handle Integer target handle in the destination context. This is one of the - :py:ref:`standard-handles`, or a dynamically generated handle used to + :ref:`standard-handles`, or a dynamically generated handle used to receive a one-time reply, such as the return value of a function call. .. attribute:: reply_to @@ -143,12 +143,12 @@ Message Class .. py:method:: __init__ (\**kwargs) - Construct a message from from the supplied `kwargs`. :py:attr:`src_id` - and :py:attr:`auth_id` are always set to :py:data:`mitogen.context_id`. + Construct a message from from the supplied `kwargs`. :attr:`src_id` + and :attr:`auth_id` are always set to :data:`mitogen.context_id`. .. py:classmethod:: pickled (obj, \**kwargs) - Construct a pickled message, setting :py:attr:`data` to the + Construct a pickled message, setting :attr:`data` to the serialization of `obj`, and setting remaining fields using `kwargs`. :returns: @@ -156,10 +156,10 @@ Message Class .. method:: unpickle (throw=True) - Unpickle :py:attr:`data`, optionally raising any exceptions present. + Unpickle :attr:`data`, optionally raising any exceptions present. :param bool throw: - If :py:data:`True`, raise exceptions, otherwise it is the caller's + If :data:`True`, raise exceptions, otherwise it is the caller's responsibility. :raises mitogen.core.CallError: @@ -169,8 +169,8 @@ Message Class .. method:: reply (obj, router=None, \**kwargs) - Compose a reply to this message and send it using :py:attr:`router`, or - `router` is :py:attr:`router` is :data:`None`. + Compose a reply to this message and send it using :attr:`router`, or + `router` is :attr:`router` is :data:`None`. :param obj: Either a :class:`Message`, or an object to be serialized in order @@ -190,8 +190,8 @@ Router Class .. class:: Router Route messages between parent and child contexts, and invoke handlers - defined on our parent context. :py:meth:`Router.route() ` straddles - the :py:class:`Broker ` and user threads, it is safe + defined on our parent context. :meth:`Router.route() ` straddles + the :class:`Broker ` and user threads, it is safe to call anywhere. **Note:** This is the somewhat limited core version of the Router class @@ -217,7 +217,7 @@ Router Class .. method:: stream_by_id (dst_id) - Return the :py:class:`mitogen.core.Stream` that should be used to + Return the :class:`mitogen.core.Stream` that should be used to communicate with `dst_id`. If a specific route for `dst_id` is not known, a reference to the parent context's stream is returned. @@ -260,24 +260,24 @@ Router Class :param function policy: Function invoked as `policy(msg, stream)` where `msg` is a - :py:class:`mitogen.core.Message` about to be delivered, and - `stream` is the :py:class:`mitogen.core.Stream` on which it was - received. The function must return :py:data:`True`, otherwise an + :class:`mitogen.core.Message` about to be delivered, and + `stream` is the :class:`mitogen.core.Stream` on which it was + received. The function must return :data:`True`, otherwise an error is logged and delivery is refused. Two built-in policy functions exist: - * :py:func:`mitogen.core.has_parent_authority`: requires the + * :func:`mitogen.core.has_parent_authority`: requires the message arrived from a parent context, or a context acting with a parent context's authority (``auth_id``). - * :py:func:`mitogen.parent.is_immediate_child`: requires the + * :func:`mitogen.parent.is_immediate_child`: requires the message arrived from an immediately connected child, for use in messaging patterns where either something becomes buggy or insecure by permitting indirect upstream communication. In case of refusal, and the message's ``reply_to`` field is - nonzero, a :py:class:`mitogen.core.CallError` is delivered to the + nonzero, a :class:`mitogen.core.CallError` is delivered to the sender indicating refusal occurred. :return: @@ -297,7 +297,7 @@ Router Class destination is the local context, then arrange for it to be dispatched using the local handlers. - This is a lower overhead version of :py:meth:`route` that may only be + This is a lower overhead version of :meth:`route` that may only be called from the I/O multiplexer thread. :param mitogen.core.Stream stream: @@ -308,11 +308,11 @@ Router Class .. method:: route(msg) - Arrange for the :py:class:`Message` `msg` to be delivered to its + Arrange for the :class:`Message` `msg` to be delivered to its destination using any relevant downstream context, or if none is found, by forwarding the message upstream towards the master context. If `msg` is destined for the local context, it is dispatched using the handles - registered with :py:meth:`add_handler`. + registered with :meth:`add_handler`. This may be called from any thread. @@ -321,7 +321,7 @@ Router Class .. class:: Router (broker=None) - Extend :py:class:`mitogen.core.Router` with functionality useful to + Extend :class:`mitogen.core.Router` with functionality useful to masters, and child contexts who later become masters. Currently when this class is required, the target context's router is upgraded at runtime. @@ -334,16 +334,16 @@ Router Class customers or projects. :param mitogen.master.Broker broker: - :py:class:`Broker` instance to use. If not specified, a private - :py:class:`Broker` is created. + :class:`Broker` instance to use. If not specified, a private + :class:`Broker` is created. .. attribute:: profiling When :data:`True`, cause the broker thread and any subsequent broker and main threads existing in any child to write ``/tmp/mitogen.stats...log`` containing a - :py:mod:`cProfile` dump on graceful exit. Must be set prior to - construction of any :py:class:`Broker`, e.g. via: + :mod:`cProfile` dump on graceful exit. Must be set prior to + construction of any :class:`Broker`, e.g. via: .. code:: @@ -378,7 +378,7 @@ Router Class and router, and responds to function calls identically to children created using other methods. - For long-lived processes, :py:meth:`local` is always better as it + For long-lived processes, :meth:`local` is always better as it guarantees a pristine interpreter state that inherited little from the parent. Forking should only be used in performance-sensitive scenarios where short-lived children must be spawned to isolate potentially buggy @@ -420,10 +420,10 @@ Router Class immediate copy-on-write to large portions of the process heap. * Locks held in the parent causing random deadlocks in the child, such - as when another thread emits a log entry via the :py:mod:`logging` - package concurrent to another thread calling :py:meth:`fork`. + as when another thread emits a log entry via the :mod:`logging` + package concurrent to another thread calling :meth:`fork`. - * Objects existing in Thread-Local Storage of every non-:py:meth:`fork` + * Objects existing in Thread-Local Storage of every non-:meth:`fork` thread becoming permanently inaccessible, and never having their object destructors called, including TLS usage by native extension code, triggering many new variants of all the issues above. @@ -434,16 +434,16 @@ Router Class case, children continually reuse the same state due to repeatedly forking from a static parent. - :py:meth:`fork` cleans up Mitogen-internal objects, in addition to - locks held by the :py:mod:`logging` package, reseeds - :py:func:`random.random`, and the OpenSSL PRNG via - :py:func:`ssl.RAND_add`, but only if the :py:mod:`ssl` module is + :meth:`fork` cleans up Mitogen-internal objects, in addition to + locks held by the :mod:`logging` package, reseeds + :func:`random.random`, and the OpenSSL PRNG via + :func:`ssl.RAND_add`, but only if the :mod:`ssl` module is already loaded. You must arrange for your program's state, including any third party packages in use, to be cleaned up by specifying an `on_fork` function. The associated stream implementation is - :py:class:`mitogen.fork.Stream`. + :class:`mitogen.fork.Stream`. :param function on_fork: Function invoked as `on_fork()` from within the child process. This @@ -459,19 +459,19 @@ Router Class serialization. :param Context via: - Same as the `via` parameter for :py:meth:`local`. + Same as the `via` parameter for :meth:`local`. :param bool debug: - Same as the `debug` parameter for :py:meth:`local`. + Same as the `debug` parameter for :meth:`local`. :param bool profiling: - Same as the `profiling` parameter for :py:meth:`local`. + Same as the `profiling` parameter for :meth:`local`. .. method:: local (remote_name=None, python_path=None, debug=False, connect_timeout=None, profiling=False, via=None) Construct a context on the local machine as a subprocess of the current process. The associated stream implementation is - :py:class:`mitogen.master.Stream`. + :class:`mitogen.master.Stream`. :param str remote_name: The ``argv[0]`` suffix for the new process. If `remote_name` is @@ -493,9 +493,9 @@ Router Class another tool, such as ``["/usr/bin/env", "python"]``. :param bool debug: - If :data:`True`, arrange for debug logging (:py:meth:`enable_debug`) to + If :data:`True`, arrange for debug logging (:meth:`enable_debug`) to be enabled in the new context. Automatically :data:`True` when - :py:meth:`enable_debug` has been called, but may be used + :meth:`enable_debug` has been called, but may be used selectively otherwise. :param bool unidirectional: @@ -510,14 +510,14 @@ Router Class healthy. Defaults to 30 seconds. :param bool profiling: - If :data:`True`, arrange for profiling (:py:data:`profiling`) to be + If :data:`True`, arrange for profiling (:data:`profiling`) to be enabled in the new context. Automatically :data:`True` when - :py:data:`profiling` is :data:`True`, but may be used selectively + :data:`profiling` is :data:`True`, but may be used selectively otherwise. :param mitogen.core.Context via: If not :data:`None`, arrange for construction to occur via RPCs - made to the context `via`, and for :py:data:`ADD_ROUTE + made to the context `via`, and for :data:`ADD_ROUTE ` messages to be generated as appropriate. .. code-block:: python @@ -534,7 +534,7 @@ Router Class The ``doas`` process is started in a newly allocated pseudo-terminal, and supports typing interactive passwords. - Accepts all parameters accepted by :py:meth:`local`, in addition to: + Accepts all parameters accepted by :meth:`local`, in addition to: :param str username: Username to use, defaults to ``root``. @@ -559,7 +559,7 @@ Router Class temporary new Docker container using the ``docker`` program. One of `container` or `image` must be specified. - Accepts all parameters accepted by :py:meth:`local`, in addition to: + Accepts all parameters accepted by :meth:`local`, in addition to: :param str container: Existing container to connect to. Defaults to :data:`None`. @@ -578,7 +578,7 @@ Router Class Construct a context on the local machine within a FreeBSD jail using the ``jexec`` program. - Accepts all parameters accepted by :py:meth:`local`, in addition to: + Accepts all parameters accepted by :meth:`local`, in addition to: :param str container: Existing container to connect to. Defaults to :data:`None`. @@ -594,7 +594,7 @@ Router Class Construct a context on the local machine within an LXC classic container using the ``lxc-attach`` program. - Accepts all parameters accepted by :py:meth:`local`, in addition to: + Accepts all parameters accepted by :meth:`local`, in addition to: :param str container: Existing container to connect to. Defaults to :data:`None`. @@ -608,7 +608,7 @@ Router Class Construct a context on the local machine within a LXD container using the ``lxc`` program. - Accepts all parameters accepted by :py:meth:`local`, in addition to: + Accepts all parameters accepted by :meth:`local`, in addition to: :param str container: Existing container to connect to. Defaults to :data:`None`. @@ -656,7 +656,7 @@ Router Class ``su`` process is started in a newly allocated pseudo-terminal, and supports typing interactive passwords. - Accepts all parameters accepted by :py:meth:`local`, in addition to: + Accepts all parameters accepted by :meth:`local`, in addition to: :param str username: Username to pass to ``su``, defaults to ``root``. @@ -683,7 +683,7 @@ Router Class The ``sudo`` process is started in a newly allocated pseudo-terminal, and supports typing interactive passwords. - Accepts all parameters accepted by :py:meth:`local`, in addition to: + Accepts all parameters accepted by :meth:`local`, in addition to: :param str username: Username to pass to sudo as the ``-u`` parameter, defaults to @@ -694,28 +694,32 @@ Router Class :param str password: The password to use if/when sudo requests it. Depending on the sudo configuration, this is either the current account password or the - target account password. :py:class:`mitogen.sudo.PasswordError` + target account password. :class:`mitogen.sudo.PasswordError` will be raised if sudo requests a password but none is provided. :param bool set_home: - If :py:data:`True`, request ``sudo`` set the ``HOME`` environment + If :data:`True`, request ``sudo`` set the ``HOME`` environment variable to match the target UNIX account. :param bool preserve_env: - If :py:data:`True`, request ``sudo`` to preserve the environment of + If :data:`True`, request ``sudo`` to preserve the environment of the parent process. :param list sudo_args: - Arguments in the style of :py:data:`sys.argv` that would normally + Arguments in the style of :data:`sys.argv` that would normally be passed to ``sudo``. The arguments are parsed in-process to set equivalent parameters. Re-parsing ensures unsupported options cause - :py:class:`mitogen.core.StreamError` to be raised, and that + :class:`mitogen.core.StreamError` to be raised, and that attributes of the stream match the actual behaviour of ``sudo``. .. method:: ssh (hostname, username=None, ssh_path=None, port=None, check_host_keys='enforce', password=None, identity_file=None, identities_only=True, compression=True, \**kwargs) - Construct a remote context over a ``ssh`` invocation. The ``ssh`` - process is started in a newly allocated pseudo-terminal, and supports - typing interactive passwords. + Construct a remote context over an OpenSSH ``ssh`` invocation. - Accepts all parameters accepted by :py:meth:`local`, in addition to: + By default, the ``ssh`` process is started in a newly allocated + pseudo-terminal to support typing interactive passwords, however when + making many connections, this may be disabled by specifying + `batch_mode=True`, as most operating systems have a conservative upper + limit on the number of pseudo-terminals that may exist. + + Accepts all parameters accepted by :meth:`local`, in addition to: :param str username: The SSH username; default is unspecified, which causes SSH to pick @@ -737,7 +741,7 @@ Router Class unknown hosts cause a connection failure. :param str password: Password to type if/when ``ssh`` requests it. If not specified and - a password is requested, :py:class:`mitogen.ssh.PasswordError` is + a password is requested, :class:`mitogen.ssh.PasswordError` is raised. :param str identity_file: Path to an SSH private key file to use for authentication. Default @@ -755,12 +759,17 @@ Router Class present in ``~/.ssh``. This ensures authentication attempts only occur using the supplied password or SSH key. :param bool compression: - If :py:data:`True`, enable ``ssh`` compression support. Compression + If :data:`True`, enable ``ssh`` compression support. Compression has a minimal effect on the size of modules transmitted, as they are already compressed, however it has a large effect on every remaining message in the otherwise uncompressed stream protocol, such as function call arguments and return values. - :parama int ssh_debug_level: + :param bool batch_mode: + If :data:`True`, disable pseudo-terminal allocation. When + :data:`True`, the `password=` parameter may not be used, since no + PTY exists to enter the password, and the `check_host_keys=` + parameter may not be set to `accept`. + :param int ssh_debug_level: Optional integer `0..3` indicating the SSH client debug level. :raises mitogen.ssh.PasswordError: A password was requested but none was specified, or the specified @@ -808,12 +817,12 @@ Context Class The message. :returns: - :py:class:`mitogen.core.Receiver` configured to receive any replies + :class:`mitogen.core.Receiver` configured to receive any replies sent to the message's `reply_to` handle. .. method:: send_await (msg, deadline=None) - As with :py:meth:`send_async`, but expect a single reply + As with :meth:`send_async`, but expect a single reply (`persist=False`) delivered within `deadline` seconds. :param mitogen.core.Message msg: @@ -830,7 +839,7 @@ Context Class .. class:: Context - Extend :py:class:`mitogen.core.Router` with functionality useful to + Extend :class:`mitogen.core.Router` with functionality useful to masters, and child contexts who later become parents. Currently when this class is required, the target context's router is upgraded at runtime. @@ -843,7 +852,7 @@ Context Class terminate a hung context using this method. This will be fixed shortly. :param bool wait: - If :py:data:`True`, block the calling thread until the context has + If :data:`True`, block the calling thread until the context has completely terminated. :returns: If `wait` is :data:`False`, returns a :class:`mitogen.core.Latch` @@ -888,7 +897,7 @@ Context Class Function keyword arguments, if any. See :ref:`serialization-rules` for permitted types. :returns: - :py:class:`mitogen.core.Receiver` configured to receive the result + :class:`mitogen.core.Receiver` configured to receive the result of the invocation: .. code-block:: python @@ -903,11 +912,11 @@ Context Class Asynchronous calls may be dispatched in parallel to multiple contexts and consumed as they complete using - :py:class:`mitogen.select.Select`. + :class:`mitogen.select.Select`. .. method:: call (fn, \*args, \*\*kwargs) - Equivalent to :py:meth:`call_async(fn, \*args, \**kwargs).get().unpickle() + Equivalent to :meth:`call_async(fn, \*args, \**kwargs).get().unpickle() `. :returns: @@ -935,7 +944,7 @@ Receiver Class Receivers are used to wait for pickled responses from another context to be sent to a handle registered in this context. A receiver may be single-use - (as in the case of :py:meth:`mitogen.parent.Context.call_async`) or + (as in the case of :meth:`mitogen.parent.Context.call_async`) or multiple use. :param mitogen.core.Router router: @@ -959,12 +968,12 @@ Receiver Class If not :data:`None`, a reference to a function invoked as `notify(receiver)` when a new message is delivered to this receiver. - Used by :py:class:`mitogen.select.Select` to implement waiting on + Used by :class:`mitogen.select.Select` to implement waiting on multiple receivers. .. py:method:: to_sender () - Return a :py:class:`mitogen.core.Sender` configured to deliver messages + Return a :class:`mitogen.core.Sender` configured to deliver messages to this receiver. Since a Sender can be serialized, this makes it convenient to pass `(context_id, handle)` pairs around:: @@ -981,15 +990,15 @@ Receiver Class .. py:method:: empty () - Return :data:`True` if calling :py:meth:`get` would block. + Return :data:`True` if calling :meth:`get` would block. - As with :py:class:`Queue.Queue`, :data:`True` may be returned even - though a subsequent call to :py:meth:`get` will succeed, since a - message may be posted at any moment between :py:meth:`empty` and - :py:meth:`get`. + As with :class:`Queue.Queue`, :data:`True` may be returned even + though a subsequent call to :meth:`get` will succeed, since a + message may be posted at any moment between :meth:`empty` and + :meth:`get`. - :py:meth:`empty` is only useful to avoid a race while installing - :py:attr:`notify`: + :meth:`empty` is only useful to avoid a race while installing + :attr:`notify`: .. code-block:: python @@ -1003,8 +1012,8 @@ Receiver Class .. py:method:: close () - Cause :py:class:`mitogen.core.ChannelError` to be raised in any thread - waiting in :py:meth:`get` on this receiver. + Cause :class:`mitogen.core.ChannelError` to be raised in any thread + waiting in :meth:`get` on this receiver. .. py:method:: get (timeout=None) @@ -1022,17 +1031,17 @@ Receiver Class :returns: `(msg, data)` tuple, where `msg` is the - :py:class:`mitogen.core.Message` that was received, and `data` is + :class:`mitogen.core.Message` that was received, and `data` is its unpickled data part. .. py:method:: get_data (timeout=None) - Like :py:meth:`get`, except only return the data part. + Like :meth:`get`, except only return the data part. .. py:method:: __iter__ () Block and yield `(msg, data)` pairs delivered to this receiver until - :py:class:`mitogen.core.ChannelError` is raised. + :class:`mitogen.core.ChannelError` is raised. Sender Class @@ -1043,10 +1052,10 @@ Sender Class .. class:: Sender (context, dst_handle) Senders are used to send pickled messages to a handle in another context, - it is the inverse of :py:class:`mitogen.core.Sender`. + it is the inverse of :class:`mitogen.core.Sender`. Senders may be serialized, making them convenient to wire up data flows. - See :py:meth:`mitogen.core.Receiver.to_sender` for more information. + See :meth:`mitogen.core.Receiver.to_sender` for more information. :param mitogen.core.Context context: Context to send messages to. @@ -1055,7 +1064,7 @@ Sender Class .. py:method:: close () - Send a dead message to the remote end, causing :py:meth:`ChannelError` + Send a dead message to the remote end, causing :meth:`ChannelError` to be raised in any waiting thread. .. py:method:: send (data) @@ -1074,11 +1083,11 @@ Select Class Support scatter/gather asynchronous calls and waiting on multiple receivers, channels, and sub-Selects. Accepts a sequence of - :py:class:`mitogen.core.Receiver` or :py:class:`mitogen.select.Select` + :class:`mitogen.core.Receiver` or :class:`mitogen.select.Select` instances and returns the first value posted to any receiver or select. If `oneshot` is :data:`True`, then remove each receiver as it yields a - result; since :py:meth:`__iter__` terminates once the final receiver is + result; since :meth:`__iter__` terminates once the final receiver is removed, this makes it convenient to respond to calls made in parallel: .. code-block:: python @@ -1093,7 +1102,7 @@ Select Class # Iteration ends when last Receiver yields a result. print('Received total %s from %s receivers' % (total, len(recvs))) - :py:class:`Select` may drive a long-running scheduler: + :class:`Select` may drive a long-running scheduler: .. code-block:: python @@ -1104,7 +1113,7 @@ Select Class for context, workfunc in get_new_work(): select.add(context.call_async(workfunc)) - :py:class:`Select` may be nested: + :class:`Select` may be nested: .. code-block:: python @@ -1122,11 +1131,11 @@ Select Class .. py:classmethod:: all (it) - Take an iterable of receivers and retrieve a :py:class:`Message` from + Take an iterable of receivers and retrieve a :class:`Message` from each, returning the result of calling `msg.unpickle()` on each in turn. Results are returned in the order they arrived. - This is sugar for handling batch :py:class:`Context.call_async` + This is sugar for handling batch :class:`Context.call_async` invocations: .. code-block:: python @@ -1146,28 +1155,28 @@ Select Class for context in contexts) Result processing happens concurrently to new results arriving, so - :py:meth:`all` should always be faster. + :meth:`all` should always be faster. .. py:method:: get (timeout=None, block=True) Fetch the next available value from any receiver, or raise - :py:class:`mitogen.core.TimeoutError` if no value is available within + :class:`mitogen.core.TimeoutError` if no value is available within `timeout` seconds. - On success, the message's :py:attr:`receiver + On success, the message's :attr:`receiver ` attribute is set to the receiver. :param float timeout: Timeout in seconds. :param bool block: - If :py:data:`False`, immediately raise - :py:class:`mitogen.core.TimeoutError` if the select is empty. + If :data:`False`, immediately raise + :class:`mitogen.core.TimeoutError` if the select is empty. :return: - :py:class:`mitogen.core.Message` + :class:`mitogen.core.Message` :raises mitogen.core.TimeoutError: Timeout was reached. :raises mitogen.core.LatchError: - :py:meth:`close` has been called, and the underlying latch is no + :meth:`close` has been called, and the underlying latch is no longer valid. .. py:method:: __bool__ () @@ -1178,8 +1187,8 @@ Select Class Remove the select's notifier function from each registered receiver, mark the associated latch as closed, and cause any thread currently - sleeping in :py:meth:`get` to be woken with - :py:class:`mitogen.core.LatchError`. + sleeping in :meth:`get` to be woken with + :class:`mitogen.core.LatchError`. This is necessary to prevent memory leaks in long-running receivers. It is called automatically when the Python :keyword:`with` statement is @@ -1187,35 +1196,35 @@ Select Class .. py:method:: empty () - Return :data:`True` if calling :py:meth:`get` would block. + Return :data:`True` if calling :meth:`get` would block. - As with :py:class:`Queue.Queue`, :data:`True` may be returned even - though a subsequent call to :py:meth:`get` will succeed, since a - message may be posted at any moment between :py:meth:`empty` and - :py:meth:`get`. + As with :class:`Queue.Queue`, :data:`True` may be returned even + though a subsequent call to :meth:`get` will succeed, since a + message may be posted at any moment between :meth:`empty` and + :meth:`get`. - :py:meth:`empty` may return :data:`False` even when :py:meth:`get` + :meth:`empty` may return :data:`False` even when :meth:`get` would block if another thread has drained a receiver added to this select. This can be avoided by only consuming each receiver from a single thread. .. py:method:: __iter__ (self) - Yield the result of :py:meth:`get` until no receivers remain in the + Yield the result of :meth:`get` until no receivers remain in the select, either because `oneshot` is :data:`True`, or each receiver was - explicitly removed via :py:meth:`remove`. + explicitly removed via :meth:`remove`. .. py:method:: add (recv) - Add the :py:class:`mitogen.core.Receiver` or - :py:class:`mitogen.core.Channel` `recv` to the select. + Add the :class:`mitogen.core.Receiver` or + :class:`mitogen.core.Channel` `recv` to the select. .. py:method:: remove (recv) - Remove the :py:class:`mitogen.core.Receiver` or - :py:class:`mitogen.core.Channel` `recv` from the select. Note that if - the receiver has notified prior to :py:meth:`remove`, then it will - still be returned by a subsequent :py:meth:`get`. This may change in a + Remove the :class:`mitogen.core.Receiver` or + :class:`mitogen.core.Channel` `recv` from the select. Note that if + the receiver has notified prior to :meth:`remove`, then it will + still be returned by a subsequent :meth:`get`. This may change in a future version. @@ -1226,7 +1235,7 @@ Channel Class .. class:: Channel (router, context, dst_handle, handle=None) - A channel inherits from :py:class:`mitogen.core.Sender` and + A channel inherits from :class:`mitogen.core.Sender` and `mitogen.core.Receiver` to provide bidirectional functionality. Since all handles aren't known until after both ends are constructed, for @@ -1248,8 +1257,8 @@ Broker Class .. attribute:: shutdown_timeout = 3.0 - Seconds grace to allow :py:class:`streams ` to shutdown - gracefully before force-disconnecting them during :py:meth:`shutdown`. + Seconds grace to allow :class:`streams ` to shutdown + gracefully before force-disconnecting them during :meth:`shutdown`. .. method:: defer (func, \*args, \*kwargs) @@ -1259,26 +1268,26 @@ Broker Class .. method:: start_receive (stream) - Mark the :py:attr:`receive_side ` on `stream` as + Mark the :attr:`receive_side ` on `stream` as ready for reading. Safe to call from any thread. When the associated file descriptor becomes ready for reading, - :py:meth:`BasicStream.on_receive` will be called. + :meth:`BasicStream.on_receive` will be called. .. method:: stop_receive (stream) - Mark the :py:attr:`receive_side ` on `stream` as + Mark the :attr:`receive_side ` on `stream` as not ready for reading. Safe to call from any thread. .. method:: _start_transmit (stream) - Mark the :py:attr:`transmit_side ` on `stream` as + Mark the :attr:`transmit_side ` on `stream` as ready for writing. Must only be called from the Broker thread. When the associated file descriptor becomes ready for writing, - :py:meth:`BasicStream.on_transmit` will be called. + :meth:`BasicStream.on_transmit` will be called. .. method:: stop_receive (stream) - Mark the :py:attr:`transmit_side ` on `stream` as + Mark the :attr:`transmit_side ` on `stream` as not ready for writing. Safe to call from any thread. .. method:: shutdown @@ -1288,12 +1297,12 @@ Broker Class .. method:: join Wait for the broker to stop, expected to be called after - :py:meth:`shutdown`. + :meth:`shutdown`. .. method:: keep_alive - Return :data:`True` if any reader's :py:attr:`Side.keep_alive` - attribute is :data:`True`, or any :py:class:`Context` is still + Return :data:`True` if any reader's :attr:`Side.keep_alive` + attribute is :data:`True`, or any :class:`Context` is still registered that is not the master. Used to delay shutdown while some important work is in progress (e.g. log draining). @@ -1301,11 +1310,11 @@ Broker Class .. method:: _broker_main - Handle events until :py:meth:`shutdown`. On shutdown, invoke - :py:meth:`Stream.on_shutdown` for every active stream, then allow up to - :py:attr:`shutdown_timeout` seconds for the streams to unregister + Handle events until :meth:`shutdown`. On shutdown, invoke + :meth:`Stream.on_shutdown` for every active stream, then allow up to + :attr:`shutdown_timeout` seconds for the streams to unregister themselves before forcefully calling - :py:meth:`Stream.on_disconnect`. + :meth:`Stream.on_disconnect`. .. currentmodule:: mitogen.master @@ -1321,7 +1330,7 @@ Broker Class :param bool install_watcher: If :data:`True`, an additional thread is started to monitor the - lifetime of the main thread, triggering :py:meth:`shutdown` + lifetime of the main thread, triggering :meth:`shutdown` automatically in case the user forgets to call it, or their code crashed. @@ -1332,8 +1341,8 @@ Broker Class .. attribute:: shutdown_timeout = 5.0 - Seconds grace to allow :py:class:`streams ` to shutdown - gracefully before force-disconnecting them during :py:meth:`shutdown`. + Seconds grace to allow :class:`streams ` to shutdown + gracefully before force-disconnecting them during :meth:`shutdown`. Utility Functions @@ -1349,7 +1358,7 @@ A random assortment of utility functions useful on masters and children. Many tools love to subclass built-in types in order to implement useful functionality, such as annotating the safety of a Unicode string, or adding additional methods to a dict. However, cPickle loves to preserve those - subtypes during serialization, resulting in CallError during :py:meth:`call + subtypes during serialization, resulting in CallError during :meth:`call ` in the target when it tries to deserialize the data. @@ -1369,12 +1378,12 @@ A random assortment of utility functions useful on masters and children. Remove all entries mentioning ``site-packages`` or ``Extras`` from the system path. Used primarily for testing on OS X within a virtualenv, where - OS X bundles some ancient version of the :py:mod:`six` module. + OS X bundles some ancient version of the :mod:`six` module. .. currentmodule:: mitogen.utils .. function:: log_to_file (path=None, io=False, level='INFO') - Install a new :py:class:`logging.Handler` writing applications logs to the + Install a new :class:`logging.Handler` writing applications logs to the filesystem. Useful when debugging slave IO problems. Parameters to this function may be overridden at runtime using environment @@ -1382,14 +1391,14 @@ A random assortment of utility functions useful on masters and children. :param str path: If not :data:`None`, a filesystem path to write logs to. Otherwise, - logs are written to :py:data:`sys.stderr`. + logs are written to :data:`sys.stderr`. :param bool io: If :data:`True`, include extremely verbose IO logs in the output. Useful for debugging hangs, less useful for debugging application code. :param str level: - Name of the :py:mod:`logging` package constant that is the minimum + Name of the :mod:`logging` package constant that is the minimum level to log at. Useful levels are ``DEBUG``, ``INFO``, ``WARNING``, and ``ERROR``. @@ -1397,7 +1406,7 @@ A random assortment of utility functions useful on masters and children. .. function:: run_with_router(func, \*args, \**kwargs) Arrange for `func(router, \*args, \**kwargs)` to run with a temporary - :py:class:`mitogen.master.Router`, ensuring the Router and Broker are + :class:`mitogen.master.Router`, ensuring the Router and Broker are correctly shut down during normal or exceptional return. :returns: @@ -1406,7 +1415,7 @@ A random assortment of utility functions useful on masters and children. .. currentmodule:: mitogen.utils .. decorator:: with_router - Decorator version of :py:func:`run_with_router`. Example: + Decorator version of :func:`run_with_router`. Example: .. code-block:: python diff --git a/mitogen/ssh.py b/mitogen/ssh.py index 38e12531..2650d32f 100644 --- a/mitogen/ssh.py +++ b/mitogen/ssh.py @@ -111,7 +111,6 @@ class HostKeyError(mitogen.core.StreamError): class Stream(mitogen.parent.Stream): - create_child = staticmethod(mitogen.parent.hybrid_tty_create_child) child_is_immediate_subprocess = False #: Default to whatever is available as 'python' on the remote machine, @@ -121,8 +120,8 @@ class Stream(mitogen.parent.Stream): #: Number of -v invocations to pass on command line. ssh_debug_level = 0 - #: Once connected, points to the corresponding TtyLogStream, allowing it to - #: be disconnected at the same time this stream is being torn down. + #: If batch_mode=False, points to the corresponding TtyLogStream, allowing + #: it to be disconnected at the same time this stream is being torn down. tty_stream = None #: The path to the SSH binary. @@ -137,15 +136,27 @@ class Stream(mitogen.parent.Stream): ssh_args = None check_host_keys_msg = 'check_host_keys= must be set to accept, enforce or ignore' + batch_mode_check_host_keys_msg = ( + 'check_host_keys cannot be set to "accept" when batch mode is ' + 'enabled, since batch mode disables PTY allocation.' + ) + batch_mode_password_msg = ( + 'A password cannot be set when batch mode is enabled, ' + 'since batch mode disables PTY allocation.' + ) def construct(self, hostname, username=None, ssh_path=None, port=None, check_host_keys='enforce', password=None, identity_file=None, compression=True, ssh_args=None, keepalive_enabled=True, - keepalive_count=3, keepalive_interval=15, + keepalive_count=3, keepalive_interval=15, batch_mode=False, identities_only=True, ssh_debug_level=None, **kwargs): super(Stream, self).construct(**kwargs) if check_host_keys not in ('accept', 'enforce', 'ignore'): raise ValueError(self.check_host_keys_msg) + if check_host_keys == 'accept' and batch_mode: + raise ValueError(self.batch_mode_check_host_keys_msg) + if password is not None and batch_mode: + raise ValueError(self.batch_mode_password_msg) self.hostname = hostname self.username = username @@ -158,6 +169,14 @@ class Stream(mitogen.parent.Stream): self.keepalive_enabled = keepalive_enabled self.keepalive_count = keepalive_count self.keepalive_interval = keepalive_interval + self.batch_mode = batch_mode + if self.batch_mode: + self.create_child = mitogen.parent.create_child + self.create_child_args = { + 'merge_stdio': True, + } + else: + self.create_child = mitogen.parent.hybrid_tty_create_child if ssh_path: self.ssh_path = ssh_path if ssh_args: @@ -166,7 +185,8 @@ class Stream(mitogen.parent.Stream): self.ssh_debug_level = ssh_debug_level def on_disconnect(self, broker): - self.tty_stream.on_disconnect(broker) + if self.tty_stream is not None: + self.tty_stream.on_disconnect(broker) super(Stream, self).on_disconnect(broker) def get_boot_command(self): @@ -193,10 +213,15 @@ class Stream(mitogen.parent.Stream): '-o', 'ServerAliveInterval %s' % (self.keepalive_interval,), '-o', 'ServerAliveCountMax %s' % (self.keepalive_count,), ] + if self.batch_mode: + bits += ['-o', 'BatchMode yes'] if self.check_host_keys == 'enforce': bits += ['-o', 'StrictHostKeyChecking yes'] if self.check_host_keys == 'accept': - bits += ['-o', 'StrictHostKeyChecking ask'] + if self.batch_mode: + bits += ['-o', 'StrictHostKeyChecking no'] + else: + bits += ['-o', 'StrictHostKeyChecking ask'] elif self.check_host_keys == 'ignore': bits += [ '-o', 'StrictHostKeyChecking no', @@ -240,19 +265,23 @@ class Stream(mitogen.parent.Stream): # with ours. raise HostKeyError(self.hostkey_config_msg) + def _ec0_received(self): + if self.tty_stream is not None: + self._router.broker.start_receive(self.tty_stream) + return super(Stream, self)._ec0_received() + def _connect_bootstrap(self, extra_fd): - self.tty_stream = mitogen.parent.TtyLogStream(extra_fd, self) + fds = [self.receive_side.fd] + if extra_fd is not None: + self.tty_stream = mitogen.parent.TtyLogStream(extra_fd, self) + fds.append(extra_fd) - password_sent = False - it = mitogen.parent.iter_read( - fds=[self.receive_side.fd, extra_fd], - deadline=self.connect_deadline - ) + it = mitogen.parent.iter_read(fds=fds, deadline=self.connect_deadline) + password_sent = False for buf, partial in filter_debug(self, it): LOG.debug('%r: received %r', self, buf) if buf.endswith(self.EC0_MARKER): - self._router.broker.start_receive(self.tty_stream) self._ec0_received() return elif HOSTKEY_REQ_PROMPT in buf.lower(): diff --git a/tests/data/fakessh.py b/tests/data/fakessh.py index 08a5da3e..415425af 100755 --- a/tests/data/fakessh.py +++ b/tests/data/fakessh.py @@ -6,6 +6,45 @@ import shlex import subprocess import sys + +HOST_KEY_ASK_MSG = """ +The authenticity of host '[91.121.165.123]:9122 ([91.121.165.123]:9122)' can't be established. +ECDSA key fingerprint is SHA256:JvfPvazZzQ9/CUdKN7tiYlNZtDRdEgDsYVIzOgPrsR4. +Are you sure you want to continue connecting (yes/no)? +""".strip('\n') + +HOST_KEY_STRICT_MSG = """Host key verification failed.\n""" + + +def tty(msg): + fp = open('/dev/tty', 'w', 0) + fp.write(msg) + fp.close() + + +def stderr(msg): + fp = open('/dev/stderr', 'w', 0) + fp.write(msg) + fp.close() + + +def confirm(msg): + tty(msg) + fp = open('/dev/tty', 'r', 0) + try: + return fp.readline() + finally: + fp.close() + + +if os.getenv('FAKESSH_MODE') == 'ask': + assert 'y\n' == confirm(HOST_KEY_ASK_MSG) + +if os.getenv('FAKESSH_MODE') == 'strict': + stderr(HOST_KEY_STRICT_MSG) + sys.exit(255) + + parser = optparse.OptionParser() parser.add_option('--user', '-l', action='store') parser.add_option('-o', dest='options', action='append') diff --git a/tests/ssh_test.py b/tests/ssh_test.py index a514c8ea..a305ec70 100644 --- a/tests/ssh_test.py +++ b/tests/ssh_test.py @@ -1,3 +1,4 @@ +import os import sys import mitogen @@ -13,9 +14,9 @@ import plain_old_module class FakeSshTest(testlib.RouterMixin, unittest2.TestCase): def test_okay(self): context = self.router.ssh( - hostname='hostname', - username='mitogen__has_sudo', - ssh_path=testlib.data_path('fakessh.py'), + hostname='hostname', + username='mitogen__has_sudo', + ssh_path=testlib.data_path('fakessh.py'), ) #context.call(mitogen.utils.log_to_file, '/tmp/log') #context.call(mitogen.utils.disable_site_packages) @@ -123,6 +124,86 @@ class BannerTest(testlib.DockerMixin, unittest2.TestCase): self.assertEquals(name, context.name) +class BatchModeTest(testlib.DockerMixin, testlib.TestCase): + stream_class = mitogen.ssh.Stream + # + # Test that: + # + # - batch_mode=false, host_key_checking=accept + # - batch_mode=false, host_key_checking=enforce + # - batch_mode=false, host_key_checking=ignore + # + # - batch_mode=true, host_key_checking=accept + # - batch_mode=true, host_key_checking=enforce + # - batch_mode=true, host_key_checking=ignore + # - batch_mode=true, password is not None + # + def fake_ssh(self, FAKESSH_MODE=None, **kwargs): + os.environ['FAKESSH_MODE'] = str(FAKESSH_MODE) + try: + return self.router.ssh( + hostname='hostname', + username='mitogen__has_sudo', + ssh_path=testlib.data_path('fakessh.py'), + **kwargs + ) + finally: + del os.environ['FAKESSH_MODE'] + + def test_false_accept(self): + # Should succeed. + self.fake_ssh(FAKESSH_MODE='ask', check_host_keys='accept') + + def test_false_enforce(self): + # Should succeed. + self.fake_ssh(check_host_keys='enforce') + + def test_false_ignore(self): + # Should succeed. + self.fake_ssh(check_host_keys='ignore') + + def test_false_password(self): + # Should succeed. + self.docker_ssh(username='mitogen__has_sudo_nopw', + password='has_sudo_nopw_password') + + def test_true_accept(self): + e = self.assertRaises(ValueError, + lambda: self.fake_ssh(check_host_keys='accept', batch_mode=True) + ) + self.assertEquals(e.args[0], + self.stream_class.batch_mode_check_host_keys_msg) + + def test_true_enforce(self): + e = self.assertRaises(mitogen.ssh.HostKeyError, + lambda: self.docker_ssh( + batch_mode=True, + check_host_keys='enforce', + ssh_args=['-o', 'UserKnownHostsFile /dev/null'], + ) + ) + self.assertEquals(e.args[0], self.stream_class.hostkey_failed_msg) + + def test_true_ignore(self): + e = self.assertRaises(mitogen.ssh.HostKeyError, + lambda: self.fake_ssh( + FAKESSH_MODE='strict', + batch_mode=True, + check_host_keys='ignore', + ) + ) + self.assertEquals(e.args[0], self.stream_class.hostkey_failed_msg) + + def test_true_password(self): + e = self.assertRaises(ValueError, + lambda: self.fake_ssh( + password='nope', + batch_mode=True, + ) + ) + self.assertEquals(e.args[0], self.stream_class.batch_mode_password_msg) + + if __name__ == '__main__': unittest2.main()