archive
/
mitogen
mirror of https://github.com/mitogen-hq/mitogen.git
1
0
Fork 0
Code Releases Activity

issue #337: ssh: support disabling PTY allocation

Browse Source 
`.ssh(batch_mode=True)`
pull/372/head
David Wilson 6 years ago
parent c4c6ae88a4
commit 5004207705
4 changed files with 327 additions and 169 deletions
Show Stats Download Patch File Download Diff File

315
docs/api.rst
Unescape Escape View File

@ -35,9 +35,9 @@ mitogen.core
Decorator that marks a function or class method to automatically receive a
kwarg named `econtext`, referencing the
:py:class:`mitogen.core.ExternalContext` active in the context in which the
:class:`mitogen.core.ExternalContext` active in the context in which the
function is being invoked in. The decorator is only meaningful when the
function is invoked via :py:data:`CALL_FUNCTION
function is invoked via :data:`CALL_FUNCTION
<mitogen.core.CALL_FUNCTION>`.
When the function is invoked directly, `econtext` must still be passed to
@ -47,10 +47,10 @@ mitogen.core
.. decorator:: takes_router
Decorator that marks a function or class method to automatically receive a
kwarg named `router`, referencing the :py:class:`mitogen.core.Router`
kwarg named `router`, referencing the :class:`mitogen.core.Router`
active in the context in which the function is being invoked in. The
decorator is only meaningful when the function is invoked via
:py:data:`CALL_FUNCTION <mitogen.core.CALL_FUNCTION>`.
:data:`CALL_FUNCTION <mitogen.core.CALL_FUNCTION>`.
When the function is invoked directly, `router` must still be passed to it
explicitly.
@ -94,18 +94,18 @@ Message Class
.. attribute:: router
The :py:class:`mitogen.core.Router` responsible for routing the
message. This is :py:data:`None` for locally originated messages.
The :class:`mitogen.core.Router` responsible for routing the
message. This is :data:`None` for locally originated messages.
.. attribute:: receiver
The :py:class:`mitogen.core.Receiver` over which the message was last
received. Part of the :py:class:`mitogen.select.Select` interface.
Defaults to :py:data:`None`.
The :class:`mitogen.core.Receiver` over which the message was last
received. Part of the :class:`mitogen.select.Select` interface.
Defaults to :data:`None`.
.. attribute:: dst_id
Integer target context ID. :py:class:`mitogen.core.Router` delivers
Integer target context ID. :class:`mitogen.core.Router` delivers
messages locally when their :attr:`dst_id` matches
:data:`mitogen.context_id`, otherwise they are routed up or downstream.
@ -117,12 +117,12 @@ Message Class
.. attribute:: auth_id
The context ID under whose authority the message is acting. See
:py:ref:`source-verification`.
:ref:`source-verification`.
.. attribute:: handle
Integer target handle in the destination context. This is one of the
:py:ref:`standard-handles`, or a dynamically generated handle used to
:ref:`standard-handles`, or a dynamically generated handle used to
receive a one-time reply, such as the return value of a function call.
.. attribute:: reply_to
@ -143,12 +143,12 @@ Message Class
.. py:method:: __init__ (\**kwargs)
Construct a message from from the supplied `kwargs`. :py:attr:`src_id`
and :py:attr:`auth_id` are always set to :py:data:`mitogen.context_id`.
Construct a message from from the supplied `kwargs`. :attr:`src_id`
and :attr:`auth_id` are always set to :data:`mitogen.context_id`.
.. py:classmethod:: pickled (obj, \**kwargs)
Construct a pickled message, setting :py:attr:`data` to the
Construct a pickled message, setting :attr:`data` to the
serialization of `obj`, and setting remaining fields using `kwargs`.
:returns:
@ -156,10 +156,10 @@ Message Class
.. method:: unpickle (throw=True)
Unpickle :py:attr:`data`, optionally raising any exceptions present.
Unpickle :attr:`data`, optionally raising any exceptions present.
:param bool throw:
If :py:data:`True`, raise exceptions, otherwise it is the caller's
If :data:`True`, raise exceptions, otherwise it is the caller's
responsibility.
:raises mitogen.core.CallError:
@ -169,8 +169,8 @@ Message Class
.. method:: reply (obj, router=None, \**kwargs)
Compose a reply to this message and send it using :py:attr:`router`, or
`router` is :py:attr:`router` is :data:`None`.
Compose a reply to this message and send it using :attr:`router`, or
`router` is :attr:`router` is :data:`None`.
:param obj:
Either a :class:`Message`, or an object to be serialized in order
@ -190,8 +190,8 @@ Router Class
.. class:: Router
Route messages between parent and child contexts, and invoke handlers
defined on our parent context. :py:meth:`Router.route() <route>` straddles
the :py:class:`Broker <mitogen.core.Broker>` and user threads, it is safe
defined on our parent context. :meth:`Router.route() <route>` straddles
the :class:`Broker <mitogen.core.Broker>` and user threads, it is safe
to call anywhere.
**Note:** This is the somewhat limited core version of the Router class
@ -217,7 +217,7 @@ Router Class
.. method:: stream_by_id (dst_id)
Return the :py:class:`mitogen.core.Stream` that should be used to
Return the :class:`mitogen.core.Stream` that should be used to
communicate with `dst_id`. If a specific route for `dst_id` is not
known, a reference to the parent context's stream is returned.
@ -260,24 +260,24 @@ Router Class
:param function policy:
Function invoked as `policy(msg, stream)` where `msg` is a
:py:class:`mitogen.core.Message` about to be delivered, and
`stream` is the :py:class:`mitogen.core.Stream` on which it was
received. The function must return :py:data:`True`, otherwise an
:class:`mitogen.core.Message` about to be delivered, and
`stream` is the :class:`mitogen.core.Stream` on which it was
received. The function must return :data:`True`, otherwise an
error is logged and delivery is refused.
Two built-in policy functions exist:
* :py:func:`mitogen.core.has_parent_authority`: requires the
* :func:`mitogen.core.has_parent_authority`: requires the
message arrived from a parent context, or a context acting with a
parent context's authority (``auth_id``).
* :py:func:`mitogen.parent.is_immediate_child`: requires the
* :func:`mitogen.parent.is_immediate_child`: requires the
message arrived from an immediately connected child, for use in
messaging patterns where either something becomes buggy or
insecure by permitting indirect upstream communication.
In case of refusal, and the message's ``reply_to`` field is
nonzero, a :py:class:`mitogen.core.CallError` is delivered to the
nonzero, a :class:`mitogen.core.CallError` is delivered to the
sender indicating refusal occurred.
:return:
@ -297,7 +297,7 @@ Router Class
destination is the local context, then arrange for it to be dispatched
using the local handlers.
This is a lower overhead version of :py:meth:`route` that may only be
This is a lower overhead version of :meth:`route` that may only be
called from the I/O multiplexer thread.
:param mitogen.core.Stream stream:
@ -308,11 +308,11 @@ Router Class
.. method:: route(msg)
Arrange for the :py:class:`Message` `msg` to be delivered to its
Arrange for the :class:`Message` `msg` to be delivered to its
destination using any relevant downstream context, or if none is found,
by forwarding the message upstream towards the master context. If `msg`
is destined for the local context, it is dispatched using the handles
registered with :py:meth:`add_handler`.
registered with :meth:`add_handler`.
This may be called from any thread.
@ -321,7 +321,7 @@ Router Class
.. class:: Router (broker=None)
Extend :py:class:`mitogen.core.Router` with functionality useful to
Extend :class:`mitogen.core.Router` with functionality useful to
masters, and child contexts who later become masters. Currently when this
class is required, the target context's router is upgraded at runtime.
@ -334,16 +334,16 @@ Router Class
customers or projects.
:param mitogen.master.Broker broker:
:py:class:`Broker` instance to use. If not specified, a private
:py:class:`Broker` is created.
:class:`Broker` instance to use. If not specified, a private
:class:`Broker` is created.
.. attribute:: profiling
When :data:`True`, cause the broker thread and any subsequent broker
and main threads existing in any child to write
``/tmp/mitogen.stats.<pid>.<thread_name>.log`` containing a
:py:mod:`cProfile` dump on graceful exit. Must be set prior to
construction of any :py:class:`Broker`, e.g. via:
:mod:`cProfile` dump on graceful exit. Must be set prior to
construction of any :class:`Broker`, e.g. via:
.. code::
@ -378,7 +378,7 @@ Router Class
and router, and responds to function calls identically to children
created using other methods.
For long-lived processes, :py:meth:`local` is always better as it
For long-lived processes, :meth:`local` is always better as it
guarantees a pristine interpreter state that inherited little from the
parent. Forking should only be used in performance-sensitive scenarios
where short-lived children must be spawned to isolate potentially buggy
@ -420,10 +420,10 @@ Router Class
immediate copy-on-write to large portions of the process heap.
* Locks held in the parent causing random deadlocks in the child, such
as when another thread emits a log entry via the :py:mod:`logging`
package concurrent to another thread calling :py:meth:`fork`.
as when another thread emits a log entry via the :mod:`logging`
package concurrent to another thread calling :meth:`fork`.
* Objects existing in Thread-Local Storage of every non-:py:meth:`fork`
* Objects existing in Thread-Local Storage of every non-:meth:`fork`
thread becoming permanently inaccessible, and never having their
object destructors called, including TLS usage by native extension
code, triggering many new variants of all the issues above.
@ -434,16 +434,16 @@ Router Class
case, children continually reuse the same state due to repeatedly
forking from a static parent.
:py:meth:`fork` cleans up Mitogen-internal objects, in addition to
locks held by the :py:mod:`logging` package, reseeds
:py:func:`random.random`, and the OpenSSL PRNG via
:py:func:`ssl.RAND_add`, but only if the :py:mod:`ssl` module is
:meth:`fork` cleans up Mitogen-internal objects, in addition to
locks held by the :mod:`logging` package, reseeds
:func:`random.random`, and the OpenSSL PRNG via
:func:`ssl.RAND_add`, but only if the :mod:`ssl` module is
already loaded. You must arrange for your program's state, including
any third party packages in use, to be cleaned up by specifying an
`on_fork` function.
The associated stream implementation is
:py:class:`mitogen.fork.Stream`.
:class:`mitogen.fork.Stream`.
:param function on_fork:
Function invoked as `on_fork()` from within the child process. This
@ -459,19 +459,19 @@ Router Class
serialization.
:param Context via:
Same as the `via` parameter for :py:meth:`local`.
Same as the `via` parameter for :meth:`local`.
:param bool debug:
Same as the `debug` parameter for :py:meth:`local`.
Same as the `debug` parameter for :meth:`local`.
:param bool profiling:
Same as the `profiling` parameter for :py:meth:`local`.
Same as the `profiling` parameter for :meth:`local`.
.. method:: local (remote_name=None, python_path=None, debug=False, connect_timeout=None, profiling=False, via=None)
Construct a context on the local machine as a subprocess of the current
process. The associated stream implementation is
:py:class:`mitogen.master.Stream`.
:class:`mitogen.master.Stream`.
:param str remote_name:
The ``argv[0]`` suffix for the new process. If `remote_name` is
@ -493,9 +493,9 @@ Router Class
another tool, such as ``["/usr/bin/env", "python"]``.
:param bool debug:
If :data:`True`, arrange for debug logging (:py:meth:`enable_debug`) to
If :data:`True`, arrange for debug logging (:meth:`enable_debug`) to
be enabled in the new context. Automatically :data:`True` when
:py:meth:`enable_debug` has been called, but may be used
:meth:`enable_debug` has been called, but may be used
selectively otherwise.
:param bool unidirectional:
@ -510,14 +510,14 @@ Router Class
healthy. Defaults to 30 seconds.
:param bool profiling:
If :data:`True`, arrange for profiling (:py:data:`profiling`) to be
If :data:`True`, arrange for profiling (:data:`profiling`) to be
enabled in the new context. Automatically :data:`True` when
:py:data:`profiling` is :data:`True`, but may be used selectively
:data:`profiling` is :data:`True`, but may be used selectively
otherwise.
:param mitogen.core.Context via:
If not :data:`None`, arrange for construction to occur via RPCs
made to the context `via`, and for :py:data:`ADD_ROUTE
made to the context `via`, and for :data:`ADD_ROUTE
<mitogen.core.ADD_ROUTE>` messages to be generated as appropriate.
.. code-block:: python
@ -534,7 +534,7 @@ Router Class
The ``doas`` process is started in a newly allocated pseudo-terminal,
and supports typing interactive passwords.
Accepts all parameters accepted by :py:meth:`local`, in addition to:
Accepts all parameters accepted by :meth:`local`, in addition to:
:param str username:
Username to use, defaults to ``root``.
@ -559,7 +559,7 @@ Router Class
temporary new Docker container using the ``docker`` program. One of
`container` or `image` must be specified.
Accepts all parameters accepted by :py:meth:`local`, in addition to:
Accepts all parameters accepted by :meth:`local`, in addition to:
:param str container:
Existing container to connect to. Defaults to :data:`None`.
@ -578,7 +578,7 @@ Router Class
Construct a context on the local machine within a FreeBSD jail using
the ``jexec`` program.
Accepts all parameters accepted by :py:meth:`local`, in addition to:
Accepts all parameters accepted by :meth:`local`, in addition to:
:param str container:
Existing container to connect to. Defaults to :data:`None`.
@ -594,7 +594,7 @@ Router Class
Construct a context on the local machine within an LXC classic
container using the ``lxc-attach`` program.
Accepts all parameters accepted by :py:meth:`local`, in addition to:
Accepts all parameters accepted by :meth:`local`, in addition to:
:param str container:
Existing container to connect to. Defaults to :data:`None`.
@ -608,7 +608,7 @@ Router Class
Construct a context on the local machine within a LXD container using
the ``lxc`` program.
Accepts all parameters accepted by :py:meth:`local`, in addition to:
Accepts all parameters accepted by :meth:`local`, in addition to:
:param str container:
Existing container to connect to. Defaults to :data:`None`.
@ -656,7 +656,7 @@ Router Class
``su`` process is started in a newly allocated pseudo-terminal, and
supports typing interactive passwords.
Accepts all parameters accepted by :py:meth:`local`, in addition to:
Accepts all parameters accepted by :meth:`local`, in addition to:
:param str username:
Username to pass to ``su``, defaults to ``root``.
@ -683,7 +683,7 @@ Router Class
The ``sudo`` process is started in a newly allocated pseudo-terminal,
and supports typing interactive passwords.
Accepts all parameters accepted by :py:meth:`local`, in addition to:
Accepts all parameters accepted by :meth:`local`, in addition to:
:param str username:
Username to pass to sudo as the ``-u`` parameter, defaults to
@ -694,28 +694,32 @@ Router Class
:param str password:
The password to use if/when sudo requests it. Depending on the sudo
configuration, this is either the current account password or the
target account password. :py:class:`mitogen.sudo.PasswordError`
target account password. :class:`mitogen.sudo.PasswordError`
will be raised if sudo requests a password but none is provided.
:param bool set_home:
If :py:data:`True`, request ``sudo`` set the ``HOME`` environment
If :data:`True`, request ``sudo`` set the ``HOME`` environment
variable to match the target UNIX account.
:param bool preserve_env:
If :py:data:`True`, request ``sudo`` to preserve the environment of
If :data:`True`, request ``sudo`` to preserve the environment of
the parent process.
:param list sudo_args:
Arguments in the style of :py:data:`sys.argv` that would normally
Arguments in the style of :data:`sys.argv` that would normally
be passed to ``sudo``. The arguments are parsed in-process to set
equivalent parameters. Re-parsing ensures unsupported options cause
:py:class:`mitogen.core.StreamError` to be raised, and that
:class:`mitogen.core.StreamError` to be raised, and that
attributes of the stream match the actual behaviour of ``sudo``.
.. method:: ssh (hostname, username=None, ssh_path=None, port=None, check_host_keys='enforce', password=None, identity_file=None, identities_only=True, compression=True, \**kwargs)
Construct a remote context over a ``ssh`` invocation. The ``ssh``
process is started in a newly allocated pseudo-terminal, and supports
typing interactive passwords.
Construct a remote context over an OpenSSH ``ssh`` invocation.
Accepts all parameters accepted by :py:meth:`local`, in addition to:
By default, the ``ssh`` process is started in a newly allocated
pseudo-terminal to support typing interactive passwords, however when
making many connections, this may be disabled by specifying
`batch_mode=True`, as most operating systems have a conservative upper
limit on the number of pseudo-terminals that may exist.
Accepts all parameters accepted by :meth:`local`, in addition to:
:param str username:
The SSH username; default is unspecified, which causes SSH to pick
@ -737,7 +741,7 @@ Router Class
unknown hosts cause a connection failure.
:param str password:
Password to type if/when ``ssh`` requests it. If not specified and
a password is requested, :py:class:`mitogen.ssh.PasswordError` is
a password is requested, :class:`mitogen.ssh.PasswordError` is
raised.
:param str identity_file:
Path to an SSH private key file to use for authentication. Default
@ -755,12 +759,17 @@ Router Class
present in ``~/.ssh``. This ensures authentication attempts only
occur using the supplied password or SSH key.
:param bool compression:
If :py:data:`True`, enable ``ssh`` compression support. Compression
If :data:`True`, enable ``ssh`` compression support. Compression
has a minimal effect on the size of modules transmitted, as they
are already compressed, however it has a large effect on every
remaining message in the otherwise uncompressed stream protocol,
such as function call arguments and return values.
:parama int ssh_debug_level:
:param bool batch_mode:
If :data:`True`, disable pseudo-terminal allocation. When
:data:`True`, the `password=` parameter may not be used, since no
PTY exists to enter the password, and the `check_host_keys=`
parameter may not be set to `accept`.
:param int ssh_debug_level:
Optional integer `0..3` indicating the SSH client debug level.
:raises mitogen.ssh.PasswordError:
A password was requested but none was specified, or the specified
@ -808,12 +817,12 @@ Context Class
The message.
:returns:
:py:class:`mitogen.core.Receiver` configured to receive any replies
:class:`mitogen.core.Receiver` configured to receive any replies
sent to the message's `reply_to` handle.
.. method:: send_await (msg, deadline=None)
As with :py:meth:`send_async`, but expect a single reply
As with :meth:`send_async`, but expect a single reply
(`persist=False`) delivered within `deadline` seconds.
:param mitogen.core.Message msg:
@ -830,7 +839,7 @@ Context Class
.. class:: Context
Extend :py:class:`mitogen.core.Router` with functionality useful to
Extend :class:`mitogen.core.Router` with functionality useful to
masters, and child contexts who later become parents. Currently when this
class is required, the target context's router is upgraded at runtime.
@ -843,7 +852,7 @@ Context Class
terminate a hung context using this method. This will be fixed shortly.
:param bool wait:
If :py:data:`True`, block the calling thread until the context has
If :data:`True`, block the calling thread until the context has
completely terminated.
:returns:
If `wait` is :data:`False`, returns a :class:`mitogen.core.Latch`
@ -888,7 +897,7 @@ Context Class
Function keyword arguments, if any. See :ref:`serialization-rules`
for permitted types.
:returns:
:py:class:`mitogen.core.Receiver` configured to receive the result
:class:`mitogen.core.Receiver` configured to receive the result
of the invocation:
.. code-block:: python
@ -903,11 +912,11 @@ Context Class
Asynchronous calls may be dispatched in parallel to multiple
contexts and consumed as they complete using
:py:class:`mitogen.select.Select`.
:class:`mitogen.select.Select`.
.. method:: call (fn, \*args, \*\*kwargs)
Equivalent to :py:meth:`call_async(fn, \*args, \**kwargs).get().unpickle()
Equivalent to :meth:`call_async(fn, \*args, \**kwargs).get().unpickle()
<call_async>`.
:returns:
@ -935,7 +944,7 @@ Receiver Class
Receivers are used to wait for pickled responses from another context to be
sent to a handle registered in this context. A receiver may be single-use
(as in the case of :py:meth:`mitogen.parent.Context.call_async`) or
(as in the case of :meth:`mitogen.parent.Context.call_async`) or
multiple use.
:param mitogen.core.Router router:
@ -959,12 +968,12 @@ Receiver Class
If not :data:`None`, a reference to a function invoked as
`notify(receiver)` when a new message is delivered to this receiver.
Used by :py:class:`mitogen.select.Select` to implement waiting on
Used by :class:`mitogen.select.Select` to implement waiting on
multiple receivers.
.. py:method:: to_sender ()
Return a :py:class:`mitogen.core.Sender` configured to deliver messages
Return a :class:`mitogen.core.Sender` configured to deliver messages
to this receiver. Since a Sender can be serialized, this makes it
convenient to pass `(context_id, handle)` pairs around::
@ -981,15 +990,15 @@ Receiver Class
.. py:method:: empty ()
Return :data:`True` if calling :py:meth:`get` would block.
Return :data:`True` if calling :meth:`get` would block.
As with :py:class:`Queue.Queue`, :data:`True` may be returned even
though a subsequent call to :py:meth:`get` will succeed, since a
message may be posted at any moment between :py:meth:`empty` and
:py:meth:`get`.
As with :class:`Queue.Queue`, :data:`True` may be returned even
though a subsequent call to :meth:`get` will succeed, since a
message may be posted at any moment between :meth:`empty` and
:meth:`get`.
:py:meth:`empty` is only useful to avoid a race while installing
:py:attr:`notify`:
:meth:`empty` is only useful to avoid a race while installing
:attr:`notify`:
.. code-block:: python
@ -1003,8 +1012,8 @@ Receiver Class
.. py:method:: close ()
Cause :py:class:`mitogen.core.ChannelError` to be raised in any thread
waiting in :py:meth:`get` on this receiver.
Cause :class:`mitogen.core.ChannelError` to be raised in any thread
waiting in :meth:`get` on this receiver.
.. py:method:: get (timeout=None)
@ -1022,17 +1031,17 @@ Receiver Class
:returns:
`(msg, data)` tuple, where `msg` is the
:py:class:`mitogen.core.Message` that was received, and `data` is
:class:`mitogen.core.Message` that was received, and `data` is
its unpickled data part.
.. py:method:: get_data (timeout=None)
Like :py:meth:`get`, except only return the data part.
Like :meth:`get`, except only return the data part.
.. py:method:: __iter__ ()
Block and yield `(msg, data)` pairs delivered to this receiver until
:py:class:`mitogen.core.ChannelError` is raised.
:class:`mitogen.core.ChannelError` is raised.
Sender Class
@ -1043,10 +1052,10 @@ Sender Class
.. class:: Sender (context, dst_handle)
Senders are used to send pickled messages to a handle in another context,
it is the inverse of :py:class:`mitogen.core.Sender`.
it is the inverse of :class:`mitogen.core.Sender`.
Senders may be serialized, making them convenient to wire up data flows.
See :py:meth:`mitogen.core.Receiver.to_sender` for more information.
See :meth:`mitogen.core.Receiver.to_sender` for more information.
:param mitogen.core.Context context:
Context to send messages to.
@ -1055,7 +1064,7 @@ Sender Class
.. py:method:: close ()
Send a dead message to the remote end, causing :py:meth:`ChannelError`
Send a dead message to the remote end, causing :meth:`ChannelError`
to be raised in any waiting thread.
.. py:method:: send (data)
@ -1074,11 +1083,11 @@ Select Class
Support scatter/gather asynchronous calls and waiting on multiple
receivers, channels, and sub-Selects. Accepts a sequence of
:py:class:`mitogen.core.Receiver` or :py:class:`mitogen.select.Select`
:class:`mitogen.core.Receiver` or :class:`mitogen.select.Select`
instances and returns the first value posted to any receiver or select.
If `oneshot` is :data:`True`, then remove each receiver as it yields a
result; since :py:meth:`__iter__` terminates once the final receiver is
result; since :meth:`__iter__` terminates once the final receiver is
removed, this makes it convenient to respond to calls made in parallel:
.. code-block:: python
@ -1093,7 +1102,7 @@ Select Class
# Iteration ends when last Receiver yields a result.
print('Received total %s from %s receivers' % (total, len(recvs)))
:py:class:`Select` may drive a long-running scheduler:
:class:`Select` may drive a long-running scheduler:
.. code-block:: python
@ -1104,7 +1113,7 @@ Select Class
for context, workfunc in get_new_work():
select.add(context.call_async(workfunc))
:py:class:`Select` may be nested:
:class:`Select` may be nested:
.. code-block:: python
@ -1122,11 +1131,11 @@ Select Class
.. py:classmethod:: all (it)
Take an iterable of receivers and retrieve a :py:class:`Message` from
Take an iterable of receivers and retrieve a :class:`Message` from
each, returning the result of calling `msg.unpickle()` on each in turn.
Results are returned in the order they arrived.
This is sugar for handling batch :py:class:`Context.call_async`
This is sugar for handling batch :class:`Context.call_async`
invocations:
.. code-block:: python
@ -1146,28 +1155,28 @@ Select Class
for context in contexts)
Result processing happens concurrently to new results arriving, so
:py:meth:`all` should always be faster.
:meth:`all` should always be faster.
.. py:method:: get (timeout=None, block=True)
Fetch the next available value from any receiver, or raise
:py:class:`mitogen.core.TimeoutError` if no value is available within
:class:`mitogen.core.TimeoutError` if no value is available within
`timeout` seconds.
On success, the message's :py:attr:`receiver
On success, the message's :attr:`receiver
<mitogen.core.Message.receiver>` attribute is set to the receiver.
:param float timeout:
Timeout in seconds.
:param bool block:
If :py:data:`False`, immediately raise
:py:class:`mitogen.core.TimeoutError` if the select is empty.
If :data:`False`, immediately raise
:class:`mitogen.core.TimeoutError` if the select is empty.
:return:
:py:class:`mitogen.core.Message`
:class:`mitogen.core.Message`
:raises mitogen.core.TimeoutError:
Timeout was reached.
:raises mitogen.core.LatchError:
:py:meth:`close` has been called, and the underlying latch is no
:meth:`close` has been called, and the underlying latch is no
longer valid.
.. py:method:: __bool__ ()
@ -1178,8 +1187,8 @@ Select Class
Remove the select's notifier function from each registered receiver,
mark the associated latch as closed, and cause any thread currently
sleeping in :py:meth:`get` to be woken with
:py:class:`mitogen.core.LatchError`.
sleeping in :meth:`get` to be woken with
:class:`mitogen.core.LatchError`.
This is necessary to prevent memory leaks in long-running receivers. It
is called automatically when the Python :keyword:`with` statement is
@ -1187,35 +1196,35 @@ Select Class
.. py:method:: empty ()
Return :data:`True` if calling :py:meth:`get` would block.
Return :data:`True` if calling :meth:`get` would block.
As with :py:class:`Queue.Queue`, :data:`True` may be returned even
though a subsequent call to :py:meth:`get` will succeed, since a
message may be posted at any moment between :py:meth:`empty` and
:py:meth:`get`.
As with :class:`Queue.Queue`, :data:`True` may be returned even
though a subsequent call to :meth:`get` will succeed, since a
message may be posted at any moment between :meth:`empty` and
:meth:`get`.
:py:meth:`empty` may return :data:`False` even when :py:meth:`get`
:meth:`empty` may return :data:`False` even when :meth:`get`
would block if another thread has drained a receiver added to this
select. This can be avoided by only consuming each receiver from a
single thread.
.. py:method:: __iter__ (self)
Yield the result of :py:meth:`get` until no receivers remain in the
Yield the result of :meth:`get` until no receivers remain in the
select, either because `oneshot` is :data:`True`, or each receiver was
explicitly removed via :py:meth:`remove`.
explicitly removed via :meth:`remove`.
.. py:method:: add (recv)
Add the :py:class:`mitogen.core.Receiver` or
:py:class:`mitogen.core.Channel` `recv` to the select.
Add the :class:`mitogen.core.Receiver` or
:class:`mitogen.core.Channel` `recv` to the select.
.. py:method:: remove (recv)
Remove the :py:class:`mitogen.core.Receiver` or
:py:class:`mitogen.core.Channel` `recv` from the select. Note that if
the receiver has notified prior to :py:meth:`remove`, then it will
still be returned by a subsequent :py:meth:`get`. This may change in a
Remove the :class:`mitogen.core.Receiver` or
:class:`mitogen.core.Channel` `recv` from the select. Note that if
the receiver has notified prior to :meth:`remove`, then it will
still be returned by a subsequent :meth:`get`. This may change in a
future version.
@ -1226,7 +1235,7 @@ Channel Class
.. class:: Channel (router, context, dst_handle, handle=None)
A channel inherits from :py:class:`mitogen.core.Sender` and
A channel inherits from :class:`mitogen.core.Sender` and
`mitogen.core.Receiver` to provide bidirectional functionality.
Since all handles aren't known until after both ends are constructed, for
@ -1248,8 +1257,8 @@ Broker Class
.. attribute:: shutdown_timeout = 3.0
Seconds grace to allow :py:class:`streams <Stream>` to shutdown
gracefully before force-disconnecting them during :py:meth:`shutdown`.
Seconds grace to allow :class:`streams <Stream>` to shutdown
gracefully before force-disconnecting them during :meth:`shutdown`.
.. method:: defer (func, \*args, \*kwargs)
@ -1259,26 +1268,26 @@ Broker Class
.. method:: start_receive (stream)
Mark the :py:attr:`receive_side <Stream.receive_side>` on `stream` as
Mark the :attr:`receive_side <Stream.receive_side>` on `stream` as
ready for reading. Safe to call from any thread. When the associated
file descriptor becomes ready for reading,
:py:meth:`BasicStream.on_receive` will be called.
:meth:`BasicStream.on_receive` will be called.
.. method:: stop_receive (stream)
Mark the :py:attr:`receive_side <Stream.receive_side>` on `stream` as
Mark the :attr:`receive_side <Stream.receive_side>` on `stream` as
not ready for reading. Safe to call from any thread.
.. method:: _start_transmit (stream)
Mark the :py:attr:`transmit_side <Stream.transmit_side>` on `stream` as
Mark the :attr:`transmit_side <Stream.transmit_side>` on `stream` as
ready for writing. Must only be called from the Broker thread. When the
associated file descriptor becomes ready for writing,
:py:meth:`BasicStream.on_transmit` will be called.
:meth:`BasicStream.on_transmit` will be called.
.. method:: stop_receive (stream)
Mark the :py:attr:`transmit_side <Stream.receive_side>` on `stream` as
Mark the :attr:`transmit_side <Stream.receive_side>` on `stream` as
not ready for writing. Safe to call from any thread.
.. method:: shutdown
@ -1288,12 +1297,12 @@ Broker Class
.. method:: join
Wait for the broker to stop, expected to be called after
:py:meth:`shutdown`.
:meth:`shutdown`.
.. method:: keep_alive
Return :data:`True` if any reader's :py:attr:`Side.keep_alive`
attribute is :data:`True`, or any :py:class:`Context` is still
Return :data:`True` if any reader's :attr:`Side.keep_alive`
attribute is :data:`True`, or any :class:`Context` is still
registered that is not the master. Used to delay shutdown while some
important work is in progress (e.g. log draining).
@ -1301,11 +1310,11 @@ Broker Class
.. method:: _broker_main
Handle events until :py:meth:`shutdown`. On shutdown, invoke
:py:meth:`Stream.on_shutdown` for every active stream, then allow up to
:py:attr:`shutdown_timeout` seconds for the streams to unregister
Handle events until :meth:`shutdown`. On shutdown, invoke
:meth:`Stream.on_shutdown` for every active stream, then allow up to
:attr:`shutdown_timeout` seconds for the streams to unregister
themselves before forcefully calling
:py:meth:`Stream.on_disconnect`.
:meth:`Stream.on_disconnect`.
.. currentmodule:: mitogen.master
@ -1321,7 +1330,7 @@ Broker Class
:param bool install_watcher:
If :data:`True`, an additional thread is started to monitor the
lifetime of the main thread, triggering :py:meth:`shutdown`
lifetime of the main thread, triggering :meth:`shutdown`
automatically in case the user forgets to call it, or their code
crashed.
@ -1332,8 +1341,8 @@ Broker Class
.. attribute:: shutdown_timeout = 5.0
Seconds grace to allow :py:class:`streams <Stream>` to shutdown
gracefully before force-disconnecting them during :py:meth:`shutdown`.
Seconds grace to allow :class:`streams <Stream>` to shutdown
gracefully before force-disconnecting them during :meth:`shutdown`.
Utility Functions
@ -1349,7 +1358,7 @@ A random assortment of utility functions useful on masters and children.
Many tools love to subclass built-in types in order to implement useful
functionality, such as annotating the safety of a Unicode string, or adding
additional methods to a dict. However, cPickle loves to preserve those
subtypes during serialization, resulting in CallError during :py:meth:`call
subtypes during serialization, resulting in CallError during :meth:`call
<mitogen.parent.Context.call>` in the target when it tries to deserialize
the data.
@ -1369,12 +1378,12 @@ A random assortment of utility functions useful on masters and children.
Remove all entries mentioning ``site-packages`` or ``Extras`` from the
system path. Used primarily for testing on OS X within a virtualenv, where
OS X bundles some ancient version of the :py:mod:`six` module.
OS X bundles some ancient version of the :mod:`six` module.
.. currentmodule:: mitogen.utils
.. function:: log_to_file (path=None, io=False, level='INFO')
Install a new :py:class:`logging.Handler` writing applications logs to the
Install a new :class:`logging.Handler` writing applications logs to the
filesystem. Useful when debugging slave IO problems.
Parameters to this function may be overridden at runtime using environment
@ -1382,14 +1391,14 @@ A random assortment of utility functions useful on masters and children.
:param str path:
If not :data:`None`, a filesystem path to write logs to. Otherwise,
logs are written to :py:data:`sys.stderr`.
logs are written to :data:`sys.stderr`.
:param bool io:
If :data:`True`, include extremely verbose IO logs in the output.
Useful for debugging hangs, less useful for debugging application code.
:param str level:
Name of the :py:mod:`logging` package constant that is the minimum
Name of the :mod:`logging` package constant that is the minimum
level to log at. Useful levels are ``DEBUG``, ``INFO``, ``WARNING``,
and ``ERROR``.
@ -1397,7 +1406,7 @@ A random assortment of utility functions useful on masters and children.
.. function:: run_with_router(func, \*args, \**kwargs)
Arrange for `func(router, \*args, \**kwargs)` to run with a temporary
:py:class:`mitogen.master.Router`, ensuring the Router and Broker are
:class:`mitogen.master.Router`, ensuring the Router and Broker are
correctly shut down during normal or exceptional return.
:returns:
@ -1406,7 +1415,7 @@ A random assortment of utility functions useful on masters and children.
.. currentmodule:: mitogen.utils
.. decorator:: with_router
Decorator version of :py:func:`run_with_router`. Example:
Decorator version of :func:`run_with_router`. Example:
.. code-block:: python

55
mitogen/ssh.py
Unescape Escape View File

@ -111,7 +111,6 @@ class HostKeyError(mitogen.core.StreamError):
class Stream(mitogen.parent.Stream):
create_child = staticmethod(mitogen.parent.hybrid_tty_create_child)
child_is_immediate_subprocess = False
#: Default to whatever is available as 'python' on the remote machine,
@ -121,8 +120,8 @@ class Stream(mitogen.parent.Stream):
#: Number of -v invocations to pass on command line.
ssh_debug_level = 0
#: Once connected, points to the corresponding TtyLogStream, allowing it to
#: be disconnected at the same time this stream is being torn down.
#: If batch_mode=False, points to the corresponding TtyLogStream, allowing
#: it to be disconnected at the same time this stream is being torn down.
tty_stream = None
#: The path to the SSH binary.
@ -137,15 +136,27 @@ class Stream(mitogen.parent.Stream):
ssh_args = None
check_host_keys_msg = 'check_host_keys= must be set to accept, enforce or ignore'
batch_mode_check_host_keys_msg = (
'check_host_keys cannot be set to "accept" when batch mode is '
'enabled, since batch mode disables PTY allocation.'
)
batch_mode_password_msg = (
'A password cannot be set when batch mode is enabled, '
'since batch mode disables PTY allocation.'
)
def construct(self, hostname, username=None, ssh_path=None, port=None,
check_host_keys='enforce', password=None, identity_file=None,
compression=True, ssh_args=None, keepalive_enabled=True,
keepalive_count=3, keepalive_interval=15,
keepalive_count=3, keepalive_interval=15, batch_mode=False,
identities_only=True, ssh_debug_level=None, **kwargs):
super(Stream, self).construct(**kwargs)
if check_host_keys not in ('accept', 'enforce', 'ignore'):
raise ValueError(self.check_host_keys_msg)
if check_host_keys == 'accept' and batch_mode:
raise ValueError(self.batch_mode_check_host_keys_msg)
if password is not None and batch_mode:
raise ValueError(self.batch_mode_password_msg)
self.hostname = hostname
self.username = username
@ -158,6 +169,14 @@ class Stream(mitogen.parent.Stream):
self.keepalive_enabled = keepalive_enabled
self.keepalive_count = keepalive_count
self.keepalive_interval = keepalive_interval
self.batch_mode = batch_mode
if self.batch_mode:
self.create_child = mitogen.parent.create_child
self.create_child_args = {
'merge_stdio': True,
}
else:
self.create_child = mitogen.parent.hybrid_tty_create_child
if ssh_path:
self.ssh_path = ssh_path
if ssh_args:
@ -166,7 +185,8 @@ class Stream(mitogen.parent.Stream):
self.ssh_debug_level = ssh_debug_level
def on_disconnect(self, broker):
self.tty_stream.on_disconnect(broker)
if self.tty_stream is not None:
self.tty_stream.on_disconnect(broker)
super(Stream, self).on_disconnect(broker)
def get_boot_command(self):
@ -193,10 +213,15 @@ class Stream(mitogen.parent.Stream):
'-o', 'ServerAliveInterval %s' % (self.keepalive_interval,),
'-o', 'ServerAliveCountMax %s' % (self.keepalive_count,),
]
if self.batch_mode:
bits += ['-o', 'BatchMode yes']
if self.check_host_keys == 'enforce':
bits += ['-o', 'StrictHostKeyChecking yes']
if self.check_host_keys == 'accept':
bits += ['-o', 'StrictHostKeyChecking ask']
if self.batch_mode:
bits += ['-o', 'StrictHostKeyChecking no']
else:
bits += ['-o', 'StrictHostKeyChecking ask']
elif self.check_host_keys == 'ignore':
bits += [
'-o', 'StrictHostKeyChecking no',
@ -240,19 +265,23 @@ class Stream(mitogen.parent.Stream):
# with ours.
raise HostKeyError(self.hostkey_config_msg)
def _ec0_received(self):
if self.tty_stream is not None:
self._router.broker.start_receive(self.tty_stream)
return super(Stream, self)._ec0_received()
def _connect_bootstrap(self, extra_fd):
self.tty_stream = mitogen.parent.TtyLogStream(extra_fd, self)
fds = [self.receive_side.fd]
if extra_fd is not None:
self.tty_stream = mitogen.parent.TtyLogStream(extra_fd, self)
fds.append(extra_fd)
password_sent = False
it = mitogen.parent.iter_read(
fds=[self.receive_side.fd, extra_fd],
deadline=self.connect_deadline
)
it = mitogen.parent.iter_read(fds=fds, deadline=self.connect_deadline)
password_sent = False
for buf, partial in filter_debug(self, it):
LOG.debug('%r: received %r', self, buf)
if buf.endswith(self.EC0_MARKER):
self._router.broker.start_receive(self.tty_stream)
self._ec0_received()
return
elif HOSTKEY_REQ_PROMPT in buf.lower():

39
tests/data/fakessh.py
Unescape Escape View File

@ -6,6 +6,45 @@ import shlex
import subprocess
import sys
HOST_KEY_ASK_MSG = """
The authenticity of host '[91.121.165.123]:9122 ([91.121.165.123]:9122)' can't be established.
ECDSA key fingerprint is SHA256:JvfPvazZzQ9/CUdKN7tiYlNZtDRdEgDsYVIzOgPrsR4.
Are you sure you want to continue connecting (yes/no)?
""".strip('\n')
HOST_KEY_STRICT_MSG = """Host key verification failed.\n"""
def tty(msg):
fp = open('/dev/tty', 'w', 0)
fp.write(msg)
fp.close()
def stderr(msg):
fp = open('/dev/stderr', 'w', 0)
fp.write(msg)
fp.close()
def confirm(msg):
tty(msg)
fp = open('/dev/tty', 'r', 0)
try:
return fp.readline()
finally:
fp.close()
if os.getenv('FAKESSH_MODE') == 'ask':
assert 'y\n' == confirm(HOST_KEY_ASK_MSG)
if os.getenv('FAKESSH_MODE') == 'strict':
stderr(HOST_KEY_STRICT_MSG)
sys.exit(255)
parser = optparse.OptionParser()
parser.add_option('--user', '-l', action='store')
parser.add_option('-o', dest='options', action='append')

87
tests/ssh_test.py
Unescape Escape View File

@ -1,3 +1,4 @@
import os
import sys
import mitogen
@ -13,9 +14,9 @@ import plain_old_module
class FakeSshTest(testlib.RouterMixin, unittest2.TestCase):
def test_okay(self):
context = self.router.ssh(
hostname='hostname',
username='mitogen__has_sudo',
ssh_path=testlib.data_path('fakessh.py'),
hostname='hostname',
username='mitogen__has_sudo',
ssh_path=testlib.data_path('fakessh.py'),
)
#context.call(mitogen.utils.log_to_file, '/tmp/log')
#context.call(mitogen.utils.disable_site_packages)
@ -123,6 +124,86 @@ class BannerTest(testlib.DockerMixin, unittest2.TestCase):
self.assertEquals(name, context.name)
class BatchModeTest(testlib.DockerMixin, testlib.TestCase):
stream_class = mitogen.ssh.Stream
#
# Test that:
#
# - batch_mode=false, host_key_checking=accept
# - batch_mode=false, host_key_checking=enforce
# - batch_mode=false, host_key_checking=ignore
#
# - batch_mode=true, host_key_checking=accept
# - batch_mode=true, host_key_checking=enforce
# - batch_mode=true, host_key_checking=ignore
# - batch_mode=true, password is not None
#
def fake_ssh(self, FAKESSH_MODE=None, **kwargs):
os.environ['FAKESSH_MODE'] = str(FAKESSH_MODE)
try:
return self.router.ssh(
hostname='hostname',
username='mitogen__has_sudo',
ssh_path=testlib.data_path('fakessh.py'),
**kwargs
)
finally:
del os.environ['FAKESSH_MODE']
def test_false_accept(self):
# Should succeed.
self.fake_ssh(FAKESSH_MODE='ask', check_host_keys='accept')
def test_false_enforce(self):
# Should succeed.
self.fake_ssh(check_host_keys='enforce')
def test_false_ignore(self):
# Should succeed.
self.fake_ssh(check_host_keys='ignore')
def test_false_password(self):
# Should succeed.
self.docker_ssh(username='mitogen__has_sudo_nopw',
password='has_sudo_nopw_password')
def test_true_accept(self):
e = self.assertRaises(ValueError,
lambda: self.fake_ssh(check_host_keys='accept', batch_mode=True)
)
self.assertEquals(e.args[0],
self.stream_class.batch_mode_check_host_keys_msg)
def test_true_enforce(self):
e = self.assertRaises(mitogen.ssh.HostKeyError,
lambda: self.docker_ssh(
batch_mode=True,
check_host_keys='enforce',
ssh_args=['-o', 'UserKnownHostsFile /dev/null'],
)
)
self.assertEquals(e.args[0], self.stream_class.hostkey_failed_msg)
def test_true_ignore(self):
e = self.assertRaises(mitogen.ssh.HostKeyError,
lambda: self.fake_ssh(
FAKESSH_MODE='strict',
batch_mode=True,
check_host_keys='ignore',
)
)
self.assertEquals(e.args[0], self.stream_class.hostkey_failed_msg)
def test_true_password(self):
e = self.assertRaises(ValueError,
lambda: self.fake_ssh(
password='nope',
batch_mode=True,
)
)
self.assertEquals(e.args[0], self.stream_class.batch_mode_password_msg)
if __name__ == '__main__':
unittest2.main()

Write Preview
Loading…
Cancel
Save