archive
/
mitogen
mirror of https://github.com/mitogen-hq/mitogen.git
1
0
Fork 0
Code Releases Activity

docs: major Ansible page update.

Browse Source
pull/219/head
David Wilson 7 years ago
parent fa4746f656
commit 3ce6b36932
1 changed files with 350 additions and 262 deletions
Show Stats Download Patch File Download Diff File

612
docs/ansible.rst
Unescape Escape View File

@ -1,64 +1,57 @@
Ansible Extension
=================
.. image:: images/ansible/cell_division.png .. image:: images/ansible/cell_division.png
:align: right :align: right
An extension to `Ansible`_ is included that implements host connections over Mitogen for Ansible
Mitogen, replacing embedded shell invocations with pure-Python equivalents ===================
invoked via highly efficient remote procedure calls tunnelled over SSH. No
changes are required to the target hosts.
The extension is approaching a generally dependable state, and works well for
many real-world playbooks. `Bug reports`_ in this area are very welcome –
Ansible is a huge beast, and only significant testing will prove the
extension's soundness.
Divergence from Ansible's normal behaviour is considered a bug, so please An extension to `Ansible`_ is included that implements connections over
report anything you notice, regardless of how inconsequential it may seem. Mitogen, replacing embedded shell invocations with pure-Python equivalents
invoked via highly efficient remote procedure calls to persistent interpreters
tunnelled over SSH. No changes are required to target hosts.
The extension is approaching stability and real-world testing is now
encouraged. `Bug reports`_ are welcome: Ansible is huge, and only wide testing
will ensure soundness.
.. _Ansible: https://www.ansible.com/ .. _Ansible: https://www.ansible.com/
.. _Bug reports: https://goo.gl/yLKZiJ .. _Bug reports: https://goo.gl/yLKZiJ
Overview Overview
-------- --------
You should **expect a 1.25x - 7x speedup** and a **CPU usage reduction of at **Expect a 1.25x - 7x speedup** and a **CPU usage reduction of at least 2x**,
least 2x**, depending on network conditions, the specific modules executed, and depending on network conditions, modules executed, and time already spent by
time spent by the target host already doing useful work. Mitogen cannot speed targets on useful work. Mitogen cannot improve a module once it is executing,
up a module once it is executing, it can only ensure the module executes as it can only ensure the module executes as quickly as possible.
quickly as possible.
* **One connection is used per target**, in addition to one sudo invocation per
* **A single SSH connection is used for each target host**, in addition to one user account. This is much better than SSH multiplexing combined with
sudo invocation per distinct user account. Subsequent playbook steps always pipelining, as significant state can be maintained in RAM between steps, and
reuse the same connection. This is much better than SSH multiplexing combined system logs aren't spammed with repeat authentication events.
with pipelining, as significant state can be maintained in RAM between steps,
and the system logs aren't filled with spam from repeat SSH and sudo * **A single network roundtrip is used** to execute a step whose code already
invocations. exists in RAM on the target. Eliminating multiplexed SSH channel creation
saves 5 ms runtime per 1 ms of network latency for every playbook step.
* **A single Python interpreter is used** per host and sudo account combination
for the duration of the run, avoiding the repeat cost of invoking multiple * **Processes are aggressively reused**, avoiding the cost of invoking Python
interpreters and recompiling imports, saving 300-800 ms for every playbook and recompiling imports, saving 300-800 ms for every playbook step.
step.
* Code is ephemerally cached in RAM, **reducing bandwidth usage by an order
* Remote interpreters reuse Mitogen's module import mechanism, caching uploaded of magnitude** compared to SSH pipelining, with around 5x fewer frames
dependencies between steps at the host and user account level. As a traversing the network in a typical run.
consequence, **bandwidth usage is consistently an order of magnitude lower**
compared to SSH pipelining, and around 5x fewer frames are required to * **No writes to the target's filesystem occur**, unless explicitly triggered
traverse the wire for a run to complete successfully. by a playbook step. In all typical configurations, Ansible repeatedly
rewrites and extracts ZIP files to multiple temporary directories on the
* **No writes to the target host's filesystem occur**, unless explicitly target. Since no temporary files are used, security issues relating to those
triggered by a playbook step. In all typical configurations, Ansible files in cross-account scenarios are entirely avoided.
repeatedly rewrites and extracts ZIP files to multiple temporary directories
on the target host. Since no temporary files are used, security issues
relating to those files in cross-account scenarios are entirely avoided.
Demo Demo
---- ~~~~
This demonstrates Ansible running a subset of the Mitogen integration tests This demonstrates Ansible running a subset of the Mitogen integration tests
concurrent to an equivalent run using the extension. concurrent to an equivalent run using the extension.
@ -71,7 +64,7 @@ concurrent to an equivalent run using the extension.
Testimonials Testimonials
------------ ~~~~~~~~~~~~
* "With mitogen **my playbook runtime went from 45 minutes to just under 3 * "With mitogen **my playbook runtime went from 45 minutes to just under 3
minutes**. Awesome work!" minutes**. Awesome work!"
@ -96,14 +89,11 @@ Testimonials
Installation Installation
------------ ------------
.. caution:: 1. Thoroughly review the documented behavioural differences.
2. Verify Ansible 2.4/2.5 and Python 2.7 are listed in ``ansible --version``
Please review the behavioural differences documented below prior to use. output.
3. Download and extract https://github.com/dw/mitogen/archive/master.zip
1. Verify Ansible 2.4 and Python 2.7 are listed in the output of ``ansible 4. Modify ``ansible.cfg``:
--version``
2. Download and extract https://github.com/dw/mitogen/archive/master.zip
3. Modify ``ansible.cfg``:
.. code-block:: dosini .. code-block:: dosini
@ -111,70 +101,256 @@ Installation
strategy_plugins = /path/to/mitogen-master/ansible_mitogen/plugins/strategy strategy_plugins = /path/to/mitogen-master/ansible_mitogen/plugins/strategy
strategy = mitogen_linear strategy = mitogen_linear
The ``strategy`` key is optional. If omitted, you can set the The ``strategy`` key is optional. If omitted, the
``ANSIBLE_STRATEGY=mitogen_linear`` environment variable on a per-run basis. ``ANSIBLE_STRATEGY=mitogen_linear`` environment variable can be set on a
Like ``mitogen_linear``, the ``mitogen_free`` strategy also exists to mimic per-run basis. Like ``mitogen_linear``, the ``mitogen_free`` strategy exists
the built-in ``free`` strategy. to mimic the ``free`` strategy.
Noteworthy Differences
----------------------
* Ansible 2.4 and 2.5 are supported. File bugs to register interest in older
releases.
4. Cross your fingers and try it. * The ``sudo`` become method is available and ``su`` is planned. File bugs to
register interest in additional methods.
* The ``ssh``, ``local`` and ``docker`` connection types are available, with
more planned. File bugs to register interest.
Limitations * Local commands execute in a reuseable interpreter created identically to
----------- interpreters on targets. Presently one interpreter per ``become_user``
exists, and so only one local action may execute simultaneously.
* Only Ansible 2.4 is being used for development, with occasional tests under Ansible usually permits up to ``forks`` simultaneous local actions. Any
2.5, 2.3 and 2.2. It should be more than possible to fully support at least long-running local actions that execute for every target will experience
2.3, if not also 2.2. artificial serialization, causing slowdown equivalent to `task_duration *
num_targets`. This will be fixed soon.
* Only the ``sudo`` become method is available, however adding new methods is * Asynchronous jobs presently exist only for the duration of a run, and time
straightforward, and eventually at least ``su`` will be included. limits are not implemented.
* The extension's performance benefits do not scale perfectly linearly with the * Due to use of :func:`select.select` the IO multiplexer breaks down around 100
number of targets. This is a subject of ongoing investigation and targets, expect performance degradation as this number is approached and
improvements will appear in time. errant behaviour as it is exceeded. A replacement will appear soon.
* "Module Replacer" style modules are not yet supported. These rarely appear in * The undocumented ability to extend :mod:`ansible.module_utils` by supplying a
practice, and light Github code searches failed to reveal many examples of ``module_utils`` directory alongside a custom new-style module is not yet
them. supported.
* "Module Replacer" style modules are not supported. These rarely appear in
practice, and light web searches failed to reveal many examples of them.
Behavioural Differences * Ansible permits up to ``forks`` connections to be setup in parallel, whereas
----------------------- in Mitogen this is handled by a fixed-size thread pool. Up to 16 connections
may be established in parallel by default, this can be modified by setting
the ``MITOGEN_POOL_SIZE`` environment variable.
* Ansible permits up to ``forks`` SSH connections to be setup simultaneously, * Performance does not scale perfectly linearly with target count. This will
whereas in Mitogen this is handled by a thread pool. Eventually this pool improve over time.
will become per-CPU, but meanwhile, a maximum of 16 SSH connections may be
established simultaneously by default. This can be increased or decreased
setting the ``MITOGEN_POOL_SIZE`` environment variable.
* Mitogen treats connection timeouts for the SSH and become steps of a task * Timeouts normally apply to the combined runtime of the SSH and become steps
invocation separately, meaning that in some circumstances the configured of a task. As Mitogen treats SSH and sudo distincly, during a failure the
timeout may appear to be doubled. This is since Mitogen internally treats the effective timeout may appear to double.
creation of an SSH account context separately to the creation of a sudo
account context proxied via that SSH account.
A future revision may detect a sudo account context created immediately
following its parent SSH account, and try to emulate Ansible's existing
timeout semantics.
* Local commands are executed in a reuseable Python interpreter created New Features & Notes
identically to interpreters used on remote hosts. At present only one such --------------------
interpreter per ``become_user`` exists, and so only one local action may be
executed simultaneously per local user account.
Connection Delegation
~~~~~~~~~~~~~~~~~~~~~
.. image:: images/jumpbox.png
:align: right
Included is a preview of **Connection Delegation**, a Mitogen-specific
implementation of `stackable connection plug-ins`_. This enables multi-hop
connections via a bastion, or Docker connections delegated via their host
machine, where reaching the host may itself entail recursive delegation.
.. _Stackable connection plug-ins: https://github.com/ansible/proposals/issues/25
Unlike with SSH forwarding Ansible has complete visibility of the final
topology, declarative configuration via static/dynamic inventory is possible,
and data can be cached and re-served, and code executed on every intermediary.
For example when targeting Docker containers on a remote machine, each module
need only be uploaded once for the first task and container that requires it,
then cached and served from the SSH account for every future task in any
container.
.. raw:: html
<div style="clear: both;"></div>
.. caution::
Connection delegation is a work in progress, bug reports are welcome.
Ansible usually permits up to ``ansible.cfg:forks`` simultaneous local * While imports are cached on intermediaries, module scripts are needlessly
actions. Any long-running local actions that execute for every target will reuploaded for each target. Fixing this is equivalent to implementing
experience artificial serialization, causing slowdown equivalent to **Topology-Aware File Synchronization**, so it may remain unfixed until
`task_duration * num_targets`. This will be fixed soon. that feature is started.
* Delegated connection setup is single-threaded; only one connection can be
constructed in parallel per intermediary.
* Unbounded queue RAM growth may occur in an intermediary during large file
transfers if the link between any two hops is slower than the link
between the controller and the first hop.
* Inferring the configuration of intermediaries may be buggy, manifesting
as duplicate connections between hops, due to not perfectly replicating
the configuration Ansible would normally use for the intermediary.
* The extension does not understand the difference between a delegated
connection and a ``become_user``. If interpreter recycling kicks in, a
delegated connection could be prematurely recycled.
To enable connection delegation, set ``mitogen_via=<inventory name>`` on the
command line, or as host and group variables.
.. code-block:: ini
# Docker container on web1.dc1 is reachable via web1.dc1.
[app-containers.web1.dc1]
app1.web1.dc1 ansible_host=app1 ansible_connection=docker mitogen_via=web1.dc1
# Web servers in DC1 are reachable via bastion.dc1
[dc1]
web1.dc1
web2.dc1
web3.dc1
[dc1:vars]
mitogen_via = bastion.dc1
# Web servers in DC2 are reachable via bastion.dc2
[dc2]
web1.dc2
web2.dc2
web3.dc2
[dc2:vars]
mitogen_via = bastion.dc2
# Prod bastions are reachable via a corporate network gateway.
[bastions]
bastion.dc1 mitogen_via=corp-gateway.internal
bastion.dc2 mitogen_via=corp-gateway.internal
[corp-gateway]
corp-gateway.internal
File Transfer
~~~~~~~~~~~~~
Normally a tool like ``scp`` is used to copy a file with the ``copy`` or
``template`` actions, or when uploading modules with pipelining disabled. With
Mitogen copies are implemented natively using the same interpreters, connection
tree, and routed message bus that carries RPCs.
This permits streaming directly between endpoints regardless of execution
environment, without necessitating temporary copies in intermediary accounts or
machines, for example when ``become`` is active, or in the presence of
connection delegation. It also neatly avoids the problem of securely sharing
temporary files between accounts and machines.
One roundtrip is required to initiate a transfer. For any tool that operates
via SSH multiplexing, 5 are required to configure the associated IO channel, in
addition to the time needed to start the local and remote processes. A complete
localhost invocation of ``scp`` requires around 15 ms.
As the implementation is self-contained, it is simple to make future
improvements like prioritizing transfers, supporting resume, or displaying
progress bars.
Interpreter Reuse
~~~~~~~~~~~~~~~~~
Python interpreters are aggressively reused to execute modules. While this
works well, it violates an unwritten assumption, and so it is possible an
earlier module execution could cause a subsequent module to fail, or for
unrelated modules to interact poorly due to bad hygiene, such as
monkey-patching that becomes stacked over repeat invocations.
Before reporting a bug relating to a misbehaving module, please re-run with
``-e mitogen_task_isolation=fork`` to see if the problem abates. This may be
set per-task, paying attention to the possibility an earlier task may be the
true cause of a failure.
.. code-block:: yaml
- name: My task.
broken_module:
some_option: true
vars:
mitogen_task_isolation: fork
* Asynchronous jobs exist only for the duration of a run, and cannot be If forking solves your problem, **please report a bug regardless**, as an
queried by subsequent ansible-playbook invocations. Since the ability to internal list can be updated to prevent others bumping into the same problem.
query job IDs across runs relied on an implementation detail, it is not
expected this will break any real-world playbooks.
Interpreter Recycling
~~~~~~~~~~~~~~~~~~~~~
There is a per-target limit on the number of interpreters. Once 20 exist, the
youngest is terminated before starting any new interpreter, preventing
situations like below from triggering memory exhaustion.
.. code-block:: yaml
- hosts: corp_boxes
vars:
user_directory: [
# 10,000 corporate user accounts
]
tasks:
- name: Create user bashrc
become: true
vars:
ansible_become_user: "{{item}}"
copy:
src: bashrc
dest: "~{{item}}/.bashrc"
with_items: "{{user_directory}}"
The youngest is chosen to preserve useful accounts like ``root`` and
``postgresql`` that often appear early in a run, however it is simple to
construct a playbook that defeats this strategy. A future version will key
interpreters on the identity of their creating task, avoiding useful account
recycling in every scenario.
To modify the limit, set the ``MITOGEN_MAX_INTERPRETERS`` environment variable.
Standard IO
~~~~~~~~~~~
Ansible uses pseudo TTYs for most invocations to allow it to type interactive
passwords, however pseudo TTYs are disabled where standard input is required or
``sudo`` is not in use. Additionally when SSH multiplexing is enabled, a string
like ``Shared connection to localhost closed\r\n`` appears in ``stderr`` of
every invocation.
Mitogen does not naturally require either of these, as command output is always
embedded within framed messages, and it can simply call :py:func:`pty.openpty`
in any location an interactive password must be typed.
A major downside to Ansible's behaviour is that ``stdout`` and ``stderr`` are
merged together into a single ``stdout`` variable, with carriage returns
inserted in the output by the TTY layer. However ugly, the extension emulates
this precisely, to avoid breaking playbooks that expect text to appear in
specific variables with a particular linefeed style.
How Modules Execute How Modules Execute
------------------- ~~~~~~~~~~~~~~~~~~~
Ansible usually modifies, recompresses and reuploads modules every time they Ansible usually modifies, recompresses and reuploads modules every time they
run on a target, work that must be repeated by the controller for every run on a target, work that must be repeated by the controller for every
@ -218,47 +394,51 @@ cached in RAM for the remainder of the run.
key2=repr(value2)[ ..]] "``. key2=repr(value2)[ ..]] "``.
Sample Profiles Runtime Patches
--------------- ~~~~~~~~~~~~~~~
Local VM connection
~~~~~~~~~~~~~~~~~~~
This demonstrates Mitogen vs. connection pipelining to a local VM, executing Three small runtime patches are employed in ``strategy.py`` to hook into
the 100 simple repeated steps of ``run_hostname_100_times.yml`` from the desirable locations, in order to override uses of shell, the module executor,
examples directory. Mitogen requires **43x less bandwidth and 4.25x less and the mechanism for selecting a connection plug-in. While it is hoped the
time**. patches can be avoided in future, for interesting versions of Ansible deployed
today this simply is not possible, and so they continue to be required.
.. image:: images/ansible/run_hostname_100_times.png The patches are concise and behave conservatively, including by disabling
themselves when non-Mitogen connections are in use. Additional third party
plug-ins are unlikely to attempt similar patches, so the risk to an established
configuration should be minimal.
Kathmandu to Paris Flag Emulation
~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~
This is a full Django application playbook over a ~180ms link between Kathmandu Mitogen re-parses ``sudo_flags``, ``become_flags``, and ``ssh_flags`` using
and Paris. Aside from large pauses where the host performs useful work, the option parsers extracted from `sudo(1)` and `ssh(1)` in order to emulate their
high latency of this link means Mitogen only manages a 1.7x speedup. equivalent semantics. This allows:
Many early roundtrips are due to inefficiencies in Mitogen's importer that will * robust support for common ``ansible.cfg`` tricks without reconfiguration,
be fixed over time, however the majority, comprising at least 10 seconds, are such as forwarding SSH agents across ``sudo`` invocations,
due to idling while the host's previous result and next command are in-flight * reporting on conflicting flag combinations,
on the network. * reporting on unsupported flag combinations,
* internally special-casing certain behaviour (like recursive agent forwarding)
without boring the user with the details,
* avoiding opening the extension up to untestable scenarios where users can
insert arbitrary garbage between Mitogen and the components it integrates
with,
* precise emulation by an alternative implementation, for example if Mitogen
grew support for Paramiko.
The initial extension lays groundwork for exciting structural changes to the
execution model: a future version will tackle latency head-on by delegating
some control flow to the target host, melding the performance and scalability
benefits of pull-based operation with the management simplicity of push-based
operation.
.. image:: images/ansible/costapp.png Supported Variables
-------------------
Matching Ansible's model, variables are treated on a per-task basis, causing
establishment of additional reuseable interpreters as necessary to match the
configuration of each task.
SSH Variables
-------------
Matching Ansible's existing model, these variables are treated on a per-task SSH
basis, causing establishment of additional reuseable interpreters as necessary ~~~
to match the configuration of each task.
This list will grow as more missing pieces are discovered. This list will grow as more missing pieces are discovered.
@ -272,8 +452,8 @@ This list will grow as more missing pieces are discovered.
* ``ssh_args``, ``ssh_common_args``, ``ssh_extra_args`` * ``ssh_args``, ``ssh_common_args``, ``ssh_extra_args``
Sudo Variables Sudo
-------------- ~~~~
* ``ansible_python_interpreter`` * ``ansible_python_interpreter``
* ``ansible_sudo_exe``, ``ansible_become_exe`` * ``ansible_sudo_exe``, ``ansible_become_exe``
@ -283,31 +463,23 @@ Sudo Variables
* ansible.cfg: ``timeout`` * ansible.cfg: ``timeout``
Docker Variables Docker
---------------- ~~~~~~
Note: Docker support is only intended for developer testing, it might disappear
entirely prior to a stable release.
* ansible_host Docker support has received relatively little testing, expect increased
probability of surprises for the time being.
* ``ansible_host``
Chat on IRC
-----------
Some users and developers hang out on the
`#mitogen <https://webchat.freenode.net/?channels=mitogen>`_ channel on the
FreeNode IRC network.
Debugging Debugging
--------- ---------
Normally with Ansible, diagnostics and use of the :py:mod:`logging` package Diagnostics and use of the :py:mod:`logging` package output on the target
output on the target machine are discarded. With Mitogen, all of this is machine are usually discarded. With Mitogen, all of this is captured and
captured and returned to the host machine, where it can be viewed as desired returned to the controller, where it can be viewed as desired with ``-vvv``.
with ``-vvv``. Basic high level logs are produced with ``-vvv``, with logging Basic high level logs are produced with ``-vvv``, with logging of all IO on the
of all IO on the controller with ``-vvvv`` or higher. controller with ``-vvvv`` or higher.
Although use of standard IO and the logging package on the target is forwarded Although use of standard IO and the logging package on the target is forwarded
to the controller, it is not possible to receive IO activity logs, as the to the controller, it is not possible to receive IO activity logs, as the
@ -320,129 +492,45 @@ When file-based logging is enabled, one file per context will be created on the
local machine and every target machine, as ``/tmp/mitogen.<pid>.log``. local machine and every target machine, as ``/tmp/mitogen.<pid>.log``.
Implementation Notes Getting Help
-------------------- ~~~~~~~~~~~~
Some users and developers hang out on the
Interpreter Reuse `#mitogen <https://webchat.freenode.net/?channels=mitogen>`_ channel on the
~~~~~~~~~~~~~~~~~ FreeNode IRC network.
The extension aggressively reuses the single target Python interpreter to
execute every module. While this generally works well, it violates an unwritten
assumption regarding Ansible modules, and so it is possible a buggy module
could cause a run to fail, or for unrelated modules to interact with each other
due to bad hygiene.
Before reporting a bug relating to a module behaving incorrectly, please re-run
your playbook with ``-e mitogen_task_isolation=fork`` to see if the problem
abates. This may also be set on a per-task basis:
::
- name: My task.
broken_module:
some_option: true
vars:
mitogen_task_isolation: fork
If forking fixes your problem, **please report a bug regardless**, as an
internal list can be updated to prevent users bumping into the same problem in
future.
Interpreter Recycling
~~~~~~~~~~~~~~~~~~~~~
The extension limits the number of persistent interpreters in use. When the
limit is reached, the youngest interpreter is terminated before starting a new
interpreter, preventing situations like below from triggering memory
exhaustion.
.. code-block:: yaml
- hosts: corp_boxes
vars:
user_directory: [
# 10,000 corporate user accounts
]
tasks:
- name: Create user bashrc
become: true
vars:
ansible_become_user: "{{item}}"
copy:
src: bashrc
dest: "~{{item}}/.bashrc"
with_items: "{{user_directory}}"
This recycling does not occur for direct connections from the controller, and
it is keyed on a per-target basis, i.e. up to 20 interpreters may exist for
each directly connected target.
The youngest interpreter is chosen to preserve useful accounts, like "root" or
"postgresql" that tend to appear early in a run, however it is simple to
construct a playbook that defeats this strategy. A future version will key
interpreters on the identity of their creating task, file and/or playbook,
avoiding useful account recycling in every scenario.
To raise or lower the limit from 20, set the ``MITOGEN_MAX_INTERPRETERS``
environment variable to a new value.
Runtime Patches
~~~~~~~~~~~~~~~
Three small runtime patches are employed in ``strategy.py`` to hook into
desirable locations, in order to override uses of shell, the module executor,
and the mechanism for selecting a connection plug-in. While it is hoped the
patches can be avoided in future, for interesting versions of Ansible deployed
today this simply is not possible, and so they continue to be required.
The patches are concise and behave conservatively, including by disabling
themselves when non-Mitogen connections are in use. Additional third party
plug-ins are unlikely to attempt similar patches, so the risk to an established
configuration should be minimal.
Sample Profiles
---------------
Standard IO Local VM connection
~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~
Ansible uses pseudo TTYs for most invocations, to allow it to handle typing This demonstrates Mitogen vs. connection pipelining to a local VM, executing
passwords interactively, however it disables pseudo TTYs for certain commands the 100 simple repeated steps of ``run_hostname_100_times.yml`` from the
where standard input is required or ``sudo`` is not in use. Additionally when examples directory. Mitogen requires **43x less bandwidth and 4.25x less
SSH multiplexing is enabled, a string like ``Shared connection to localhost time**.
closed\r\n`` appears in ``stderr`` of every invocation.
Mitogen does not naturally require either of these, as command output is .. image:: images/ansible/run_hostname_100_times.png
embedded within the SSH stream, and it can simply call :py:func:`pty.openpty`
in every location an interactive password must be typed.
A major downside to Ansible's behaviour is that ``stdout`` and ``stderr`` are
merged together into a single ``stdout`` variable, with carriage returns
inserted in the output by the TTY layer. However ugly, the extension emulates
all of this behaviour precisely, to avoid breaking playbooks that expect
certain text to appear in certain variables with certain linefeed characters.
See `Ansible#14377`_ for related discussion. Kathmandu to Paris
~~~~~~~~~~~~~~~~~~
.. _Ansible#14377: https://github.com/ansible/ansible/issues/14377 This is a full Django application playbook over a ~180ms link between Kathmandu
and Paris. Aside from large pauses where the host performs useful work, the
high latency of this link means Mitogen only manages a 1.7x speedup.
Many early roundtrips are due to inefficiencies in Mitogen's importer that will
be fixed over time, however the majority, comprising at least 10 seconds, are
due to idling while the host's previous result and next command are in-flight
on the network.
Flag Emulation The initial extension lays groundwork for exciting structural changes to the
~~~~~~~~~~~~~~ execution model: a future version will tackle latency head-on by delegating
some control flow to the target host, melding the performance and scalability
benefits of pull-based operation with the management simplicity of push-based
operation.
Mitogen re-parses ``sudo_flags``, ``become_flags``, and ``ssh_flags`` using .. image:: images/ansible/costapp.png
option parsers extracted from `sudo(1)` and `ssh(1)` in order to emulate their
equivalent semantics. This allows:
* robust support for common ``ansible.cfg`` tricks without reconfiguration,
such as forwarding SSH agents across ``sudo`` invocations,
* reporting on conflicting flag combinations,
* reporting on unsupported flag combinations,
* internally special-casing certain behaviour (like recursive agent forwarding)
without boring the user with the details,
* avoiding opening the extension up to untestable scenarios where users can
insert arbitrary garbage between Mitogen and the components it integrates
with,
* precise emulation by an alternative implementation, for example if Mitogen
grew support for Paramiko.

Write Preview
Loading…
Cancel
Save