diff --git a/docs/ansible.rst b/docs/ansible.rst index c8b0d12f..205e5e3c 100644 --- a/docs/ansible.rst +++ b/docs/ansible.rst @@ -921,6 +921,154 @@ logging is necessary. File-based logging can be enabled by setting enabled, one file per context will be created on the local machine and every target machine, as ``/tmp/mitogen..log``. + +Common Problems +~~~~~~~~~~~~~~~ + +The most common bug reports fall into the following categories, so it is worth +checking whether you can categorize a problem using the tools provided before +reporting it: + +**Missed/Incorrect Configuration Variables** + In some cases Ansible may support a configuration variable that Mitogen + does not yet support, or Mitogen supports, but the support is broken. For + example, Mitogen may pick the wrong username or SSH parameters. + + To detect this, use the special ``mitogen_get_stack`` action described + below to verify all the configuration variables Mitogen has chosen for the + connection make sense. + +**Process Environment Differences** + Mitogen's process model differs significantly to Ansible's in certain + places. In the past, bugs have been reported because Ansible plug-ins + modify an environment variable after Mitogen processes are started + +**Variable Expansion Differences** + To avoid many classes of bugs, Mitogen avoids shell wherever possible. + Ansible however is traditionally built on shell, and it is often difficult + to tell just how many times a configuration parameter will pass through + shell expansion and quoting, and in what context before it is used. + + Due to this, in some circumstances Mitogen may parse some expanded + variables differently, for example, in the wrong user account. Careful + review of ``-vvv`` and ``mitogen_ssh_debug_level`` logs can reveal this. + For example in the past, Mitogen used a different method of expanding + ``~/.ssh/id_rsa``, causing authentication to fail when ``ansible-playbook`` + was run via ``sudo -E``. + +**External Tool Integration Differences** + Mitogen reimplements any aspect of Ansible that involves integrating with + SSH, sudo, Docker, or related tools. For this reason, sometimes its support + for those tools doffers or is less mature than in Ansible. + + In the past Mitogen has had bug reports due to failing to recognize a + particular variation of a login or password prompt on an exotic or + non-English operating system, or confusing a login banner for a password + prompt. Careful review of ``-vvv`` logs help identify these cases, as + Mitogen logs all strings it receives during connection, and how it + interprets them. + + +.. _mitogen-get-stack: + +The `mitogen_get_stack` Action +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When a Mitogen strategy is loaded, a special ``mitogen_get_stack`` action is +available that returns a concise description of the connection configuration as +extracted from Ansible and passed to the core library. Using it, you can learn +whether a problem lies in the Ansible extension or deeper in library code. + +The action may be used in a playbook as ``mitogen_get_stack:`` just like a +regular module, or directly from the command-line:: + + $ ANSIBLE_STRATEGY=mitogen_linear ansible -m mitogen_get_stack -b -k k3 + SSH password: + k3 | SUCCESS => { + "changed": true, + "result": [ + { + "kwargs": { + "check_host_keys": "enforce", + "connect_timeout": 10, + "hostname": "k3", + "identities_only": false, + "identity_file": null, + "password": "mysecretpassword", + "port": null, + "python_path": null, + "ssh_args": [ + "-C", + "-o", + "ControlMaster=auto", + "-o", + "ControlPersist=60s" + ], + "ssh_debug_level": null, + "ssh_path": "ssh", + "username": null + }, + "method": "ssh" + }, + { + "enable_lru": true, + "kwargs": { + "connect_timeout": 10, + "password": null, + "python_path": null, + "sudo_args": [ + "-H", + "-S", + "-n" + ], + "sudo_path": null, + "username": "root" + }, + "method": "sudo" + } + ] + } + +Each object in the list represents a single 'hop' in the connection, from +nearest to furthest. Unlike in Ansible, the core library treats ``become`` +steps and SSH steps identically, so they are represented distinctly in the +output. + +The presence of ``null`` means no explicit value was extracted from Ansible, +and either the Mitogen library or SSH will choose a value for the parameter. In +the example above, Mitogen will choose ``/usr/bin/python`` for ``python_path``, +and SSH will choose ``22`` for ``port``, or whatever ``Port`` it parses from +``~/.ssh/config``. Note the presence of ``null`` may indicate the extension +failed to extract the correct value. + +When using ``mitogen_get_stack`` to diagnose a problem, pay special attention +to ensuring the invocation exactly matches the problematic task. For example, +if the failing task has ``delegate_to:`` or ``become:`` enabled, the +``mitogen_get_stack`` invocation must include those statements in order for the +output to be accurate. + +If a playbook cannot start at all, you may need to temporarily use +``gather_facts: no`` to allow the first task to proceed. This action does not +create connections, so if it is the first task, it is still possible to review +its output. + + +The `mitogen_ssh_debug_level` Variable +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Mitogen has support for capturing SSH diagnostic logs, and integrating them +into the regular debug log output produced when ``-vvv`` is active. This +provides a single audit trail of every component active during SSH +authentication. + +Particularly for authentication failures, setting this variable to 3, in +combination with ``-vvv``, allows review of every parameter passed to SSH, and +review of every action SSH attempted during authentication. + +For example, this method can be used to ascertain whether SSH attempted agent +authentication, or what private key files it was able to access and which it tried. + + .. _diagnosing-hangs: Diagnosing Hangs diff --git a/docs/changelog.rst b/docs/changelog.rst index 3c7350b5..95123a69 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -154,6 +154,11 @@ Enhancements ``mitogen_host_pinned`` strategy wraps the ``host_pinned`` strategy introduced in Ansible 2.7. +* `#412 `_: to simplify diagnosing + issues with connection configuration, Mitogen ships with a + ``mitogen_get_stack`` action that is automatically added to the action + plug-in path. See :ref:`mitogen-get-stack` for more information. + * `#415 `_: the interface employed for in-process queues was changed from `kqueue `_ / @@ -164,6 +169,7 @@ Enhancements a runtime improvement in many-host runs. + Fixes ^^^^^ @@ -210,6 +216,11 @@ Fixes * `#410 `_: the sudo method supports the SELinux ``--type`` and ``--role`` options. +* `#412 `_: connection delegation and + ``delegate_to:`` handling suffered a major regression in 0.2.3. The 0.2.2 + behaviour has been restored, and further work has been made to improve the + compatibility of connection delegation's configuration building methods. + * `#420 `_: if a :class:`Connection` was constructed in the Ansible top-level process, for example while executing ``meta: reset_connection``, resources could become undesirably shared in