Loading the ansible_mitogen Ansible plugins apparently doesn't follow the same
rules as importing a Python module. So sys.path manipulations in __init__.py
weren't fired when Ansible tried to load the plugins from a /custom/path that
wasn't already on sys.path.
This wasn't picked up by the test because CI always installs Mitogen as a
Python package (in a virtual env).
This reverses 6145508312.
This replaces `mitogen.master.scan_code_imports()` with
`mitogen.imports.codeobj_imports()`. The Python 3.x implementation now uses
`str.find()`, relying on Python >= 3.6 "widecode" format. Behaviour and
semantics should be unchanged. Now implementations are approx
- 1.5 x faster on Python 2.x
- 2 - 3 x faster on Python 3.x
Before
```console
$ ./tests/bench/scan_code
scan_code_imports python2.7 100 loops, best of 3: 3.19 msec per loop
scan_code_imports python3.9 500 loops, best of 5: 685 usec per loop
scan_code_imports python3.10 500 loops, best of 5: 727 usec per loop
scan_code_imports python3.11 500 loops, best of 5: 601 usec per loop
scan_code_imports python3.12 500 loops, best of 5: 609 usec per loop
scan_code_imports python3.13 500 loops, best of 5: 586 usec per loop
```
After
```console
codeobj_imports python2.7 1000 loops, best of 3: 1.98 msec per loop
codeobj_imports python3.9 1000 loops, best of 5: 302 usec per loop
codeobj_imports python3.10 1000 loops, best of 5: 297 usec per loop
codeobj_imports python3.11 1000 loops, best of 5: 243 usec per loop
codeobj_imports python3.12 1000 loops, best of 5: 278 usec per loop
codeobj_imports python3.13 1000 loops, best of 5: 259 usec per loop
```
```console
$ uname -a
Darwin kintha 24.6.0 Darwin Kernel Version 24.6.0: Mon Jul 14 11:30:29 PDT
2025; root:xnu-11417.140.69~1/RELEASE_ARM64_T6000 arm64
```
Mitogen was leaving the stdout and stderr of subprocesses in non-blocking
mode. When Python code ran in the remote process created by Mitogen calls such
as `print(long_string)` or `os.stout.write(bigger_than_the_buffer)` sometimes
raised `BlockingIOError`, or similar.
This change
- Removes code in `mitogen.core.Side` that set blocking/non-blocking mode
- Adds blocking/non-blocking control to `os.mitogen.pipe()` and a new
function `mitogen.core.socketpair()`
- Replaces `mitogen.core.set_block` and `mitogen.core.set_nonblock`
with `mitogen.core.set_blocking`, mirroring `os.set_blocking`
- Updates call sites as appropriate
- Adds tests for new functions and arguments
- Adds a regression test for subprocess stdio blocking/non-blocking
fixes#712
In vanilla Ansible >= 12 (ansible-core 2.19)
- ssh connection plugin `verbosity` controls `ssh [-v[v[v]]]`
- config option `DEFAULT_VERBOSITY` controls whether that output is displayed
In vanilla Ansible <= 11 (ansible-core <= 2.18)
- `DEFAULT_VERBOSITY` controls both `ssh` verbosity & display verbositty
As of this change
- Mitogen + Ansible >= 12 behaviour matches vanilla Ansible >= 12.
- Mitogen + Ansible <= 11 behaviour remains unchanged
- `DEFAULT_VERBOSITY` only controls display verbosity.
- Mitogen + Ansible respect the Ansible variable `mitogen_ssh_debug_level`
I've chosen not to retroactively replicate the old vanilla Ansible behaviour
in Mitogen + Ansible <= 11 cases. I'm pretty sure it was an oversight,
rather than a design choice, but Ansible+Mitogen with `ANSIBLE_VERBOSITY=3`
is already very verbose.
fixes#1282
See
- https://docs.ansible.com/ansible/latest/reference_appendices/config.html#default-verbosity
- https://docs.ansible.com/ansible/devel/collections/ansible/builtin/ssh_connection.html#parameter-verbosity
The function is Ansible >= 12 (ansible-core >= 2.19). See #1274 for analysis
of `json.dumps()` vs `jsonify()` differences. This change is a middle ground
between full backward compatibility and using `json.dumps()` unadorned.
- if `data` is `None`, then it will still be transferred as `{}` on older
versions of Ansible, but 'null' in newer releases. Cases where 'null'
caused a problem are suspected/reported, but no reproducers are available.
- `ensure_ascii=True` will be still be tried, with fallback. I believe this
is only relevant on Python 2.x.
- `sort_keys=True` will no longer be used.
- No indentation/pretty printing will be applied, this remains unchanged
fixes#1274
This adapts PR #740 by @extmind (afe0026890),
which augmented the call to `Connection.get_task_var()` with
`C.config.get_config_value('INTERPRETER_PYTHON'` as a default. Instead this
*replaces* the call to `Connection.get_task_var()`. The aim is greater
simplicity by disentangling templating of a configured interpreter path and
discovery of an interpreter when none is configured. I think this also reduces
the number of times `Connection._get_task_vars()` is called, so reducing the
number of times we do the ugly stack frame inspection.
I've also added test cases.
Co-authored-by: Lars Beckers <lars@extmind.de>
Ansible >= 4 (ansible-core >= 2.11) the SSH plugin has a `timeout` option and
with variable `ansible_ssh_timeout`, but not a `ansible_timeout` variable.
The local plugin has no such option or variable(s). However `ansible_timeout`
is backfilled for all conection plugins, by legacy mechanisms that populate
the play context attribute:
- `ansible.constants.COMMON_CONNECTION_VARS`
- `ansible.constants.MAGIC_VARIABLE_MAPPING`
The `timeout` keyword is for task completion timeout, not connection timeout.
The previous commit (53b4881628 in PR 1200) was
not intended to change these values, but some WIP slipped through. This
partially reverts that commit so the two changes (moving the monkey patch,
making the monkey patch more capable) exist in distinct commits.
This tightens up our monkey patching `Connection._action` so it's only applied
during `meta: reset_connection` & promptly removed. This fixes "'int' object
has no attribute 'template'" when `ansible.plugins.action.wait_for_connection`
or other code calls `ansible.plugins.connection.ConnectionBase.reset()`.
This could also have switched to `templar=templar` on the temporary action,
rather than `templar=0`, but it's not strictly necessary to fix this bug. I
anticipate other changes doing so soon, to improve interpreter discovery &
templated python interpreter path support.
A twist - for the connection option "host" the corresponding legacy
PlayContext attribute is PlayContext.remote_addr. This may be the only case
where a connection option name and the PlayContext attribute name differ.
Adding a the tt-ssh-executable test target uncovered an Ansible bug during
`meta: reset_connection` tasks. So this commit includes a workaround for
affected versions of Ansible.
Uses the same fallback for (mitogen_sudo et al) as become_exe (see #1173).
The new `Spec.become_flags()` is not yet explicitly tested. Note that it
returns a string (matching the Ansible option of the same name), whereas
`Spec.sudo_args()` returns a list.
refs #1083
Some ansible_mitogen connection plugins look more like become plugins (e.g.
mitogen_sudo) & use become plugin options. For now there's special handling in
PlayContextSpec._become_option(). Further design/discussion can go in #1173.
Refs #1087.
This reads the become username from the `become_user` attribute of the play
context, to the `"become_user"` option of the loaded become plugin. This has
been supported by vanilla Ansible since Ansible 2.10 (ansible-base 2.10).
To support this I've also switched from using the `play_context.become` (a
bool), to `connection.become` (an instance of the appropriate) become plugin.
New tests have been added, modelled on those for templated connection
parameters (see #1147, #1153, #1159).
See
- 480b106d65
refs #1083
Co-authored-by: mordek <m.pirog@bonasoft.pl>
This switches `ansible_mitogen.transport_config.PlayContextSpec.password()` to
Ansible's plugin option framework. As a result
- The relatively recent `ansible_ssh_password` variable is now respected.
- The SSH connection password can be templated and specified as a play
variable. Task variables will probably also work, but testing was blocked
by #1132.
There is a chance this change will cause a regression in another connection
plugin (e.g. mitogen_docker), but nothing turned up in the test suite.
I intend ot migrate other connection configuration to
`ansible_mitogen.transport_config.PlayContextSpec._connect_option()`, the next
candidate is the remote port.
fixes#1106
Rough guidelines, in decending preference:
- Use mitogen.core if possible
- Use ansible.module_utils.six if possible
- Embed a getattr() or try/except
viewkeys() et al can't be brought into mitogen.core because that package still
targets Python 2.4. dict.viewkeys() were introduced in Python 2.7.
Alex Willmer