Previously the command size could very depanding on the current username, hostname, and process pid.
Before
```
SSH command size: 759
Preamble (mitogen.core + econtext) size: 18227 (17.80KiB)
...
```
After
SSH command size: 755
Preamble (mitogen.core + econtext) size: 18227 (17.80KiB)
...
```
This replaces `mitogen.master.scan_code_imports()` with
`mitogen.imports.codeobj_imports()`. The Python 3.x implementation now uses
`str.find()`, relying on Python >= 3.6 "widecode" format. Behaviour and
semantics should be unchanged. Now implementations are approx
- 1.5 x faster on Python 2.x
- 2 - 3 x faster on Python 3.x
Before
```console
$ ./tests/bench/scan_code
scan_code_imports python2.7 100 loops, best of 3: 3.19 msec per loop
scan_code_imports python3.9 500 loops, best of 5: 685 usec per loop
scan_code_imports python3.10 500 loops, best of 5: 727 usec per loop
scan_code_imports python3.11 500 loops, best of 5: 601 usec per loop
scan_code_imports python3.12 500 loops, best of 5: 609 usec per loop
scan_code_imports python3.13 500 loops, best of 5: 586 usec per loop
```
After
```console
codeobj_imports python2.7 1000 loops, best of 3: 1.98 msec per loop
codeobj_imports python3.9 1000 loops, best of 5: 302 usec per loop
codeobj_imports python3.10 1000 loops, best of 5: 297 usec per loop
codeobj_imports python3.11 1000 loops, best of 5: 243 usec per loop
codeobj_imports python3.12 1000 loops, best of 5: 278 usec per loop
codeobj_imports python3.13 1000 loops, best of 5: 259 usec per loop
```
```console
$ uname -a
Darwin kintha 24.6.0 Darwin Kernel Version 24.6.0: Mon Jul 14 11:30:29 PDT
2025; root:xnu-11417.140.69~1/RELEASE_ARM64_T6000 arm64
```
This covers existing behaviours of `mitogen.master.scan_code_imports()` some
of which are relied on, some not, but regardless weren't tested. Notably
- Explicit relative imports return level > 0
- Imports inside `class` and `def` are excluded
- Imports inside other blocks are included
- Python 3.x prunes impossible if/else branches (previously unknown)
It also
- Decouples the test results from the implementation details of the unit test.
- Fixes a missing import
- Fixes at least one Python 2.4 incompatibility (use of with block)
Mitogen was leaving the stdout and stderr of subprocesses in non-blocking
mode. When Python code ran in the remote process created by Mitogen calls such
as `print(long_string)` or `os.stout.write(bigger_than_the_buffer)` sometimes
raised `BlockingIOError`, or similar.
This change
- Removes code in `mitogen.core.Side` that set blocking/non-blocking mode
- Adds blocking/non-blocking control to `os.mitogen.pipe()` and a new
function `mitogen.core.socketpair()`
- Replaces `mitogen.core.set_block` and `mitogen.core.set_nonblock`
with `mitogen.core.set_blocking`, mirroring `os.set_blocking`
- Updates call sites as appropriate
- Adds tests for new functions and arguments
- Adds a regression test for subprocess stdio blocking/non-blocking
fixes#712
This is to prevent job names being truncated in the Github Actions web UI. So
it is obvious at a glance which jobs have failed. Previously one had to click
into the details to know which job was which, leading to confusion and wasted
time.
This also
- removes braced ranges in `testenv.setenv`. They appear not to be supported
by tox (see https://github.com/tox-dev/tox/issues/3571)
- fixes the env var `DEFAULT_STDOUT_CALLBACK` -> `ANSIBLE_STDOUT_CALLBACK`
as a result of these test output format was previously not as intended for
some Ansible versions.
In vanilla Ansible >= 12 (ansible-core 2.19)
- ssh connection plugin `verbosity` controls `ssh [-v[v[v]]]`
- config option `DEFAULT_VERBOSITY` controls whether that output is displayed
In vanilla Ansible <= 11 (ansible-core <= 2.18)
- `DEFAULT_VERBOSITY` controls both `ssh` verbosity & display verbositty
As of this change
- Mitogen + Ansible >= 12 behaviour matches vanilla Ansible >= 12.
- Mitogen + Ansible <= 11 behaviour remains unchanged
- `DEFAULT_VERBOSITY` only controls display verbosity.
- Mitogen + Ansible respect the Ansible variable `mitogen_ssh_debug_level`
I've chosen not to retroactively replicate the old vanilla Ansible behaviour
in Mitogen + Ansible <= 11 cases. I'm pretty sure it was an oversight,
rather than a design choice, but Ansible+Mitogen with `ANSIBLE_VERBOSITY=3`
is already very verbose.
fixes#1282
See
- https://docs.ansible.com/ansible/latest/reference_appendices/config.html#default-verbosity
- https://docs.ansible.com/ansible/devel/collections/ansible/builtin/ssh_connection.html#parameter-verbosity
Ansible 12 (ansible-core 2.19) has gained support for specifying an SSH
password, without requiring `sshpass`. It specifies the environment variable
`SSH_ASKPASS` such that `ansible` itself is called.
Mitogen is already able to support this. This change provides test coverage of
the new feature by not installing `sshpass` on macOS runners. when Ansible 12
is under test. Ubuntu runners come with `sshpass` pre-installed.
Required Ansible is also bumped to the latest pre-releases, for relevant
fixes.
Note that tests/ansible/integration/ssh/templated_by_play_taskvar.yml was
previously erroniously being skipped with ansible-core 2.19.0a<N> and
2.19.0b<N>.
fixes#1293
refs #1175
Ansible >= 12 (ansible-core >= 2.19) deprecates `stdout_callback=yaml`,
superceded by `callback_result_format=yaml`. There is a change in behaviour:
`callback_result_format` applies to output of both `ansible-playbook` _and_
`ansible`.
Tests that run `ansible` in a subprocess are now explicitly configured to use
json (even if they don't inspect that output yet) for more assert-able output
across all versions of Ansible.
The function is Ansible >= 12 (ansible-core >= 2.19). See #1274 for analysis
of `json.dumps()` vs `jsonify()` differences. This change is a middle ground
between full backward compatibility and using `json.dumps()` unadorned.
- if `data` is `None`, then it will still be transferred as `{}` on older
versions of Ansible, but 'null' in newer releases. Cases where 'null'
caused a problem are suspected/reported, but no reproducers are available.
- `ensure_ascii=True` will be still be tried, with fallback. I believe this
is only relevant on Python 2.x.
- `sort_keys=True` will no longer be used.
- No indentation/pretty printing will be applied, this remains unchanged
fixes#1274
Python 2.7 (distro package) and 3.6 (pyenv managed) jobs run on Ubuntu 22.04.
More recent Pythons (distro or Github provided) run on 24.04.
fixes#1256
Ansible tasks that run locally (e.g. `connection: local`, `delegate_to:
localhost`) must now specify their `ansible_python_interpreter`, typically as
`{{ ansible_playbook_python }}`; otherwise the system Python on the controller
(e.g. `/usr/bin/python`) is likely to be used and this is often outside the
version range supported by the Ansible verison under test. If this occurs then
the symptom is often a failure to import a builtin from
`ansible.module_utils.six.moves`, e.g.
```
fatal: [target-centos6-1]: FAILED! => changed=true
cmd:
- ansible
- -m
- shell
- -c
- local
- -a
- whoami
- -i
- /tmp/mitogen_ci_ansibled3llejls/hosts
- test-targets
delta: '0:00:02.076385'
end: '2025-04-17 17:27:02.561500'
msg: non-zero return code
rc: 8
start: '2025-04-17 17:27:00.485115'
stderr: |-
stderr_lines: <omitted>
stdout: |-
An exception occurred during task execution. To see the full traceback,
use -vvv. The error was: from ansible.module_utils.six.moves import
map, reduce, shlex_quote
```
Each grouping gets an independant dir, e.g.
- ansible -> /tmp/mitogen_ci_ansible
- debops -> /tmp/mitogen_ci_debops
Importing ci_lib no longer creates a temporary directory as a side effect.
Alex Willmer