Commit Graph

3949 Commits (ce0c5967005167ebad7bbf2e16f266fe477f817e)
 

Author SHA1 Message Date
Alex Willmer ce0c596700
Merge pull request #1338 from moreati/release-v0.3.28
Release v0.3.28
3 months ago
Alex Willmer 2791abe17a Merge commit '9f9b37d' into release-v0.3.28 3 months ago
Alex Willmer 9f9b37d1ad Prepare v0.3.28 3 months ago
Alex Willmer f6902dd05d
Merge pull request #1336 from Nihlus/freeipa-fixes
Add FreeIPA client modules to the always-fork list
3 months ago
Alex Willmer 2736f38c4b docs: Changelog for FreeIPA client modules -> ALWAYS_FORK_MODULES 3 months ago
Jarl Gullberg 59d5d74abd
Add FreeIPA client modules to the always-fork list. 3 months ago
Alex Willmer 36569792bc
Merge pull request #1307 from moreati/issue1306-investigate
mitogen: Fix non-blocking IO errors in first stage of bootstrap
3 months ago
Alex Willmer 85d6046f2f mitogen: Fix non-blocking IO errors in first stage of bootstrap
When /etc/sudoers has log_output (or similar) enabled the process spawned by
`ctx.sudo()` via `mitogen.parent.Connection.start_child()` receives a stdin
that is in non-blocking mode. The immediate symptom is that `os.openfd(0,
...).read(n)` sometimes returns `None`, causing the first stage to raise an
unhandled TypeError.

The fix (for now) is to use `select.select()` in a while loop to read stdin.
This increases the command size slightly, but I think it's a reasonable
tradeoff until/unless the cause is more fully understood.

All CI tests are now run with sudoers log_output enabled, in order to catch
regressions. `first_stage_test.CommandLineTest` has been amended, because it
relied on implementation details of the bootstrap process that are no longer
true.

Before
```
SSH command size: 755
Preamble (mitogen.core + econtext) size: 18227 (17.80KiB)

                        Original           Minimized           Compressed
mitogen.core         152218 148.7KiB  68437 66.8KiB 45.0%  18124 17.7KiB 11.9%
mitogen.parent        98853  96.5KiB  51103 49.9KiB 51.7%  12881 12.6KiB 13.0%
mitogen.fork           8445   8.2KiB   4139  4.0KiB 49.0%   1652  1.6KiB 19.6%
mitogen.ssh           10827  10.6KiB   6893  6.7KiB 63.7%   2099  2.0KiB 19.4%
mitogen.sudo          12089  11.8KiB   5924  5.8KiB 49.0%   2249  2.2KiB 18.6%
mitogen.select        12325  12.0KiB   2929  2.9KiB 23.8%    964  0.9KiB  7.8%
mitogen.service       41581  40.6KiB  22398 21.9KiB 53.9%   5847  5.7KiB 14.1%
mitogen.fakessh       15767  15.4KiB   8149  8.0KiB 51.7%   2676  2.6KiB 17.0%
mitogen.master        55317  54.0KiB  28846 28.2KiB 52.1%   7528  7.4KiB 13.6%
```

After
```
SSH command size: 798
Preamble (mitogen.core + econtext) size: 18227 (17.80KiB)

                        Original           Minimized           Compressed
mitogen.core         152218 148.7KiB  68437 66.8KiB 45.0%  18124 17.7KiB 11.9%
mitogen.parent        98944  96.6KiB  51180 50.0KiB 51.7%  12910 12.6KiB 13.0%
mitogen.fork           8445   8.2KiB   4139  4.0KiB 49.0%   1652  1.6KiB 19.6%
mitogen.ssh           10827  10.6KiB   6893  6.7KiB 63.7%   2099  2.0KiB 19.4%
mitogen.sudo          12089  11.8KiB   5924  5.8KiB 49.0%   2249  2.2KiB 18.6%
mitogen.select        12325  12.0KiB   2929  2.9KiB 23.8%    964  0.9KiB  7.8%
mitogen.service       41581  40.6KiB  22398 21.9KiB 53.9%   5847  5.7KiB 14.1%
mitogen.fakessh       15767  15.4KiB   8149  8.0KiB 51.7%   2676  2.6KiB 17.0%
mitogen.master        55317  54.0KiB  28846 28.2KiB 52.1%   7528  7.4KiB 13.6%
```
3 months ago
Alex Willmer c508bfb58b tests: Check stdio is blocking in sudo contexts
refs #712
3 months ago
Alex Willmer 76f6eb741d tests: Count bytes written in stdio_test.StdIOTest
This is mainly for peace of mind. With all this non-blocking IO investigation
I'm getting a bit paranoid wrt file objects.

refs #712
3 months ago
Alex Willmer 3dfaf83ce7 preamble_size: Fix variability of command & preamble(?) size
Previously the command size could very depanding on the current username, hostname, and process pid.

Before
```
SSH command size: 759
Preamble (mitogen.core + econtext) size: 18227 (17.80KiB)
...
```

After
SSH command size: 755
Preamble (mitogen.core + econtext) size: 18227 (17.80KiB)
...
```
3 months ago
Alex Willmer 936b08dd08 preamble_size: Include mitogen.core and clarify bootstrap size
After:
SSH command size: 759
Preamble (mitogen.core + econtext) size: 18227 (17.80KiB)

                        Original           Minimized           Compressed
mitogen.core         152218 148.7KiB  68437 66.8KiB 45.0%  18124 17.7KiB 11.9%
mitogen.parent        98853  96.5KiB  51103 49.9KiB 51.7%  12881 12.6KiB 13.0%
mitogen.fork           8445   8.2KiB   4139  4.0KiB 49.0%   1652  1.6KiB 19.6%
mitogen.ssh           10827  10.6KiB   6893  6.7KiB 63.7%   2099  2.0KiB 19.4%
mitogen.sudo          12089  11.8KiB   5924  5.8KiB 49.0%   2249  2.2KiB 18.6%
mitogen.select        12325  12.0KiB   2929  2.9KiB 23.8%    964  0.9KiB  7.8%
mitogen.service       41581  40.6KiB  22398 21.9KiB 53.9%   5847  5.7KiB 14.1%
mitogen.fakessh       15767  15.4KiB   8149  8.0KiB 51.7%   2676  2.6KiB 17.0%
mitogen.master        55317  54.0KiB  28846 28.2KiB 52.1%   7528  7.4KiB 13.6%
3 months ago
Alex Willmer 30d8a38a3b preamble_size: Consolidate table formatting, align columns better
Before
./preamble_size.py
SSH command size: 759
Bootstrap (mitogen.core) size: 18227 (17.80KiB)

                              Original          Minimized           Compressed
mitogen.parent            98853 96.5KiB  51103 49.9KiB 51.7%  12881 12.6KiB 13.0%
mitogen.fork               8445  8.2KiB   4139  4.0KiB 49.0%   1652  1.6KiB 19.6%
mitogen.ssh               10827 10.6KiB   6893  6.7KiB 63.7%   2099  2.0KiB 19.4%
mitogen.sudo              12089 11.8KiB   5924  5.8KiB 49.0%   2249  2.2KiB 18.6%
mitogen.select            12325 12.0KiB   2929  2.9KiB 23.8%    964  0.9KiB 7.8%
mitogen.service           41581 40.6KiB  22398 21.9KiB 53.9%   5847  5.7KiB 14.1%
mitogen.fakessh           15767 15.4KiB   8149  8.0KiB 51.7%   2676  2.6KiB 17.0%
mitogen.master            55317 54.0KiB  28846 28.2KiB 52.1%   7528  7.4KiB 13.6%

After:
SSH command size: 759
Bootstrap (mitogen.core) size: 18227 (17.80KiB)

                        Original           Minimized           Compressed
mitogen.parent        98853  96.5KiB  51103 49.9KiB 51.7%  12881 12.6KiB 13.0%
mitogen.fork           8445   8.2KiB   4139  4.0KiB 49.0%   1652  1.6KiB 19.6%
mitogen.ssh           10827  10.6KiB   6893  6.7KiB 63.7%   2099  2.0KiB 19.4%
mitogen.sudo          12089  11.8KiB   5924  5.8KiB 49.0%   2249  2.2KiB 18.6%
mitogen.select        12325  12.0KiB   2929  2.9KiB 23.8%    964  0.9KiB  7.8%
mitogen.service       41581  40.6KiB  22398 21.9KiB 53.9%   5847  5.7KiB 14.1%
mitogen.fakessh       15767  15.4KiB   8149  8.0KiB 51.7%   2676  2.6KiB 17.0%
mitogen.master        55317  54.0KiB  28846 28.2KiB 52.1%   7528  7.4KiB 13.6%
3 months ago
Alex Willmer e4e2c6caaf CI: Move sudo test users defaults into /etc/sudoers.d
Prep for reusing it in non-Ansible tests
3 months ago
Alex Willmer 5abdde1117 CI: Report sudo version on Ansible targets 3 months ago
Alex Willmer dc7fae973b CI: Fix ci_lib and test_lib have_<cmd>() when <cmd> exits abnormally
We were not raising CalledProcessError when exit status != 0.
3 months ago
Alex Willmer ced63f96d9
Merge pull request #1332 from moreati/release-v0.3.27
Release v0.3.27
3 months ago
Alex Willmer 885c6de65e
Merge pull request #1331 from moreati/prep-v0.3.27
Prep v0.3.27
3 months ago
Alex Willmer cbf47967b1 Merge commit '154331e455a3153252e44fe41a9e0b3d08ccf477' into release-v0.3.27 3 months ago
Alex Willmer 07d1078010 Begin v0.3.28dev 3 months ago
Alex Willmer 154331e455 Prepare v0.3.27 3 months ago
Alex Willmer b8d3f86b12
Merge pull request #1328 from moreati/issue1325-scan_code_imports-refactor
mitogen: Refactor `mitogen.master.scan_code_imports()` -> `mitogen.imports.codeobj_imports()`
3 months ago
Alex Willmer 0e5f47f145 mitogen: Refactor scan_code_imports() as mitogen.imports.codeobj_imports()
This replaces `mitogen.master.scan_code_imports()` with
`mitogen.imports.codeobj_imports()`. The Python 3.x implementation now uses
`str.find()`, relying on Python >= 3.6 "widecode" format. Behaviour and
semantics should be unchanged. Now implementations are approx
- 1.5 x faster on Python 2.x
- 2 - 3 x faster on Python 3.x

Before
```console
$ ./tests/bench/scan_code
scan_code_imports python2.7  100 loops, best of 3: 3.19 msec per loop
scan_code_imports python3.9  500 loops, best of 5: 685 usec per loop
scan_code_imports python3.10  500 loops, best of 5: 727 usec per loop
scan_code_imports python3.11  500 loops, best of 5: 601 usec per loop
scan_code_imports python3.12  500 loops, best of 5: 609 usec per loop
scan_code_imports python3.13  500 loops, best of 5: 586 usec per loop
```

After
```console
codeobj_imports python2.7  1000 loops, best of 3: 1.98 msec per loop
codeobj_imports python3.9  1000 loops, best of 5: 302 usec per loop
codeobj_imports python3.10  1000 loops, best of 5: 297 usec per loop
codeobj_imports python3.11  1000 loops, best of 5: 243 usec per loop
codeobj_imports python3.12  1000 loops, best of 5: 278 usec per loop
codeobj_imports python3.13  1000 loops, best of 5: 259 usec per loop
```
```console
$ uname -a
Darwin kintha 24.6.0 Darwin Kernel Version 24.6.0: Mon Jul 14 11:30:29 PDT
2025; root:xnu-11417.140.69~1/RELEASE_ARM64_T6000 arm64
```
4 months ago
Alex Willmer 3093d0bb2d tests: Add scan_code_imports benchmark
```console
$ ./tests/bench/scan_code
scan_code_imports python2.7  100 loops, best of 3: 3.19 msec per loop
scan_code_imports python3.9  500 loops, best of 5: 685 usec per loop
scan_code_imports python3.10  500 loops, best of 5: 727 usec per loop
scan_code_imports python3.11  500 loops, best of 5: 601 usec per loop
scan_code_imports python3.12  500 loops, best of 5: 609 usec per loop
scan_code_imports python3.13  500 loops, best of 5: 586 usec per loop
```
4 months ago
Alex Willmer 2fd88298ae tests: Improve master_test.ScanCodeImportsTest coverage
This covers existing behaviours of `mitogen.master.scan_code_imports()` some
of which are relied on, some not, but regardless weren't tested. Notably
- Explicit relative imports return level > 0
- Imports inside `class` and `def` are excluded
- Imports inside other blocks are included
- Python 3.x prunes impossible if/else branches (previously unknown)

It also
- Decouples the test results from the implementation details of the unit test.
- Fixes a missing import
- Fixes at least one Python 2.4 incompatibility (use of with block)
4 months ago
Alex Willmer 1386529493
Merge pull request #1330 from moreati/gha-workflow-cleanup
CI: Refactor and de-duplicate Github Actions workflow
4 months ago
Alex Willmer 618eccc0f3 CI: Set macOS failed logins limit of mitogen test users to 1000
refs #1315
4 months ago
Alex Willmer 9e3377c0a8 CI: Combine build deps & tooling steps 4 months ago
Alex Willmer e3241912f7 CI: Factor out .ci/show_python_versions 4 months ago
Alex Willmer 9b6fc117f9 CI: Remove unused python_version vars from Ubuntu 22.04 jobs 4 months ago
Alex Willmer f77db1da59
Merge pull request #1321 from moreati/release-v0.3.26
Release v0.3.26
4 months ago
Alex Willmer 4cad51a629
Merge pull request #1320 from moreati/prepare-v0.3.26
Prepare v0.3.26
4 months ago
Alex Willmer fd1d45568a Merge commit '5908936' into release-v0.3.26 4 months ago
Alex Willmer 7fb7567809 Begin v0.3.27dev 4 months ago
Alex Willmer 5908936f8c Prepare v0.3.26 4 months ago
Alex Willmer 64feda250e
Merge pull request #1300 from moreati/issue712-stdout-non-blocking
stdio EAGAIN investigation
4 months ago
Alex Willmer 17bee70dc2 mitogen: Fix BlockingIOError & EAGAIN in subprocess stdio
Mitogen was leaving the stdout and stderr of subprocesses in non-blocking
mode. When Python code ran in the remote process created by Mitogen calls such
as `print(long_string)` or `os.stout.write(bigger_than_the_buffer)` sometimes
raised `BlockingIOError`, or similar.

This change
- Removes code in `mitogen.core.Side` that set blocking/non-blocking mode
- Adds blocking/non-blocking control to `os.mitogen.pipe()` and a new
  function `mitogen.core.socketpair()`
- Replaces `mitogen.core.set_block` and `mitogen.core.set_nonblock`
  with `mitogen.core.set_blocking`, mirroring `os.set_blocking`
- Updates call sites as appropriate
- Adds tests for new functions and arguments
- Adds a regression test for subprocess stdio blocking/non-blocking

fixes #712
4 months ago
Alex Willmer 4529a217e8
Merge pull request #1301 from moreati/repear-delay-simplify
mitogen: Simplify `mitogen.parent.Reaper._calc_delay()` calculation
4 months ago
Alex Willmer fde2dda87e mitogen: Simplify `mitogen.parent.Reaper._calc_delay()` calculation 4 months ago
Alex Willmer 65db935c57
Merge pull request #1311 from moreati/issue1309
ansible_mitogen: Fix `become_method=doas`, add tests
4 months ago
Alex Willmer 868d77a402 ansible_mitogen: Fix become_method=doas, add tests 4 months ago
Alex Willmer d6e74ad663
Merge pull request #1319 from moreati/ci-job-names
CI: Abbreviate Github Actions job names
4 months ago
Alex Willmer 32f6d0c358 CI: Abbreviate Github Actions job names
This is to prevent job names being truncated in the Github Actions web UI. So
it is obvious at a glance which jobs have failed. Previously one had to click
into the details to know which job was which, leading to confusion and wasted
time.

This also

- removes braced ranges in `testenv.setenv`. They appear not to be supported
  by tox (see https://github.com/tox-dev/tox/issues/3571)
- fixes the env var `DEFAULT_STDOUT_CALLBACK` -> `ANSIBLE_STDOUT_CALLBACK`

as a result of these test output format was previously not as intended for
some Ansible versions.
4 months ago
Alex Willmer 021d712edc
Merge pull request #1316 from moreati/release-v0.3.25
Release v0.3.25
4 months ago
Alex Willmer 536ab7d78e Merge commit 'dfafa14' into release-v0.3.25 4 months ago
Alex Willmer 53ab2b2a4d
Merge pull request #1313 from moreati/prep-0.3.25
Prep 0.3.25
4 months ago
Alex Willmer 7f84874755 Begin 0.3.26dev 4 months ago
Alex Willmer dfafa1430e Prepare v0.3.25 4 months ago
Alex Willmer d240a78af3
Merge pull request #1308 from moreati/prep-v0.3.25b1
Prepare v0.3.25b1
4 months ago
Alex Willmer fa22aa1c8e Resume 0.3.25dev 4 months ago