Commit Graph

3872 Commits (5b04049ce204b524994dd76ef3a4c214ffb9d9eb)
 

Author SHA1 Message Date
Marc Hartmayer 5b04049ce2 mitogen/core: Unpickle user defined classes from the (trusted) parent
The parent has other possibilities than doing attacks via malicious messages.
Therefore it's okay to trust the messages from the parents and use unsecure
unpickling methods.

Signed-off-by: Marc Hartmayer <mhartmay@linux.ibm.com>
2 months ago
Marc Hartmayer 5490cd74b4 tests: add test for passing user defined classes and instances from parent -> child
It's well known that unpickling data received from an untrusted source is not
secure. But since children trust their parents and ancestors, unpickling data
received from a parent or ancestor should be supported. Add a test for this use
case.

Signed-off-by: Marc Hartmayer <mhartmay@linux.ibm.com>
2 months ago
Alex Willmer 3338a651a6
Merge pull request #1339 from moreati/prepare-v0.3.29
Prepare v0.3.29
4 months ago
Alex Willmer 6071fb58c9 Begin 0.3.30dev 4 months ago
Alex Willmer e670bf0ebd Prepare v0.3.29 4 months ago
Alex Willmer f6451bf795
Merge pull request #1287 from moreati/issue1242-py3.14
Python 3.14 support
4 months ago
Alex Willmer 090952a987 Python 3.14 support 4 months ago
Alex Willmer d27275ad46 ci: Set global max failed logins on macOS 4 months ago
Alex Willmer 1b00ca2581 tests: Bump dependency versions 4 months ago
Alex Willmer f4f646a00a
Merge pull request #1337 from moreati/prepare-v0.3.28
Prepare v0.3.28
4 months ago
Alex Willmer b03c1f3d87 Begin 0.3.29dev 4 months ago
Alex Willmer 9f9b37d1ad Prepare v0.3.28 4 months ago
Alex Willmer f6902dd05d
Merge pull request #1336 from Nihlus/freeipa-fixes
Add FreeIPA client modules to the always-fork list
4 months ago
Alex Willmer 2736f38c4b docs: Changelog for FreeIPA client modules -> ALWAYS_FORK_MODULES 4 months ago
Jarl Gullberg 59d5d74abd
Add FreeIPA client modules to the always-fork list. 4 months ago
Alex Willmer 36569792bc
Merge pull request #1307 from moreati/issue1306-investigate
mitogen: Fix non-blocking IO errors in first stage of bootstrap
5 months ago
Alex Willmer 85d6046f2f mitogen: Fix non-blocking IO errors in first stage of bootstrap
When /etc/sudoers has log_output (or similar) enabled the process spawned by
`ctx.sudo()` via `mitogen.parent.Connection.start_child()` receives a stdin
that is in non-blocking mode. The immediate symptom is that `os.openfd(0,
...).read(n)` sometimes returns `None`, causing the first stage to raise an
unhandled TypeError.

The fix (for now) is to use `select.select()` in a while loop to read stdin.
This increases the command size slightly, but I think it's a reasonable
tradeoff until/unless the cause is more fully understood.

All CI tests are now run with sudoers log_output enabled, in order to catch
regressions. `first_stage_test.CommandLineTest` has been amended, because it
relied on implementation details of the bootstrap process that are no longer
true.

Before
```
SSH command size: 755
Preamble (mitogen.core + econtext) size: 18227 (17.80KiB)

                        Original           Minimized           Compressed
mitogen.core         152218 148.7KiB  68437 66.8KiB 45.0%  18124 17.7KiB 11.9%
mitogen.parent        98853  96.5KiB  51103 49.9KiB 51.7%  12881 12.6KiB 13.0%
mitogen.fork           8445   8.2KiB   4139  4.0KiB 49.0%   1652  1.6KiB 19.6%
mitogen.ssh           10827  10.6KiB   6893  6.7KiB 63.7%   2099  2.0KiB 19.4%
mitogen.sudo          12089  11.8KiB   5924  5.8KiB 49.0%   2249  2.2KiB 18.6%
mitogen.select        12325  12.0KiB   2929  2.9KiB 23.8%    964  0.9KiB  7.8%
mitogen.service       41581  40.6KiB  22398 21.9KiB 53.9%   5847  5.7KiB 14.1%
mitogen.fakessh       15767  15.4KiB   8149  8.0KiB 51.7%   2676  2.6KiB 17.0%
mitogen.master        55317  54.0KiB  28846 28.2KiB 52.1%   7528  7.4KiB 13.6%
```

After
```
SSH command size: 798
Preamble (mitogen.core + econtext) size: 18227 (17.80KiB)

                        Original           Minimized           Compressed
mitogen.core         152218 148.7KiB  68437 66.8KiB 45.0%  18124 17.7KiB 11.9%
mitogen.parent        98944  96.6KiB  51180 50.0KiB 51.7%  12910 12.6KiB 13.0%
mitogen.fork           8445   8.2KiB   4139  4.0KiB 49.0%   1652  1.6KiB 19.6%
mitogen.ssh           10827  10.6KiB   6893  6.7KiB 63.7%   2099  2.0KiB 19.4%
mitogen.sudo          12089  11.8KiB   5924  5.8KiB 49.0%   2249  2.2KiB 18.6%
mitogen.select        12325  12.0KiB   2929  2.9KiB 23.8%    964  0.9KiB  7.8%
mitogen.service       41581  40.6KiB  22398 21.9KiB 53.9%   5847  5.7KiB 14.1%
mitogen.fakessh       15767  15.4KiB   8149  8.0KiB 51.7%   2676  2.6KiB 17.0%
mitogen.master        55317  54.0KiB  28846 28.2KiB 52.1%   7528  7.4KiB 13.6%
```
5 months ago
Alex Willmer c508bfb58b tests: Check stdio is blocking in sudo contexts
refs #712
5 months ago
Alex Willmer 76f6eb741d tests: Count bytes written in stdio_test.StdIOTest
This is mainly for peace of mind. With all this non-blocking IO investigation
I'm getting a bit paranoid wrt file objects.

refs #712
5 months ago
Alex Willmer 3dfaf83ce7 preamble_size: Fix variability of command & preamble(?) size
Previously the command size could very depanding on the current username, hostname, and process pid.

Before
```
SSH command size: 759
Preamble (mitogen.core + econtext) size: 18227 (17.80KiB)
...
```

After
SSH command size: 755
Preamble (mitogen.core + econtext) size: 18227 (17.80KiB)
...
```
5 months ago
Alex Willmer 936b08dd08 preamble_size: Include mitogen.core and clarify bootstrap size
After:
SSH command size: 759
Preamble (mitogen.core + econtext) size: 18227 (17.80KiB)

                        Original           Minimized           Compressed
mitogen.core         152218 148.7KiB  68437 66.8KiB 45.0%  18124 17.7KiB 11.9%
mitogen.parent        98853  96.5KiB  51103 49.9KiB 51.7%  12881 12.6KiB 13.0%
mitogen.fork           8445   8.2KiB   4139  4.0KiB 49.0%   1652  1.6KiB 19.6%
mitogen.ssh           10827  10.6KiB   6893  6.7KiB 63.7%   2099  2.0KiB 19.4%
mitogen.sudo          12089  11.8KiB   5924  5.8KiB 49.0%   2249  2.2KiB 18.6%
mitogen.select        12325  12.0KiB   2929  2.9KiB 23.8%    964  0.9KiB  7.8%
mitogen.service       41581  40.6KiB  22398 21.9KiB 53.9%   5847  5.7KiB 14.1%
mitogen.fakessh       15767  15.4KiB   8149  8.0KiB 51.7%   2676  2.6KiB 17.0%
mitogen.master        55317  54.0KiB  28846 28.2KiB 52.1%   7528  7.4KiB 13.6%
5 months ago
Alex Willmer 30d8a38a3b preamble_size: Consolidate table formatting, align columns better
Before
./preamble_size.py
SSH command size: 759
Bootstrap (mitogen.core) size: 18227 (17.80KiB)

                              Original          Minimized           Compressed
mitogen.parent            98853 96.5KiB  51103 49.9KiB 51.7%  12881 12.6KiB 13.0%
mitogen.fork               8445  8.2KiB   4139  4.0KiB 49.0%   1652  1.6KiB 19.6%
mitogen.ssh               10827 10.6KiB   6893  6.7KiB 63.7%   2099  2.0KiB 19.4%
mitogen.sudo              12089 11.8KiB   5924  5.8KiB 49.0%   2249  2.2KiB 18.6%
mitogen.select            12325 12.0KiB   2929  2.9KiB 23.8%    964  0.9KiB 7.8%
mitogen.service           41581 40.6KiB  22398 21.9KiB 53.9%   5847  5.7KiB 14.1%
mitogen.fakessh           15767 15.4KiB   8149  8.0KiB 51.7%   2676  2.6KiB 17.0%
mitogen.master            55317 54.0KiB  28846 28.2KiB 52.1%   7528  7.4KiB 13.6%

After:
SSH command size: 759
Bootstrap (mitogen.core) size: 18227 (17.80KiB)

                        Original           Minimized           Compressed
mitogen.parent        98853  96.5KiB  51103 49.9KiB 51.7%  12881 12.6KiB 13.0%
mitogen.fork           8445   8.2KiB   4139  4.0KiB 49.0%   1652  1.6KiB 19.6%
mitogen.ssh           10827  10.6KiB   6893  6.7KiB 63.7%   2099  2.0KiB 19.4%
mitogen.sudo          12089  11.8KiB   5924  5.8KiB 49.0%   2249  2.2KiB 18.6%
mitogen.select        12325  12.0KiB   2929  2.9KiB 23.8%    964  0.9KiB  7.8%
mitogen.service       41581  40.6KiB  22398 21.9KiB 53.9%   5847  5.7KiB 14.1%
mitogen.fakessh       15767  15.4KiB   8149  8.0KiB 51.7%   2676  2.6KiB 17.0%
mitogen.master        55317  54.0KiB  28846 28.2KiB 52.1%   7528  7.4KiB 13.6%
5 months ago
Alex Willmer e4e2c6caaf CI: Move sudo test users defaults into /etc/sudoers.d
Prep for reusing it in non-Ansible tests
5 months ago
Alex Willmer 5abdde1117 CI: Report sudo version on Ansible targets 5 months ago
Alex Willmer dc7fae973b CI: Fix ci_lib and test_lib have_<cmd>() when <cmd> exits abnormally
We were not raising CalledProcessError when exit status != 0.
5 months ago
Alex Willmer 885c6de65e
Merge pull request #1331 from moreati/prep-v0.3.27
Prep v0.3.27
5 months ago
Alex Willmer 07d1078010 Begin v0.3.28dev 5 months ago
Alex Willmer 154331e455 Prepare v0.3.27 5 months ago
Alex Willmer b8d3f86b12
Merge pull request #1328 from moreati/issue1325-scan_code_imports-refactor
mitogen: Refactor `mitogen.master.scan_code_imports()` -> `mitogen.imports.codeobj_imports()`
5 months ago
Alex Willmer 0e5f47f145 mitogen: Refactor scan_code_imports() as mitogen.imports.codeobj_imports()
This replaces `mitogen.master.scan_code_imports()` with
`mitogen.imports.codeobj_imports()`. The Python 3.x implementation now uses
`str.find()`, relying on Python >= 3.6 "widecode" format. Behaviour and
semantics should be unchanged. Now implementations are approx
- 1.5 x faster on Python 2.x
- 2 - 3 x faster on Python 3.x

Before
```console
$ ./tests/bench/scan_code
scan_code_imports python2.7  100 loops, best of 3: 3.19 msec per loop
scan_code_imports python3.9  500 loops, best of 5: 685 usec per loop
scan_code_imports python3.10  500 loops, best of 5: 727 usec per loop
scan_code_imports python3.11  500 loops, best of 5: 601 usec per loop
scan_code_imports python3.12  500 loops, best of 5: 609 usec per loop
scan_code_imports python3.13  500 loops, best of 5: 586 usec per loop
```

After
```console
codeobj_imports python2.7  1000 loops, best of 3: 1.98 msec per loop
codeobj_imports python3.9  1000 loops, best of 5: 302 usec per loop
codeobj_imports python3.10  1000 loops, best of 5: 297 usec per loop
codeobj_imports python3.11  1000 loops, best of 5: 243 usec per loop
codeobj_imports python3.12  1000 loops, best of 5: 278 usec per loop
codeobj_imports python3.13  1000 loops, best of 5: 259 usec per loop
```
```console
$ uname -a
Darwin kintha 24.6.0 Darwin Kernel Version 24.6.0: Mon Jul 14 11:30:29 PDT
2025; root:xnu-11417.140.69~1/RELEASE_ARM64_T6000 arm64
```
5 months ago
Alex Willmer 3093d0bb2d tests: Add scan_code_imports benchmark
```console
$ ./tests/bench/scan_code
scan_code_imports python2.7  100 loops, best of 3: 3.19 msec per loop
scan_code_imports python3.9  500 loops, best of 5: 685 usec per loop
scan_code_imports python3.10  500 loops, best of 5: 727 usec per loop
scan_code_imports python3.11  500 loops, best of 5: 601 usec per loop
scan_code_imports python3.12  500 loops, best of 5: 609 usec per loop
scan_code_imports python3.13  500 loops, best of 5: 586 usec per loop
```
5 months ago
Alex Willmer 2fd88298ae tests: Improve master_test.ScanCodeImportsTest coverage
This covers existing behaviours of `mitogen.master.scan_code_imports()` some
of which are relied on, some not, but regardless weren't tested. Notably
- Explicit relative imports return level > 0
- Imports inside `class` and `def` are excluded
- Imports inside other blocks are included
- Python 3.x prunes impossible if/else branches (previously unknown)

It also
- Decouples the test results from the implementation details of the unit test.
- Fixes a missing import
- Fixes at least one Python 2.4 incompatibility (use of with block)
5 months ago
Alex Willmer 1386529493
Merge pull request #1330 from moreati/gha-workflow-cleanup
CI: Refactor and de-duplicate Github Actions workflow
5 months ago
Alex Willmer 618eccc0f3 CI: Set macOS failed logins limit of mitogen test users to 1000
refs #1315
5 months ago
Alex Willmer 9e3377c0a8 CI: Combine build deps & tooling steps 5 months ago
Alex Willmer e3241912f7 CI: Factor out .ci/show_python_versions 5 months ago
Alex Willmer 9b6fc117f9 CI: Remove unused python_version vars from Ubuntu 22.04 jobs 5 months ago
Alex Willmer 4cad51a629
Merge pull request #1320 from moreati/prepare-v0.3.26
Prepare v0.3.26
5 months ago
Alex Willmer 7fb7567809 Begin v0.3.27dev 5 months ago
Alex Willmer 5908936f8c Prepare v0.3.26 5 months ago
Alex Willmer 64feda250e
Merge pull request #1300 from moreati/issue712-stdout-non-blocking
stdio EAGAIN investigation
5 months ago
Alex Willmer 17bee70dc2 mitogen: Fix BlockingIOError & EAGAIN in subprocess stdio
Mitogen was leaving the stdout and stderr of subprocesses in non-blocking
mode. When Python code ran in the remote process created by Mitogen calls such
as `print(long_string)` or `os.stout.write(bigger_than_the_buffer)` sometimes
raised `BlockingIOError`, or similar.

This change
- Removes code in `mitogen.core.Side` that set blocking/non-blocking mode
- Adds blocking/non-blocking control to `os.mitogen.pipe()` and a new
  function `mitogen.core.socketpair()`
- Replaces `mitogen.core.set_block` and `mitogen.core.set_nonblock`
  with `mitogen.core.set_blocking`, mirroring `os.set_blocking`
- Updates call sites as appropriate
- Adds tests for new functions and arguments
- Adds a regression test for subprocess stdio blocking/non-blocking

fixes #712
5 months ago
Alex Willmer 4529a217e8
Merge pull request #1301 from moreati/repear-delay-simplify
mitogen: Simplify `mitogen.parent.Reaper._calc_delay()` calculation
5 months ago
Alex Willmer fde2dda87e mitogen: Simplify `mitogen.parent.Reaper._calc_delay()` calculation 5 months ago
Alex Willmer 65db935c57
Merge pull request #1311 from moreati/issue1309
ansible_mitogen: Fix `become_method=doas`, add tests
5 months ago
Alex Willmer 868d77a402 ansible_mitogen: Fix become_method=doas, add tests 5 months ago
Alex Willmer d6e74ad663
Merge pull request #1319 from moreati/ci-job-names
CI: Abbreviate Github Actions job names
5 months ago
Alex Willmer 32f6d0c358 CI: Abbreviate Github Actions job names
This is to prevent job names being truncated in the Github Actions web UI. So
it is obvious at a glance which jobs have failed. Previously one had to click
into the details to know which job was which, leading to confusion and wasted
time.

This also

- removes braced ranges in `testenv.setenv`. They appear not to be supported
  by tox (see https://github.com/tox-dev/tox/issues/3571)
- fixes the env var `DEFAULT_STDOUT_CALLBACK` -> `ANSIBLE_STDOUT_CALLBACK`

as a result of these test output format was previously not as intended for
some Ansible versions.
5 months ago
Alex Willmer 53ab2b2a4d
Merge pull request #1313 from moreati/prep-0.3.25
Prep 0.3.25
5 months ago
Alex Willmer 7f84874755 Begin 0.3.26dev 5 months ago