Commit Graph

3878 Commits (dad28e8b4a1436b09a2ae7e0a8cc12a4490f495c)
 

Author SHA1 Message Date
Marc Hartmayer dad28e8b4a tests: Add a test case that verifies behavior when the log record factory is modified
The test currently fails with the following error:

  $ PYTHONPATH=$(pwd)/tests:$PYTHONPATH python3 -m unittest -v log_handler_test
  ...
  test_logrecordfactory (log_handler_test.LogRecordFactoryTest.test_logrecordfactory) ... --- Logging error ---
  Traceback (most recent call last):
    File "/usr/lib/python3.12/logging/__init__.py", line 464, in format
      return self._format(record)
             ^^^^^^^^^^^^^^^^^^^^
    File "/usr/lib/python3.12/logging/__init__.py", line 460, in _format
      return self._fmt % values
             ~~~~~~~~~~^~~~~~~~
  KeyError: 'custom_attribute'

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/usr/lib/python3.12/logging/__init__.py", line 1160, in emit
      msg = self.format(record)
            ^^^^^^^^^^^^^^^^^^^
    File "/usr/lib/python3.12/logging/__init__.py", line 999, in format
      return fmt.format(record)
             ^^^^^^^^^^^^^^^^^^
    File "/usr/lib/python3.12/logging/__init__.py", line 999, in format
      return fmt.format(record)
             ^^^^^^^^^^^^^^^^^^
    File "/usr/lib/python3.12/logging/__init__.py", line 706, in format
      s = self.formatMessage(record)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/lib/python3.12/logging/__init__.py", line 675, in formatMessage
      return self._style.format(record)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/lib/python3.12/logging/__init__.py", line 466, in format
      raise ValueError('Formatting field not found in record: %s' % e)
  ValueError: Formatting field not found in record: 'custom_attribute'

Signed-off-by: Marc Hartmayer <mhartmay@linux.ibm.com>
1 month ago
Alex Willmer 3c648f7df8
Merge pull request #1345 from moreati/oi-mate-wheres-your-loicense
Fix and formalise license metadata
1 month ago
Alex Willmer 01baec8347 Declare license as SPDX identifier in metadata
Fixes warning seen during packaging operations

```
➜  mitogen git:(master) ✗ uv build --sdist
Building source distribution...
...
!!

        ********************************************************************************
        Please consider removing the following classifiers in favor of a SPDX license expression:

        License :: OSI Approved :: BSD License

        See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license for details.
        ********************************************************************************

!!
  self._finalize_license_expression()
running egg_info
...
```
1 month ago
Alex Willmer 85f0c33dc5 Correct mitogen.imports.* licenses 1 month ago
Alex Willmer 8f66aa5fcd
Merge pull request #1248 from moreati/sys.path
ansible_mitogen: De-duplicate sys.path manipulations by Ansible plugins
1 month ago
Alex Willmer 6145508312 ansible_mitogen: De-duplicate sys.path manipulations by Ansible plugins 1 month ago
Alex Willmer 9701424be0
Merge pull request #1266 from moreati/imports-cleanup
Imports cleanup
1 month ago
Alex Willmer 7d5f63ccbf Cleanup unused and missing imports 1 month ago
Alex Willmer 3338a651a6
Merge pull request #1339 from moreati/prepare-v0.3.29
Prepare v0.3.29
3 months ago
Alex Willmer 6071fb58c9 Begin 0.3.30dev 3 months ago
Alex Willmer e670bf0ebd Prepare v0.3.29 3 months ago
Alex Willmer f6451bf795
Merge pull request #1287 from moreati/issue1242-py3.14
Python 3.14 support
3 months ago
Alex Willmer 090952a987 Python 3.14 support 3 months ago
Alex Willmer d27275ad46 ci: Set global max failed logins on macOS 3 months ago
Alex Willmer 1b00ca2581 tests: Bump dependency versions 3 months ago
Alex Willmer f4f646a00a
Merge pull request #1337 from moreati/prepare-v0.3.28
Prepare v0.3.28
3 months ago
Alex Willmer b03c1f3d87 Begin 0.3.29dev 3 months ago
Alex Willmer 9f9b37d1ad Prepare v0.3.28 3 months ago
Alex Willmer f6902dd05d
Merge pull request #1336 from Nihlus/freeipa-fixes
Add FreeIPA client modules to the always-fork list
3 months ago
Alex Willmer 2736f38c4b docs: Changelog for FreeIPA client modules -> ALWAYS_FORK_MODULES 3 months ago
Jarl Gullberg 59d5d74abd
Add FreeIPA client modules to the always-fork list. 3 months ago
Alex Willmer 36569792bc
Merge pull request #1307 from moreati/issue1306-investigate
mitogen: Fix non-blocking IO errors in first stage of bootstrap
3 months ago
Alex Willmer 85d6046f2f mitogen: Fix non-blocking IO errors in first stage of bootstrap
When /etc/sudoers has log_output (or similar) enabled the process spawned by
`ctx.sudo()` via `mitogen.parent.Connection.start_child()` receives a stdin
that is in non-blocking mode. The immediate symptom is that `os.openfd(0,
...).read(n)` sometimes returns `None`, causing the first stage to raise an
unhandled TypeError.

The fix (for now) is to use `select.select()` in a while loop to read stdin.
This increases the command size slightly, but I think it's a reasonable
tradeoff until/unless the cause is more fully understood.

All CI tests are now run with sudoers log_output enabled, in order to catch
regressions. `first_stage_test.CommandLineTest` has been amended, because it
relied on implementation details of the bootstrap process that are no longer
true.

Before
```
SSH command size: 755
Preamble (mitogen.core + econtext) size: 18227 (17.80KiB)

                        Original           Minimized           Compressed
mitogen.core         152218 148.7KiB  68437 66.8KiB 45.0%  18124 17.7KiB 11.9%
mitogen.parent        98853  96.5KiB  51103 49.9KiB 51.7%  12881 12.6KiB 13.0%
mitogen.fork           8445   8.2KiB   4139  4.0KiB 49.0%   1652  1.6KiB 19.6%
mitogen.ssh           10827  10.6KiB   6893  6.7KiB 63.7%   2099  2.0KiB 19.4%
mitogen.sudo          12089  11.8KiB   5924  5.8KiB 49.0%   2249  2.2KiB 18.6%
mitogen.select        12325  12.0KiB   2929  2.9KiB 23.8%    964  0.9KiB  7.8%
mitogen.service       41581  40.6KiB  22398 21.9KiB 53.9%   5847  5.7KiB 14.1%
mitogen.fakessh       15767  15.4KiB   8149  8.0KiB 51.7%   2676  2.6KiB 17.0%
mitogen.master        55317  54.0KiB  28846 28.2KiB 52.1%   7528  7.4KiB 13.6%
```

After
```
SSH command size: 798
Preamble (mitogen.core + econtext) size: 18227 (17.80KiB)

                        Original           Minimized           Compressed
mitogen.core         152218 148.7KiB  68437 66.8KiB 45.0%  18124 17.7KiB 11.9%
mitogen.parent        98944  96.6KiB  51180 50.0KiB 51.7%  12910 12.6KiB 13.0%
mitogen.fork           8445   8.2KiB   4139  4.0KiB 49.0%   1652  1.6KiB 19.6%
mitogen.ssh           10827  10.6KiB   6893  6.7KiB 63.7%   2099  2.0KiB 19.4%
mitogen.sudo          12089  11.8KiB   5924  5.8KiB 49.0%   2249  2.2KiB 18.6%
mitogen.select        12325  12.0KiB   2929  2.9KiB 23.8%    964  0.9KiB  7.8%
mitogen.service       41581  40.6KiB  22398 21.9KiB 53.9%   5847  5.7KiB 14.1%
mitogen.fakessh       15767  15.4KiB   8149  8.0KiB 51.7%   2676  2.6KiB 17.0%
mitogen.master        55317  54.0KiB  28846 28.2KiB 52.1%   7528  7.4KiB 13.6%
```
3 months ago
Alex Willmer c508bfb58b tests: Check stdio is blocking in sudo contexts
refs #712
3 months ago
Alex Willmer 76f6eb741d tests: Count bytes written in stdio_test.StdIOTest
This is mainly for peace of mind. With all this non-blocking IO investigation
I'm getting a bit paranoid wrt file objects.

refs #712
3 months ago
Alex Willmer 3dfaf83ce7 preamble_size: Fix variability of command & preamble(?) size
Previously the command size could very depanding on the current username, hostname, and process pid.

Before
```
SSH command size: 759
Preamble (mitogen.core + econtext) size: 18227 (17.80KiB)
...
```

After
SSH command size: 755
Preamble (mitogen.core + econtext) size: 18227 (17.80KiB)
...
```
3 months ago
Alex Willmer 936b08dd08 preamble_size: Include mitogen.core and clarify bootstrap size
After:
SSH command size: 759
Preamble (mitogen.core + econtext) size: 18227 (17.80KiB)

                        Original           Minimized           Compressed
mitogen.core         152218 148.7KiB  68437 66.8KiB 45.0%  18124 17.7KiB 11.9%
mitogen.parent        98853  96.5KiB  51103 49.9KiB 51.7%  12881 12.6KiB 13.0%
mitogen.fork           8445   8.2KiB   4139  4.0KiB 49.0%   1652  1.6KiB 19.6%
mitogen.ssh           10827  10.6KiB   6893  6.7KiB 63.7%   2099  2.0KiB 19.4%
mitogen.sudo          12089  11.8KiB   5924  5.8KiB 49.0%   2249  2.2KiB 18.6%
mitogen.select        12325  12.0KiB   2929  2.9KiB 23.8%    964  0.9KiB  7.8%
mitogen.service       41581  40.6KiB  22398 21.9KiB 53.9%   5847  5.7KiB 14.1%
mitogen.fakessh       15767  15.4KiB   8149  8.0KiB 51.7%   2676  2.6KiB 17.0%
mitogen.master        55317  54.0KiB  28846 28.2KiB 52.1%   7528  7.4KiB 13.6%
3 months ago
Alex Willmer 30d8a38a3b preamble_size: Consolidate table formatting, align columns better
Before
./preamble_size.py
SSH command size: 759
Bootstrap (mitogen.core) size: 18227 (17.80KiB)

                              Original          Minimized           Compressed
mitogen.parent            98853 96.5KiB  51103 49.9KiB 51.7%  12881 12.6KiB 13.0%
mitogen.fork               8445  8.2KiB   4139  4.0KiB 49.0%   1652  1.6KiB 19.6%
mitogen.ssh               10827 10.6KiB   6893  6.7KiB 63.7%   2099  2.0KiB 19.4%
mitogen.sudo              12089 11.8KiB   5924  5.8KiB 49.0%   2249  2.2KiB 18.6%
mitogen.select            12325 12.0KiB   2929  2.9KiB 23.8%    964  0.9KiB 7.8%
mitogen.service           41581 40.6KiB  22398 21.9KiB 53.9%   5847  5.7KiB 14.1%
mitogen.fakessh           15767 15.4KiB   8149  8.0KiB 51.7%   2676  2.6KiB 17.0%
mitogen.master            55317 54.0KiB  28846 28.2KiB 52.1%   7528  7.4KiB 13.6%

After:
SSH command size: 759
Bootstrap (mitogen.core) size: 18227 (17.80KiB)

                        Original           Minimized           Compressed
mitogen.parent        98853  96.5KiB  51103 49.9KiB 51.7%  12881 12.6KiB 13.0%
mitogen.fork           8445   8.2KiB   4139  4.0KiB 49.0%   1652  1.6KiB 19.6%
mitogen.ssh           10827  10.6KiB   6893  6.7KiB 63.7%   2099  2.0KiB 19.4%
mitogen.sudo          12089  11.8KiB   5924  5.8KiB 49.0%   2249  2.2KiB 18.6%
mitogen.select        12325  12.0KiB   2929  2.9KiB 23.8%    964  0.9KiB  7.8%
mitogen.service       41581  40.6KiB  22398 21.9KiB 53.9%   5847  5.7KiB 14.1%
mitogen.fakessh       15767  15.4KiB   8149  8.0KiB 51.7%   2676  2.6KiB 17.0%
mitogen.master        55317  54.0KiB  28846 28.2KiB 52.1%   7528  7.4KiB 13.6%
3 months ago
Alex Willmer e4e2c6caaf CI: Move sudo test users defaults into /etc/sudoers.d
Prep for reusing it in non-Ansible tests
3 months ago
Alex Willmer 5abdde1117 CI: Report sudo version on Ansible targets 3 months ago
Alex Willmer dc7fae973b CI: Fix ci_lib and test_lib have_<cmd>() when <cmd> exits abnormally
We were not raising CalledProcessError when exit status != 0.
3 months ago
Alex Willmer 885c6de65e
Merge pull request #1331 from moreati/prep-v0.3.27
Prep v0.3.27
3 months ago
Alex Willmer 07d1078010 Begin v0.3.28dev 4 months ago
Alex Willmer 154331e455 Prepare v0.3.27 4 months ago
Alex Willmer b8d3f86b12
Merge pull request #1328 from moreati/issue1325-scan_code_imports-refactor
mitogen: Refactor `mitogen.master.scan_code_imports()` -> `mitogen.imports.codeobj_imports()`
4 months ago
Alex Willmer 0e5f47f145 mitogen: Refactor scan_code_imports() as mitogen.imports.codeobj_imports()
This replaces `mitogen.master.scan_code_imports()` with
`mitogen.imports.codeobj_imports()`. The Python 3.x implementation now uses
`str.find()`, relying on Python >= 3.6 "widecode" format. Behaviour and
semantics should be unchanged. Now implementations are approx
- 1.5 x faster on Python 2.x
- 2 - 3 x faster on Python 3.x

Before
```console
$ ./tests/bench/scan_code
scan_code_imports python2.7  100 loops, best of 3: 3.19 msec per loop
scan_code_imports python3.9  500 loops, best of 5: 685 usec per loop
scan_code_imports python3.10  500 loops, best of 5: 727 usec per loop
scan_code_imports python3.11  500 loops, best of 5: 601 usec per loop
scan_code_imports python3.12  500 loops, best of 5: 609 usec per loop
scan_code_imports python3.13  500 loops, best of 5: 586 usec per loop
```

After
```console
codeobj_imports python2.7  1000 loops, best of 3: 1.98 msec per loop
codeobj_imports python3.9  1000 loops, best of 5: 302 usec per loop
codeobj_imports python3.10  1000 loops, best of 5: 297 usec per loop
codeobj_imports python3.11  1000 loops, best of 5: 243 usec per loop
codeobj_imports python3.12  1000 loops, best of 5: 278 usec per loop
codeobj_imports python3.13  1000 loops, best of 5: 259 usec per loop
```
```console
$ uname -a
Darwin kintha 24.6.0 Darwin Kernel Version 24.6.0: Mon Jul 14 11:30:29 PDT
2025; root:xnu-11417.140.69~1/RELEASE_ARM64_T6000 arm64
```
4 months ago
Alex Willmer 3093d0bb2d tests: Add scan_code_imports benchmark
```console
$ ./tests/bench/scan_code
scan_code_imports python2.7  100 loops, best of 3: 3.19 msec per loop
scan_code_imports python3.9  500 loops, best of 5: 685 usec per loop
scan_code_imports python3.10  500 loops, best of 5: 727 usec per loop
scan_code_imports python3.11  500 loops, best of 5: 601 usec per loop
scan_code_imports python3.12  500 loops, best of 5: 609 usec per loop
scan_code_imports python3.13  500 loops, best of 5: 586 usec per loop
```
4 months ago
Alex Willmer 2fd88298ae tests: Improve master_test.ScanCodeImportsTest coverage
This covers existing behaviours of `mitogen.master.scan_code_imports()` some
of which are relied on, some not, but regardless weren't tested. Notably
- Explicit relative imports return level > 0
- Imports inside `class` and `def` are excluded
- Imports inside other blocks are included
- Python 3.x prunes impossible if/else branches (previously unknown)

It also
- Decouples the test results from the implementation details of the unit test.
- Fixes a missing import
- Fixes at least one Python 2.4 incompatibility (use of with block)
4 months ago
Alex Willmer 1386529493
Merge pull request #1330 from moreati/gha-workflow-cleanup
CI: Refactor and de-duplicate Github Actions workflow
4 months ago
Alex Willmer 618eccc0f3 CI: Set macOS failed logins limit of mitogen test users to 1000
refs #1315
4 months ago
Alex Willmer 9e3377c0a8 CI: Combine build deps & tooling steps 4 months ago
Alex Willmer e3241912f7 CI: Factor out .ci/show_python_versions 4 months ago
Alex Willmer 9b6fc117f9 CI: Remove unused python_version vars from Ubuntu 22.04 jobs 4 months ago
Alex Willmer 4cad51a629
Merge pull request #1320 from moreati/prepare-v0.3.26
Prepare v0.3.26
4 months ago
Alex Willmer 7fb7567809 Begin v0.3.27dev 4 months ago
Alex Willmer 5908936f8c Prepare v0.3.26 4 months ago
Alex Willmer 64feda250e
Merge pull request #1300 from moreati/issue712-stdout-non-blocking
stdio EAGAIN investigation
4 months ago
Alex Willmer 17bee70dc2 mitogen: Fix BlockingIOError & EAGAIN in subprocess stdio
Mitogen was leaving the stdout and stderr of subprocesses in non-blocking
mode. When Python code ran in the remote process created by Mitogen calls such
as `print(long_string)` or `os.stout.write(bigger_than_the_buffer)` sometimes
raised `BlockingIOError`, or similar.

This change
- Removes code in `mitogen.core.Side` that set blocking/non-blocking mode
- Adds blocking/non-blocking control to `os.mitogen.pipe()` and a new
  function `mitogen.core.socketpair()`
- Replaces `mitogen.core.set_block` and `mitogen.core.set_nonblock`
  with `mitogen.core.set_blocking`, mirroring `os.set_blocking`
- Updates call sites as appropriate
- Adds tests for new functions and arguments
- Adds a regression test for subprocess stdio blocking/non-blocking

fixes #712
4 months ago
Alex Willmer 4529a217e8
Merge pull request #1301 from moreati/repear-delay-simplify
mitogen: Simplify `mitogen.parent.Reaper._calc_delay()` calculation
4 months ago
Alex Willmer fde2dda87e mitogen: Simplify `mitogen.parent.Reaper._calc_delay()` calculation 4 months ago