c9eb6e54e2
Begin 0.3.34dev
2 months ago
4a442f503e
Prepare v0.3.33
2 months ago
61a7fa1fee
Begin 0.3.33dev
2 months ago
28ea4780db
Prepare v0.3.32
2 months ago
83c5ab1900
mitogen: Send first stage parameters as argv (796 bytes -> 822)
...
Benefit: The base64 lump is now static for a given Mitogen version, and the
variable parts are more visible. This will make debugging, auditting, and
allow-listing a bit easier.
Potential benefit: generate the base64 once, at build time or startup. Rather
than once per connection.
Cost: Bootstrap command is 26 bytes longer.
```
➜ mitogen git:(boot-cmd--argv) ✗ ./preamble_size.py
SSH command size: 822
Preamble (mitogen.core + econtext) size: 18230 (17.80KiB)
Original Minimized Compressed
mitogen.core 152237 148.7KiB 68453 66.8KiB 45.0% 18130 17.7KiB 11.9%
mitogen.parent 98746 96.4KiB 51215 50.0KiB 51.9% 12922 12.6KiB 13.1%
mitogen.fork 8445 8.2KiB 4139 4.0KiB 49.0% 1652 1.6KiB 19.6%
mitogen.ssh 10847 10.6KiB 6913 6.8KiB 63.7% 2102 2.1KiB 19.4%
mitogen.sudo 12089 11.8KiB 5924 5.8KiB 49.0% 2249 2.2KiB 18.6%
mitogen.select 12325 12.0KiB 2929 2.9KiB 23.8% 964 0.9KiB 7.8%
mitogen.service 41581 40.6KiB 22398 21.9KiB 53.9% 5847 5.7KiB 14.1%
mitogen.fakessh 15753 15.4KiB 8135 7.9KiB 51.6% 2672 2.6KiB 17.0%
mitogen.master 52891 51.7KiB 27586 26.9KiB 52.2% 7129 7.0KiB 13.5%
```
2 months ago
3b7a75dfaf
mitogen: Send first stage as argv (786 bytes -> 796 bytes)
...
This saves one layer of quoting/quote escaping in the bootstrap command and a
string interpolation per connection. The cost is an increasing the bootstrap
command by 10 bytes. I like the tradeoff. I could be convinced to revert it.
```console
➜ mitogen git:(boot-cmd--argv) ✗ ./preamble_size.py
SSH command size: 796
Preamble (mitogen.core + econtext) size: 18230 (17.80KiB)
Original Minimized Compressed
mitogen.core 152237 148.7KiB 68453 66.8KiB 45.0% 18130 17.7KiB 11.9%
mitogen.parent 99181 96.9KiB 51384 50.2KiB 51.8% 12956 12.7KiB 13.1%
mitogen.fork 8445 8.2KiB 4139 4.0KiB 49.0% 1652 1.6KiB 19.6%
mitogen.ssh 10847 10.6KiB 6913 6.8KiB 63.7% 2102 2.1KiB 19.4%
mitogen.sudo 12089 11.8KiB 5924 5.8KiB 49.0% 2249 2.2KiB 18.6%
mitogen.select 12325 12.0KiB 2929 2.9KiB 23.8% 964 0.9KiB 7.8%
mitogen.service 41581 40.6KiB 22398 21.9KiB 53.9% 5847 5.7KiB 14.1%
mitogen.fakessh 15753 15.4KiB 8135 7.9KiB 51.6% 2672 2.6KiB 17.0%
mitogen.master 52891 51.7KiB 27586 26.9KiB 52.2% 7129 7.0KiB 13.5%
```
2 months ago
191abd492a
mitogen: Compress first stage without header or checksum (790 bytes -> 786)
...
```console
➜ mitogen git:(boot-cmd--argv) ✗ ./preamble_size.py
SSH command size: 786
Preamble (mitogen.core + econtext) size: 18230 (17.80KiB)
Original Minimized Compressed
mitogen.core 152237 148.7KiB 68453 66.8KiB 45.0% 18130 17.7KiB 11.9%
mitogen.parent 99166 96.8KiB 51375 50.2KiB 51.8% 12957 12.7KiB 13.1%
mitogen.fork 8445 8.2KiB 4139 4.0KiB 49.0% 1652 1.6KiB 19.6%
mitogen.ssh 10847 10.6KiB 6913 6.8KiB 63.7% 2102 2.1KiB 19.4%
mitogen.sudo 12089 11.8KiB 5924 5.8KiB 49.0% 2249 2.2KiB 18.6%
mitogen.select 12325 12.0KiB 2929 2.9KiB 23.8% 964 0.9KiB 7.8%
mitogen.service 41581 40.6KiB 22398 21.9KiB 53.9% 5847 5.7KiB 14.1%
mitogen.fakessh 15753 15.4KiB 8135 7.9KiB 51.6% 2672 2.6KiB 17.0%
mitogen.master 52891 51.7KiB 27586 26.9KiB 52.2% 7129 7.0KiB 13.5%
```
Confirmed Python 2.4 supports this use of zlib.compressobj, despite lack of
mention in https://docs.python.org/2.4/lib/module-zlib.html
```pycon
Python 2.4.6 (#2 , Apr 29 2018, 11:16:24)
[GCC 7.3.0] on linux4
Type "help", "copyright", "credits" or "license" for more information.
>>> import zlib
>>> c=zlib.compressobj(zlib.Z_BEST_COMPRESSION,zlib.DEFLATED,-zlib.MAX_WBITS)
>>> c.compress('qwertyuiop') + c.flush()
'+,O-*\xa9,\xcd\xcc/\x00\x00'
```
2 months ago
408946adbe
mitogen: Golf 8 bytes from bootstrap first stage (798 -> 790)
...
Before
```
SSH command size: 798
Preamble (mitogen.core + econtext) size: 18230 (17.80KiB)
Original Minimized Compressed
mitogen.core 152237 148.7KiB 68453 66.8KiB 45.0% 18130 17.7KiB 11.9%
mitogen.parent 99020 96.7KiB 51247 50.0KiB 51.8% 12910 12.6KiB 13.0%
mitogen.fork 8445 8.2KiB 4139 4.0KiB 49.0% 1652 1.6KiB 19.6%
mitogen.ssh 10847 10.6KiB 6913 6.8KiB 63.7% 2102 2.1KiB 19.4%
mitogen.sudo 12089 11.8KiB 5924 5.8KiB 49.0% 2249 2.2KiB 18.6%
mitogen.select 12325 12.0KiB 2929 2.9KiB 23.8% 964 0.9KiB 7.8%
mitogen.service 41581 40.6KiB 22398 21.9KiB 53.9% 5847 5.7KiB 14.1%
mitogen.fakessh 15753 15.4KiB 8135 7.9KiB 51.6% 2672 2.6KiB 17.0%
mitogen.master 52891 51.7KiB 27586 26.9KiB 52.2% 7129 7.0KiB 13.5%
```
After
```
SSH command size: 790
Preamble (mitogen.core + econtext) size: 18230 (17.80KiB)
Original Minimized Compressed
mitogen.core 152237 148.7KiB 68453 66.8KiB 45.0% 18130 17.7KiB 11.9%
mitogen.parent 99020 96.7KiB 51247 50.0KiB 51.8% 12903 12.6KiB 13.0%
mitogen.fork 8445 8.2KiB 4139 4.0KiB 49.0% 1652 1.6KiB 19.6%
mitogen.ssh 10847 10.6KiB 6913 6.8KiB 63.7% 2102 2.1KiB 19.4%
mitogen.sudo 12089 11.8KiB 5924 5.8KiB 49.0% 2249 2.2KiB 18.6%
mitogen.select 12325 12.0KiB 2929 2.9KiB 23.8% 964 0.9KiB 7.8%
mitogen.service 41581 40.6KiB 22398 21.9KiB 53.9% 5847 5.7KiB 14.1%
mitogen.fakessh 15753 15.4KiB 8135 7.9KiB 51.6% 2672 2.6KiB 17.0%
mitogen.master 52891 51.7KiB 27586 26.9KiB 52.2% 7129 7.0KiB 13.5%
```
2 months ago
e4e82f53a1
Begin 0.3.32dev
2 months ago
77b7a31949
Prepare v0.3.31
2 months ago
c72acfd966
Begin v0.3.31.dev
2 months ago
1e90ff25ee
Prepare v0.3.30
2 months ago
24745183ed
master: Fix LogForwarder in case an own LogRecordFactory is used
...
Since Python 3.2 the log record factory can be changed by using
`logging.setLogRecordFactory` [1]. Therefore use `logging.makeLogRecord` as
recommended in the documentation:
"LogRecord instances are created automatically by the Logger every time
something is logged, and can be created manually via makeLogRecord() (for
example, from a pickled event received over the wire)." [2]
This fixes the test case
`log_handler_test.LogRecordFactoryTest.test_logrecordfactory`.
[1] https://docs.python.org/3/library/logging.html#logging.setLogRecordFactory
[2] https://docs.python.org/3/library/logging.html#logrecord-objects
Signed-off-by: Marc Hartmayer <mhartmay@linux.ibm.com>
2 months ago
85f0c33dc5
Correct mitogen.imports.* licenses
2 months ago
7d5f63ccbf
Cleanup unused and missing imports
2 months ago
6071fb58c9
Begin 0.3.30dev
4 months ago
e670bf0ebd
Prepare v0.3.29
4 months ago
090952a987
Python 3.14 support
4 months ago
b03c1f3d87
Begin 0.3.29dev
4 months ago
9f9b37d1ad
Prepare v0.3.28
4 months ago
85d6046f2f
mitogen: Fix non-blocking IO errors in first stage of bootstrap
...
When /etc/sudoers has log_output (or similar) enabled the process spawned by
`ctx.sudo()` via `mitogen.parent.Connection.start_child()` receives a stdin
that is in non-blocking mode. The immediate symptom is that `os.openfd(0,
...).read(n)` sometimes returns `None`, causing the first stage to raise an
unhandled TypeError.
The fix (for now) is to use `select.select()` in a while loop to read stdin.
This increases the command size slightly, but I think it's a reasonable
tradeoff until/unless the cause is more fully understood.
All CI tests are now run with sudoers log_output enabled, in order to catch
regressions. `first_stage_test.CommandLineTest` has been amended, because it
relied on implementation details of the bootstrap process that are no longer
true.
Before
```
SSH command size: 755
Preamble (mitogen.core + econtext) size: 18227 (17.80KiB)
Original Minimized Compressed
mitogen.core 152218 148.7KiB 68437 66.8KiB 45.0% 18124 17.7KiB 11.9%
mitogen.parent 98853 96.5KiB 51103 49.9KiB 51.7% 12881 12.6KiB 13.0%
mitogen.fork 8445 8.2KiB 4139 4.0KiB 49.0% 1652 1.6KiB 19.6%
mitogen.ssh 10827 10.6KiB 6893 6.7KiB 63.7% 2099 2.0KiB 19.4%
mitogen.sudo 12089 11.8KiB 5924 5.8KiB 49.0% 2249 2.2KiB 18.6%
mitogen.select 12325 12.0KiB 2929 2.9KiB 23.8% 964 0.9KiB 7.8%
mitogen.service 41581 40.6KiB 22398 21.9KiB 53.9% 5847 5.7KiB 14.1%
mitogen.fakessh 15767 15.4KiB 8149 8.0KiB 51.7% 2676 2.6KiB 17.0%
mitogen.master 55317 54.0KiB 28846 28.2KiB 52.1% 7528 7.4KiB 13.6%
```
After
```
SSH command size: 798
Preamble (mitogen.core + econtext) size: 18227 (17.80KiB)
Original Minimized Compressed
mitogen.core 152218 148.7KiB 68437 66.8KiB 45.0% 18124 17.7KiB 11.9%
mitogen.parent 98944 96.6KiB 51180 50.0KiB 51.7% 12910 12.6KiB 13.0%
mitogen.fork 8445 8.2KiB 4139 4.0KiB 49.0% 1652 1.6KiB 19.6%
mitogen.ssh 10827 10.6KiB 6893 6.7KiB 63.7% 2099 2.0KiB 19.4%
mitogen.sudo 12089 11.8KiB 5924 5.8KiB 49.0% 2249 2.2KiB 18.6%
mitogen.select 12325 12.0KiB 2929 2.9KiB 23.8% 964 0.9KiB 7.8%
mitogen.service 41581 40.6KiB 22398 21.9KiB 53.9% 5847 5.7KiB 14.1%
mitogen.fakessh 15767 15.4KiB 8149 8.0KiB 51.7% 2676 2.6KiB 17.0%
mitogen.master 55317 54.0KiB 28846 28.2KiB 52.1% 7528 7.4KiB 13.6%
```
5 months ago
07d1078010
Begin v0.3.28dev
5 months ago
154331e455
Prepare v0.3.27
5 months ago
0e5f47f145
mitogen: Refactor scan_code_imports() as mitogen.imports.codeobj_imports()
...
This replaces `mitogen.master.scan_code_imports()` with
`mitogen.imports.codeobj_imports()`. The Python 3.x implementation now uses
`str.find()`, relying on Python >= 3.6 "widecode" format. Behaviour and
semantics should be unchanged. Now implementations are approx
- 1.5 x faster on Python 2.x
- 2 - 3 x faster on Python 3.x
Before
```console
$ ./tests/bench/scan_code
scan_code_imports python2.7 100 loops, best of 3: 3.19 msec per loop
scan_code_imports python3.9 500 loops, best of 5: 685 usec per loop
scan_code_imports python3.10 500 loops, best of 5: 727 usec per loop
scan_code_imports python3.11 500 loops, best of 5: 601 usec per loop
scan_code_imports python3.12 500 loops, best of 5: 609 usec per loop
scan_code_imports python3.13 500 loops, best of 5: 586 usec per loop
```
After
```console
codeobj_imports python2.7 1000 loops, best of 3: 1.98 msec per loop
codeobj_imports python3.9 1000 loops, best of 5: 302 usec per loop
codeobj_imports python3.10 1000 loops, best of 5: 297 usec per loop
codeobj_imports python3.11 1000 loops, best of 5: 243 usec per loop
codeobj_imports python3.12 1000 loops, best of 5: 278 usec per loop
codeobj_imports python3.13 1000 loops, best of 5: 259 usec per loop
```
```console
$ uname -a
Darwin kintha 24.6.0 Darwin Kernel Version 24.6.0: Mon Jul 14 11:30:29 PDT
2025; root:xnu-11417.140.69~1/RELEASE_ARM64_T6000 arm64
```
5 months ago
7fb7567809
Begin v0.3.27dev
5 months ago
5908936f8c
Prepare v0.3.26
5 months ago
17bee70dc2
mitogen: Fix BlockingIOError & EAGAIN in subprocess stdio
...
Mitogen was leaving the stdout and stderr of subprocesses in non-blocking
mode. When Python code ran in the remote process created by Mitogen calls such
as `print(long_string)` or `os.stout.write(bigger_than_the_buffer)` sometimes
raised `BlockingIOError`, or similar.
This change
- Removes code in `mitogen.core.Side` that set blocking/non-blocking mode
- Adds blocking/non-blocking control to `os.mitogen.pipe()` and a new
function `mitogen.core.socketpair()`
- Replaces `mitogen.core.set_block` and `mitogen.core.set_nonblock`
with `mitogen.core.set_blocking`, mirroring `os.set_blocking`
- Updates call sites as appropriate
- Adds tests for new functions and arguments
- Adds a regression test for subprocess stdio blocking/non-blocking
fixes #712
5 months ago
fde2dda87e
mitogen: Simplify `mitogen.parent.Reaper._calc_delay()` calculation
5 months ago
7f84874755
Begin 0.3.26dev
5 months ago
dfafa1430e
Prepare v0.3.25
5 months ago
65a81121c5
Begin 0.3.25dev
7 months ago
229fd67e97
Prepare v0.3.24
7 months ago
566a4c1e3c
mitogen: log child file objects rather than descriptors
...
The repr() of file objects is more self descriptive, and includes the fd.
7 months ago
6cec613daa
mitogen: Only close stdio file descriptors that were open at process startup
...
File descriptors 0, 1, and 2 are usually stdin, stdout, stderr; but not
always. If a process is started without one of these then the first descriptor
allocated by the process opening a file or socket will be allocated an fd <=
STDERR_FILENO. This isn't common, but it does occur, e.g. Windows GUI apps
started without being connected to a console, controller side plugins run
under Ansible 12 (ansible-core 2.19).
In such cases the corresponding sys attribute (e.g. sys.stderr) will be None.
refs #1258
See also
- https://docs.python.org/3/library/sys.html#sys.__stdin__
- https://docs.ansible.com/ansible/devel/porting_guides/porting_guide_12.html#porting-guide-for-v12-0-0a1
- https://github.com/ansible/ansible/pull/82770
- https://github.com/python/typeshed/issues/11778
- https://gist.github.com/moreati/034fef45f73d809d9411a8a63eca34d6
7 months ago
49fea37879
mitogen: Use pty.STD*_FILENO constants
...
Makes it more obvious and easier to find where stdin, stdout, and stderr file
descriptors are being interacted with.
7 months ago
8faae13d7e
mitogen: Improve readability of stderr initialisation in create_child()
...
There should be no behaviour change.
7 months ago
c146682e2e
Begin 0.3.24dev
8 months ago
8e25944c94
Prepare v0.3.23
8 months ago
927fb172d8
mitogen: Log skipped termios attributes
...
refs #1121
11 months ago
5f42da36f3
mitogen: Deduplicate cfmakeraw() flags
...
refs #1121
11 months ago
657e40b982
Begin 0.3.23dev
11 months ago
ae703b97a7
Prepare v0.3.22
11 months ago
67219c309a
mitogen: Fix unclosed file in first stage
12 months ago
6fcb7aae96
mitogen: Replace uses of deprecated `pkgutil.find_loader()`
...
fixes #1111
12 months ago
5c76941d1e
Begin 0.3.22dev
12 months ago
8b7354cb3a
Prepare v0.3.21
12 months ago
3ee6a0ff93
Begin v0.3.21dev
1 year ago
f1f8ad11ee
Prepare v0.3.20
1 year ago
ffb1709630
Begin v0.3.20dev
1 year ago
24ba734ff9
Prepare v0.3.19
1 year ago
Alex Willmer