efdd82d1ab
mitogen: Streamline Poller classes and Latch.poller_class selection
...
This
- Clarifies and corrects docstrings and comments based on investigation for #957
- Removes unused `Poller*._repr` attributes
- Eliminates some uses of `getattr()`
- Introduces `mitogen.parent.POLLERS` & `mitogen.parent.POLLER_LIGHTWEIGHT`
Preamble size change
```
@@ -1,7 +1,7 @@
SSH command size: 759
-Bootstrap (mitogen.core) size: 17862 (17.44KiB)
+Bootstrap (mitogen.core) size: 17934 (17.51KiB)
Original Minimized Compressed
-mitogen.parent 98171 95.9KiB 50569 49.4KiB 51.5% 12742 12.4KiB 13.0%
+mitogen.parent 96979 94.7KiB 49844 48.7KiB 51.4% 12697 12.4KiB 13.1%
mitogen.fork 8436 8.2KiB 4130 4.0KiB 49.0% 1648 1.6KiB 19.5%
mitogen.ssh 10892 10.6KiB 6952 6.8KiB 63.8% 2113 2.1KiB 19.4%
```
4 weeks ago
3a31a7d886
mitogen: Workaround CPython importlib PermissionError when cwd is unreadable
...
On macOS when using a become plugin as an unprivileged user, to another
unprivileged user it is likely that the current working directory can't be
read. In this case os.cwd() raises PermissionError.
On versions of Python currently in the wild (March 2024, CPython <= 3.13) if
any non-builtin or non-frozen module (e.g. zlib, base64) is imported then
`importlib._bootstrap_external.PathFinder._path_importer_cache()` attempts to
call os.cwd() without catching PermissionError.
The previous comment about needing an extra .encode() appears to be wrong,
atleast for Python 3.x >= 3.6.
Command size increased by 54 bytes, bootstrap by 804 bytes. Changed from
codecs module to binascii & zlib because they're extensions, and importing
them triggers fewer supporting imports (e.g. encodings module).
Before
```
✗ ./preamble_size.py
SSH command size: 705
Bootstrap (mitogen.core) size: 17078 (16.68KiB)
Original Minimized Compressed
mitogen.parent 97884 95.6KiB 50515 49.3KiB 51.6% 12727 12.4KiB
13.0%
mitogen.fork 8436 8.2KiB 4130 4.0KiB 49.0% 1648 1.6KiB
19.5%
mitogen.ssh 10892 10.6KiB 6952 6.8KiB 63.8% 2113 2.1KiB
19.4%
mitogen.sudo 12089 11.8KiB 5924 5.8KiB 49.0% 2249 2.2KiB
18.6%
mitogen.select 12325 12.0KiB 2929 2.9KiB 23.8% 964 0.9KiB
7.8%
mitogen.service 41699 40.7KiB 22477 22.0KiB 53.9% 5885 5.7KiB
14.1%
mitogen.fakessh 15577 15.2KiB 7989 7.8KiB 51.3% 2623 2.6KiB
16.8%
mitogen.master 51398 50.2KiB 25715 25.1KiB 50.0% 6886 6.7KiB
13.4%
```
After
```
✗ ./preamble_size.py
SSH command size: 759
Bootstrap (mitogen.core) size: 17882 (17.46KiB)
Original Minimized Compressed
mitogen.parent 98173 95.9KiB 50571 49.4KiB 51.5% 12747 12.4KiB
13.0%
mitogen.fork 8436 8.2KiB 4130 4.0KiB 49.0% 1648 1.6KiB
19.5%
mitogen.ssh 10892 10.6KiB 6952 6.8KiB 63.8% 2113 2.1KiB
19.4%
mitogen.sudo 12089 11.8KiB 5924 5.8KiB 49.0% 2249 2.2KiB
18.6%
mitogen.select 12325 12.0KiB 2929 2.9KiB 23.8% 964 0.9KiB
7.8%
mitogen.service 41699 40.7KiB 22477 22.0KiB 53.9% 5885 5.7KiB
14.1%
mitogen.fakessh 15577 15.2KiB 7989 7.8KiB 51.3% 2623 2.6KiB
16.8%
mitogen.master 56116 54.8KiB 29427 28.7KiB 52.4% 7627 7.4KiB
13.6%
```
Fixes #885
Refs https://github.com/python/cpython/issues/115911
2 months ago
e2f4d9275c
tests: Fix ansible_python_interpreter & discovered_interpreter_python tests on macOS
...
Should account for fiddling in mitogen.parent.Connection._first_stage() and
symlinks. I won't be surprised if it breaks again soon and often.
3 months ago
63457b4866
docs: Update external URLs (e.g. dw/mitogen -> mitogen-hq/mitogen)
...
Found with sphinx-build -b linkcheck. Not all flagged URLs have been changed,
e.g. Ansible plugins, deleted Github users.
9 months ago
109feec6d5
Fix lints found by flake8
2 years ago
552819e765
mitogen.parent: Detect and avoid Python2.7 wrapper on macOS 11 & 12
...
Without this errors such as the following occur
```
✗ MITOGEN_LOG_LEVEL=DEBUG python3 foo.py
Python: execv: (null): No such file or directory
Traceback (most recent call last):
File "foo.py", line 16, in <module>
target = router.local(python_path='/usr/bin/python2.7', debug=True)
File "/Users/alex/src/mitogen2/mitogen/parent.py", line 2486, in local
return self.connect(u'local', **kwargs)
File "/Users/alex/src/mitogen2/mitogen/parent.py", line 2446, in connect
return self._connect(klass, **mitogen.core.Kwargs(kwargs))
File "/Users/alex/src/mitogen2/mitogen/parent.py", line 2426, in _connect
conn.connect(context=context)
File "/Users/alex/src/mitogen2/mitogen/parent.py", line 1708, in connect
raise self.exception
mitogen.parent.EofError: EOF on stream; last 100 lines received:
MITO000
MITO001
```
Before
```
$ ./preamble_size.py
SSH command size: 625
Bootstrap (mitogen.core) size: 17007 (16.61KiB)
Original Minimized Compressed
mitogen.parent 97496 95.2KiB 50355 49.2KiB 51.6% 12663 12.4KiB 13.0%
mitogen.fork 8436 8.2KiB 4130 4.0KiB 49.0% 1648 1.6KiB 19.5%
mitogen.ssh 10892 10.6KiB 6952 6.8KiB 63.8% 2113 2.1KiB 19.4%
mitogen.sudo 12089 11.8KiB 5924 5.8KiB 49.0% 2249 2.2KiB 18.6%
mitogen.select 12325 12.0KiB 2929 2.9KiB 23.8% 964 0.9KiB 7.8%
mitogen.service 41644 40.7KiB 22431 21.9KiB 53.9% 5886 5.7KiB 14.1%
mitogen.fakessh 15599 15.2KiB 8011 7.8KiB 51.4% 2624 2.6KiB 16.8%
mitogen.master 48732 47.6KiB 24569 24.0KiB 50.4% 6768 6.6KiB 13.9%
```
After
```
$ ./preamble_size.py
SSH command size: 705
Bootstrap (mitogen.core) size: 17007 (16.61KiB)
Original Minimized Compressed
mitogen.parent 97885 95.6KiB 50516 49.3KiB 51.6% 12728 12.4KiB 13.0%
mitogen.fork 8436 8.2KiB 4130 4.0KiB 49.0% 1648 1.6KiB 19.5%
mitogen.ssh 10892 10.6KiB 6952 6.8KiB 63.8% 2113 2.1KiB 19.4%
mitogen.sudo 12089 11.8KiB 5924 5.8KiB 49.0% 2249 2.2KiB 18.6%
mitogen.select 12325 12.0KiB 2929 2.9KiB 23.8% 964 0.9KiB 7.8%
mitogen.service 41644 40.7KiB 22431 21.9KiB 53.9% 5886 5.7KiB 14.1%
mitogen.fakessh 15599 15.2KiB 8011 7.8KiB 51.4% 2624 2.6KiB 16.8%
mitogen.master 48733 47.6KiB 24570 24.0KiB 50.4% 6771 6.6KiB 13.9%
```
2 years ago
e6bc53ca3a
mitogen.parent: Remove tabs from minimized first stage
...
Saves 4 bytes
Before
```
$ python preamble_size.py
SSH command size: 629
Bootstrap (mitogen.core) size: 17007 (16.61KiB)
Original Minimized Compressed
mitogen.parent 97497 95.2KiB 50356 49.2KiB 51.6% 12665 12.4KiB 13.0%
mitogen.fork 8436 8.2KiB 4130 4.0KiB 49.0% 1648 1.6KiB 19.5%
mitogen.ssh 10892 10.6KiB 6952 6.8KiB 63.8% 2113 2.1KiB 19.4%
mitogen.sudo 12089 11.8KiB 5924 5.8KiB 49.0% 2249 2.2KiB 18.6%
mitogen.select 12325 12.0KiB 2929 2.9KiB 23.8% 964 0.9KiB 7.8%
mitogen.service 41644 40.7KiB 22431 21.9KiB 53.9% 5886 5.7KiB 14.1%
mitogen.fakessh 15599 15.2KiB 8011 7.8KiB 51.4% 2624 2.6KiB 16.8%
mitogen.master 48732 47.6KiB 24569 24.0KiB 50.4% 6768 6.6KiB 13.9%
```
After
```
$ python preamble_size.py
SSH command size: 625
Bootstrap (mitogen.core) size: 17007 (16.61KiB)
Original Minimized Compressed
mitogen.parent 97496 95.2KiB 50355 49.2KiB 51.6% 12663 12.4KiB 13.0%
mitogen.fork 8436 8.2KiB 4130 4.0KiB 49.0% 1648 1.6KiB 19.5%
mitogen.ssh 10892 10.6KiB 6952 6.8KiB 63.8% 2113 2.1KiB 19.4%
mitogen.sudo 12089 11.8KiB 5924 5.8KiB 49.0% 2249 2.2KiB 18.6%
mitogen.select 12325 12.0KiB 2929 2.9KiB 23.8% 964 0.9KiB 7.8%
mitogen.service 41644 40.7KiB 22431 21.9KiB 53.9% 5886 5.7KiB 14.1%
mitogen.fakessh 15599 15.2KiB 8011 7.8KiB 51.4% 2624 2.6KiB 16.8%
mitogen.master 48732 47.6KiB 24569 24.0KiB 50.4% 6768 6.6KiB 13.9%
```
3 years ago
dd6d73db37
mitogen.parent: Eliminate use of platform module in first stage
...
This reduces the size of the initial SSH command by 204 bytes, & may fix errors
running Mitogen on macOS. AFAICT platform was used but not imported.
Before
```
$ python ./preamble_size.py
SSH command size: 833
Bootstrap (mitogen.core) size: 17007 (16.61KiB)
Original Minimized Compressed
mitogen.parent 97565 95.3KiB 50427 49.2KiB 51.7% 12689 12.4KiB 13.0%
mitogen.fork 8436 8.2KiB 4130 4.0KiB 49.0% 1648 1.6KiB 19.5%
mitogen.ssh 10892 10.6KiB 6952 6.8KiB 63.8% 2113 2.1KiB 19.4%
mitogen.sudo 12089 11.8KiB 5924 5.8KiB 49.0% 2249 2.2KiB 18.6%
mitogen.select 12325 12.0KiB 2929 2.9KiB 23.8% 964 0.9KiB 7.8%
mitogen.service 41644 40.7KiB 22431 21.9KiB 53.9% 5886 5.7KiB 14.1%
mitogen.fakessh 15599 15.2KiB 8011 7.8KiB 51.4% 2624 2.6KiB 16.8%
mitogen.master 48732 47.6KiB 24569 24.0KiB 50.4% 6768 6.6KiB 13.9%
```
After
```
$ python preamble_size.py
SSH command size: 629
Bootstrap (mitogen.core) size: 17007 (16.61KiB)
Original Minimized Compressed
mitogen.parent 97543 95.3KiB 50357 49.2KiB 51.6% 12665 12.4KiB 13.0%
mitogen.fork 8436 8.2KiB 4130 4.0KiB 49.0% 1648 1.6KiB 19.5%
mitogen.ssh 10892 10.6KiB 6952 6.8KiB 63.8% 2113 2.1KiB 19.4%
mitogen.sudo 12089 11.8KiB 5924 5.8KiB 49.0% 2249 2.2KiB 18.6%
mitogen.select 12325 12.0KiB 2929 2.9KiB 23.8% 964 0.9KiB 7.8%
mitogen.service 41644 40.7KiB 22431 21.9KiB 53.9% 5886 5.7KiB 14.1%
mitogen.fakessh 15599 15.2KiB 8011 7.8KiB 51.4% 2624 2.6KiB 16.8%
mitogen.master 48732 47.6KiB 24569 24.0KiB 50.4% 6768 6.6KiB 13.9%
```
3 years ago
84c567e265
Add podman connection support
...
Shameless copy of buildah connection with modifications of invocation to
fit podman CLI.
Signed-off-by: Denis Zalevskiy <dez@aiven.io>
3 years ago
518324c371
fix regression in Darwin 19 (OSX 10.15+) ansible python interpreter detection
4 years ago
d6329f3446
Merge devel/290 @ 79b979ec8544ef5d8620c64068d4a42fabf50415
5 years ago
be4f1bdb50
issue #646 : add extra logging to assertions and start_child()
5 years ago
b21be478e4
issue #627 : add test and tweak Reaper behaviour.
5 years ago
e86f371b93
issue #621 : send ADD_ROUTE earlier and add test for early logging.
...
Logs were broken because ADD_ROUTE was being sent *after* messages
started flowing from the new child. That's an explicit non-goal of the
design, so fix it.
5 years ago
8cbaa98ff9
docs: more hyperlinks
5 years ago
e0d9b8d1e1
docs: a few more internals.rst additions
5 years ago
3b63da670f
Fix up another handful of LGTM errors.
5 years ago
4fa760cd21
issue #613 : add tests for all the weird shutdown methods
5 years ago
57012e0f72
Add mitogen.core.now() and use it everywhere; closes #614 .
5 years ago
93e8d5dfcc
docs: fix Sphinx warnings, add LogHandler, more docstrings
5 years ago
444b7d6d97
parent: use protocol for getting remote_id
...
Fixes 8d1b01d8ef
5 years ago
3231c62a66
parent: fix get_log_level() for split out loggers.
5 years ago
95b067a114
parent: docstring fixes
5 years ago
f4cee16526
parent: zombie reaping v3
...
Improvements:
- Refactored off Process, separately testable without a connection
- Don't delay Broker shutdown indefinitely for detached children
5 years ago
709a0c013f
issue #410 : fix test failure due to obsolete parentfp/childfp
5 years ago
9c0cb44ee9
issue #170 : replace Timer.cancelled with Timer.active
...
It's more flexable: False can represent 'cancelled' or 'expired',
whereas setting cancelled=True for an expired timer didn't feel right.
5 years ago
e8b1bf5909
issue #410 : automatically work around SELinux braindamage.
5 years ago
ad590f3321
parent: docstring formatting
5 years ago
a79d2bd50b
docs: another round of docstring cleanups.
5 years ago
feb1654305
docs: many more internals.rst tidyups
5 years ago
f0782ccd42
[stream-refactor] get caught up on internals.rst updates
5 years ago
b76da4698b
parent: move subprocess creation to mux thread too
...
Now connect() really is a pure blocking wrapper.
5 years ago
5298e87548
Split out and make readable more log messages across both packages
5 years ago
aa06b960f5
parent: define Connection behaviour during Broker.shutdown()
...
- Connection attempt fails reliably, and it fails with CancelledError
- Add new mitogen.core.unlisten()
- Add test.
5 years ago
45a3014fd4
parent: decode logged stdout as UTF-8.
5 years ago
9035884c77
ansible: abstract worker process model.
...
Move all details of broker/router setup out of connection.py, instead
deferring it to a WorkerModel class exported by process.py via
get_worker_model(). The running strategy can override the configured
worker model via _get_worker_model().
ClassicWorkerModel is installed by default, which implements the
extension's existing process model.
Add optional support for the third party setproctitle module, so
children have pretty names in ps output.
Add optional support for per-CPU multiplexers to classic runs.
5 years ago
6b8a7cbcc4
[stream-refactor] parent: fix crash on graceful shutdown
...
Now it's possible for stream.protocol to not refer to MitogenProtocol,
move the signal handler to a MitogenProtocol subclass instead.
Fixes a crash where CTRL+C during child bootstrap would print
AttributeError.
5 years ago
2ccdeeeb87
parent: tidy up create_socketpair()
5 years ago
c02358698b
[stream-refactor] don't abort Connection until all buffers are empty
5 years ago
93342ba60c
Normalize docstring formatting
5 years ago
65e31f63fe
[stream-refactor] fix Py2.4 failure by implementing missing Timer method
5 years ago
4eecc08047
[stream-refactor] merge stdout+stderr when reporting EofError
...
Fixes sudo regression
5 years ago
93abbcaf7a
[stream-refactor] fix crash in runner/forking_active.yml
5 years ago
8769c3ce24
[stream-refactor] more readable log string format
5 years ago
d411003b64
[stream-refactor] dont doubly log last partial line
5 years ago
1fb3852fa6
[stream-refactor] fix crash when no stderr present.
5 years ago
f039c81bb0
[stream-refactor] rename Process attrs, fix up more create_child_test
5 years ago
8d1b01d8ef
Refactor Stream, introduce quasi-asynchronous connect, much more
...
Split Stream into many, many classes
* mitogen.parent.Connection: Handles connection setup logic only.
* Maintain references to stdout and stderr streams.
* Manages TimerList timer to cancel connection attempt after
deadline
* Blocking setup code replaced by async equivalents running on the
broker
* mitogen.parent.Options: Tracks connection-specific options. This
keeps the connection class small, but more importantly, it is
generic to the future desire to build and execute command lines
without starting a full connection.
* mitogen.core.Protocol: Handles program behaviour relating to events
on a stream. Protocol performs no IO of its own, instead deferring
it to Stream and Side. This makes testing much easier, and means
libssh can reimplement Stream and Side to reuse MitogenProtocol
* mitogen.core.MitogenProtocol: Guts of the old Mitogen stream
implementtion
* mitogen.core.BufferedWriter: Guts of the old Mitogen buffered
transmit implementation, made generic
* mitogen.core.DelineatedProtocol: Guts of the old IoLogger, knows how
to split up input and pass it on to a
on_line_received()/on_partial_line_received() callback.
* mitogen.parent.BootstrapProtocol: Asynchronous equivalent of the old
blocking connect code. Waits for various prompts (MITO001 etc) and
writes the bootstrap using a BufferedWriter. On success, switches
the stream to MitogenProtocol.
* mitogen.core.Message: move encoding parts of MitogenProtocol out to
Message (where it belongs) and write a bunch of new tests for
pickling.
* The bizarre Stream.construct() is gone now, Option.__init__ is its
own constructor. Should fix many LGTM errors.
* Update all connection methods: Every connection method is updated to
use async logic, defining protocols as required to handle interactive
prompts like in SSH or su. Add new real integration tests for at least
doas and su.
* Eliminate manual fd management: File descriptors are trapped in file
objects at their point of origin, and Side is updated to use file
objects rather than raw descriptors. This eliminates a whole class of
bugs where unrelated FDs could be closed by the wrong component. Now
an FD's open/closed status is fused to it everywhere in the library.
* Halve file descriptor usage: now FD open/close state is tracked by
its file object, we don't need to duplicate FDs everywhere so that
receive/transmit side can be closed independently. Instead both sides
back on to the same file object. Closes #26 , Closes #470 .
* Remove most uses of dup/dup2: Closes #256 . File descriptors are
trapped in a common file object and shared among classes. The
remaining few uses for dup/dup2 are as close to minimal as possible.
* Introduce mitogen.parent.Process: uniform interface for subprocesses
created either via mitogen.fork or the subprocess module. Remove all
the crap where we steal a pid from subprocess guts. Now we use
subprocess to manage its processes as it should be. Closes #169 by
using the new Timers facility to poll for a slow-to-exit subprocess.
* Fix su password race: Closes #363 . DelineatedProtocol naturally
retries partially received lines, preventing the cause of the original
race.
* Delete old blocking IO utility functions
iter_read()/write_all()/discard_until().
Closes #26
Closes #147
Closes #169
Closes #256
Closes #363
Closes #419
Closes #470
5 years ago
0f7bbcece9
parent: remove unused Timer parameter.
5 years ago
f66611dc83
parent: docstring improvements, cfmakeraw() regression.
5 years ago
Alex Willmer