efdd82d1ab
mitogen: Streamline Poller classes and Latch.poller_class selection
...
This
- Clarifies and corrects docstrings and comments based on investigation for #957
- Removes unused `Poller*._repr` attributes
- Eliminates some uses of `getattr()`
- Introduces `mitogen.parent.POLLERS` & `mitogen.parent.POLLER_LIGHTWEIGHT`
Preamble size change
```
@@ -1,7 +1,7 @@
SSH command size: 759
-Bootstrap (mitogen.core) size: 17862 (17.44KiB)
+Bootstrap (mitogen.core) size: 17934 (17.51KiB)
Original Minimized Compressed
-mitogen.parent 98171 95.9KiB 50569 49.4KiB 51.5% 12742 12.4KiB 13.0%
+mitogen.parent 96979 94.7KiB 49844 48.7KiB 51.4% 12697 12.4KiB 13.1%
mitogen.fork 8436 8.2KiB 4130 4.0KiB 49.0% 1648 1.6KiB 19.5%
mitogen.ssh 10892 10.6KiB 6952 6.8KiB 63.8% 2113 2.1KiB 19.4%
```
2 weeks ago
37ebce7e6e
Begin 0.3.8dev
4 weeks ago
a3644963c4
Prepare v0.3.7
4 weeks ago
933477fcbe
Begin 0.3.7dev
1 month ago
5d789faee5
Prepare 0.3.6
1 month ago
d7979c3597
mitogen: Raise TypeError on `mitogen.utils.cast(custom_str)` failures
...
If casting a string fails then raise a TypeError. This is potentially an API
breaking change; chosen as the lesser evil vs. allowing silent errors.
`cast()` relies on `bytes(obj)` & `str(obj)` returning the respective
supertype. That's no longer the case for `AnsibleUnsafeBytes` &
`AnsibleUnsafeText`; since fixes/mitigations for CVE-2023-5764.
fixes #1046 , refs #977
See also
- https://github.com/advisories/GHSA-7j69-qfc3-2fq9
- https://github.com/ansible/ansible/pull/82293
1 month ago
fc24b3f25e
Start v0.3.6 development
2 months ago
e97ab2f597
Prepare v0.3.5
2 months ago
5ad3d14ceb
mitogen: Support PEP 451 ModuleSpec API, required for Python 3.12
...
importlib.machinery.ModuleSpec and find_spec() were introduced in Python 3.4
under PEP 451. They replace the find_module() API of PEP 302, which was
deprecated from Python 3.4. They were removed in Python 3.12 along with the
imp module.
This change adds support for the PEP 451 APIs. Mitogen should no longer import
imp on Python versions that support ModuleSpec. Tests have been added to cover
the new APIs.
CI jobs have been added to cover Python 3.x on macOS.
Refs #1033
Co-authored-by: Witold Baryluk <witold.baryluk@gmail.com>
2 months ago
3a31a7d886
mitogen: Workaround CPython importlib PermissionError when cwd is unreadable
...
On macOS when using a become plugin as an unprivileged user, to another
unprivileged user it is likely that the current working directory can't be
read. In this case os.cwd() raises PermissionError.
On versions of Python currently in the wild (March 2024, CPython <= 3.13) if
any non-builtin or non-frozen module (e.g. zlib, base64) is imported then
`importlib._bootstrap_external.PathFinder._path_importer_cache()` attempts to
call os.cwd() without catching PermissionError.
The previous comment about needing an extra .encode() appears to be wrong,
atleast for Python 3.x >= 3.6.
Command size increased by 54 bytes, bootstrap by 804 bytes. Changed from
codecs module to binascii & zlib because they're extensions, and importing
them triggers fewer supporting imports (e.g. encodings module).
Before
```
✗ ./preamble_size.py
SSH command size: 705
Bootstrap (mitogen.core) size: 17078 (16.68KiB)
Original Minimized Compressed
mitogen.parent 97884 95.6KiB 50515 49.3KiB 51.6% 12727 12.4KiB
13.0%
mitogen.fork 8436 8.2KiB 4130 4.0KiB 49.0% 1648 1.6KiB
19.5%
mitogen.ssh 10892 10.6KiB 6952 6.8KiB 63.8% 2113 2.1KiB
19.4%
mitogen.sudo 12089 11.8KiB 5924 5.8KiB 49.0% 2249 2.2KiB
18.6%
mitogen.select 12325 12.0KiB 2929 2.9KiB 23.8% 964 0.9KiB
7.8%
mitogen.service 41699 40.7KiB 22477 22.0KiB 53.9% 5885 5.7KiB
14.1%
mitogen.fakessh 15577 15.2KiB 7989 7.8KiB 51.3% 2623 2.6KiB
16.8%
mitogen.master 51398 50.2KiB 25715 25.1KiB 50.0% 6886 6.7KiB
13.4%
```
After
```
✗ ./preamble_size.py
SSH command size: 759
Bootstrap (mitogen.core) size: 17882 (17.46KiB)
Original Minimized Compressed
mitogen.parent 98173 95.9KiB 50571 49.4KiB 51.5% 12747 12.4KiB
13.0%
mitogen.fork 8436 8.2KiB 4130 4.0KiB 49.0% 1648 1.6KiB
19.5%
mitogen.ssh 10892 10.6KiB 6952 6.8KiB 63.8% 2113 2.1KiB
19.4%
mitogen.sudo 12089 11.8KiB 5924 5.8KiB 49.0% 2249 2.2KiB
18.6%
mitogen.select 12325 12.0KiB 2929 2.9KiB 23.8% 964 0.9KiB
7.8%
mitogen.service 41699 40.7KiB 22477 22.0KiB 53.9% 5885 5.7KiB
14.1%
mitogen.fakessh 15577 15.2KiB 7989 7.8KiB 51.3% 2623 2.6KiB
16.8%
mitogen.master 56116 54.8KiB 29427 28.7KiB 52.4% 7627 7.4KiB
13.6%
```
Fixes #885
Refs https://github.com/python/cpython/issues/115911
2 months ago
e2f4d9275c
tests: Fix ansible_python_interpreter & discovered_interpreter_python tests on macOS
...
Should account for fiddling in mitogen.parent.Connection._first_stage() and
symlinks. I won't be surprised if it breaks again soon and often.
2 months ago
fc3e788cb4
non functional: Add comments about imp module removal in Python 3.12
3 months ago
63457b4866
docs: Update external URLs (e.g. dw/mitogen -> mitogen-hq/mitogen)
...
Found with sphinx-build -b linkcheck. Not all flagged URLs have been changed,
e.g. Ansible plugins, deleted Github users.
9 months ago
6aa4fd3573
docs: Fix generation of static website
...
Bare minimum syntax errors and requirements constraints to work with Netlify
hosting.
9 months ago
455fd2bcdf
Bump version
10 months ago
f18f5165cd
Prep for v0.3.4
10 months ago
ad4b686836
master.PkgutilMethod: Skip module loaders that raise ValueError
...
e.g. in Ansible 6, ansible-core 2.13
```
[mux 2717] 23:39:11.342416 D mitogen: PkgutilMethod(): _AnsibleCollectionLoader(path=None).get_file_name('ansible.plugins') failed: ValueError('_AnsibleCollectionLoader(path=None) cannot find files for ansible.plugins, only ansible_collections.ansible.builtin.plugins')
```
2 years ago
d71fb672e8
Begin v0.3.4.dev0
2 years ago
c0d3deeac5
Prepare v0.3.3
2 years ago
e101cc4f44
mitogen.utils: Preserve docstring of functions decorated @with_router
...
Co-authored-by: Rezart Qelibari <gast-kontakt+mitogen@astzweig.de>
Replaces #837
Fixes #836
2 years ago
109feec6d5
Fix lints found by flake8
2 years ago
18c89de5a9
Remove unused module imports
2 years ago
65809a6f0f
mitogen: Handle Python 3.10 threading depreactions
2 years ago
d2ca8a9423
master.ParentEnumerationMethod: Require matching pkg.__name__
...
Co-authored-by: Stefano Rivera <stefano@rivera.za.net>
When the requested module (e.g. ansible.module_utils.distro)
- is provided by another module *e.g. distro)
- that itself was a package (e.g. distro 1.7.0)
At runtime
- ansible/module_utils/distro/__init__.py executes
- if https://pypi.org/project/distro/ is present, it's loaded as
ansible.module_utils.distro
- otherwise ansible/module_utils/distro/_distro.py is loaded
ParentEnumerationMethod would wrongly use whatever was in
sys.modules['ansible.module_utils.distro]. Instead we should ascend to
the first parent that has fullname == sys.modules[fullname].__name__.
Then descend to the appropriate .py file on disk.
This bug didn't show up before because until distro 1.7.0 (Feb 2022) the
top-level distro module was a module (distro.py) not a package
(distro/__init__.py)
fixes #906
2 years ago
47699e15aa
master.SysModulesMethod: log rejection reasons
2 years ago
0fa0a93f55
master.PkgutilMethod: log rejection reasons
2 years ago
60c4ae5599
Add notes on imports and importlib
2 years ago
5b8f7dd1be
Start v0.3.3 development
2 years ago
e8c3fe7881
Fix Trove classifier, bump version
...
fixes #891
(cherry picked from commit 1a84184838
2 years ago
9a19f0ed16
mitogen.master: Fix stdlib discovery when sys.prefix contains symlinks
...
This can be he case when Python has been installed with Homebrew.
2 years ago
552819e765
mitogen.parent: Detect and avoid Python2.7 wrapper on macOS 11 & 12
...
Without this errors such as the following occur
```
✗ MITOGEN_LOG_LEVEL=DEBUG python3 foo.py
Python: execv: (null): No such file or directory
Traceback (most recent call last):
File "foo.py", line 16, in <module>
target = router.local(python_path='/usr/bin/python2.7', debug=True)
File "/Users/alex/src/mitogen2/mitogen/parent.py", line 2486, in local
return self.connect(u'local', **kwargs)
File "/Users/alex/src/mitogen2/mitogen/parent.py", line 2446, in connect
return self._connect(klass, **mitogen.core.Kwargs(kwargs))
File "/Users/alex/src/mitogen2/mitogen/parent.py", line 2426, in _connect
conn.connect(context=context)
File "/Users/alex/src/mitogen2/mitogen/parent.py", line 1708, in connect
raise self.exception
mitogen.parent.EofError: EOF on stream; last 100 lines received:
MITO000
MITO001
```
Before
```
$ ./preamble_size.py
SSH command size: 625
Bootstrap (mitogen.core) size: 17007 (16.61KiB)
Original Minimized Compressed
mitogen.parent 97496 95.2KiB 50355 49.2KiB 51.6% 12663 12.4KiB 13.0%
mitogen.fork 8436 8.2KiB 4130 4.0KiB 49.0% 1648 1.6KiB 19.5%
mitogen.ssh 10892 10.6KiB 6952 6.8KiB 63.8% 2113 2.1KiB 19.4%
mitogen.sudo 12089 11.8KiB 5924 5.8KiB 49.0% 2249 2.2KiB 18.6%
mitogen.select 12325 12.0KiB 2929 2.9KiB 23.8% 964 0.9KiB 7.8%
mitogen.service 41644 40.7KiB 22431 21.9KiB 53.9% 5886 5.7KiB 14.1%
mitogen.fakessh 15599 15.2KiB 8011 7.8KiB 51.4% 2624 2.6KiB 16.8%
mitogen.master 48732 47.6KiB 24569 24.0KiB 50.4% 6768 6.6KiB 13.9%
```
After
```
$ ./preamble_size.py
SSH command size: 705
Bootstrap (mitogen.core) size: 17007 (16.61KiB)
Original Minimized Compressed
mitogen.parent 97885 95.6KiB 50516 49.3KiB 51.6% 12728 12.4KiB 13.0%
mitogen.fork 8436 8.2KiB 4130 4.0KiB 49.0% 1648 1.6KiB 19.5%
mitogen.ssh 10892 10.6KiB 6952 6.8KiB 63.8% 2113 2.1KiB 19.4%
mitogen.sudo 12089 11.8KiB 5924 5.8KiB 49.0% 2249 2.2KiB 18.6%
mitogen.select 12325 12.0KiB 2929 2.9KiB 23.8% 964 0.9KiB 7.8%
mitogen.service 41644 40.7KiB 22431 21.9KiB 53.9% 5886 5.7KiB 14.1%
mitogen.fakessh 15599 15.2KiB 8011 7.8KiB 51.4% 2624 2.6KiB 16.8%
mitogen.master 48733 47.6KiB 24570 24.0KiB 50.4% 6771 6.6KiB 13.9%
```
2 years ago
e6bc53ca3a
mitogen.parent: Remove tabs from minimized first stage
...
Saves 4 bytes
Before
```
$ python preamble_size.py
SSH command size: 629
Bootstrap (mitogen.core) size: 17007 (16.61KiB)
Original Minimized Compressed
mitogen.parent 97497 95.2KiB 50356 49.2KiB 51.6% 12665 12.4KiB 13.0%
mitogen.fork 8436 8.2KiB 4130 4.0KiB 49.0% 1648 1.6KiB 19.5%
mitogen.ssh 10892 10.6KiB 6952 6.8KiB 63.8% 2113 2.1KiB 19.4%
mitogen.sudo 12089 11.8KiB 5924 5.8KiB 49.0% 2249 2.2KiB 18.6%
mitogen.select 12325 12.0KiB 2929 2.9KiB 23.8% 964 0.9KiB 7.8%
mitogen.service 41644 40.7KiB 22431 21.9KiB 53.9% 5886 5.7KiB 14.1%
mitogen.fakessh 15599 15.2KiB 8011 7.8KiB 51.4% 2624 2.6KiB 16.8%
mitogen.master 48732 47.6KiB 24569 24.0KiB 50.4% 6768 6.6KiB 13.9%
```
After
```
$ python preamble_size.py
SSH command size: 625
Bootstrap (mitogen.core) size: 17007 (16.61KiB)
Original Minimized Compressed
mitogen.parent 97496 95.2KiB 50355 49.2KiB 51.6% 12663 12.4KiB 13.0%
mitogen.fork 8436 8.2KiB 4130 4.0KiB 49.0% 1648 1.6KiB 19.5%
mitogen.ssh 10892 10.6KiB 6952 6.8KiB 63.8% 2113 2.1KiB 19.4%
mitogen.sudo 12089 11.8KiB 5924 5.8KiB 49.0% 2249 2.2KiB 18.6%
mitogen.select 12325 12.0KiB 2929 2.9KiB 23.8% 964 0.9KiB 7.8%
mitogen.service 41644 40.7KiB 22431 21.9KiB 53.9% 5886 5.7KiB 14.1%
mitogen.fakessh 15599 15.2KiB 8011 7.8KiB 51.4% 2624 2.6KiB 16.8%
mitogen.master 48732 47.6KiB 24569 24.0KiB 50.4% 6768 6.6KiB 13.9%
```
3 years ago
dd6d73db37
mitogen.parent: Eliminate use of platform module in first stage
...
This reduces the size of the initial SSH command by 204 bytes, & may fix errors
running Mitogen on macOS. AFAICT platform was used but not imported.
Before
```
$ python ./preamble_size.py
SSH command size: 833
Bootstrap (mitogen.core) size: 17007 (16.61KiB)
Original Minimized Compressed
mitogen.parent 97565 95.3KiB 50427 49.2KiB 51.7% 12689 12.4KiB 13.0%
mitogen.fork 8436 8.2KiB 4130 4.0KiB 49.0% 1648 1.6KiB 19.5%
mitogen.ssh 10892 10.6KiB 6952 6.8KiB 63.8% 2113 2.1KiB 19.4%
mitogen.sudo 12089 11.8KiB 5924 5.8KiB 49.0% 2249 2.2KiB 18.6%
mitogen.select 12325 12.0KiB 2929 2.9KiB 23.8% 964 0.9KiB 7.8%
mitogen.service 41644 40.7KiB 22431 21.9KiB 53.9% 5886 5.7KiB 14.1%
mitogen.fakessh 15599 15.2KiB 8011 7.8KiB 51.4% 2624 2.6KiB 16.8%
mitogen.master 48732 47.6KiB 24569 24.0KiB 50.4% 6768 6.6KiB 13.9%
```
After
```
$ python preamble_size.py
SSH command size: 629
Bootstrap (mitogen.core) size: 17007 (16.61KiB)
Original Minimized Compressed
mitogen.parent 97543 95.3KiB 50357 49.2KiB 51.6% 12665 12.4KiB 13.0%
mitogen.fork 8436 8.2KiB 4130 4.0KiB 49.0% 1648 1.6KiB 19.5%
mitogen.ssh 10892 10.6KiB 6952 6.8KiB 63.8% 2113 2.1KiB 19.4%
mitogen.sudo 12089 11.8KiB 5924 5.8KiB 49.0% 2249 2.2KiB 18.6%
mitogen.select 12325 12.0KiB 2929 2.9KiB 23.8% 964 0.9KiB 7.8%
mitogen.service 41644 40.7KiB 22431 21.9KiB 53.9% 5886 5.7KiB 14.1%
mitogen.fakessh 15599 15.2KiB 8011 7.8KiB 51.4% 2624 2.6KiB 16.8%
mitogen.master 48732 47.6KiB 24569 24.0KiB 50.4% 6768 6.6KiB 13.9%
```
3 years ago
84c567e265
Add podman connection support
...
Shameless copy of buildah connection with modifications of invocation to
fit podman CLI.
Signed-off-by: Denis Zalevskiy <dez@aiven.io>
3 years ago
da536e8ae1
Fix stdlib typos that would cause NameError or AttributeError exceptions
3 years ago
da3772cbff
Increment version to 0.3.1.dev0
3 years ago
9b1617f26f
Bump version to 0.3.0rc1
3 years ago
fd8c2d3702
Merge branch 'master' into master
3 years ago
bce3bab3e8
Add the msvcrt moduleto the default module deny list
...
Commit https://github.com/python/cpython/commit/880d42a3b247 (first
released in Python 3.8a0) moved an import of msvcrt from an if <win32>
block, into a try/except block. So now the import is tried even on Linux
or MacOS.
https://docs.python.org/3/library/msvcrt.html is a Windows specific
builtin.
3 years ago
ceb0a94467
Explain why the Python 2.x thread module is blacklisted
...
Based on the original commit I believe it is only an optimization.
However I could be wrong. I intend to request review of this part.
3 years ago
6b5a46ccd0
Fixes RuntimeWarning for improper bufferring on local connection
3 years ago
4657979210
adding clarifying comments
3 years ago
f886293a6a
Merge branch 'master' into collectionsSupport
4 years ago
b0ce29dcfd
ssh: Match newer ssh host key prompt that accepts the fingerprint
...
This fixes an ERROR in test_accept_enforce_host_keys() while running the
test suite.
Fixes #756
4 years ago
518324c371
fix regression in Darwin 19 (OSX 10.15+) ansible python interpreter detection
4 years ago
33e176d62e
add support for ansible_collections site-package (from pip ansible==2.10.0 install) + switch to ansible 2.10.0 rather than github tag
4 years ago
1ffb319300
remove debugging
4 years ago
207e36194d
try and get some visibility into test failures
4 years ago
5b3d90dac4
see if sys.path is being loaded properly on azure
4 years ago
4dea09a924
check sys.path issue
4 years ago
Alex Willmer