importlib.machinery.ModuleSpec and find_spec() were introduced in Python 3.4
under PEP 451. They replace the find_module() API of PEP 302, which was
deprecated from Python 3.4. They were removed in Python 3.12 along with the
imp module.
This change adds support for the PEP 451 APIs. Mitogen should no longer import
imp on Python versions that support ModuleSpec. Tests have been added to cover
the new APIs.
CI jobs have been added to cover Python 3.x on macOS.
Refs #1033
Co-authored-by: Witold Baryluk <witold.baryluk@gmail.com>
e.g. in Ansible 6, ansible-core 2.13
```
[mux 2717] 23:39:11.342416 D mitogen: PkgutilMethod(): _AnsibleCollectionLoader(path=None).get_file_name('ansible.plugins') failed: ValueError('_AnsibleCollectionLoader(path=None) cannot find files for ansible.plugins, only ansible_collections.ansible.builtin.plugins')
```
Co-authored-by: Stefano Rivera <stefano@rivera.za.net>
When the requested module (e.g. ansible.module_utils.distro)
- is provided by another module *e.g. distro)
- that itself was a package (e.g. distro 1.7.0)
At runtime
- ansible/module_utils/distro/__init__.py executes
- if https://pypi.org/project/distro/ is present, it's loaded as
ansible.module_utils.distro
- otherwise ansible/module_utils/distro/_distro.py is loaded
ParentEnumerationMethod would wrongly use whatever was in
sys.modules['ansible.module_utils.distro]. Instead we should ascend to
the first parent that has fullname == sys.modules[fullname].__name__.
Then descend to the appropriate .py file on disk.
This bug didn't show up before because until distro 1.7.0 (Feb 2022) the
top-level distro module was a module (distro.py) not a package
(distro/__init__.py)
fixes#906
Given a message sent on "ssh.foo" to "mypkg.mymod", instead of logging
it to "mitogen.ctx.ssh.foo" in the master process, with the message
prefixed with the original logger name, instead log it to
"mypkg.mymod.[ssh.foo]", permitting normal logging package filtering
features to work as they usually do.
This also helps tidy up logging output a little bit.
Split Stream into many, many classes
* mitogen.parent.Connection: Handles connection setup logic only.
* Maintain references to stdout and stderr streams.
* Manages TimerList timer to cancel connection attempt after
deadline
* Blocking setup code replaced by async equivalents running on the
broker
* mitogen.parent.Options: Tracks connection-specific options. This
keeps the connection class small, but more importantly, it is
generic to the future desire to build and execute command lines
without starting a full connection.
* mitogen.core.Protocol: Handles program behaviour relating to events
on a stream. Protocol performs no IO of its own, instead deferring
it to Stream and Side. This makes testing much easier, and means
libssh can reimplement Stream and Side to reuse MitogenProtocol
* mitogen.core.MitogenProtocol: Guts of the old Mitogen stream
implementtion
* mitogen.core.BufferedWriter: Guts of the old Mitogen buffered
transmit implementation, made generic
* mitogen.core.DelineatedProtocol: Guts of the old IoLogger, knows how
to split up input and pass it on to a
on_line_received()/on_partial_line_received() callback.
* mitogen.parent.BootstrapProtocol: Asynchronous equivalent of the old
blocking connect code. Waits for various prompts (MITO001 etc) and
writes the bootstrap using a BufferedWriter. On success, switches
the stream to MitogenProtocol.
* mitogen.core.Message: move encoding parts of MitogenProtocol out to
Message (where it belongs) and write a bunch of new tests for
pickling.
* The bizarre Stream.construct() is gone now, Option.__init__ is its
own constructor. Should fix many LGTM errors.
* Update all connection methods: Every connection method is updated to
use async logic, defining protocols as required to handle interactive
prompts like in SSH or su. Add new real integration tests for at least
doas and su.
* Eliminate manual fd management: File descriptors are trapped in file
objects at their point of origin, and Side is updated to use file
objects rather than raw descriptors. This eliminates a whole class of
bugs where unrelated FDs could be closed by the wrong component. Now
an FD's open/closed status is fused to it everywhere in the library.
* Halve file descriptor usage: now FD open/close state is tracked by
its file object, we don't need to duplicate FDs everywhere so that
receive/transmit side can be closed independently. Instead both sides
back on to the same file object. Closes#26, Closes#470.
* Remove most uses of dup/dup2: Closes#256. File descriptors are
trapped in a common file object and shared among classes. The
remaining few uses for dup/dup2 are as close to minimal as possible.
* Introduce mitogen.parent.Process: uniform interface for subprocesses
created either via mitogen.fork or the subprocess module. Remove all
the crap where we steal a pid from subprocess guts. Now we use
subprocess to manage its processes as it should be. Closes#169 by
using the new Timers facility to poll for a slow-to-exit subprocess.
* Fix su password race: Closes#363. DelineatedProtocol naturally
retries partially received lines, preventing the cause of the original
race.
* Delete old blocking IO utility functions
iter_read()/write_all()/discard_until().
Closes#26Closes#147Closes#169Closes#256Closes#363Closes#419Closes#470
Now it's possible to find both packages and modules when the
sys.modules[...] state for the package/module is junk. Previously only
modules were possible.
This also refactors things to make writing better tests for all these
cases much simpler.
Regardless of the version of simplejson loaded in the master, load up
the ModuleResponder cache with our 2.4-compatible version.
To cope with simplejson being loaded due to modules like ec2_group that
try to import it before importing 'json', also update target.py to
remove it from the whitelist if a local 'json' module import succeeds.
Alex Willmer