`mitogen.parent.POLLER_LIGHTWEIGHT` will normally be `PollPoller`, falling
back to `EpollPoller`, `KqueuePoller`, or `Poller`.
Fixes#957
Co-authored-by: Luca Berruti <nadirio@gmail.com>
Co-authored-by: Philippe Kueck <bqobccy6ejnq2bqvmebqiwqha4cs4@protected32.unixadm.org>
Follwing fixes in Ansible 7-9 for CVE-2023-5764 cating `AnsibleUnsafeBytes` &
`AnsibleUnsafeText` to `bytes()` or `str()` requires special handling. The
handling is Ansible specific, so it shouldn't go in the mitogen package but
rather the ansible_mitogen package.
`ansible_mitogen.utils.unsafe.cast()` is most like `mitogen.utils.cast()`.
During development it began as `ansible_mitogen.utils.unsafe.unwrap_var()`,
closer to an inverse of `ansible.utils.unsafe_procy.wrap_var()`. Future
enhancements may move in this direction.
refs #977, refs #1046
See also
- https://github.com/advisories/GHSA-7j69-qfc3-2fq9
- https://github.com/ansible/ansible/pull/82293
- https://github.com/mitogen-hq/mitogen/wiki/AnsibleUnsafe-notes
If casting a string fails then raise a TypeError. This is potentially an API
breaking change; chosen as the lesser evil vs. allowing silent errors.
`cast()` relies on `bytes(obj)` & `str(obj)` returning the respective
supertype. That's no longer the case for `AnsibleUnsafeBytes` &
`AnsibleUnsafeText`; since fixes/mitigations for CVE-2023-5764.
fixes#1046, refs #977
See also
- https://github.com/advisories/GHSA-7j69-qfc3-2fq9
- https://github.com/ansible/ansible/pull/82293
Follwing fixes in Ansible 7-9 for CVE-2023-5764 cating `AnsibleUnsafeBytes` &
`AnsibleUnsafeText` to `bytes()` or `str()` requires special handling. The
handling is Ansible specific, so it shouldn't go in the mitogen package but
rather the ansible_mitogen package.
`ansible_mitogen.utils.unsafe.cast()` is most like `mitogen.utils.cast()`.
During development it began as `ansible_mitogen.utils.unsafe.unwrap_var()`,
closer to an inverse of `ansible.utils.unsafe_procy.wrap_var()`. Future
enhancements may move in this direction.
refs #977, refs #1046
See also
- https://github.com/advisories/GHSA-7j69-qfc3-2fq9
- https://github.com/ansible/ansible/pull/82293
- https://github.com/mitogen-hq/mitogen/wiki/AnsibleUnsafe-notes
If casting a string fails then raise a TypeError. This is potentially an API
breaking change; chosen as the lesser evil vs. allowing silent errors.
`cast()` relies on `bytes(obj)` & `str(obj)` returning the respective
supertype. That's no longer the case for `AnsibleUnsafeBytes` &
`AnsibleUnsafeText`; since fixes/mitigations for CVE-2023-5764.
fixes#1046, refs #977
See also
- https://github.com/advisories/GHSA-7j69-qfc3-2fq9
- https://github.com/ansible/ansible/pull/82293
Most of the necessary changes were made in recent PEP 451 commits. This bumps
the CI jobs, and declares the support. Test dependendancies are bumped to
latest supportted/available versions.
refs #1033
importlib.machinery.ModuleSpec and find_spec() were introduced in Python 3.4
under PEP 451. They replace the find_module() API of PEP 302, which was
deprecated from Python 3.4. They were removed in Python 3.12 along with the
imp module.
This change adds support for the PEP 451 APIs. Mitogen should no longer import
imp on Python versions that support ModuleSpec. Tests have been added to cover
the new APIs.
CI jobs have been added to cover Python 3.x on macOS.
Refs #1033
Co-authored-by: Witold Baryluk <witold.baryluk@gmail.com>
This prevents unreleased versions appearing on the website (e.g. 0.3.5.dev0),
but introduces the risk of forgetting to update the website after a release.
A better fix requires deeper design/workflow thought.
refs #1028
faulthandler is a stdlib module in Python 3.3+. For a long time a PyPI package
of the same name was available for earlier Python releases. That package has
since been removed from PyPI, and the source respoitory archived. So we should
not rely on it.
fixes#983 refs #970
Refs #925#969
I'm not 100% confident that merely removing this is the full fix,
without substituting something else. I am sure keeping it would be
the greater of two evils. __del__() should be avoided on general
principal, and it's associated with multiple intermittant CI
failures, plus multiple user reported issues.
This reapplies an earlier change, when this plugin was first introduced to
Mitogen. The plugin was updated to fix
[DEPRECATION WARNING]: The '_remote_checksum()' method is deprecated.
I've elected to short-circuit the if statemtn logic, rather than
deleting/unindenting, to make the code delta much smaller. This should make it
easier to maintain/update.
Fixes#915
Co-authored-by: Stefano Rivera <stefano@rivera.za.net>
When the requested module (e.g. ansible.module_utils.distro)
- is provided by another module *e.g. distro)
- that itself was a package (e.g. distro 1.7.0)
At runtime
- ansible/module_utils/distro/__init__.py executes
- if https://pypi.org/project/distro/ is present, it's loaded as
ansible.module_utils.distro
- otherwise ansible/module_utils/distro/_distro.py is loaded
ParentEnumerationMethod would wrongly use whatever was in
sys.modules['ansible.module_utils.distro]. Instead we should ascend to
the first parent that has fullname == sys.modules[fullname].__name__.
Then descend to the appropriate .py file on disk.
This bug didn't show up before because until distro 1.7.0 (Feb 2022) the
top-level distro module was a module (distro.py) not a package
(distro/__init__.py)
fixes#906
Also
- Simplifies adding support for additional Ansible versions
- Unifies Python package versioning in CI and local test environments
- Matches Python versions tested, with those declared in setup.py
- Expands targets covered by automated Ansible tests to
- centos6, centos8
- debian9, debian11
- ubuntu1604, ubuntu2004
This reverse shell was historically used to debug CI jobs interactively.
It is not used anymore, and may be causing jobs to hang, then timeout.
There is no reason to keep it, and removing it simplifies CI jobs.
Additionally it has been reported as flagged by security scanners, so
removing it makes Mitogen easier to package/adopt.
fixes#847
The Travis Linux distribution must be upgraded because the Trusty
(Ubuntu 14.04) image does not have Python 3.9. Xenial (Ubuntu 16.04) is
the earliest version that offers Python 3.9.
I have not chosen a later release, in order to aid restoration of Python
2.4 - 2.6 tests.
CALLBACK_VERSION et al are documented as required in
https://docs.ansible.com/ansible/2.10/dev_guide/developing_plugins.html#callback-plugins.
The need for document_fragment is noted in
cfa8075537/lib/ansible/plugins/callback/default.py (L28-L32)Fixes#758
This addresses the following error, seen while running
`ansible_tests.py`.
```
TASK [Gathering Facts gather_timeout=10, gather_subset=['all']]
****************
task path:
/home/alex/src/mitogen/tests/ansible/regression/issue_109__target_has_old_ansible_installed.yml:4
[WARNING]: Failure using method (v2_runner_on_start) in callback plugin
(<ansible.plugins.callback.nice_stdout.CallbackModule object at
0x7f76b3dad090>): 'show_per_host_start'
Callback Exception:
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/executor/task_queue_manager.py",
line 372, in send_callback
method(*new_args, **kwargs)
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/plugins/callback/default.py",
line 240, in v2_runner_on_start
if self.get_option('show_per_host_start'):
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/plugins/callback/__init__.py",
line 91, in get_option
return self._plugin_options[k]
Callback Exception:
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/executor/task_queue_manager.py",
line 372, in send_callback
method(*new_args, **kwargs)
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/plugins/callback/default.py",
line 240, in v2_runner_on_start
if self.get_option('show_per_host_start'):
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/plugins/callback/__init__.py",
line 91, in get_option
return self._plugin_options[k]
[task 339882] 00:00:08.172036 D ansible_mitogen.affinity: CPU mask for
WorkerProcess: 0x000004
Callback Exception:
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/executor/task_queue_manager.py",
line 372, in send_callback
method(*new_args, **kwargs)
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/plugins/callback/default.py",
line 240, in v2_runner_on_start
if self.get_option('show_per_host_start'):
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/plugins/callback/__init__.py",
line 91, in get_option
return self._plugin_options[k]
```
* origin/v028: (383 commits)
Bump version for release.
docs: update Changelog for 0.2.8.
issue #627: add test and tweak Reaper behaviour.
docs: lots more changelog concision
docs: changelog concision
docs: more changelog tweaks
docs: reorder chapters
docs: versionless <title>
docs: update supported Ansible version, mention unsupported features
docs: changelog fixes/tweaks
issue #590: update Changelog.
issue #621: send ADD_ROUTE earlier and add test for early logging.
issue #590: whoops, import missing test modules
issue #590: rework ParentEnumerationMethod to recursively handle bad modules
issue #627: reduce the default pool size in a child to 2.
tests: add a few extra service tests.
docs: some more hyperlink joy
docs: more hyperlinks
docs: add domainrefs plugin to make link aliases everywhere \o/
docs: link IS_DEAD in changelog
docs: tweaks to better explain changelog race
issue #533: update routing to account for DEL_ROUTE propagation race
tests: use defer_sync() Rather than defer() + ancient sync_with_broker()
tests: one case from doas_test was invoking su
tests: hide memory-mapped files from lsof output
issue #615: remove meaningless test
issue #625: ignore SIGINT within MuxProcess
issue #625: use exec() instead of subprocess in mitogen_ansible_playbook
issue #615: regression test
issue #615: update Changelog.
issue #615: ensure 4GB max_message_size is configured for task workers.
issue #615: update Changelog.
issue #615: route a dead message to recipients when no reply is expected
issue #615: fetch_file() might be called with AnsibleUnicode.
issue #615: redirect 'fetch' action to 'mitogen_fetch'.
issue #615: extricate slurp brainwrong from mitogen_fetch
issue #615: ansible: import Ansible fetch.py action plug-in
issue #533: include object identity of Stream in repr()
docs: lots more changelog
issue #595: add buildah to docs and changelog.
docs: a few more internals.rst additions
ci: update to Ansible 2.8.3
tests: another random string changed in 2.8.3
tests: fix sudo_flags_failure for Ansible 2.8.3
ci: fix procps command line format warning
Whoops, merge together lgtm.yml and .lgtm.yml
issue #440: log Python version during bootstrap.
docs: update changelog
issue #558: disable test on OSX to cope with boundless mediocrity
issue #558, #582: preserve remote tmpdir if caller did not supply one
issue #613: must await 'exit' and 'disconnect' in wait=False test
Import LGTM config to disable some stuff
Fix up another handful of LGTM errors.
tests: work around AnsibleModule.run_command() race.
docs: mention another __main__ safeguard
docs: tweaks
formatting error
docs: make Sphinx install soft fail on Python 2.
issue #598: allow disabling preempt in terraform
issue #598: update Changelog.
issue #605: update Changelog.
issue #605: ansible: share a sem_t instead of a pthread_mutex_t
issue #613: add tests for all the weird shutdown methods
Add mitogen.core.now() and use it everywhere; closes#614.
docs: move decorator docs into core.py and use autodecorator
preamble_size: make it work on Python 3.
docs: upgrade Sphinx to 2.1.2, require Python 3 to build docs.
docs: fix Sphinx warnings, add LogHandler, more docstrings
docs: tidy up some Changelog text
issue #615: fix up FileService tests for new logic
issue #615: another Py3x fix.
issue #615: Py3x fix.
issue #615: update Changelog.
issue #615: use FileService for target->controll file transfers
issue #482: another Py3 fix
ci: try removing exclude: to make Azure jobs work again
compat: fix Py2.4 SyntaxError
issue #482: remove 'ssh' from checked processes
ci: Py3 fix
issue #279: add one more test for max_message_size
issue #482: ci: add stray process checks to all jobs
tests: fix format string error
core: MitogenProtocol.is_privileged was not set in children
issue #482: tests: fail DockerMixin tests if stray processes exist
docs: update Changelog.
issue #586: update Changelog.
docs: update Changelog.
[security] core: undirectional routing wasn't respected in some cases
docs: tidy up Select.all()
issue #612: update Changelog.
master: fix TypeError
pkgutil: fix Python3 compatibility
parent: use protocol for getting remote_id
docs: merge signals.rst into internals.rst
os_fork: do not attempt to cork the active thread.
parent: fix get_log_level() for split out loggers.
issue #547: fix service_test failures.
issue #547: update Changelog.
issue #547: core/service: race/deadlock-free service pool init
docs: update Changelog.
...
Ansible has no blocking services running, or really any service that
would have an outsized benefit from multiple IO waiters. Probably we
only need 1, but let's start with 2 just in case.
The undocumented 'tmp' parameter controls whether _execute_module()
would delete anything on 2.3, so mimic that. This means
_execute_remove_stat() calls will not blow away the temp directory,
which broke the unarchive plugin.
Its functionality was duplicated by _on_broker_exit() somewhere along
the way, and nothing has referred to it in a long time. I have no idea
how this happened.
Merge its docstring into _on_broker_exit() and delete it, remove the
Router "shutdown" signal after confirming it has no users, and move all
the Router-originated error messages together in a block at the top of
the class.
Already covered by router_test.AddHandlerTest.test_dead_message_sent_at_shutdown
* origin/026:
docs: update Changelog for release.
Bump version for release.
issue #555: ansible: workaround ancient reload(sys) hack.
issue #554: mitogen_action_script fix
issue #554: fix Ansible 2.4 compatibility
issue #554: don't rely on tmp_path autoremoval in test.
issue #554: track and remove multiple make_tmp_path() calls.
docs: update Changelog.
docs: drastically simplify install/changelog.
issue #552: include process identity in log messages.
issue #550: update Changelog.
issue #550: parent: add explanatory comment.
issue #550: fix up TTY ioctls on WSL 2016 Anniversary Update
docs: update Changelog.
service: make service list optional.
docs: update Changelog; closes#548.
issue #548: always treat transport=smart as 'ssh' for mitogen_via=.
docs: better intro paragraph.
.ci: copy private key file to tempdir.
os_fork: more doc tweaks
os_fork: more doc tweaks
os_fork: yet more doc tidyup
os_fork: more doc tweaks
os_fork: clean up docs
.ci: import soak scripts.
.ci: allow containers for different jobs to run simultaneously
os_fork: python 3 fixes and tests.
issue #535: activate Corker on 2.4 in master too.
issue #535: update Changelog.
issue #535: wire mitogen.os_fork into Broker and Pool.
issue #535: parent: add create_socketpair(size=..) parameter.
issue #535: introduce mitogen.os_fork module and Corker class.
issue #535: docs: update Changelog
issue #535: service: support Pool.defer() like Broker.defer()
issue #535: core: unicode.encode() may take importer lock on 2.x
issue #535: docs: fix up Select doc
issue #535: docs: update Changelog.
issue #535: core/select: support selecting from Latches.
core: increase cookie field lengths to 64-bit; closes#545.
tests: ensure serialization restrictions are in effect
tests/bench: set process affinity in throughput.py.
docs: update copyright year.
docs: update Changelog.
core: Make Latch.put(obj=) optional.
docs: change 'unreleased' Changelog format and add a hint.
docs: update Changelog; closes#542.
issue #542: return of select poller, new selection logic
issue #542: .ci: move some tests to Azure and enable Mac job.
ansible: create stub __init__.py for sdist.
This is the most minimal change for what might be relatively minimal
edge case. Alternative is replacing reload(), but let's not do that yet.
Closes#555
Philippe Kueck