Commit Graph

831 Commits (27214517a724431fb5dc73255b608bc4fe078479)

Author SHA1 Message Date
Alex Willmer 34088a8b7f ansible_mitogen: Consolidate Python 2 & 3 compatibility
Rough guidelines, in decending preference:
- Use mitogen.core if possible
- Use ansible.module_utils.six if possible
- Embed a getattr() or try/except

viewkeys() et al can't be brought into mitogen.core because that package still
targets Python 2.4. dict.viewkeys() were introduced in Python 2.7.
2 months ago
Alex Willmer 0a908d76da ansible_mitogen: Remove fallback imports for Ansible < 2.10 2 months ago
Alex Willmer b1fd6038bf ansible_mitogen: Remove Python 2.4 and 2.5 backward compatibility fallbacks
Because ansible_mitogen >= 0.3 supports Ansible >= 2.10 and Ansible 2.10
requires supports Python >= 2.7 on controllers and Python >= 2.6 on targets
these are dead weight.

See
- https://docs.ansible.com/ansible/latest/reference_appendices/release_and_maintenance.html#ansible-core-support-matrix
- tox.ini
2 months ago
Alex Willmer c6cf08ab39 mitogen: Consolidate back compatibility fallbacks and polyfills in mitogen.core
This saves some bytes on the wire ad simplifies reasoning about the code.
2 months ago
Alex Willmer ce6297b0e9 Begin v0.3.11 2 months ago
Alex Willmer cea2e7b98d Prepare v0.3.10 2 months ago
Gaige B Paulsen 2ba1b2b3f8
Fix: termios.error: (22, 'Invalid argument') during `become` on Solaris/Illumos/SmartOS (#1089)
This fixes compatibility with Solaris/Illumos/SmartOS, addressing an issue that shows up most frequently with become. The issue was mostly due to differences in how the TTY driver is handled and the pty driver not supporting echo on both sides of the pipe (as designed, from a Solaris point of view).

Fixes #950

Co-authored-by: Alex Willmer <alex@moreati.org.uk>
2 months ago
Alex Willmer d032c591c2 tests: Retry container process check during teardown
I'm about 75% sure the check is an unavoidable race condition, see
https://github.com/mitogen-hq/mitogen/issues/694#issuecomment-2338001694. If
it occurs again, then reopen the issue.

Fixes #694
2 months ago
Alex Willmer 79ed797bad tests: Test templating of ansible_ssh_common_args et al
refs #905
3 months ago
Alex Willmer 5af6534a70 tests: Test AnsibleUnsafeText when copying files larger SMALL_FILE_LIMIT
The bug was fixed in a previous commit by Jonathan Rosser. This adds testing.
The bug is only triggered when the copy module is used inside a `with_items:`
loop and the destination filename has an extension. A `loop:` loop is not
sufficient.

refs #1110
3 months ago
Alex Willmer 5ab872f289 ansible_mitogen: Add regression test for ActionModuleMixin._remote_chmod()
Adapted from Jonathon's reproducer in #1087.
3 months ago
Alex Willmer d15051b187 Begin v0.3.10dev 3 months ago
Alex Willmer 6fbad3ae7d Prepare v0.3.9 3 months ago
Alex Willmer c948e6668a docs: Remove email form from website
https://networkgenomics.com is no longer served, so the form submission would
fail.
3 months ago
Alex Willmer 2edcb31996 docs: Remove piwik analytics hooks from website
https://networkgenomics.com is no longer served, so the javascript and other
attempts to beacon or redirect result in HTTP errors.
3 months ago
Alex Willmer 357fe38766 Ansible 10 (ansible-core 2.17) support
Notably
- Python 2.7 and 3.6 are no longer supported by Ansible on targets
- The yum module has been removed, and redirected to dnf
- _INTERPRETER_PYTHON_DISTRO_MAP has been neutered. Interpreter discovery
  always favours specific `python3.<x>` interpreters in decending version
  order, then generic `python3` or `python`.
- Add the ability for an action plugin to call self._execute_module(*,
  ignore_unknown_opts=True) to execute a module with options that may not be
  supported for the version being called.

https://docs.ansible.com/ansible/devel/porting_guides/porting_guide_10.html
https://github.com/ansible-community/ansible-build-data/blob/main/10/CHANGELOG-v10.md
https://github.com/ansible/ansible/blob/stable-2.17/changelogs/CHANGELOG-v2.17.rst

fixes #1074, refs #1082

Co-authored-by: Claude Becker <becker@phys.ethz.ch>
3 months ago
Alex Willmer 40695f413b ansible_mitogen: Respect ansible_facts.discovered_interpreter_python more
fixes #1097
3 months ago
Alex Willmer 62cde17150 Start v0.3.9 development 4 months ago
Alex Willmer e334b50d9d Prepare v0.3.8 4 months ago
Alex Willmer 924dbd6f0c CI: Migrate macOS integration tests to macOS 12, drop Python 2.7 jobs
macOS 11 is not longer an available runner on Azure Devops. The minimum is now
macOS 12. This runner does not have Python 2.7 installed, so running them
would require a custom install - which I'm declaring too much effort for too
little gain.

refs #1090
4 months ago
Alex Willmer 05d98e5b49 tests: Speed up ssh timeout tests 6 months ago
Philippe Kueck ec05e542b4
Fix 'ansible_host_key_checking' and 'ansible_ssh_host_key_checking' for
adding new hosts to the inventory using 'add_hosts'

Co-authored-by: Alex Willmer <alex@moreati.org.uk>
7 months ago
Alex Willmer 4996ec2f09 ansible_mitogen: Fix "filedescriptor out of range in select()" in WorkerProcess
`mitogen.parent.POLLER_LIGHTWEIGHT` will normally be `PollPoller`, falling
back to `EpollPoller`, `KqueuePoller`, or `Poller`.

Fixes #957

Co-authored-by: Luca Berruti <nadirio@gmail.com>
Co-authored-by: Philippe Kueck <bqobccy6ejnq2bqvmebqiwqha4cs4@protected32.unixadm.org>
7 months ago
Alex Willmer d5e9186289 ansible_mitogen: Fix --ask-become-pass, add test coverage
Previously f1503874de fixed the priority of
ansible_become_pass over ansible_become_password, but broke --ask-become-pass.
Fixes #952.
7 months ago
Alex Willmer 37ebce7e6e Begin 0.3.8dev 8 months ago
Alex Willmer a3644963c4 Prepare v0.3.7 8 months ago
Alex Willmer cca651da1f ansible_mitogen: Ansible 9 (ansible-core 2.16) support 8 months ago
Alex Willmer 45c42d386a tests: Replace uses of ``include:``, unify skipping of mitogen only tests
The tag mitogen_only is only informational for now. It may be possible to use
it with ANSIBLE_SKIP_TAGS in the future.
8 months ago
Alex Willmer fa1d21747f ansible_mitogen: Declare Ansible 8 (ansible-core 2.15) support
refs #1021
8 months ago
Alex Willmer 933477fcbe Begin 0.3.7dev 8 months ago
Alex Willmer 5d789faee5 Prepare 0.3.6 8 months ago
Alex Willmer b822f20007 ansible_mitogen: Handle AnsibleUnsafeText et al in Ansible >= 7
Follwing fixes in Ansible 7-9 for CVE-2023-5764 cating `AnsibleUnsafeBytes` &
`AnsibleUnsafeText` to `bytes()` or `str()` requires special handling. The
handling is Ansible specific, so it shouldn't go in the mitogen package but
rather the ansible_mitogen package.

`ansible_mitogen.utils.unsafe.cast()` is most like `mitogen.utils.cast()`.
During development it began as `ansible_mitogen.utils.unsafe.unwrap_var()`,
closer to an inverse of `ansible.utils.unsafe_procy.wrap_var()`. Future
enhancements may move in this direction.

refs #977, refs #1046

See also
- https://github.com/advisories/GHSA-7j69-qfc3-2fq9
- https://github.com/ansible/ansible/pull/82293
- https://github.com/mitogen-hq/mitogen/wiki/AnsibleUnsafe-notes
8 months ago
Alex Willmer d7979c3597 mitogen: Raise TypeError on `mitogen.utils.cast(custom_str)` failures
If casting a string fails then raise a TypeError. This is potentially an API
breaking change; chosen as the lesser evil vs. allowing silent errors.

`cast()` relies on `bytes(obj)` & `str(obj)` returning the respective
supertype. That's no longer the case for `AnsibleUnsafeBytes` &
`AnsibleUnsafeText`; since fixes/mitigations for  CVE-2023-5764.

fixes #1046, refs #977

See also
- https://github.com/advisories/GHSA-7j69-qfc3-2fq9
- https://github.com/ansible/ansible/pull/82293
8 months ago
Orion Poplawski dfc3c7d516 ansible_mitogen: Add Ansible 7 support
Co-authored-by: Orion Poplawski <orion@nwra.com>
8 months ago
Alex Willmer 50efa53f8f
docs: Correct PEP 451 hyperlink 8 months ago
Alex Willmer fc24b3f25e Start v0.3.6 development 8 months ago
Alex Willmer e97ab2f597 Prepare v0.3.5 8 months ago
Alex Willmer 123efa7510 mitogen: Support Python 3.12
Most of the necessary changes were made in recent PEP 451 commits. This bumps
the CI jobs, and declares the support. Test dependendancies are bumped to
latest supportted/available versions.

refs #1033
8 months ago
Alex Willmer 5ad3d14ceb mitogen: Support PEP 451 ModuleSpec API, required for Python 3.12
importlib.machinery.ModuleSpec and find_spec() were introduced in Python 3.4
under PEP 451. They replace the find_module() API of PEP 302, which was
deprecated from Python 3.4. They were removed in Python 3.12 along with the
imp module.

This change adds support for the PEP 451 APIs. Mitogen should no longer import
imp on Python versions that support ModuleSpec. Tests have been added to cover
the new APIs.

CI jobs have been added to cover Python 3.x on macOS.

Refs #1033
Co-authored-by: Witold Baryluk <witold.baryluk@gmail.com>
8 months ago
Alex Willmer 3a31a7d886 mitogen: Workaround CPython importlib PermissionError when cwd is unreadable
On macOS when using a become plugin as an unprivileged user, to another
unprivileged user it is likely that the current working directory can't be
read. In this case os.cwd() raises PermissionError.

On versions of Python currently in the wild (March 2024, CPython <= 3.13) if
any non-builtin or non-frozen module (e.g. zlib, base64) is imported then
`importlib._bootstrap_external.PathFinder._path_importer_cache()` attempts to
call os.cwd() without catching PermissionError.

The previous comment about needing an extra .encode() appears to be wrong,
atleast for Python 3.x >= 3.6.

Command size increased by 54 bytes, bootstrap by 804 bytes. Changed from
codecs module to binascii & zlib because they're extensions, and importing
them triggers fewer supporting imports (e.g. encodings module).

Before

```
✗ ./preamble_size.py
SSH command size: 705
Bootstrap (mitogen.core) size: 17078 (16.68KiB)

                              Original          Minimized           Compressed
mitogen.parent            97884 95.6KiB  50515 49.3KiB 51.6%  12727 12.4KiB
13.0%
mitogen.fork               8436  8.2KiB   4130  4.0KiB 49.0%   1648  1.6KiB
19.5%
mitogen.ssh               10892 10.6KiB   6952  6.8KiB 63.8%   2113  2.1KiB
19.4%
mitogen.sudo              12089 11.8KiB   5924  5.8KiB 49.0%   2249  2.2KiB
18.6%
mitogen.select            12325 12.0KiB   2929  2.9KiB 23.8%    964  0.9KiB
7.8%
mitogen.service           41699 40.7KiB  22477 22.0KiB 53.9%   5885  5.7KiB
14.1%
mitogen.fakessh           15577 15.2KiB   7989  7.8KiB 51.3%   2623  2.6KiB
16.8%
mitogen.master            51398 50.2KiB  25715 25.1KiB 50.0%   6886  6.7KiB
13.4%
```

After

```
✗ ./preamble_size.py
SSH command size: 759
Bootstrap (mitogen.core) size: 17882 (17.46KiB)

                              Original          Minimized           Compressed
mitogen.parent            98173 95.9KiB  50571 49.4KiB 51.5%  12747 12.4KiB
13.0%
mitogen.fork               8436  8.2KiB   4130  4.0KiB 49.0%   1648  1.6KiB
19.5%
mitogen.ssh               10892 10.6KiB   6952  6.8KiB 63.8%   2113  2.1KiB
19.4%
mitogen.sudo              12089 11.8KiB   5924  5.8KiB 49.0%   2249  2.2KiB
18.6%
mitogen.select            12325 12.0KiB   2929  2.9KiB 23.8%    964  0.9KiB
7.8%
mitogen.service           41699 40.7KiB  22477 22.0KiB 53.9%   5885  5.7KiB
14.1%
mitogen.fakessh           15577 15.2KiB   7989  7.8KiB 51.3%   2623  2.6KiB
16.8%
mitogen.master            56116 54.8KiB  29427 28.7KiB 52.4%   7627  7.4KiB
13.6%
```

Fixes #885
Refs https://github.com/python/cpython/issues/115911
8 months ago
Alex Willmer b7188c1cad docs: Decouple website download version from package version
This prevents unreleased versions appearing on the website (e.g. 0.3.5.dev0),
but introduces the risk of forgetting to update the website after a release.
A better fix requires deeper design/workflow thought.

refs #1028
1 year ago
Alex Willmer e580258071 docs: Bypass networkgenomics.com/try/ -> PyPI redirect
refs #1028
1 year ago
Alex Willmer 63457b4866 docs: Update external URLs (e.g. dw/mitogen -> mitogen-hq/mitogen)
Found with sphinx-build -b linkcheck. Not all flagged URLs have been changed,
e.g. Ansible plugins, deleted Github users.
1 year ago
Alex Willmer 6aa4fd3573 docs: Fix generation of static website
Bare minimum syntax errors and requirements constraints to work with Netlify
hosting.
1 year ago
Nerijus Baliūnas 4089e875a9 Add Python 3.11 support
Co-authored-by: Alex Willmer <alex@moreati.org.uk>
1 year ago
Alex Willmer 455fd2bcdf Bump version 1 year ago
Alex Willmer f18f5165cd Prep for v0.3.4 1 year ago
Alex Willmer 19b79f7ab5 CI: Fix tests on Linux, Ansible tests targetting Debian 9 & 11
Without Ubuntu 20.04 virtualenv package being installed pip was installing a
version of virtualenv that couldn't create the Tox environment for Python 2.7.

> Successfully installed distlib-0.3.6 filelock-3.12.2 platformdirs-3.8.0
> pluggy-1.2.0 py-1.11.0 tomli-2.0.1 tox-3.28.0 virtualenv-20.23.1
> Finishing: Install tooling
> ...
> py27-mode_mitogen-distro_centos6 create: /home/vsts/work/1/s/.tox/py27-
> mode_mitogen-distro_centos6
> ERROR: invocation failed (exit code 1), logfile: /home/vsts/work/1/s/.tox/
> py27-mode_mitogen-distro_centos6/log/py27-mode_mitogen-distro_centos6-0.log
> ================================== log start
> ===================================
> RuntimeError: failed to query /usr/bin/python2.7 with code 1 err:
> '  File "/home/vsts/.local/lib/python3.8/site-packages/virtualenv/discovery/
> py_info.py", line 24\n    return list(OrderedDict.fromkeys(["",
> *os.environ.get("PATHEXT", "").lower().split(os.pathsep)]))\n
> ^\nSyntaxError: invalid syntax\n'
1 year ago
Alex Willmer 21cb4a3472 CI: Remove faulthandler fallback requirement
faulthandler is a stdlib module in Python 3.3+. For a long time a PyPI package
of the same name was available for earlier Python releases. That package has
since been removed from PyPI, and the source respoitory archived. So we should
not rely on it.
fixes #983 refs #970
2 years ago
Alex Willmer 1871f2a9b1 Remove vendored mitogen.compat.simplejson
Python 2.6 added json to the stdlib. We no longer support Python <= 2.7 in
Mitogen 0.3.x, so this fallback is unneeded complexity. Fixes #659
2 years ago