Commit Graph

787 Commits (2c7f2b4a9994ee2c564e4bf227d08efe2f7bc94d)

Author SHA1 Message Date
Alex Willmer 2c7f2b4a99 Experimental: mitogen: Remove minification of Python source code
Done on the basis that sending a few more bytes of the network is quicker than
tokenizing and untokeninzing the modules.
1 year ago
Alex Willmer 455fd2bcdf Bump version 1 year ago
Alex Willmer f18f5165cd Prep for v0.3.4 1 year ago
Alex Willmer 19b79f7ab5 CI: Fix tests on Linux, Ansible tests targetting Debian 9 & 11
Without Ubuntu 20.04 virtualenv package being installed pip was installing a
version of virtualenv that couldn't create the Tox environment for Python 2.7.

> Successfully installed distlib-0.3.6 filelock-3.12.2 platformdirs-3.8.0
> pluggy-1.2.0 py-1.11.0 tomli-2.0.1 tox-3.28.0 virtualenv-20.23.1
> Finishing: Install tooling
> ...
> py27-mode_mitogen-distro_centos6 create: /home/vsts/work/1/s/.tox/py27-
> mode_mitogen-distro_centos6
> ERROR: invocation failed (exit code 1), logfile: /home/vsts/work/1/s/.tox/
> py27-mode_mitogen-distro_centos6/log/py27-mode_mitogen-distro_centos6-0.log
> ================================== log start
> ===================================
> RuntimeError: failed to query /usr/bin/python2.7 with code 1 err:
> '  File "/home/vsts/.local/lib/python3.8/site-packages/virtualenv/discovery/
> py_info.py", line 24\n    return list(OrderedDict.fromkeys(["",
> *os.environ.get("PATHEXT", "").lower().split(os.pathsep)]))\n
> ^\nSyntaxError: invalid syntax\n'
1 year ago
Alex Willmer 21cb4a3472 CI: Remove faulthandler fallback requirement
faulthandler is a stdlib module in Python 3.3+. For a long time a PyPI package
of the same name was available for earlier Python releases. That package has
since been removed from PyPI, and the source respoitory archived. So we should
not rely on it.
fixes #983 refs #970
2 years ago
Alex Willmer 1871f2a9b1 Remove vendored mitogen.compat.simplejson
Python 2.6 added json to the stdlib. We no longer support Python <= 2.7 in
Mitogen 0.3.x, so this fallback is unneeded complexity. Fixes #659
2 years ago
Alex Willmer 0af2ce8c30 Remove ansible_mitogen Connection.close() workaround
Refs #925 #969

I'm not 100% confident that merely removing this is the full fix,
without substituting something else. I am sure keeping it would be
the greater of two evils. __del__() should be avoided on general
principal, and it's associated with multiple intermittant CI
failures, plus multiple user reported issues.
2 years ago
Alex Willmer e8ad12e881 Ansible 6 support
fixes #929
2 years ago
SAADY Yousef c1e72b8225 Fix typo changelog.rst 2 years ago
David Mehren a30a743ce7 Add ansible.builtin.dnf to ALWAYS_FORK_MODULES
The new fully qualified name of the DNF module needs to also be added to the list.

Fixes #832
2 years ago
Alex Willmer d71fb672e8 Begin v0.3.4.dev0 3 years ago
Alex Willmer c0d3deeac5 Prepare v0.3.3 3 years ago
Alex Willmer 25ea6dde02 ansible_mitogen: Allow mitogen_fetch to bypass slurp module
This reapplies an earlier change, when this plugin was first introduced to
Mitogen. The plugin was updated to fix

[DEPRECATION WARNING]: The '_remote_checksum()' method is deprecated.

I've elected to short-circuit the if statemtn logic, rather than
deleting/unindenting, to make the code delta much smaller. This should make it
easier to maintain/update.

Fixes #915
3 years ago
Alex Willmer e101cc4f44 mitogen.utils: Preserve docstring of functions decorated @with_router
Co-authored-by: Rezart Qelibari <gast-kontakt+mitogen@astzweig.de>

Replaces #837
Fixes #836
3 years ago
Alex Willmer 96e20a09d6 ansible_mitogen: Add podman connection plugin 3 years ago
Alex Willmer 0417d4d73a Replace os.system() with subprocess.check_call()
Non-zero return codes should raise an exception, not pass silently.
3 years ago
Alex Willmer 2a95d039ab Python 3.10 support 3 years ago
Alex Willmer d2ca8a9423 master.ParentEnumerationMethod: Require matching pkg.__name__
Co-authored-by: Stefano Rivera <stefano@rivera.za.net>

When the requested module (e.g. ansible.module_utils.distro)
- is provided by another module *e.g. distro)
- that itself was a package (e.g. distro 1.7.0)

At runtime
- ansible/module_utils/distro/__init__.py executes
- if https://pypi.org/project/distro/ is present, it's loaded as
ansible.module_utils.distro
- otherwise ansible/module_utils/distro/_distro.py is loaded

ParentEnumerationMethod would wrongly use whatever was in
sys.modules['ansible.module_utils.distro]. Instead we should ascend to
the first parent that has fullname == sys.modules[fullname].__name__.
Then descend to the appropriate .py file on disk.

This bug didn't show up before because until distro 1.7.0 (Feb 2022) the
top-level distro module was a module (distro.py) not a package
(distro/__init__.py)

fixes #906
3 years ago
Alex Willmer 5b8f7dd1be
Start v0.3.3 development 3 years ago
Alex Willmer e8c3fe7881
Fix Trove classifier, bump version
fixes #891

(cherry picked from commit 1a84184838)
3 years ago
Klaus Zerwes 54b9115f20
Update docs/ansible_detailed.rst
fixed supported ansible / python version listing

Co-authored-by: Alex Willmer <alex@moreati.org.uk>
3 years ago
Klaus Zerwes 56c341790c prepare support for ansible community 5 aka. core 2.12 3 years ago
Alex Willmer 552819e765 mitogen.parent: Detect and avoid Python2.7 wrapper on macOS 11 & 12
Without this errors such as the following occur
```
✗ MITOGEN_LOG_LEVEL=DEBUG python3 foo.py
Python: execv: (null): No such file or directory
Traceback (most recent call last):
  File "foo.py", line 16, in <module>
    target = router.local(python_path='/usr/bin/python2.7', debug=True)
  File "/Users/alex/src/mitogen2/mitogen/parent.py", line 2486, in local
    return self.connect(u'local', **kwargs)
  File "/Users/alex/src/mitogen2/mitogen/parent.py", line 2446, in connect
    return self._connect(klass, **mitogen.core.Kwargs(kwargs))
  File "/Users/alex/src/mitogen2/mitogen/parent.py", line 2426, in _connect
    conn.connect(context=context)
  File "/Users/alex/src/mitogen2/mitogen/parent.py", line 1708, in connect
    raise self.exception
mitogen.parent.EofError: EOF on stream; last 100 lines received:
MITO000
MITO001
```

Before
```
$ ./preamble_size.py
SSH command size: 625
Bootstrap (mitogen.core) size: 17007 (16.61KiB)

                              Original          Minimized           Compressed
mitogen.parent            97496 95.2KiB  50355 49.2KiB 51.6%  12663 12.4KiB 13.0%
mitogen.fork               8436  8.2KiB   4130  4.0KiB 49.0%   1648  1.6KiB 19.5%
mitogen.ssh               10892 10.6KiB   6952  6.8KiB 63.8%   2113  2.1KiB 19.4%
mitogen.sudo              12089 11.8KiB   5924  5.8KiB 49.0%   2249  2.2KiB 18.6%
mitogen.select            12325 12.0KiB   2929  2.9KiB 23.8%    964  0.9KiB 7.8%
mitogen.service           41644 40.7KiB  22431 21.9KiB 53.9%   5886  5.7KiB 14.1%
mitogen.fakessh           15599 15.2KiB   8011  7.8KiB 51.4%   2624  2.6KiB 16.8%
mitogen.master            48732 47.6KiB  24569 24.0KiB 50.4%   6768  6.6KiB 13.9%
```

After
```
$ ./preamble_size.py
SSH command size: 705
Bootstrap (mitogen.core) size: 17007 (16.61KiB)

                              Original          Minimized           Compressed
mitogen.parent            97885 95.6KiB  50516 49.3KiB 51.6%  12728 12.4KiB 13.0%
mitogen.fork               8436  8.2KiB   4130  4.0KiB 49.0%   1648  1.6KiB 19.5%
mitogen.ssh               10892 10.6KiB   6952  6.8KiB 63.8%   2113  2.1KiB 19.4%
mitogen.sudo              12089 11.8KiB   5924  5.8KiB 49.0%   2249  2.2KiB 18.6%
mitogen.select            12325 12.0KiB   2929  2.9KiB 23.8%    964  0.9KiB 7.8%
mitogen.service           41644 40.7KiB  22431 21.9KiB 53.9%   5886  5.7KiB 14.1%
mitogen.fakessh           15599 15.2KiB   8011  7.8KiB 51.4%   2624  2.6KiB 16.8%
mitogen.master            48733 47.6KiB  24570 24.0KiB 50.4%   6771  6.6KiB 13.9%
```
3 years ago
Alex Willmer 465ac8abff ansible: Fix AttributeError in kubectl connection 3 years ago
Alex Willmer e194a6367f ci: Fix version comparisons involving double digits
See https://gist.github.com/moreati/e7507c5b606b12ec0ddafcb7c8debbf1
3 years ago
Alex Willmer e6bc53ca3a mitogen.parent: Remove tabs from minimized first stage
Saves 4 bytes

Before
```
$ python preamble_size.py
SSH command size: 629
Bootstrap (mitogen.core) size: 17007 (16.61KiB)

                              Original          Minimized           Compressed
mitogen.parent            97497 95.2KiB  50356 49.2KiB 51.6%  12665 12.4KiB 13.0%
mitogen.fork               8436  8.2KiB   4130  4.0KiB 49.0%   1648  1.6KiB 19.5%
mitogen.ssh               10892 10.6KiB   6952  6.8KiB 63.8%   2113  2.1KiB 19.4%
mitogen.sudo              12089 11.8KiB   5924  5.8KiB 49.0%   2249  2.2KiB 18.6%
mitogen.select            12325 12.0KiB   2929  2.9KiB 23.8%    964  0.9KiB 7.8%
mitogen.service           41644 40.7KiB  22431 21.9KiB 53.9%   5886  5.7KiB 14.1%
mitogen.fakessh           15599 15.2KiB   8011  7.8KiB 51.4%   2624  2.6KiB 16.8%
mitogen.master            48732 47.6KiB  24569 24.0KiB 50.4%   6768  6.6KiB 13.9%
```

After
```
$ python preamble_size.py
SSH command size: 625
Bootstrap (mitogen.core) size: 17007 (16.61KiB)

                              Original          Minimized           Compressed
mitogen.parent            97496 95.2KiB  50355 49.2KiB 51.6%  12663 12.4KiB 13.0%
mitogen.fork               8436  8.2KiB   4130  4.0KiB 49.0%   1648  1.6KiB 19.5%
mitogen.ssh               10892 10.6KiB   6952  6.8KiB 63.8%   2113  2.1KiB 19.4%
mitogen.sudo              12089 11.8KiB   5924  5.8KiB 49.0%   2249  2.2KiB 18.6%
mitogen.select            12325 12.0KiB   2929  2.9KiB 23.8%    964  0.9KiB 7.8%
mitogen.service           41644 40.7KiB  22431 21.9KiB 53.9%   5886  5.7KiB 14.1%
mitogen.fakessh           15599 15.2KiB   8011  7.8KiB 51.4%   2624  2.6KiB 16.8%
mitogen.master            48732 47.6KiB  24569 24.0KiB 50.4%   6768  6.6KiB 13.9%
```
3 years ago
Alex Willmer dd6d73db37 mitogen.parent: Eliminate use of platform module in first stage
This reduces the size of the initial SSH command by 204 bytes, & may fix errors
running Mitogen on  macOS. AFAICT platform was used but not imported.

Before
```
$ python ./preamble_size.py
SSH command size: 833
Bootstrap (mitogen.core) size: 17007 (16.61KiB)

                              Original          Minimized           Compressed
mitogen.parent            97565 95.3KiB  50427 49.2KiB 51.7%  12689 12.4KiB 13.0%
mitogen.fork               8436  8.2KiB   4130  4.0KiB 49.0%   1648  1.6KiB 19.5%
mitogen.ssh               10892 10.6KiB   6952  6.8KiB 63.8%   2113  2.1KiB 19.4%
mitogen.sudo              12089 11.8KiB   5924  5.8KiB 49.0%   2249  2.2KiB 18.6%
mitogen.select            12325 12.0KiB   2929  2.9KiB 23.8%    964  0.9KiB 7.8%
mitogen.service           41644 40.7KiB  22431 21.9KiB 53.9%   5886  5.7KiB 14.1%
mitogen.fakessh           15599 15.2KiB   8011  7.8KiB 51.4%   2624  2.6KiB 16.8%
mitogen.master            48732 47.6KiB  24569 24.0KiB 50.4%   6768  6.6KiB 13.9%
```

After
```
$ python preamble_size.py
SSH command size: 629
Bootstrap (mitogen.core) size: 17007 (16.61KiB)

                              Original          Minimized           Compressed
mitogen.parent            97543 95.3KiB  50357 49.2KiB 51.6%  12665 12.4KiB 13.0%
mitogen.fork               8436  8.2KiB   4130  4.0KiB 49.0%   1648  1.6KiB 19.5%
mitogen.ssh               10892 10.6KiB   6952  6.8KiB 63.8%   2113  2.1KiB 19.4%
mitogen.sudo              12089 11.8KiB   5924  5.8KiB 49.0%   2249  2.2KiB 18.6%
mitogen.select            12325 12.0KiB   2929  2.9KiB 23.8%    964  0.9KiB 7.8%
mitogen.service           41644 40.7KiB  22431 21.9KiB 53.9%   5886  5.7KiB 14.1%
mitogen.fakessh           15599 15.2KiB   8011  7.8KiB 51.4%   2624  2.6KiB 16.8%
mitogen.master            48732 47.6KiB  24569 24.0KiB 50.4%   6768  6.6KiB 13.9%
```
3 years ago
Alex Willmer c61c063b4f Support for Ansible 3 & 4
fixes #834

Co-authored-by: Claude Becker (@upekkha)
Co-authored-by: Dolph Mathews (@dolph)
3 years ago
Denis Zalevskiy 84c567e265
Add podman connection support
Shameless copy of buildah connection with modifications of invocation to
fit podman CLI.

Signed-off-by: Denis Zalevskiy <dez@aiven.io>
3 years ago
Denis Zalevskiy 10caa4a104
Fix typo in buildah context creation docs
Signed-off-by: Denis Zalevskiy <dez@aiven.io>
3 years ago
Alex Willmer c5c65ab0d2 ci: Switch Continuous Integration to Tox
Also
 - Simplifies adding support for additional Ansible versions
 - Unifies Python package versioning in CI and local test environments
 - Matches Python versions tested, with those declared in setup.py
 - Expands targets covered by automated Ansible tests to
    - centos6, centos8
    - debian9, debian11
    - ubuntu1604, ubuntu2004
3 years ago
Alex Willmer e76eefb8be Fix miscellaneous spelling/formatting 3 years ago
Alex Willmer da3772cbff Increment version to 0.3.1.dev0 3 years ago
Alex Willmer a990eb3d77 CI: Remove obsolete reverse shell historically used to debug CI
This reverse shell was historically used to debug CI jobs interactively.
It is not used anymore, and may be causing jobs to hang, then timeout.
There is no reason to keep it, and removing it simplifies CI jobs.

Additionally it has been reported as flagged by security scanners, so
removing it makes Mitogen easier to package/adopt.

fixes #847
3 years ago
Philippe Kueck bdbc9fe827
changelog entry for #827 3 years ago
Steven Robertson 234dde5fc1 check Ansible version before loaders are loaded 4 years ago
Alex Willmer bce3bab3e8 Add the msvcrt moduleto the default module deny list
Commit https://github.com/python/cpython/commit/880d42a3b247 (first
released in Python 3.8a0) moved an import of msvcrt from an if <win32>
block, into a try/except block. So now the import is tried even on Linux
or MacOS.

https://docs.python.org/3/library/msvcrt.html is a Windows specific
builtin.
4 years ago
Alex Willmer 1e72ebaf8b tests: Update test coverage to Python 3.9
The Travis Linux distribution must be upgraded because the Trusty
(Ubuntu 14.04) image does not have Python 3.9. Xenial (Ubuntu 16.04) is
the earliest version that offers Python 3.9.

I have not chosen a later release, in order to aid restoration of Python
2.4 - 2.6 tests.
4 years ago
David Wilson fad6eb1b47 Bump copyright year and use generic author name (testing Travis) 4 years ago
Steven Robertson f489478127 code cleanup + adds 0.2.10 + 0.3.0 changelog 4 years ago
Steven Robertson 8d3da2dbd2
Merge branch 'master' into patch-1 4 years ago
Alex Willmer 79b4c0f815 tests: Fix AttributeError in callback plugins used by test suite
CALLBACK_VERSION et al are documented as required in
https://docs.ansible.com/ansible/2.10/dev_guide/developing_plugins.html#callback-plugins.
The need for document_fragment is noted in
cfa8075537/lib/ansible/plugins/callback/default.py (L28-L32)

Fixes #758

This addresses the following error, seen while running
`ansible_tests.py`.

```
TASK [Gathering Facts gather_timeout=10, gather_subset=['all']]
****************
task path:
/home/alex/src/mitogen/tests/ansible/regression/issue_109__target_has_old_ansible_installed.yml:4
[WARNING]: Failure using method (v2_runner_on_start) in callback plugin
(<ansible.plugins.callback.nice_stdout.CallbackModule object at
0x7f76b3dad090>): 'show_per_host_start'
Callback Exception:
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/executor/task_queue_manager.py",
line 372, in send_callback
    method(*new_args, **kwargs)
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/plugins/callback/default.py",
line 240, in v2_runner_on_start
    if self.get_option('show_per_host_start'):
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/plugins/callback/__init__.py",
line 91, in get_option
    return self._plugin_options[k]
Callback Exception:
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/executor/task_queue_manager.py",
line 372, in send_callback
    method(*new_args, **kwargs)
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/plugins/callback/default.py",
line 240, in v2_runner_on_start
    if self.get_option('show_per_host_start'):
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/plugins/callback/__init__.py",
line 91, in get_option
    return self._plugin_options[k]
[task 339882] 00:00:08.172036 D ansible_mitogen.affinity: CPU mask for
WorkerProcess: 0x000004
Callback Exception:
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/executor/task_queue_manager.py",
line 372, in send_callback
    method(*new_args, **kwargs)
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/plugins/callback/default.py",
line 240, in v2_runner_on_start
    if self.get_option('show_per_host_start'):
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/plugins/callback/__init__.py",
line 91, in get_option
    return self._plugin_options[k]
```
4 years ago
Alex Willmer b0ce29dcfd ssh: Match newer ssh host key prompt that accepts the fingerprint
This fixes an ERROR in test_accept_enforce_host_keys() while running the
test suite.

Fixes #756
4 years ago
Steven Robertson 7d4a57d16a
Merge branch 'master' into patch-1 4 years ago
Nicolas Mattia 4d48f140a8 Fix typo in Ansible documentation 4 years ago
Luiz Ribeiro 0e47280e43
Fix mitogen_ssh_keepalive_interval documentation 5 years ago
Scott Buchanan fa7c0191ee
correct latest ansible version supported 5 years ago
Steven Robertson 2d7e019dd2
Update docs/ansible_detailed.rst
Co-Authored-By: Jacob Floyd <cognifloyd@gmail.com>
5 years ago
Steven Robertson 1dddfadf32 removing note saying ansible 2.8 interpreter discovery doesn't work 5 years ago
Steven Robertson e8f3154cab Merge branch 'master' into complexAnsiblePythonInterpreterArg 5 years ago