Commit Graph

776 Commits (pr683)

Author SHA1 Message Date
Alex Willmer 70a6d0d7dc Get mitogen.fakessh module working again
Fixes include

- Setting cloexec flag on pipe files, using set_inheritable on sockets,
  and close_fds=False on subprocess.Popen to work around file
  descriptors not being inheritable by default in new versions of python

- Adding mitogen.exit_status variable and avoiding os.kill call so fake
  'ssh' script is able to exit cleanly with correct status code

- Fixing broken os.dup call in ExternalContext._setup_master when input
  and output streams have the same descriptor

- Updating fakessh module to do necessary python3 string/byte
  conversions, and use updated mitogen Protocol, Stream, and Router apis

- Simplifying fakessh startup sequence so there aren't unnecessary
  differences between ways control and data handles are passed, and ways
  master and slave processes are initialized

- Fixing shutdown race conditions where subprocess exit handling or
  stdin EOF handling could result in a truncated stdout stream

- Updating and adding a lot of docstrings and comments

- Adding Process.proc is None / is not None assertions to be clear about
  which parts of fakessh.Process code are specific to the slave process,
  and which parts are specific to the master process.

- Re-enabling unit test case and updating an outdated file path so it
  passes
3 years ago
Alex Willmer 25ea6dde02 ansible_mitogen: Allow mitogen_fetch to bypass slurp module
This reapplies an earlier change, when this plugin was first introduced to
Mitogen. The plugin was updated to fix

[DEPRECATION WARNING]: The '_remote_checksum()' method is deprecated.

I've elected to short-circuit the if statemtn logic, rather than
deleting/unindenting, to make the code delta much smaller. This should make it
easier to maintain/update.

Fixes #915
3 years ago
Alex Willmer e101cc4f44 mitogen.utils: Preserve docstring of functions decorated @with_router
Co-authored-by: Rezart Qelibari <gast-kontakt+mitogen@astzweig.de>

Replaces #837
Fixes #836
3 years ago
Alex Willmer 96e20a09d6 ansible_mitogen: Add podman connection plugin 3 years ago
Alex Willmer 0417d4d73a Replace os.system() with subprocess.check_call()
Non-zero return codes should raise an exception, not pass silently.
3 years ago
Alex Willmer 2a95d039ab Python 3.10 support 3 years ago
Alex Willmer d2ca8a9423 master.ParentEnumerationMethod: Require matching pkg.__name__
Co-authored-by: Stefano Rivera <stefano@rivera.za.net>

When the requested module (e.g. ansible.module_utils.distro)
- is provided by another module *e.g. distro)
- that itself was a package (e.g. distro 1.7.0)

At runtime
- ansible/module_utils/distro/__init__.py executes
- if https://pypi.org/project/distro/ is present, it's loaded as
ansible.module_utils.distro
- otherwise ansible/module_utils/distro/_distro.py is loaded

ParentEnumerationMethod would wrongly use whatever was in
sys.modules['ansible.module_utils.distro]. Instead we should ascend to
the first parent that has fullname == sys.modules[fullname].__name__.
Then descend to the appropriate .py file on disk.

This bug didn't show up before because until distro 1.7.0 (Feb 2022) the
top-level distro module was a module (distro.py) not a package
(distro/__init__.py)

fixes #906
3 years ago
Alex Willmer 5b8f7dd1be
Start v0.3.3 development 3 years ago
Alex Willmer e8c3fe7881
Fix Trove classifier, bump version
fixes #891

(cherry picked from commit 1a84184838)
3 years ago
Klaus Zerwes 54b9115f20
Update docs/ansible_detailed.rst
fixed supported ansible / python version listing

Co-authored-by: Alex Willmer <alex@moreati.org.uk>
3 years ago
Klaus Zerwes 56c341790c prepare support for ansible community 5 aka. core 2.12 3 years ago
Alex Willmer 552819e765 mitogen.parent: Detect and avoid Python2.7 wrapper on macOS 11 & 12
Without this errors such as the following occur
```
✗ MITOGEN_LOG_LEVEL=DEBUG python3 foo.py
Python: execv: (null): No such file or directory
Traceback (most recent call last):
  File "foo.py", line 16, in <module>
    target = router.local(python_path='/usr/bin/python2.7', debug=True)
  File "/Users/alex/src/mitogen2/mitogen/parent.py", line 2486, in local
    return self.connect(u'local', **kwargs)
  File "/Users/alex/src/mitogen2/mitogen/parent.py", line 2446, in connect
    return self._connect(klass, **mitogen.core.Kwargs(kwargs))
  File "/Users/alex/src/mitogen2/mitogen/parent.py", line 2426, in _connect
    conn.connect(context=context)
  File "/Users/alex/src/mitogen2/mitogen/parent.py", line 1708, in connect
    raise self.exception
mitogen.parent.EofError: EOF on stream; last 100 lines received:
MITO000
MITO001
```

Before
```
$ ./preamble_size.py
SSH command size: 625
Bootstrap (mitogen.core) size: 17007 (16.61KiB)

                              Original          Minimized           Compressed
mitogen.parent            97496 95.2KiB  50355 49.2KiB 51.6%  12663 12.4KiB 13.0%
mitogen.fork               8436  8.2KiB   4130  4.0KiB 49.0%   1648  1.6KiB 19.5%
mitogen.ssh               10892 10.6KiB   6952  6.8KiB 63.8%   2113  2.1KiB 19.4%
mitogen.sudo              12089 11.8KiB   5924  5.8KiB 49.0%   2249  2.2KiB 18.6%
mitogen.select            12325 12.0KiB   2929  2.9KiB 23.8%    964  0.9KiB 7.8%
mitogen.service           41644 40.7KiB  22431 21.9KiB 53.9%   5886  5.7KiB 14.1%
mitogen.fakessh           15599 15.2KiB   8011  7.8KiB 51.4%   2624  2.6KiB 16.8%
mitogen.master            48732 47.6KiB  24569 24.0KiB 50.4%   6768  6.6KiB 13.9%
```

After
```
$ ./preamble_size.py
SSH command size: 705
Bootstrap (mitogen.core) size: 17007 (16.61KiB)

                              Original          Minimized           Compressed
mitogen.parent            97885 95.6KiB  50516 49.3KiB 51.6%  12728 12.4KiB 13.0%
mitogen.fork               8436  8.2KiB   4130  4.0KiB 49.0%   1648  1.6KiB 19.5%
mitogen.ssh               10892 10.6KiB   6952  6.8KiB 63.8%   2113  2.1KiB 19.4%
mitogen.sudo              12089 11.8KiB   5924  5.8KiB 49.0%   2249  2.2KiB 18.6%
mitogen.select            12325 12.0KiB   2929  2.9KiB 23.8%    964  0.9KiB 7.8%
mitogen.service           41644 40.7KiB  22431 21.9KiB 53.9%   5886  5.7KiB 14.1%
mitogen.fakessh           15599 15.2KiB   8011  7.8KiB 51.4%   2624  2.6KiB 16.8%
mitogen.master            48733 47.6KiB  24570 24.0KiB 50.4%   6771  6.6KiB 13.9%
```
3 years ago
Alex Willmer 465ac8abff ansible: Fix AttributeError in kubectl connection 3 years ago
Alex Willmer e194a6367f ci: Fix version comparisons involving double digits
See https://gist.github.com/moreati/e7507c5b606b12ec0ddafcb7c8debbf1
3 years ago
Alex Willmer e6bc53ca3a mitogen.parent: Remove tabs from minimized first stage
Saves 4 bytes

Before
```
$ python preamble_size.py
SSH command size: 629
Bootstrap (mitogen.core) size: 17007 (16.61KiB)

                              Original          Minimized           Compressed
mitogen.parent            97497 95.2KiB  50356 49.2KiB 51.6%  12665 12.4KiB 13.0%
mitogen.fork               8436  8.2KiB   4130  4.0KiB 49.0%   1648  1.6KiB 19.5%
mitogen.ssh               10892 10.6KiB   6952  6.8KiB 63.8%   2113  2.1KiB 19.4%
mitogen.sudo              12089 11.8KiB   5924  5.8KiB 49.0%   2249  2.2KiB 18.6%
mitogen.select            12325 12.0KiB   2929  2.9KiB 23.8%    964  0.9KiB 7.8%
mitogen.service           41644 40.7KiB  22431 21.9KiB 53.9%   5886  5.7KiB 14.1%
mitogen.fakessh           15599 15.2KiB   8011  7.8KiB 51.4%   2624  2.6KiB 16.8%
mitogen.master            48732 47.6KiB  24569 24.0KiB 50.4%   6768  6.6KiB 13.9%
```

After
```
$ python preamble_size.py
SSH command size: 625
Bootstrap (mitogen.core) size: 17007 (16.61KiB)

                              Original          Minimized           Compressed
mitogen.parent            97496 95.2KiB  50355 49.2KiB 51.6%  12663 12.4KiB 13.0%
mitogen.fork               8436  8.2KiB   4130  4.0KiB 49.0%   1648  1.6KiB 19.5%
mitogen.ssh               10892 10.6KiB   6952  6.8KiB 63.8%   2113  2.1KiB 19.4%
mitogen.sudo              12089 11.8KiB   5924  5.8KiB 49.0%   2249  2.2KiB 18.6%
mitogen.select            12325 12.0KiB   2929  2.9KiB 23.8%    964  0.9KiB 7.8%
mitogen.service           41644 40.7KiB  22431 21.9KiB 53.9%   5886  5.7KiB 14.1%
mitogen.fakessh           15599 15.2KiB   8011  7.8KiB 51.4%   2624  2.6KiB 16.8%
mitogen.master            48732 47.6KiB  24569 24.0KiB 50.4%   6768  6.6KiB 13.9%
```
3 years ago
Alex Willmer dd6d73db37 mitogen.parent: Eliminate use of platform module in first stage
This reduces the size of the initial SSH command by 204 bytes, & may fix errors
running Mitogen on  macOS. AFAICT platform was used but not imported.

Before
```
$ python ./preamble_size.py
SSH command size: 833
Bootstrap (mitogen.core) size: 17007 (16.61KiB)

                              Original          Minimized           Compressed
mitogen.parent            97565 95.3KiB  50427 49.2KiB 51.7%  12689 12.4KiB 13.0%
mitogen.fork               8436  8.2KiB   4130  4.0KiB 49.0%   1648  1.6KiB 19.5%
mitogen.ssh               10892 10.6KiB   6952  6.8KiB 63.8%   2113  2.1KiB 19.4%
mitogen.sudo              12089 11.8KiB   5924  5.8KiB 49.0%   2249  2.2KiB 18.6%
mitogen.select            12325 12.0KiB   2929  2.9KiB 23.8%    964  0.9KiB 7.8%
mitogen.service           41644 40.7KiB  22431 21.9KiB 53.9%   5886  5.7KiB 14.1%
mitogen.fakessh           15599 15.2KiB   8011  7.8KiB 51.4%   2624  2.6KiB 16.8%
mitogen.master            48732 47.6KiB  24569 24.0KiB 50.4%   6768  6.6KiB 13.9%
```

After
```
$ python preamble_size.py
SSH command size: 629
Bootstrap (mitogen.core) size: 17007 (16.61KiB)

                              Original          Minimized           Compressed
mitogen.parent            97543 95.3KiB  50357 49.2KiB 51.6%  12665 12.4KiB 13.0%
mitogen.fork               8436  8.2KiB   4130  4.0KiB 49.0%   1648  1.6KiB 19.5%
mitogen.ssh               10892 10.6KiB   6952  6.8KiB 63.8%   2113  2.1KiB 19.4%
mitogen.sudo              12089 11.8KiB   5924  5.8KiB 49.0%   2249  2.2KiB 18.6%
mitogen.select            12325 12.0KiB   2929  2.9KiB 23.8%    964  0.9KiB 7.8%
mitogen.service           41644 40.7KiB  22431 21.9KiB 53.9%   5886  5.7KiB 14.1%
mitogen.fakessh           15599 15.2KiB   8011  7.8KiB 51.4%   2624  2.6KiB 16.8%
mitogen.master            48732 47.6KiB  24569 24.0KiB 50.4%   6768  6.6KiB 13.9%
```
3 years ago
Alex Willmer c61c063b4f Support for Ansible 3 & 4
fixes #834

Co-authored-by: Claude Becker (@upekkha)
Co-authored-by: Dolph Mathews (@dolph)
3 years ago
Denis Zalevskiy 84c567e265
Add podman connection support
Shameless copy of buildah connection with modifications of invocation to
fit podman CLI.

Signed-off-by: Denis Zalevskiy <dez@aiven.io>
3 years ago
Denis Zalevskiy 10caa4a104
Fix typo in buildah context creation docs
Signed-off-by: Denis Zalevskiy <dez@aiven.io>
3 years ago
Alex Willmer c5c65ab0d2 ci: Switch Continuous Integration to Tox
Also
 - Simplifies adding support for additional Ansible versions
 - Unifies Python package versioning in CI and local test environments
 - Matches Python versions tested, with those declared in setup.py
 - Expands targets covered by automated Ansible tests to
    - centos6, centos8
    - debian9, debian11
    - ubuntu1604, ubuntu2004
3 years ago
Alex Willmer e76eefb8be Fix miscellaneous spelling/formatting 3 years ago
Alex Willmer da3772cbff Increment version to 0.3.1.dev0 3 years ago
Alex Willmer a990eb3d77 CI: Remove obsolete reverse shell historically used to debug CI
This reverse shell was historically used to debug CI jobs interactively.
It is not used anymore, and may be causing jobs to hang, then timeout.
There is no reason to keep it, and removing it simplifies CI jobs.

Additionally it has been reported as flagged by security scanners, so
removing it makes Mitogen easier to package/adopt.

fixes #847
3 years ago
Philippe Kueck bdbc9fe827
changelog entry for #827 3 years ago
Steven Robertson 234dde5fc1 check Ansible version before loaders are loaded 4 years ago
Alex Willmer bce3bab3e8 Add the msvcrt moduleto the default module deny list
Commit https://github.com/python/cpython/commit/880d42a3b247 (first
released in Python 3.8a0) moved an import of msvcrt from an if <win32>
block, into a try/except block. So now the import is tried even on Linux
or MacOS.

https://docs.python.org/3/library/msvcrt.html is a Windows specific
builtin.
4 years ago
Alex Willmer 1e72ebaf8b tests: Update test coverage to Python 3.9
The Travis Linux distribution must be upgraded because the Trusty
(Ubuntu 14.04) image does not have Python 3.9. Xenial (Ubuntu 16.04) is
the earliest version that offers Python 3.9.

I have not chosen a later release, in order to aid restoration of Python
2.4 - 2.6 tests.
4 years ago
David Wilson fad6eb1b47 Bump copyright year and use generic author name (testing Travis) 4 years ago
Steven Robertson f489478127 code cleanup + adds 0.2.10 + 0.3.0 changelog 4 years ago
Steven Robertson 8d3da2dbd2
Merge branch 'master' into patch-1 4 years ago
Alex Willmer 79b4c0f815 tests: Fix AttributeError in callback plugins used by test suite
CALLBACK_VERSION et al are documented as required in
https://docs.ansible.com/ansible/2.10/dev_guide/developing_plugins.html#callback-plugins.
The need for document_fragment is noted in
cfa8075537/lib/ansible/plugins/callback/default.py (L28-L32)

Fixes #758

This addresses the following error, seen while running
`ansible_tests.py`.

```
TASK [Gathering Facts gather_timeout=10, gather_subset=['all']]
****************
task path:
/home/alex/src/mitogen/tests/ansible/regression/issue_109__target_has_old_ansible_installed.yml:4
[WARNING]: Failure using method (v2_runner_on_start) in callback plugin
(<ansible.plugins.callback.nice_stdout.CallbackModule object at
0x7f76b3dad090>): 'show_per_host_start'
Callback Exception:
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/executor/task_queue_manager.py",
line 372, in send_callback
    method(*new_args, **kwargs)
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/plugins/callback/default.py",
line 240, in v2_runner_on_start
    if self.get_option('show_per_host_start'):
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/plugins/callback/__init__.py",
line 91, in get_option
    return self._plugin_options[k]
Callback Exception:
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/executor/task_queue_manager.py",
line 372, in send_callback
    method(*new_args, **kwargs)
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/plugins/callback/default.py",
line 240, in v2_runner_on_start
    if self.get_option('show_per_host_start'):
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/plugins/callback/__init__.py",
line 91, in get_option
    return self._plugin_options[k]
[task 339882] 00:00:08.172036 D ansible_mitogen.affinity: CPU mask for
WorkerProcess: 0x000004
Callback Exception:
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/executor/task_queue_manager.py",
line 372, in send_callback
    method(*new_args, **kwargs)
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/plugins/callback/default.py",
line 240, in v2_runner_on_start
    if self.get_option('show_per_host_start'):
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/plugins/callback/__init__.py",
line 91, in get_option
    return self._plugin_options[k]
```
4 years ago
Alex Willmer b0ce29dcfd ssh: Match newer ssh host key prompt that accepts the fingerprint
This fixes an ERROR in test_accept_enforce_host_keys() while running the
test suite.

Fixes #756
4 years ago
Steven Robertson 7d4a57d16a
Merge branch 'master' into patch-1 4 years ago
Nicolas Mattia 4d48f140a8 Fix typo in Ansible documentation 4 years ago
Luiz Ribeiro 0e47280e43
Fix mitogen_ssh_keepalive_interval documentation 5 years ago
Scott Buchanan fa7c0191ee
correct latest ansible version supported 5 years ago
Steven Robertson 2d7e019dd2
Update docs/ansible_detailed.rst
Co-Authored-By: Jacob Floyd <cognifloyd@gmail.com>
5 years ago
Steven Robertson 1dddfadf32 removing note saying ansible 2.8 interpreter discovery doesn't work 5 years ago
Steven Robertson e8f3154cab Merge branch 'master' into complexAnsiblePythonInterpreterArg 5 years ago
David Wilson d6329f3446 Merge devel/290 @ 79b979ec8544ef5d8620c64068d4a42fabf50415 5 years ago
Steven Robertson bca9d9bf67 added info to ansible_detailed.rst explaining new ansible_python_interpreter functionality 5 years ago
David Wilson e3c514d906 issue #633: update Changelog. 5 years ago
David Wilson ed33236c71 docs: , -> : 5 years ago
David Wilson 9b79986859 docs: use new manpage alias in one more place 5 years ago
David Wilson 7a391022ef docs: update Changelog for 0.2.8. 5 years ago
David Wilson 4bd0a234ce docs: lots more changelog concision 5 years ago
David Wilson 8a11dea075 docs: changelog concision 5 years ago
David Wilson efe91092dc docs: more changelog tweaks 5 years ago
David Wilson 52c070efc0 docs: reorder chapters 5 years ago
David Wilson 8844d5f5cc docs: versionless <title> 5 years ago