Commit Graph

3476 Commits (123efa751074f5d7e922efcb73e7a0673cc92b09)
 

Author SHA1 Message Date
Alex Willmer 123efa7510 mitogen: Support Python 3.12
Most of the necessary changes were made in recent PEP 451 commits. This bumps
the CI jobs, and declares the support. Test dependendancies are bumped to
latest supportted/available versions.

refs #1033
8 months ago
Alex Willmer fe8a3a71fc ansible_mitogen: Remove use of distutils, which was removed in Python 3.12 8 months ago
Alex Willmer 92c00d913e tests: Skip "discovered python matches invoked" on macOS 11/Python 2.7/Vanilla 8 months ago
Alex Willmer 5ad3d14ceb mitogen: Support PEP 451 ModuleSpec API, required for Python 3.12
importlib.machinery.ModuleSpec and find_spec() were introduced in Python 3.4
under PEP 451. They replace the find_module() API of PEP 302, which was
deprecated from Python 3.4. They were removed in Python 3.12 along with the
imp module.

This change adds support for the PEP 451 APIs. Mitogen should no longer import
imp on Python versions that support ModuleSpec. Tests have been added to cover
the new APIs.

CI jobs have been added to cover Python 3.x on macOS.

Refs #1033
Co-authored-by: Witold Baryluk <witold.baryluk@gmail.com>
8 months ago
Alex Willmer 3a31a7d886 mitogen: Workaround CPython importlib PermissionError when cwd is unreadable
On macOS when using a become plugin as an unprivileged user, to another
unprivileged user it is likely that the current working directory can't be
read. In this case os.cwd() raises PermissionError.

On versions of Python currently in the wild (March 2024, CPython <= 3.13) if
any non-builtin or non-frozen module (e.g. zlib, base64) is imported then
`importlib._bootstrap_external.PathFinder._path_importer_cache()` attempts to
call os.cwd() without catching PermissionError.

The previous comment about needing an extra .encode() appears to be wrong,
atleast for Python 3.x >= 3.6.

Command size increased by 54 bytes, bootstrap by 804 bytes. Changed from
codecs module to binascii & zlib because they're extensions, and importing
them triggers fewer supporting imports (e.g. encodings module).

Before

```
✗ ./preamble_size.py
SSH command size: 705
Bootstrap (mitogen.core) size: 17078 (16.68KiB)

                              Original          Minimized           Compressed
mitogen.parent            97884 95.6KiB  50515 49.3KiB 51.6%  12727 12.4KiB
13.0%
mitogen.fork               8436  8.2KiB   4130  4.0KiB 49.0%   1648  1.6KiB
19.5%
mitogen.ssh               10892 10.6KiB   6952  6.8KiB 63.8%   2113  2.1KiB
19.4%
mitogen.sudo              12089 11.8KiB   5924  5.8KiB 49.0%   2249  2.2KiB
18.6%
mitogen.select            12325 12.0KiB   2929  2.9KiB 23.8%    964  0.9KiB
7.8%
mitogen.service           41699 40.7KiB  22477 22.0KiB 53.9%   5885  5.7KiB
14.1%
mitogen.fakessh           15577 15.2KiB   7989  7.8KiB 51.3%   2623  2.6KiB
16.8%
mitogen.master            51398 50.2KiB  25715 25.1KiB 50.0%   6886  6.7KiB
13.4%
```

After

```
✗ ./preamble_size.py
SSH command size: 759
Bootstrap (mitogen.core) size: 17882 (17.46KiB)

                              Original          Minimized           Compressed
mitogen.parent            98173 95.9KiB  50571 49.4KiB 51.5%  12747 12.4KiB
13.0%
mitogen.fork               8436  8.2KiB   4130  4.0KiB 49.0%   1648  1.6KiB
19.5%
mitogen.ssh               10892 10.6KiB   6952  6.8KiB 63.8%   2113  2.1KiB
19.4%
mitogen.sudo              12089 11.8KiB   5924  5.8KiB 49.0%   2249  2.2KiB
18.6%
mitogen.select            12325 12.0KiB   2929  2.9KiB 23.8%    964  0.9KiB
7.8%
mitogen.service           41699 40.7KiB  22477 22.0KiB 53.9%   5885  5.7KiB
14.1%
mitogen.fakessh           15577 15.2KiB   7989  7.8KiB 51.3%   2623  2.6KiB
16.8%
mitogen.master            56116 54.8KiB  29427 28.7KiB 52.4%   7627  7.4KiB
13.6%
```

Fixes #885
Refs https://github.com/python/cpython/issues/115911
8 months ago
Alex Willmer 1031551dd9 tests: Clarify transport config tests optimisation & correct value
The ini inventory parser doesn't support comments after a value, so the value
parsed was "python3000  # Not expected to exist".
9 months ago
Alex Willmer 2973d90670 tests: Enable su tests under vanilla Ansible >= 2.11
cwd_show was useful when debugging these tests, worth keeping around.
9 months ago
Alex Willmer e2f4d9275c tests: Fix ansible_python_interpreter & discovered_interpreter_python tests on macOS
Should account for fiddling in mitogen.parent.Connection._first_stage() and
symlinks. I won't be surprised if it breaks again soon and often.
9 months ago
Alex Willmer c2ad52e54e tests: Fix tests using get_url across Python versions
Using https:// requires certificate store management and additional parameter
passing that changed across Ansible and Python versions. Using http:// allows
the same tests to be used across wider spans of Python version on the
controller, and Python verison on the targets.

Python 3.12 on a target + get_uri needs Ansible >= 8 (ansible-core >= 2.15).
Python 3.12 removed deprecated httplib.HTTPSConnection() arguments.
https://github.com/ansible/ansible/pull/80751
9 months ago
Alex Willmer a6a5c5bb97 tests: Clarify status/purpose of Python 2.x era Ansible Module workaround 9 months ago
Alex Willmer 2839954559 tests: Account for /tmp symlink in virtualenv test on macOS 9 months ago
Alex Willmer adfd4e17f3 tests: Declare inventory file types to Visual Studio Code and Vim
Works with the VS Code modeline extension. Enables syntax highlighting.
9 months ago
Alex Willmer 591152bef0 tests: Avoid intermittant 2 hour timeout in new style Ansible module tests
This has been lurking for years, raising it's head at unpredictable times.
This change doesn't fix it, but it should make it a lot less mysterious.
9 months ago
Alex Willmer a6c89751f9 tests: Cleanup ansible-lint errors & warnings in user creation playbook
Task " Install slow profile for one account" removed because it duplicates
earlier work.
9 months ago
Alex Willmer 8b574f234d tests: Report Ansible controller parameters before image prep & user creation 9 months ago
Alex Willmer bde7f062b9 tests: Fix Ansible module shebangs
With https://github.com/ansible/ansible/pull/76677 Ansible
fixed shebang substitution for Ansible modules and tightened
up what shebang is allowed.

Changing these fixes the tests using them with vanilla Ansible.

https://docs.ansible.com/ansible/latest/dev_guide/testing/sanity/shebang.html
9 months ago
Alex Willmer 9a9dd66ba0 Ignore Ansible retry files 9 months ago
Alex Willmer fc3e788cb4 non functional: Add comments about imp module removal in Python 3.12 10 months ago
Alex Willmer f9a6748154 ci: Fix Python 2.7 builds on macOS 11
With current macOS 11 runner images (20231216.1) the `python` on `$PATH` is
Python 3.12 and setuptools isn't installed by default. E.g.

```
python -mtox -e "py27-mode_localhost-ansible4"
========================== Starting Command Output ===========================
/bin/bash --noprofile --norc
/Users/runner/work/_temp/93a29c4c-f606-45e4-8dbd-a4a5f51b8730.sh
GLOB sdist-make: /Users/runner/work/1/s/setup.py
ERROR: invocation failed (exit code 1), logfile:
/Users/runner/work/1/s/.tox/log/GLOB-0.log
================================== log start
===================================
Traceback (most recent call last):
  File "/Users/runner/work/1/s/setup.py", line 32, in <module>
    from setuptools import find_packages, setup
ModuleNotFoundError: No module named 'setuptools'
```

Installing setuptools under Python 3.12 chooses package versions incompatible
with Python 2.7. Additionally Mitogen isn't yet compatible with Python 3.12
(#1033), so tests that call a local context with `python` fail.
11 months ago
Alex Willmer b7188c1cad docs: Decouple website download version from package version
This prevents unreleased versions appearing on the website (e.g. 0.3.5.dev0),
but introduces the risk of forgetting to update the website after a release.
A better fix requires deeper design/workflow thought.

refs #1028
1 year ago
Alex Willmer e580258071 docs: Bypass networkgenomics.com/try/ -> PyPI redirect
refs #1028
1 year ago
Alex Willmer 798032b979
Merge pull request #1027 from moreati/pyver-token
ci: Authenticate UsePythonVersion requests to Github
1 year ago
Alex Willmer 3f105d5169 ci: Authenticate UsePythonVersion requests to Github
This should address the warning in Azure Pipelines

> You should provide GitHub token if you want to download a python release.
> Otherwise you may hit the GitHub anonymous download limit.

The token is provided from a secret variable in the pipeline.
1 year ago
Alex Willmer d839cbfaf2
Merge pull request #1026 from moreati/netlify
docs: Fix generating static website on Netlify
1 year ago
Alex Willmer 63457b4866 docs: Update external URLs (e.g. dw/mitogen -> mitogen-hq/mitogen)
Found with sphinx-build -b linkcheck. Not all flagged URLs have been changed,
e.g. Ansible plugins, deleted Github users.
1 year ago
Alex Willmer 6aa4fd3573 docs: Fix generation of static website
Bare minimum syntax errors and requirements constraints to work with Netlify
hosting.
1 year ago
Alex Willmer 85f9261c9a
Merge pull request #1019 from moreati/pr987
Add Python 3.11 support
1 year ago
Nerijus Baliūnas 4089e875a9 Add Python 3.11 support
Co-authored-by: Alex Willmer <alex@moreati.org.uk>
1 year ago
Alex Willmer 57b5be3589
Merge pull request #1016 from moreati/pr987
Prepare for Python 3.11
1 year ago
Alex Willmer 49c54386b3 tests: Only use subprocess32 package on Python 2.x
This is how the package documentation recommends and it's less likely to
interfere with new features in stdlib subprocess module.
1 year ago
Alex Willmer 98d110ed16 tests: Bump Django ModuleFinder test cases
Preperation for Python 3.11 support
1 year ago
Alex Willmer 6258365df6 tests: Handle square bracket IPv6 in `docker port` output
Fixes
```
======================================================================
ERROR: setUpClass (ssh_test.BannerTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vsts/work/1/s/tests/testlib.py", line 625, in setUpClass
    cls.dockerized_ssh = DockerizedSshDaemon(**daemon_args)
  File "/home/vsts/work/1/s/tests/testlib.py", line 553, in __init__
    self.start_container()
  File "/home/vsts/work/1/s/tests/testlib.py", line 533, in start_container
    self._get_container_port()
  File "/home/vsts/work/1/s/tests/testlib.py", line 510, in _get_container_port
    self.port = int(bport)
ValueError: invalid literal for int() with base 10: ':]:32770'
```
1 year ago
Alex Willmer 88f15a8169
Merge pull request #1006 from moreati/test-on-22.04
tests: Support Ubuntu 22.04 as test suite runner (controller)
1 year ago
Alex Willmer 270c3a25de tests: Support Ubuntu 22.04 as test suite runner (controller)
To do so the test suite allows a weak cryptographic alogorithm (SHA1) to be
used, principally for CentOS 6 targets. This can be removed if/when support
for older (legacy) targets is dropped.

Only the test suite enables this known weak alogorithm. Mitogen as-shipped
doesn't enable or disable algorithms.
1 year ago
Alex Willmer ec212a10d8
Merge pull request #1002 from moreati/prep-0.3.4
Prepare 0.3.4
1 year ago
Alex Willmer 455fd2bcdf Bump version 1 year ago
Alex Willmer f18f5165cd Prep for v0.3.4 1 year ago
Alex Willmer 5602445709
Merge pull request #1001 from moreati/ci-fixup
CI: Fix tests on Linux, Ansible tests targetting Debian 9 & 11
1 year ago
Alex Willmer 19b79f7ab5 CI: Fix tests on Linux, Ansible tests targetting Debian 9 & 11
Without Ubuntu 20.04 virtualenv package being installed pip was installing a
version of virtualenv that couldn't create the Tox environment for Python 2.7.

> Successfully installed distlib-0.3.6 filelock-3.12.2 platformdirs-3.8.0
> pluggy-1.2.0 py-1.11.0 tomli-2.0.1 tox-3.28.0 virtualenv-20.23.1
> Finishing: Install tooling
> ...
> py27-mode_mitogen-distro_centos6 create: /home/vsts/work/1/s/.tox/py27-
> mode_mitogen-distro_centos6
> ERROR: invocation failed (exit code 1), logfile: /home/vsts/work/1/s/.tox/
> py27-mode_mitogen-distro_centos6/log/py27-mode_mitogen-distro_centos6-0.log
> ================================== log start
> ===================================
> RuntimeError: failed to query /usr/bin/python2.7 with code 1 err:
> '  File "/home/vsts/.local/lib/python3.8/site-packages/virtualenv/discovery/
> py_info.py", line 24\n    return list(OrderedDict.fromkeys(["",
> *os.environ.get("PATHEXT", "").lower().split(os.pathsep)]))\n
> ^\nSyntaxError: invalid syntax\n'
1 year ago
Alex Willmer 330375be32
Merge pull request #980 from InsanePrawn/transport_lxc_fix
ansible_mitogen: correct typo in MitogenViaSpec.mitogen_lxc_path()
2 years ago
InsanePrawn 317a2abd57 ansible_mitogen: correct typo in MitogenViaSpec.mitogen_lxc_path()
self.host_vars -> self._host_vars
2 years ago
Alex Willmer d488b5ebb3
Merge pull request #965 from moreati/evict-simplejson
Remove vendored mitogen.compat.simplejson
2 years ago
Alex Willmer 8151577b75 CI: Limit to Tox < 4.0 to avoid plugin incompatibility
I'm abandoning tox-factor because having any [tox] requires = ... causes tox
3.x to create an isolated virtualenv for running tox itself. Since Tox 4.x was
released that virtualenv gets it, which is incompatible with the tox-factor
plugin.
e.g.
```
Traceback (most recent call last):
File
"/Users/runner/work/1/s/.tox/.tox/lib/python3.10/site-packages/tox_factor/compat.py",
line 2, in <module>
    from tox.config.parallel import ENV_VAR_KEY_PUBLIC as TOX_PARALLEL_ENV
ModuleNotFoundError: No module named 'tox.config.parallel'
```
2 years ago
Alex Willmer 21cb4a3472 CI: Remove faulthandler fallback requirement
faulthandler is a stdlib module in Python 3.3+. For a long time a PyPI package
of the same name was available for earlier Python releases. That package has
since been removed from PyPI, and the source respoitory archived. So we should
not rely on it.
fixes #983 refs #970
2 years ago
Alex Willmer 1871f2a9b1 Remove vendored mitogen.compat.simplejson
Python 2.6 added json to the stdlib. We no longer support Python <= 2.7 in
Mitogen 0.3.x, so this fallback is unneeded complexity. Fixes #659
2 years ago
Alex Willmer a47b9f3631
Merge pull request #969 from moreati/transport_config_python_undiscover
tests: Speed up transport config tests by avoiding interpreter discovery
2 years ago
Alex Willmer 392c41d160 ci: Ignore WALinuxAgent processes in Azure Piplines CI runner 2 years ago
Alex Willmer 0af2ce8c30 Remove ansible_mitogen Connection.close() workaround
Refs #925 #969

I'm not 100% confident that merely removing this is the full fix,
without substituting something else. I am sure keeping it would be
the greater of two evils. __del__() should be avoided on general
principal, and it's associated with multiple intermittant CI
failures, plus multiple user reported issues.
2 years ago
Alex Willmer 03acf40315 tests: Speed up transport config tests by avoiding interpreter discovery
Reduced execution time of tests/ansible/integration/transport_config/all.yml
from 11 minutes to 49 seconds.
2 years ago
Alex Willmer 572636a9d3
Merge pull request #964 from moreati/test-tweaks
Test tweaks
2 years ago