This reduces the number of jobs from 48 to 24. The Mitogen part of the test
suite has been parameterized on the Linux container targets to be run against.
Both the Ansible tests & Mitogen tests now use the same source of truth to
control which targets to use: environment variable MITOGEN_TEST_DISTRO_SPECS.
This replaces the two mutually exclusive env vars DISTRO and DISTROS. I've
also removed vestgial traces of an unused env var MITOGEN_TEST_DISTRO.
Parameterization adapted from
https://eli.thegreenplace.net/2014/04/02/dynamically-generating-python-test-cases
refs #1058, #1059
Preperation for migrating from Azure DevOps with Amazon Elastic Container
Registry (AWS ECR), to GitHub Actions with GitHub Container Registry (GHCR).
DebOps tests are not currently being run, the updates to .ci/debops*.py are
best effort only.
To avoid rate limiting errors, CI (currently Azure Devops) logs into the
container registry (currently AWS ECR). Outside CI this is unnnecessary and
makes it harder to run the tests, because very few people have access to a
suitable AWS secret token.
Following this change `aws ecr-public get-login-password` will only be run if
the environment variable $TF_BUILD==True. This is set by Azure Pipelines
jobs. If the CI platform is changed then another indicator should be used.
https://adamj.eu/tech/2020/03/09/detect-if-your-tests-are-running-on-ci/
macOS 11 is not longer an available runner on Azure Devops. The minimum is now
macOS 12. This runner does not have Python 2.7 installed, so running them
would require a custom install - which I'm declaring too much effort for too
little gain.
refs #1090
Most of the necessary changes were made in recent PEP 451 commits. This bumps
the CI jobs, and declares the support. Test dependendancies are bumped to
latest supportted/available versions.
refs #1033
importlib.machinery.ModuleSpec and find_spec() were introduced in Python 3.4
under PEP 451. They replace the find_module() API of PEP 302, which was
deprecated from Python 3.4. They were removed in Python 3.12 along with the
imp module.
This change adds support for the PEP 451 APIs. Mitogen should no longer import
imp on Python versions that support ModuleSpec. Tests have been added to cover
the new APIs.
CI jobs have been added to cover Python 3.x on macOS.
Refs #1033
Co-authored-by: Witold Baryluk <witold.baryluk@gmail.com>
With current macOS 11 runner images (20231216.1) the `python` on `$PATH` is
Python 3.12 and setuptools isn't installed by default. E.g.
```
python -mtox -e "py27-mode_localhost-ansible4"
========================== Starting Command Output ===========================
/bin/bash --noprofile --norc
/Users/runner/work/_temp/93a29c4c-f606-45e4-8dbd-a4a5f51b8730.sh
GLOB sdist-make: /Users/runner/work/1/s/setup.py
ERROR: invocation failed (exit code 1), logfile:
/Users/runner/work/1/s/.tox/log/GLOB-0.log
================================== log start
===================================
Traceback (most recent call last):
File "/Users/runner/work/1/s/setup.py", line 32, in <module>
from setuptools import find_packages, setup
ModuleNotFoundError: No module named 'setuptools'
```
Installing setuptools under Python 3.12 chooses package versions incompatible
with Python 2.7. Additionally Mitogen isn't yet compatible with Python 3.12
(#1033), so tests that call a local context with `python` fail.
This should address the warning in Azure Pipelines
> You should provide GitHub token if you want to download a python release.
> Otherwise you may hit the GitHub anonymous download limit.
The token is provided from a secret variable in the pipeline.
I'm abandoning tox-factor because having any [tox] requires = ... causes tox
3.x to create an isolated virtualenv for running tox itself. Since Tox 4.x was
released that virtualenv gets it, which is incompatible with the tox-factor
plugin.
e.g.
```
Traceback (most recent call last):
File
"/Users/runner/work/1/s/.tox/.tox/lib/python3.10/site-packages/tox_factor/compat.py",
line 2, in <module>
from tox.config.parallel import ENV_VAR_KEY_PUBLIC as TOX_PARALLEL_ENV
ModuleNotFoundError: No module named 'tox.config.parallel'
```
Should result in
- better isolation of faults
- easier retries of failed jobs
- quicker test completion, to to more efficient queueing.
This does not expand test coverage, merely rearranges the tests already run.
Also
- Simplifies adding support for additional Ansible versions
- Unifies Python package versioning in CI and local test environments
- Matches Python versions tested, with those declared in setup.py
- Expands targets covered by automated Ansible tests to
- centos6, centos8
- debian9, debian11
- ubuntu1604, ubuntu2004
This reverse shell was historically used to debug CI jobs interactively.
It is not used anymore, and may be causing jobs to hang, then timeout.
There is no reason to keep it, and removing it simplifies CI jobs.
Additionally it has been reported as flagged by security scanners, so
removing it makes Mitogen easier to package/adopt.
fixes#847
Alex Willmer