To avoid rate limiting errors, CI (currently Azure Devops) logs into the
container registry (currently AWS ECR). Outside CI this is unnnecessary and
makes it harder to run the tests, because very few people have access to a
suitable AWS secret token.
Following this change `aws ecr-public get-login-password` will only be run if
the environment variable $TF_BUILD==True. This is set by Azure Pipelines
jobs. If the CI platform is changed then another indicator should be used.
https://adamj.eu/tech/2020/03/09/detect-if-your-tests-are-running-on-ci/
macOS 11 is not longer an available runner on Azure Devops. The minimum is now
macOS 12. This runner does not have Python 2.7 installed, so running them
would require a custom install - which I'm declaring too much effort for too
little gain.
refs #1090
Most of the necessary changes were made in recent PEP 451 commits. This bumps
the CI jobs, and declares the support. Test dependendancies are bumped to
latest supportted/available versions.
refs #1033
importlib.machinery.ModuleSpec and find_spec() were introduced in Python 3.4
under PEP 451. They replace the find_module() API of PEP 302, which was
deprecated from Python 3.4. They were removed in Python 3.12 along with the
imp module.
This change adds support for the PEP 451 APIs. Mitogen should no longer import
imp on Python versions that support ModuleSpec. Tests have been added to cover
the new APIs.
CI jobs have been added to cover Python 3.x on macOS.
Refs #1033
Co-authored-by: Witold Baryluk <witold.baryluk@gmail.com>
With current macOS 11 runner images (20231216.1) the `python` on `$PATH` is
Python 3.12 and setuptools isn't installed by default. E.g.
```
python -mtox -e "py27-mode_localhost-ansible4"
========================== Starting Command Output ===========================
/bin/bash --noprofile --norc
/Users/runner/work/_temp/93a29c4c-f606-45e4-8dbd-a4a5f51b8730.sh
GLOB sdist-make: /Users/runner/work/1/s/setup.py
ERROR: invocation failed (exit code 1), logfile:
/Users/runner/work/1/s/.tox/log/GLOB-0.log
================================== log start
===================================
Traceback (most recent call last):
File "/Users/runner/work/1/s/setup.py", line 32, in <module>
from setuptools import find_packages, setup
ModuleNotFoundError: No module named 'setuptools'
```
Installing setuptools under Python 3.12 chooses package versions incompatible
with Python 2.7. Additionally Mitogen isn't yet compatible with Python 3.12
(#1033), so tests that call a local context with `python` fail.
This should address the warning in Azure Pipelines
> You should provide GitHub token if you want to download a python release.
> Otherwise you may hit the GitHub anonymous download limit.
The token is provided from a secret variable in the pipeline.
I'm abandoning tox-factor because having any [tox] requires = ... causes tox
3.x to create an isolated virtualenv for running tox itself. Since Tox 4.x was
released that virtualenv gets it, which is incompatible with the tox-factor
plugin.
e.g.
```
Traceback (most recent call last):
File
"/Users/runner/work/1/s/.tox/.tox/lib/python3.10/site-packages/tox_factor/compat.py",
line 2, in <module>
from tox.config.parallel import ENV_VAR_KEY_PUBLIC as TOX_PARALLEL_ENV
ModuleNotFoundError: No module named 'tox.config.parallel'
```
Should result in
- better isolation of faults
- easier retries of failed jobs
- quicker test completion, to to more efficient queueing.
This does not expand test coverage, merely rearranges the tests already run.
Also
- Simplifies adding support for additional Ansible versions
- Unifies Python package versioning in CI and local test environments
- Matches Python versions tested, with those declared in setup.py
- Expands targets covered by automated Ansible tests to
- centos6, centos8
- debian9, debian11
- ubuntu1604, ubuntu2004
This reverse shell was historically used to debug CI jobs interactively.
It is not used anymore, and may be causing jobs to hang, then timeout.
There is no reason to keep it, and removing it simplifies CI jobs.
Additionally it has been reported as flagged by security scanners, so
removing it makes Mitogen easier to package/adopt.
fixes#847
Attempt to fix
Collecting hdrhistogram==0.6.1 (from -r tests/ansible/requirements.txt
(line 2))
Downloading
2422ad2ad90047d3b9f73061e8696f/hdrhistogram-0.6.1.tar.gz
(61kB)
Complete output from command python setup.py egg_info:
Couldn't find index page for 'pbr' (maybe misspelled?)
sources=['src/python-codec.c'])]
File "/usr/lib/python3.5/distutils/core.py", line 108, in setup
_setup_distribution = dist = klass(attrs)
File "/tmp/venv/lib/python3.5/site-packages/setuptools/dist.py",
line 315, in __init__
self.fetch_build_eggs(attrs['setup_requires'])
File "/tmp/venv/lib/python3.5/site-packages/setuptools/dist.py",
line 361, in fetch_build_eggs
replace_conflicting=True,
File
"/tmp/venv/lib/python3.5/site-packages/pkg_resources/__init__.py", line
850, in resolve
dist = best[req.key] = env.best_match(req, ws, installer)
File
"/tmp/venv/lib/python3.5/site-packages/pkg_resources/__init__.py", line
1122, in best_match
return self.obtain(req, installer)
File
"/tmp/venv/lib/python3.5/site-packages/pkg_resources/__init__.py", line
1134, in obtain
return installer(requirement)
File "/tmp/venv/lib/python3.5/site-packages/setuptools/dist.py",
line 429, in fetch_build_egg
return cmd.easy_install(req)
File
"/tmp/venv/lib/python3.5/site-packages/setuptools/command/easy_install.py",
line 659, in easy_install
raise DistutilsError(msg)
distutils.errors.DistutilsError: Could not find suitable
distribution for Requirement.parse('pbr')
A few changes are bundled in this
- Ansible 2.10.x and Mitogen 0.3.x are used to build nearly all images
(Ansile 2.3.x is retained for CentOS 5, because it uses Python 2.4).
- Tox is used to install/run Ansible, replacing build_docker_images.py
- A static inventory, identifying containers by name rather than ID.
- debian-test image is renamed to debian9-test
- debian9-test image is now based on debian:9
- centos6-test image is now based on moreati/centos6-vault
following the same scheme as centos5-test.
- Images are now uploaded to Amazon Elastic Container Registry (ECR).
See #809.
- Debian recommended packages aren't installed (e.g. build-essential)
- Python 2.x and Python 3.x are installed wherever available.
- Python Virtualenv is installed wherever available.
The Travis Linux distribution must be upgraded because the Trusty
(Ubuntu 14.04) image does not have Python 3.9. Xenial (Ubuntu 16.04) is
the earliest version that offers Python 3.9.
I have not chosen a later release, in order to aid restoration of Python
2.4 - 2.6 tests.
Alex Willmer