Commit Graph

200 Commits (d839cbfaf26201f6afc0582d2a4b84a6723c46a7)

Author SHA1 Message Date
Nerijus Baliūnas 4089e875a9 Add Python 3.11 support
Co-authored-by: Alex Willmer <alex@moreati.org.uk>
1 year ago
Alex Willmer 49c54386b3 tests: Only use subprocess32 package on Python 2.x
This is how the package documentation recommends and it's less likely to
interfere with new features in stdlib subprocess module.
1 year ago
Alex Willmer 19b79f7ab5 CI: Fix tests on Linux, Ansible tests targetting Debian 9 & 11
Without Ubuntu 20.04 virtualenv package being installed pip was installing a
version of virtualenv that couldn't create the Tox environment for Python 2.7.

> Successfully installed distlib-0.3.6 filelock-3.12.2 platformdirs-3.8.0
> pluggy-1.2.0 py-1.11.0 tomli-2.0.1 tox-3.28.0 virtualenv-20.23.1
> Finishing: Install tooling
> ...
> py27-mode_mitogen-distro_centos6 create: /home/vsts/work/1/s/.tox/py27-
> mode_mitogen-distro_centos6
> ERROR: invocation failed (exit code 1), logfile: /home/vsts/work/1/s/.tox/
> py27-mode_mitogen-distro_centos6/log/py27-mode_mitogen-distro_centos6-0.log
> ================================== log start
> ===================================
> RuntimeError: failed to query /usr/bin/python2.7 with code 1 err:
> '  File "/home/vsts/.local/lib/python3.8/site-packages/virtualenv/discovery/
> py_info.py", line 24\n    return list(OrderedDict.fromkeys(["",
> *os.environ.get("PATHEXT", "").lower().split(os.pathsep)]))\n
> ^\nSyntaxError: invalid syntax\n'
1 year ago
Alex Willmer 8151577b75 CI: Limit to Tox < 4.0 to avoid plugin incompatibility
I'm abandoning tox-factor because having any [tox] requires = ... causes tox
3.x to create an isolated virtualenv for running tox itself. Since Tox 4.x was
released that virtualenv gets it, which is incompatible with the tox-factor
plugin.
e.g.
```
Traceback (most recent call last):
File
"/Users/runner/work/1/s/.tox/.tox/lib/python3.10/site-packages/tox_factor/compat.py",
line 2, in <module>
    from tox.config.parallel import ENV_VAR_KEY_PUBLIC as TOX_PARALLEL_ENV
ModuleNotFoundError: No module named 'tox.config.parallel'
```
2 years ago
Alex Willmer 392c41d160 ci: Ignore WALinuxAgent processes in Azure Piplines CI runner 2 years ago
Alex Willmer edd2868ef6 tests: Don't rely on facts when setting become
They won't be available if the play is first, and hence no facts have been
gather in previous play(s), e.g. due to --start-at-task
2 years ago
Alex Willmer 39dfd2dfe8 ci: Upgrade VM Images to macOS 11 and Ubuntu 20.04 2 years ago
Alex Willmer e8ad12e881 Ansible 6 support
fixes #929
2 years ago
Alex Willmer 195b400087 ci: Drop Ansible 3 tests to free up CI capacity 2 years ago
Alex Willmer 63e39c1ac5 ci: Remove traces of Ansible < 2.10 (not supported in 0.3.x) 2 years ago
Alex Willmer 64819ecb5f tests: Regression test for #776 (package/yum/dnf module called twice) 2 years ago
Alex Willmer c32577295a tests: Check and/or suppress stderr of subprocesses, reduce shell=True uses 2 years ago
Alex Willmer 18c89de5a9 Remove unused module imports 3 years ago
Alex Willmer 0417d4d73a Replace os.system() with subprocess.check_call()
Non-zero return codes should raise an exception, not pass silently.
3 years ago
Alex Willmer 2a95d039ab Python 3.10 support 3 years ago
Alex Willmer c9318a26f6 tests: Suppress pip version warnings 3 years ago
Alex Willmer 491458673b tests: Manage ANSIBLE_STRATEGY with Tox 3 years ago
Alex Willmer d424895344 ci: Drop Mac1014 jobs
The VM Image has been retired by Azure Pipelines.
3 years ago
Alex Willmer 77c306199c ci: Explicitly test on macOS-11 VM Image
As of writing this macOS-latest isn't quite yet upgraded, but warnings about
it are spewing out.
3 years ago
Klaus Zerwes b0610cb63c issue #874 ansible 2.12 requires min python 3.8 on the controller 3 years ago
Klaus Zerwes 40fc1f8a25 issue #874 try to implement the testst for ansible 2.12 3 years ago
Alex Willmer cadd393cd7 ci: Expand macOS coverage to 10.14, latest release on Azure Pipelines
10.14 will be retired on Azure mid December 2021. I want to get so test runs
while we can.
3 years ago
Alex Willmer 7bdec56a70 ci: One Tox environment per Azure job
Should result in
- better isolation of faults
- easier retries of failed jobs
- quicker test completion, to to more efficient queueing.

This does not expand test coverage, merely rearranges the tests already run.
3 years ago
Alex Willmer c61c063b4f Support for Ansible 3 & 4
fixes #834

Co-authored-by: Claude Becker (@upekkha)
Co-authored-by: Dolph Mathews (@dolph)
3 years ago
Alex Willmer c5c65ab0d2 ci: Switch Continuous Integration to Tox
Also
 - Simplifies adding support for additional Ansible versions
 - Unifies Python package versioning in CI and local test environments
 - Matches Python versions tested, with those declared in setup.py
 - Expands targets covered by automated Ansible tests to
    - centos6, centos8
    - debian9, debian11
    - ubuntu1604, ubuntu2004
3 years ago
Alex Willmer e76eefb8be Fix miscellaneous spelling/formatting 3 years ago
Alex Willmer 39d02e9d59 ci: Don't install pycparser and idna seperately
AFAICT no longer necessary. These install fine as part of requirements.txt.
3 years ago
Alex Willmer 3ba1625a99 ci: Reduce verbosity of Ansible integration tests 3 years ago
Alex Willmer 9e9161cbb9 CI: Clamp ansible-base to < 2.10.14
refs #861
3 years ago
Alex Willmer a990eb3d77 CI: Remove obsolete reverse shell historically used to debug CI
This reverse shell was historically used to debug CI jobs interactively.
It is not used anymore, and may be causing jobs to hang, then timeout.
There is no reason to keep it, and removing it simplifies CI jobs.

Additionally it has been reported as flagged by security scanners, so
removing it makes Mitogen easier to package/adopt.

fixes #847
3 years ago
Alex Willmer 042e46682e Abandon Travis CI We have no credits, and its a PITA to maintain two CI platforms. The tests running on Travis provide very little extra coverage vs Azure Pipelines alone 3 years ago
Alex Willmer 0db7467e4a tests: Authenticate to AWS Elastic Container Registry for `docker pull`
This should increase the rate limit from 1/s to 10/s. AWS
credentials are obtained from environment variables in Azure
pipelines, or the default AWS CLI profile.

https://docs.aws.amazon.com/AmazonECR/latest/public/public-service-quotas.html
https://docs.aws.amazon.com/AmazonECR/latest/public/public-registries.html#public-registry-auth
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html
3 years ago
Alex Willmer 3d350643a9 tests: Tighten requirements.txt, due to new package releases
Attempt to fix

Collecting hdrhistogram==0.6.1 (from -r tests/ansible/requirements.txt
(line 2))
Downloading
2422ad2ad90047d3b9f73061e8696f/hdrhistogram-0.6.1.tar.gz
(61kB)
    Complete output from command python setup.py egg_info:
    Couldn't find index page for 'pbr' (maybe misspelled?)
        sources=['src/python-codec.c'])]
      File "/usr/lib/python3.5/distutils/core.py", line 108, in setup
        _setup_distribution = dist = klass(attrs)
File "/tmp/venv/lib/python3.5/site-packages/setuptools/dist.py",
line 315, in __init__
        self.fetch_build_eggs(attrs['setup_requires'])
File "/tmp/venv/lib/python3.5/site-packages/setuptools/dist.py",
line 361, in fetch_build_eggs
        replace_conflicting=True,
File
"/tmp/venv/lib/python3.5/site-packages/pkg_resources/__init__.py", line
850, in resolve
        dist = best[req.key] = env.best_match(req, ws, installer)
File
"/tmp/venv/lib/python3.5/site-packages/pkg_resources/__init__.py", line
1122, in best_match
        return self.obtain(req, installer)
File
"/tmp/venv/lib/python3.5/site-packages/pkg_resources/__init__.py", line
1134, in obtain
        return installer(requirement)
File "/tmp/venv/lib/python3.5/site-packages/setuptools/dist.py",
line 429, in fetch_build_egg
        return cmd.easy_install(req)
File
"/tmp/venv/lib/python3.5/site-packages/setuptools/command/easy_install.py",
line 659, in easy_install
        raise DistutilsError(msg)
distutils.errors.DistutilsError: Could not find suitable
distribution for Requirement.parse('pbr')
3 years ago
Alex Willmer a8e8cf91cb tests: Rebuild Docker containers
A few changes are bundled in this
 - Ansible 2.10.x and Mitogen 0.3.x are used to build nearly all images
   (Ansile 2.3.x is retained for CentOS 5, because it uses Python 2.4).
 - Tox is used to install/run Ansible, replacing build_docker_images.py
 - A static inventory, identifying containers by name rather than ID.
 - debian-test image is renamed to debian9-test
 - debian9-test image is now based on debian:9
 - centos6-test image is now based on moreati/centos6-vault
   following the same scheme as centos5-test.
 - Images are now uploaded to Amazon Elastic Container Registry (ECR).
   See #809.
 - Debian recommended packages aren't installed (e.g. build-essential)
 - Python 2.x and Python 3.x are installed wherever available.
 - Python Virtualenv is installed wherever available.
4 years ago
Alex Willmer e67e4b83b4 tests: Fix throttling of Docker pulls
fixes #809
4 years ago
Alex Willmer b71fbe528c
Merge branch 'master' into distro-vs-distros 4 years ago
Alex Willmer 1e72ebaf8b tests: Update test coverage to Python 3.9
The Travis Linux distribution must be upgraded because the Trusty
(Ubuntu 14.04) image does not have Python 3.9. Xenial (Ubuntu 16.04) is
the earliest version that offers Python 3.9.

I have not chosen a later release, in order to aid restoration of Python
2.4 - 2.6 tests.
4 years ago
Alex Willmer 7cc9145360
Merge branch 'master' into tests-resource_intensive-tag 4 years ago
Alex Willmer f5b3fab04c
Merge branch 'master' into docstrings 4 years ago
Alex Willmer 1a0951c664
Merge branch 'master' into tests-resource_intensive-tag 4 years ago
Alex Willmer 9111778823
Merge branch 'master' into docker-diet 4 years ago
Alex Willmer 25446cd698 tests: Throttle docker pulls for Ansible tests
I hope this will prevent AWS Elastic Container Registry returning 429:
Too Many Requests error, due to the burst of activity from multiple VMs,
when a CI build begins.
4 years ago
Alex Willmer 99235071a3 tests: docstrings for ci_lib 4 years ago
Alex Willmer 1c7e107809 tests: Use AWS Elastic Container registry for test images
This replaces use of Docker Hub, which now rate limits API calls and
causing CI builds to fail.

refs #791
4 years ago
Alex Willmer adbed36dd9 tests: Avoid double docker pull on Azure Pipes 4 years ago
Alex Willmer bb271d8d00 tests: Tag resource intensive tasks, to skip when strategy=linear
The ansible_mitogen test suite takes over an hour when Ansible is not
accelerated by Mitogen. This change aims to reduce that by skipping
tests with a large number of iterations when the linear strategy is
chosen.

The tagged tests are intended to uncover Mitogen resource leaks. Since
Mitogen is not invoked when strategy=linear, the slight reduction in
test coverage is an acceptable trade off.
4 years ago
Alex Willmer a64e966ab9 tests: Add unofficial Tox environments for Ansible tests
These are not part of the official testing regime (tests run for pull
requests). I find them convenient for local development.

Limitations
- Python 2.7+ only. No Python 2.4, 2.5, or 2.6.
- Requires Pythons pre-installed (e.g. DeadSnakes, pyenv)
- No coverage of alternate controller OS (e.g. MacOS)

The environments tested by default are

py27-mode_ansible-ansible2.10
py36-mode_ansible-ansible2.10
py39-mode_ansible-ansible2.10
py27-mode_mitogen
py36-mode_mitogen
py39-mode_mitogen
py27-mode_mitogen-distro_centos7
py36-mode_mitogen-distro_centos7
py39-mode_mitogen-distro_centos7
4 years ago
Alex Willmer 9dce5fe056 tests: Correct DISTRO vs DISTROS usage in CI configurations
The environment variable DISTRO is used to set the target Docker image
used when running the Mitogen unit tests. DISTROS is used for the
Ansible integration tests. VER sets the version of Ansible that is
installed on the controller.

When MODE=Mitogen
- there is no need to set VER, because nothing that installs Ansible
- it does not make sense to set DISTROS.
4 years ago
Steven Robertson 91f55a60bc vanilla ansible is now running but is really slow; bump timeout 4 years ago
Steven Robertson 90f40ada6c try vanilla ansible 2.10 on Mac 4 years ago