This reduces the number of jobs from 48 to 24. The Mitogen part of the test
suite has been parameterized on the Linux container targets to be run against.
Both the Ansible tests & Mitogen tests now use the same source of truth to
control which targets to use: environment variable MITOGEN_TEST_DISTRO_SPECS.
This replaces the two mutually exclusive env vars DISTRO and DISTROS. I've
also removed vestgial traces of an unused env var MITOGEN_TEST_DISTRO.
Parameterization adapted from
https://eli.thegreenplace.net/2014/04/02/dynamically-generating-python-test-cases
refs #1058, #1059
The password is provided as a variable because there is no corresponding
keyword. I get the impression that keywords are considered a legacy mechanism,
so most (new) options are only overridable by variables.
The port is proved as a variable for now, to test remote_name in isolation.
These targets are not used by any active tests, and the large numbers of hosts
multiply the size of the taskvars disctionary in memory to many (10s) MiB.
refs #1058
By switching to block style (`|`) with clip (no `-` or `+`) the failure
messages don't require quoting and gain a single trailing newline. This causes
Ansible to print them as block style, when using the yaml stdout callback
plugin. As a result the values have one less layer of quoting and quote
escaping, making them much easier to read.
This switches `ansible_mitogen.transport_config.PlayContextSpec.password()` to
Ansible's plugin option framework. As a result
- The relatively recent `ansible_ssh_password` variable is now respected.
- The SSH connection password can be templated and specified as a play
variable. Task variables will probably also work, but testing was blocked
by #1132.
There is a chance this change will cause a regression in another connection
plugin (e.g. mitogen_docker), but nothing turned up in the test suite.
I intend ot migrate other connection configuration to
`ansible_mitogen.transport_config.PlayContextSpec._connect_option()`, the next
candidate is the remote port.
fixes#1106
This replaces the use of `os.path.realpath()` which gave incorrect results on
macOS - depending on the exact Python build, Python version, macOS version,
installation method, and phase of the moon.
realpath information kept around to aid debugging.
CI containers lack the necessary `setfacl` command. This has not previously
been noticed because no vanilla Ansible jobs were being run on Linux, only on
macOS.
refs #1118
Preperation for migrating from Azure DevOps with Amazon Elastic Container
Registry (AWS ECR), to GitHub Actions with GitHub Container Registry (GHCR).
DebOps tests are not currently being run, the updates to .ci/debops*.py are
best effort only.
Python 3.x emits `ResourceWarning`s if certains resources aren't correctly
closed. Due to the way Mitogen has been terminating child processes this has
been occurring.
```
test_dev_tty_open_succeeds
(create_child_test.TtyCreateChildTest.test_dev_tty_open_succeeds) ...
/opt/hostedtoolcache/Python/3.12.5/x64/lib/python3.12/subprocess.py:1127:
ResourceWarning: subprocess 3313 is still running
_warn("subprocess %s is still running" % self.pid,
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok
```
During garbage collection subprocess.Popen() objects emit
ResourceWarning("subprocess 123 is still running")
if proc.returncode hasn't been set. Typically calling proc.wait() does so,
once the sub-process has exited. Calling os.waitpid(proc.pid, 0) also waits
for the sub-process to exit, but it doesn't update proc.returncode, so the
ResourceWarning is still emitted.
This change exposes `subprocess.Popen` methods on
`mitogen.parent.PopenProcess`, so that the returncode can be set.
See https://gist.github.com/moreati/b8d157ff82cb15234bece4033accc5e5
This
- Removes the indirection of calling ansible in a sub-shell
- Includes vanilla Ansible, which was previously skipped
- Tests whether ansible_ssh_pass overrides ansible_password, as it should
As a one off I've the new tests against vanilla Ansible 2.10 through Ansible
10, to confirm the baseline priorities have remained unchanged all releases
currently supported by Mitogen 0.3.x.
The bug was fixed in a previous commit by Jonathan Rosser. This adds testing.
The bug is only triggered when the copy module is used inside a `with_items:`
loop and the destination filename has an extension. A `loop:` loop is not
sufficient.
refs #1110
This is in anticipation of #1110, which only exhibits inside a with_items:
loop. For this refactor `loop:` is used, to confirm the refactored tests are
still correct. A subsequent commit will change them to with_items.
The content of the files and their SHA1 checksums are unchanged.
Relying on the virtualenv default or hardcoding "python" results in a Python
2.x virtualenv on some targets (e.g. debian10-test). This caused a failure
when testing with Ansible >= 10 (ansible-core >= 2.17), which have dropped
Python 2.x support.
refs #1074
By setting ansible_python_interpreter for these fictious hosts we avoid
Ansible trying and failing to connect to them in a attempt to populate
ansible_facts.discovered_interpreter_python. This speeds up these tests by
avoiding a timeout.
It is also a necessary pre-requisite for Ansible 10 (ansible-core 2.17). In
that release no hardcoded fallback is used, failure to determine a valid
Python interpreter is a fatal error.
refs #1074
Pip 72 was released yesterday (2024-07-28), dropping `setup.py test` support.
hdrhistogram 0.6.1 requires it to install.
For now constrain Pip to earlier releases, so our tests can be run.
refs #1090
macOS 11 is not longer an available runner on Azure Devops. The minimum is now
macOS 12. This runner does not have Python 2.7 installed, so running them
would require a custom install - which I'm declaring too much effort for too
little gain.
refs #1090
CentOS 8 has reached EOL. Packages are no longer mirrored or maintained. A
historic snapshot of the packages is kept on vault.centos.org.
refs #1088, #1090
Follwing fixes in Ansible 7-9 for CVE-2023-5764 cating `AnsibleUnsafeBytes` &
`AnsibleUnsafeText` to `bytes()` or `str()` requires special handling. The
handling is Ansible specific, so it shouldn't go in the mitogen package but
rather the ansible_mitogen package.
`ansible_mitogen.utils.unsafe.cast()` is most like `mitogen.utils.cast()`.
During development it began as `ansible_mitogen.utils.unsafe.unwrap_var()`,
closer to an inverse of `ansible.utils.unsafe_procy.wrap_var()`. Future
enhancements may move in this direction.
refs #977, refs #1046
See also
- https://github.com/advisories/GHSA-7j69-qfc3-2fq9
- https://github.com/ansible/ansible/pull/82293
- https://github.com/mitogen-hq/mitogen/wiki/AnsibleUnsafe-notes
If casting a string fails then raise a TypeError. This is potentially an API
breaking change; chosen as the lesser evil vs. allowing silent errors.
`cast()` relies on `bytes(obj)` & `str(obj)` returning the respective
supertype. That's no longer the case for `AnsibleUnsafeBytes` &
`AnsibleUnsafeText`; since fixes/mitigations for CVE-2023-5764.
fixes#1046, refs #977
See also
- https://github.com/advisories/GHSA-7j69-qfc3-2fq9
- https://github.com/ansible/ansible/pull/82293
Most of the necessary changes were made in recent PEP 451 commits. This bumps
the CI jobs, and declares the support. Test dependendancies are bumped to
latest supportted/available versions.
refs #1033
importlib.machinery.ModuleSpec and find_spec() were introduced in Python 3.4
under PEP 451. They replace the find_module() API of PEP 302, which was
deprecated from Python 3.4. They were removed in Python 3.12 along with the
imp module.
This change adds support for the PEP 451 APIs. Mitogen should no longer import
imp on Python versions that support ModuleSpec. Tests have been added to cover
the new APIs.
CI jobs have been added to cover Python 3.x on macOS.
Refs #1033
Co-authored-by: Witold Baryluk <witold.baryluk@gmail.com>
Using https:// requires certificate store management and additional parameter
passing that changed across Ansible and Python versions. Using http:// allows
the same tests to be used across wider spans of Python version on the
controller, and Python verison on the targets.
Python 3.12 on a target + get_uri needs Ansible >= 8 (ansible-core >= 2.15).
Python 3.12 removed deprecated httplib.HTTPSConnection() arguments.
https://github.com/ansible/ansible/pull/80751
Fixes
```
======================================================================
ERROR: setUpClass (ssh_test.BannerTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/vsts/work/1/s/tests/testlib.py", line 625, in setUpClass
cls.dockerized_ssh = DockerizedSshDaemon(**daemon_args)
File "/home/vsts/work/1/s/tests/testlib.py", line 553, in __init__
self.start_container()
File "/home/vsts/work/1/s/tests/testlib.py", line 533, in start_container
self._get_container_port()
File "/home/vsts/work/1/s/tests/testlib.py", line 510, in _get_container_port
self.port = int(bport)
ValueError: invalid literal for int() with base 10: ':]:32770'
```
To do so the test suite allows a weak cryptographic alogorithm (SHA1) to be
used, principally for CentOS 6 targets. This can be removed if/when support
for older (legacy) targets is dropped.
Only the test suite enables this known weak alogorithm. Mitogen as-shipped
doesn't enable or disable algorithms.
faulthandler is a stdlib module in Python 3.3+. For a long time a PyPI package
of the same name was available for earlier Python releases. That package has
since been removed from PyPI, and the source respoitory archived. So we should
not rely on it.
fixes#983 refs #970
Until Ansible 2.9 it looks like ansible_become_password had higher priority.
From Ansible 2.10 ansible_become_pass has higher priority [1]. Mitogen was not
respecting this.
I may need to rework this further, instatiating the become plugin may have
slowed down execution.
[1] Based on testing with
```
[ubuntus]
become-pass-pass ansible_become_pass=1234
become-pass-password ansible_become_password=1234
become-pass-both ansible_become_password=wrong ansible_become_pass=1234
[ubuntus:vars]
ansible_host=ubuntu2004.local
ansible_user=ubuntu
```
```
- hosts: ubuntus
gather_facts: false
become: true
tasks:
- ping:
```
This reapplies an earlier change, when this plugin was first introduced to
Mitogen. The plugin was updated to fix
[DEPRECATION WARNING]: The '_remote_checksum()' method is deprecated.
I've elected to short-circuit the if statemtn logic, rather than
deleting/unindenting, to make the code delta much smaller. This should make it
easier to maintain/update.
Fixes#915
Attempt to fix
Collecting hdrhistogram==0.6.1 (from -r tests/ansible/requirements.txt
(line 2))
Downloading
2422ad2ad90047d3b9f73061e8696f/hdrhistogram-0.6.1.tar.gz
(61kB)
Complete output from command python setup.py egg_info:
Couldn't find index page for 'pbr' (maybe misspelled?)
sources=['src/python-codec.c'])]
File "/usr/lib/python3.5/distutils/core.py", line 108, in setup
_setup_distribution = dist = klass(attrs)
File "/tmp/venv/lib/python3.5/site-packages/setuptools/dist.py",
line 315, in __init__
self.fetch_build_eggs(attrs['setup_requires'])
File "/tmp/venv/lib/python3.5/site-packages/setuptools/dist.py",
line 361, in fetch_build_eggs
replace_conflicting=True,
File
"/tmp/venv/lib/python3.5/site-packages/pkg_resources/__init__.py", line
850, in resolve
dist = best[req.key] = env.best_match(req, ws, installer)
File
"/tmp/venv/lib/python3.5/site-packages/pkg_resources/__init__.py", line
1122, in best_match
return self.obtain(req, installer)
File
"/tmp/venv/lib/python3.5/site-packages/pkg_resources/__init__.py", line
1134, in obtain
return installer(requirement)
File "/tmp/venv/lib/python3.5/site-packages/setuptools/dist.py",
line 429, in fetch_build_egg
return cmd.easy_install(req)
File
"/tmp/venv/lib/python3.5/site-packages/setuptools/command/easy_install.py",
line 659, in easy_install
raise DistutilsError(msg)
distutils.errors.DistutilsError: Could not find suitable
distribution for Requirement.parse('pbr')
From cpuset(7):
> For masks longer than one word, a comma separator is used between
> words. Words are displayed in big-endian order, which has the most
> significant bit first. The hex digits within a word are also in
> big-endian order.
A few changes are bundled in this
- Ansible 2.10.x and Mitogen 0.3.x are used to build nearly all images
(Ansile 2.3.x is retained for CentOS 5, because it uses Python 2.4).
- Tox is used to install/run Ansible, replacing build_docker_images.py
- A static inventory, identifying containers by name rather than ID.
- debian-test image is renamed to debian9-test
- debian9-test image is now based on debian:9
- centos6-test image is now based on moreati/centos6-vault
following the same scheme as centos5-test.
- Images are now uploaded to Amazon Elastic Container Registry (ECR).
See #809.
- Debian recommended packages aren't installed (e.g. build-essential)
- Python 2.x and Python 3.x are installed wherever available.
- Python Virtualenv is installed wherever available.
The Travis Linux distribution must be upgraded because the Trusty
(Ubuntu 14.04) image does not have Python 3.9. Xenial (Ubuntu 16.04) is
the earliest version that offers Python 3.9.
I have not chosen a later release, in order to aid restoration of Python
2.4 - 2.6 tests.
The ansible_mitogen test suite takes over an hour when Ansible is not
accelerated by Mitogen. This change aims to reduce that by skipping
tests with a large number of iterations when the linear strategy is
chosen.
The tagged tests are intended to uncover Mitogen resource leaks. Since
Mitogen is not invoked when strategy=linear, the slight reduction in
test coverage is an acceptable trade off.
Alex Willmer