The wrong base was used when calculating the mode. So the file became world
readable and writable on a CI runner, until
ansible/integration/ssh/variables.yml happened to correct it near the end of
the integration tests.
I believe this was the only instance.
```console
mitogen git:(issue1182) ✗ ag --python 'int\(.+7\)' . .ci | wc -l
0
```
fixes#1182
Adding a the tt-ssh-executable test target uncovered an Ansible bug during
`meta: reset_connection` tasks. So this commit includes a workaround for
affected versions of Ansible.
Uses the same fallback for (mitogen_sudo et al) as become_exe (see #1173).
The new `Spec.become_flags()` is not yet explicitly tested. Note that it
returns a string (matching the Ansible option of the same name), whereas
`Spec.sudo_args()` returns a list.
refs #1083
Some ansible_mitogen connection plugins look more like become plugins (e.g.
mitogen_sudo) & use become plugin options. For now there's special handling in
PlayContextSpec._become_option(). Further design/discussion can go in #1173.
Refs #1087.
The tasks in tests/imageprep/_user_accounts.yml that create users did not
specify a primary group for those users - this left the decision to Ansible's
user module, and/or the underlying OS. In Ansible 9+ (ansible-core 2.16+ the
user module defaults to primary group "staff." Earlier don't supply a default,
which releases probably results in a primary group nameed "None" (due to
stringifying the Python singleton of the same name), or whatever the macOS
Directory Services has for no data/NULL.
The invalid GID 4294967295 (MAX_UINT32 == 2**32-1) in the sudo error probably
enters the mix via something similar to sudo CVE-2019-14287.
Fixes#692
See
- https://github.com/ansible/ansible/pull/79999
- https://github.com/ansible/ansible/commit/c69c83c962f987c78af98da0746527df
- https://www.sudo.ws/security/advisories/minus_1_uid/
> Bruce Wayne : [confused] Am I meant to understand any of that?
> Lucius Fox : Not at all, I just wanted you to know how hard it was.
> -- Batman Begins
This reads the become username from the `become_user` attribute of the play
context, to the `"become_user"` option of the loaded become plugin. This has
been supported by vanilla Ansible since Ansible 2.10 (ansible-base 2.10).
To support this I've also switched from using the `play_context.become` (a
bool), to `connection.become` (an instance of the appropriate) become plugin.
New tests have been added, modelled on those for templated connection
parameters (see #1147, #1153, #1159).
See
- 480b106d65
refs #1083
Co-authored-by: mordek <m.pirog@bonasoft.pl>
This reduces the number of jobs from 48 to 24. The Mitogen part of the test
suite has been parameterized on the Linux container targets to be run against.
Both the Ansible tests & Mitogen tests now use the same source of truth to
control which targets to use: environment variable MITOGEN_TEST_DISTRO_SPECS.
This replaces the two mutually exclusive env vars DISTRO and DISTROS. I've
also removed vestgial traces of an unused env var MITOGEN_TEST_DISTRO.
Parameterization adapted from
https://eli.thegreenplace.net/2014/04/02/dynamically-generating-python-test-cases
refs #1058, #1059
The password is provided as a variable because there is no corresponding
keyword. I get the impression that keywords are considered a legacy mechanism,
so most (new) options are only overridable by variables.
The port is proved as a variable for now, to test remote_name in isolation.
These targets are not used by any active tests, and the large numbers of hosts
multiply the size of the taskvars disctionary in memory to many (10s) MiB.
refs #1058
By switching to block style (`|`) with clip (no `-` or `+`) the failure
messages don't require quoting and gain a single trailing newline. This causes
Ansible to print them as block style, when using the yaml stdout callback
plugin. As a result the values have one less layer of quoting and quote
escaping, making them much easier to read.
This switches `ansible_mitogen.transport_config.PlayContextSpec.password()` to
Ansible's plugin option framework. As a result
- The relatively recent `ansible_ssh_password` variable is now respected.
- The SSH connection password can be templated and specified as a play
variable. Task variables will probably also work, but testing was blocked
by #1132.
There is a chance this change will cause a regression in another connection
plugin (e.g. mitogen_docker), but nothing turned up in the test suite.
I intend ot migrate other connection configuration to
`ansible_mitogen.transport_config.PlayContextSpec._connect_option()`, the next
candidate is the remote port.
fixes#1106
This replaces the use of `os.path.realpath()` which gave incorrect results on
macOS - depending on the exact Python build, Python version, macOS version,
installation method, and phase of the moon.
realpath information kept around to aid debugging.
CI containers lack the necessary `setfacl` command. This has not previously
been noticed because no vanilla Ansible jobs were being run on Linux, only on
macOS.
refs #1118
Preperation for migrating from Azure DevOps with Amazon Elastic Container
Registry (AWS ECR), to GitHub Actions with GitHub Container Registry (GHCR).
DebOps tests are not currently being run, the updates to .ci/debops*.py are
best effort only.
Python 3.x emits `ResourceWarning`s if certains resources aren't correctly
closed. Due to the way Mitogen has been terminating child processes this has
been occurring.
```
test_dev_tty_open_succeeds
(create_child_test.TtyCreateChildTest.test_dev_tty_open_succeeds) ...
/opt/hostedtoolcache/Python/3.12.5/x64/lib/python3.12/subprocess.py:1127:
ResourceWarning: subprocess 3313 is still running
_warn("subprocess %s is still running" % self.pid,
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok
```
During garbage collection subprocess.Popen() objects emit
ResourceWarning("subprocess 123 is still running")
if proc.returncode hasn't been set. Typically calling proc.wait() does so,
once the sub-process has exited. Calling os.waitpid(proc.pid, 0) also waits
for the sub-process to exit, but it doesn't update proc.returncode, so the
ResourceWarning is still emitted.
This change exposes `subprocess.Popen` methods on
`mitogen.parent.PopenProcess`, so that the returncode can be set.
See https://gist.github.com/moreati/b8d157ff82cb15234bece4033accc5e5
This
- Removes the indirection of calling ansible in a sub-shell
- Includes vanilla Ansible, which was previously skipped
- Tests whether ansible_ssh_pass overrides ansible_password, as it should
As a one off I've the new tests against vanilla Ansible 2.10 through Ansible
10, to confirm the baseline priorities have remained unchanged all releases
currently supported by Mitogen 0.3.x.
The bug was fixed in a previous commit by Jonathan Rosser. This adds testing.
The bug is only triggered when the copy module is used inside a `with_items:`
loop and the destination filename has an extension. A `loop:` loop is not
sufficient.
refs #1110
This is in anticipation of #1110, which only exhibits inside a with_items:
loop. For this refactor `loop:` is used, to confirm the refactored tests are
still correct. A subsequent commit will change them to with_items.
The content of the files and their SHA1 checksums are unchanged.
Relying on the virtualenv default or hardcoding "python" results in a Python
2.x virtualenv on some targets (e.g. debian10-test). This caused a failure
when testing with Ansible >= 10 (ansible-core >= 2.17), which have dropped
Python 2.x support.
refs #1074
By setting ansible_python_interpreter for these fictious hosts we avoid
Ansible trying and failing to connect to them in a attempt to populate
ansible_facts.discovered_interpreter_python. This speeds up these tests by
avoiding a timeout.
It is also a necessary pre-requisite for Ansible 10 (ansible-core 2.17). In
that release no hardcoded fallback is used, failure to determine a valid
Python interpreter is a fatal error.
refs #1074
Pip 72 was released yesterday (2024-07-28), dropping `setup.py test` support.
hdrhistogram 0.6.1 requires it to install.
For now constrain Pip to earlier releases, so our tests can be run.
refs #1090
macOS 11 is not longer an available runner on Azure Devops. The minimum is now
macOS 12. This runner does not have Python 2.7 installed, so running them
would require a custom install - which I'm declaring too much effort for too
little gain.
refs #1090
CentOS 8 has reached EOL. Packages are no longer mirrored or maintained. A
historic snapshot of the packages is kept on vault.centos.org.
refs #1088, #1090
Follwing fixes in Ansible 7-9 for CVE-2023-5764 cating `AnsibleUnsafeBytes` &
`AnsibleUnsafeText` to `bytes()` or `str()` requires special handling. The
handling is Ansible specific, so it shouldn't go in the mitogen package but
rather the ansible_mitogen package.
`ansible_mitogen.utils.unsafe.cast()` is most like `mitogen.utils.cast()`.
During development it began as `ansible_mitogen.utils.unsafe.unwrap_var()`,
closer to an inverse of `ansible.utils.unsafe_procy.wrap_var()`. Future
enhancements may move in this direction.
refs #977, refs #1046
See also
- https://github.com/advisories/GHSA-7j69-qfc3-2fq9
- https://github.com/ansible/ansible/pull/82293
- https://github.com/mitogen-hq/mitogen/wiki/AnsibleUnsafe-notes
If casting a string fails then raise a TypeError. This is potentially an API
breaking change; chosen as the lesser evil vs. allowing silent errors.
`cast()` relies on `bytes(obj)` & `str(obj)` returning the respective
supertype. That's no longer the case for `AnsibleUnsafeBytes` &
`AnsibleUnsafeText`; since fixes/mitigations for CVE-2023-5764.
fixes#1046, refs #977
See also
- https://github.com/advisories/GHSA-7j69-qfc3-2fq9
- https://github.com/ansible/ansible/pull/82293
Most of the necessary changes were made in recent PEP 451 commits. This bumps
the CI jobs, and declares the support. Test dependendancies are bumped to
latest supportted/available versions.
refs #1033
importlib.machinery.ModuleSpec and find_spec() were introduced in Python 3.4
under PEP 451. They replace the find_module() API of PEP 302, which was
deprecated from Python 3.4. They were removed in Python 3.12 along with the
imp module.
This change adds support for the PEP 451 APIs. Mitogen should no longer import
imp on Python versions that support ModuleSpec. Tests have been added to cover
the new APIs.
CI jobs have been added to cover Python 3.x on macOS.
Refs #1033
Co-authored-by: Witold Baryluk <witold.baryluk@gmail.com>
Using https:// requires certificate store management and additional parameter
passing that changed across Ansible and Python versions. Using http:// allows
the same tests to be used across wider spans of Python version on the
controller, and Python verison on the targets.
Python 3.12 on a target + get_uri needs Ansible >= 8 (ansible-core >= 2.15).
Python 3.12 removed deprecated httplib.HTTPSConnection() arguments.
https://github.com/ansible/ansible/pull/80751
Fixes
```
======================================================================
ERROR: setUpClass (ssh_test.BannerTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/vsts/work/1/s/tests/testlib.py", line 625, in setUpClass
cls.dockerized_ssh = DockerizedSshDaemon(**daemon_args)
File "/home/vsts/work/1/s/tests/testlib.py", line 553, in __init__
self.start_container()
File "/home/vsts/work/1/s/tests/testlib.py", line 533, in start_container
self._get_container_port()
File "/home/vsts/work/1/s/tests/testlib.py", line 510, in _get_container_port
self.port = int(bport)
ValueError: invalid literal for int() with base 10: ':]:32770'
```
To do so the test suite allows a weak cryptographic alogorithm (SHA1) to be
used, principally for CentOS 6 targets. This can be removed if/when support
for older (legacy) targets is dropped.
Only the test suite enables this known weak alogorithm. Mitogen as-shipped
doesn't enable or disable algorithms.
faulthandler is a stdlib module in Python 3.3+. For a long time a PyPI package
of the same name was available for earlier Python releases. That package has
since been removed from PyPI, and the source respoitory archived. So we should
not rely on it.
fixes#983 refs #970
Until Ansible 2.9 it looks like ansible_become_password had higher priority.
From Ansible 2.10 ansible_become_pass has higher priority [1]. Mitogen was not
respecting this.
I may need to rework this further, instatiating the become plugin may have
slowed down execution.
[1] Based on testing with
```
[ubuntus]
become-pass-pass ansible_become_pass=1234
become-pass-password ansible_become_password=1234
become-pass-both ansible_become_password=wrong ansible_become_pass=1234
[ubuntus:vars]
ansible_host=ubuntu2004.local
ansible_user=ubuntu
```
```
- hosts: ubuntus
gather_facts: false
become: true
tasks:
- ping:
```
Alex Willmer