Commit Graph

81 Commits (682faf85fc406ada7e54b6c2092cda9301940c2c)

Author SHA1 Message Date
Alex Willmer e0103eb66c CI: Add OS release coverage: AlmaLinux 9 1 week ago
Alex Willmer 7996a03a37 ci: Bootstrap Debian like containers with python-apt or python3-apt
The Ansible apt module requires it
1 week ago
Alex Willmer 5b6b076c4e ci: Avoid ansible_virtualization_type to check for docker targets
It's not consistant across Ansible versions, particular the oldest ones. This
may have contributed to older test images containing usernames from the host
OS that they were built on (e.g. dmw, alex).
1 week ago
Alex Willmer 4ecafc564d ci: Use command: true as noop handler
meta: noop failed on older Ansibles (e.g. 2.3)
1 week ago
Alex Willmer 56dce28906 ci: Dont show arguments in task name during image prep
A bit to noisy for my taste
1 week ago
Alex Willmer ff973775ce ci: Push new container images to GitHub Container Registry 1 week ago
Alex Willmer 5ffdbb5999 ci: Add Alma 9, Debian 12, Ubuntu 22.04, & Ubuntu 24.04 to image prep 1 week ago
Alex Willmer 01e24f9ddf ci: Use highest supported Ansible version during image prep
It was necessary to split setup.yml because there is no common subset of
supported include/import keywords across Ansible 2.3 - 2.11. The yaml stdout
callback is unavailabe in Ansible 2.3.
1 week ago
Alex Willmer 22e7046cf6 ci: Run image-prep as fast as possible
Mitogen maintainer(s) got better laptops in the last decaade or so.
1 week ago
Alex Willmer cc8a39864d ci: Only install default Python 3.x during image prep
Newer images will shortly be generated, so these higher Python versions aren't
needed anymore.
1 week ago
Alex Willmer 40fbfe58fc ci: Install doas package during image prep, delete vendored doas
Debian 11 is the earliest Debian release with such a package. Ubuntu first
included it in 22.04 CentOS doesn't have it.
1 week ago
Alex Willmer b353980699 ci: Tighten Ansible error checking during image prep 1 week ago
Alex Willmer 3fe9b9bd87 ci: Install setfacl for vanilla Ansible unprivileged become 1 week ago
Alex Willmer cfbb7f884e ci: Add playbook to configure container host for image prep 1 week ago
Alex Willmer a1b5d4941e ci: Use upstream base images for image prep
This eliminates use of third-party *-vault images and performs repository
config during image prep.

The Apache httpd proxy is necessary because https://vault.centos.org now only
accepts TLS 1.x connections, and CentOS 5 can only do upto SSL 3.0. It is
developed to run on Debian 11.
1 week ago
Alex Willmer e32c90a63e ci: Factor out package installation role 1 week ago
Alex Willmer a143787c02 ci: Handle custom package repositories in bootstrap role 1 week ago
Alex Willmer bcc726d3b7 ci: Handle dnf packages in bootstrap role 1 week ago
Alex Willmer 780f8af1a4 ci: Factor out image prep bootstrap as a role
Promoting the script to a full template will fix some whitespace errors later.
1 week ago
Alex Willmer d1c4217db0 ci: Wait for fresh image prep containers to start 1 week ago
Alex Willmer 09b972e96e ci: Fix ansible-lint complaints in image prep playbooks 1 week ago
Alex Willmer f966b3e5c6 CI: Remove lingering stdout_callback=yaml in macOS jobs
Support removed in Ansible 13 (ansible-core 2.20).
refs #1285, #1291
2 weeks ago
Alex Willmer d27275ad46 ci: Set global max failed logins on macOS 3 months ago
Alex Willmer 85d6046f2f mitogen: Fix non-blocking IO errors in first stage of bootstrap
When /etc/sudoers has log_output (or similar) enabled the process spawned by
`ctx.sudo()` via `mitogen.parent.Connection.start_child()` receives a stdin
that is in non-blocking mode. The immediate symptom is that `os.openfd(0,
...).read(n)` sometimes returns `None`, causing the first stage to raise an
unhandled TypeError.

The fix (for now) is to use `select.select()` in a while loop to read stdin.
This increases the command size slightly, but I think it's a reasonable
tradeoff until/unless the cause is more fully understood.

All CI tests are now run with sudoers log_output enabled, in order to catch
regressions. `first_stage_test.CommandLineTest` has been amended, because it
relied on implementation details of the bootstrap process that are no longer
true.

Before
```
SSH command size: 755
Preamble (mitogen.core + econtext) size: 18227 (17.80KiB)

                        Original           Minimized           Compressed
mitogen.core         152218 148.7KiB  68437 66.8KiB 45.0%  18124 17.7KiB 11.9%
mitogen.parent        98853  96.5KiB  51103 49.9KiB 51.7%  12881 12.6KiB 13.0%
mitogen.fork           8445   8.2KiB   4139  4.0KiB 49.0%   1652  1.6KiB 19.6%
mitogen.ssh           10827  10.6KiB   6893  6.7KiB 63.7%   2099  2.0KiB 19.4%
mitogen.sudo          12089  11.8KiB   5924  5.8KiB 49.0%   2249  2.2KiB 18.6%
mitogen.select        12325  12.0KiB   2929  2.9KiB 23.8%    964  0.9KiB  7.8%
mitogen.service       41581  40.6KiB  22398 21.9KiB 53.9%   5847  5.7KiB 14.1%
mitogen.fakessh       15767  15.4KiB   8149  8.0KiB 51.7%   2676  2.6KiB 17.0%
mitogen.master        55317  54.0KiB  28846 28.2KiB 52.1%   7528  7.4KiB 13.6%
```

After
```
SSH command size: 798
Preamble (mitogen.core + econtext) size: 18227 (17.80KiB)

                        Original           Minimized           Compressed
mitogen.core         152218 148.7KiB  68437 66.8KiB 45.0%  18124 17.7KiB 11.9%
mitogen.parent        98944  96.6KiB  51180 50.0KiB 51.7%  12910 12.6KiB 13.0%
mitogen.fork           8445   8.2KiB   4139  4.0KiB 49.0%   1652  1.6KiB 19.6%
mitogen.ssh           10827  10.6KiB   6893  6.7KiB 63.7%   2099  2.0KiB 19.4%
mitogen.sudo          12089  11.8KiB   5924  5.8KiB 49.0%   2249  2.2KiB 18.6%
mitogen.select        12325  12.0KiB   2929  2.9KiB 23.8%    964  0.9KiB  7.8%
mitogen.service       41581  40.6KiB  22398 21.9KiB 53.9%   5847  5.7KiB 14.1%
mitogen.fakessh       15767  15.4KiB   8149  8.0KiB 51.7%   2676  2.6KiB 17.0%
mitogen.master        55317  54.0KiB  28846 28.2KiB 52.1%   7528  7.4KiB 13.6%
```
3 months ago
Alex Willmer e4e2c6caaf CI: Move sudo test users defaults into /etc/sudoers.d
Prep for reusing it in non-Ansible tests
3 months ago
Alex Willmer 618eccc0f3 CI: Set macOS failed logins limit of mitogen test users to 1000
refs #1315
4 months ago
Alex Willmer 3cba11a126 CI: Fix ansible_version comparison with ansible-core 2.19.0rc1
Note that tests/ansible/integration/ssh/templated_by_play_taskvar.yml was
previously erroniously being skipped with ansible-core 2.19.0a<N> and
2.19.0b<N>.

fixes #1293
refs #1175
5 months ago
Alex Willmer 2598941384 tests: Add Debian 11/bullseye security archive signing key
Tests that install packages are failing due to repos/packages that are signed
with this key.

```console
$ wget https://ftp-master.debian.org/keys/archive-key-11-security.asc
--2025-06-17 14:36:04--  https://ftp-master.debian.org/keys/archive-key-11-security.asc
Resolving ftp-master.debian.org (ftp-master.debian.org)... 192.91.235.231
Connecting to ftp-master.debian.org (ftp-master.debian.org)|192.91.235.231|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 11873 (12K) [application/pgp-keys]
Saving to: ‘archive-key-11-security.asc’

archive-key-11-security.asc 100%[=================>]  11.59K  --.-KB/s    in 0.002s

2025-06-17 14:36:05 (6.64 MB/s) - ‘archive-key-11-security.asc’ saved [11873/11873]

$ sha256sum archive-key-11-security.asc
716e79393c724d14ecba8be46e99ecbe1b689f67ceff3cb3cab28f6e69e8b8b8  archive-key-11-security.asc
$ cp archive-key-11-security.asc \
     ~/src/mitogen/tests/image_prep/roles/package_manager/files/debian-archive-bullseye-security-automatic.asc
```
6 months ago
Alex Willmer fdbd1a8c9b ci: Configure package managers using a role
This allows code sharing between integration tests and test image prep.
9 months ago
Alex Willmer 1e12edbf95 ci: Use file module to set mitogen__readonly_homedir permissions 9 months ago
Alex Willmer f2e0e552ac ci: Fix sshd configuration during image prep
This will allow image preparation using Ansible versions that
- predate ansible_facts.*
- predate loop keyword
- predate collections
9 months ago
Alex Willmer 20e23b5bd9 ci: Name all image prep plays 9 months ago
Alex Willmer 8e58c4a759 ci: Decouple image prep from Ansible controller reporting
This will allow image preparation using Ansible versions that predate
import_playbook.
9 months ago
Alex Willmer 78b440104e CI: Validate sudoers file 10 months ago
Alex Willmer c92df356e6 CI: Consolidate sudoers config tasks 10 months ago
Alex Willmer 11d2d70fd8 CI: Use native Ansible support to hide macOS users 10 months ago
Alex Willmer 5283e6756b CI: Statically specify test usernames and group names
This makes it easier to grep for a username and to discover how the user was
create. Hence it should be easier to understand/debug tests.
10 months ago
Alex Willmer 3a1b5ec620 CI: Increase sshd MaxAuthRetries to 50 on macOS runners
refs #1186
1 year ago
Alex Willmer 8cfcb66cda CI: Refactor sshd configuration into a role
Prep for applying it to macOS 13 GitHub runners.

refs #1186
1 year ago
Alex Willmer 8a34b925a4 tests: Re-enable become/sudo tests, fix them on macOS runners
The tasks in tests/imageprep/_user_accounts.yml that create users did not
specify a primary group for those users - this left the decision to Ansible's
user module, and/or the underlying OS. In Ansible 9+ (ansible-core 2.16+ the
user module defaults to primary group "staff." Earlier don't supply a default,
which releases probably results in a primary group nameed "None" (due to
stringifying the Python singleton of the same name), or whatever the macOS
Directory Services has for no data/NULL.

The invalid GID 4294967295 (MAX_UINT32 == 2**32-1) in the sudo error probably
enters the mix via something similar to sudo CVE-2019-14287.

Fixes #692

See
- https://github.com/ansible/ansible/pull/79999
- https://github.com/ansible/ansible/commit/c69c83c962f987c78af98da0746527df
- https://www.sudo.ws/security/advisories/minus_1_uid/

> Bruce Wayne : [confused]  Am I meant to understand any of that?
> Lucius Fox : Not at all, I just wanted you to know how hard it was.
> -- Batman Begins
1 year ago
Alex Willmer 1773c9aba6 trivia: Fix trailing whitespace 1 year ago
Alex Willmer 8b92e09655 ci: Extract container registry location into variables
Preperation for migrating from Azure DevOps with Amazon Elastic Container
Registry (AWS ECR), to GitHub Actions with GitHub Container Registry (GHCR).

DebOps tests are not currently being run, the updates to .ci/debops*.py are
best effort only.
1 year ago
Alex Willmer 45c42d386a tests: Replace uses of ``include:``, unify skipping of mitogen only tests
The tag mitogen_only is only informational for now. It may be possible to use
it with ANSIBLE_SKIP_TAGS in the future.
2 years ago
Alex Willmer b822f20007 ansible_mitogen: Handle AnsibleUnsafeText et al in Ansible >= 7
Follwing fixes in Ansible 7-9 for CVE-2023-5764 cating `AnsibleUnsafeBytes` &
`AnsibleUnsafeText` to `bytes()` or `str()` requires special handling. The
handling is Ansible specific, so it shouldn't go in the mitogen package but
rather the ansible_mitogen package.

`ansible_mitogen.utils.unsafe.cast()` is most like `mitogen.utils.cast()`.
During development it began as `ansible_mitogen.utils.unsafe.unwrap_var()`,
closer to an inverse of `ansible.utils.unsafe_procy.wrap_var()`. Future
enhancements may move in this direction.

refs #977, refs #1046

See also
- https://github.com/advisories/GHSA-7j69-qfc3-2fq9
- https://github.com/ansible/ansible/pull/82293
- https://github.com/mitogen-hq/mitogen/wiki/AnsibleUnsafe-notes
2 years ago
Alex Willmer a6c89751f9 tests: Cleanup ansible-lint errors & warnings in user creation playbook
Task " Install slow profile for one account" removed because it duplicates
earlier work.
2 years ago
Alex Willmer 8b574f234d tests: Report Ansible controller parameters before image prep & user creation 2 years ago
Alex Willmer ac7505d624 tests: Add centos 8; debian 10, 11; ubuntu 16.04, 18.04, 20.04 test images 5 years ago
Alex Willmer 6bf58c3cfb tests: Don't add local user to Docker containers 5 years ago
Alex Willmer a8e8cf91cb tests: Rebuild Docker containers
A few changes are bundled in this
 - Ansible 2.10.x and Mitogen 0.3.x are used to build nearly all images
   (Ansile 2.3.x is retained for CentOS 5, because it uses Python 2.4).
 - Tox is used to install/run Ansible, replacing build_docker_images.py
 - A static inventory, identifying containers by name rather than ID.
 - debian-test image is renamed to debian9-test
 - debian9-test image is now based on debian:9
 - centos6-test image is now based on moreati/centos6-vault
   following the same scheme as centos5-test.
 - Images are now uploaded to Amazon Elastic Container Registry (ECR).
   See #809.
 - Debian recommended packages aren't installed (e.g. build-essential)
 - Python 2.x and Python 3.x are installed wherever available.
 - Python Virtualenv is installed wherever available.
5 years ago
Steven Robertson 72e6abf6db attempt at fixing 'sudo runas gid invalid value' since the sudo command looks correct 6 years ago