Commit Graph

68 Commits (cfbb7f884e28f55400023f60ab879d850b64d719)

Author SHA1 Message Date
Alex Willmer cfbb7f884e ci: Add playbook to configure container host for image prep 1 week ago
Alex Willmer a1b5d4941e ci: Use upstream base images for image prep
This eliminates use of third-party *-vault images and performs repository
config during image prep.

The Apache httpd proxy is necessary because https://vault.centos.org now only
accepts TLS 1.x connections, and CentOS 5 can only do upto SSL 3.0. It is
developed to run on Debian 11.
1 week ago
Alex Willmer e32c90a63e ci: Factor out package installation role 1 week ago
Alex Willmer a143787c02 ci: Handle custom package repositories in bootstrap role 1 week ago
Alex Willmer bcc726d3b7 ci: Handle dnf packages in bootstrap role 1 week ago
Alex Willmer 780f8af1a4 ci: Factor out image prep bootstrap as a role
Promoting the script to a full template will fix some whitespace errors later.
1 week ago
Alex Willmer d1c4217db0 ci: Wait for fresh image prep containers to start 1 week ago
Alex Willmer 09b972e96e ci: Fix ansible-lint complaints in image prep playbooks 1 week ago
Alex Willmer f966b3e5c6 CI: Remove lingering stdout_callback=yaml in macOS jobs
Support removed in Ansible 13 (ansible-core 2.20).
refs #1285, #1291
2 weeks ago
Alex Willmer d27275ad46 ci: Set global max failed logins on macOS 3 months ago
Alex Willmer 85d6046f2f mitogen: Fix non-blocking IO errors in first stage of bootstrap
When /etc/sudoers has log_output (or similar) enabled the process spawned by
`ctx.sudo()` via `mitogen.parent.Connection.start_child()` receives a stdin
that is in non-blocking mode. The immediate symptom is that `os.openfd(0,
...).read(n)` sometimes returns `None`, causing the first stage to raise an
unhandled TypeError.

The fix (for now) is to use `select.select()` in a while loop to read stdin.
This increases the command size slightly, but I think it's a reasonable
tradeoff until/unless the cause is more fully understood.

All CI tests are now run with sudoers log_output enabled, in order to catch
regressions. `first_stage_test.CommandLineTest` has been amended, because it
relied on implementation details of the bootstrap process that are no longer
true.

Before
```
SSH command size: 755
Preamble (mitogen.core + econtext) size: 18227 (17.80KiB)

                        Original           Minimized           Compressed
mitogen.core         152218 148.7KiB  68437 66.8KiB 45.0%  18124 17.7KiB 11.9%
mitogen.parent        98853  96.5KiB  51103 49.9KiB 51.7%  12881 12.6KiB 13.0%
mitogen.fork           8445   8.2KiB   4139  4.0KiB 49.0%   1652  1.6KiB 19.6%
mitogen.ssh           10827  10.6KiB   6893  6.7KiB 63.7%   2099  2.0KiB 19.4%
mitogen.sudo          12089  11.8KiB   5924  5.8KiB 49.0%   2249  2.2KiB 18.6%
mitogen.select        12325  12.0KiB   2929  2.9KiB 23.8%    964  0.9KiB  7.8%
mitogen.service       41581  40.6KiB  22398 21.9KiB 53.9%   5847  5.7KiB 14.1%
mitogen.fakessh       15767  15.4KiB   8149  8.0KiB 51.7%   2676  2.6KiB 17.0%
mitogen.master        55317  54.0KiB  28846 28.2KiB 52.1%   7528  7.4KiB 13.6%
```

After
```
SSH command size: 798
Preamble (mitogen.core + econtext) size: 18227 (17.80KiB)

                        Original           Minimized           Compressed
mitogen.core         152218 148.7KiB  68437 66.8KiB 45.0%  18124 17.7KiB 11.9%
mitogen.parent        98944  96.6KiB  51180 50.0KiB 51.7%  12910 12.6KiB 13.0%
mitogen.fork           8445   8.2KiB   4139  4.0KiB 49.0%   1652  1.6KiB 19.6%
mitogen.ssh           10827  10.6KiB   6893  6.7KiB 63.7%   2099  2.0KiB 19.4%
mitogen.sudo          12089  11.8KiB   5924  5.8KiB 49.0%   2249  2.2KiB 18.6%
mitogen.select        12325  12.0KiB   2929  2.9KiB 23.8%    964  0.9KiB  7.8%
mitogen.service       41581  40.6KiB  22398 21.9KiB 53.9%   5847  5.7KiB 14.1%
mitogen.fakessh       15767  15.4KiB   8149  8.0KiB 51.7%   2676  2.6KiB 17.0%
mitogen.master        55317  54.0KiB  28846 28.2KiB 52.1%   7528  7.4KiB 13.6%
```
4 months ago
Alex Willmer e4e2c6caaf CI: Move sudo test users defaults into /etc/sudoers.d
Prep for reusing it in non-Ansible tests
4 months ago
Alex Willmer 618eccc0f3 CI: Set macOS failed logins limit of mitogen test users to 1000
refs #1315
4 months ago
Alex Willmer 3cba11a126 CI: Fix ansible_version comparison with ansible-core 2.19.0rc1
Note that tests/ansible/integration/ssh/templated_by_play_taskvar.yml was
previously erroniously being skipped with ansible-core 2.19.0a<N> and
2.19.0b<N>.

fixes #1293
refs #1175
5 months ago
Alex Willmer 2598941384 tests: Add Debian 11/bullseye security archive signing key
Tests that install packages are failing due to repos/packages that are signed
with this key.

```console
$ wget https://ftp-master.debian.org/keys/archive-key-11-security.asc
--2025-06-17 14:36:04--  https://ftp-master.debian.org/keys/archive-key-11-security.asc
Resolving ftp-master.debian.org (ftp-master.debian.org)... 192.91.235.231
Connecting to ftp-master.debian.org (ftp-master.debian.org)|192.91.235.231|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 11873 (12K) [application/pgp-keys]
Saving to: ‘archive-key-11-security.asc’

archive-key-11-security.asc 100%[=================>]  11.59K  --.-KB/s    in 0.002s

2025-06-17 14:36:05 (6.64 MB/s) - ‘archive-key-11-security.asc’ saved [11873/11873]

$ sha256sum archive-key-11-security.asc
716e79393c724d14ecba8be46e99ecbe1b689f67ceff3cb3cab28f6e69e8b8b8  archive-key-11-security.asc
$ cp archive-key-11-security.asc \
     ~/src/mitogen/tests/image_prep/roles/package_manager/files/debian-archive-bullseye-security-automatic.asc
```
6 months ago
Alex Willmer fdbd1a8c9b ci: Configure package managers using a role
This allows code sharing between integration tests and test image prep.
9 months ago
Alex Willmer 1e12edbf95 ci: Use file module to set mitogen__readonly_homedir permissions 9 months ago
Alex Willmer f2e0e552ac ci: Fix sshd configuration during image prep
This will allow image preparation using Ansible versions that
- predate ansible_facts.*
- predate loop keyword
- predate collections
9 months ago
Alex Willmer 20e23b5bd9 ci: Name all image prep plays 9 months ago
Alex Willmer 8e58c4a759 ci: Decouple image prep from Ansible controller reporting
This will allow image preparation using Ansible versions that predate
import_playbook.
9 months ago
Alex Willmer 78b440104e CI: Validate sudoers file 10 months ago
Alex Willmer c92df356e6 CI: Consolidate sudoers config tasks 10 months ago
Alex Willmer 11d2d70fd8 CI: Use native Ansible support to hide macOS users 10 months ago
Alex Willmer 5283e6756b CI: Statically specify test usernames and group names
This makes it easier to grep for a username and to discover how the user was
create. Hence it should be easier to understand/debug tests.
10 months ago
Alex Willmer 3a1b5ec620 CI: Increase sshd MaxAuthRetries to 50 on macOS runners
refs #1186
1 year ago
Alex Willmer 8cfcb66cda CI: Refactor sshd configuration into a role
Prep for applying it to macOS 13 GitHub runners.

refs #1186
1 year ago
Alex Willmer 8a34b925a4 tests: Re-enable become/sudo tests, fix them on macOS runners
The tasks in tests/imageprep/_user_accounts.yml that create users did not
specify a primary group for those users - this left the decision to Ansible's
user module, and/or the underlying OS. In Ansible 9+ (ansible-core 2.16+ the
user module defaults to primary group "staff." Earlier don't supply a default,
which releases probably results in a primary group nameed "None" (due to
stringifying the Python singleton of the same name), or whatever the macOS
Directory Services has for no data/NULL.

The invalid GID 4294967295 (MAX_UINT32 == 2**32-1) in the sudo error probably
enters the mix via something similar to sudo CVE-2019-14287.

Fixes #692

See
- https://github.com/ansible/ansible/pull/79999
- https://github.com/ansible/ansible/commit/c69c83c962f987c78af98da0746527df
- https://www.sudo.ws/security/advisories/minus_1_uid/

> Bruce Wayne : [confused]  Am I meant to understand any of that?
> Lucius Fox : Not at all, I just wanted you to know how hard it was.
> -- Batman Begins
1 year ago
Alex Willmer 1773c9aba6 trivia: Fix trailing whitespace 1 year ago
Alex Willmer 8b92e09655 ci: Extract container registry location into variables
Preperation for migrating from Azure DevOps with Amazon Elastic Container
Registry (AWS ECR), to GitHub Actions with GitHub Container Registry (GHCR).

DebOps tests are not currently being run, the updates to .ci/debops*.py are
best effort only.
1 year ago
Alex Willmer 45c42d386a tests: Replace uses of ``include:``, unify skipping of mitogen only tests
The tag mitogen_only is only informational for now. It may be possible to use
it with ANSIBLE_SKIP_TAGS in the future.
2 years ago
Alex Willmer b822f20007 ansible_mitogen: Handle AnsibleUnsafeText et al in Ansible >= 7
Follwing fixes in Ansible 7-9 for CVE-2023-5764 cating `AnsibleUnsafeBytes` &
`AnsibleUnsafeText` to `bytes()` or `str()` requires special handling. The
handling is Ansible specific, so it shouldn't go in the mitogen package but
rather the ansible_mitogen package.

`ansible_mitogen.utils.unsafe.cast()` is most like `mitogen.utils.cast()`.
During development it began as `ansible_mitogen.utils.unsafe.unwrap_var()`,
closer to an inverse of `ansible.utils.unsafe_procy.wrap_var()`. Future
enhancements may move in this direction.

refs #977, refs #1046

See also
- https://github.com/advisories/GHSA-7j69-qfc3-2fq9
- https://github.com/ansible/ansible/pull/82293
- https://github.com/mitogen-hq/mitogen/wiki/AnsibleUnsafe-notes
2 years ago
Alex Willmer a6c89751f9 tests: Cleanup ansible-lint errors & warnings in user creation playbook
Task " Install slow profile for one account" removed because it duplicates
earlier work.
2 years ago
Alex Willmer 8b574f234d tests: Report Ansible controller parameters before image prep & user creation 2 years ago
Alex Willmer ac7505d624 tests: Add centos 8; debian 10, 11; ubuntu 16.04, 18.04, 20.04 test images 5 years ago
Alex Willmer 6bf58c3cfb tests: Don't add local user to Docker containers 5 years ago
Alex Willmer a8e8cf91cb tests: Rebuild Docker containers
A few changes are bundled in this
 - Ansible 2.10.x and Mitogen 0.3.x are used to build nearly all images
   (Ansile 2.3.x is retained for CentOS 5, because it uses Python 2.4).
 - Tox is used to install/run Ansible, replacing build_docker_images.py
 - A static inventory, identifying containers by name rather than ID.
 - debian-test image is renamed to debian9-test
 - debian9-test image is now based on debian:9
 - centos6-test image is now based on moreati/centos6-vault
   following the same scheme as centos5-test.
 - Images are now uploaded to Amazon Elastic Container Registry (ECR).
   See #809.
 - Debian recommended packages aren't installed (e.g. build-essential)
 - Python 2.x and Python 3.x are installed wherever available.
 - Python Virtualenv is installed wherever available.
5 years ago
Steven Robertson 72e6abf6db attempt at fixing 'sudo runas gid invalid value' since the sudo command looks correct 6 years ago
David Wilson e4321f81a0 issue #600: /etc/environment may be non-ASCII in an unknown encoding 6 years ago
David Wilson ebb4a7ca6a issue #543: dumb fix for file vs. stat :( 6 years ago
David Wilson f3915b5f40 issue #543: disable host key checking 6 years ago
David Wilson 0741876392 issue #543: Hide Mitogen test users from gdm 6 years ago
David Wilson 0e55bb3eb7 image_prep: ensure Mac users can SSH without manual intervention 6 years ago
David Wilson 501cfca82b issue #543: make localhost_ansible_tests run locally 6 years ago
David Wilson 87440ec6f7 [stream-refactor] Debian Docker container image initctl 6 years ago
David Wilson 2f950b3bda [stream-refactor] allow doas_test to succeed on CentOS
Unlike on Debian, some environment variables that tickle
getpass.getuser() are being inherited. So use getuid() instead.

Also install the doas binary on CentOS. CI was changed (I believe) to
shrink the configuration matrix, and now these tests run on CentOS too.
6 years ago
David Wilson f0065d76d8 [stream-refactor] add descriptive task names to _container_prep 6 years ago
David Wilson 4524f03a48 issue #271: add mitogen__permdenied user to Docker image. 6 years ago
David Wilson d8dc5420ce tests: install OpenBSD doas port in Debian image.
To allow fancy new improved doas_test.
6 years ago
David Wilson 60fe3fd6f5 issue #429: enable en_US locale to unbreak debops test. 7 years ago
David Wilson 7531af3ee0 issue #499: fix another mind-numbingly stupid vanilla inconsistency 7 years ago