You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

History

Alex Willmer 85d6046f2f mitogen: Fix non-blocking IO errors in first stage of bootstrap When /etc/sudoers has log_output (or similar) enabled the process spawned by `ctx.sudo()` via `mitogen.parent.Connection.start_child()` receives a stdin that is in non-blocking mode. The immediate symptom is that `os.openfd(0, ...).read(n)` sometimes returns `None`, causing the first stage to raise an unhandled TypeError. The fix (for now) is to use `select.select()` in a while loop to read stdin. This increases the command size slightly, but I think it's a reasonable tradeoff until/unless the cause is more fully understood. All CI tests are now run with sudoers log_output enabled, in order to catch regressions. `first_stage_test.CommandLineTest` has been amended, because it relied on implementation details of the bootstrap process that are no longer true. Before ``` SSH command size: 755 Preamble (mitogen.core + econtext) size: 18227 (17.80KiB) Original Minimized Compressed mitogen.core 152218 148.7KiB 68437 66.8KiB 45.0% 18124 17.7KiB 11.9% mitogen.parent 98853 96.5KiB 51103 49.9KiB 51.7% 12881 12.6KiB 13.0% mitogen.fork 8445 8.2KiB 4139 4.0KiB 49.0% 1652 1.6KiB 19.6% mitogen.ssh 10827 10.6KiB 6893 6.7KiB 63.7% 2099 2.0KiB 19.4% mitogen.sudo 12089 11.8KiB 5924 5.8KiB 49.0% 2249 2.2KiB 18.6% mitogen.select 12325 12.0KiB 2929 2.9KiB 23.8% 964 0.9KiB 7.8% mitogen.service 41581 40.6KiB 22398 21.9KiB 53.9% 5847 5.7KiB 14.1% mitogen.fakessh 15767 15.4KiB 8149 8.0KiB 51.7% 2676 2.6KiB 17.0% mitogen.master 55317 54.0KiB 28846 28.2KiB 52.1% 7528 7.4KiB 13.6% ``` After ``` SSH command size: 798 Preamble (mitogen.core + econtext) size: 18227 (17.80KiB) Original Minimized Compressed mitogen.core 152218 148.7KiB 68437 66.8KiB 45.0% 18124 17.7KiB 11.9% mitogen.parent 98944 96.6KiB 51180 50.0KiB 51.7% 12910 12.6KiB 13.0% mitogen.fork 8445 8.2KiB 4139 4.0KiB 49.0% 1652 1.6KiB 19.6% mitogen.ssh 10827 10.6KiB 6893 6.7KiB 63.7% 2099 2.0KiB 19.4% mitogen.sudo 12089 11.8KiB 5924 5.8KiB 49.0% 2249 2.2KiB 18.6% mitogen.select 12325 12.0KiB 2929 2.9KiB 23.8% 964 0.9KiB 7.8% mitogen.service 41581 40.6KiB 22398 21.9KiB 53.9% 5847 5.7KiB 14.1% mitogen.fakessh 15767 15.4KiB 8149 8.0KiB 51.7% 2676 2.6KiB 17.0% mitogen.master 55317 54.0KiB 28846 28.2KiB 52.1% 7528 7.4KiB 13.6% ```		3 months ago
..
soak	.ci: import soak scripts.	7 years ago
README.md	ci: Reduce number of Jobs by parameterizing Mitogen Docker SSH tests	1 year ago
ansible_tests.py	CI: Consolidate directory path constants	10 months ago
bash_functions	CI: Combine build deps & tooling steps	4 months ago
ci_lib.py	mitogen: Fix non-blocking IO errors in first stage of bootstrap	3 months ago
debops_common_install.py	ci: Extract container registry location into variables	1 year ago
debops_common_tests.py	CI: Don't share temporary directory between test groupings	10 months ago
install_sshpass	CI: Test SSH password authentication without sshpass command	5 months ago
localhost_ansible_tests.py	CI: Test SSH password authentication without sshpass command	5 months ago
mitogen_py24_install.py	ci: Move container registry authentication to an Azure Devops step	1 year ago
mitogen_py24_tests.py	ci: Reduce number of Jobs by parameterizing Mitogen Docker SSH tests	1 year ago
mitogen_tests.py	mitogen: Fix non-blocking IO errors in first stage of bootstrap	3 months ago
show_python_versions	CI: Factor out .ci/show_python_versions	4 months ago

README.md

`.ci`

This directory contains scripts for Continuous Integration platforms. Currently GitHub Actions, but ideally they will also run on any Debian-like machine.

The scripts are usually split into _install and _test steps. The _install step will damage your machine, the _test step will just run the tests the way CI runs them.

There is a common library, ci_lib.py, which just centralized a bunch of random macros and also environment parsing.

Some of the scripts allow you to pass extra flags through to the component under test, e.g. ../../.ci/ansible_tests.py -vvv will run with verbose.

Hack these scripts until your heart is content. There is no pride to be found here, just necessity.

`ci_lib.run_batches()`

There are some weird looking functions to extract more paralellism from the build. The above function takes lists of strings, arranging for the strings in each list to run in order, but for the lists to run in parallel. That's great for doing setup.py install while pulling a Docker container, for example.

Environment Variables

MITOGEN_TEST_DISTRO_SPECS: a space delimited list of distro specs to run the tests against. (e.g. centos6 ubuntu2004-py3*4). Each spec determines the Linux distribution, target Python interepreter & number of instances. Only distributions with a pre-built Linux container image can be used.
- debian-py3: when generating Ansible inventory file, set ansible_python_interpreter to python3, i.e. run a test where the target interpreter is Python 3.
- debian*16: generate 16 Docker containers running Debian. Also works with -py3.
MITOGEN_TEST_IMAGE_TEMPLATE: specifies the Linux container image name, and hence the container registry used for test targets.