8d1b01d8ef
Split Stream into many, many classes
* mitogen.parent.Connection: Handles connection setup logic only.
* Maintain references to stdout and stderr streams.
* Manages TimerList timer to cancel connection attempt after
deadline
* Blocking setup code replaced by async equivalents running on the
broker
* mitogen.parent.Options: Tracks connection-specific options. This
keeps the connection class small, but more importantly, it is
generic to the future desire to build and execute command lines
without starting a full connection.
* mitogen.core.Protocol: Handles program behaviour relating to events
on a stream. Protocol performs no IO of its own, instead deferring
it to Stream and Side. This makes testing much easier, and means
libssh can reimplement Stream and Side to reuse MitogenProtocol
* mitogen.core.MitogenProtocol: Guts of the old Mitogen stream
implementtion
* mitogen.core.BufferedWriter: Guts of the old Mitogen buffered
transmit implementation, made generic
* mitogen.core.DelineatedProtocol: Guts of the old IoLogger, knows how
to split up input and pass it on to a
on_line_received()/on_partial_line_received() callback.
* mitogen.parent.BootstrapProtocol: Asynchronous equivalent of the old
blocking connect code. Waits for various prompts (MITO001 etc) and
writes the bootstrap using a BufferedWriter. On success, switches
the stream to MitogenProtocol.
* mitogen.core.Message: move encoding parts of MitogenProtocol out to
Message (where it belongs) and write a bunch of new tests for
pickling.
* The bizarre Stream.construct() is gone now, Option.__init__ is its
own constructor. Should fix many LGTM errors.
* Update all connection methods: Every connection method is updated to
use async logic, defining protocols as required to handle interactive
prompts like in SSH or su. Add new real integration tests for at least
doas and su.
* Eliminate manual fd management: File descriptors are trapped in file
objects at their point of origin, and Side is updated to use file
objects rather than raw descriptors. This eliminates a whole class of
bugs where unrelated FDs could be closed by the wrong component. Now
an FD's open/closed status is fused to it everywhere in the library.
* Halve file descriptor usage: now FD open/close state is tracked by
its file object, we don't need to duplicate FDs everywhere so that
receive/transmit side can be closed independently. Instead both sides
back on to the same file object. Closes #26, Closes #470.
* Remove most uses of dup/dup2: Closes #256. File descriptors are
trapped in a common file object and shared among classes. The
remaining few uses for dup/dup2 are as close to minimal as possible.
* Introduce mitogen.parent.Process: uniform interface for subprocesses
created either via mitogen.fork or the subprocess module. Remove all
the crap where we steal a pid from subprocess guts. Now we use
subprocess to manage its processes as it should be. Closes #169 by
using the new Timers facility to poll for a slow-to-exit subprocess.
* Fix su password race: Closes #363. DelineatedProtocol naturally
retries partially received lines, preventing the cause of the original
race.
* Delete old blocking IO utility functions
iter_read()/write_all()/discard_until().
Closes #26
Closes #147
Closes #169
Closes #256
Closes #363
Closes #419
Closes #470
|
5 years ago | |
|---|---|---|
| .. | ||
| ansible | 5 years ago | |
| bench | 5 years ago | |
| data | 5 years ago | |
| image_prep | 5 years ago | |
| soak | 6 years ago | |
| README.md | 6 years ago | |
| __init__.py | 6 years ago | |
| broker_test.py | 5 years ago | |
| buildah_test.py | 5 years ago | |
| call_error_test.py | 5 years ago | |
| call_function_test.py | 5 years ago | |
| channel_test.py | 6 years ago | |
| context_test.py | 5 years ago | |
| doas_test.py | 5 years ago | |
| docker_test.py | 5 years ago | |
| error_test.py | 5 years ago | |
| fakessh_test.py | 5 years ago | |
| file_service_test.py | 5 years ago | |
| first_stage_test.py | 5 years ago | |
| fork_test.py | 5 years ago | |
| id_allocation_test.py | 6 years ago | |
| importer_test.py | 5 years ago | |
| io_op_test.py | 5 years ago | |
| iter_split_test.py | 5 years ago | |
| jail_test.py | 5 years ago | |
| kubectl_test.py | 6 years ago | |
| latch_test.py | 5 years ago | |
| local_test.py | 6 years ago | |
| lxc_test.py | 5 years ago | |
| lxd_test.py | 5 years ago | |
| master_test.py | 6 years ago | |
| message_test.py | 5 years ago | |
| minify_test.py | 5 years ago | |
| mitogen_protocol_test.py | 5 years ago | |
| module_finder_test.py | 5 years ago | |
| nested_test.py | 6 years ago | |
| os_fork_test.py | 5 years ago | |
| parent_test.py | 5 years ago | |
| policy_function_test.py | 5 years ago | |
| poller_test.py | 5 years ago | |
| polyfill_functions_test.py | 5 years ago | |
| push_file_service_test.py | 5 years ago | |
| receiver_test.py | 5 years ago | |
| requirements.txt | 6 years ago | |
| responder_test.py | 5 years ago | |
| router_test.py | 5 years ago | |
| select_test.py | 5 years ago | |
| service_test.py | 5 years ago | |
| setns_test.py | 5 years ago | |
| signals_test.py | 5 years ago | |
| ssh_test.py | 5 years ago | |
| su_test.py | 5 years ago | |
| sudo_test.py | 5 years ago | |
| testlib.py | 5 years ago | |
| timer_test.py | 5 years ago | |
| two_three_compat_test.py | 5 years ago | |
| types_test.py | 5 years ago | |
| unix_test.py | 5 years ago | |
| utils_test.py | 5 years ago | |
README.md
Warning
This directory is full of disorganized crap, including random hacks I checked in that I'd like to turn into tests. The effort to write tests only really started in September 2017. Pull requests in this area are very welcome!
Running The Tests
Your computer should have an Internet connection, and the docker command
line tool should be able to connect to a working Docker daemon (localhost or
elsewhere for OS X etc.) that can run new images.
The IP address of the Docker daemon must allow exposing ports from running containers, e.g. it should not be firewalled or port forwarded.
If in doubt, just install Docker on a Linux box in the default configuration and run the tests there.
Steps To Prepare Development Environment
- Get the code
git clone https://github.com/dw/mitogen.git - Go into the working directory
cd mitogen - Establish the docker image
./tests/build_docker_image.py - Build the virtual environment
virtualenv ../venv - Enable the virtual environment we just built
source ../venv/bin/activate - Install Mitogen in pip editable mode
pip install -e . - Run
test
Selecting a target distribution
Docker target images exist for testing against CentOS and Debian, with the
default being Debian. To select CentOS, specify MITOGEN_TEST_DISTRO=centos in
the environment.
User Accounts
A set of standard user account names are used by in the Docker container and
also by Ansible's osx_setup.yml.
root
In the Docker image only, the password is "rootpassword".
mitogen__has_sudo
The login password is "has_sudo_password" and the account is capable of
sudoing to root, by supplying the account password to sudo.
mitogen__has_sudo_pubkey
The login password is "has_sudo_pubkey_password". Additionally
tests/data/docker/mitogen__has_sudo_pubkey.key SSH key may be used to log
in. It can sudo the same as mitogen__has_sudo.
mitogen__has_sudo_nopw
The login password is "has_sudo_nopw_password". It can sudo to root without
supplying a password. It has explicit sudoers rules forcing it to require a
password for other accounts.
mitogen__pw_required
The login password is "pw_required_password". When "sudo -u" is used to
target this account, its password must be entered rather than the login
user's password.
mitogen__require_tty
The login password is "require_tty_password". When "sudo -u" is used to
target this account, the parent session requires a TTY.
mitogen__require_tty_pw_required
The login password is "require_tty_pw_required_password". When "sudo -u" is
used to target this account, the parent session requires a TTY and the
account password must be entered.
mitogen__user1 .. mitogen__user5
These accounts do not have passwords set. They exist to test the Ansible
interpreter recycling logic.
mitogen__sudo1 .. mitogen__sudo4
May passwordless sudo to any account.
mitogen__webapp
A plain old account with no sudo access, used as the target for fakessh
tests.
Ansible Integration Test Environment
The integration tests expect to be run against a either one of the Docker images, or a similar target with the same set of UNIX accounts and sudo rules.
The login account should be able to sudo to root witout a password.