Commit Graph

1082 Commits (master)

Author SHA1 Message Date
David Wilson 711f46aee9 tests: move tty_create_child tests together 5 years ago
David Wilson aa06b960f5 parent: define Connection behaviour during Broker.shutdown()
- Connection attempt fails reliably, and it fails with CancelledError
- Add new mitogen.core.unlisten()
- Add test.
5 years ago
David Wilson 19b259a45f issue #549: skip Docker tests if Docker is unavailable 5 years ago
David Wilson dcfd733e6f issue #549: remove Linux-specific assumptions from create_child_test
Some stat fields are implementation-specific, little value even testing
them on Linux
5 years ago
David Wilson e4321f81a0 issue #600: /etc/environment may be non-ASCII in an unknown encoding 5 years ago
David Wilson b75198396d Fix unit_Test.ClientTest following 108015aa22
Closes #604
5 years ago
David Wilson 28b4d63e49 tests: terraform tweaks 5 years ago
David Wilson 5ed0b93669 tests: update gcloud.py to match terraform config 5 years ago
David Wilson de2e1ec184 tests/ansible/gcloud: terraform conf for load testing 5 years ago
David Wilson 108015aa22 ansible: gracefully handle failure to connect to MuxProcess
It's possible to hit an ugly exception during early CTRL+C
5 years ago
David Wilson a9d3fdf6b7 ansible: fix affinity tests for 5ae45f6612390bbc888b65964fb5c218feed1679 5 years ago
David Wilson ebb4a7ca6a issue #543: dumb fix for file vs. stat :( 5 years ago
David Wilson f3915b5f40 issue #543: disable host key checking 5 years ago
David Wilson 0741876392 issue #543: Hide Mitogen test users from gdm 5 years ago
David Wilson 17d0e1b315 issue #543: skip test that's hard to do on Mac 5 years ago
David Wilson 0e55bb3eb7 image_prep: ensure Mac users can SSH without manual intervention 5 years ago
David Wilson 501cfca82b issue #543: make localhost_ansible_tests run locally 5 years ago
David Wilson 115145555e [linear2] fix another test relying on Connection.parent 5 years ago
David Wilson a39169ff16 [linear2] fix mitogen_shutdown_all service context access 5 years ago
David Wilson 1fca0b7a94 [linear2] fix MuxProcess test fixture and some merge fallout 5 years ago
David Wilson 9035884c77 ansible: abstract worker process model.
Move all details of broker/router setup out of connection.py, instead
deferring it to a WorkerModel class exported by process.py via
get_worker_model(). The running strategy can override the configured
worker model via _get_worker_model().

ClassicWorkerModel is installed by default, which implements the
extension's existing process model.

Add optional support for the third party setproctitle module, so
children have pretty names in ps output.

Add optional support for per-CPU multiplexers to classic runs.
5 years ago
David Wilson 6da991fae9 [stream-refactor] Py3.x test fixes 5 years ago
David Wilson 87440ec6f7 [stream-refactor] Debian Docker container image initctl 5 years ago
David Wilson 0e6de532de [stream-refactor] fix testlib assertion format string 5 years ago
David Wilson 5f7ab220cb [stream-refactor] statically link doas binary using musl
So it can run on CentOS 5
5 years ago
David Wilson 462c4ff59f [stream-refactor] stop writing to /tmp/foo in fd_check.py. 5 years ago
David Wilson cd0a557602 [stream-refactor] yet another 2.4 issue in create_child_test 5 years ago
David Wilson 54987100b2 [stream-refactor] allow up to 30 seconds to connect in unix_test
It reliably fails when running on a (intentionally) heavily loaded
machine
5 years ago
David Wilson db9066fbfb [stream-refactor] mark setns module as requiring Python >2.4 5 years ago
David Wilson 856dfcebcd [stream-refactor] another 2.4 fix for create_child_test 5 years ago
David Wilson bed5931194 [stream-refactor] remove one more getuser() usage 5 years ago
David Wilson 2f950b3bda [stream-refactor] allow doas_test to succeed on CentOS
Unlike on Debian, some environment variables that tickle
getpass.getuser() are being inherited. So use getuid() instead.

Also install the doas binary on CentOS. CI was changed (I believe) to
shrink the configuration matrix, and now these tests run on CentOS too.
5 years ago
David Wilson 2ba3973bc5 Pin idna==2.7 when running on Python<2.7. 5 years ago
David Wilson 07f3179e58 [stream-refactor] Py2.4 compat fix for iter_split_test. 5 years ago
David Wilson f0065d76d8 [stream-refactor] add descriptive task names to _container_prep 5 years ago
David Wilson bd80d4b0af [stream-refactor] avoid os.wait3() for Py2.4. 5 years ago
David Wilson 1aceacf89e [stream-refactor] replace old detach_popen() reference 5 years ago
David Wilson 300f8b2ff9 ansible: fixturize creation of MuxProcess
This relies on the previous commit resetting global variables.

Update clean_shutdown() to handle duplicate calls, due to tests
repeatedly installing it.
5 years ago
David Wilson 869e04af10 [stream-refactor] import fd_check.py used by create_child_test 5 years ago
David Wilson 1069ca43d6 [stream-refactor] port mitogen.buildah, added to master since work began 5 years ago
David Wilson f43e24e970 [stream-refactor] repaired rest of create_child_test. 5 years ago
David Wilson f039c81bb0 [stream-refactor] rename Process attrs, fix up more create_child_test 5 years ago
David Wilson cfe337b3c0 [stream-refactor] import incomplete create_child_test 5 years ago
David Wilson 9c38093aa7 issue #482: tests: check for zombie process after test. 5 years ago
David Wilson 3a1125a7bd issue #363: add test. 5 years ago
David Wilson baafc746fe tests: clean up old-style SSH exception catch 5 years ago
David Wilson 4524f03a48 issue #271: add mitogen__permdenied user to Docker image. 5 years ago
David Wilson acade4ce88 ssh: fix issue #271 regression due to refactor, add test. 5 years ago
David Wilson 8d1b01d8ef Refactor Stream, introduce quasi-asynchronous connect, much more
Split Stream into many, many classes

  * mitogen.parent.Connection: Handles connection setup logic only.
    * Maintain references to stdout and stderr streams.
    * Manages TimerList timer to cancel connection attempt after
      deadline
    * Blocking setup code replaced by async equivalents running on the
      broker

  * mitogen.parent.Options: Tracks connection-specific options. This
    keeps the connection class small, but more importantly, it is
    generic to the future desire to build and execute command lines
    without starting a full connection.

  * mitogen.core.Protocol: Handles program behaviour relating to events
    on a stream. Protocol performs no IO of its own, instead deferring
    it to Stream and Side. This makes testing much easier, and means
    libssh can reimplement Stream and Side to reuse MitogenProtocol

  * mitogen.core.MitogenProtocol: Guts of the old Mitogen stream
    implementtion

  * mitogen.core.BufferedWriter: Guts of the old Mitogen buffered
    transmit implementation, made generic

  * mitogen.core.DelineatedProtocol: Guts of the old IoLogger, knows how
    to split up input and pass it on to a
    on_line_received()/on_partial_line_received() callback.

  * mitogen.parent.BootstrapProtocol: Asynchronous equivalent of the old
    blocking connect code. Waits for various prompts (MITO001 etc) and
    writes the bootstrap using a BufferedWriter. On success, switches
    the stream to MitogenProtocol.

  * mitogen.core.Message: move encoding parts of MitogenProtocol out to
    Message (where it belongs) and write a bunch of new tests for
    pickling.

  * The bizarre Stream.construct() is gone now, Option.__init__ is its
    own constructor. Should fix many LGTM errors.

* Update all connection methods:  Every connection method is updated to
  use async logic, defining protocols as required to handle interactive
  prompts like in SSH or su. Add new real integration tests for at least
  doas and su.

* Eliminate manual fd management: File descriptors are trapped in file
  objects at their point of origin, and Side is updated to use file
  objects rather than raw descriptors. This eliminates a whole class of
  bugs where unrelated FDs could be closed by the wrong component. Now
  an FD's open/closed status is fused to it everywhere in the library.

* Halve file descriptor usage: now FD open/close state is tracked by
  its file object, we don't need to duplicate FDs everywhere so that
  receive/transmit side can be closed independently. Instead both sides
  back on to the same file object. Closes #26, Closes #470.

* Remove most uses of dup/dup2: Closes #256. File descriptors are
  trapped in a common file object and shared among classes. The
  remaining few uses for dup/dup2 are as close to minimal as possible.

* Introduce mitogen.parent.Process: uniform interface for subprocesses
  created either via mitogen.fork or the subprocess module. Remove all
  the crap where we steal a pid from subprocess guts. Now we use
  subprocess to manage its processes as it should be. Closes #169 by
  using the new Timers facility to poll for a slow-to-exit subprocess.

* Fix su password race: Closes #363. DelineatedProtocol naturally
  retries partially received lines, preventing the cause of the original
  race.

* Delete old blocking IO utility functions
  iter_read()/write_all()/discard_until().

Closes #26
Closes #147
Closes #169
Closes #256
Closes #363
Closes #419
Closes #470
5 years ago
David Wilson 37beb3a5c5 core: teach iter_split() to break on callback returning False. 5 years ago
David Wilson bf77d4ab1d testlib: have LogCapturer.raw() return unicode on 2.x. 5 years ago
David Wilson fb23ecae01 tests: jail_test fixes. 5 years ago
David Wilson 12569ad70a tests/bench: import ssh-roundtrip.py. 5 years ago
David Wilson bb0c98f451 tests: note location of related tests. 5 years ago
David Wilson eff8179591 tests: add real test for doas. 5 years ago
David Wilson d8dc5420ce tests: install OpenBSD doas port in Debian image.
To allow fancy new improved doas_test.
5 years ago
David Wilson de9a8b2a80 tests: add setns_test that works if password localhost sudo works. 5 years ago
David Wilson bbf0b22493 Import minimal jail_test. 5 years ago
David Wilson d1f5e0663d core: move message encoding to Message.pack(), add+refactor tests.
The old inline pack is still present in the old location but will be
removed in a followup commit.
5 years ago
David Wilson 3ab7998861 tests: add some UTF-8 to ssh_login_banner to encourage breakage. 5 years ago
David Wilson f6d26c5acb testlib: disable lsof warnings due to Docker crap 5 years ago
David Wilson 70ff4b674c parent: discard cancelled events in TimerList.get_timeout().
Otherwise get_timeout() keeps broker alive via keep_alive() for a
cancelled timer during shutdown.
5 years ago
David Wilson 5aca9d6c3f core: split out iter_split() for use in parent.py. 5 years ago
David Wilson 2fbc77a155 issue #170: implement timers. 5 years ago
Jordan Webb d71bdd1694
Add buildah test and stub 6 years ago
David Wilson ee7dae7514 ci: Another round of fixes for random Ansible UI breakage in 2.7/2.8 6 years ago
David Wilson ad5a80f200 Use virtualenv Python for stub connections to workaround problem
../data/stubs/stub-kubectl.py exec -it localhost -- /usr/bin/python -c "...":

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    LookupError: unknown encoding: base64

It's not clear why this is happening. "stub-kubectl.py" is executed with
the 2.7 virtualenv, while the exec() that happens inside stub-kubectl
was for "/usr/bin/python".

That second Python can't find chunks of its stdlib:

    stat("/usr/lib/python2.7/encodings/base64", 0x7ffde8744c60) = -1 ENOENT (No such file or directory)
    open("/usr/lib/python2.7/encodings/base64.so", O_RDONLY) = -1 ENOENT (No such file or directory)
    open("/usr/lib/python2.7/encodings/base64module.so", O_RDONLY) = -1 ENOENT (No such file or directory)
    open("/usr/lib/python2.7/encodings/base64.py", O_RDONLY) = -1 ENOENT (No such file or directory)
    open("/usr/lib/python2.7/encodings/base64.pyc", O_RDONLY) = -1 ENOENT (No such file or directory)
    write(2, "Traceback (most recent call last):\n", 35) = 35
    write(2, "  File \"<string>\", line 1, in <module>\n", 39) = 39
6 years ago
David Wilson 08e7fe4f80 tests: add 2.8 format async error timeout message 6 years ago
David Wilson 418fc15e80 tests: allow running without hdrhistograms library. 6 years ago
David Wilson 3620fce071 issue #593: expose configurables for SSH keepalive and increase the default 6 years ago
David Wilson e90c05dc9d issue #591: fix test for Ansible 2.3. 6 years ago
David Wilson 0b7fd3f290 issue #591: ansible: restore CWD prior to AnsibleModule initialization. 6 years ago
David Wilson fe7c3610ff issue #590: disable distro test on vanilla 6 years ago
David Wilson f1287e6e9a tests: Py3.x fix. 6 years ago
David Wilson a1f2ec222d issue #590: fix test for <2.8 Ansibles. 6 years ago
David Wilson 90105e2031 tests: Py3.x fix. 6 years ago
David Wilson e94200aeb9 issue #590: actually run Ansible test. 6 years ago
David Wilson 50cdf63c27 tests: Py3.x fix. 6 years ago
David Wilson 2f29f3e8e6 tests: rearrange test modules again, they're used in multiple places
They're back on sys.path.
6 years ago
David Wilson 2f68a5a660 module_finder_test: mask one more difference between unit2 vs. direct start 6 years ago
David Wilson cf1e7129a7 issue #590: add dummy package for new test. 6 years ago
David Wilson c1db0d3858 issue #590: move example modules to module_finder/, fix/add tests 6 years ago
David Wilson 7a5c436a39 issue #590: Ansible test for module_utils.distro use. 6 years ago
David Wilson 8f940e2ccb issue #590: teach importer to handle self-replacing modules 6 years ago
David Wilson 61b651bdd2 issue #587: workaround for ansible/ansible#56629 6 years ago
David Wilson 8a18d1862e issue #587: "state: absent" was removed in 2.8.0
ansible/cc9c72d6f845710b24e952670b534a57f6948513
6 years ago
David Wilson 92b4724010 issue #587: consistent become_exe() behaviour for older Ansibles. 6 years ago
David Wilson a25fa566a1 issue #587: update stub_connections/ test to use new doas var 6 years ago
David Wilson a3be746865 issue #587: update MODULE FAILURE message format for post >2.7 6 years ago
David Wilson 54b5fdf761 issue #587: fix syntax error due to presence of comment 6 years ago
David Wilson ce80e326fe issue #587: disable deprecation_warnings for CI.
Our use case justifies it
6 years ago
David Wilson b8e1b4df51 issue #587: 2.8 whitespace handling was improved.
b72e989e18
6 years ago
David Wilson c616359a17 Import include processing bench 6 years ago
David Wilson f30a4c05c8 issue #581: expose mitogen_mask_remote_name variable. 6 years ago
David Wilson 65deb3feac issue #575: fix exception text rendering 6 years ago
David Wilson 3ff6123483 issue #557: support correct cpu_set_t size 6 years ago
David Wilson 2bd0bbd4df issue #555: ansible: workaround ancient reload(sys) hack.
This is the most minimal change for what might be relatively minimal
edge case. Alternative is replacing reload(), but let's not do that yet.

Closes #555
6 years ago
David Wilson 30b8172573 issue #554: mitogen_action_script fix 6 years ago
David Wilson 6309774be2 issue #554: fix Ansible 2.4 compatibility 6 years ago
David Wilson f36b4b47bf issue #554: don't rely on tmp_path autoremoval in test.
Ansible doesn't do this, so we shouldn't either.
6 years ago
David Wilson 7743e57ff3 issue #554: track and remove multiple make_tmp_path() calls. 6 years ago
David Wilson 26e6194d0a issue #548: always treat transport=smart as 'ssh' for mitogen_via=.
The idea behind transport=smart is to select between paramiko and
OpenSSH given the availability of connection multiplexing and/or OSX
kernel bugs. We need to make no such choice.
6 years ago
David Wilson c413d53144 os_fork: python 3 fixes and tests. 6 years ago
David Wilson 18b984a0b4 issue #535: activate Corker on 2.4 in master too. 6 years ago
David Wilson 06e52ca89f issue #535: wire mitogen.os_fork into Broker and Pool. 6 years ago
David Wilson b3f592acee issue #535: core/select: support selecting from Latches. 6 years ago
David Wilson e517810e5a tests: ensure serialization restrictions are in effect 6 years ago
David Wilson ffdf31edd7 tests/bench: set process affinity in throughput.py. 6 years ago
David Wilson 9bcd2ec56c issue #542: return of select poller, new selection logic 6 years ago
David Wilson 0aa4c9d8fc issue #542: .ci: move some tests to Azure and enable Mac job. 6 years ago
David Wilson 5ed445c4aa issue #537: disable just the trivial LinuxPolicyTest on Travis. 6 years ago
David Wilson 8f9c67daf1 ansible: refactor affinity class and add abstract tests. 6 years ago
David Wilson d4c0250083 issue #532: PushFileService race.
There has always been a race in PushFileService since given a parent
asked to forward modules to two children via some intermediary:

    interm = router.local()
    c1 = router.local(via=interm)
    c2 = router.local(via=interm)

    service.propagate_to(c1, 'foo/bar.py')
    service.propagate_to(c2, 'foo/bar.py')

Two calls will be emitted to 'interm':

    PushFileService.store_and_forward(c1, 'foo/bar.py', [blob])
    PushFileService.store(c2, 'foo/bar.py')

Which will be processed in-order up to the point where service pool
threads in 'interm' are woken to process the message.

While it is guaranteed store_and_forward() will be processed first, no
guarantee existed that its assigned pool thread would wake and take
_lock first, thus it was possible for forward() to win the race, and for
a request to arrive to forward a file that had not been placed in local
cache yet.

Here we get rid of SerializedInvoker entirely, as it is partially to
blame for hiding the race: SerializedInvoker can only ensure no two
messages are processed simultaneously, it cannot ensure the messages are
processed in their intended order.

Instead, teach forward() that it may be called before
store_and_forward(), and if that is the case, to place the forward
request on to _waiters alongside any local threads blocked in get().
6 years ago
David Wilson 7fd0d34910 tests/ansible: Spec.port() test & mitogen_via= fix.
ansible_ssh_port was not respected.
6 years ago
David Wilson b5b23e8f3d tests/ansible: Spec.become_pass() test. 6 years ago
David Wilson 7b3ed52fd8 tests/ansible: Spec.become_user() test. 6 years ago
David Wilson 1c955a9876 ansible: capture stderr stream of async tasks. Closes #540. 6 years ago
David Wilson 8ae6ca1d5b tests/ansible: Spec.become_method() test & mitogen_via= fix.
ansible_become_method hostvar was not taken into account.
6 years ago
David Wilson 8ba75d82ec tests/ansible: Spec.become() test 6 years ago
David Wilson d1cadf8ac8 tests/ansible: Spec.password() test, document interactive pw limitation. 6 years ago
David Wilson 21ad299d7b tests/ansible: Spec.remote_user() test & mitogen_via= fix.
ansible_ssh_user precedence was incorrect.
6 years ago
David Wilson 748f5f675d tests/ansible: Spec.remote_addr() test & mitogen_via= fix.
ansible_ssh_host was not respected.
6 years ago
David Wilson 8fd641c442 tests/ansible: Spec.transport() test. 6 years ago
David Wilson e1df98168c issue #536: add mitogen_via= tests too. 6 years ago
David Wilson 2f29c76eec tests: for 2.3 compatibility, disable gcloud.py for now 6 years ago
David Wilson b3f20f54e6 issue #536: disable transport_config tests on vanilla 6 years ago
David Wilson 1d43e187e8 issue #536: connection_delegation/ tests were erroneously broken
While fixing delegate_to, this un-hardwiring of /usr/bin/python
happened. It was always incorrect.
6 years ago
David Wilson 0dfcf5560b tests: define MITOGEN_INVENTORY_FILE even if -i unspecified.
To fix running tests locally.
6 years ago
David Wilson 01faed708d issue #536: add tests for each ansible_python_interpreter case. 6 years ago
David Wilson 86f9572ef7 issue #536: stop defining explicit localhost in inventory.
This was needed at some point in the past, but the tests don't seem to
care about it any more. We'll fix any CI breakage by changing the tests,
since verifying implicit localhost behaviour is important.
6 years ago
David Wilson 81e386586f tests: allow running Ansible tests locally without -udmw again. 6 years ago
David Wilson 4506c528ca issue #481: add test. 6 years ago
David Wilson b263e01867 issue #481: avoid crash if disconnect occurs during forward_modules() 6 years ago
David Wilson 2aed0a0c6c tests: add new compression parameter to mitogen_get_stack results 6 years ago
David Wilson 6859186069 tests: disable affinity_test on Travis :/ 6 years ago
David Wilson 19ae7b7a84 issue #508: fix responder stats test due to new smaller parent.py. 6 years ago
David Wilson 953a910881 issue #508: tests: skip minify_test Py2.4/2.5 for profiler.py. 6 years ago
David Wilson 263f530496 tests: fix fallout from 36fb318adf5c56e729296c3efce84f4dd75ced4e 6 years ago
David Wilson 8026e54b9b tests: move affinity_test to Ansible tests. 6 years ago
David Wilson e703522010 issue #505: docs: add new detail graph for one scenario. 6 years ago
David Wilson b2bae9deed tests: pad out localhost-* 6 years ago
David Wilson fda7d5c276 bench: tidy up and cpu-pin some more files. 6 years ago
David Wilson 29adfbea59 tests: add localhost-x100 6 years ago
David Wilson 0e193c223c issue #508: master: minify all Mitogen/ansible_mitogen sources.
Minify-safe files are marked with a magical "# !mitogen: minify_safe"
comment anywhere in the file, which activates the minifier. The result
is naturally cached by ModuleResponder, therefore lru_cache is gone too.

Given:

    import os, mitogen
    @mitogen.main()
    def main(router):
        c = router.ssh(hostname='k3')
        c.call(os.getpid)
        router.sudo(via=c)

SSH footprint drops from 56.2 KiB to 42.75 KiB (-23.9%)
Ansible "shell: hostname" drops 149.26 KiB to 117.42 KiB (-21.3%)
6 years ago
David Wilson b72d139bd2 tests: make soak testing work reliably on vanilla.
It claims to disable host key checking, but it doesn't.
6 years ago
David Wilson b59a47aecb tests: make mitogen_shutdown_all be run_once by default. 6 years ago
David Wilson e77048ec2d utils: pad out reset_affinity() and integrate with detach_popen() 6 years ago
David Wilson 2209d880e4 ansible/bench: make end= configurable. 6 years ago
David Wilson 954f874085 issue #527: catch new-style module tracebacks like vanilla. 6 years ago
David Wilson 60fe3fd6f5 issue #429: enable en_US locale to unbreak debops test. 6 years ago
David Wilson 7531af3ee0 issue #499: fix another mind-numbingly stupid vanilla inconsistency 6 years ago
David Wilson a40946297f issue #497: do our best to cope with crap upstream code 6 years ago
David Wilson 2fdbd0cfcd ssh: fix test to match updated log format. 6 years ago
David Wilson 6e9f8e829e issue #429: teach sudo about every know i18n password string. 6 years ago
David Wilson 960e505f07 issue #429: install i18n-related bits in test images. 6 years ago
David Wilson eb93f82d05 tests: ensure file is closed in connection_test. 6 years ago
David Wilson 73979043ad gcloud: small updates 6 years ago
David Wilson 59068ca955 tests: give ansible/gcloud/ its own requirements file. 6 years ago
David Wilson 9df314f9c5 issue #499: another totally moronic implementation difference 6 years ago
David Wilson 53794469a0 issue #499: disable new test on vanilla. 6 years ago
David Wilson a1121c5a84 issue #499: respect C.BECOME_ALLOW_SAME_USER. 6 years ago
David Wilson b254051416 ansible: add test to ensure UNIX socket is cleaned on exit; closes #488. 6 years ago
David Wilson 628e8f4466 Bump PyYaML to a 3.7-compatible version. 6 years ago
David Wilson 3435f24e8d issue #479: ModuleFinder special case for __main__ on Py3.x. 6 years ago
David Wilson a7da1b048b tests: just disable the test. 6 years ago
David Wilson cdb1434809 tests: hopefully fix this dumb test for the final time 6 years ago
David Wilson a67cc85bdb issue #477: use MITOGEN_INVENTORY_FILE everywhere. 6 years ago
David Wilson ea23d7fd54 issue #477: hacksmash weird 2.3 inventory_file var issue. 6 years ago
David Wilson 9aff8edf50 issue #477: one more conditional test. 6 years ago
David Wilson e133b8dbb1 issue #477: some more conditional tests. 6 years ago
David Wilson 599da0689a issue #477 / ansible: avoid a race in async job startup.
Ansible 2.3/Python 2.4 work revealed there is no guarantee a slow target
will have written the initial job status file out before a fast
controller makes an initial check for it. Therefore, provide AsyncRunner
with a sender it should send a message to when the initial job file has
been written.

As a bonus, also catch and report exceptions happening early in
AsyncRunner, rather than leaving them to end up in -vvv output.
6 years ago
David Wilson c61498202e issue #477: use assert_equal for nicer debug. 6 years ago
David Wilson 5e4066bb64 issue #477: add Connection.homedir test. 6 years ago
David Wilson 8385c38cff issue #477: paper over Ansible 2.3 flag handling difference 6 years ago
David Wilson b67e4e118e issue #477: update forking_correct_parent for subprocess isolation 6 years ago
David Wilson abb0319bbb issue #477: get rid of perl JSON module requirement. 6 years ago
David Wilson 7d1b0e0807 issue #477: Ansible 2.3 did not support gather_facts min subset. 6 years ago
David Wilson b0ec398755 issue #477: CentOS 5 image requires perl installed too. 6 years ago
David Wilson 0a9c3dcc40 issue #477: missing stub-su.py from 137f5fa6c5 6 years ago
David Wilson 2f087859bc issue #477: 2.4-compatible syntax. 6 years ago
David Wilson 7abe5ff564 issue #477: clearing glibc caches is not possible on Py2.4. 6 years ago
David Wilson 9377fed96b issue #477: install simplejson for vanilla tests. 6 years ago
David Wilson 137f5fa6c5 issue #477: add basic su_test and Py2.4 polyfill. 6 years ago
David Wilson d9efeb950a issue #477: import updated Python build scripts
Now using Docker to get correct libc.
6 years ago
David Wilson ba59b80bea issue #477: initial Python 2.4.6 build for CI. 6 years ago
David Wilson 217e65dcbc issue #477: enable git-lfs for tests/data/*.tar.bz2. 6 years ago
David Wilson ffb1b842db issue #477: import build script for Python 2.4.6. 6 years ago
David Wilson 767331d76f issue #477: disable Django parts of module_finder_test on 2.4. 6 years ago
David Wilson b6634194af issue #477: clean up globals after unix_test. 6 years ago
David Wilson 30d933b299 issue #477: remove unused pytest bits from importer_test. 6 years ago
David Wilson 69c27a33fc issue #477: remove fork use from unix_test. 6 years ago
David Wilson fedd294f96 issue #485: import new throuhgput bench 6 years ago
David Wilson 6281f231cb issue #477: more fork removal 6 years ago
David Wilson 25bffc424d issue #477: util/fakessh/two_three_compat fixes. 6 years ago
David Wilson 61297c24e5 issue #477: call_function_test fixes for 2.4. 6 years ago
David Wilson dd30a907ce issue #477: promote setup_gil() to mitogen.utils
This is since ansible_mitogen/process.py is 2.6-only, and I want to use
setup_gil() in 2.4 code.
6 years ago
David Wilson cef7c1ccc7 issue #477: fix lxc_test any polyfill import. 6 years ago
David Wilson f420d876f2 issue #477: stop using fork in responder_test. 6 years ago
David Wilson 9d5b3d9e02 issue #477: stop using fork in service_test. 6 years ago
David Wilson 101e1c17b1 issue #477: stop using fork() in parent_test, compatible enumerate(). 6 years ago
David Wilson 9f10695ee2 issue #477: Popen.terminate() polyfill for Py2.4. 6 years ago
David Wilson b7f39f26cd issue #477: stop using .fork() in router_test, one small 2.4 fix. 6 years ago
David Wilson 9248b6d2b6 issue #477: old Py zlib did not include extended exception text. 6 years ago
David Wilson b831b4ab31 issue #477: stop using router.fork() in receiver_test 6 years ago
David Wilson a33767d31e issue #477: any() polyfill for lxc_test. 6 years ago
David Wilson 3b4f184298 issue #477: replace type(e) -> __class__ for an exception
Due to old style exception classes.
6 years ago
David Wilson b985224ff6 issue #477: old Mock does not throw side_effect exceptions from a list 6 years ago
David Wilson 6a2f88d6a3 issue #477: make mitogen.fork unsupported on Py<2.6. 6 years ago
David Wilson 51294db52d issue #477: fix 2 runner tests on Ansible 2.7. 6 years ago
David Wilson 01db198122 issue #477: fix new KwargsTest on Python 3.x. 6 years ago
David Wilson d4897d37bb issue #477: ModuleFinder now returns Unicode module names. 6 years ago
David Wilson c443244986 issue #477: Ansible 2.3 compatible regression/all.yml. 6 years ago
David Wilson 532ae92029 issue #477: Ansible 2.3 requires placeholder module for assert_equals 6 years ago
David Wilson 572db1a385 issue #477: build a CentOS 5/Py2.4 container + playbook compat fixes. 6 years ago
David Wilson 0b0ae5c971 issue #477: backport various test modules to Python 2.4. 6 years ago
David Wilson 462a8567e5 issue #477: add all() polyfill to custom_python_detect_environmnet 6 years ago
David Wilson e9706a4a09 issue #477: _update_linecache() must append newlines. 6 years ago
David Wilson 19b708e141 issue #415, #477: Poller must handle POLLHUP too.
Linux will fire poll() with simply the POLLHUP bit set even though it
was not requested, resulting in an infinite loop.
6 years ago
David Wilson 97a96f5dd8 issue #477: rename and add tests for polyfill functions. 6 years ago
David Wilson dc01f9e47e issue #477: Ansible 2.3 module output format difference.
Don't test for keys it doesn't set.
6 years ago
David Wilson b6062afe24 issue #477: Ansible 2.3 cannot use when: on an include.
So just use our magic action module to reset the connection instead.
6 years ago
David Wilson c1a4597e32 issue #477: tests: use Ansible 2.3-compatible include rather than import
This will break with Ansible 2.8. Probably going to end up with a regex
hack for CI.
6 years ago
David Wilson 83aff538b3 issue #477: testlib: Py2.4 did not have BaseException. 6 years ago
David Wilson be8562ce09 issue #477: Ansible 2.3 had stricter arg spec format. 6 years ago
David Wilson 0ee8ee78b8 issue #477: Py2.4 cannot tolerate unicode kwargs. 6 years ago
David Wilson 4388e794ce issue #477: Py2.4: enumerate() may return stopped threads. 6 years ago
David Wilson 07401d767a issue #477: Python 2.4 type(exc) returns old-style instance. 6 years ago
David Wilson 2ead451277 issue #477: Python 2.4 lacked str.partition. 6 years ago
David Wilson 2ec19f2374 issue #477: Python 2.4 lacked Thread.name. 6 years ago
David Wilson 2f016b5c7b issue #477: Python 2.4 lacked context managers. 6 years ago
David Wilson 66b64cae2c issue #477: older Ansibles had no vars plugin base class. 6 years ago
David Wilson d6945443b7 tests: add exact test for issue 251; closes #251. 6 years ago
David Wilson 23866084d7 issue #412: promote "mitogen_get_stack" to the main extension.
This is to make it easier for users to diagnose their own problems.
6 years ago
David Wilson 2ad05f1238 issue #251, #412, #434: fix connection configuration brainwrong
This refactors connection.py to pull the two huge dict-building
functions out into new transport_transport_config.PlayContextSpec and
MitogenViaSpec classes, leaving a lot more room to breath in both files
to figure out exactly how connection configuration should work.

The changes made in 1f21a30 / 3d58832 are updated or completely removed,
the original change was misguided, in a bid to fix connection delegation
taking variables from the wrong place when delegate_to was active.

The Python path no longer defaults to '/usr/bin/python', this does not
appear to be Ansible's normal behaviour. This has changed several times,
so it may have to change again, and it may cause breakage after release.

Connection delegation respects the c.DEFAULT_REMOTE_USER whereas the
previous version simply tried to fetch whatever was in the
'ansible_user' hostvar. Many more connection delegation variables closer
match vanilla's handling, but this still requires more work. Some of the
variables need access to the command line, and upstream are in the
process of changing all that stuff around.
6 years ago
David Wilson 8414ff21ca issue #434: tests: set a default remote_user in ansible.cfg. 6 years ago
David Wilson 1b8748a8d9 tests: use assert_equal in more places. 6 years ago
David Wilson d72567b15b tests: make assert_equal work on newer Ansibles. 6 years ago
David Wilson 8891b48080 tests: convert stack_construction.yml to assert_equal. 6 years ago
David Wilson 4256d2aa4b tests: make fork_histogram optional 6 years ago
David Wilson 17eff064b0 tests: use assert_equal in delegate_to_template.yml. 6 years ago
David Wilson 91c9aff9ff tests: import assert_equal action. 6 years ago
David Wilson 630c058a89 tests: rename 'delegation/' to 'connection_delegation/' 6 years ago
David Wilson f5d9af80ef tests: some more utility function tests + flake8. 6 years ago
David Wilson 0ba8cc7b61 tests: clean up / deduplicate Ansible inventory. 6 years ago
David Wilson 1555f23215 tests: add some more helper function tests. 6 years ago
David Wilson 9d9add3339 issue #414: reenable test. 6 years ago
David Wilson fcc403cc2f core: make Receiver.to_sender() use Router.myself(). 6 years ago
David Wilson 4bdf60326c issue #424: ansible: make put_file() raise AnsibleFileNotFound 6 years ago
David Wilson 06415bb720 issue #310: fix test failures, teach old import method new tricks
- don't try anything unless something really lives in sys.modules by
  that name
- non-ASCII files are possible
- the unimportable thing might be an extension module, we don't want
  that
6 years ago
David Wilson 6af1a64cce master: handle crazy non-modules in sys.modules again; closes #310. 6 years ago
David Wilson 7c33979e25 issue #415: add IPC latency bench. 6 years ago
David Wilson a8c056c27b issue #436: decode debug log lines on Python 3. 6 years ago
David Wilson 8380baf41b issue #461: Ansible 2.3 placeholder modules for action plug-ins. 6 years ago
David Wilson 822978520f issue #446: update Receiver.__iter__ to match
iter() previously relied on the fake dead message being enqueued.
6 years ago
David Wilson 386d869d5f tests: fix responder_test after removing unused imports 6 years ago
David Wilson 6592598999 tests: stray socket file left during unix_test. 6 years ago
David Wilson cce1dbf3b1 tests: quieten a bunch of spam printed during run 6 years ago
David Wilson b0e7c1a315 tests: fix /etc/environment test on vanilla
Use "meta: reset_connection" to ensure /etc/environment is reloaded.
Looks like this entire feature can be ripped out!
6 years ago
David Wilson 26fdbf70e0 tests: good_load_module_size check can't be exact
Since message contains pathnames that vary across machines
6 years ago
David Wilson ae03d5f4f1 issue #446: basic tests for Receiver._on_receive(). 6 years ago
David Wilson bcd9827c3b core: Latch.empty() improvements
- throw LatchError if the latch is closed.
- wrap with the lock to avoid unexpected weirdness.
6 years ago
David Wilson 388649df97 core: Receiver.close() now wakes all threads; closes #446. 6 years ago
David Wilson f2f41809ae issue #459: initial get_stats() implementation 6 years ago
David Wilson 1d97493fcd tests: fallout from #447. 6 years ago
David Wilson de719fa249 core: throw error on duplicate add_handler(); closes #447. 6 years ago
David Wilson dc92e529bc service: unregister receiver at shutdown; closes #445. 6 years ago
David Wilson 85e965118d Merge remote-tracking branch 'origin/issue462'
* origin/issue462:
  issue #462: docs: update Changelog.
  parent: cope with broken /dev/pts on Linux; closes #462.
6 years ago
David Wilson 9da6e6a9a8 tests: don't call Router.shutdown() twice.
Due to 497234e7 / issue #456.
6 years ago
David Wilson b2b7e7b06b tests: file_service_test fixes
Due to 6713b90acc
6 years ago
David Wilson a8921bb6ff tests: fix scaling in fork_histogram 6 years ago
David Wilson 6936b93b43 tests: import fork_histogram.py. 6 years ago
David Wilson 5a96d13fd8 issue #426: fix all.yml sorting, one more delegate_to 6 years ago
David Wilson bd82fa1986 issue #426: fix low_level_execute_command.yml breakage. 6 years ago
David Wilson a6e6bc4c71 issue #426: to_text filter. 6 years ago
David Wilson 9d87f03ab4 issue #426: disable Ansible smart transport. 6 years ago
David Wilson 49d37bfa2d issue #426: remove LANG and LC_ALL during tests. 6 years ago
David Wilson a6e6fd14be issue #426: more 2->3 test fixes. 6 years ago
David Wilson 67f710fd9b issue #426: use delegate_to in fixup_perms2 and copy.yml 6 years ago
David Wilson a4c7a98dd9 parent: cope with broken /dev/pts on Linux; closes #462. 6 years ago
David Wilson 3179951f5c issue #454: fix AttributeError and atexit.yml test. 6 years ago
David Wilson 1eb08fb5c5 core: docstring tidyups 6 years ago
David Wilson 497234e782 issue #456: core: raise error during defer() if Broker shutdown 6 years ago
David Wilson f20e0bbac1 service: Allow registering path prefixes with FileService.
e.g. service.register_prefix('/') disables all security checks.
6 years ago
David Wilson 6fafc0a631 select: Python 2.x required __nonzero__ for bool(Select) 6 years ago
David Wilson 300cb41e2e core: detect stream corruption. Closes #438. 6 years ago
David Wilson 3876590aa1 parent: add descriptive errors for unsupported call() types.
Closes #439.
6 years ago
David Wilson 22de7f0e72 issue #436: fix string parsing of mitogen_ssh_debug_level
It can be a string when specified on the command line.
6 years ago
David Wilson 44d6ca771a issue #426: fix local/delegate_to issue 6 years ago
David Wilson 934d8ac139 issue #426: fix 2->3 issue in plain_old_module. 6 years ago
David Wilson 3206d59c87 issue #426: teach DockerMixin to allow selecting interpreter 6 years ago
David Wilson f1661abe4e tests: make IterReadTest a little more robust 6 years ago
David Wilson bef4b0c962 tests: fix copy.yml title 6 years ago
David Wilson 2a6dbb038f Pythonize, split out and parallelize all jobs. 6 years ago
David Wilson b60a6d0f3b Split dev_requirements.txt up according to test mode. 6 years ago
David Wilson fccf424140 tests: Pythonize print_env.sh. 6 years ago
David Wilson 8f1f3de123 tests: Pythonize env_wrapper.sh, clean up local_test 6 years ago
David Wilson a717c5406c tests: split etc_environment test in two
Turns out nobody supports ~/.pam_environment any more. Keep the
behaviour around for the time being.
6 years ago
David Wilson 4b61e5af02 tests: run FD and thread checks on every test case.
Trying to hunt down weirdness on Azure.
6 years ago
David Wilson 374fd72dbb tests: disable mtime test on vanilla 6 years ago
David Wilson 79ca67aadd tests: disable connection tests for non-Mitogen 6 years ago
David Wilson 8972dbb7b9 tests: more Ansible fixes. 6 years ago
David Wilson acf0b04876 tests: run some playbooks against only one target. 6 years ago
David Wilson ee2d10375d tests: don't run reset_connection tests on <2.5.6. 6 years ago
David Wilson fcdfd5f107 tests: fix disconnect_cleanup.yml target count assumption 6 years ago
David Wilson 1bb239189b tests: another attempt at working paths. 6 years ago
David Wilson 5f815ec6c4 tests: try to fix PATH problem on Travis. 6 years ago
David Wilson f5f72b958f tests: avoid -u command line parameter conflict 6 years ago
David Wilson 0c3e48468b tests: run disconnect_during_module.yml in subprocess
Avoid entire run failing with unreachable
6 years ago
David Wilson 35092c5d35 tests: Unicode/bytes fixes for integration/connection/exec_command.yml 6 years ago
David Wilson 4d443e654b tests: replace another shell script. 6 years ago
David Wilson 9ad022107e issue #414: disable test until rest of CI is healthy 6 years ago
David Wilson b29c8eaf2a tests: allow passing -vvv to build_docker_images. 6 years ago
David Wilson f87553b165 tests: must set ansible_become_pass in synchronize.yml. 6 years ago
David Wilson 816da64df5 tests: show task args in image_prep 6 years ago
David Wilson 174b685d16 tests: CentOS 6 lacks %wheel in sudo by default. 6 years ago
David Wilson 574fc27a9c issue #414: import test / reproduction. 6 years ago
David Wilson c286f4f107 Add tests/ansible/requirements.txt to Tox. 6 years ago
David Wilson cd6486b0e9 tests: fix more DisconnectTest raciness. 6 years ago
David Wilson e180d310b5 tests: fix fork_test compat on 3.x. 6 years ago
David Wilson 3f46c9569c tests: 3.x syntax compat for tests/data/stubs/ 6 years ago
David Wilson 27a4001f4f tests: handle NameError when faulthandler is not installed. 6 years ago
David Wilson 5482b4d528 tests: poller_test 3.x fix. 6 years ago
David Wilson 3836c6a220 tests/bench: run roundtrip.py a ton more to reduce variance 6 years ago
David Wilson d1b7c232bf tests: image_prep needs sudo 6 years ago
David Wilson cf97932fad core: dead messages have optional body, use it everywhere; closes #387. 6 years ago
David Wilson 802de6a8d5 issue #406: clean up DiagLogStream handling and connect() failure.
When Stream.connect() fails, have it just use on_disconnect(). Now there
is a single disconnect cleanup path.

Remove cutpasted DiagLogStream setup/destruction, and move it into the
base class (temporarily), and only manage the lifetime of its underlying
FD via Side.close().  This cures another EBADF failure.
6 years ago
David Wilson e01c8f2891 issue #406: 3.x syntax fixes. 6 years ago
David Wilson 003526ef7b issue #406: fix thread leaks in unix_test too. 6 years ago
David Wilson eae1bdba4e tests: make minify_test print something useful on failure 6 years ago
David Wilson 802efa6ea6 issue #406: ensure broker_test waits for broker exit. 6 years ago
David Wilson 175fc377d2 tests: remove hard-wired SSL paths from fork_test. 6 years ago
David Wilson 3da4b1a420 tests: verify only main/watcher threads exist at teardown 6 years ago
David Wilson b0dd628f07 issue #406: parent_test fixes, NameError in log_fd_calls(). 6 years ago
David Wilson 6ff1e001da issue #406: log socketpair calls too. 6 years ago
David Wilson 70c550f50c issue #406: close stdout pipes in parent_test 6 years ago
David Wilson 9b3cb55a8b issue #4096: import log_fd_calls() helper. 6 years ago
David Wilson 8a0b343760 issue #406: test for FD leak after every TestCase 6 years ago
David Wilson e9a6e4c3d2 issue #406: add test. 6 years ago
David Wilson 5eff8ea4fb tests: make result_shell_echo_hi compare less of the timedelta; closes #361
Assuming less than one second is too much to ask from Travis.
6 years ago
David Wilson 5be9a55bf4 core: allow Context to be pickled by non-Mitogen pickler. 6 years ago
David Wilson c148c869e6 issue #76, #370: add disconnect cleanup test 6 years ago
David Wilson aeec2b5054 tests: pad out minify_test to verify all internal modules 6 years ago
David Wilson f2d288bb1e tests: ensure minify() result can be compiled for all of core. 6 years ago
David Wilson fd326f5ad7 tests: stub tests for doas/mitogen_doas 6 years ago
David Wilson cbd4129cb9 tests: fix paramiko_unblemished.yml 6 years ago
David Wilson e8fc9e490f tests: update osa_delegate_to_self to match connection parameters 6 years ago
David Wilson 6c71c5bfef issue #369: disable reset_connection on Ansible<2.5.6
https://github.com/ansible/ansible/issues/27520
6 years ago
David Wilson 8ed72e7e7b issue #369: avoid Ansible 2.5 bug (cond_reset_warn missing method) 6 years ago
David Wilson ab4ccc6b92 issue #369: don't mass-kill all SSH clients in reconnection.yml
It breaks my new development environment :)
6 years ago
David Wilson fcdf4a0f35 Import missing stub-sudo.py. 6 years ago