Commit Graph

725 Commits (bbf0b22493c80af08f1220c68cc3785159815f8b)

Author SHA1 Message Date
David Wilson ffdf31edd7 tests/bench: set process affinity in throughput.py. 6 years ago
David Wilson 9bcd2ec56c issue #542: return of select poller, new selection logic 6 years ago
David Wilson 0aa4c9d8fc issue #542: .ci: move some tests to Azure and enable Mac job. 6 years ago
David Wilson 5ed445c4aa issue #537: disable just the trivial LinuxPolicyTest on Travis. 6 years ago
David Wilson 8f9c67daf1 ansible: refactor affinity class and add abstract tests. 6 years ago
David Wilson d4c0250083 issue #532: PushFileService race.
There has always been a race in PushFileService since given a parent
asked to forward modules to two children via some intermediary:

    interm = router.local()
    c1 = router.local(via=interm)
    c2 = router.local(via=interm)

    service.propagate_to(c1, 'foo/bar.py')
    service.propagate_to(c2, 'foo/bar.py')

Two calls will be emitted to 'interm':

    PushFileService.store_and_forward(c1, 'foo/bar.py', [blob])
    PushFileService.store(c2, 'foo/bar.py')

Which will be processed in-order up to the point where service pool
threads in 'interm' are woken to process the message.

While it is guaranteed store_and_forward() will be processed first, no
guarantee existed that its assigned pool thread would wake and take
_lock first, thus it was possible for forward() to win the race, and for
a request to arrive to forward a file that had not been placed in local
cache yet.

Here we get rid of SerializedInvoker entirely, as it is partially to
blame for hiding the race: SerializedInvoker can only ensure no two
messages are processed simultaneously, it cannot ensure the messages are
processed in their intended order.

Instead, teach forward() that it may be called before
store_and_forward(), and if that is the case, to place the forward
request on to _waiters alongside any local threads blocked in get().
6 years ago
David Wilson 7fd0d34910 tests/ansible: Spec.port() test & mitogen_via= fix.
ansible_ssh_port was not respected.
6 years ago
David Wilson b5b23e8f3d tests/ansible: Spec.become_pass() test. 6 years ago
David Wilson 7b3ed52fd8 tests/ansible: Spec.become_user() test. 6 years ago
David Wilson 1c955a9876 ansible: capture stderr stream of async tasks. Closes #540. 6 years ago
David Wilson 8ae6ca1d5b tests/ansible: Spec.become_method() test & mitogen_via= fix.
ansible_become_method hostvar was not taken into account.
6 years ago
David Wilson 8ba75d82ec tests/ansible: Spec.become() test 6 years ago
David Wilson d1cadf8ac8 tests/ansible: Spec.password() test, document interactive pw limitation. 6 years ago
David Wilson 21ad299d7b tests/ansible: Spec.remote_user() test & mitogen_via= fix.
ansible_ssh_user precedence was incorrect.
6 years ago
David Wilson 748f5f675d tests/ansible: Spec.remote_addr() test & mitogen_via= fix.
ansible_ssh_host was not respected.
6 years ago
David Wilson 8fd641c442 tests/ansible: Spec.transport() test. 6 years ago
David Wilson e1df98168c issue #536: add mitogen_via= tests too. 6 years ago
David Wilson 2f29c76eec tests: for 2.3 compatibility, disable gcloud.py for now 6 years ago
David Wilson b3f20f54e6 issue #536: disable transport_config tests on vanilla 6 years ago
David Wilson 1d43e187e8 issue #536: connection_delegation/ tests were erroneously broken
While fixing delegate_to, this un-hardwiring of /usr/bin/python
happened. It was always incorrect.
6 years ago
David Wilson 0dfcf5560b tests: define MITOGEN_INVENTORY_FILE even if -i unspecified.
To fix running tests locally.
6 years ago
David Wilson 01faed708d issue #536: add tests for each ansible_python_interpreter case. 6 years ago
David Wilson 86f9572ef7 issue #536: stop defining explicit localhost in inventory.
This was needed at some point in the past, but the tests don't seem to
care about it any more. We'll fix any CI breakage by changing the tests,
since verifying implicit localhost behaviour is important.
6 years ago
David Wilson 81e386586f tests: allow running Ansible tests locally without -udmw again. 6 years ago
David Wilson 4506c528ca issue #481: add test. 6 years ago
David Wilson b263e01867 issue #481: avoid crash if disconnect occurs during forward_modules() 6 years ago
David Wilson 2aed0a0c6c tests: add new compression parameter to mitogen_get_stack results 6 years ago
David Wilson 6859186069 tests: disable affinity_test on Travis :/ 6 years ago
David Wilson 19ae7b7a84 issue #508: fix responder stats test due to new smaller parent.py. 6 years ago
David Wilson 953a910881 issue #508: tests: skip minify_test Py2.4/2.5 for profiler.py. 6 years ago
David Wilson 263f530496 tests: fix fallout from 36fb318adf5c56e729296c3efce84f4dd75ced4e 6 years ago
David Wilson 8026e54b9b tests: move affinity_test to Ansible tests. 6 years ago
David Wilson e703522010 issue #505: docs: add new detail graph for one scenario. 6 years ago
David Wilson b2bae9deed tests: pad out localhost-* 6 years ago
David Wilson fda7d5c276 bench: tidy up and cpu-pin some more files. 6 years ago
David Wilson 29adfbea59 tests: add localhost-x100 6 years ago
David Wilson 0e193c223c issue #508: master: minify all Mitogen/ansible_mitogen sources.
Minify-safe files are marked with a magical "# !mitogen: minify_safe"
comment anywhere in the file, which activates the minifier. The result
is naturally cached by ModuleResponder, therefore lru_cache is gone too.

Given:

    import os, mitogen
    @mitogen.main()
    def main(router):
        c = router.ssh(hostname='k3')
        c.call(os.getpid)
        router.sudo(via=c)

SSH footprint drops from 56.2 KiB to 42.75 KiB (-23.9%)
Ansible "shell: hostname" drops 149.26 KiB to 117.42 KiB (-21.3%)
6 years ago
David Wilson b72d139bd2 tests: make soak testing work reliably on vanilla.
It claims to disable host key checking, but it doesn't.
6 years ago
David Wilson b59a47aecb tests: make mitogen_shutdown_all be run_once by default. 6 years ago
David Wilson e77048ec2d utils: pad out reset_affinity() and integrate with detach_popen() 6 years ago
David Wilson 2209d880e4 ansible/bench: make end= configurable. 6 years ago
David Wilson 954f874085 issue #527: catch new-style module tracebacks like vanilla. 6 years ago
David Wilson 60fe3fd6f5 issue #429: enable en_US locale to unbreak debops test. 6 years ago
David Wilson 7531af3ee0 issue #499: fix another mind-numbingly stupid vanilla inconsistency 6 years ago
David Wilson a40946297f issue #497: do our best to cope with crap upstream code 6 years ago
David Wilson 2fdbd0cfcd ssh: fix test to match updated log format. 6 years ago
David Wilson 6e9f8e829e issue #429: teach sudo about every know i18n password string. 6 years ago
David Wilson 960e505f07 issue #429: install i18n-related bits in test images. 6 years ago
David Wilson eb93f82d05 tests: ensure file is closed in connection_test. 6 years ago
David Wilson 73979043ad gcloud: small updates 6 years ago
David Wilson 59068ca955 tests: give ansible/gcloud/ its own requirements file. 6 years ago
David Wilson 9df314f9c5 issue #499: another totally moronic implementation difference 6 years ago
David Wilson 53794469a0 issue #499: disable new test on vanilla. 6 years ago
David Wilson a1121c5a84 issue #499: respect C.BECOME_ALLOW_SAME_USER. 6 years ago
David Wilson b254051416 ansible: add test to ensure UNIX socket is cleaned on exit; closes #488. 6 years ago
David Wilson 628e8f4466 Bump PyYaML to a 3.7-compatible version. 6 years ago
David Wilson 3435f24e8d issue #479: ModuleFinder special case for __main__ on Py3.x. 6 years ago
David Wilson a7da1b048b tests: just disable the test. 6 years ago
David Wilson cdb1434809 tests: hopefully fix this dumb test for the final time 6 years ago
David Wilson a67cc85bdb issue #477: use MITOGEN_INVENTORY_FILE everywhere. 6 years ago
David Wilson ea23d7fd54 issue #477: hacksmash weird 2.3 inventory_file var issue. 6 years ago
David Wilson 9aff8edf50 issue #477: one more conditional test. 6 years ago
David Wilson e133b8dbb1 issue #477: some more conditional tests. 6 years ago
David Wilson 599da0689a issue #477 / ansible: avoid a race in async job startup.
Ansible 2.3/Python 2.4 work revealed there is no guarantee a slow target
will have written the initial job status file out before a fast
controller makes an initial check for it. Therefore, provide AsyncRunner
with a sender it should send a message to when the initial job file has
been written.

As a bonus, also catch and report exceptions happening early in
AsyncRunner, rather than leaving them to end up in -vvv output.
6 years ago
David Wilson c61498202e issue #477: use assert_equal for nicer debug. 6 years ago
David Wilson 5e4066bb64 issue #477: add Connection.homedir test. 6 years ago
David Wilson 8385c38cff issue #477: paper over Ansible 2.3 flag handling difference 6 years ago
David Wilson b67e4e118e issue #477: update forking_correct_parent for subprocess isolation 6 years ago
David Wilson abb0319bbb issue #477: get rid of perl JSON module requirement. 6 years ago
David Wilson 7d1b0e0807 issue #477: Ansible 2.3 did not support gather_facts min subset. 6 years ago
David Wilson b0ec398755 issue #477: CentOS 5 image requires perl installed too. 6 years ago
David Wilson 0a9c3dcc40 issue #477: missing stub-su.py from 137f5fa6c5 6 years ago
David Wilson 2f087859bc issue #477: 2.4-compatible syntax. 6 years ago
David Wilson 7abe5ff564 issue #477: clearing glibc caches is not possible on Py2.4. 6 years ago
David Wilson 9377fed96b issue #477: install simplejson for vanilla tests. 6 years ago
David Wilson 137f5fa6c5 issue #477: add basic su_test and Py2.4 polyfill. 6 years ago
David Wilson d9efeb950a issue #477: import updated Python build scripts
Now using Docker to get correct libc.
6 years ago
David Wilson ba59b80bea issue #477: initial Python 2.4.6 build for CI. 6 years ago
David Wilson 217e65dcbc issue #477: enable git-lfs for tests/data/*.tar.bz2. 6 years ago
David Wilson ffb1b842db issue #477: import build script for Python 2.4.6. 6 years ago
David Wilson 767331d76f issue #477: disable Django parts of module_finder_test on 2.4. 6 years ago
David Wilson b6634194af issue #477: clean up globals after unix_test. 6 years ago
David Wilson 30d933b299 issue #477: remove unused pytest bits from importer_test. 6 years ago
David Wilson 69c27a33fc issue #477: remove fork use from unix_test. 6 years ago
David Wilson fedd294f96 issue #485: import new throuhgput bench 6 years ago
David Wilson 6281f231cb issue #477: more fork removal 6 years ago
David Wilson 25bffc424d issue #477: util/fakessh/two_three_compat fixes. 6 years ago
David Wilson 61297c24e5 issue #477: call_function_test fixes for 2.4. 6 years ago
David Wilson dd30a907ce issue #477: promote setup_gil() to mitogen.utils
This is since ansible_mitogen/process.py is 2.6-only, and I want to use
setup_gil() in 2.4 code.
6 years ago
David Wilson cef7c1ccc7 issue #477: fix lxc_test any polyfill import. 6 years ago
David Wilson f420d876f2 issue #477: stop using fork in responder_test. 6 years ago
David Wilson 9d5b3d9e02 issue #477: stop using fork in service_test. 6 years ago
David Wilson 101e1c17b1 issue #477: stop using fork() in parent_test, compatible enumerate(). 6 years ago
David Wilson 9f10695ee2 issue #477: Popen.terminate() polyfill for Py2.4. 6 years ago
David Wilson b7f39f26cd issue #477: stop using .fork() in router_test, one small 2.4 fix. 6 years ago
David Wilson 9248b6d2b6 issue #477: old Py zlib did not include extended exception text. 6 years ago
David Wilson b831b4ab31 issue #477: stop using router.fork() in receiver_test 6 years ago
David Wilson a33767d31e issue #477: any() polyfill for lxc_test. 6 years ago
David Wilson 3b4f184298 issue #477: replace type(e) -> __class__ for an exception
Due to old style exception classes.
6 years ago
David Wilson b985224ff6 issue #477: old Mock does not throw side_effect exceptions from a list 6 years ago