Commit Graph

774 Commits (de2e1ec184ed21507b5bf546deefdf0c7a73c933)

Author SHA1 Message Date
David Wilson d1f5e0663d core: move message encoding to Message.pack(), add+refactor tests.
The old inline pack is still present in the old location but will be
removed in a followup commit.
5 years ago
David Wilson 3ab7998861 tests: add some UTF-8 to ssh_login_banner to encourage breakage. 5 years ago
David Wilson f6d26c5acb testlib: disable lsof warnings due to Docker crap 5 years ago
David Wilson 70ff4b674c parent: discard cancelled events in TimerList.get_timeout().
Otherwise get_timeout() keeps broker alive via keep_alive() for a
cancelled timer during shutdown.
5 years ago
David Wilson 5aca9d6c3f core: split out iter_split() for use in parent.py. 5 years ago
David Wilson 2fbc77a155 issue #170: implement timers. 5 years ago
Jordan Webb d71bdd1694
Add buildah test and stub 5 years ago
David Wilson ee7dae7514 ci: Another round of fixes for random Ansible UI breakage in 2.7/2.8 5 years ago
David Wilson ad5a80f200 Use virtualenv Python for stub connections to workaround problem
../data/stubs/stub-kubectl.py exec -it localhost -- /usr/bin/python -c "...":

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    LookupError: unknown encoding: base64

It's not clear why this is happening. "stub-kubectl.py" is executed with
the 2.7 virtualenv, while the exec() that happens inside stub-kubectl
was for "/usr/bin/python".

That second Python can't find chunks of its stdlib:

    stat("/usr/lib/python2.7/encodings/base64", 0x7ffde8744c60) = -1 ENOENT (No such file or directory)
    open("/usr/lib/python2.7/encodings/base64.so", O_RDONLY) = -1 ENOENT (No such file or directory)
    open("/usr/lib/python2.7/encodings/base64module.so", O_RDONLY) = -1 ENOENT (No such file or directory)
    open("/usr/lib/python2.7/encodings/base64.py", O_RDONLY) = -1 ENOENT (No such file or directory)
    open("/usr/lib/python2.7/encodings/base64.pyc", O_RDONLY) = -1 ENOENT (No such file or directory)
    write(2, "Traceback (most recent call last):\n", 35) = 35
    write(2, "  File \"<string>\", line 1, in <module>\n", 39) = 39
5 years ago
David Wilson 08e7fe4f80 tests: add 2.8 format async error timeout message 5 years ago
David Wilson 418fc15e80 tests: allow running without hdrhistograms library. 5 years ago
David Wilson 3620fce071 issue #593: expose configurables for SSH keepalive and increase the default 5 years ago
David Wilson e90c05dc9d issue #591: fix test for Ansible 2.3. 5 years ago
David Wilson 0b7fd3f290 issue #591: ansible: restore CWD prior to AnsibleModule initialization. 5 years ago
David Wilson fe7c3610ff issue #590: disable distro test on vanilla 5 years ago
David Wilson f1287e6e9a tests: Py3.x fix. 5 years ago
David Wilson a1f2ec222d issue #590: fix test for <2.8 Ansibles. 5 years ago
David Wilson 90105e2031 tests: Py3.x fix. 5 years ago
David Wilson e94200aeb9 issue #590: actually run Ansible test. 5 years ago
David Wilson 50cdf63c27 tests: Py3.x fix. 5 years ago
David Wilson 2f29f3e8e6 tests: rearrange test modules again, they're used in multiple places
They're back on sys.path.
5 years ago
David Wilson 2f68a5a660 module_finder_test: mask one more difference between unit2 vs. direct start 5 years ago
David Wilson cf1e7129a7 issue #590: add dummy package for new test. 5 years ago
David Wilson c1db0d3858 issue #590: move example modules to module_finder/, fix/add tests 5 years ago
David Wilson 7a5c436a39 issue #590: Ansible test for module_utils.distro use. 5 years ago
David Wilson 8f940e2ccb issue #590: teach importer to handle self-replacing modules 5 years ago
David Wilson 61b651bdd2 issue #587: workaround for ansible/ansible#56629 5 years ago
David Wilson 8a18d1862e issue #587: "state: absent" was removed in 2.8.0
ansible/cc9c72d6f845710b24e952670b534a57f6948513
5 years ago
David Wilson 92b4724010 issue #587: consistent become_exe() behaviour for older Ansibles. 5 years ago
David Wilson a25fa566a1 issue #587: update stub_connections/ test to use new doas var 5 years ago
David Wilson a3be746865 issue #587: update MODULE FAILURE message format for post >2.7 5 years ago
David Wilson 54b5fdf761 issue #587: fix syntax error due to presence of comment 5 years ago
David Wilson ce80e326fe issue #587: disable deprecation_warnings for CI.
Our use case justifies it
5 years ago
David Wilson b8e1b4df51 issue #587: 2.8 whitespace handling was improved.
b72e989e18
5 years ago
David Wilson c616359a17 Import include processing bench 5 years ago
David Wilson f30a4c05c8 issue #581: expose mitogen_mask_remote_name variable. 5 years ago
David Wilson 65deb3feac issue #575: fix exception text rendering 5 years ago
David Wilson 3ff6123483 issue #557: support correct cpu_set_t size 5 years ago
David Wilson 2bd0bbd4df issue #555: ansible: workaround ancient reload(sys) hack.
This is the most minimal change for what might be relatively minimal
edge case. Alternative is replacing reload(), but let's not do that yet.

Closes #555
5 years ago
David Wilson 30b8172573 issue #554: mitogen_action_script fix 5 years ago
David Wilson 6309774be2 issue #554: fix Ansible 2.4 compatibility 5 years ago
David Wilson f36b4b47bf issue #554: don't rely on tmp_path autoremoval in test.
Ansible doesn't do this, so we shouldn't either.
5 years ago
David Wilson 7743e57ff3 issue #554: track and remove multiple make_tmp_path() calls. 5 years ago
David Wilson 26e6194d0a issue #548: always treat transport=smart as 'ssh' for mitogen_via=.
The idea behind transport=smart is to select between paramiko and
OpenSSH given the availability of connection multiplexing and/or OSX
kernel bugs. We need to make no such choice.
5 years ago
David Wilson c413d53144 os_fork: python 3 fixes and tests. 5 years ago
David Wilson 18b984a0b4 issue #535: activate Corker on 2.4 in master too. 5 years ago
David Wilson 06e52ca89f issue #535: wire mitogen.os_fork into Broker and Pool. 5 years ago
David Wilson b3f592acee issue #535: core/select: support selecting from Latches. 5 years ago
David Wilson e517810e5a tests: ensure serialization restrictions are in effect 5 years ago
David Wilson ffdf31edd7 tests/bench: set process affinity in throughput.py. 5 years ago
David Wilson 9bcd2ec56c issue #542: return of select poller, new selection logic 5 years ago
David Wilson 0aa4c9d8fc issue #542: .ci: move some tests to Azure and enable Mac job. 5 years ago
David Wilson 5ed445c4aa issue #537: disable just the trivial LinuxPolicyTest on Travis. 5 years ago
David Wilson 8f9c67daf1 ansible: refactor affinity class and add abstract tests. 5 years ago
David Wilson d4c0250083 issue #532: PushFileService race.
There has always been a race in PushFileService since given a parent
asked to forward modules to two children via some intermediary:

    interm = router.local()
    c1 = router.local(via=interm)
    c2 = router.local(via=interm)

    service.propagate_to(c1, 'foo/bar.py')
    service.propagate_to(c2, 'foo/bar.py')

Two calls will be emitted to 'interm':

    PushFileService.store_and_forward(c1, 'foo/bar.py', [blob])
    PushFileService.store(c2, 'foo/bar.py')

Which will be processed in-order up to the point where service pool
threads in 'interm' are woken to process the message.

While it is guaranteed store_and_forward() will be processed first, no
guarantee existed that its assigned pool thread would wake and take
_lock first, thus it was possible for forward() to win the race, and for
a request to arrive to forward a file that had not been placed in local
cache yet.

Here we get rid of SerializedInvoker entirely, as it is partially to
blame for hiding the race: SerializedInvoker can only ensure no two
messages are processed simultaneously, it cannot ensure the messages are
processed in their intended order.

Instead, teach forward() that it may be called before
store_and_forward(), and if that is the case, to place the forward
request on to _waiters alongside any local threads blocked in get().
5 years ago
David Wilson 7fd0d34910 tests/ansible: Spec.port() test & mitogen_via= fix.
ansible_ssh_port was not respected.
5 years ago
David Wilson b5b23e8f3d tests/ansible: Spec.become_pass() test. 5 years ago
David Wilson 7b3ed52fd8 tests/ansible: Spec.become_user() test. 5 years ago
David Wilson 1c955a9876 ansible: capture stderr stream of async tasks. Closes #540. 5 years ago
David Wilson 8ae6ca1d5b tests/ansible: Spec.become_method() test & mitogen_via= fix.
ansible_become_method hostvar was not taken into account.
5 years ago
David Wilson 8ba75d82ec tests/ansible: Spec.become() test 5 years ago
David Wilson d1cadf8ac8 tests/ansible: Spec.password() test, document interactive pw limitation. 5 years ago
David Wilson 21ad299d7b tests/ansible: Spec.remote_user() test & mitogen_via= fix.
ansible_ssh_user precedence was incorrect.
5 years ago
David Wilson 748f5f675d tests/ansible: Spec.remote_addr() test & mitogen_via= fix.
ansible_ssh_host was not respected.
5 years ago
David Wilson 8fd641c442 tests/ansible: Spec.transport() test. 5 years ago
David Wilson e1df98168c issue #536: add mitogen_via= tests too. 5 years ago
David Wilson 2f29c76eec tests: for 2.3 compatibility, disable gcloud.py for now 5 years ago
David Wilson b3f20f54e6 issue #536: disable transport_config tests on vanilla 5 years ago
David Wilson 1d43e187e8 issue #536: connection_delegation/ tests were erroneously broken
While fixing delegate_to, this un-hardwiring of /usr/bin/python
happened. It was always incorrect.
5 years ago
David Wilson 0dfcf5560b tests: define MITOGEN_INVENTORY_FILE even if -i unspecified.
To fix running tests locally.
5 years ago
David Wilson 01faed708d issue #536: add tests for each ansible_python_interpreter case. 5 years ago
David Wilson 86f9572ef7 issue #536: stop defining explicit localhost in inventory.
This was needed at some point in the past, but the tests don't seem to
care about it any more. We'll fix any CI breakage by changing the tests,
since verifying implicit localhost behaviour is important.
5 years ago
David Wilson 81e386586f tests: allow running Ansible tests locally without -udmw again. 5 years ago
David Wilson 4506c528ca issue #481: add test. 5 years ago
David Wilson b263e01867 issue #481: avoid crash if disconnect occurs during forward_modules() 5 years ago
David Wilson 2aed0a0c6c tests: add new compression parameter to mitogen_get_stack results 5 years ago
David Wilson 6859186069 tests: disable affinity_test on Travis :/ 5 years ago
David Wilson 19ae7b7a84 issue #508: fix responder stats test due to new smaller parent.py. 5 years ago
David Wilson 953a910881 issue #508: tests: skip minify_test Py2.4/2.5 for profiler.py. 5 years ago
David Wilson 263f530496 tests: fix fallout from 36fb318adf5c56e729296c3efce84f4dd75ced4e 5 years ago
David Wilson 8026e54b9b tests: move affinity_test to Ansible tests. 5 years ago
David Wilson e703522010 issue #505: docs: add new detail graph for one scenario. 5 years ago
David Wilson b2bae9deed tests: pad out localhost-* 5 years ago
David Wilson fda7d5c276 bench: tidy up and cpu-pin some more files. 5 years ago
David Wilson 29adfbea59 tests: add localhost-x100 5 years ago
David Wilson 0e193c223c issue #508: master: minify all Mitogen/ansible_mitogen sources.
Minify-safe files are marked with a magical "# !mitogen: minify_safe"
comment anywhere in the file, which activates the minifier. The result
is naturally cached by ModuleResponder, therefore lru_cache is gone too.

Given:

    import os, mitogen
    @mitogen.main()
    def main(router):
        c = router.ssh(hostname='k3')
        c.call(os.getpid)
        router.sudo(via=c)

SSH footprint drops from 56.2 KiB to 42.75 KiB (-23.9%)
Ansible "shell: hostname" drops 149.26 KiB to 117.42 KiB (-21.3%)
5 years ago
David Wilson b72d139bd2 tests: make soak testing work reliably on vanilla.
It claims to disable host key checking, but it doesn't.
5 years ago
David Wilson b59a47aecb tests: make mitogen_shutdown_all be run_once by default. 5 years ago
David Wilson e77048ec2d utils: pad out reset_affinity() and integrate with detach_popen() 5 years ago
David Wilson 2209d880e4 ansible/bench: make end= configurable. 5 years ago
David Wilson 954f874085 issue #527: catch new-style module tracebacks like vanilla. 5 years ago
David Wilson 60fe3fd6f5 issue #429: enable en_US locale to unbreak debops test. 6 years ago
David Wilson 7531af3ee0 issue #499: fix another mind-numbingly stupid vanilla inconsistency 6 years ago
David Wilson a40946297f issue #497: do our best to cope with crap upstream code 6 years ago
David Wilson 2fdbd0cfcd ssh: fix test to match updated log format. 6 years ago
David Wilson 6e9f8e829e issue #429: teach sudo about every know i18n password string. 6 years ago
David Wilson 960e505f07 issue #429: install i18n-related bits in test images. 6 years ago
David Wilson eb93f82d05 tests: ensure file is closed in connection_test. 6 years ago
David Wilson 73979043ad gcloud: small updates 6 years ago
David Wilson 59068ca955 tests: give ansible/gcloud/ its own requirements file. 6 years ago