Commit Graph

451 Commits (4dfbe82e76cd6ef7158069e1dfdc5755ef7cba9d)

Author SHA1 Message Date
David Wilson 001e3fee86 issue #536: restore correct Python interpreter selection behaviour. 6 years ago
David Wilson 05b1ccb658 ansible: stash PID files in CWD if requested for debugging. 6 years ago
David Wilson eb67fbe9d2 ansible: double the default pool size.
Tempted to push this up to 64, but let's do it incrementally just in
case.
6 years ago
David Wilson b89e53fd70 ansible: raise error with correct exception type. 6 years ago
David Wilson 0e193c223c issue #508: master: minify all Mitogen/ansible_mitogen sources.
Minify-safe files are marked with a magical "# !mitogen: minify_safe"
comment anywhere in the file, which activates the minifier. The result
is naturally cached by ModuleResponder, therefore lru_cache is gone too.

Given:

    import os, mitogen
    @mitogen.main()
    def main(router):
        c = router.ssh(hostname='k3')
        c.call(os.getpid)
        router.sudo(via=c)

SSH footprint drops from 56.2 KiB to 42.75 KiB (-23.9%)
Ansible "shell: hostname" drops 149.26 KiB to 117.42 KiB (-21.3%)
6 years ago
David Wilson 7badb4a25b ansible: hacky parser to alow bools to be specified on command line 6 years ago
David Wilson b499fbe29b ansible: add mitogen_ssh_compression variable. 6 years ago
David Wilson a2ae4ed696 SyntaxError. 6 years ago
David Wilson a9d48a8fdc ansible: don't pin controller if <4 cores. 6 years ago
David Wilson 4531338b12 ansible: document and make affinity stuff portable to non-Linux
Portable as in does nothing for the time at least for now.
6 years ago
David Wilson de5c050707 ansible: fix affinity.py test failure on 2 cores. 6 years ago
David Wilson 00ae90b2b2 ansible: preheat PluginLoader caches before fork.
This has been broken for some time, but somehow it has become noticeable
on recent Ansible.

loop-100-tasks.yml before:
      15.532724001 seconds time elapsed
       8.453850000 seconds user
       5.808627000 seconds sys

loop-100-tasks.yml after:
       8.991635735 seconds time elapsed
       5.059232000 seconds user
       2.578842000 seconds sys
6 years ago
David Wilson 7b129e8576 ansible: use Poller for WorkerProcess; closes #491. 6 years ago
David Wilson c6d5aa29ba ansible: new multiplexer/workers configuration
Following on from 152effc26c9a5918cb7ead7a97fe7fa7f81b6764,

* Pin mux to CPU 0
* Pin top-level CPU 1
* Pin workers sequentially to CPU 2..n

Nets 19.5% improvement on issue_140__thread_pileup.yml when targetting
64 Docker containers on the same 8 core/16 thread machine.

Before (prior to last scheme, no affinity at all):

    2294528.731458      task-clock (msec)         #    6.443 CPUs utilized
        10,429,745      context-switches          #    0.005 M/sec
         2,049,618      cpu-migrations            #    0.893 K/sec
         8,258,952      page-faults               #    0.004 M/sec
 5,532,719,253,824      cycles                    #    2.411 GHz                      (83.35%)
 3,267,471,616,230      instructions              #    0.59  insn per cycle
                                                  #    1.22  stalled cycles per insn  (83.35%)
   662,006,455,943      branches                  #  288.515 M/sec                    (83.33%)
    39,453,895,977      branch-misses             #    5.96% of all branches          (83.37%)

     356.148064576 seconds time elapsed

After:

    2226463.958975      task-clock (msec)         #    7.784 CPUs utilized
         9,831,466      context-switches          #    0.004 M/sec
           180,065      cpu-migrations            #    0.081 K/sec
         5,082,278      page-faults               #    0.002 M/sec
 5,592,548,587,259      cycles                    #    2.512 GHz                      (83.35%)
 3,135,038,855,414      instructions              #    0.56  insn per cycle
                                                  #    1.32  stalled cycles per insn  (83.32%)
   636,397,509,232      branches                  #  285.833 M/sec                    (83.30%)
    39,135,441,790      branch-misses             #    6.15% of all branches          (83.35%)

     286.036681644 seconds time elapsed
6 years ago
David Wilson 1b909e8697 ansible: pin connection multiplexer to a single core
Nets a reliable 8% improvement in issue_140__thread_pileup.yml when
targetting 64 Docker containers on the same 8 core/16 thread machine.

Before:
    2294528.731458      task-clock (msec)         #    6.443 CPUs utilized
        10,429,745      context-switches          #    0.005 M/sec
         2,049,618      cpu-migrations            #    0.893 K/sec
         8,258,952      page-faults               #    0.004 M/sec
 5,532,719,253,824      cycles                    #    2.411 GHz                      (83.35%)
 4,001,276,805,120      stalled-cycles-frontend   #   72.32% frontend cycles idle     (83.30%)
 2,024,159,442,463      stalled-cycles-backend    #   36.59% backend cycles idle      (66.65%)
 3,267,471,616,230      instructions              #    0.59  insn per cycle
                                                  #    1.22  stalled cycles per insn  (83.35%)
   662,006,455,943      branches                  #  288.515 M/sec                    (83.33%)
    39,453,895,977      branch-misses             #    5.96% of all branches          (83.37%)

     356.148064576 seconds time elapsed

After:
    2208247.938562      task-clock (msec)         #    6.735 CPUs utilized
         8,489,840      context-switches          #    0.004 M/sec
         1,432,967      cpu-migrations            #    0.649 K/sec
         7,508,957      page-faults               #    0.003 M/sec
 5,477,293,750,357      cycles                    #    2.480 GHz                      (83.31%)
 3,984,360,350,811      stalled-cycles-frontend   #   72.74% frontend cycles idle     (83.32%)
 1,976,646,418,711      stalled-cycles-backend    #   36.09% backend cycles idle      (66.64%)
 3,196,197,480,792      instructions              #    0.58  insn per cycle
                                                  #    1.25  stalled cycles per insn  (83.36%)
   648,247,332,967      branches                  #  293.557 M/sec                    (83.35%)
    39,004,881,070      branch-misses             #    6.02% of all branches          (83.37%)

     327.876903668 seconds time elapsed
6 years ago
David Wilson e587396e70 ansible: hook strategy and worker processes into profiler 6 years ago
David Wilson 84944a9a61 ansible: ensure MuxProcess MITOGEN_PROFILING results reach disk.
This has been broken for quite some time.
6 years ago
David Wilson 954f874085 issue #527: catch new-style module tracebacks like vanilla. 6 years ago
David Wilson a1121c5a84 issue #499: respect C.BECOME_ALLOW_SAME_USER. 6 years ago
David Wilson be6ab52fe1 issue #488: fix shutdown damage caused in 6ca2677de5
os._exit() subverted calm shutdown, meaning unix.Listener never had a
chance to cleanup its socket.

Move unix.Listener socket cleanup into its class so it is automatic
during shutdown, rather than cutpasted for each consumer.

Disable the watcher thread in the MuxProcess, it is useless.

Add .sock extension to /tmp/mitogen_unix_*, so we can write a test.
6 years ago
David Wilson 38a553d42d issue #490: prevent double close() destroying unrelated Connection. 6 years ago
David Wilson e7fe95af88 issue #477: fix sudo_args selection. 6 years ago
David Wilson 599da0689a issue #477 / ansible: avoid a race in async job startup.
Ansible 2.3/Python 2.4 work revealed there is no guarantee a slow target
will have written the initial job status file out before a fast
controller makes an initial check for it. Therefore, provide AsyncRunner
with a sender it should send a message to when the initial job file has
been written.

As a bonus, also catch and report exceptions happening early in
AsyncRunner, rather than leaving them to end up in -vvv output.
6 years ago
David Wilson 0175052099 issue #477: fix source of become_flags on 2.3. 6 years ago
David Wilson 97f3cfe4f4 issue #477: target.file_exists() wrapper.
os.path.exists physical module name varies across major Python versions.
6 years ago
David Wilson 8f5b65f7ec issue #477: introduce subprocess isolation.
Since Python 2.4 fork is so defective, we must use subprocesses for
mitogen_task_isolation=fork. This has plenty of upside, since the long
term goal is to dump forking altogether. This allows a gentle
introduction of its replacement.
6 years ago
David Wilson b9924683ac ansible: docstring fixes. 6 years ago
David Wilson 75f53faf8c issue #477: shlex.split() in 2.4 required bytes input. 6 years ago
David Wilson dc1d4251e3 ansible: synchronize module needs '.docker_cmd' attr for Docker plugin. 6 years ago
David Wilson dd30a907ce issue #477: promote setup_gil() to mitogen.utils
This is since ansible_mitogen/process.py is 2.6-only, and I want to use
setup_gil() in 2.4 code.
6 years ago
David Wilson bba5d82fc4 issue #477: fix another str/bytes mixup. 6 years ago
David Wilson c9ff4c6506 issue #477: fix 3.x failure in new target.set_file_mode() function. 6 years ago
David Wilson aa5d37af89 issue #477: fix 3.x failure in new target.set_file_mode() function. 6 years ago
David Wilson 84a0424749 issue #477: fix 3.x test regressions. 6 years ago
David Wilson 81f15028a7 issue #477: backport ansible_mitogen.runner to 2.4. 6 years ago
David Wilson 256628c149 issue #477: backport ansible_mitogen/target.py to Python2.4 6 years ago
David Wilson a48ee3a536 issue #477: vendorize the last 2.4-compatible simplejson
This is in part so image_prep can run against an ancient CentOS 5 image
without any upfront help, and in part simply because it's very easy to
support.
6 years ago
David Wilson 59dd0dc814 issue #477: serve up junk ansible/__init__.py just like Ansible. 6 years ago
David Wilson f52f7f6a62 ansible: merge away unused return values / functions. 6 years ago
David Wilson b5c14cecda docs: docs and docstrings. 6 years ago
David Wilson e767de3f15 issue #412: force-verbose output for mitogen_get_stack. 6 years ago
David Wilson 23866084d7 issue #412: promote "mitogen_get_stack" to the main extension.
This is to make it easier for users to diagnose their own problems.
6 years ago
David Wilson 115c3c5657 issue #412: add docstrings/boilerplate to transport_config.py. 6 years ago
David Wilson 2ad05f1238 issue #251, #412, #434: fix connection configuration brainwrong
This refactors connection.py to pull the two huge dict-building
functions out into new transport_transport_config.PlayContextSpec and
MitogenViaSpec classes, leaving a lot more room to breath in both files
to figure out exactly how connection configuration should work.

The changes made in 1f21a30 / 3d58832 are updated or completely removed,
the original change was misguided, in a bid to fix connection delegation
taking variables from the wrong place when delegate_to was active.

The Python path no longer defaults to '/usr/bin/python', this does not
appear to be Ansible's normal behaviour. This has changed several times,
so it may have to change again, and it may cause breakage after release.

Connection delegation respects the c.DEFAULT_REMOTE_USER whereas the
previous version simply tried to fetch whatever was in the
'ansible_user' hostvar. Many more connection delegation variables closer
match vanilla's handling, but this still requires more work. Some of the
variables need access to the command line, and upstream are in the
process of changing all that stuff around.
6 years ago
David Wilson 6ca2677de5 ansible: fix test failure during process exit.
======================================================================
ERROR: tests.connection_test (unittest2.loader._FailedTest)
----------------------------------------------------------------------
Traceback (most recent call last):
ImportError: Failed to import test module: tests.connection_test
Traceback (most recent call last):
  File "/home/dmw/src/mitogen/.venv/local/lib/python2.7/site-packages/unittest2/loader.py", line 456, in _find_test_path
    module = self._get_module_from_name(name)
  File "/home/dmw/src/mitogen/.venv/local/lib/python2.7/site-packages/unittest2/loader.py", line 395, in _get_module_from_name
    __import__(name)
RuntimeError: not holding the import lock
6 years ago
David Wilson 6915af502e issue #61: unused import (reported by LGTM) 6 years ago
David Wilson 907c4556c8 issue #61: unused variable (reported by LGTM) 6 years ago
David Wilson f67960848b issue #61: unused import (reported by LGTM) 6 years ago
David Wilson b8ca015b83 issue #61: unused variable (reported by LGTM) 6 years ago
David Wilson bcc7bb7128 issue #61: unused import (reported by LGTM) 6 years ago
David Wilson 4bdf60326c issue #424: ansible: make put_file() raise AnsibleFileNotFound 6 years ago
David Wilson 3f31b166f6 issue #461: Ansible 2.3-compatible _get_candidate_temp_dirs(). 6 years ago
David Wilson 9401c34f64 issue #461: Ansible 2.3 did not have _load_name. 6 years ago
David Wilson 04755c3321 issue #426: tighten up PushFileService types.
Bytes/Unicode mixing caused a hang, so prevent bytes entirely.
6 years ago
David Wilson 1e9f3446c8 issue #426: big hack so reset_connection has task_var access 6 years ago
David Wilson ca9ae4590c issue #426: TemporaryEnvironment must coerce to Unicode. 6 years ago
David Wilson 4bc0d0ec13 issue #426: apply_mode_spec() must handle bytes. 6 years ago
David Wilson 3179951f5c issue #454: fix AttributeError and atexit.yml test. 6 years ago
David Wilson cd01957995 issue #397, #454: pick out only shutil.rmtree() calls from atexit. 6 years ago
David Wilson 09aa27a6a7 ansible: initial mitogen_host_pinned strategy. 6 years ago
David Wilson 22de7f0e72 issue #436: fix string parsing of mitogen_ssh_debug_level
It can be a string when specified on the command line.
6 years ago
David Wilson 87ddead0b2 issue #435, #362: ansible: avoid touching setrlimit() on target.
This replaces the previous method for capping poorly Popen()
performance, instead entirely monkey-patching the problem function
rather than simply working around it.
6 years ago
David Wilson b599253291 issue #435: increase file descriptor limit by 8x.
This is a temporary solution at best.
6 years ago
David Wilson 16911c9464 ansible: fix 3.x TypeError regression. 6 years ago
David Wilson 50241a922f ansible: call on_fork() on broker shutdown; closes #420. 6 years ago
David Wilson 35092c5d35 tests: Unicode/bytes fixes for integration/connection/exec_command.yml 6 years ago
David Wilson 76ec4f201c issue #413: paper over harmless duplicate del_route()
Ideally it would only be called once, and in future maybe it can, but
right now we need to cope with these cases:

* Downstream parent notifies us of disconnection (DEL_ROUTE)
* We notify ourself of disconnection
* We notify ourself and so does downstream parent

It's case 3 that causes the error.
6 years ago
David Wilson bac28bc5ca issue #76, #370: add fix for disconnect cleanup test
Simply listen to RouteMonitor's Context "disconnect"  and forget
contexts according to RouteMonitor's rules, rather than duplicate them
(and screw it up).
6 years ago
David Wilson c148c869e6 issue #76, #370: add disconnect cleanup test 6 years ago
David Wilson 58c0e45661 issue #400: rework the monkeypatch. 6 years ago
David Wilson c9ecc82f85 issue #400: add logic to work around AWX callback bug. 6 years ago
David Wilson 6c71c5bfef issue #369: disable reset_connection on Ansible<2.5.6
https://github.com/ansible/ansible/issues/27520
6 years ago
David Wilson c4aec22a33 issue #369: fix one more _reset() reference. 6 years ago
David Wilson 6107ebdc0d issue #396: fix compatibility with Connection._reset(). 6 years ago
David Wilson d280bba02b issue #369: fix KeyError during new context start.
Update _via_by_context earlier; fixes:

    Traceback (most recent call last):
      File "/Users/dmw/src/mitogen/mitogen/service.py", line 519, in _on_service_call
	return invoker.invoke(method_name, kwargs, msg)
      File "/Users/dmw/src/mitogen/mitogen/service.py", line 253, in invoke
	response = self._invoke(method_name, kwargs, msg)
      File "/Users/dmw/src/mitogen/mitogen/service.py", line 239, in _invoke
	ret = method(**kwargs)
      File "/Users/dmw/src/mitogen/ansible_mitogen/services.py", line 454, in get
	reraise(*result)
      File "/Users/dmw/src/mitogen/ansible_mitogen/services.py", line 412, in _wait_or_start
	response = self._connect(key, spec, via=via)
      File "/Users/dmw/src/mitogen/ansible_mitogen/services.py", line 363, in _connect
	self._update_lru(context, spec, via)
      File "/Users/dmw/src/mitogen/ansible_mitogen/services.py", line 266, in _update_lru
	self._update_lru_unlocked(new_context, spec, via)
      File "/Users/dmw/src/mitogen/ansible_mitogen/services.py", line 253, in _update_lru_unlocked
	if self._refs_by_context[context] == 0:
    KeyError: Context(1008, u'ssh.localhost.sudo.mitogen__user3')
6 years ago
David Wilson 54445470e2 issue #409: add missing path config variables to severa plugins
So every method can be redirected to a stub implementation.
6 years ago
David Wilson a68675af8b issue #409: fix reference error in kubectl.py. 6 years ago
David Wilson 05f9fb4dd8 issue #409: don't run kubectl test in <2.5. 6 years ago
David Wilson 18af1dfb51 ansible: kubectl_path argument appears in wrong connection method
Closes #409.
6 years ago
David Wilson 96f000c5ea ansible: tilde-expand SSH key before passing to SSH; closes #334. 6 years ago
David Wilson 536690760d issue #369: teach CallChain to reset the connection. 6 years ago
David Wilson 33412927f5 issue #369: refactor Connection to support reset()
Now the tests pass.
6 years ago
David Wilson 9b7c958e2e issue #369: refactor ContextService to support reset(). 6 years ago
David Wilson d0f5671887 ansible: split key_from_dict() out into free function. 6 years ago
David Wilson 0dc3f8accf ansible: fix another target.py format string. 6 years ago
David Wilson 7e04ee8af9 ansible: fix is_good_temp_dir() log format 6 years ago
David Wilson fba52a0edf issue #76: add API for ansible_mitogen to get route list
Earlier commit moved Stream.routes attribute into a private map
belonging to RouteMonitor, to make upgrades smoother. This adds a new
accessor method to RouteMonitor.
6 years ago
David Wilson 7fd9fb0014 issue #397: fix another case where stray tmpdirs can be left behind.
Newer Ansibles use atexit.register() to invoke cleanup, so we need to
run those registrations after each run.
6 years ago
David Wilson 1b17aa1d1a ansible: fix temp cleanup regression and add test; closes #397. 6 years ago
David Wilson 9d070541d9 ansible: try to create tempdir if missing.
Closes #358.
6 years ago
David Wilson 4c81eba599 Merge commit 'refs/pull/377/head' of github.com:dw/mitogen into dmw
(Pull #377)

Changes:
- additional_parameters -> extra_args
- Merge with kubectl changes from dmw branch
- Update docs
- Remove unused username class member
- Avoid mutable kubectl_args class member
- Use six.iteritems
6 years ago
David Wilson 7a00e1cc87 issue #360: missing locks around shutdown and LRU management. 6 years ago
David Wilson 498db57ec8 issue #360: ansible: missing lock around ContextService.put(). 6 years ago
David Wilson f8bf780e21 issue #362: Py3.x fixes. 6 years ago
David Wilson f8b6c774dd issue #362: cap max open files in children. 6 years ago
David Wilson 5521945bd2 ansible: temporary files take 5. 6 years ago
Yannig Perré 17548d1e49 [Enhancement] handle kubectl vars from Ansible connector.
This change allows the kubectl connector to support the same options as
Ansible's original connector.

The playbook sample comes with an example of a pod containing two containers
and checking that moving from one container to another, the version of Python
changes as expected.
6 years ago
Yannig Perré 6828926a36 Kubernetes connection support for mitogen. 6 years ago
David Wilson 638b196a45 ansible: fix put_file() for large temporary files.
Reverts 49736b3a, large file copies can't avoid the RTT.

The parent stack must be blocked while FileService progresses, as unlike
the small file path, it does not make a snapshot of the (possibly
temporary) file passed by the action plug-in. So we need to keep that
file alive while the service runs.

Add a new integration test and a new soak test to cover both.
6 years ago
David Wilson 9ff34afafe ansible: fix regression. 6 years ago