Commit Graph

2332 Commits (b263e01867b10caf39685d02388cd82620f7103a)
 

Author SHA1 Message Date
David Wilson b263e01867 issue #481: avoid crash if disconnect occurs during forward_modules() 6 years ago
David Wilson 92b1648729 Add a few more important modules to preamble_size.py. 6 years ago
David Wilson 6dba55624e .ci: add verbiage for run_batches() too. 6 years ago
David Wilson 9c7d93885d .ci: add README.md. 6 years ago
David Wilson 32161f3df5 docs: update thanks 6 years ago
David Wilson 4f6c57b6a8 docs: lose "approaching stability" language, we're pretty good now 6 years ago
David Wilson 7b046c9fba docs: fix changelog syntax/order/"20KB" 6 years ago
David Wilson 1345f05627 Merge remote-tracking branch 'origin/dmw'
* origin/dmw:
  tests: add new compression parameter to mitogen_get_stack results
  tests: disable affinity_test on Travis :/
  issue #508: fix responder stats test due to new smaller parent.py.
  issue #508: tests: skip minify_test Py2.4/2.5 for profiler.py.
  tests: fix fallout from 36fb318adf5c56e729296c3efce84f4dd75ced4e
  issue #520: add AIX auth failure string to su.
  tests: move affinity_test to Ansible tests.
  core: cProfile is not available in 2.4.
  issue #505: docs: add new detail graph for one scenario.
  docs: update and re-record profile graphs in docs; closes #505
  service: fix PushFileService exception
  tests: pad out localhost-*
  service: start pool shutdown on broker shutdown.
  master: .encode() needed for Py3.
  ansible: stash PID files in CWD if requested for debugging.
  issue #508: master: minify_safe_re must be bytes for Py3.
  bench: tidy up and cpu-pin some more files.
  tests: add localhost-x100
  ansible: double the default pool size.
  ansible: raise error with correct exception type.
  issue #508: master: minify all Mitogen/ansible_mitogen sources.
  parent: PartialZlib docstrings.
  ansible: hacky parser to alow bools to be specified on command line
  parent: pre-cache bootstrap if possible.
  docs: update Changelog.
  ansible: add mitogen_ssh_compression variable.
  service: PushFileService never recorded a file as sent.
  parent: synchronize get_core_source()
  service: use correct profile aggregation name.
  SyntaxError.
  ansible: don't pin controller if <4 cores.
  tests: make soak testing work reliably on vanilla.
  docs: changelog tidyups.
  ansible: document and make affinity stuff portable to non-Linux
  ansible: fix affinity.py test failure on 2 cores.
  ansible: preheat PluginLoader caches before fork.
  tests: make mitogen_shutdown_all be run_once by default.
  docs: update Changelog.
  ansible: use Poller for WorkerProcess; closes #491.
  ansible: new multiplexer/workers configuration
  docs: update Changelog.
  docs: update Changelog.
  ansible: pin connection multiplexer to a single core
  utils: pad out reset_affinity() and integrate with detach_popen()
  utils: import reset_affinity() function.
  master: set Router.profiling if MITOGEN_PROFILING variable present.
  parent: don't kill children when profiling is active.
  ansible: hook strategy and worker processes into profiler
  profiler: import from linear2 branch
  core: tidy up existing profiling code and support MITOGEN_PROFILE_FMT
  issue #260: redundant if statement.
  ansible: ensure MuxProcess MITOGEN_PROFILING results reach disk.
  ansible/bench: make end= configurable.
  master: cache sent/forwarded module names
6 years ago
David Wilson 2aed0a0c6c tests: add new compression parameter to mitogen_get_stack results 6 years ago
David Wilson 6859186069 tests: disable affinity_test on Travis :/ 6 years ago
David Wilson 19ae7b7a84 issue #508: fix responder stats test due to new smaller parent.py. 6 years ago
David Wilson 953a910881 issue #508: tests: skip minify_test Py2.4/2.5 for profiler.py. 6 years ago
David Wilson 263f530496 tests: fix fallout from 36fb318adf5c56e729296c3efce84f4dd75ced4e 6 years ago
David Wilson 4abd34e7a6 issue #520: add AIX auth failure string to su. 6 years ago
David Wilson 8026e54b9b tests: move affinity_test to Ansible tests. 6 years ago
David Wilson 5ae7464011 core: cProfile is not available in 2.4. 6 years ago
David Wilson e703522010 issue #505: docs: add new detail graph for one scenario. 6 years ago
David Wilson c279454502 docs: update and re-record profile graphs in docs; closes #505
pcaps moved to 'pcaps' branch.
6 years ago
David Wilson 54835d4c9b service: fix PushFileService exception
[costapp]

ERROR! [pid 25135] 21:10:56.284733 E mitogen.ctx.ssh.35.200.203.48: mitogen: While calling no-reply method PushFileService.forward
Traceback (most recent call last):
  File "master:/home/dmw/src/mitogen/mitogen/service.py", line 260, in _invoke
    ret = method(**kwargs)
  File "master:/home/dmw/src/mitogen/mitogen/service.py", line 718, in forward
    self._forward(path, context)
  File "master:/home/dmw/src/mitogen/mitogen/service.py", line 633, in _forward
    stream = self.router.stream_by_id(context.context_id)
AttributeError: 'unicode' object has no attribute 'context_id'
^C [ERROR]: User interrupted execution
6 years ago
David Wilson b2bae9deed tests: pad out localhost-* 6 years ago
David Wilson 23d7a961e7 service: start pool shutdown on broker shutdown. 6 years ago
David Wilson 85cfa3b0f5 master: .encode() needed for Py3. 6 years ago
David Wilson 05b1ccb658 ansible: stash PID files in CWD if requested for debugging. 6 years ago
David Wilson 1d509d03ff issue #508: master: minify_safe_re must be bytes for Py3. 6 years ago
David Wilson fda7d5c276 bench: tidy up and cpu-pin some more files. 6 years ago
David Wilson 29adfbea59 tests: add localhost-x100 6 years ago
David Wilson eb67fbe9d2 ansible: double the default pool size.
Tempted to push this up to 64, but let's do it incrementally just in
case.
6 years ago
David Wilson b89e53fd70 ansible: raise error with correct exception type. 6 years ago
David Wilson 0e193c223c issue #508: master: minify all Mitogen/ansible_mitogen sources.
Minify-safe files are marked with a magical "# !mitogen: minify_safe"
comment anywhere in the file, which activates the minifier. The result
is naturally cached by ModuleResponder, therefore lru_cache is gone too.

Given:

    import os, mitogen
    @mitogen.main()
    def main(router):
        c = router.ssh(hostname='k3')
        c.call(os.getpid)
        router.sudo(via=c)

SSH footprint drops from 56.2 KiB to 42.75 KiB (-23.9%)
Ansible "shell: hostname" drops 149.26 KiB to 117.42 KiB (-21.3%)
6 years ago
David Wilson cfb94e463f parent: PartialZlib docstrings. 6 years ago
David Wilson 7badb4a25b ansible: hacky parser to alow bools to be specified on command line 6 years ago
David Wilson 9adc38d8ec parent: pre-cache bootstrap if possible.
When the interpreter is modern enough, use zlib.compressobj() to
pre-compress the unchanging parts of the bootstrap once, then use
compressobj.copy() to append just the context's config during stream
construction.

Before: 100 loops, best of 3: 5.81 msec per loop
After: 10000 loops, best of 3: 35.9 usec per loop

With 100 targets this is enough to knock 6 seconds off startup, at 500
targets it becomes half a minute.

Test 'program':
        python -m timeit -s '
                import mitogen.parent as p;
                import mitogen.master as m;
                r=m.Router();
                s=p.Stream(r, 0, max_message_size=1);
                r.broker.shutdown()'\
                \
                's.get_preamble()'
6 years ago
David Wilson e167f6373c docs: update Changelog. 6 years ago
David Wilson b499fbe29b ansible: add mitogen_ssh_compression variable. 6 years ago
David Wilson d6c4a983e1 service: PushFileService never recorded a file as sent.
Ansible modules were being resent continuously - but only the main
script module, and any custom modutils if any were present.

Wire footprint drops by ~1/3rd for a 500 task run of 'shell: hostname':

-rw-r--r-- 1 root root 584K Jan 31 22:06 500mito-before2
-rw-r--r-- 1 root root 434K Jan 31 22:04 500mito-filesbugonly
6 years ago
David Wilson 7ca927608c parent: synchronize get_core_source()
Single task 100 SSH target run, before:

        3533181 function calls (3533083 primitive calls) in 616.688 seconds
        User time (seconds): 32.52
        System time (seconds): 2.71
        Percent of CPU this job got: 64%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:54.88

After:

        451602 function calls (451504 primitive calls) in 570.746 seconds
        User time (seconds): 29.48
        System time (seconds): 2.81
        Percent of CPU this job got: 67%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:48.20
6 years ago
David Wilson 2399a9e621 service: use correct profile aggregation name. 6 years ago
David Wilson a2ae4ed696 SyntaxError. 6 years ago
David Wilson a9d48a8fdc ansible: don't pin controller if <4 cores. 6 years ago
David Wilson b72d139bd2 tests: make soak testing work reliably on vanilla.
It claims to disable host key checking, but it doesn't.
6 years ago
David Wilson 2153e195c3 docs: changelog tidyups. 6 years ago
David Wilson 4531338b12 ansible: document and make affinity stuff portable to non-Linux
Portable as in does nothing for the time at least for now.
6 years ago
David Wilson de5c050707 ansible: fix affinity.py test failure on 2 cores. 6 years ago
David Wilson 00ae90b2b2 ansible: preheat PluginLoader caches before fork.
This has been broken for some time, but somehow it has become noticeable
on recent Ansible.

loop-100-tasks.yml before:
      15.532724001 seconds time elapsed
       8.453850000 seconds user
       5.808627000 seconds sys

loop-100-tasks.yml after:
       8.991635735 seconds time elapsed
       5.059232000 seconds user
       2.578842000 seconds sys
6 years ago
David Wilson b59a47aecb tests: make mitogen_shutdown_all be run_once by default. 6 years ago
David Wilson f06152db97 docs: update Changelog. 6 years ago
David Wilson 7b129e8576 ansible: use Poller for WorkerProcess; closes #491. 6 years ago
David Wilson c6d5aa29ba ansible: new multiplexer/workers configuration
Following on from 152effc26c9a5918cb7ead7a97fe7fa7f81b6764,

* Pin mux to CPU 0
* Pin top-level CPU 1
* Pin workers sequentially to CPU 2..n

Nets 19.5% improvement on issue_140__thread_pileup.yml when targetting
64 Docker containers on the same 8 core/16 thread machine.

Before (prior to last scheme, no affinity at all):

    2294528.731458      task-clock (msec)         #    6.443 CPUs utilized
        10,429,745      context-switches          #    0.005 M/sec
         2,049,618      cpu-migrations            #    0.893 K/sec
         8,258,952      page-faults               #    0.004 M/sec
 5,532,719,253,824      cycles                    #    2.411 GHz                      (83.35%)
 3,267,471,616,230      instructions              #    0.59  insn per cycle
                                                  #    1.22  stalled cycles per insn  (83.35%)
   662,006,455,943      branches                  #  288.515 M/sec                    (83.33%)
    39,453,895,977      branch-misses             #    5.96% of all branches          (83.37%)

     356.148064576 seconds time elapsed

After:

    2226463.958975      task-clock (msec)         #    7.784 CPUs utilized
         9,831,466      context-switches          #    0.004 M/sec
           180,065      cpu-migrations            #    0.081 K/sec
         5,082,278      page-faults               #    0.002 M/sec
 5,592,548,587,259      cycles                    #    2.512 GHz                      (83.35%)
 3,135,038,855,414      instructions              #    0.56  insn per cycle
                                                  #    1.32  stalled cycles per insn  (83.32%)
   636,397,509,232      branches                  #  285.833 M/sec                    (83.30%)
    39,135,441,790      branch-misses             #    6.15% of all branches          (83.35%)

     286.036681644 seconds time elapsed
6 years ago
David Wilson 8f6e6b3940 docs: update Changelog. 6 years ago
David Wilson 90573b1963 docs: update Changelog. 6 years ago