Commit Graph

2316 Commits (e7035220102238c7e823946f3bbd6174dc87b57e)
 

Author SHA1 Message Date
David Wilson e703522010 issue #505: docs: add new detail graph for one scenario. 6 years ago
David Wilson c279454502 docs: update and re-record profile graphs in docs; closes #505
pcaps moved to 'pcaps' branch.
6 years ago
David Wilson 54835d4c9b service: fix PushFileService exception
[costapp]

ERROR! [pid 25135] 21:10:56.284733 E mitogen.ctx.ssh.35.200.203.48: mitogen: While calling no-reply method PushFileService.forward
Traceback (most recent call last):
  File "master:/home/dmw/src/mitogen/mitogen/service.py", line 260, in _invoke
    ret = method(**kwargs)
  File "master:/home/dmw/src/mitogen/mitogen/service.py", line 718, in forward
    self._forward(path, context)
  File "master:/home/dmw/src/mitogen/mitogen/service.py", line 633, in _forward
    stream = self.router.stream_by_id(context.context_id)
AttributeError: 'unicode' object has no attribute 'context_id'
^C [ERROR]: User interrupted execution
6 years ago
David Wilson b2bae9deed tests: pad out localhost-* 6 years ago
David Wilson 23d7a961e7 service: start pool shutdown on broker shutdown. 6 years ago
David Wilson 85cfa3b0f5 master: .encode() needed for Py3. 6 years ago
David Wilson 05b1ccb658 ansible: stash PID files in CWD if requested for debugging. 6 years ago
David Wilson 1d509d03ff issue #508: master: minify_safe_re must be bytes for Py3. 6 years ago
David Wilson fda7d5c276 bench: tidy up and cpu-pin some more files. 6 years ago
David Wilson 29adfbea59 tests: add localhost-x100 6 years ago
David Wilson eb67fbe9d2 ansible: double the default pool size.
Tempted to push this up to 64, but let's do it incrementally just in
case.
6 years ago
David Wilson b89e53fd70 ansible: raise error with correct exception type. 6 years ago
David Wilson 0e193c223c issue #508: master: minify all Mitogen/ansible_mitogen sources.
Minify-safe files are marked with a magical "# !mitogen: minify_safe"
comment anywhere in the file, which activates the minifier. The result
is naturally cached by ModuleResponder, therefore lru_cache is gone too.

Given:

    import os, mitogen
    @mitogen.main()
    def main(router):
        c = router.ssh(hostname='k3')
        c.call(os.getpid)
        router.sudo(via=c)

SSH footprint drops from 56.2 KiB to 42.75 KiB (-23.9%)
Ansible "shell: hostname" drops 149.26 KiB to 117.42 KiB (-21.3%)
6 years ago
David Wilson cfb94e463f parent: PartialZlib docstrings. 6 years ago
David Wilson 7badb4a25b ansible: hacky parser to alow bools to be specified on command line 6 years ago
David Wilson 9adc38d8ec parent: pre-cache bootstrap if possible.
When the interpreter is modern enough, use zlib.compressobj() to
pre-compress the unchanging parts of the bootstrap once, then use
compressobj.copy() to append just the context's config during stream
construction.

Before: 100 loops, best of 3: 5.81 msec per loop
After: 10000 loops, best of 3: 35.9 usec per loop

With 100 targets this is enough to knock 6 seconds off startup, at 500
targets it becomes half a minute.

Test 'program':
        python -m timeit -s '
                import mitogen.parent as p;
                import mitogen.master as m;
                r=m.Router();
                s=p.Stream(r, 0, max_message_size=1);
                r.broker.shutdown()'\
                \
                's.get_preamble()'
6 years ago
David Wilson e167f6373c docs: update Changelog. 6 years ago
David Wilson b499fbe29b ansible: add mitogen_ssh_compression variable. 6 years ago
David Wilson d6c4a983e1 service: PushFileService never recorded a file as sent.
Ansible modules were being resent continuously - but only the main
script module, and any custom modutils if any were present.

Wire footprint drops by ~1/3rd for a 500 task run of 'shell: hostname':

-rw-r--r-- 1 root root 584K Jan 31 22:06 500mito-before2
-rw-r--r-- 1 root root 434K Jan 31 22:04 500mito-filesbugonly
6 years ago
David Wilson 7ca927608c parent: synchronize get_core_source()
Single task 100 SSH target run, before:

        3533181 function calls (3533083 primitive calls) in 616.688 seconds
        User time (seconds): 32.52
        System time (seconds): 2.71
        Percent of CPU this job got: 64%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:54.88

After:

        451602 function calls (451504 primitive calls) in 570.746 seconds
        User time (seconds): 29.48
        System time (seconds): 2.81
        Percent of CPU this job got: 67%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:48.20
6 years ago
David Wilson 2399a9e621 service: use correct profile aggregation name. 6 years ago
David Wilson a2ae4ed696 SyntaxError. 6 years ago
David Wilson a9d48a8fdc ansible: don't pin controller if <4 cores. 6 years ago
David Wilson b72d139bd2 tests: make soak testing work reliably on vanilla.
It claims to disable host key checking, but it doesn't.
6 years ago
David Wilson 2153e195c3 docs: changelog tidyups. 6 years ago
David Wilson 4531338b12 ansible: document and make affinity stuff portable to non-Linux
Portable as in does nothing for the time at least for now.
6 years ago
David Wilson de5c050707 ansible: fix affinity.py test failure on 2 cores. 6 years ago
David Wilson 00ae90b2b2 ansible: preheat PluginLoader caches before fork.
This has been broken for some time, but somehow it has become noticeable
on recent Ansible.

loop-100-tasks.yml before:
      15.532724001 seconds time elapsed
       8.453850000 seconds user
       5.808627000 seconds sys

loop-100-tasks.yml after:
       8.991635735 seconds time elapsed
       5.059232000 seconds user
       2.578842000 seconds sys
6 years ago
David Wilson b59a47aecb tests: make mitogen_shutdown_all be run_once by default. 6 years ago
David Wilson f06152db97 docs: update Changelog. 6 years ago
David Wilson 7b129e8576 ansible: use Poller for WorkerProcess; closes #491. 6 years ago
David Wilson c6d5aa29ba ansible: new multiplexer/workers configuration
Following on from 152effc26c9a5918cb7ead7a97fe7fa7f81b6764,

* Pin mux to CPU 0
* Pin top-level CPU 1
* Pin workers sequentially to CPU 2..n

Nets 19.5% improvement on issue_140__thread_pileup.yml when targetting
64 Docker containers on the same 8 core/16 thread machine.

Before (prior to last scheme, no affinity at all):

    2294528.731458      task-clock (msec)         #    6.443 CPUs utilized
        10,429,745      context-switches          #    0.005 M/sec
         2,049,618      cpu-migrations            #    0.893 K/sec
         8,258,952      page-faults               #    0.004 M/sec
 5,532,719,253,824      cycles                    #    2.411 GHz                      (83.35%)
 3,267,471,616,230      instructions              #    0.59  insn per cycle
                                                  #    1.22  stalled cycles per insn  (83.35%)
   662,006,455,943      branches                  #  288.515 M/sec                    (83.33%)
    39,453,895,977      branch-misses             #    5.96% of all branches          (83.37%)

     356.148064576 seconds time elapsed

After:

    2226463.958975      task-clock (msec)         #    7.784 CPUs utilized
         9,831,466      context-switches          #    0.004 M/sec
           180,065      cpu-migrations            #    0.081 K/sec
         5,082,278      page-faults               #    0.002 M/sec
 5,592,548,587,259      cycles                    #    2.512 GHz                      (83.35%)
 3,135,038,855,414      instructions              #    0.56  insn per cycle
                                                  #    1.32  stalled cycles per insn  (83.32%)
   636,397,509,232      branches                  #  285.833 M/sec                    (83.30%)
    39,135,441,790      branch-misses             #    6.15% of all branches          (83.35%)

     286.036681644 seconds time elapsed
6 years ago
David Wilson 8f6e6b3940 docs: update Changelog. 6 years ago
David Wilson 90573b1963 docs: update Changelog. 6 years ago
David Wilson 1b909e8697 ansible: pin connection multiplexer to a single core
Nets a reliable 8% improvement in issue_140__thread_pileup.yml when
targetting 64 Docker containers on the same 8 core/16 thread machine.

Before:
    2294528.731458      task-clock (msec)         #    6.443 CPUs utilized
        10,429,745      context-switches          #    0.005 M/sec
         2,049,618      cpu-migrations            #    0.893 K/sec
         8,258,952      page-faults               #    0.004 M/sec
 5,532,719,253,824      cycles                    #    2.411 GHz                      (83.35%)
 4,001,276,805,120      stalled-cycles-frontend   #   72.32% frontend cycles idle     (83.30%)
 2,024,159,442,463      stalled-cycles-backend    #   36.59% backend cycles idle      (66.65%)
 3,267,471,616,230      instructions              #    0.59  insn per cycle
                                                  #    1.22  stalled cycles per insn  (83.35%)
   662,006,455,943      branches                  #  288.515 M/sec                    (83.33%)
    39,453,895,977      branch-misses             #    5.96% of all branches          (83.37%)

     356.148064576 seconds time elapsed

After:
    2208247.938562      task-clock (msec)         #    6.735 CPUs utilized
         8,489,840      context-switches          #    0.004 M/sec
         1,432,967      cpu-migrations            #    0.649 K/sec
         7,508,957      page-faults               #    0.003 M/sec
 5,477,293,750,357      cycles                    #    2.480 GHz                      (83.31%)
 3,984,360,350,811      stalled-cycles-frontend   #   72.74% frontend cycles idle     (83.32%)
 1,976,646,418,711      stalled-cycles-backend    #   36.09% backend cycles idle      (66.64%)
 3,196,197,480,792      instructions              #    0.58  insn per cycle
                                                  #    1.25  stalled cycles per insn  (83.36%)
   648,247,332,967      branches                  #  293.557 M/sec                    (83.35%)
    39,004,881,070      branch-misses             #    6.02% of all branches          (83.37%)

     327.876903668 seconds time elapsed
6 years ago
David Wilson e77048ec2d utils: pad out reset_affinity() and integrate with detach_popen() 6 years ago
David Wilson b0e3807b16 utils: import reset_affinity() function. 6 years ago
David Wilson cafdfdb8e2 master: set Router.profiling if MITOGEN_PROFILING variable present. 6 years ago
David Wilson 568044438d parent: don't kill children when profiling is active. 6 years ago
David Wilson e587396e70 ansible: hook strategy and worker processes into profiler 6 years ago
David Wilson 5dc85c05f9 profiler: import from linear2 branch 6 years ago
David Wilson 960392cf50 core: tidy up existing profiling code and support MITOGEN_PROFILE_FMT 6 years ago
David Wilson 3606580f05 issue #260: redundant if statement. 6 years ago
David Wilson 84944a9a61 ansible: ensure MuxProcess MITOGEN_PROFILING results reach disk.
This has been broken for quite some time.
6 years ago
David Wilson 2209d880e4 ansible/bench: make end= configurable. 6 years ago
David Wilson b8a0e0f929 master: cache sent/forwarded module names
On 32x Docker run of issue_140__thread_pileup.yml

Before:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   384807    3.595    0.000   12.207    0.000 /home/dmw/src/mitogen/mitogen/master.py:896(_forward_one_module)
  1218352    4.867    0.000    7.302    0.000 /home/dmw/src/mitogen/mitogen/master.py:853(_send_module_and_related)

After:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   384807    3.839    0.000    6.543    0.000 /home/dmw/src/mitogen/mitogen/master.py:902(_forward_one_module)
  1218352    0.723    0.000    0.898    0.000 /home/dmw/src/mitogen/mitogen/master.py:856(_send_module_and_related)
6 years ago
dw 9a41e17270
Merge pull request #530 from moreati/cover-us
Aggregate code coverage data across all tox runs
6 years ago
Alex Willmer 2b1035b500
Merge branch 'master' into cover-us 6 years ago
Alex Willmer 2fa84e096f Aggregate code coverage data across tox all runs
Fixes #529
6 years ago
David Wilson a6fd080b3a Merge remote-tracking branch 'origin/issue527'
* origin/issue527:
  docs: update Changelog; closes #527.
  issue #527: catch new-style module tracebacks like vanilla.
6 years ago