Commit Graph

3104 Commits (fbb92e461f524f606a25dac7684f39cf5c440b25)
 

Author SHA1 Message Date
David Wilson 0e193c223c issue #508: master: minify all Mitogen/ansible_mitogen sources.
Minify-safe files are marked with a magical "# !mitogen: minify_safe"
comment anywhere in the file, which activates the minifier. The result
is naturally cached by ModuleResponder, therefore lru_cache is gone too.

Given:

    import os, mitogen
    @mitogen.main()
    def main(router):
        c = router.ssh(hostname='k3')
        c.call(os.getpid)
        router.sudo(via=c)

SSH footprint drops from 56.2 KiB to 42.75 KiB (-23.9%)
Ansible "shell: hostname" drops 149.26 KiB to 117.42 KiB (-21.3%)
6 years ago
David Wilson cfb94e463f parent: PartialZlib docstrings. 6 years ago
David Wilson 7badb4a25b ansible: hacky parser to alow bools to be specified on command line 6 years ago
David Wilson 9adc38d8ec parent: pre-cache bootstrap if possible.
When the interpreter is modern enough, use zlib.compressobj() to
pre-compress the unchanging parts of the bootstrap once, then use
compressobj.copy() to append just the context's config during stream
construction.

Before: 100 loops, best of 3: 5.81 msec per loop
After: 10000 loops, best of 3: 35.9 usec per loop

With 100 targets this is enough to knock 6 seconds off startup, at 500
targets it becomes half a minute.

Test 'program':
        python -m timeit -s '
                import mitogen.parent as p;
                import mitogen.master as m;
                r=m.Router();
                s=p.Stream(r, 0, max_message_size=1);
                r.broker.shutdown()'\
                \
                's.get_preamble()'
6 years ago
David Wilson e167f6373c docs: update Changelog. 6 years ago
David Wilson b499fbe29b ansible: add mitogen_ssh_compression variable. 6 years ago
David Wilson d6c4a983e1 service: PushFileService never recorded a file as sent.
Ansible modules were being resent continuously - but only the main
script module, and any custom modutils if any were present.

Wire footprint drops by ~1/3rd for a 500 task run of 'shell: hostname':

-rw-r--r-- 1 root root 584K Jan 31 22:06 500mito-before2
-rw-r--r-- 1 root root 434K Jan 31 22:04 500mito-filesbugonly
6 years ago
David Wilson 7ca927608c parent: synchronize get_core_source()
Single task 100 SSH target run, before:

        3533181 function calls (3533083 primitive calls) in 616.688 seconds
        User time (seconds): 32.52
        System time (seconds): 2.71
        Percent of CPU this job got: 64%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:54.88

After:

        451602 function calls (451504 primitive calls) in 570.746 seconds
        User time (seconds): 29.48
        System time (seconds): 2.81
        Percent of CPU this job got: 67%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:48.20
6 years ago
David Wilson 2399a9e621 service: use correct profile aggregation name. 6 years ago
David Wilson a2ae4ed696 SyntaxError. 6 years ago
David Wilson a9d48a8fdc ansible: don't pin controller if <4 cores. 6 years ago
David Wilson b72d139bd2 tests: make soak testing work reliably on vanilla.
It claims to disable host key checking, but it doesn't.
6 years ago
David Wilson 2153e195c3 docs: changelog tidyups. 6 years ago
David Wilson 4531338b12 ansible: document and make affinity stuff portable to non-Linux
Portable as in does nothing for the time at least for now.
6 years ago
David Wilson de5c050707 ansible: fix affinity.py test failure on 2 cores. 6 years ago
David Wilson 00ae90b2b2 ansible: preheat PluginLoader caches before fork.
This has been broken for some time, but somehow it has become noticeable
on recent Ansible.

loop-100-tasks.yml before:
      15.532724001 seconds time elapsed
       8.453850000 seconds user
       5.808627000 seconds sys

loop-100-tasks.yml after:
       8.991635735 seconds time elapsed
       5.059232000 seconds user
       2.578842000 seconds sys
6 years ago
David Wilson b59a47aecb tests: make mitogen_shutdown_all be run_once by default. 6 years ago
David Wilson f06152db97 docs: update Changelog. 6 years ago
David Wilson 7b129e8576 ansible: use Poller for WorkerProcess; closes #491. 6 years ago
David Wilson c6d5aa29ba ansible: new multiplexer/workers configuration
Following on from 152effc26c9a5918cb7ead7a97fe7fa7f81b6764,

* Pin mux to CPU 0
* Pin top-level CPU 1
* Pin workers sequentially to CPU 2..n

Nets 19.5% improvement on issue_140__thread_pileup.yml when targetting
64 Docker containers on the same 8 core/16 thread machine.

Before (prior to last scheme, no affinity at all):

    2294528.731458      task-clock (msec)         #    6.443 CPUs utilized
        10,429,745      context-switches          #    0.005 M/sec
         2,049,618      cpu-migrations            #    0.893 K/sec
         8,258,952      page-faults               #    0.004 M/sec
 5,532,719,253,824      cycles                    #    2.411 GHz                      (83.35%)
 3,267,471,616,230      instructions              #    0.59  insn per cycle
                                                  #    1.22  stalled cycles per insn  (83.35%)
   662,006,455,943      branches                  #  288.515 M/sec                    (83.33%)
    39,453,895,977      branch-misses             #    5.96% of all branches          (83.37%)

     356.148064576 seconds time elapsed

After:

    2226463.958975      task-clock (msec)         #    7.784 CPUs utilized
         9,831,466      context-switches          #    0.004 M/sec
           180,065      cpu-migrations            #    0.081 K/sec
         5,082,278      page-faults               #    0.002 M/sec
 5,592,548,587,259      cycles                    #    2.512 GHz                      (83.35%)
 3,135,038,855,414      instructions              #    0.56  insn per cycle
                                                  #    1.32  stalled cycles per insn  (83.32%)
   636,397,509,232      branches                  #  285.833 M/sec                    (83.30%)
    39,135,441,790      branch-misses             #    6.15% of all branches          (83.35%)

     286.036681644 seconds time elapsed
6 years ago
David Wilson 8f6e6b3940 docs: update Changelog. 6 years ago
David Wilson 90573b1963 docs: update Changelog. 6 years ago
David Wilson 1b909e8697 ansible: pin connection multiplexer to a single core
Nets a reliable 8% improvement in issue_140__thread_pileup.yml when
targetting 64 Docker containers on the same 8 core/16 thread machine.

Before:
    2294528.731458      task-clock (msec)         #    6.443 CPUs utilized
        10,429,745      context-switches          #    0.005 M/sec
         2,049,618      cpu-migrations            #    0.893 K/sec
         8,258,952      page-faults               #    0.004 M/sec
 5,532,719,253,824      cycles                    #    2.411 GHz                      (83.35%)
 4,001,276,805,120      stalled-cycles-frontend   #   72.32% frontend cycles idle     (83.30%)
 2,024,159,442,463      stalled-cycles-backend    #   36.59% backend cycles idle      (66.65%)
 3,267,471,616,230      instructions              #    0.59  insn per cycle
                                                  #    1.22  stalled cycles per insn  (83.35%)
   662,006,455,943      branches                  #  288.515 M/sec                    (83.33%)
    39,453,895,977      branch-misses             #    5.96% of all branches          (83.37%)

     356.148064576 seconds time elapsed

After:
    2208247.938562      task-clock (msec)         #    6.735 CPUs utilized
         8,489,840      context-switches          #    0.004 M/sec
         1,432,967      cpu-migrations            #    0.649 K/sec
         7,508,957      page-faults               #    0.003 M/sec
 5,477,293,750,357      cycles                    #    2.480 GHz                      (83.31%)
 3,984,360,350,811      stalled-cycles-frontend   #   72.74% frontend cycles idle     (83.32%)
 1,976,646,418,711      stalled-cycles-backend    #   36.09% backend cycles idle      (66.64%)
 3,196,197,480,792      instructions              #    0.58  insn per cycle
                                                  #    1.25  stalled cycles per insn  (83.36%)
   648,247,332,967      branches                  #  293.557 M/sec                    (83.35%)
    39,004,881,070      branch-misses             #    6.02% of all branches          (83.37%)

     327.876903668 seconds time elapsed
6 years ago
David Wilson e77048ec2d utils: pad out reset_affinity() and integrate with detach_popen() 6 years ago
David Wilson b0e3807b16 utils: import reset_affinity() function. 6 years ago
David Wilson cafdfdb8e2 master: set Router.profiling if MITOGEN_PROFILING variable present. 6 years ago
David Wilson 568044438d parent: don't kill children when profiling is active. 6 years ago
David Wilson e587396e70 ansible: hook strategy and worker processes into profiler 6 years ago
David Wilson 5dc85c05f9 profiler: import from linear2 branch 6 years ago
David Wilson 960392cf50 core: tidy up existing profiling code and support MITOGEN_PROFILE_FMT 6 years ago
David Wilson 3606580f05 issue #260: redundant if statement. 6 years ago
David Wilson 84944a9a61 ansible: ensure MuxProcess MITOGEN_PROFILING results reach disk.
This has been broken for quite some time.
6 years ago
David Wilson 2209d880e4 ansible/bench: make end= configurable. 6 years ago
David Wilson b8a0e0f929 master: cache sent/forwarded module names
On 32x Docker run of issue_140__thread_pileup.yml

Before:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   384807    3.595    0.000   12.207    0.000 /home/dmw/src/mitogen/mitogen/master.py:896(_forward_one_module)
  1218352    4.867    0.000    7.302    0.000 /home/dmw/src/mitogen/mitogen/master.py:853(_send_module_and_related)

After:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   384807    3.839    0.000    6.543    0.000 /home/dmw/src/mitogen/mitogen/master.py:902(_forward_one_module)
  1218352    0.723    0.000    0.898    0.000 /home/dmw/src/mitogen/mitogen/master.py:856(_send_module_and_related)
6 years ago
dw 9a41e17270
Merge pull request #530 from moreati/cover-us
Aggregate code coverage data across all tox runs
6 years ago
Alex Willmer 2b1035b500
Merge branch 'master' into cover-us 6 years ago
Alex Willmer 2fa84e096f Aggregate code coverage data across tox all runs
Fixes #529
6 years ago
David Wilson a6fd080b3a Merge remote-tracking branch 'origin/issue527'
* origin/issue527:
  docs: update Changelog; closes #527.
  issue #527: catch new-style module tracebacks like vanilla.
6 years ago
Alex Willmer e7d9ec86f9 Allow independant control of coverage erase and reporting 6 years ago
Alex Willmer 6f17ede28b Fix incorrect attempt to use coverage
Looks like this was a copy and paste mistake
6 years ago
David Wilson 8a2dcbf674 docs: update Changelog; closes #527. 6 years ago
David Wilson 954f874085 issue #527: catch new-style module tracebacks like vanilla. 6 years ago
dw e8e66d6fd0
Merge pull request #526 from moreati/py36-deprecationwarning
Py36 deprecationwarning
6 years ago
dw ea83a1b677
Merge pull request #525 from moreati/coverage_report_always
Generate coverage report even if some tests fail
6 years ago
Alex Willmer 0d725f9cd8 Fix DeprecationWarning in mitogen.utils.run_with_router()
```
14:20:32.186270 W py.warnings:
/home/alex/src/mitogen/mitogen/utils.py:152: DeprecationWarning: invalid
escape sequence \*
  """
```
6 years ago
Alex Willmer e7d70b109b Generate coverage report even if some tests fail
`set -o errexit` was exiting run_tests prematurely if any test failed,
so the coverage report was not always generated.
6 years ago
David Wilson d763570ca2 Merge remote-tracking branch 'origin/dmw'
* origin/dmw:
  ci: fix incorrect partition/rpartition from 8a4caea84f
  issue #260: hide force-disconnect messages.
  issue #498: fix shutdown crash
  issue #260: avoid start_transmit()/on_transmit()/stop_transmit()
  core: ensure broker profiling output reaches disk
  master: keep is_stdlib_path() result as negative cache entry
  ci: Allow DISTROS="debian*32" variable, and KEEP=1
6 years ago
David Wilson 49a8745a45 ci: fix incorrect partition/rpartition from 8a4caea84f 6 years ago
David Wilson 696cee57dd issue #260: hide force-disconnect messages.
Parent is always force-disconnected, and now it shows up quite visibly
on every exit.
6 years ago
David Wilson 26c6c6d048 issue #498: fix shutdown crash
Traceback (most recent call last):
      File "<stdin>", line 2707, in _invoke
      File "<stdin>", line 2480, in _on_del_route
    NameError: global name 'target_id' is not defined
6 years ago