23d7a961e7
service: start pool shutdown on broker shutdown.
6 years ago
85cfa3b0f5
master: .encode() needed for Py3.
6 years ago
05b1ccb658
ansible: stash PID files in CWD if requested for debugging.
6 years ago
1d509d03ff
issue #508 : master: minify_safe_re must be bytes for Py3.
6 years ago
fda7d5c276
bench: tidy up and cpu-pin some more files.
6 years ago
29adfbea59
tests: add localhost-x100
6 years ago
eb67fbe9d2
ansible: double the default pool size.
...
Tempted to push this up to 64, but let's do it incrementally just in
case.
6 years ago
b89e53fd70
ansible: raise error with correct exception type.
6 years ago
0e193c223c
issue #508 : master: minify all Mitogen/ansible_mitogen sources.
...
Minify-safe files are marked with a magical "# !mitogen: minify_safe"
comment anywhere in the file, which activates the minifier. The result
is naturally cached by ModuleResponder, therefore lru_cache is gone too.
Given:
import os, mitogen
@mitogen.main()
def main(router):
c = router.ssh(hostname='k3')
c.call(os.getpid)
router.sudo(via=c)
SSH footprint drops from 56.2 KiB to 42.75 KiB (-23.9%)
Ansible "shell: hostname" drops 149.26 KiB to 117.42 KiB (-21.3%)
6 years ago
cfb94e463f
parent: PartialZlib docstrings.
6 years ago
7badb4a25b
ansible: hacky parser to alow bools to be specified on command line
6 years ago
9adc38d8ec
parent: pre-cache bootstrap if possible.
...
When the interpreter is modern enough, use zlib.compressobj() to
pre-compress the unchanging parts of the bootstrap once, then use
compressobj.copy() to append just the context's config during stream
construction.
Before: 100 loops, best of 3: 5.81 msec per loop
After: 10000 loops, best of 3: 35.9 usec per loop
With 100 targets this is enough to knock 6 seconds off startup, at 500
targets it becomes half a minute.
Test 'program':
python -m timeit -s '
import mitogen.parent as p;
import mitogen.master as m;
r=m.Router();
s=p.Stream(r, 0, max_message_size=1);
r.broker.shutdown()'\
\
's.get_preamble()'
6 years ago
e167f6373c
docs: update Changelog.
6 years ago
b499fbe29b
ansible: add mitogen_ssh_compression variable.
6 years ago
d6c4a983e1
service: PushFileService never recorded a file as sent.
...
Ansible modules were being resent continuously - but only the main
script module, and any custom modutils if any were present.
Wire footprint drops by ~1/3rd for a 500 task run of 'shell: hostname':
-rw-r--r-- 1 root root 584K Jan 31 22:06 500mito-before2
-rw-r--r-- 1 root root 434K Jan 31 22:04 500mito-filesbugonly
6 years ago
7ca927608c
parent: synchronize get_core_source()
...
Single task 100 SSH target run, before:
3533181 function calls (3533083 primitive calls) in 616.688 seconds
User time (seconds): 32.52
System time (seconds): 2.71
Percent of CPU this job got: 64%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:54.88
After:
451602 function calls (451504 primitive calls) in 570.746 seconds
User time (seconds): 29.48
System time (seconds): 2.81
Percent of CPU this job got: 67%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:48.20
6 years ago
2399a9e621
service: use correct profile aggregation name.
6 years ago
a2ae4ed696
SyntaxError.
6 years ago
a9d48a8fdc
ansible: don't pin controller if <4 cores.
6 years ago
b72d139bd2
tests: make soak testing work reliably on vanilla.
...
It claims to disable host key checking, but it doesn't.
6 years ago
2153e195c3
docs: changelog tidyups.
6 years ago
4531338b12
ansible: document and make affinity stuff portable to non-Linux
...
Portable as in does nothing for the time at least for now.
6 years ago
de5c050707
ansible: fix affinity.py test failure on 2 cores.
6 years ago
00ae90b2b2
ansible: preheat PluginLoader caches before fork.
...
This has been broken for some time, but somehow it has become noticeable
on recent Ansible.
loop-100-tasks.yml before:
15.532724001 seconds time elapsed
8.453850000 seconds user
5.808627000 seconds sys
loop-100-tasks.yml after:
8.991635735 seconds time elapsed
5.059232000 seconds user
2.578842000 seconds sys
6 years ago
b59a47aecb
tests: make mitogen_shutdown_all be run_once by default.
6 years ago
f06152db97
docs: update Changelog.
6 years ago
7b129e8576
ansible: use Poller for WorkerProcess; closes #491 .
6 years ago
c6d5aa29ba
ansible: new multiplexer/workers configuration
...
Following on from 152effc26c9a5918cb7ead7a97fe7fa7f81b6764,
* Pin mux to CPU 0
* Pin top-level CPU 1
* Pin workers sequentially to CPU 2..n
Nets 19.5% improvement on issue_140__thread_pileup.yml when targetting
64 Docker containers on the same 8 core/16 thread machine.
Before (prior to last scheme, no affinity at all):
2294528.731458 task-clock (msec) # 6.443 CPUs utilized
10,429,745 context-switches # 0.005 M/sec
2,049,618 cpu-migrations # 0.893 K/sec
8,258,952 page-faults # 0.004 M/sec
5,532,719,253,824 cycles # 2.411 GHz (83.35%)
3,267,471,616,230 instructions # 0.59 insn per cycle
# 1.22 stalled cycles per insn (83.35%)
662,006,455,943 branches # 288.515 M/sec (83.33%)
39,453,895,977 branch-misses # 5.96% of all branches (83.37%)
356.148064576 seconds time elapsed
After:
2226463.958975 task-clock (msec) # 7.784 CPUs utilized
9,831,466 context-switches # 0.004 M/sec
180,065 cpu-migrations # 0.081 K/sec
5,082,278 page-faults # 0.002 M/sec
5,592,548,587,259 cycles # 2.512 GHz (83.35%)
3,135,038,855,414 instructions # 0.56 insn per cycle
# 1.32 stalled cycles per insn (83.32%)
636,397,509,232 branches # 285.833 M/sec (83.30%)
39,135,441,790 branch-misses # 6.15% of all branches (83.35%)
286.036681644 seconds time elapsed
6 years ago
8f6e6b3940
docs: update Changelog.
6 years ago
90573b1963
docs: update Changelog.
6 years ago
1b909e8697
ansible: pin connection multiplexer to a single core
...
Nets a reliable 8% improvement in issue_140__thread_pileup.yml when
targetting 64 Docker containers on the same 8 core/16 thread machine.
Before:
2294528.731458 task-clock (msec) # 6.443 CPUs utilized
10,429,745 context-switches # 0.005 M/sec
2,049,618 cpu-migrations # 0.893 K/sec
8,258,952 page-faults # 0.004 M/sec
5,532,719,253,824 cycles # 2.411 GHz (83.35%)
4,001,276,805,120 stalled-cycles-frontend # 72.32% frontend cycles idle (83.30%)
2,024,159,442,463 stalled-cycles-backend # 36.59% backend cycles idle (66.65%)
3,267,471,616,230 instructions # 0.59 insn per cycle
# 1.22 stalled cycles per insn (83.35%)
662,006,455,943 branches # 288.515 M/sec (83.33%)
39,453,895,977 branch-misses # 5.96% of all branches (83.37%)
356.148064576 seconds time elapsed
After:
2208247.938562 task-clock (msec) # 6.735 CPUs utilized
8,489,840 context-switches # 0.004 M/sec
1,432,967 cpu-migrations # 0.649 K/sec
7,508,957 page-faults # 0.003 M/sec
5,477,293,750,357 cycles # 2.480 GHz (83.31%)
3,984,360,350,811 stalled-cycles-frontend # 72.74% frontend cycles idle (83.32%)
1,976,646,418,711 stalled-cycles-backend # 36.09% backend cycles idle (66.64%)
3,196,197,480,792 instructions # 0.58 insn per cycle
# 1.25 stalled cycles per insn (83.36%)
648,247,332,967 branches # 293.557 M/sec (83.35%)
39,004,881,070 branch-misses # 6.02% of all branches (83.37%)
327.876903668 seconds time elapsed
6 years ago
e77048ec2d
utils: pad out reset_affinity() and integrate with detach_popen()
6 years ago
b0e3807b16
utils: import reset_affinity() function.
6 years ago
cafdfdb8e2
master: set Router.profiling if MITOGEN_PROFILING variable present.
6 years ago
568044438d
parent: don't kill children when profiling is active.
6 years ago
e587396e70
ansible: hook strategy and worker processes into profiler
6 years ago
5dc85c05f9
profiler: import from linear2 branch
6 years ago
960392cf50
core: tidy up existing profiling code and support MITOGEN_PROFILE_FMT
6 years ago
3606580f05
issue #260 : redundant if statement.
6 years ago
84944a9a61
ansible: ensure MuxProcess MITOGEN_PROFILING results reach disk.
...
This has been broken for quite some time.
6 years ago
2209d880e4
ansible/bench: make end= configurable.
6 years ago
b8a0e0f929
master: cache sent/forwarded module names
...
On 32x Docker run of issue_140__thread_pileup.yml
Before:
ncalls tottime percall cumtime percall filename:lineno(function)
384807 3.595 0.000 12.207 0.000 /home/dmw/src/mitogen/mitogen/master.py:896(_forward_one_module)
1218352 4.867 0.000 7.302 0.000 /home/dmw/src/mitogen/mitogen/master.py:853(_send_module_and_related)
After:
ncalls tottime percall cumtime percall filename:lineno(function)
384807 3.839 0.000 6.543 0.000 /home/dmw/src/mitogen/mitogen/master.py:902(_forward_one_module)
1218352 0.723 0.000 0.898 0.000 /home/dmw/src/mitogen/mitogen/master.py:856(_send_module_and_related)
6 years ago
9a41e17270
Merge pull request #530 from moreati/cover-us
...
Aggregate code coverage data across all tox runs
6 years ago
2b1035b500
Merge branch 'master' into cover-us
6 years ago
2fa84e096f
Aggregate code coverage data across tox all runs
...
Fixes #529
6 years ago
a6fd080b3a
Merge remote-tracking branch 'origin/issue527'
...
* origin/issue527:
docs: update Changelog; closes #527 .
issue #527 : catch new-style module tracebacks like vanilla.
6 years ago
e7d9ec86f9
Allow independant control of coverage erase and reporting
6 years ago
6f17ede28b
Fix incorrect attempt to use coverage
...
Looks like this was a copy and paste mistake
6 years ago
8a2dcbf674
docs: update Changelog; closes #527 .
6 years ago
954f874085
issue #527 : catch new-style module tracebacks like vanilla.
6 years ago
David Wilson