7badb4a25b
ansible: hacky parser to alow bools to be specified on command line
6 years ago
9adc38d8ec
parent: pre-cache bootstrap if possible.
...
When the interpreter is modern enough, use zlib.compressobj() to
pre-compress the unchanging parts of the bootstrap once, then use
compressobj.copy() to append just the context's config during stream
construction.
Before: 100 loops, best of 3: 5.81 msec per loop
After: 10000 loops, best of 3: 35.9 usec per loop
With 100 targets this is enough to knock 6 seconds off startup, at 500
targets it becomes half a minute.
Test 'program':
python -m timeit -s '
import mitogen.parent as p;
import mitogen.master as m;
r=m.Router();
s=p.Stream(r, 0, max_message_size=1);
r.broker.shutdown()'\
\
's.get_preamble()'
6 years ago
e167f6373c
docs: update Changelog.
6 years ago
b499fbe29b
ansible: add mitogen_ssh_compression variable.
6 years ago
d6c4a983e1
service: PushFileService never recorded a file as sent.
...
Ansible modules were being resent continuously - but only the main
script module, and any custom modutils if any were present.
Wire footprint drops by ~1/3rd for a 500 task run of 'shell: hostname':
-rw-r--r-- 1 root root 584K Jan 31 22:06 500mito-before2
-rw-r--r-- 1 root root 434K Jan 31 22:04 500mito-filesbugonly
6 years ago
7ca927608c
parent: synchronize get_core_source()
...
Single task 100 SSH target run, before:
3533181 function calls (3533083 primitive calls) in 616.688 seconds
User time (seconds): 32.52
System time (seconds): 2.71
Percent of CPU this job got: 64%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:54.88
After:
451602 function calls (451504 primitive calls) in 570.746 seconds
User time (seconds): 29.48
System time (seconds): 2.81
Percent of CPU this job got: 67%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:48.20
6 years ago
2399a9e621
service: use correct profile aggregation name.
6 years ago
a2ae4ed696
SyntaxError.
6 years ago
a9d48a8fdc
ansible: don't pin controller if <4 cores.
6 years ago
b72d139bd2
tests: make soak testing work reliably on vanilla.
...
It claims to disable host key checking, but it doesn't.
6 years ago
2153e195c3
docs: changelog tidyups.
6 years ago
4531338b12
ansible: document and make affinity stuff portable to non-Linux
...
Portable as in does nothing for the time at least for now.
6 years ago
de5c050707
ansible: fix affinity.py test failure on 2 cores.
6 years ago
00ae90b2b2
ansible: preheat PluginLoader caches before fork.
...
This has been broken for some time, but somehow it has become noticeable
on recent Ansible.
loop-100-tasks.yml before:
15.532724001 seconds time elapsed
8.453850000 seconds user
5.808627000 seconds sys
loop-100-tasks.yml after:
8.991635735 seconds time elapsed
5.059232000 seconds user
2.578842000 seconds sys
6 years ago
b59a47aecb
tests: make mitogen_shutdown_all be run_once by default.
6 years ago
f06152db97
docs: update Changelog.
6 years ago
7b129e8576
ansible: use Poller for WorkerProcess; closes #491 .
6 years ago
c6d5aa29ba
ansible: new multiplexer/workers configuration
...
Following on from 152effc26c9a5918cb7ead7a97fe7fa7f81b6764,
* Pin mux to CPU 0
* Pin top-level CPU 1
* Pin workers sequentially to CPU 2..n
Nets 19.5% improvement on issue_140__thread_pileup.yml when targetting
64 Docker containers on the same 8 core/16 thread machine.
Before (prior to last scheme, no affinity at all):
2294528.731458 task-clock (msec) # 6.443 CPUs utilized
10,429,745 context-switches # 0.005 M/sec
2,049,618 cpu-migrations # 0.893 K/sec
8,258,952 page-faults # 0.004 M/sec
5,532,719,253,824 cycles # 2.411 GHz (83.35%)
3,267,471,616,230 instructions # 0.59 insn per cycle
# 1.22 stalled cycles per insn (83.35%)
662,006,455,943 branches # 288.515 M/sec (83.33%)
39,453,895,977 branch-misses # 5.96% of all branches (83.37%)
356.148064576 seconds time elapsed
After:
2226463.958975 task-clock (msec) # 7.784 CPUs utilized
9,831,466 context-switches # 0.004 M/sec
180,065 cpu-migrations # 0.081 K/sec
5,082,278 page-faults # 0.002 M/sec
5,592,548,587,259 cycles # 2.512 GHz (83.35%)
3,135,038,855,414 instructions # 0.56 insn per cycle
# 1.32 stalled cycles per insn (83.32%)
636,397,509,232 branches # 285.833 M/sec (83.30%)
39,135,441,790 branch-misses # 6.15% of all branches (83.35%)
286.036681644 seconds time elapsed
6 years ago
8f6e6b3940
docs: update Changelog.
6 years ago
90573b1963
docs: update Changelog.
6 years ago
1b909e8697
ansible: pin connection multiplexer to a single core
...
Nets a reliable 8% improvement in issue_140__thread_pileup.yml when
targetting 64 Docker containers on the same 8 core/16 thread machine.
Before:
2294528.731458 task-clock (msec) # 6.443 CPUs utilized
10,429,745 context-switches # 0.005 M/sec
2,049,618 cpu-migrations # 0.893 K/sec
8,258,952 page-faults # 0.004 M/sec
5,532,719,253,824 cycles # 2.411 GHz (83.35%)
4,001,276,805,120 stalled-cycles-frontend # 72.32% frontend cycles idle (83.30%)
2,024,159,442,463 stalled-cycles-backend # 36.59% backend cycles idle (66.65%)
3,267,471,616,230 instructions # 0.59 insn per cycle
# 1.22 stalled cycles per insn (83.35%)
662,006,455,943 branches # 288.515 M/sec (83.33%)
39,453,895,977 branch-misses # 5.96% of all branches (83.37%)
356.148064576 seconds time elapsed
After:
2208247.938562 task-clock (msec) # 6.735 CPUs utilized
8,489,840 context-switches # 0.004 M/sec
1,432,967 cpu-migrations # 0.649 K/sec
7,508,957 page-faults # 0.003 M/sec
5,477,293,750,357 cycles # 2.480 GHz (83.31%)
3,984,360,350,811 stalled-cycles-frontend # 72.74% frontend cycles idle (83.32%)
1,976,646,418,711 stalled-cycles-backend # 36.09% backend cycles idle (66.64%)
3,196,197,480,792 instructions # 0.58 insn per cycle
# 1.25 stalled cycles per insn (83.36%)
648,247,332,967 branches # 293.557 M/sec (83.35%)
39,004,881,070 branch-misses # 6.02% of all branches (83.37%)
327.876903668 seconds time elapsed
6 years ago
e77048ec2d
utils: pad out reset_affinity() and integrate with detach_popen()
6 years ago
b0e3807b16
utils: import reset_affinity() function.
6 years ago
cafdfdb8e2
master: set Router.profiling if MITOGEN_PROFILING variable present.
6 years ago
568044438d
parent: don't kill children when profiling is active.
6 years ago
e587396e70
ansible: hook strategy and worker processes into profiler
6 years ago
5dc85c05f9
profiler: import from linear2 branch
6 years ago
960392cf50
core: tidy up existing profiling code and support MITOGEN_PROFILE_FMT
6 years ago
3606580f05
issue #260 : redundant if statement.
6 years ago
84944a9a61
ansible: ensure MuxProcess MITOGEN_PROFILING results reach disk.
...
This has been broken for quite some time.
6 years ago
2209d880e4
ansible/bench: make end= configurable.
6 years ago
b8a0e0f929
master: cache sent/forwarded module names
...
On 32x Docker run of issue_140__thread_pileup.yml
Before:
ncalls tottime percall cumtime percall filename:lineno(function)
384807 3.595 0.000 12.207 0.000 /home/dmw/src/mitogen/mitogen/master.py:896(_forward_one_module)
1218352 4.867 0.000 7.302 0.000 /home/dmw/src/mitogen/mitogen/master.py:853(_send_module_and_related)
After:
ncalls tottime percall cumtime percall filename:lineno(function)
384807 3.839 0.000 6.543 0.000 /home/dmw/src/mitogen/mitogen/master.py:902(_forward_one_module)
1218352 0.723 0.000 0.898 0.000 /home/dmw/src/mitogen/mitogen/master.py:856(_send_module_and_related)
6 years ago
9a41e17270
Merge pull request #530 from moreati/cover-us
...
Aggregate code coverage data across all tox runs
6 years ago
2b1035b500
Merge branch 'master' into cover-us
6 years ago
2fa84e096f
Aggregate code coverage data across tox all runs
...
Fixes #529
6 years ago
a6fd080b3a
Merge remote-tracking branch 'origin/issue527'
...
* origin/issue527:
docs: update Changelog; closes #527 .
issue #527 : catch new-style module tracebacks like vanilla.
6 years ago
e7d9ec86f9
Allow independant control of coverage erase and reporting
6 years ago
6f17ede28b
Fix incorrect attempt to use coverage
...
Looks like this was a copy and paste mistake
6 years ago
8a2dcbf674
docs: update Changelog; closes #527 .
6 years ago
954f874085
issue #527 : catch new-style module tracebacks like vanilla.
6 years ago
e8e66d6fd0
Merge pull request #526 from moreati/py36-deprecationwarning
...
Py36 deprecationwarning
6 years ago
ea83a1b677
Merge pull request #525 from moreati/coverage_report_always
...
Generate coverage report even if some tests fail
6 years ago
0d725f9cd8
Fix DeprecationWarning in mitogen.utils.run_with_router()
...
```
14:20:32.186270 W py.warnings:
/home/alex/src/mitogen/mitogen/utils.py:152: DeprecationWarning: invalid
escape sequence \*
"""
```
6 years ago
e7d70b109b
Generate coverage report even if some tests fail
...
`set -o errexit` was exiting run_tests prematurely if any test failed,
so the coverage report was not always generated.
6 years ago
d763570ca2
Merge remote-tracking branch 'origin/dmw'
...
* origin/dmw:
ci: fix incorrect partition/rpartition from 8a4caea84f#260 : hide force-disconnect messages.
issue #498 : fix shutdown crash
issue #260 : avoid start_transmit()/on_transmit()/stop_transmit()
core: ensure broker profiling output reaches disk
master: keep is_stdlib_path() result as negative cache entry
ci: Allow DISTROS="debian*32" variable, and KEEP=1
6 years ago
49a8745a45
ci: fix incorrect partition/rpartition from 8a4caea84f
6 years ago
696cee57dd
issue #260 : hide force-disconnect messages.
...
Parent is always force-disconnected, and now it shows up quite visibly
on every exit.
6 years ago
26c6c6d048
issue #498 : fix shutdown crash
...
Traceback (most recent call last):
File "<stdin>", line 2707, in _invoke
File "<stdin>", line 2480, in _on_del_route
NameError: global name 'target_id' is not defined
6 years ago
a18a083c94
issue #260 : avoid start_transmit()/on_transmit()/stop_transmit()
...
Previous transmit sequence was:
Router._async_route -> Stream._send -> Broker._start_transmit ->
Broker.loop -> Stream.on_transmit -> socket.write ->
Broker.stop_transmit
New sequence, when socket buffer can hold message is:
Router._async_route -> Stream._send -> socket.write
bench/roundtrip.py
Before: 240 usec
after: 178 usec
Stat before:
5088.276050 task-clock (msec) # 0.997 CPUs utilized
185,568 context-switches # 0.036 M/sec
0 cpu-migrations # 0.000 K/sec
18,923 page-faults # 0.004 M/sec
13,063,871,501 cycles # 2.567 GHz
12,834,579,684 instructions # 0.98 insn per cycle
2,669,820,684 branches # 524.700 M/sec
107,296,033 branch-misses # 4.02% of all branches
5.105018296 seconds time elapsed
2.350970000 seconds user
0.345497000 seconds sys
Stat after:
4019.208047 task-clock (msec) # 0.998 CPUs utilized
249,471 context-switches # 0.062 M/sec
0 cpu-migrations # 0.000 K/sec
20,990 page-faults # 0.005 M/sec
10,312,535,979 cycles # 2.566 GHz
11,586,365,995 instructions # 1.12 insn per cycle
2,392,933,370 branches # 595.374 M/sec
75,432,205 branch-misses # 3.15% of all branches
4.028763347 seconds time elapsed
3.367051000 seconds user
0.652962000 seconds sys
6 years ago
b161408598
Merge commit 'fe74577'
...
* commit 'fe74577':
Use develop mode in tox
issue #429 : fix sudo regression.
misc: rename to scripts. tab completion!!
core: Latch._wake improvements
issue #498 : prevent crash on double 'disconnect' signal.
issue #413 : don't double-propagate DEL_ROUTE to parent.
issue #498 : wrap Router dict mutations in a lock
issue #429 : enable en_US locale to unbreak debops test.
issue #499 : fix another mind-numbingly stupid vanilla inconsistency
issue #497 : do our best to cope with crap upstream code
ssh: fix test to match updated log format.
issue #429 : update Changelog.
issue #429 : update Changelog.
issue #429 : teach sudo about every know i18n password string.
issue #429 : install i18n-related bits in test images.
ssh: tidy up logs and stream names.
tests: ensure file is closed in connection_test.
gcloud: small updates
tests: give ansible/gcloud/ its own requirements file.
6 years ago
David Wilson