00ae90b2b2
ansible: preheat PluginLoader caches before fork.
...
This has been broken for some time, but somehow it has become noticeable
on recent Ansible.
loop-100-tasks.yml before:
15.532724001 seconds time elapsed
8.453850000 seconds user
5.808627000 seconds sys
loop-100-tasks.yml after:
8.991635735 seconds time elapsed
5.059232000 seconds user
2.578842000 seconds sys
6 years ago
b59a47aecb
tests: make mitogen_shutdown_all be run_once by default.
6 years ago
f06152db97
docs: update Changelog.
6 years ago
7b129e8576
ansible: use Poller for WorkerProcess; closes #491 .
6 years ago
c6d5aa29ba
ansible: new multiplexer/workers configuration
...
Following on from 152effc26c9a5918cb7ead7a97fe7fa7f81b6764,
* Pin mux to CPU 0
* Pin top-level CPU 1
* Pin workers sequentially to CPU 2..n
Nets 19.5% improvement on issue_140__thread_pileup.yml when targetting
64 Docker containers on the same 8 core/16 thread machine.
Before (prior to last scheme, no affinity at all):
2294528.731458 task-clock (msec) # 6.443 CPUs utilized
10,429,745 context-switches # 0.005 M/sec
2,049,618 cpu-migrations # 0.893 K/sec
8,258,952 page-faults # 0.004 M/sec
5,532,719,253,824 cycles # 2.411 GHz (83.35%)
3,267,471,616,230 instructions # 0.59 insn per cycle
# 1.22 stalled cycles per insn (83.35%)
662,006,455,943 branches # 288.515 M/sec (83.33%)
39,453,895,977 branch-misses # 5.96% of all branches (83.37%)
356.148064576 seconds time elapsed
After:
2226463.958975 task-clock (msec) # 7.784 CPUs utilized
9,831,466 context-switches # 0.004 M/sec
180,065 cpu-migrations # 0.081 K/sec
5,082,278 page-faults # 0.002 M/sec
5,592,548,587,259 cycles # 2.512 GHz (83.35%)
3,135,038,855,414 instructions # 0.56 insn per cycle
# 1.32 stalled cycles per insn (83.32%)
636,397,509,232 branches # 285.833 M/sec (83.30%)
39,135,441,790 branch-misses # 6.15% of all branches (83.35%)
286.036681644 seconds time elapsed
6 years ago
8f6e6b3940
docs: update Changelog.
6 years ago
90573b1963
docs: update Changelog.
6 years ago
1b909e8697
ansible: pin connection multiplexer to a single core
...
Nets a reliable 8% improvement in issue_140__thread_pileup.yml when
targetting 64 Docker containers on the same 8 core/16 thread machine.
Before:
2294528.731458 task-clock (msec) # 6.443 CPUs utilized
10,429,745 context-switches # 0.005 M/sec
2,049,618 cpu-migrations # 0.893 K/sec
8,258,952 page-faults # 0.004 M/sec
5,532,719,253,824 cycles # 2.411 GHz (83.35%)
4,001,276,805,120 stalled-cycles-frontend # 72.32% frontend cycles idle (83.30%)
2,024,159,442,463 stalled-cycles-backend # 36.59% backend cycles idle (66.65%)
3,267,471,616,230 instructions # 0.59 insn per cycle
# 1.22 stalled cycles per insn (83.35%)
662,006,455,943 branches # 288.515 M/sec (83.33%)
39,453,895,977 branch-misses # 5.96% of all branches (83.37%)
356.148064576 seconds time elapsed
After:
2208247.938562 task-clock (msec) # 6.735 CPUs utilized
8,489,840 context-switches # 0.004 M/sec
1,432,967 cpu-migrations # 0.649 K/sec
7,508,957 page-faults # 0.003 M/sec
5,477,293,750,357 cycles # 2.480 GHz (83.31%)
3,984,360,350,811 stalled-cycles-frontend # 72.74% frontend cycles idle (83.32%)
1,976,646,418,711 stalled-cycles-backend # 36.09% backend cycles idle (66.64%)
3,196,197,480,792 instructions # 0.58 insn per cycle
# 1.25 stalled cycles per insn (83.36%)
648,247,332,967 branches # 293.557 M/sec (83.35%)
39,004,881,070 branch-misses # 6.02% of all branches (83.37%)
327.876903668 seconds time elapsed
6 years ago
e77048ec2d
utils: pad out reset_affinity() and integrate with detach_popen()
6 years ago
b0e3807b16
utils: import reset_affinity() function.
6 years ago
cafdfdb8e2
master: set Router.profiling if MITOGEN_PROFILING variable present.
6 years ago
568044438d
parent: don't kill children when profiling is active.
6 years ago
e587396e70
ansible: hook strategy and worker processes into profiler
6 years ago
5dc85c05f9
profiler: import from linear2 branch
6 years ago
960392cf50
core: tidy up existing profiling code and support MITOGEN_PROFILE_FMT
6 years ago
3606580f05
issue #260 : redundant if statement.
6 years ago
84944a9a61
ansible: ensure MuxProcess MITOGEN_PROFILING results reach disk.
...
This has been broken for quite some time.
6 years ago
2209d880e4
ansible/bench: make end= configurable.
6 years ago
b8a0e0f929
master: cache sent/forwarded module names
...
On 32x Docker run of issue_140__thread_pileup.yml
Before:
ncalls tottime percall cumtime percall filename:lineno(function)
384807 3.595 0.000 12.207 0.000 /home/dmw/src/mitogen/mitogen/master.py:896(_forward_one_module)
1218352 4.867 0.000 7.302 0.000 /home/dmw/src/mitogen/mitogen/master.py:853(_send_module_and_related)
After:
ncalls tottime percall cumtime percall filename:lineno(function)
384807 3.839 0.000 6.543 0.000 /home/dmw/src/mitogen/mitogen/master.py:902(_forward_one_module)
1218352 0.723 0.000 0.898 0.000 /home/dmw/src/mitogen/mitogen/master.py:856(_send_module_and_related)
6 years ago
9a41e17270
Merge pull request #530 from moreati/cover-us
...
Aggregate code coverage data across all tox runs
6 years ago
2b1035b500
Merge branch 'master' into cover-us
6 years ago
2fa84e096f
Aggregate code coverage data across tox all runs
...
Fixes #529
6 years ago
a6fd080b3a
Merge remote-tracking branch 'origin/issue527'
...
* origin/issue527:
docs: update Changelog; closes #527 .
issue #527 : catch new-style module tracebacks like vanilla.
6 years ago
e7d9ec86f9
Allow independant control of coverage erase and reporting
6 years ago
6f17ede28b
Fix incorrect attempt to use coverage
...
Looks like this was a copy and paste mistake
6 years ago
8a2dcbf674
docs: update Changelog; closes #527 .
6 years ago
954f874085
issue #527 : catch new-style module tracebacks like vanilla.
6 years ago
e8e66d6fd0
Merge pull request #526 from moreati/py36-deprecationwarning
...
Py36 deprecationwarning
6 years ago
ea83a1b677
Merge pull request #525 from moreati/coverage_report_always
...
Generate coverage report even if some tests fail
6 years ago
0d725f9cd8
Fix DeprecationWarning in mitogen.utils.run_with_router()
...
```
14:20:32.186270 W py.warnings:
/home/alex/src/mitogen/mitogen/utils.py:152: DeprecationWarning: invalid
escape sequence \*
"""
```
6 years ago
e7d70b109b
Generate coverage report even if some tests fail
...
`set -o errexit` was exiting run_tests prematurely if any test failed,
so the coverage report was not always generated.
6 years ago
d763570ca2
Merge remote-tracking branch 'origin/dmw'
...
* origin/dmw:
ci: fix incorrect partition/rpartition from 8a4caea84f#260 : hide force-disconnect messages.
issue #498 : fix shutdown crash
issue #260 : avoid start_transmit()/on_transmit()/stop_transmit()
core: ensure broker profiling output reaches disk
master: keep is_stdlib_path() result as negative cache entry
ci: Allow DISTROS="debian*32" variable, and KEEP=1
6 years ago
49a8745a45
ci: fix incorrect partition/rpartition from 8a4caea84f
6 years ago
696cee57dd
issue #260 : hide force-disconnect messages.
...
Parent is always force-disconnected, and now it shows up quite visibly
on every exit.
6 years ago
26c6c6d048
issue #498 : fix shutdown crash
...
Traceback (most recent call last):
File "<stdin>", line 2707, in _invoke
File "<stdin>", line 2480, in _on_del_route
NameError: global name 'target_id' is not defined
6 years ago
a18a083c94
issue #260 : avoid start_transmit()/on_transmit()/stop_transmit()
...
Previous transmit sequence was:
Router._async_route -> Stream._send -> Broker._start_transmit ->
Broker.loop -> Stream.on_transmit -> socket.write ->
Broker.stop_transmit
New sequence, when socket buffer can hold message is:
Router._async_route -> Stream._send -> socket.write
bench/roundtrip.py
Before: 240 usec
after: 178 usec
Stat before:
5088.276050 task-clock (msec) # 0.997 CPUs utilized
185,568 context-switches # 0.036 M/sec
0 cpu-migrations # 0.000 K/sec
18,923 page-faults # 0.004 M/sec
13,063,871,501 cycles # 2.567 GHz
12,834,579,684 instructions # 0.98 insn per cycle
2,669,820,684 branches # 524.700 M/sec
107,296,033 branch-misses # 4.02% of all branches
5.105018296 seconds time elapsed
2.350970000 seconds user
0.345497000 seconds sys
Stat after:
4019.208047 task-clock (msec) # 0.998 CPUs utilized
249,471 context-switches # 0.062 M/sec
0 cpu-migrations # 0.000 K/sec
20,990 page-faults # 0.005 M/sec
10,312,535,979 cycles # 2.566 GHz
11,586,365,995 instructions # 1.12 insn per cycle
2,392,933,370 branches # 595.374 M/sec
75,432,205 branch-misses # 3.15% of all branches
4.028763347 seconds time elapsed
3.367051000 seconds user
0.652962000 seconds sys
6 years ago
b161408598
Merge commit 'fe74577'
...
* commit 'fe74577':
Use develop mode in tox
issue #429 : fix sudo regression.
misc: rename to scripts. tab completion!!
core: Latch._wake improvements
issue #498 : prevent crash on double 'disconnect' signal.
issue #413 : don't double-propagate DEL_ROUTE to parent.
issue #498 : wrap Router dict mutations in a lock
issue #429 : enable en_US locale to unbreak debops test.
issue #499 : fix another mind-numbingly stupid vanilla inconsistency
issue #497 : do our best to cope with crap upstream code
ssh: fix test to match updated log format.
issue #429 : update Changelog.
issue #429 : update Changelog.
issue #429 : teach sudo about every know i18n password string.
issue #429 : install i18n-related bits in test images.
ssh: tidy up logs and stream names.
tests: ensure file is closed in connection_test.
gcloud: small updates
tests: give ansible/gcloud/ its own requirements file.
6 years ago
d4a0b70e15
core: ensure broker profiling output reaches disk
...
Profiler hasn't been used much since the hard exit was added.
6 years ago
dc4b27c6bf
master: keep is_stdlib_path() result as negative cache entry
...
On 32x Docker run of issue_140__thread_pileup.yml
Before:
ncalls tottime percall cumtime percall filename:lineno(function)
1218500 1.716 0.000 7.325 0.000 /home/dmw/src/mitogen/mitogen/master.py:118(is_stdlib_path)
After:
ncalls tottime percall cumtime percall filename:lineno(function)
166 0.000 0.000 0.001 0.000 /home/dmw/src/mitogen/mitogen/master.py:123(is_stdlib_path)
6 years ago
8a4caea84f
ci: Allow DISTROS="debian*32" variable, and KEEP=1
6 years ago
fe745779ba
Use develop mode in tox
6 years ago
fca5513610
issue #429 : fix sudo regression.
6 years ago
ae8173a29f
misc: rename to scripts. tab completion!!
6 years ago
0c0d34241b
core: Latch._wake improvements
...
os.write() can fail with EINTR due to signals, so wrap it in
io_op(). Closes #483 .
Masking EBADF looks like it is/was almost certainly papering over a bug,
remove it and suffer the bug reports. Closes #495 .
6 years ago
407294cd79
issue #498 : prevent crash on double 'disconnect' signal.
...
Fixes:
ERROR! [pid 1096] 23:31:48.363215 E mitogen: _broker_main() crashed
Traceback (most recent call last):
File "/home/dmw/src/mitogen/mitogen/core.py", line 2917, in _broker_main
self._loop_once()
File "/home/dmw/src/mitogen/mitogen/core.py", line 2875, in _loop_once
self._call(side.stream, func)
File "/home/dmw/src/mitogen/mitogen/core.py", line 2860, in _call
stream.on_disconnect(self)
File "/home/dmw/src/mitogen/mitogen/parent.py", line 1161, in on_disconnect
super(Stream, self).on_disconnect(broker)
File "/home/dmw/src/mitogen/mitogen/core.py", line 1534, in on_disconnect
fire(self, 'disconnect')
File "/home/dmw/src/mitogen/mitogen/core.py", line 390, in fire
func(*args, **kwargs)
File "/home/dmw/src/mitogen/mitogen/parent.py", line 1794, in <lambda>
func=lambda: self._on_stream_disconnect(stream),
File "/home/dmw/src/mitogen/mitogen/parent.py", line 1810, in _on_stream_disconnect
routes = self._routes_by_stream.pop(stream)
KeyError: mitogen.ssh.Stream('ssh.localhost:2236')
6 years ago
9aa845669c
issue #413 : don't double-propagate DEL_ROUTE to parent.
...
propagate_up() sends ADD_ROUTE and DEL_ROUTE
propagate_down() sends only DEL_ROUTE, but didn't bother checking if
up() had sent it already.
Fixes:
ERROR! [pid 41060] 17:55:30.739159 E mitogen.ctx.ssh.localhost:
mitogen: RouteMonitor(): received DEL_ROUTE for 6081 from
mitogen.fork.Stream(u'fork.41142'), expected
mitogen.core.Stream('parent')
6 years ago
e2478dcb9f
issue #498 : wrap Router dict mutations in a lock
6 years ago
60fe3fd6f5
issue #429 : enable en_US locale to unbreak debops test.
6 years ago
7531af3ee0
issue #499 : fix another mind-numbingly stupid vanilla inconsistency
6 years ago
a40946297f
issue #497 : do our best to cope with crap upstream code
6 years ago
David Wilson