* origin/v024: (662 commits)
docs: update Changelog release date.
Bump version for release.
docs: update Changelog; closes#481
issue #481: core: preserve stderr TTY FD if one is present.
issue #481: avoid crash if disconnect occurs during forward_modules()
Add a few more important modules to preamble_size.py.
.ci: add verbiage for run_batches() too.
.ci: add README.md.
docs: update thanks
docs: lose "approaching stability" language, we're pretty good now
docs: fix changelog syntax/order/"20KB"
tests: add new compression parameter to mitogen_get_stack results
tests: disable affinity_test on Travis :/
issue #508: fix responder stats test due to new smaller parent.py.
issue #508: tests: skip minify_test Py2.4/2.5 for profiler.py.
tests: fix fallout from 36fb318adf5c56e729296c3efce84f4dd75ced4e
issue #520: add AIX auth failure string to su.
tests: move affinity_test to Ansible tests.
core: cProfile is not available in 2.4.
issue #505: docs: add new detail graph for one scenario.
docs: update and re-record profile graphs in docs; closes#505
service: fix PushFileService exception
tests: pad out localhost-*
service: start pool shutdown on broker shutdown.
master: .encode() needed for Py3.
ansible: stash PID files in CWD if requested for debugging.
issue #508: master: minify_safe_re must be bytes for Py3.
bench: tidy up and cpu-pin some more files.
tests: add localhost-x100
ansible: double the default pool size.
ansible: raise error with correct exception type.
issue #508: master: minify all Mitogen/ansible_mitogen sources.
parent: PartialZlib docstrings.
ansible: hacky parser to alow bools to be specified on command line
parent: pre-cache bootstrap if possible.
docs: update Changelog.
ansible: add mitogen_ssh_compression variable.
service: PushFileService never recorded a file as sent.
parent: synchronize get_core_source()
service: use correct profile aggregation name.
SyntaxError.
ansible: don't pin controller if <4 cores.
tests: make soak testing work reliably on vanilla.
docs: changelog tidyups.
ansible: document and make affinity stuff portable to non-Linux
ansible: fix affinity.py test failure on 2 cores.
ansible: preheat PluginLoader caches before fork.
tests: make mitogen_shutdown_all be run_once by default.
docs: update Changelog.
ansible: use Poller for WorkerProcess; closes#491.
ansible: new multiplexer/workers configuration
docs: update Changelog.
docs: update Changelog.
ansible: pin connection multiplexer to a single core
utils: pad out reset_affinity() and integrate with detach_popen()
utils: import reset_affinity() function.
master: set Router.profiling if MITOGEN_PROFILING variable present.
parent: don't kill children when profiling is active.
ansible: hook strategy and worker processes into profiler
profiler: import from linear2 branch
core: tidy up existing profiling code and support MITOGEN_PROFILE_FMT
issue #260: redundant if statement.
ansible: ensure MuxProcess MITOGEN_PROFILING results reach disk.
ansible/bench: make end= configurable.
master: cache sent/forwarded module names
Aggregate code coverage data across tox all runs
Allow independant control of coverage erase and reporting
Fix incorrect attempt to use coverage
docs: update Changelog; closes#527.
issue #527: catch new-style module tracebacks like vanilla.
Fix DeprecationWarning in mitogen.utils.run_with_router()
Generate coverage report even if some tests fail
ci: fix incorrect partition/rpartition from 8a4caea84f
issue #260: hide force-disconnect messages.
issue #498: fix shutdown crash
issue #260: avoid start_transmit()/on_transmit()/stop_transmit()
core: ensure broker profiling output reaches disk
master: keep is_stdlib_path() result as negative cache entry
ci: Allow DISTROS="debian*32" variable, and KEEP=1
Use develop mode in tox
issue #429: fix sudo regression.
misc: rename to scripts. tab completion!!
core: Latch._wake improvements
issue #498: prevent crash on double 'disconnect' signal.
issue #413: don't double-propagate DEL_ROUTE to parent.
issue #498: wrap Router dict mutations in a lock
issue #429: enable en_US locale to unbreak debops test.
issue #499: fix another mind-numbingly stupid vanilla inconsistency
issue #497: do our best to cope with crap upstream code
ssh: fix test to match updated log format.
issue #429: update Changelog.
issue #429: update Changelog.
issue #429: teach sudo about every know i18n password string.
issue #429: install i18n-related bits in test images.
ssh: tidy up logs and stream names.
tests: ensure file is closed in connection_test.
gcloud: small updates
tests: give ansible/gcloud/ its own requirements file.
issue #499: another totally moronic implementation difference
issue #499: disable new test on vanilla.
...
* origin/master: (661 commits)
Bump version for release.
docs: update Changelog; closes#481
issue #481: core: preserve stderr TTY FD if one is present.
issue #481: avoid crash if disconnect occurs during forward_modules()
Add a few more important modules to preamble_size.py.
.ci: add verbiage for run_batches() too.
.ci: add README.md.
docs: update thanks
docs: lose "approaching stability" language, we're pretty good now
docs: fix changelog syntax/order/"20KB"
tests: add new compression parameter to mitogen_get_stack results
tests: disable affinity_test on Travis :/
issue #508: fix responder stats test due to new smaller parent.py.
issue #508: tests: skip minify_test Py2.4/2.5 for profiler.py.
tests: fix fallout from 36fb318adf5c56e729296c3efce84f4dd75ced4e
issue #520: add AIX auth failure string to su.
tests: move affinity_test to Ansible tests.
core: cProfile is not available in 2.4.
issue #505: docs: add new detail graph for one scenario.
docs: update and re-record profile graphs in docs; closes#505
service: fix PushFileService exception
tests: pad out localhost-*
service: start pool shutdown on broker shutdown.
master: .encode() needed for Py3.
ansible: stash PID files in CWD if requested for debugging.
issue #508: master: minify_safe_re must be bytes for Py3.
bench: tidy up and cpu-pin some more files.
tests: add localhost-x100
ansible: double the default pool size.
ansible: raise error with correct exception type.
issue #508: master: minify all Mitogen/ansible_mitogen sources.
parent: PartialZlib docstrings.
ansible: hacky parser to alow bools to be specified on command line
parent: pre-cache bootstrap if possible.
docs: update Changelog.
ansible: add mitogen_ssh_compression variable.
service: PushFileService never recorded a file as sent.
parent: synchronize get_core_source()
service: use correct profile aggregation name.
SyntaxError.
ansible: don't pin controller if <4 cores.
tests: make soak testing work reliably on vanilla.
docs: changelog tidyups.
ansible: document and make affinity stuff portable to non-Linux
ansible: fix affinity.py test failure on 2 cores.
ansible: preheat PluginLoader caches before fork.
tests: make mitogen_shutdown_all be run_once by default.
docs: update Changelog.
ansible: use Poller for WorkerProcess; closes#491.
ansible: new multiplexer/workers configuration
docs: update Changelog.
docs: update Changelog.
ansible: pin connection multiplexer to a single core
utils: pad out reset_affinity() and integrate with detach_popen()
utils: import reset_affinity() function.
master: set Router.profiling if MITOGEN_PROFILING variable present.
parent: don't kill children when profiling is active.
ansible: hook strategy and worker processes into profiler
profiler: import from linear2 branch
core: tidy up existing profiling code and support MITOGEN_PROFILE_FMT
issue #260: redundant if statement.
ansible: ensure MuxProcess MITOGEN_PROFILING results reach disk.
ansible/bench: make end= configurable.
master: cache sent/forwarded module names
Aggregate code coverage data across tox all runs
Allow independant control of coverage erase and reporting
Fix incorrect attempt to use coverage
docs: update Changelog; closes#527.
issue #527: catch new-style module tracebacks like vanilla.
Fix DeprecationWarning in mitogen.utils.run_with_router()
Generate coverage report even if some tests fail
ci: fix incorrect partition/rpartition from 8a4caea84f
issue #260: hide force-disconnect messages.
issue #498: fix shutdown crash
issue #260: avoid start_transmit()/on_transmit()/stop_transmit()
core: ensure broker profiling output reaches disk
master: keep is_stdlib_path() result as negative cache entry
ci: Allow DISTROS="debian*32" variable, and KEEP=1
Use develop mode in tox
issue #429: fix sudo regression.
misc: rename to scripts. tab completion!!
core: Latch._wake improvements
issue #498: prevent crash on double 'disconnect' signal.
issue #413: don't double-propagate DEL_ROUTE to parent.
issue #498: wrap Router dict mutations in a lock
issue #429: enable en_US locale to unbreak debops test.
issue #499: fix another mind-numbingly stupid vanilla inconsistency
issue #497: do our best to cope with crap upstream code
ssh: fix test to match updated log format.
issue #429: update Changelog.
issue #429: update Changelog.
issue #429: teach sudo about every know i18n password string.
issue #429: install i18n-related bits in test images.
ssh: tidy up logs and stream names.
tests: ensure file is closed in connection_test.
gcloud: small updates
tests: give ansible/gcloud/ its own requirements file.
issue #499: another totally moronic implementation difference
issue #499: disable new test on vanilla.
docs: update Changelog; closes#499.
...
* origin/dmw:
docs: update Changelog; closes#481
issue #481: core: preserve stderr TTY FD if one is present.
issue #481: avoid crash if disconnect occurs during forward_modules()
Add a few more important modules to preamble_size.py.
.ci: add verbiage for run_batches() too.
.ci: add README.md.
docs: update thanks
docs: lose "approaching stability" language, we're pretty good now
docs: fix changelog syntax/order/"20KB"
Since 802de6a8d5, sudo on CentOS 5 had
begun failing due to a TTY FD leak in the parent process being fixed.
The old versions of sudo doesn't hang around after starting a child --
they exec the privilege-escalated child process on top of themselves,
meaning no spare copy of the TTY FD is kept alive by sudo.
When the child starts up, it replaces stdio with IoLoggers, including
the inherited stderr FD connected to DiagLogStream/the slave PTY. When
the last process closes a slave PTY, the kernel sends SIGHUP to any
processes still having it as the controlling TTY.
Therefore we must either ignore SIGHUP until the first stage has been
waited on (since the first stage also preserve the FD), or dup the
inherited TTY FD and keep it around forever.
Wasting one FD seems less annoying than modifying process signals for
all potential library users, so that is the approach taken here.
* origin/dmw:
tests: add new compression parameter to mitogen_get_stack results
tests: disable affinity_test on Travis :/
issue #508: fix responder stats test due to new smaller parent.py.
issue #508: tests: skip minify_test Py2.4/2.5 for profiler.py.
tests: fix fallout from 36fb318adf5c56e729296c3efce84f4dd75ced4e
issue #520: add AIX auth failure string to su.
tests: move affinity_test to Ansible tests.
core: cProfile is not available in 2.4.
issue #505: docs: add new detail graph for one scenario.
docs: update and re-record profile graphs in docs; closes#505
service: fix PushFileService exception
tests: pad out localhost-*
service: start pool shutdown on broker shutdown.
master: .encode() needed for Py3.
ansible: stash PID files in CWD if requested for debugging.
issue #508: master: minify_safe_re must be bytes for Py3.
bench: tidy up and cpu-pin some more files.
tests: add localhost-x100
ansible: double the default pool size.
ansible: raise error with correct exception type.
issue #508: master: minify all Mitogen/ansible_mitogen sources.
parent: PartialZlib docstrings.
ansible: hacky parser to alow bools to be specified on command line
parent: pre-cache bootstrap if possible.
docs: update Changelog.
ansible: add mitogen_ssh_compression variable.
service: PushFileService never recorded a file as sent.
parent: synchronize get_core_source()
service: use correct profile aggregation name.
SyntaxError.
ansible: don't pin controller if <4 cores.
tests: make soak testing work reliably on vanilla.
docs: changelog tidyups.
ansible: document and make affinity stuff portable to non-Linux
ansible: fix affinity.py test failure on 2 cores.
ansible: preheat PluginLoader caches before fork.
tests: make mitogen_shutdown_all be run_once by default.
docs: update Changelog.
ansible: use Poller for WorkerProcess; closes#491.
ansible: new multiplexer/workers configuration
docs: update Changelog.
docs: update Changelog.
ansible: pin connection multiplexer to a single core
utils: pad out reset_affinity() and integrate with detach_popen()
utils: import reset_affinity() function.
master: set Router.profiling if MITOGEN_PROFILING variable present.
parent: don't kill children when profiling is active.
ansible: hook strategy and worker processes into profiler
profiler: import from linear2 branch
core: tidy up existing profiling code and support MITOGEN_PROFILE_FMT
issue #260: redundant if statement.
ansible: ensure MuxProcess MITOGEN_PROFILING results reach disk.
ansible/bench: make end= configurable.
master: cache sent/forwarded module names
[costapp]
ERROR! [pid 25135] 21:10:56.284733 E mitogen.ctx.ssh.35.200.203.48: mitogen: While calling no-reply method PushFileService.forward
Traceback (most recent call last):
File "master:/home/dmw/src/mitogen/mitogen/service.py", line 260, in _invoke
ret = method(**kwargs)
File "master:/home/dmw/src/mitogen/mitogen/service.py", line 718, in forward
self._forward(path, context)
File "master:/home/dmw/src/mitogen/mitogen/service.py", line 633, in _forward
stream = self.router.stream_by_id(context.context_id)
AttributeError: 'unicode' object has no attribute 'context_id'
^C [ERROR]: User interrupted execution
Minify-safe files are marked with a magical "# !mitogen: minify_safe"
comment anywhere in the file, which activates the minifier. The result
is naturally cached by ModuleResponder, therefore lru_cache is gone too.
Given:
import os, mitogen
@mitogen.main()
def main(router):
c = router.ssh(hostname='k3')
c.call(os.getpid)
router.sudo(via=c)
SSH footprint drops from 56.2 KiB to 42.75 KiB (-23.9%)
Ansible "shell: hostname" drops 149.26 KiB to 117.42 KiB (-21.3%)
When the interpreter is modern enough, use zlib.compressobj() to
pre-compress the unchanging parts of the bootstrap once, then use
compressobj.copy() to append just the context's config during stream
construction.
Before: 100 loops, best of 3: 5.81 msec per loop
After: 10000 loops, best of 3: 35.9 usec per loop
With 100 targets this is enough to knock 6 seconds off startup, at 500
targets it becomes half a minute.
Test 'program':
python -m timeit -s '
import mitogen.parent as p;
import mitogen.master as m;
r=m.Router();
s=p.Stream(r, 0, max_message_size=1);
r.broker.shutdown()'\
\
's.get_preamble()'
Ansible modules were being resent continuously - but only the main
script module, and any custom modutils if any were present.
Wire footprint drops by ~1/3rd for a 500 task run of 'shell: hostname':
-rw-r--r-- 1 root root 584K Jan 31 22:06 500mito-before2
-rw-r--r-- 1 root root 434K Jan 31 22:04 500mito-filesbugonly
Single task 100 SSH target run, before:
3533181 function calls (3533083 primitive calls) in 616.688 seconds
User time (seconds): 32.52
System time (seconds): 2.71
Percent of CPU this job got: 64%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:54.88
After:
451602 function calls (451504 primitive calls) in 570.746 seconds
User time (seconds): 29.48
System time (seconds): 2.81
Percent of CPU this job got: 67%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:48.20
David Wilson