Commit Graph

2335 Commits (fa0c25bb2de95b6c338c2d2fbcd190add8aea146)
 

Author SHA1 Message Date
David Wilson fa0c25bb2d Bump version for release. 6 years ago
David Wilson b12539f99b docs: update Changelog; closes #481 6 years ago
David Wilson 78ec634dab issue #481: core: preserve stderr TTY FD if one is present.
Since 802de6a8d5, sudo on CentOS 5 had
begun failing due to a TTY FD leak in the parent process being fixed.

The old versions of sudo doesn't hang around after starting a child --
they exec the privilege-escalated child process on top of themselves,
meaning no spare copy of the TTY FD is kept alive by sudo.

When the child starts up, it replaces stdio with IoLoggers, including
the inherited stderr FD connected to DiagLogStream/the slave PTY. When
the last process closes a slave PTY, the kernel sends SIGHUP to any
processes still having it as the controlling TTY.

Therefore we must either ignore SIGHUP until the first stage has been
waited on (since the first stage also preserve the FD), or dup the
inherited TTY FD and keep it around forever.

Wasting one FD seems less annoying than modifying process signals for
all potential library users, so that is the approach taken here.
6 years ago
David Wilson b263e01867 issue #481: avoid crash if disconnect occurs during forward_modules() 6 years ago
David Wilson 92b1648729 Add a few more important modules to preamble_size.py. 6 years ago
David Wilson 6dba55624e .ci: add verbiage for run_batches() too. 6 years ago
David Wilson 9c7d93885d .ci: add README.md. 6 years ago
David Wilson 32161f3df5 docs: update thanks 6 years ago
David Wilson 4f6c57b6a8 docs: lose "approaching stability" language, we're pretty good now 6 years ago
David Wilson 7b046c9fba docs: fix changelog syntax/order/"20KB" 6 years ago
David Wilson 1345f05627 Merge remote-tracking branch 'origin/dmw'
* origin/dmw:
  tests: add new compression parameter to mitogen_get_stack results
  tests: disable affinity_test on Travis :/
  issue #508: fix responder stats test due to new smaller parent.py.
  issue #508: tests: skip minify_test Py2.4/2.5 for profiler.py.
  tests: fix fallout from 36fb318adf5c56e729296c3efce84f4dd75ced4e
  issue #520: add AIX auth failure string to su.
  tests: move affinity_test to Ansible tests.
  core: cProfile is not available in 2.4.
  issue #505: docs: add new detail graph for one scenario.
  docs: update and re-record profile graphs in docs; closes #505
  service: fix PushFileService exception
  tests: pad out localhost-*
  service: start pool shutdown on broker shutdown.
  master: .encode() needed for Py3.
  ansible: stash PID files in CWD if requested for debugging.
  issue #508: master: minify_safe_re must be bytes for Py3.
  bench: tidy up and cpu-pin some more files.
  tests: add localhost-x100
  ansible: double the default pool size.
  ansible: raise error with correct exception type.
  issue #508: master: minify all Mitogen/ansible_mitogen sources.
  parent: PartialZlib docstrings.
  ansible: hacky parser to alow bools to be specified on command line
  parent: pre-cache bootstrap if possible.
  docs: update Changelog.
  ansible: add mitogen_ssh_compression variable.
  service: PushFileService never recorded a file as sent.
  parent: synchronize get_core_source()
  service: use correct profile aggregation name.
  SyntaxError.
  ansible: don't pin controller if <4 cores.
  tests: make soak testing work reliably on vanilla.
  docs: changelog tidyups.
  ansible: document and make affinity stuff portable to non-Linux
  ansible: fix affinity.py test failure on 2 cores.
  ansible: preheat PluginLoader caches before fork.
  tests: make mitogen_shutdown_all be run_once by default.
  docs: update Changelog.
  ansible: use Poller for WorkerProcess; closes #491.
  ansible: new multiplexer/workers configuration
  docs: update Changelog.
  docs: update Changelog.
  ansible: pin connection multiplexer to a single core
  utils: pad out reset_affinity() and integrate with detach_popen()
  utils: import reset_affinity() function.
  master: set Router.profiling if MITOGEN_PROFILING variable present.
  parent: don't kill children when profiling is active.
  ansible: hook strategy and worker processes into profiler
  profiler: import from linear2 branch
  core: tidy up existing profiling code and support MITOGEN_PROFILE_FMT
  issue #260: redundant if statement.
  ansible: ensure MuxProcess MITOGEN_PROFILING results reach disk.
  ansible/bench: make end= configurable.
  master: cache sent/forwarded module names
6 years ago
David Wilson 2aed0a0c6c tests: add new compression parameter to mitogen_get_stack results 6 years ago
David Wilson 6859186069 tests: disable affinity_test on Travis :/ 6 years ago
David Wilson 19ae7b7a84 issue #508: fix responder stats test due to new smaller parent.py. 6 years ago
David Wilson 953a910881 issue #508: tests: skip minify_test Py2.4/2.5 for profiler.py. 6 years ago
David Wilson 263f530496 tests: fix fallout from 36fb318adf5c56e729296c3efce84f4dd75ced4e 6 years ago
David Wilson 4abd34e7a6 issue #520: add AIX auth failure string to su. 6 years ago
David Wilson 8026e54b9b tests: move affinity_test to Ansible tests. 6 years ago
David Wilson 5ae7464011 core: cProfile is not available in 2.4. 6 years ago
David Wilson e703522010 issue #505: docs: add new detail graph for one scenario. 6 years ago
David Wilson c279454502 docs: update and re-record profile graphs in docs; closes #505
pcaps moved to 'pcaps' branch.
6 years ago
David Wilson 54835d4c9b service: fix PushFileService exception
[costapp]

ERROR! [pid 25135] 21:10:56.284733 E mitogen.ctx.ssh.35.200.203.48: mitogen: While calling no-reply method PushFileService.forward
Traceback (most recent call last):
  File "master:/home/dmw/src/mitogen/mitogen/service.py", line 260, in _invoke
    ret = method(**kwargs)
  File "master:/home/dmw/src/mitogen/mitogen/service.py", line 718, in forward
    self._forward(path, context)
  File "master:/home/dmw/src/mitogen/mitogen/service.py", line 633, in _forward
    stream = self.router.stream_by_id(context.context_id)
AttributeError: 'unicode' object has no attribute 'context_id'
^C [ERROR]: User interrupted execution
6 years ago
David Wilson b2bae9deed tests: pad out localhost-* 6 years ago
David Wilson 23d7a961e7 service: start pool shutdown on broker shutdown. 6 years ago
David Wilson 85cfa3b0f5 master: .encode() needed for Py3. 6 years ago
David Wilson 05b1ccb658 ansible: stash PID files in CWD if requested for debugging. 6 years ago
David Wilson 1d509d03ff issue #508: master: minify_safe_re must be bytes for Py3. 6 years ago
David Wilson fda7d5c276 bench: tidy up and cpu-pin some more files. 6 years ago
David Wilson 29adfbea59 tests: add localhost-x100 6 years ago
David Wilson eb67fbe9d2 ansible: double the default pool size.
Tempted to push this up to 64, but let's do it incrementally just in
case.
6 years ago
David Wilson b89e53fd70 ansible: raise error with correct exception type. 6 years ago
David Wilson 0e193c223c issue #508: master: minify all Mitogen/ansible_mitogen sources.
Minify-safe files are marked with a magical "# !mitogen: minify_safe"
comment anywhere in the file, which activates the minifier. The result
is naturally cached by ModuleResponder, therefore lru_cache is gone too.

Given:

    import os, mitogen
    @mitogen.main()
    def main(router):
        c = router.ssh(hostname='k3')
        c.call(os.getpid)
        router.sudo(via=c)

SSH footprint drops from 56.2 KiB to 42.75 KiB (-23.9%)
Ansible "shell: hostname" drops 149.26 KiB to 117.42 KiB (-21.3%)
6 years ago
David Wilson cfb94e463f parent: PartialZlib docstrings. 6 years ago
David Wilson 7badb4a25b ansible: hacky parser to alow bools to be specified on command line 6 years ago
David Wilson 9adc38d8ec parent: pre-cache bootstrap if possible.
When the interpreter is modern enough, use zlib.compressobj() to
pre-compress the unchanging parts of the bootstrap once, then use
compressobj.copy() to append just the context's config during stream
construction.

Before: 100 loops, best of 3: 5.81 msec per loop
After: 10000 loops, best of 3: 35.9 usec per loop

With 100 targets this is enough to knock 6 seconds off startup, at 500
targets it becomes half a minute.

Test 'program':
        python -m timeit -s '
                import mitogen.parent as p;
                import mitogen.master as m;
                r=m.Router();
                s=p.Stream(r, 0, max_message_size=1);
                r.broker.shutdown()'\
                \
                's.get_preamble()'
6 years ago
David Wilson e167f6373c docs: update Changelog. 6 years ago
David Wilson b499fbe29b ansible: add mitogen_ssh_compression variable. 6 years ago
David Wilson d6c4a983e1 service: PushFileService never recorded a file as sent.
Ansible modules were being resent continuously - but only the main
script module, and any custom modutils if any were present.

Wire footprint drops by ~1/3rd for a 500 task run of 'shell: hostname':

-rw-r--r-- 1 root root 584K Jan 31 22:06 500mito-before2
-rw-r--r-- 1 root root 434K Jan 31 22:04 500mito-filesbugonly
6 years ago
David Wilson 7ca927608c parent: synchronize get_core_source()
Single task 100 SSH target run, before:

        3533181 function calls (3533083 primitive calls) in 616.688 seconds
        User time (seconds): 32.52
        System time (seconds): 2.71
        Percent of CPU this job got: 64%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:54.88

After:

        451602 function calls (451504 primitive calls) in 570.746 seconds
        User time (seconds): 29.48
        System time (seconds): 2.81
        Percent of CPU this job got: 67%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:48.20
6 years ago
David Wilson 2399a9e621 service: use correct profile aggregation name. 6 years ago
David Wilson a2ae4ed696 SyntaxError. 6 years ago
David Wilson a9d48a8fdc ansible: don't pin controller if <4 cores. 6 years ago
David Wilson b72d139bd2 tests: make soak testing work reliably on vanilla.
It claims to disable host key checking, but it doesn't.
6 years ago
David Wilson 2153e195c3 docs: changelog tidyups. 6 years ago
David Wilson 4531338b12 ansible: document and make affinity stuff portable to non-Linux
Portable as in does nothing for the time at least for now.
6 years ago
David Wilson de5c050707 ansible: fix affinity.py test failure on 2 cores. 6 years ago
David Wilson 00ae90b2b2 ansible: preheat PluginLoader caches before fork.
This has been broken for some time, but somehow it has become noticeable
on recent Ansible.

loop-100-tasks.yml before:
      15.532724001 seconds time elapsed
       8.453850000 seconds user
       5.808627000 seconds sys

loop-100-tasks.yml after:
       8.991635735 seconds time elapsed
       5.059232000 seconds user
       2.578842000 seconds sys
6 years ago
David Wilson b59a47aecb tests: make mitogen_shutdown_all be run_once by default. 6 years ago
David Wilson f06152db97 docs: update Changelog. 6 years ago
David Wilson 7b129e8576 ansible: use Poller for WorkerProcess; closes #491. 6 years ago