4f23f0bec1
issue #590 : update comment to indicate the hack is permanent
6 years ago
1a92995a24
issue #590 : include nasty workaround for sys.modules junk
6 years ago
92b4724010
issue #587 : consistent become_exe() behaviour for older Ansibles.
6 years ago
f35194fe0f
issue #587 : mitogen_doas should not become_exe for doas_path
...
Looks like this has always been wrong - when used as a connection
method, PlayContext.become_method/become_exe may hold totally unrelated
data.
6 years ago
c1c8d5c31e
issue #587 : 2.8 PlayContext lacks sudo_flags attribute.
...
This is a huge bodge.
6 years ago
e11b251c75
issue #587 : 2.8 PluginLoader.get() introduced new collection_list kwarg
6 years ago
46dde95962
issue #587 : 2.8 PlayContext.connection no longer contains connection name
...
Not clear what the intention is here. Either need to ferret it out of
some other location, or just stop preloading the connection class in the
top-level process.
6 years ago
4a614c3950
issue #587 : bump max Ansible version
6 years ago
f105a81e20
ansible: descriptive version check during startup.
6 years ago
f30a4c05c8
issue #581 : expose mitogen_mask_remote_name variable.
6 years ago
65deb3feac
issue #575 : fix exception text rendering
6 years ago
34fb9da1be
issue #570 : add firewalld to always-fork list for now.
6 years ago
3ff6123483
issue #557 : support correct cpu_set_t size
6 years ago
2bd0bbd4df
issue #555 : ansible: workaround ancient reload(sys) hack.
...
This is the most minimal change for what might be relatively minimal
edge case. Alternative is replacing reload(), but let's not do that yet.
Closes #555
6 years ago
6309774be2
issue #554 : fix Ansible 2.4 compatibility
6 years ago
7743e57ff3
issue #554 : track and remove multiple make_tmp_path() calls.
6 years ago
7dacb68eeb
issue #552 : include process identity in log messages.
6 years ago
26e6194d0a
issue #548 : always treat transport=smart as 'ssh' for mitogen_via=.
...
The idea behind transport=smart is to select between paramiko and
OpenSSH given the availability of connection multiplexing and/or OSX
kernel bugs. We need to make no such choice.
6 years ago
458a4faa97
ansible: create stub __init__.py for sdist.
...
This went into 0.2.5 sdist tarball but it's not checked in.
6 years ago
8f9c67daf1
ansible: refactor affinity class and add abstract tests.
6 years ago
0f30808234
ansible: quiesce boto logger; closes #541 .
6 years ago
7fd0d34910
tests/ansible: Spec.port() test & mitogen_via= fix.
...
ansible_ssh_port was not respected.
6 years ago
1f77d24bec
Update copyright year everywhere.
6 years ago
b5b23e8f3d
tests/ansible: Spec.become_pass() test.
6 years ago
ae5a471e31
issue #539 : disable logger propagation.
6 years ago
1c955a9876
ansible: capture stderr stream of async tasks. Closes #540 .
6 years ago
7ff4e6694c
issue #536 : rework how 2.3-compatible simplejson is served
...
Regardless of the version of simplejson loaded in the master, load up
the ModuleResponder cache with our 2.4-compatible version.
To cope with simplejson being loaded due to modules like ec2_group that
try to import it before importing 'json', also update target.py to
remove it from the whitelist if a local 'json' module import succeeds.
6 years ago
8ae6ca1d5b
tests/ansible: Spec.become_method() test & mitogen_via= fix.
...
ansible_become_method hostvar was not taken into account.
6 years ago
d1cadf8ac8
tests/ansible: Spec.password() test, document interactive pw limitation.
6 years ago
21ad299d7b
tests/ansible: Spec.remote_user() test & mitogen_via= fix.
...
ansible_ssh_user precedence was incorrect.
6 years ago
748f5f675d
tests/ansible: Spec.remote_addr() test & mitogen_via= fix.
...
ansible_ssh_host was not respected.
6 years ago
e1df98168c
issue #536 : add mitogen_via= tests too.
6 years ago
604b418412
ansible: fix a crash on 2.3 when mitogen_via= host is missing.
6 years ago
001e3fee86
issue #536 : restore correct Python interpreter selection behaviour.
6 years ago
05b1ccb658
ansible: stash PID files in CWD if requested for debugging.
6 years ago
eb67fbe9d2
ansible: double the default pool size.
...
Tempted to push this up to 64, but let's do it incrementally just in
case.
6 years ago
b89e53fd70
ansible: raise error with correct exception type.
6 years ago
0e193c223c
issue #508 : master: minify all Mitogen/ansible_mitogen sources.
...
Minify-safe files are marked with a magical "# !mitogen: minify_safe"
comment anywhere in the file, which activates the minifier. The result
is naturally cached by ModuleResponder, therefore lru_cache is gone too.
Given:
import os, mitogen
@mitogen.main()
def main(router):
c = router.ssh(hostname='k3')
c.call(os.getpid)
router.sudo(via=c)
SSH footprint drops from 56.2 KiB to 42.75 KiB (-23.9%)
Ansible "shell: hostname" drops 149.26 KiB to 117.42 KiB (-21.3%)
6 years ago
7badb4a25b
ansible: hacky parser to alow bools to be specified on command line
6 years ago
b499fbe29b
ansible: add mitogen_ssh_compression variable.
6 years ago
a2ae4ed696
SyntaxError.
6 years ago
a9d48a8fdc
ansible: don't pin controller if <4 cores.
6 years ago
4531338b12
ansible: document and make affinity stuff portable to non-Linux
...
Portable as in does nothing for the time at least for now.
6 years ago
de5c050707
ansible: fix affinity.py test failure on 2 cores.
6 years ago
00ae90b2b2
ansible: preheat PluginLoader caches before fork.
...
This has been broken for some time, but somehow it has become noticeable
on recent Ansible.
loop-100-tasks.yml before:
15.532724001 seconds time elapsed
8.453850000 seconds user
5.808627000 seconds sys
loop-100-tasks.yml after:
8.991635735 seconds time elapsed
5.059232000 seconds user
2.578842000 seconds sys
6 years ago
7b129e8576
ansible: use Poller for WorkerProcess; closes #491 .
6 years ago
c6d5aa29ba
ansible: new multiplexer/workers configuration
...
Following on from 152effc26c9a5918cb7ead7a97fe7fa7f81b6764,
* Pin mux to CPU 0
* Pin top-level CPU 1
* Pin workers sequentially to CPU 2..n
Nets 19.5% improvement on issue_140__thread_pileup.yml when targetting
64 Docker containers on the same 8 core/16 thread machine.
Before (prior to last scheme, no affinity at all):
2294528.731458 task-clock (msec) # 6.443 CPUs utilized
10,429,745 context-switches # 0.005 M/sec
2,049,618 cpu-migrations # 0.893 K/sec
8,258,952 page-faults # 0.004 M/sec
5,532,719,253,824 cycles # 2.411 GHz (83.35%)
3,267,471,616,230 instructions # 0.59 insn per cycle
# 1.22 stalled cycles per insn (83.35%)
662,006,455,943 branches # 288.515 M/sec (83.33%)
39,453,895,977 branch-misses # 5.96% of all branches (83.37%)
356.148064576 seconds time elapsed
After:
2226463.958975 task-clock (msec) # 7.784 CPUs utilized
9,831,466 context-switches # 0.004 M/sec
180,065 cpu-migrations # 0.081 K/sec
5,082,278 page-faults # 0.002 M/sec
5,592,548,587,259 cycles # 2.512 GHz (83.35%)
3,135,038,855,414 instructions # 0.56 insn per cycle
# 1.32 stalled cycles per insn (83.32%)
636,397,509,232 branches # 285.833 M/sec (83.30%)
39,135,441,790 branch-misses # 6.15% of all branches (83.35%)
286.036681644 seconds time elapsed
6 years ago
1b909e8697
ansible: pin connection multiplexer to a single core
...
Nets a reliable 8% improvement in issue_140__thread_pileup.yml when
targetting 64 Docker containers on the same 8 core/16 thread machine.
Before:
2294528.731458 task-clock (msec) # 6.443 CPUs utilized
10,429,745 context-switches # 0.005 M/sec
2,049,618 cpu-migrations # 0.893 K/sec
8,258,952 page-faults # 0.004 M/sec
5,532,719,253,824 cycles # 2.411 GHz (83.35%)
4,001,276,805,120 stalled-cycles-frontend # 72.32% frontend cycles idle (83.30%)
2,024,159,442,463 stalled-cycles-backend # 36.59% backend cycles idle (66.65%)
3,267,471,616,230 instructions # 0.59 insn per cycle
# 1.22 stalled cycles per insn (83.35%)
662,006,455,943 branches # 288.515 M/sec (83.33%)
39,453,895,977 branch-misses # 5.96% of all branches (83.37%)
356.148064576 seconds time elapsed
After:
2208247.938562 task-clock (msec) # 6.735 CPUs utilized
8,489,840 context-switches # 0.004 M/sec
1,432,967 cpu-migrations # 0.649 K/sec
7,508,957 page-faults # 0.003 M/sec
5,477,293,750,357 cycles # 2.480 GHz (83.31%)
3,984,360,350,811 stalled-cycles-frontend # 72.74% frontend cycles idle (83.32%)
1,976,646,418,711 stalled-cycles-backend # 36.09% backend cycles idle (66.64%)
3,196,197,480,792 instructions # 0.58 insn per cycle
# 1.25 stalled cycles per insn (83.36%)
648,247,332,967 branches # 293.557 M/sec (83.35%)
39,004,881,070 branch-misses # 6.02% of all branches (83.37%)
327.876903668 seconds time elapsed
6 years ago
e587396e70
ansible: hook strategy and worker processes into profiler
6 years ago
84944a9a61
ansible: ensure MuxProcess MITOGEN_PROFILING results reach disk.
...
This has been broken for quite some time.
6 years ago
954f874085
issue #527 : catch new-style module tracebacks like vanilla.
6 years ago
a1121c5a84
issue #499 : respect C.BECOME_ALLOW_SAME_USER.
6 years ago
be6ab52fe1
issue #488 : fix shutdown damage caused in 6ca2677de5
...
os._exit() subverted calm shutdown, meaning unix.Listener never had a
chance to cleanup its socket.
Move unix.Listener socket cleanup into its class so it is automatic
during shutdown, rather than cutpasted for each consumer.
Disable the watcher thread in the MuxProcess, it is useless.
Add .sock extension to /tmp/mitogen_unix_*, so we can write a test.
6 years ago
38a553d42d
issue #490 : prevent double close() destroying unrelated Connection.
6 years ago
e7fe95af88
issue #477 : fix sudo_args selection.
6 years ago
599da0689a
issue #477 / ansible: avoid a race in async job startup.
...
Ansible 2.3/Python 2.4 work revealed there is no guarantee a slow target
will have written the initial job status file out before a fast
controller makes an initial check for it. Therefore, provide AsyncRunner
with a sender it should send a message to when the initial job file has
been written.
As a bonus, also catch and report exceptions happening early in
AsyncRunner, rather than leaving them to end up in -vvv output.
6 years ago
0175052099
issue #477 : fix source of become_flags on 2.3.
6 years ago
97f3cfe4f4
issue #477 : target.file_exists() wrapper.
...
os.path.exists physical module name varies across major Python versions.
6 years ago
8f5b65f7ec
issue #477 : introduce subprocess isolation.
...
Since Python 2.4 fork is so defective, we must use subprocesses for
mitogen_task_isolation=fork. This has plenty of upside, since the long
term goal is to dump forking altogether. This allows a gentle
introduction of its replacement.
6 years ago
b9924683ac
ansible: docstring fixes.
6 years ago
75f53faf8c
issue #477 : shlex.split() in 2.4 required bytes input.
6 years ago
dc1d4251e3
ansible: synchronize module needs '.docker_cmd' attr for Docker plugin.
6 years ago
dd30a907ce
issue #477 : promote setup_gil() to mitogen.utils
...
This is since ansible_mitogen/process.py is 2.6-only, and I want to use
setup_gil() in 2.4 code.
6 years ago
bba5d82fc4
issue #477 : fix another str/bytes mixup.
6 years ago
c9ff4c6506
issue #477 : fix 3.x failure in new target.set_file_mode() function.
6 years ago
aa5d37af89
issue #477 : fix 3.x failure in new target.set_file_mode() function.
6 years ago
84a0424749
issue #477 : fix 3.x test regressions.
6 years ago
81f15028a7
issue #477 : backport ansible_mitogen.runner to 2.4.
6 years ago
256628c149
issue #477 : backport ansible_mitogen/target.py to Python2.4
6 years ago
a48ee3a536
issue #477 : vendorize the last 2.4-compatible simplejson
...
This is in part so image_prep can run against an ancient CentOS 5 image
without any upfront help, and in part simply because it's very easy to
support.
6 years ago
59dd0dc814
issue #477 : serve up junk ansible/__init__.py just like Ansible.
6 years ago
f52f7f6a62
ansible: merge away unused return values / functions.
6 years ago
b5c14cecda
docs: docs and docstrings.
6 years ago
e767de3f15
issue #412 : force-verbose output for mitogen_get_stack.
6 years ago
23866084d7
issue #412 : promote "mitogen_get_stack" to the main extension.
...
This is to make it easier for users to diagnose their own problems.
6 years ago
115c3c5657
issue #412 : add docstrings/boilerplate to transport_config.py.
6 years ago
2ad05f1238
issue #251 , #412 , #434 : fix connection configuration brainwrong
...
This refactors connection.py to pull the two huge dict-building
functions out into new transport_transport_config.PlayContextSpec and
MitogenViaSpec classes, leaving a lot more room to breath in both files
to figure out exactly how connection configuration should work.
The changes made in 1f21a303d58832
6 years ago
6ca2677de5
ansible: fix test failure during process exit.
...
======================================================================
ERROR: tests.connection_test (unittest2.loader._FailedTest)
----------------------------------------------------------------------
Traceback (most recent call last):
ImportError: Failed to import test module: tests.connection_test
Traceback (most recent call last):
File "/home/dmw/src/mitogen/.venv/local/lib/python2.7/site-packages/unittest2/loader.py", line 456, in _find_test_path
module = self._get_module_from_name(name)
File "/home/dmw/src/mitogen/.venv/local/lib/python2.7/site-packages/unittest2/loader.py", line 395, in _get_module_from_name
__import__(name)
RuntimeError: not holding the import lock
6 years ago
6915af502e
issue #61 : unused import (reported by LGTM)
6 years ago
907c4556c8
issue #61 : unused variable (reported by LGTM)
6 years ago
f67960848b
issue #61 : unused import (reported by LGTM)
6 years ago
b8ca015b83
issue #61 : unused variable (reported by LGTM)
6 years ago
bcc7bb7128
issue #61 : unused import (reported by LGTM)
6 years ago
4bdf60326c
issue #424 : ansible: make put_file() raise AnsibleFileNotFound
6 years ago
3f31b166f6
issue #461 : Ansible 2.3-compatible _get_candidate_temp_dirs().
6 years ago
9401c34f64
issue #461 : Ansible 2.3 did not have _load_name.
6 years ago
04755c3321
issue #426 : tighten up PushFileService types.
...
Bytes/Unicode mixing caused a hang, so prevent bytes entirely.
6 years ago
1e9f3446c8
issue #426 : big hack so reset_connection has task_var access
6 years ago
ca9ae4590c
issue #426 : TemporaryEnvironment must coerce to Unicode.
6 years ago
4bc0d0ec13
issue #426 : apply_mode_spec() must handle bytes.
6 years ago
3179951f5c
issue #454 : fix AttributeError and atexit.yml test.
6 years ago
cd01957995
issue #397 , #454 : pick out only shutil.rmtree() calls from atexit.
6 years ago
09aa27a6a7
ansible: initial mitogen_host_pinned strategy.
6 years ago
22de7f0e72
issue #436 : fix string parsing of mitogen_ssh_debug_level
...
It can be a string when specified on the command line.
6 years ago
87ddead0b2
issue #435 , #362 : ansible: avoid touching setrlimit() on target.
...
This replaces the previous method for capping poorly Popen()
performance, instead entirely monkey-patching the problem function
rather than simply working around it.
6 years ago
b599253291
issue #435 : increase file descriptor limit by 8x.
...
This is a temporary solution at best.
6 years ago
16911c9464
ansible: fix 3.x TypeError regression.
6 years ago
50241a922f
ansible: call on_fork() on broker shutdown; closes #420 .
6 years ago
35092c5d35
tests: Unicode/bytes fixes for integration/connection/exec_command.yml
6 years ago
76ec4f201c
issue #413 : paper over harmless duplicate del_route()
...
Ideally it would only be called once, and in future maybe it can, but
right now we need to cope with these cases:
* Downstream parent notifies us of disconnection (DEL_ROUTE)
* We notify ourself of disconnection
* We notify ourself and so does downstream parent
It's case 3 that causes the error.
6 years ago
David Wilson