Commit Graph

1268 Commits (3e48f954396d3e973ef2bbc3871ddbfba4327a0a)
 

Author SHA1 Message Date
David Wilson b3a5fa70b0 core: copy debug setting to child's Router too.
core.Router doesn't pay attention to this attribute, but after
upgrade_router() has been called, the new parent.Router will.
7 years ago
David Wilson 64b60be50c tests: split runner_new_process out of runner_one_job 7 years ago
David Wilson 23b2a545cf fork: avoid another logging deadlock at startup.
The very first task /must/ be clearing out logging locks, since
_at_fork() functions call LOG.debug() via Side.close(). Additionally,
the root logger is not included in loggerDict, so we must specify it
explicitly.
7 years ago
David Wilson 785df88fa4 issue #186: core: remove long-forgotten hack.
This is likely to break something, it was definitely needed at some
point, but I never put much effort into figuring out why. Meanwhile,
Python appears to make find_module('ansible.module_utils.facts.')
requests in some circumstances, which causes us to indicate the module
exists while this hack exists.

So remove it, and let's see what breaks.
7 years ago
David Wilson 569c12a2d6 ansible: use PushFileService for module deps.
planner.py:
  * Rather than grant FileService access to a file for children, use
    PushFileService to trigger deduplicating send of the file through
    the hierarchy immediately.
  * Send the complete list of Ansible module imports to the target so
    runner.py knows which files and scripts must be loaded via
    PushFileService prior to detaching.

runner.py:
  * Teach NewStyleRunner to use the full module map to block until
    everything is loaded prior to detach().

target.py:
  * Delete old _get_file(), replace get_file() with get_small_file()
    which uses PushFileService instead.

Closes #186
7 years ago
David Wilson 7d4f4b205f ansible: update module preload list. 7 years ago
David Wilson 76beea6554 issue #186: move target._get_file into mitogen.service
For lack of a better place to keep the client function, make it a
classmethod of FileService itself for now.

The old _get_file() is removed in a subsequent commit.
7 years ago
David Wilson a3b747af1b issue #186: add PushFileService
This is like FileService but blocks until the file is pushed by a parent
context, with deduplicating behaviour at each level in the hierarchy. It
does not stream large files, so it is only suitable for small files like
Python modules.

Additionally add SerializedInvoker for use with PushFileService, which
ensures all method calls to a single service occur in sequence.
7 years ago
David Wilson 2e8c027322 issue #213: avoid service.Pool construction race
Ensure concurrent calls to service.Pool do not result in a duplicate
pool being constructed in a child.
7 years ago
David Wilson 1745c3aff0 issue #186: ansible: detach asynchronous tasks
After Runner.setup() has executed, but before the module executes. This
relies on subsequent commits to ensure all files are preloaded.
7 years ago
David Wilson 34daec4a7a core: prevent warning when CALL_FUNCTION used without reply_to
Such as when the stub CALL_SERVICE handler is used.
7 years ago
David Wilson 088a7e5cff ansible: handle "from timeout import timeout" imports.
It's not simple without executing a module to determine whether the
above refers to a submodule of a package, or an object defined within a
module.

Therefore detect when resolution of a child module yields the same path
as the parent, and ignore the result.
7 years ago
David Wilson 3aadfbcfa1 Add select to preamble_size 7 years ago
David Wilson daa9cfd0a8 ansible: MITOGEN_DUMP_THREAD_STACKS for mux process too 7 years ago
David Wilson 7162c13e05 docs: add more notes for getting_started.rst 7 years ago
David Wilson d9087c510b ansible: move FileService into mitogen.service. 7 years ago
David Wilson 9cb3878f3f nsible: remove unused master import 7 years ago
David Wilson fdbd954113 ansible: preload built-in modules in ModuleDepScanner.
For "ansible -m setup" over a 25ms link, avoids 65 roundtrips and
reduces runtime from 5.7s to 4.1s (-28%).

For "ansible -m setup" over a simulated 250 ms link, reduces runtime
from m27.015s to 0m8.254s (-69%).
7 years ago
David Wilson b577a11f86 master: fix IdAllocator log messages. 7 years ago
David Wilson f7d2eace08 tests: importer fixes 7 years ago
David Wilson 8d45e609ee ansible: preload always-requested modules.
Avoid 9 roundtrips during setup. In combination with previous change,
reduces 'ansible -m stat' execution over 25ms link from 3.7s to 3.07s.

1,3d0
< _on_get_module('ansible')
< _on_get_module('ansible.module_utils')
< _on_get_module('ansible.module_utils.basic')
69,74d65
< _on_get_module('ansible.module_utils.json_utils')
< _on_get_module('ansible.release')
< _on_get_module('ansible_mitogen')
< _on_get_module('ansible_mitogen.runner')
< _on_get_module('ansible_mitogen.target')
< _on_get_module('mitogen.fork')
7 years ago
David Wilson f7b368b1fb master: implement ModuleResponder.forward_module(). 7 years ago
David Wilson a578250bfb ansible: remove indirect master.py imports.
Avoids sending 10 modules:

77d76
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'ansible_mitogen.module_finder')
79d77
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'ansible_mitogen.services')
81,84d78
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'mitogen.compat')
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'mitogen.compat.collections')
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'mitogen.compat.functools')
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'mitogen.compat.tokenize')
86,87d79
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'mitogen.master')
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'mitogen.minify')
89,90d80
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'mitogen.select')
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'mitogen.service')
7 years ago
David Wilson 9492dbc4d7 parent: split out minify.py and add stub where master can install it.
This needs a cleaner mechanism to install it, at least this one is
documented.
7 years ago
David Wilson 325d13538f issue #196: debug: don't statically import master. 7 years ago
David Wilson cee76ee7b9 tests: gcloud setup fixes. 7 years ago
David Wilson aba6cb302a docs: add example sudoers rule
hat tip @seuf :)
7 years ago
dw c24d29d367
Merge pull request #255 from dw/dmw
Dmw
7 years ago
David Wilson 3b0addcfb0 service: v2. Closes #213 7 years ago
David Wilson 469bde63c2 parent: fix log message ordering 7 years ago
David Wilson a4ddef25a1 core: move reader/writer debug prints
They stop working with kqueue/epoll poller in the old location. Also
comment them out again, should never have been checked in uncommented.
7 years ago
David Wilson 3f595bbc7e issue #213: use import_module() in parent.py.
This dynamic import crap really needs to be ripped out of parent.py
again. Static imports work much better for the module loader too.
7 years ago
David Wilson e118963b30 issue #254: fork: take care not to rely on FD numbers. 7 years ago
David Wilson fc59f57ba2 issue #213: core: split out import_module() for use in services.py. 7 years ago
David Wilson 49fb25ee1c issue #213: core: fix shutdown crash due to member variable rename 7 years ago
David Wilson 40c6c6426f issue #213: core: fix test breakage due to log message change 7 years ago
David Wilson 2310497d55 issue #213: core: have Message.reply() log msg for zero reply_to
It's easy to call msg.reply() by accident on a message that never had
reply_to set, resulting in a "invalid handle" error message coming from
router. Instead log a more accurate message on the stack that actualy
caused the problem.
7 years ago
David Wilson 633585524f tests: don't need separate module for id_allocation_test any more
This used to be because everything would explode while importing
__main__ under py.test, but that was fixed months ago.
7 years ago
David Wilson dd48c41332 Ignore another annoying flake8 message. 7 years ago
David Wilson d2714752ee docs: tidy ups 7 years ago
David Wilson 61365236ad docs/select: fix up more references, fix headings. 7 years ago
David Wilson 4bf3d01104 select: add missing get(block=..) parameter. 7 years ago
David Wilson ddf28987a0 master: split Select() into new module to reduce wire size.
service.py currently imports master.py(+parent.py) just to get Select().
7 years ago
David Wilson 7a592d1c34 core: better Poller.__repr__ 7 years ago
David Wilson b0ce6eecd7 fork: support on_start= argument. 7 years ago
David Wilson 00edf0d66d core: have ExternalContext accept a config dict rather than kwargs.
The parameter lists had gotten out of control.
7 years ago
David Wilson eac4cc7afe tests: speed things up 7 years ago
David Wilson 863a95e860 docs: update contributors 7 years ago
David Wilson b0aa413173 tests: import benchmark reproduction setup playbook
Incomplete, also needs to mess around with Gcloud routing and handle box
setup/teardown, because that's another "well engineered" waste of time.
7 years ago
David Wilson a99c7a12f9 tests: output split groups in gcloud.py 7 years ago