If PushService.store_and_forward() loses the race to arrive at a brand
new context first, and the context's main thread is already executing a
CALL_FUNCTION that is blocked on the result of PushService, deadlock
could occur in the old scheme.
Instead (for now) simply spam a thread for each incoming message, and
use the get_or_create_pool() lock to ensure things work out in the end.
This could potentially generate a huge number of threads given the wrong
app, but we'll fix that problem when it appears.
The controller must know the ID of the forked child in order to
propagate dependencies to it, so forking+starting the module run cannot
happen entirely on the target, without some additional mechanism to
wait-and-repropagate the deps as they arrive on the target.
Rework things so that init_child() also handles starting the fork parent,
and returns it along with the context's home directory in a single round
trip.
Now master knows the identity of the fork parent, it can directly create
fork children and call run_module_async() in them. This necessitates 2
roundtrips to start an asynchronous task.
This whole thing sucks and entirely needs simplified, but for now things
almost work, so keeping it.
connection.py:
* Expect ContextService to return the entire dict return value of
init_child(). Store the fork_contxt from the return value.
planner.py:
* Rework Planner to store the invocation as an instance attribute, to
simplify method calls.
* Add Planner.get_push_files() and Planner.get_module_deps().
* Add _propagate_deps() which takes a Planner and ensures the deps it
describes are sent to a (non forked or forked) context.
* Move async task logic out of target.py and into invoke() /
_invoke_*().
process.py:
* Services no longer need references to each other. planner.py handles
sending module deps with one extra RPC.
services.py:
* Return "init_child_result" key instead of simple "home_dir" key.
* Get rid of dep propagation from ModuleDepService, it lives in
planner.py now.
target.py:
* Get rid of async task start logic, lives in planner.py now.
The very first task /must/ be clearing out logging locks, since
_at_fork() functions call LOG.debug() via Side.close(). Additionally,
the root logger is not included in loggerDict, so we must specify it
explicitly.
This is likely to break something, it was definitely needed at some
point, but I never put much effort into figuring out why. Meanwhile,
Python appears to make find_module('ansible.module_utils.facts.')
requests in some circumstances, which causes us to indicate the module
exists while this hack exists.
So remove it, and let's see what breaks.
planner.py:
* Rather than grant FileService access to a file for children, use
PushFileService to trigger deduplicating send of the file through
the hierarchy immediately.
* Send the complete list of Ansible module imports to the target so
runner.py knows which files and scripts must be loaded via
PushFileService prior to detaching.
runner.py:
* Teach NewStyleRunner to use the full module map to block until
everything is loaded prior to detach().
target.py:
* Delete old _get_file(), replace get_file() with get_small_file()
which uses PushFileService instead.
Closes#186
For lack of a better place to keep the client function, make it a
classmethod of FileService itself for now.
The old _get_file() is removed in a subsequent commit.
This is like FileService but blocks until the file is pushed by a parent
context, with deduplicating behaviour at each level in the hierarchy. It
does not stream large files, so it is only suitable for small files like
Python modules.
Additionally add SerializedInvoker for use with PushFileService, which
ensures all method calls to a single service occur in sequence.
It's not simple without executing a module to determine whether the
above refers to a submodule of a package, or an object defined within a
module.
Therefore detect when resolution of a child module yields the same path
as the parent, and ignore the result.
For "ansible -m setup" over a 25ms link, avoids 65 roundtrips and
reduces runtime from 5.7s to 4.1s (-28%).
For "ansible -m setup" over a simulated 250 ms link, reduces runtime
from m27.015s to 0m8.254s (-69%).
It's easy to call msg.reply() by accident on a message that never had
reply_to set, resulting in a "invalid handle" error message coming from
router. Instead log a more accurate message on the stack that actualy
caused the problem.
David Wilson