Commit Graph

72 Commits (cfb94e463fecd36a228cbd1fc79576dda472e537)

Author SHA1 Message Date
David Wilson 3f31b166f6 issue #461: Ansible 2.3-compatible _get_candidate_temp_dirs(). 6 years ago
David Wilson 04755c3321 issue #426: tighten up PushFileService types.
Bytes/Unicode mixing caused a hang, so prevent bytes entirely.
6 years ago
David Wilson 76ec4f201c issue #413: paper over harmless duplicate del_route()
Ideally it would only be called once, and in future maybe it can, but
right now we need to cope with these cases:

* Downstream parent notifies us of disconnection (DEL_ROUTE)
* We notify ourself of disconnection
* We notify ourself and so does downstream parent

It's case 3 that causes the error.
6 years ago
David Wilson bac28bc5ca issue #76, #370: add fix for disconnect cleanup test
Simply listen to RouteMonitor's Context "disconnect"  and forget
contexts according to RouteMonitor's rules, rather than duplicate them
(and screw it up).
6 years ago
David Wilson c148c869e6 issue #76, #370: add disconnect cleanup test 6 years ago
David Wilson d280bba02b issue #369: fix KeyError during new context start.
Update _via_by_context earlier; fixes:

    Traceback (most recent call last):
      File "/Users/dmw/src/mitogen/mitogen/service.py", line 519, in _on_service_call
	return invoker.invoke(method_name, kwargs, msg)
      File "/Users/dmw/src/mitogen/mitogen/service.py", line 253, in invoke
	response = self._invoke(method_name, kwargs, msg)
      File "/Users/dmw/src/mitogen/mitogen/service.py", line 239, in _invoke
	ret = method(**kwargs)
      File "/Users/dmw/src/mitogen/ansible_mitogen/services.py", line 454, in get
	reraise(*result)
      File "/Users/dmw/src/mitogen/ansible_mitogen/services.py", line 412, in _wait_or_start
	response = self._connect(key, spec, via=via)
      File "/Users/dmw/src/mitogen/ansible_mitogen/services.py", line 363, in _connect
	self._update_lru(context, spec, via)
      File "/Users/dmw/src/mitogen/ansible_mitogen/services.py", line 266, in _update_lru
	self._update_lru_unlocked(new_context, spec, via)
      File "/Users/dmw/src/mitogen/ansible_mitogen/services.py", line 253, in _update_lru_unlocked
	if self._refs_by_context[context] == 0:
    KeyError: Context(1008, u'ssh.localhost.sudo.mitogen__user3')
6 years ago
David Wilson 9b7c958e2e issue #369: refactor ContextService to support reset(). 6 years ago
David Wilson d0f5671887 ansible: split key_from_dict() out into free function. 6 years ago
David Wilson fba52a0edf issue #76: add API for ansible_mitogen to get route list
Earlier commit moved Stream.routes attribute into a private map
belonging to RouteMonitor, to make upgrades smoother. This adds a new
accessor method to RouteMonitor.
6 years ago
David Wilson 7a00e1cc87 issue #360: missing locks around shutdown and LRU management. 6 years ago
David Wilson 498db57ec8 issue #360: ansible: missing lock around ContextService.put(). 6 years ago
David Wilson 32751cd356 master: allow batching context switches for forward_modules()
-7 switches per task.
6 years ago
David Wilson 3c6b72b452 ansible: gracefully return (and explain) ChannelError in ContextService.
When Ansible abnormally shuts down, the broker begins
force-disconnecting every context, including those for which connection
is currently in-progress.

When that happens, .call(init_child) throws ChannelError, and that needs
returned back to the worker, assuming the worker still even exists.

This solution is incomplete: with sick nodes, it's also possible the
worker died naturally, and so the worker should perhaps respond by
retrying the connection.

Previously, the unhandled ChannelError would spam the console when e.g.
fork() began returning EAGAIN.
6 years ago
David Wilson 4afe2fdc55 issue #321: fix probable threading issue. 6 years ago
David Wilson f24f02ba06 issue #321: take remote_tmp and system_tmpdirs into account.
Can't simply ignore these settings as some users may have weird noexec
filesystems.
6 years ago
David Wilson d62e6e2a7f ansible: serialize calls to ModuleDepService.
Concurrent calls to ModuleDepService would cause significant wasted
work, as potentially all pool threads run the same uncached module dep
scan.

Without:
         3243581 function calls (3233009 primitive calls) in 4770.672 seconds

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     2523    0.011    0.000   39.849    0.016 services.py:409(scan)

With:
         2801561 function calls (2800042 primitive calls) in 5166.843 seconds

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     2506    0.009    0.000    1.967    0.001 services.py:411(scan)

Ignore timing variance due to problems with the test job.
6 years ago
David Wilson a8e4dcc98d issue #301: correct remote_tmp evaluation context.
Vanilla Ansible expands remote_tmp variables in the context of the login
account, not any become_user account.
6 years ago
David Wilson c5ea7c45a1 comments/docs: correct mitogen.master.Context -> mitogen.parent.Context. 6 years ago
David Wilson 17dda781c0 issue #317: ansible: fix log filtering in several cases
* mitogen/ansible_mitogen should only generate ERROR-level logs in
  log_path unless -vvv is enabled.
* Targets were accidentally configured to always have DEBUG set, causing
  many log messages to be sent on the wire even though they would be
  filtered in the master.

Closes #317.
6 years ago
David Wilson 410016ff47 Initial Python 3.x port work.
* ansible: use unicode_literals everywhere since it only needs to be
  compatible back to 2.6.
* compat/collections.py: delete this entirely and rip out the parts of
  functools that require it.
* Introduce serializable Kwargs dict subclass that translates keys to
  Unicode on instantiation.
* enable_debug_logging() must set _v/_vv globals.
* cStringIO does not exist in 3.x.
* Treat IOLogger and LogForwarder input as latin-1.
* Avoid ResourceWarnings in first stage by explicitly closing fps.
* Fix preamble_size.py syntax errors.
6 years ago
David Wilson caffaa79f7 issue #186: rework async/forked tasks again.
The controller must know the ID of the forked child in order to
propagate dependencies to it, so forking+starting the module run cannot
happen entirely on the target, without some additional mechanism to
wait-and-repropagate the deps as they arrive on the target.

Rework things so that init_child() also handles starting the fork parent,
and returns it along with the context's home directory in a single round
trip.

Now master knows the identity of the fork parent, it can directly create
fork children and call run_module_async() in them. This necessitates 2
roundtrips to start an asynchronous task.

This whole thing sucks and entirely needs simplified, but for now things
almost work, so keeping it.

connection.py:
  * Expect ContextService to return the entire dict return value of
    init_child(). Store the fork_contxt from the return value.

planner.py:
  * Rework Planner to store the invocation as an instance attribute, to
    simplify method calls.
  * Add Planner.get_push_files() and Planner.get_module_deps().
  * Add _propagate_deps() which takes a Planner and ensures the deps it
    describes are sent to a (non forked or forked) context.
  * Move async task logic out of target.py and into invoke() /
    _invoke_*().

process.py:
  * Services no longer need references to each other. planner.py handles
    sending module deps with one extra RPC.

services.py:
  * Return "init_child_result" key instead of simple "home_dir" key.
  * Get rid of dep propagation from ModuleDepService, it lives in
    planner.py now.

target.py:
  * Get rid of async task start logic, lives in planner.py now.
6 years ago
David Wilson 569c12a2d6 ansible: use PushFileService for module deps.
planner.py:
  * Rather than grant FileService access to a file for children, use
    PushFileService to trigger deduplicating send of the file through
    the hierarchy immediately.
  * Send the complete list of Ansible module imports to the target so
    runner.py knows which files and scripts must be loaded via
    PushFileService prior to detaching.

runner.py:
  * Teach NewStyleRunner to use the full module map to block until
    everything is loaded prior to detach().

target.py:
  * Delete old _get_file(), replace get_file() with get_small_file()
    which uses PushFileService instead.

Closes #186
6 years ago
David Wilson 7d4f4b205f ansible: update module preload list. 6 years ago
David Wilson d9087c510b ansible: move FileService into mitogen.service. 6 years ago
David Wilson 9cb3878f3f nsible: remove unused master import 6 years ago
David Wilson fdbd954113 ansible: preload built-in modules in ModuleDepScanner.
For "ansible -m setup" over a 25ms link, avoids 65 roundtrips and
reduces runtime from 5.7s to 4.1s (-28%).

For "ansible -m setup" over a simulated 250 ms link, reduces runtime
from m27.015s to 0m8.254s (-69%).
6 years ago
David Wilson 8d45e609ee ansible: preload always-requested modules.
Avoid 9 roundtrips during setup. In combination with previous change,
reduces 'ansible -m stat' execution over 25ms link from 3.7s to 3.07s.

1,3d0
< _on_get_module('ansible')
< _on_get_module('ansible.module_utils')
< _on_get_module('ansible.module_utils.basic')
69,74d65
< _on_get_module('ansible.module_utils.json_utils')
< _on_get_module('ansible.release')
< _on_get_module('ansible_mitogen')
< _on_get_module('ansible_mitogen.runner')
< _on_get_module('ansible_mitogen.target')
< _on_get_module('mitogen.fork')
6 years ago
David Wilson 3b0addcfb0 service: v2. Closes #213 6 years ago
David Wilson bb61745a1a issue #217: pass through non-custom module utils to regular importer.
This may come back to bite later, but in the meantime it avoids shipping
up to 12KiB of junk metadata for every single task invocation.

For detachment (aka. async), we must ensure the target has two types of
preloads completed (modules and module_utils files) before detaching.
6 years ago
David Wilson 30034877a5 issue #217: ansible: working, if extremely inefficient implementation 6 years ago
David Wilson 81b62d9a1a issue #217: ansible: beginnings of ModuleDepService. 6 years ago
David Wilson bd2cc0830c Enable unidirectional routing in Ansible; closes #132. 6 years ago
David Wilson 49eae23f92 issue #218: ansibe: use Secret and Blob types. 6 years ago
David Wilson f9e1905ec6 issue #199: ansible: stop writing temp files for new style modules
While adding support for non-new style module types, NewStyleRunner
began writing modules to a temporary file, and sys.argv was patched to
actually include the script filename. The argv change was never required
to fix any particular bug, and a search of the standard modules reveals
no argv users. Update argv[0] to be '', like an interactive interpreter
would have.

While fixing #210, new style runner began setting __file__ to the
temporary file path in order to allow apt.py to discover the Ansiballz
temporary directory. 5 out of 1,516 standard modules follow this
pattern, but in each case, none actually attempt to access __file__,
they just call dirname on it. Therefore do not write the contents of
file, simply set it to the path as it would exist, within a real
temporary directory.

Finally move temporary directory creation out of runner and into target.
Now a single directory exists for the duration of a run, and is emptied
by runner.py as necessary after each task invocation.

This could be further extended to stop rewriting non-new-style modules
in a with_items loop, but that's another step.

Finally the last bullet point in the documentation almost isn't a lie
again.
6 years ago
David Wilson 94e048a2e5 ansible: ensure FileService uses exact CHUNK_SIZE multiple
9.8% throughput increase with sudo.
6 years ago
David Wilson e1a3cea2f9 ansible: FileService: don't send empty last chunk 6 years ago
David Wilson 2a56c672ca ansible: FileService docstring updates. 6 years ago
David Wilson 69e5902e61 issue #212: support explicit acknowledgements in FileService. 6 years ago
David Wilson b0309b539c ansible: disable interpreter recycling for connections.
Must explicitly specify enable_lru=True in ContextService.get() to
trigger recycling.
6 years ago
David Wilson 219a202a82 issue #226: ansible: file transfer improvements
* put_data() supports setting mode and times.
* put_file() refuses to copy non-regular files (sockets, FIFOs).
* put_file() saves one RTT for <32KiB files by using put_data() and
  embedding file content in argument list.
* FileService returns dict with size/mode/owner/group/mtime/atime.
* FileService refuses to copy non-regular files.
* transfer_file() preserves file mode.
* transfer_file() preserves atime/mtime.
* transfer_file() optionally preserves ownership.
* transfer_file() optionally calls fsync().
* transfer_file() uses unique temporary file name to avoid conflicting
  with parallel transfers.
* transfer_file() ensures temporary file is deleted on any error.
* write_path() writes to a temporary file and deletes it on failure.
* write_path() uses unique temporary file name to avoid conflicting
  with parallel transfers.
* write_path() supports setting symbolic owner/group.
* write_path() optionally calls fsync().
* write_path() supports setting symbolic mode/mtime/atime.

Closes #226, #227, #229
6 years ago
David Wilson 95039eea11 ansible: make key_from_kwargs() 10x faster
It was half the cost of the service call
6 years ago
David Wilson 3fab8a3af5 ansible: connection delegation v1
This implements the first edition of Connection Delegation, where
delegating connection establishment is initially single-threaded.

ansible_mitogen/strategy.py:
ansible_mitogen/plugins/connection/*:

  Begin splitting connection.Connection into subclasses, exposing them
  directly as "mitogen_ssh", "mitogen_local", etc. connection types.

  This is far from removing strategy.py, but it's a tiny start.

ansible_mitogen/connection.py:

  * config_from_play_context() and config_from_host_vars() build up a
    huge dictionary containing either more or less PlayContext contents,
    or our best attempt at reconstructing a host's connection config
    from its hostvars, where that config is not the current
    WorkerProcess target.

    They both produce the same format with the same keys, allowing
    remaining code to have a single input format.

    These dicts contain fields named after how Ansible refers to them,
    e.g. "sudo_exe".

  * _config_from_via() parses a basic connection specification like
    "username@inventory_name" into one of the aforementioned dicts.

  * _stack_from_config() produces a list of dicts describing the order
    in which (Mitogen) connections should be established, such that each
    element is proxied via= the previous element. The dicts produced by
    this function use Mitogen keyword arguments, the former di.

    These dicts contain fields named after how Mitogen refers to them,
    e.g. "sudo_path".

  * Pass the stack to ContextService, which is responsible for actual
    setup of the full chain.

ansible_mitogen/services.py:

  Teach get() to walk the supplied stack, establishing each connection
  in turn, creating refounts for it before continuing.

  TODO: refcounting is broken in a variety of cases.
6 years ago
David Wilson 7c6ce726aa ansible: rename variable to reflect correct time unit 6 years ago
David Wilson 2f1df7f82d ansible: FileService wasn't sleeping properly.
"_schedule_pending" is a function, "_pending_by_stream" is the map we
want to test.
6 years ago
David Wilson b2abe74ab6 issue #210: run DebOps under v2.5.1 too. 6 years ago
David Wilson 89fc842ca8 ansible: typo. 6 years ago
David Wilson 21082cec40 ansible: fix ugly formatting. 6 years ago
David Wilson 376fc85000 ansible: FileService docstrings. 6 years ago
David Wilson cf30e88a3e ansible: implement missing FileService.on_shutdown() 6 years ago
David Wilson 5913be64d7 docs: remove last remaining major risk :D 6 years ago