Commit Graph

55 Commits (00ae90b2b232f2f544667eb98ca74bad3844f3c2)

Author SHA1 Message Date
David Wilson 599da0689a issue #477 / ansible: avoid a race in async job startup.
Ansible 2.3/Python 2.4 work revealed there is no guarantee a slow target
will have written the initial job status file out before a fast
controller makes an initial check for it. Therefore, provide AsyncRunner
with a sender it should send a message to when the initial job file has
been written.

As a bonus, also catch and report exceptions happening early in
AsyncRunner, rather than leaving them to end up in -vvv output.
6 years ago
David Wilson 97f3cfe4f4 issue #477: target.file_exists() wrapper.
os.path.exists physical module name varies across major Python versions.
6 years ago
David Wilson 8f5b65f7ec issue #477: introduce subprocess isolation.
Since Python 2.4 fork is so defective, we must use subprocesses for
mitogen_task_isolation=fork. This has plenty of upside, since the long
term goal is to dump forking altogether. This allows a gentle
introduction of its replacement.
6 years ago
David Wilson b9924683ac ansible: docstring fixes. 6 years ago
David Wilson c9ff4c6506 issue #477: fix 3.x failure in new target.set_file_mode() function. 6 years ago
David Wilson aa5d37af89 issue #477: fix 3.x failure in new target.set_file_mode() function. 6 years ago
David Wilson 84a0424749 issue #477: fix 3.x test regressions. 6 years ago
David Wilson 256628c149 issue #477: backport ansible_mitogen/target.py to Python2.4 6 years ago
David Wilson 4bc0d0ec13 issue #426: apply_mode_spec() must handle bytes. 6 years ago
David Wilson 87ddead0b2 issue #435, #362: ansible: avoid touching setrlimit() on target.
This replaces the previous method for capping poorly Popen()
performance, instead entirely monkey-patching the problem function
rather than simply working around it.
6 years ago
David Wilson b599253291 issue #435: increase file descriptor limit by 8x.
This is a temporary solution at best.
6 years ago
David Wilson 35092c5d35 tests: Unicode/bytes fixes for integration/connection/exec_command.yml 6 years ago
David Wilson 0dc3f8accf ansible: fix another target.py format string. 6 years ago
David Wilson 7e04ee8af9 ansible: fix is_good_temp_dir() log format 6 years ago
David Wilson 9d070541d9 ansible: try to create tempdir if missing.
Closes #358.
6 years ago
David Wilson f8bf780e21 issue #362: Py3.x fixes. 6 years ago
David Wilson f8b6c774dd issue #362: cap max open files in children. 6 years ago
David Wilson 5521945bd2 ansible: temporary files take 5. 6 years ago
David Wilson f24f02ba06 issue #321: take remote_tmp and system_tmpdirs into account.
Can't simply ignore these settings as some users may have weird noexec
filesystems.
6 years ago
David Wilson a2686b1a2c issue #321: simplify temp directory handling. 6 years ago
David Wilson f4a66194e4 ansible: Py3.x fixes for Ansible PR #41749 workaround. 6 years ago
David Wilson f8b3441431 ansible: work around Ansible PR #41749 6 years ago
David Wilson 17dda781c0 issue #317: ansible: fix log filtering in several cases
* mitogen/ansible_mitogen should only generate ERROR-level logs in
  log_path unless -vvv is enabled.
* Targets were accidentally configured to always have DEBUG set, causing
  many log messages to be sent on the wire even though they would be
  filtered in the master.

Closes #317.
6 years ago
David Wilson b6d6468c92 issue #301: support expandvars() for remote_tmp only.
Vanilla Ansible support expandvars-like expansions widely in a variety
of places. Prefer to whitelist those we need, rather than sprinkling
hellish semantics everywhere.
6 years ago
David Wilson bd65e50b2f ansible: reduce() does not exist in 3.x. 6 years ago
David Wilson 410016ff47 Initial Python 3.x port work.
* ansible: use unicode_literals everywhere since it only needs to be
  compatible back to 2.6.
* compat/collections.py: delete this entirely and rip out the parts of
  functools that require it.
* Introduce serializable Kwargs dict subclass that translates keys to
  Unicode on instantiation.
* enable_debug_logging() must set _v/_vv globals.
* cStringIO does not exist in 3.x.
* Treat IOLogger and LogForwarder input as latin-1.
* Avoid ResourceWarnings in first stage by explicitly closing fps.
* Fix preamble_size.py syntax errors.
6 years ago
David Wilson 08538d327b ansible: don't write failed job result after async timeout.
The failed job result is likely to be "interrupted system call", and we
don't want that to overwrite the SIGALRM handler's "the task timed out",
so just discard it.
7 years ago
David Wilson 3994f1b30a ansible: implment async job time limit. 7 years ago
David Wilson e35694acd5 ansible: flake8 fixes. 7 years ago
David Wilson ae20a689ef issue #186: finally enable detach. 7 years ago
David Wilson caffaa79f7 issue #186: rework async/forked tasks again.
The controller must know the ID of the forked child in order to
propagate dependencies to it, so forking+starting the module run cannot
happen entirely on the target, without some additional mechanism to
wait-and-repropagate the deps as they arrive on the target.

Rework things so that init_child() also handles starting the fork parent,
and returns it along with the context's home directory in a single round
trip.

Now master knows the identity of the fork parent, it can directly create
fork children and call run_module_async() in them. This necessitates 2
roundtrips to start an asynchronous task.

This whole thing sucks and entirely needs simplified, but for now things
almost work, so keeping it.

connection.py:
  * Expect ContextService to return the entire dict return value of
    init_child(). Store the fork_contxt from the return value.

planner.py:
  * Rework Planner to store the invocation as an instance attribute, to
    simplify method calls.
  * Add Planner.get_push_files() and Planner.get_module_deps().
  * Add _propagate_deps() which takes a Planner and ensures the deps it
    describes are sent to a (non forked or forked) context.
  * Move async task logic out of target.py and into invoke() /
    _invoke_*().

process.py:
  * Services no longer need references to each other. planner.py handles
    sending module deps with one extra RPC.

services.py:
  * Return "init_child_result" key instead of simple "home_dir" key.
  * Get rid of dep propagation from ModuleDepService, it lives in
    planner.py now.

target.py:
  * Get rid of async task start logic, lives in planner.py now.
7 years ago
David Wilson 569c12a2d6 ansible: use PushFileService for module deps.
planner.py:
  * Rather than grant FileService access to a file for children, use
    PushFileService to trigger deduplicating send of the file through
    the hierarchy immediately.
  * Send the complete list of Ansible module imports to the target so
    runner.py knows which files and scripts must be loaded via
    PushFileService prior to detaching.

runner.py:
  * Teach NewStyleRunner to use the full module map to block until
    everything is loaded prior to detach().

target.py:
  * Delete old _get_file(), replace get_file() with get_small_file()
    which uses PushFileService instead.

Closes #186
7 years ago
David Wilson 76beea6554 issue #186: move target._get_file into mitogen.service
For lack of a better place to keep the client function, make it a
classmethod of FileService itself for now.

The old _get_file() is removed in a subsequent commit.
7 years ago
David Wilson 1745c3aff0 issue #186: ansible: detach asynchronous tasks
After Runner.setup() has executed, but before the module executes. This
relies on subsequent commits to ensure all files are preloaded.
7 years ago
David Wilson d9087c510b ansible: move FileService into mitogen.service. 7 years ago
David Wilson a578250bfb ansible: remove indirect master.py imports.
Avoids sending 10 modules:

77d76
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'ansible_mitogen.module_finder')
79d77
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'ansible_mitogen.services')
81,84d78
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'mitogen.compat')
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'mitogen.compat.collections')
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'mitogen.compat.functools')
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'mitogen.compat.tokenize')
86,87d79
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'mitogen.master')
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'mitogen.minify')
89,90d80
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'mitogen.select')
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'mitogen.service')
7 years ago
David Wilson 3b0addcfb0 service: v2. Closes #213 7 years ago
David Wilson bb61745a1a issue #217: pass through non-custom module utils to regular importer.
This may come back to bite later, but in the meantime it avoids shipping
up to 12KiB of junk metadata for every single task invocation.

For detachment (aka. async), we must ensure the target has two types of
preloads completed (modules and module_utils files) before detaching.
7 years ago
David Wilson 69f58875c8 tests: more ansible fixes from recent brakage. 7 years ago
David Wilson f737ff5276 ansible: stop passing through remote_tmp variable
Ansiballz does not use remote_tmp so neither should we, per #239.
7 years ago
David Wilson f9e1905ec6 issue #199: ansible: stop writing temp files for new style modules
While adding support for non-new style module types, NewStyleRunner
began writing modules to a temporary file, and sys.argv was patched to
actually include the script filename. The argv change was never required
to fix any particular bug, and a search of the standard modules reveals
no argv users. Update argv[0] to be '', like an interactive interpreter
would have.

While fixing #210, new style runner began setting __file__ to the
temporary file path in order to allow apt.py to discover the Ansiballz
temporary directory. 5 out of 1,516 standard modules follow this
pattern, but in each case, none actually attempt to access __file__,
they just call dirname on it. Therefore do not write the contents of
file, simply set it to the path as it would exist, within a real
temporary directory.

Finally move temporary directory creation out of runner and into target.
Now a single directory exists for the duration of a run, and is emptied
by runner.py as necessary after each task invocation.

This could be further extended to stop rewriting non-new-style modules
in a with_items loop, but that's another step.

Finally the last bullet point in the documentation almost isn't a lie
again.
7 years ago
David Wilson f5d22a3ca1 core: support deleting handlers, make Receiver.close() unregister 7 years ago
David Wilson 69e5902e61 issue #212: support explicit acknowledgements in FileService. 7 years ago
David Wilson 7c5bbc5168 setns: support changing user.
To match existing third party plugin.
7 years ago
David Wilson 003f30b5a9 issue #226: test fixes. 7 years ago
David Wilson 780b63520f issue #226: don't attempt to fchmod() a pathname 7 years ago
David Wilson 219a202a82 issue #226: ansible: file transfer improvements
* put_data() supports setting mode and times.
* put_file() refuses to copy non-regular files (sockets, FIFOs).
* put_file() saves one RTT for <32KiB files by using put_data() and
  embedding file content in argument list.
* FileService returns dict with size/mode/owner/group/mtime/atime.
* FileService refuses to copy non-regular files.
* transfer_file() preserves file mode.
* transfer_file() preserves atime/mtime.
* transfer_file() optionally preserves ownership.
* transfer_file() optionally calls fsync().
* transfer_file() uses unique temporary file name to avoid conflicting
  with parallel transfers.
* transfer_file() ensures temporary file is deleted on any error.
* write_path() writes to a temporary file and deletes it on failure.
* write_path() uses unique temporary file name to avoid conflicting
  with parallel transfers.
* write_path() supports setting symbolic owner/group.
* write_path() optionally calls fsync().
* write_path() supports setting symbolic mode/mtime/atime.

Closes #226, #227, #229
7 years ago
David Wilson b5be0fd65b ansible: log _get_file() timings. 7 years ago
David Wilson cb73c44084 ansible: implement streaming in Connection.put_file().
This is the function the copy module uses.
7 years ago
David Wilson 29087018c7 ansible: implement streaming in FileService.
This commit only uses it for the target.get_file() helper, which is only
used for transferring modules. The next commit wires it into the
Connection.transfer_file() API, which is the method the copy module
uses.
7 years ago