Commit Graph

216 Commits (7d4f4b205f6b5683c6c6c46872bb823af8f6188d)

Author SHA1 Message Date
David Wilson 7d4f4b205f ansible: update module preload list. 7 years ago
David Wilson 76beea6554 issue #186: move target._get_file into mitogen.service
For lack of a better place to keep the client function, make it a
classmethod of FileService itself for now.

The old _get_file() is removed in a subsequent commit.
7 years ago
David Wilson 1745c3aff0 issue #186: ansible: detach asynchronous tasks
After Runner.setup() has executed, but before the module executes. This
relies on subsequent commits to ensure all files are preloaded.
7 years ago
David Wilson 088a7e5cff ansible: handle "from timeout import timeout" imports.
It's not simple without executing a module to determine whether the
above refers to a submodule of a package, or an object defined within a
module.

Therefore detect when resolution of a child module yields the same path
as the parent, and ignore the result.
7 years ago
David Wilson daa9cfd0a8 ansible: MITOGEN_DUMP_THREAD_STACKS for mux process too 7 years ago
David Wilson d9087c510b ansible: move FileService into mitogen.service. 7 years ago
David Wilson 9cb3878f3f nsible: remove unused master import 7 years ago
David Wilson fdbd954113 ansible: preload built-in modules in ModuleDepScanner.
For "ansible -m setup" over a 25ms link, avoids 65 roundtrips and
reduces runtime from 5.7s to 4.1s (-28%).

For "ansible -m setup" over a simulated 250 ms link, reduces runtime
from m27.015s to 0m8.254s (-69%).
7 years ago
David Wilson 8d45e609ee ansible: preload always-requested modules.
Avoid 9 roundtrips during setup. In combination with previous change,
reduces 'ansible -m stat' execution over 25ms link from 3.7s to 3.07s.

1,3d0
< _on_get_module('ansible')
< _on_get_module('ansible.module_utils')
< _on_get_module('ansible.module_utils.basic')
69,74d65
< _on_get_module('ansible.module_utils.json_utils')
< _on_get_module('ansible.release')
< _on_get_module('ansible_mitogen')
< _on_get_module('ansible_mitogen.runner')
< _on_get_module('ansible_mitogen.target')
< _on_get_module('mitogen.fork')
7 years ago
David Wilson a578250bfb ansible: remove indirect master.py imports.
Avoids sending 10 modules:

77d76
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'ansible_mitogen.module_finder')
79d77
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'ansible_mitogen.services')
81,84d78
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'mitogen.compat')
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'mitogen.compat.collections')
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'mitogen.compat.functools')
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'mitogen.compat.tokenize')
86,87d79
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'mitogen.master')
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'mitogen.minify')
89,90d80
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'mitogen.select')
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'mitogen.service')
7 years ago
David Wilson 3b0addcfb0 service: v2. Closes #213 7 years ago
David Wilson ddf28987a0 master: split Select() into new module to reduce wire size.
service.py currently imports master.py(+parent.py) just to get Select().
7 years ago
David Wilson 1eb5c20f57 ansible: add dummy init.pys so setup.py find_packages() DTRT. 7 years ago
David Wilson bb61745a1a issue #217: pass through non-custom module utils to regular importer.
This may come back to bite later, but in the meantime it avoids shipping
up to 12KiB of junk metadata for every single task invocation.

For detachment (aka. async), we must ensure the target has two types of
preloads completed (modules and module_utils files) before detaching.
7 years ago
David Wilson 30034877a5 issue #217: ansible: working, if extremely inefficient implementation 7 years ago
David Wilson 81b62d9a1a issue #217: ansible: beginnings of ModuleDepService. 7 years ago
David Wilson 0f8190eff6 issue #217: ansible: working module_finder.py 7 years ago
David Wilson 6a17042020 issue #217: ansible: merge duplicate ReplacerPlanner. 7 years ago
David Wilson bd2cc0830c Enable unidirectional routing in Ansible; closes #132. 7 years ago
David Wilson f96c552f87 issue #217: initial module scanner code.
This is sketch code, it's being done separately from mitogen.master.* to
begin with to avoid breaking what's there.
7 years ago
David Wilson 2ad0d0521d ansible: reload /etc/resolv.conf for each task.
The OpenShift installer modifies /etc/resolv.conf then tests the new
resolver configuration, however, there was no mechanism to reload
resolv.conf in our reuseable interpreter.

https://github.com/openshift/openshift-ansible/blob/release-3.9/roles/openshift_web_console/tasks/install.yml#L137

This inserts an explicit call to res_init() for every new style
invocation, with an approximate cost of ~1usec on Linux since glibc
verifies resolv.conf has changed before reloading it.

There is little to be done for users of the thread-safe resolver APIs,
their state is hidden from us. If bugs like that manifest, whack-a-mole
style 'del sys.modules[thatmod]' patches may suffice.
7 years ago
David Wilson 49eae23f92 issue #218: ansibe: use Secret and Blob types. 7 years ago
David Wilson 69f58875c8 tests: more ansible fixes from recent brakage. 7 years ago
David Wilson ad1f624750 ansible: document and rearrange Runner params.
Move emulate_tty to where it's used.
7 years ago
David Wilson 1186b927f9 ansible: remove seemingly unused raw_params
Traced git log all the way back to beginning of time, and checked
Ansible versions starting Jan 2016. Zero clue where this came from, but
the convention suggests it came from Ansible at some point.
7 years ago
David Wilson f737ff5276 ansible: stop passing through remote_tmp variable
Ansiballz does not use remote_tmp so neither should we, per #239.
7 years ago
David Wilson f9e1905ec6 issue #199: ansible: stop writing temp files for new style modules
While adding support for non-new style module types, NewStyleRunner
began writing modules to a temporary file, and sys.argv was patched to
actually include the script filename. The argv change was never required
to fix any particular bug, and a search of the standard modules reveals
no argv users. Update argv[0] to be '', like an interactive interpreter
would have.

While fixing #210, new style runner began setting __file__ to the
temporary file path in order to allow apt.py to discover the Ansiballz
temporary directory. 5 out of 1,516 standard modules follow this
pattern, but in each case, none actually attempt to access __file__,
they just call dirname on it. Therefore do not write the contents of
file, simply set it to the path as it would exist, within a real
temporary directory.

Finally move temporary directory creation out of runner and into target.
Now a single directory exists for the duration of a run, and is emptied
by runner.py as necessary after each task invocation.

This could be further extended to stop rewriting non-new-style modules
in a with_items loop, but that's another step.

Finally the last bullet point in the documentation almost isn't a lie
again.
7 years ago
David Wilson 1bc08323bf ansible: more compatible module script naming. 7 years ago
David Wilson 8bd34e1e28 ansible: gracefully report connection timeouts as StreamError. 7 years ago
David Wilson 4d1c6d2101 issue #127: ssh: reasonable solution to host key checking.
Ideally it would be possible to specify a callback function, but this is
not possible for proxied connections. So simply provide the 3 most
useful modes, defaulting to the most secure.

Closes #127. Closes #134.
7 years ago
David Wilson 9cc8b21a07 ansible: missing mitogen_su module. 7 years ago
David Wilson 78c401ba4d ansible: support su become method. 7 years ago
David Wilson 3203846708 issue #239: ansible: ignore remote_tmp in new style runner. 7 years ago
David Wilson 94e048a2e5 ansible: ensure FileService uses exact CHUNK_SIZE multiple
9.8% throughput increase with sudo.
7 years ago
David Wilson dafe12b315 ansible: fix AnsibleUnicode crash when processing "~username". 7 years ago
David Wilson e1a3cea2f9 ansible: FileService: don't send empty last chunk 7 years ago
David Wilson 2a56c672ca ansible: FileService docstring updates. 7 years ago
David Wilson f5d22a3ca1 core: support deleting handlers, make Receiver.close() unregister 7 years ago
David Wilson 69e5902e61 issue #212: support explicit acknowledgements in FileService. 7 years ago
David Wilson b0309b539c ansible: disable interpreter recycling for connections.
Must explicitly specify enable_lru=True in ContextService.get() to
trigger recycling.
7 years ago
David Wilson 65e6a44fe7 docs: add links. 7 years ago
David Wilson 90f7b4baea ansible: stub plugin documentation. 7 years ago
David Wilson c85a5b6446 ansible: make call timing more readable 7 years ago
David Wilson bba2a42e44 ansible: add mitogen_sudo method, split out connection subclasses.
Slowly moving towards real implementations in those files.
7 years ago
David Wilson 0a9126c510 ansible: connection plugins missing from previous commit. 7 years ago
David Wilson 7c5bbc5168 setns: support changing user.
To match existing third party plugin.
7 years ago
David Wilson 003f30b5a9 issue #226: test fixes. 7 years ago
David Wilson 6edb3f165d ansible: avoid a race during shutdown. 7 years ago
David Wilson 780b63520f issue #226: don't attempt to fchmod() a pathname 7 years ago
David Wilson 219a202a82 issue #226: ansible: file transfer improvements
* put_data() supports setting mode and times.
* put_file() refuses to copy non-regular files (sockets, FIFOs).
* put_file() saves one RTT for <32KiB files by using put_data() and
  embedding file content in argument list.
* FileService returns dict with size/mode/owner/group/mtime/atime.
* FileService refuses to copy non-regular files.
* transfer_file() preserves file mode.
* transfer_file() preserves atime/mtime.
* transfer_file() optionally preserves ownership.
* transfer_file() optionally calls fsync().
* transfer_file() uses unique temporary file name to avoid conflicting
  with parallel transfers.
* transfer_file() ensures temporary file is deleted on any error.
* write_path() writes to a temporary file and deletes it on failure.
* write_path() uses unique temporary file name to avoid conflicting
  with parallel transfers.
* write_path() supports setting symbolic owner/group.
* write_path() optionally calls fsync().
* write_path() supports setting symbolic mode/mtime/atime.

Closes #226, #227, #229
7 years ago