Commit Graph

416 Commits (06e52ca89f48d257059b1889bb1029b84500fa4d)

Author SHA1 Message Date
David Wilson 7e04ee8af9 ansible: fix is_good_temp_dir() log format 6 years ago
David Wilson fba52a0edf issue #76: add API for ansible_mitogen to get route list
Earlier commit moved Stream.routes attribute into a private map
belonging to RouteMonitor, to make upgrades smoother. This adds a new
accessor method to RouteMonitor.
6 years ago
David Wilson 7fd9fb0014 issue #397: fix another case where stray tmpdirs can be left behind.
Newer Ansibles use atexit.register() to invoke cleanup, so we need to
run those registrations after each run.
6 years ago
David Wilson 1b17aa1d1a ansible: fix temp cleanup regression and add test; closes #397. 6 years ago
David Wilson 9d070541d9 ansible: try to create tempdir if missing.
Closes #358.
6 years ago
David Wilson 4c81eba599 Merge commit 'refs/pull/377/head' of github.com:dw/mitogen into dmw
(Pull #377)

Changes:
- additional_parameters -> extra_args
- Merge with kubectl changes from dmw branch
- Update docs
- Remove unused username class member
- Avoid mutable kubectl_args class member
- Use six.iteritems
6 years ago
David Wilson 7a00e1cc87 issue #360: missing locks around shutdown and LRU management. 6 years ago
David Wilson 498db57ec8 issue #360: ansible: missing lock around ContextService.put(). 6 years ago
David Wilson f8bf780e21 issue #362: Py3.x fixes. 6 years ago
David Wilson f8b6c774dd issue #362: cap max open files in children. 6 years ago
David Wilson 5521945bd2 ansible: temporary files take 5. 6 years ago
Yannig Perré 17548d1e49 [Enhancement] handle kubectl vars from Ansible connector.
This change allows the kubectl connector to support the same options as
Ansible's original connector.

The playbook sample comes with an example of a pod containing two containers
and checking that moving from one container to another, the version of Python
changes as expected.
6 years ago
Yannig Perré 6828926a36 Kubernetes connection support for mitogen. 6 years ago
David Wilson 638b196a45 ansible: fix put_file() for large temporary files.
Reverts 49736b3a, large file copies can't avoid the RTT.

The parent stack must be blocked while FileService progresses, as unlike
the small file path, it does not make a snapshot of the (possibly
temporary) file passed by the action plug-in. So we need to keep that
file alive while the service runs.

Add a new integration test and a new soak test to cover both.
6 years ago
David Wilson 9ff34afafe ansible: fix regression. 6 years ago
David Wilson 90f89f95fb ansible: fix exec_command() regression. 6 years ago
David Wilson 49736b3ae8 ansible: fix FileService call, and remove another roundtrip. 6 years ago
David Wilson e241081cae ansible: stop sharing target temp_dir in runner.
This cannot work with delegate_to, since delegate_to permits multiple
concurrent tasks to be executing on the same target.
6 years ago
David Wilson 43d9815f6d ansible: use CallChain everywhere.
This replaces the 'dump to logger' behaviour of pipelined calls from
before with a call chain that returns any exception on next synchronized
call.
6 years ago
David Wilson b254eb3399 ansible: fix non-action connection instantiation.
e.g. by synchronize module.
6 years ago
David Wilson 705d77a9be ansible: remove a bunch more aliasing from connection.py. 6 years ago
David Wilson 66142e7d75 ansible: fork isolated tasks from correct parent.
Closes #355.
6 years ago
David Wilson da8c6b45b0 ansible: remove task_vars aliasing from connection.py.
Crazy spam creep.
6 years ago
David Wilson 32751cd356 master: allow batching context switches for forward_modules()
-7 switches per task.
6 years ago
David Wilson 86942b6bf9 ansible: add explanatory exception
If disconnection occurs during a Connection.call(), return
AnsibleConnectionFailure.
6 years ago
David Wilson 3c6b72b452 ansible: gracefully return (and explain) ChannelError in ContextService.
When Ansible abnormally shuts down, the broker begins
force-disconnecting every context, including those for which connection
is currently in-progress.

When that happens, .call(init_child) throws ChannelError, and that needs
returned back to the worker, assuming the worker still even exists.

This solution is incomplete: with sick nodes, it's also possible the
worker died naturally, and so the worker should perhaps respond by
retrying the connection.

Previously, the unhandled ChannelError would spam the console when e.g.
fork() began returning EAGAIN.
6 years ago
David Wilson e647adc62e ansible: copy GIL change from linear2 branch.
Reduces runtime by 25% given 100 25ms SSH targets:

    ANSIBLE_STRATEGY=mitogen \
    MITOGEN_POOL_SIZE=100 \
    /usr/bin/time -l ansible k3-x100 -m shell -a hostname

Before:
           39.56 real        35.29 user        17.24 sys
      59600896  maximum resident set size
       1784252  page reclaims
          9016  messages sent
         10382  messages received
         18774  voluntary context switches
        770070  involuntary context switches

After:
           29.79 real        22.10 user        11.77 sys
      59281408  maximum resident set size
       1725268  page reclaims
          8582  messages sent
          9959  messages received
         14582  voluntary context switches
         75280  involuntary context switches
6 years ago
David Wilson 2647f73501 ansible: bump UNIX listener default backlog, and set it to match forks.
The connection multiplexer can expect to not be scheduled at least until
every $forks worker processes has attempted a connection, so the backlog
must be able to hold every worker.
6 years ago
David Wilson 8ab11f415f ansible: better support for diagnosing hangs
* Always enable the faulthandler module in the top-level process if it
  is available.
* Make MITOGEN_DUMP_THREAD_STACKS interval configurable, to better
  handle larger runs.
* Add docs subsection on diagnosing hangs.

Conflicts:
	ansible_mitogen/process.py
6 years ago
David Wilson 9792b8b54f ansible: use template-expanded delegate_to hostname in one more location. 6 years ago
David Wilson 90c2ed03d0 ansible: fix synchronize module
Broken by recent connection delegation fixes.
6 years ago
David Wilson 7458dfae85 ansible: avoid roundtrip for small file transfers.
Calls to connect.put_file() where the file is sufficiently small enough
to fit in a single RPC proceed without waiting for an RPC response. If
the write fails the target context will log an exception, and any
subsequent step depending on the written file will fail.

I verified every built-in action plugin for file transfer calls, and
they all depend on the transferred file in the following step, so this
should be safe.

Reduces template/copy actions to 2-RTT, loop-20-templates.yml runtime
reduced from 30 seconds to 10 seconds over a 250ms link compared to
v0.2.2, and from 123 seconds compared to vanilla with pipelining
enabled.
6 years ago
David Wilson 084c0ac065 ansible: avoid roundtrip in copy action due to fixup_perms2().
On top of existing temporary files work, this reduces the number of
roundtrips required for "copy" and "template" actions from 6 to 3.
6 years ago
David Wilson e18396d54d ansible: enable profiling by default!
Thankfully this never made it into a release
6 years ago
David Wilson bce4f59138 issue #345: disable IdentitiesOnly by default. 6 years ago
David Wilson 2e3d04bbb8 issue #342: forward _create_control_path() to SSH plugin.
network_cli connection type loads the "ssh" (mitogen_ssh) plugin and
expects a private method to exist.
6 years ago
David Wilson ac9b84d237 issue #321: 2.4+ compatibility fixes, disable test on Vanilla. 6 years ago
David Wilson 4afe2fdc55 issue #321: fix probable threading issue. 6 years ago
David Wilson f24f02ba06 issue #321: take remote_tmp and system_tmpdirs into account.
Can't simply ignore these settings as some users may have weird noexec
filesystems.
6 years ago
David Wilson a2686b1a2c issue #321: simplify temp directory handling. 6 years ago
David Wilson 3d588323ff issue #340: use expanded delegate_to hostname, not template.
PlayContext.delegate_to is the unexpanded template, Ansible doesn't keep
a copy of it around anywhere convenient. We either need to re-expand it
or take the expanded version that was stored on the Task, which is what
is done here.
6 years ago
David Wilson a6995a5288 issue #338: refactor env handling into class and fix tests. 6 years ago
David Wilson 916e46621b issue #340: add connection delegation tests. 6 years ago
David Wilson 6dcd5f8998 issue #340: split up Connection._connect()
The logic was getting too busy.
6 years ago
David Wilson a1e653978b issue #340: connection delegation used wrong variable name.
When inventory name did not match remote_addr, it would attempt to SSH
to the inventory name.
6 years ago
David Wilson 370b98f960 ansible: tidy up connection.py.
- more docstrings.
- _wrap_or_none -> optional_secret()
6 years ago
David Wilson 9e572a7939 ansible: fix duplicate MuxProcess socket write.
The while: loop was necessary due to some cutpaste further on down the
file.
6 years ago
David Wilson b521f215fd ansible: handle >2.6 magic exceptions + sys.excepthook damage
Closes #332.
6 years ago
David Wilson 053c594d65 ansible: prevent logs spamming user console on exit.
Closes #331.
6 years ago
David Wilson 1f21a30e7f issue #251: ansible: watch for delegate_to during connection delegation.
This needs more work -- pretty certain that python_path and suchlike are
coming from the wrong place. Possibly we need another config_from_..()
specialized for delegate_to.
6 years ago
David Wilson 6c4b01642c ansible: don't crash when adhoc tries to run a missing module.
ansible-playbook prints a separate error during parsing stage, adhoc
performs no such check.
6 years ago
David Wilson 34a9f67115 issue #339: whoops, actually wire up new connection method. 6 years ago
David Wilson 1473f49505 ansible: emulate /etc/environment reloading behaviour of vanilla.
This change is relatively incomplete -- ideally we could snapshot
os.environ and /etc/environment at startup and respect key deletions
too, but that's a lot more work. Wait for a bug report instead.

Closes #338.
6 years ago
David Wilson 4077182fb2 ansible: plugins were missing absolute_import. 6 years ago
David Wilson 5c573f7fcb ansible: insert short sleep when MITOGEN_PROFILING active.
Hacky, but works fine.
6 years ago
David Wilson d62e6e2a7f ansible: serialize calls to ModuleDepService.
Concurrent calls to ModuleDepService would cause significant wasted
work, as potentially all pool threads run the same uncached module dep
scan.

Without:
         3243581 function calls (3233009 primitive calls) in 4770.672 seconds

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     2523    0.011    0.000   39.849    0.016 services.py:409(scan)

With:
         2801561 function calls (2800042 primitive calls) in 5166.843 seconds

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     2506    0.009    0.000    1.967    0.001 services.py:411(scan)

Ignore timing variance due to problems with the test job.
6 years ago
David Wilson 3e0de9790c issue #324: fix Python 3 fallout for custom module_utils.
Also enable at last one of its tests.
6 years ago
David Wilson f4a66194e4 ansible: Py3.x fixes for Ansible PR #41749 workaround. 6 years ago
David Wilson 6c03b83748 issue #291: don't attempt mitogen import until sys.path modified.
Given an extracted download of mitogen-2.2.tar.gz, with strategy_plugins
pointing into it, if an old version of the package was pip-installed,
then the old pip-installed package would be imported and override
whatever came from the tarball.

Instead, modify sys.path before attempting any import. This still isn't
perfect, but it's better.
6 years ago
David Wilson f8b3441431 ansible: work around Ansible PR #41749 6 years ago
David Wilson bfe9f81d0b ansible: fix RPC time logging.
Rendering call arguemtns was broken for non-positional arguments.
6 years ago
David Wilson 71b4294888 issue #291: 2.6 compat fix. 6 years ago
David Wilson b44b823c4a ansible: make _remote_expand_user() pay attention to sudoable=.. 6 years ago
David Wilson a8e4dcc98d issue #301: correct remote_tmp evaluation context.
Vanilla Ansible expands remote_tmp variables in the context of the login
account, not any become_user account.
6 years ago
David Wilson de0d526487 issue #291: restore behaviour of invoking binaries via /bin/sh
This ensures failed task output matches vanilla Ansible exactly (as it
did before starting #291).
6 years ago
David Wilson c5ea7c45a1 comments/docs: correct mitogen.master.Context -> mitogen.parent.Context. 6 years ago
David Wilson 2c74eac19a issue #291: more Ansible-compatible script invocation
When running any kind of script, rewrite the hashbang like Ansible does,
but subsequently ignore it and explicitly use a fragment of shell from
the ansible_*_interpreter variable to call the interpreter, just like
Ansible does.

This fixes hashbangs containing '/usr/bin/env A=1 bash' on Linux, where
putting that into a hashbang line results in an infinite loop.
6 years ago
David Wilson 17dda781c0 issue #317: ansible: fix log filtering in several cases
* mitogen/ansible_mitogen should only generate ERROR-level logs in
  log_path unless -vvv is enabled.
* Targets were accidentally configured to always have DEBUG set, causing
  many log messages to be sent on the wire even though they would be
  filtered in the master.

Closes #317.
6 years ago
David Wilson b6d6468c92 issue #301: support expandvars() for remote_tmp only.
Vanilla Ansible support expandvars-like expansions widely in a variety
of places. Prefer to whitelist those we need, rather than sprinkling
hellish semantics everywhere.
6 years ago
David Wilson 417e02bd6d ansible: copy Ansible's method for finding the display. 6 years ago
David Wilson e39c602fd3 issue #291: support UNIX hashbang syntax for ansible_*_interpreter.
Closes #291.
6 years ago
David Wilson 15d68b3c32 issue #309: fix environment cleanup regression.
Closes #309.
6 years ago
David Wilson 3f3c9cd001 Merge commit 'refs/pull/303/head' of github.com:dw/mitogen into dmw
issue #303
6 years ago
David Wilson b9c88d344b issue #299: ansible: fix PluginLoader.get() monkey-patch
This prototype is broken for network_cli connections.
6 years ago
napkindrawing 745d72bb1d core: support for "doas" become_method 6 years ago
David Wilson 6dd9e8984d issue #297: fix viewkeys use on 3.x. 6 years ago
David Wilson 124d8023a2 issue #297: don't bother trying to restore old CWD. 6 years ago
David Wilson d8e0c9e12c issue #297: local commands must execute with WorkerProcess environment. 6 years ago
David Wilson 012745efea issue #297: local actions must execute with fixed directory.
Local actions must execute in the the parent directory of the playbook
that defines the action.
6 years ago
David Wilson 5b03e06457 issue #294: ansible: fix mixed vanilla/Mitogen runs.
Don't bother trying to understand what damage PluginLoader has done to
ansible.plugins.* namespace, just ask it for the base class instead.
6 years ago
David Wilson 3b7a1d4f36 ansible: fix 3.x dependency scanning on OS X
On OS X with case-insensitive filenames, resolving
'ansible.module_utils.facts.base.Hardware' finds
'ansible.module_utils.facts.hardware/__init__.py', because
module_finder's procedure is completely wrong for resolving child
modules. Patch over it for now since it otherwise works for Ansible.
6 years ago
David Wilson bd65e50b2f ansible: reduce() does not exist in 3.x. 6 years ago
David Wilson 34751c38b0 ansible: cStringIO interprets unicode as a buffer obj on 2.x 6 years ago
David Wilson 0eb77b5f7c utils: always enable microsecond logging.
It's too useful, and the logs are fairly out of control already, may as
well just capture everything in the first pass.
6 years ago
David Wilson 410016ff47 Initial Python 3.x port work.
* ansible: use unicode_literals everywhere since it only needs to be
  compatible back to 2.6.
* compat/collections.py: delete this entirely and rip out the parts of
  functools that require it.
* Introduce serializable Kwargs dict subclass that translates keys to
  Unicode on instantiation.
* enable_debug_logging() must set _v/_vv globals.
* cStringIO does not exist in 3.x.
* Treat IOLogger and LogForwarder input as latin-1.
* Avoid ResourceWarnings in first stage by explicitly closing fps.
* Fix preamble_size.py syntax errors.
6 years ago
David Wilson 6025412050 issue #272: add a blacklist for noisy target loggers 7 years ago
dw 27ab051289
Merge pull request #282 from dw/issue278
Issue278
7 years ago
David Wilson 04b65020ac issue #278: ansible: support mitogen_ssh_debug_level variable. 7 years ago
David Wilson 7853b74e7f issue #280: put 'dnf' on the always fork list 7 years ago
David Wilson 2fbe1f1b54 Get integration tests running under 2.6.
Closes #270
Closes #273
7 years ago
David Wilson 08538d327b ansible: don't write failed job result after async timeout.
The failed job result is likely to be "interrupted system call", and we
don't want that to overwrite the SIGALRM handler's "the task timed out",
so just discard it.
7 years ago
David Wilson 45b748833d ansible: don't randomly fail due to temp directory cleanup.
Happens about 1 time in 3 when async task times out.
7 years ago
David Wilson 6377f2d69c issue #257: split pool shutdown and join. 7 years ago
David Wilson d33ef1866e ansible: wrap socket calls in io_op()
Breaks under signal stress test.
7 years ago
David Wilson 3994f1b30a ansible: implment async job time limit. 7 years ago
David Wilson e35694acd5 ansible: flake8 fixes. 7 years ago
David Wilson 4bd992e35a issue #186: move module code fetch back to overridden method 7 years ago
David Wilson ae20a689ef issue #186: finally enable detach. 7 years ago
David Wilson caffaa79f7 issue #186: rework async/forked tasks again.
The controller must know the ID of the forked child in order to
propagate dependencies to it, so forking+starting the module run cannot
happen entirely on the target, without some additional mechanism to
wait-and-repropagate the deps as they arrive on the target.

Rework things so that init_child() also handles starting the fork parent,
and returns it along with the context's home directory in a single round
trip.

Now master knows the identity of the fork parent, it can directly create
fork children and call run_module_async() in them. This necessitates 2
roundtrips to start an asynchronous task.

This whole thing sucks and entirely needs simplified, but for now things
almost work, so keeping it.

connection.py:
  * Expect ContextService to return the entire dict return value of
    init_child(). Store the fork_contxt from the return value.

planner.py:
  * Rework Planner to store the invocation as an instance attribute, to
    simplify method calls.
  * Add Planner.get_push_files() and Planner.get_module_deps().
  * Add _propagate_deps() which takes a Planner and ensures the deps it
    describes are sent to a (non forked or forked) context.
  * Move async task logic out of target.py and into invoke() /
    _invoke_*().

process.py:
  * Services no longer need references to each other. planner.py handles
    sending module deps with one extra RPC.

services.py:
  * Return "init_child_result" key instead of simple "home_dir" key.
  * Get rid of dep propagation from ModuleDepService, it lives in
    planner.py now.

target.py:
  * Get rid of async task start logic, lives in planner.py now.
7 years ago
David Wilson 569c12a2d6 ansible: use PushFileService for module deps.
planner.py:
  * Rather than grant FileService access to a file for children, use
    PushFileService to trigger deduplicating send of the file through
    the hierarchy immediately.
  * Send the complete list of Ansible module imports to the target so
    runner.py knows which files and scripts must be loaded via
    PushFileService prior to detaching.

runner.py:
  * Teach NewStyleRunner to use the full module map to block until
    everything is loaded prior to detach().

target.py:
  * Delete old _get_file(), replace get_file() with get_small_file()
    which uses PushFileService instead.

Closes #186
7 years ago
David Wilson 7d4f4b205f ansible: update module preload list. 7 years ago
David Wilson 76beea6554 issue #186: move target._get_file into mitogen.service
For lack of a better place to keep the client function, make it a
classmethod of FileService itself for now.

The old _get_file() is removed in a subsequent commit.
7 years ago
David Wilson 1745c3aff0 issue #186: ansible: detach asynchronous tasks
After Runner.setup() has executed, but before the module executes. This
relies on subsequent commits to ensure all files are preloaded.
7 years ago
David Wilson 088a7e5cff ansible: handle "from timeout import timeout" imports.
It's not simple without executing a module to determine whether the
above refers to a submodule of a package, or an object defined within a
module.

Therefore detect when resolution of a child module yields the same path
as the parent, and ignore the result.
7 years ago
David Wilson daa9cfd0a8 ansible: MITOGEN_DUMP_THREAD_STACKS for mux process too 7 years ago
David Wilson d9087c510b ansible: move FileService into mitogen.service. 7 years ago
David Wilson 9cb3878f3f nsible: remove unused master import 7 years ago
David Wilson fdbd954113 ansible: preload built-in modules in ModuleDepScanner.
For "ansible -m setup" over a 25ms link, avoids 65 roundtrips and
reduces runtime from 5.7s to 4.1s (-28%).

For "ansible -m setup" over a simulated 250 ms link, reduces runtime
from m27.015s to 0m8.254s (-69%).
7 years ago
David Wilson 8d45e609ee ansible: preload always-requested modules.
Avoid 9 roundtrips during setup. In combination with previous change,
reduces 'ansible -m stat' execution over 25ms link from 3.7s to 3.07s.

1,3d0
< _on_get_module('ansible')
< _on_get_module('ansible.module_utils')
< _on_get_module('ansible.module_utils.basic')
69,74d65
< _on_get_module('ansible.module_utils.json_utils')
< _on_get_module('ansible.release')
< _on_get_module('ansible_mitogen')
< _on_get_module('ansible_mitogen.runner')
< _on_get_module('ansible_mitogen.target')
< _on_get_module('mitogen.fork')
7 years ago
David Wilson a578250bfb ansible: remove indirect master.py imports.
Avoids sending 10 modules:

77d76
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'ansible_mitogen.module_finder')
79d77
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'ansible_mitogen.services')
81,84d78
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'mitogen.compat')
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'mitogen.compat.collections')
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'mitogen.compat.functools')
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'mitogen.compat.tokenize')
86,87d79
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'mitogen.master')
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'mitogen.minify')
89,90d80
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'mitogen.select')
< _send_load_module(mitogen.ssh.Stream(u'ssh.localhost'), 'mitogen.service')
7 years ago
David Wilson 3b0addcfb0 service: v2. Closes #213 7 years ago
David Wilson ddf28987a0 master: split Select() into new module to reduce wire size.
service.py currently imports master.py(+parent.py) just to get Select().
7 years ago
David Wilson 1eb5c20f57 ansible: add dummy init.pys so setup.py find_packages() DTRT. 7 years ago
David Wilson bb61745a1a issue #217: pass through non-custom module utils to regular importer.
This may come back to bite later, but in the meantime it avoids shipping
up to 12KiB of junk metadata for every single task invocation.

For detachment (aka. async), we must ensure the target has two types of
preloads completed (modules and module_utils files) before detaching.
7 years ago
David Wilson 30034877a5 issue #217: ansible: working, if extremely inefficient implementation 7 years ago
David Wilson 81b62d9a1a issue #217: ansible: beginnings of ModuleDepService. 7 years ago
David Wilson 0f8190eff6 issue #217: ansible: working module_finder.py 7 years ago
David Wilson 6a17042020 issue #217: ansible: merge duplicate ReplacerPlanner. 7 years ago
David Wilson bd2cc0830c Enable unidirectional routing in Ansible; closes #132. 7 years ago
David Wilson f96c552f87 issue #217: initial module scanner code.
This is sketch code, it's being done separately from mitogen.master.* to
begin with to avoid breaking what's there.
7 years ago
David Wilson 2ad0d0521d ansible: reload /etc/resolv.conf for each task.
The OpenShift installer modifies /etc/resolv.conf then tests the new
resolver configuration, however, there was no mechanism to reload
resolv.conf in our reuseable interpreter.

https://github.com/openshift/openshift-ansible/blob/release-3.9/roles/openshift_web_console/tasks/install.yml#L137

This inserts an explicit call to res_init() for every new style
invocation, with an approximate cost of ~1usec on Linux since glibc
verifies resolv.conf has changed before reloading it.

There is little to be done for users of the thread-safe resolver APIs,
their state is hidden from us. If bugs like that manifest, whack-a-mole
style 'del sys.modules[thatmod]' patches may suffice.
7 years ago
David Wilson 49eae23f92 issue #218: ansibe: use Secret and Blob types. 7 years ago
David Wilson 69f58875c8 tests: more ansible fixes from recent brakage. 7 years ago
David Wilson ad1f624750 ansible: document and rearrange Runner params.
Move emulate_tty to where it's used.
7 years ago
David Wilson 1186b927f9 ansible: remove seemingly unused raw_params
Traced git log all the way back to beginning of time, and checked
Ansible versions starting Jan 2016. Zero clue where this came from, but
the convention suggests it came from Ansible at some point.
7 years ago
David Wilson f737ff5276 ansible: stop passing through remote_tmp variable
Ansiballz does not use remote_tmp so neither should we, per #239.
7 years ago
David Wilson f9e1905ec6 issue #199: ansible: stop writing temp files for new style modules
While adding support for non-new style module types, NewStyleRunner
began writing modules to a temporary file, and sys.argv was patched to
actually include the script filename. The argv change was never required
to fix any particular bug, and a search of the standard modules reveals
no argv users. Update argv[0] to be '', like an interactive interpreter
would have.

While fixing #210, new style runner began setting __file__ to the
temporary file path in order to allow apt.py to discover the Ansiballz
temporary directory. 5 out of 1,516 standard modules follow this
pattern, but in each case, none actually attempt to access __file__,
they just call dirname on it. Therefore do not write the contents of
file, simply set it to the path as it would exist, within a real
temporary directory.

Finally move temporary directory creation out of runner and into target.
Now a single directory exists for the duration of a run, and is emptied
by runner.py as necessary after each task invocation.

This could be further extended to stop rewriting non-new-style modules
in a with_items loop, but that's another step.

Finally the last bullet point in the documentation almost isn't a lie
again.
7 years ago
David Wilson 1bc08323bf ansible: more compatible module script naming. 7 years ago
David Wilson 8bd34e1e28 ansible: gracefully report connection timeouts as StreamError. 7 years ago
David Wilson 4d1c6d2101 issue #127: ssh: reasonable solution to host key checking.
Ideally it would be possible to specify a callback function, but this is
not possible for proxied connections. So simply provide the 3 most
useful modes, defaulting to the most secure.

Closes #127. Closes #134.
7 years ago
David Wilson 9cc8b21a07 ansible: missing mitogen_su module. 7 years ago
David Wilson 78c401ba4d ansible: support su become method. 7 years ago
David Wilson 3203846708 issue #239: ansible: ignore remote_tmp in new style runner. 7 years ago
David Wilson 94e048a2e5 ansible: ensure FileService uses exact CHUNK_SIZE multiple
9.8% throughput increase with sudo.
7 years ago
David Wilson dafe12b315 ansible: fix AnsibleUnicode crash when processing "~username". 7 years ago
David Wilson e1a3cea2f9 ansible: FileService: don't send empty last chunk 7 years ago
David Wilson 2a56c672ca ansible: FileService docstring updates. 7 years ago
David Wilson f5d22a3ca1 core: support deleting handlers, make Receiver.close() unregister 7 years ago
David Wilson 69e5902e61 issue #212: support explicit acknowledgements in FileService. 7 years ago
David Wilson b0309b539c ansible: disable interpreter recycling for connections.
Must explicitly specify enable_lru=True in ContextService.get() to
trigger recycling.
7 years ago
David Wilson 65e6a44fe7 docs: add links. 7 years ago
David Wilson 90f7b4baea ansible: stub plugin documentation. 7 years ago
David Wilson c85a5b6446 ansible: make call timing more readable 7 years ago
David Wilson bba2a42e44 ansible: add mitogen_sudo method, split out connection subclasses.
Slowly moving towards real implementations in those files.
7 years ago
David Wilson 0a9126c510 ansible: connection plugins missing from previous commit. 7 years ago
David Wilson 7c5bbc5168 setns: support changing user.
To match existing third party plugin.
7 years ago
David Wilson 003f30b5a9 issue #226: test fixes. 7 years ago
David Wilson 6edb3f165d ansible: avoid a race during shutdown. 7 years ago
David Wilson 780b63520f issue #226: don't attempt to fchmod() a pathname 7 years ago
David Wilson 219a202a82 issue #226: ansible: file transfer improvements
* put_data() supports setting mode and times.
* put_file() refuses to copy non-regular files (sockets, FIFOs).
* put_file() saves one RTT for <32KiB files by using put_data() and
  embedding file content in argument list.
* FileService returns dict with size/mode/owner/group/mtime/atime.
* FileService refuses to copy non-regular files.
* transfer_file() preserves file mode.
* transfer_file() preserves atime/mtime.
* transfer_file() optionally preserves ownership.
* transfer_file() optionally calls fsync().
* transfer_file() uses unique temporary file name to avoid conflicting
  with parallel transfers.
* transfer_file() ensures temporary file is deleted on any error.
* write_path() writes to a temporary file and deletes it on failure.
* write_path() uses unique temporary file name to avoid conflicting
  with parallel transfers.
* write_path() supports setting symbolic owner/group.
* write_path() optionally calls fsync().
* write_path() supports setting symbolic mode/mtime/atime.

Closes #226, #227, #229
7 years ago