Commit Graph

298 Commits (42d3f96d146f068a4d90a30159f223857606963e)

Author SHA1 Message Date
David Wilson 43d9815f6d ansible: use CallChain everywhere.
This replaces the 'dump to logger' behaviour of pipelined calls from
before with a call chain that returns any exception on next synchronized
call.
6 years ago
David Wilson b254eb3399 ansible: fix non-action connection instantiation.
e.g. by synchronize module.
6 years ago
David Wilson 705d77a9be ansible: remove a bunch more aliasing from connection.py. 6 years ago
David Wilson 66142e7d75 ansible: fork isolated tasks from correct parent.
Closes #355.
6 years ago
David Wilson da8c6b45b0 ansible: remove task_vars aliasing from connection.py.
Crazy spam creep.
6 years ago
David Wilson 32751cd356 master: allow batching context switches for forward_modules()
-7 switches per task.
6 years ago
David Wilson 86942b6bf9 ansible: add explanatory exception
If disconnection occurs during a Connection.call(), return
AnsibleConnectionFailure.
6 years ago
David Wilson 3c6b72b452 ansible: gracefully return (and explain) ChannelError in ContextService.
When Ansible abnormally shuts down, the broker begins
force-disconnecting every context, including those for which connection
is currently in-progress.

When that happens, .call(init_child) throws ChannelError, and that needs
returned back to the worker, assuming the worker still even exists.

This solution is incomplete: with sick nodes, it's also possible the
worker died naturally, and so the worker should perhaps respond by
retrying the connection.

Previously, the unhandled ChannelError would spam the console when e.g.
fork() began returning EAGAIN.
6 years ago
David Wilson e647adc62e ansible: copy GIL change from linear2 branch.
Reduces runtime by 25% given 100 25ms SSH targets:

    ANSIBLE_STRATEGY=mitogen \
    MITOGEN_POOL_SIZE=100 \
    /usr/bin/time -l ansible k3-x100 -m shell -a hostname

Before:
           39.56 real        35.29 user        17.24 sys
      59600896  maximum resident set size
       1784252  page reclaims
          9016  messages sent
         10382  messages received
         18774  voluntary context switches
        770070  involuntary context switches

After:
           29.79 real        22.10 user        11.77 sys
      59281408  maximum resident set size
       1725268  page reclaims
          8582  messages sent
          9959  messages received
         14582  voluntary context switches
         75280  involuntary context switches
6 years ago
David Wilson 2647f73501 ansible: bump UNIX listener default backlog, and set it to match forks.
The connection multiplexer can expect to not be scheduled at least until
every $forks worker processes has attempted a connection, so the backlog
must be able to hold every worker.
6 years ago
David Wilson 8ab11f415f ansible: better support for diagnosing hangs
* Always enable the faulthandler module in the top-level process if it
  is available.
* Make MITOGEN_DUMP_THREAD_STACKS interval configurable, to better
  handle larger runs.
* Add docs subsection on diagnosing hangs.

Conflicts:
	ansible_mitogen/process.py
6 years ago
David Wilson 9792b8b54f ansible: use template-expanded delegate_to hostname in one more location. 6 years ago
David Wilson 90c2ed03d0 ansible: fix synchronize module
Broken by recent connection delegation fixes.
6 years ago
David Wilson 7458dfae85 ansible: avoid roundtrip for small file transfers.
Calls to connect.put_file() where the file is sufficiently small enough
to fit in a single RPC proceed without waiting for an RPC response. If
the write fails the target context will log an exception, and any
subsequent step depending on the written file will fail.

I verified every built-in action plugin for file transfer calls, and
they all depend on the transferred file in the following step, so this
should be safe.

Reduces template/copy actions to 2-RTT, loop-20-templates.yml runtime
reduced from 30 seconds to 10 seconds over a 250ms link compared to
v0.2.2, and from 123 seconds compared to vanilla with pipelining
enabled.
6 years ago
David Wilson 084c0ac065 ansible: avoid roundtrip in copy action due to fixup_perms2().
On top of existing temporary files work, this reduces the number of
roundtrips required for "copy" and "template" actions from 6 to 3.
6 years ago
David Wilson e18396d54d ansible: enable profiling by default!
Thankfully this never made it into a release
6 years ago
David Wilson bce4f59138 issue #345: disable IdentitiesOnly by default. 6 years ago
David Wilson 2e3d04bbb8 issue #342: forward _create_control_path() to SSH plugin.
network_cli connection type loads the "ssh" (mitogen_ssh) plugin and
expects a private method to exist.
6 years ago
David Wilson ac9b84d237 issue #321: 2.4+ compatibility fixes, disable test on Vanilla. 6 years ago
David Wilson 4afe2fdc55 issue #321: fix probable threading issue. 6 years ago
David Wilson f24f02ba06 issue #321: take remote_tmp and system_tmpdirs into account.
Can't simply ignore these settings as some users may have weird noexec
filesystems.
6 years ago
David Wilson a2686b1a2c issue #321: simplify temp directory handling. 6 years ago
David Wilson 3d588323ff issue #340: use expanded delegate_to hostname, not template.
PlayContext.delegate_to is the unexpanded template, Ansible doesn't keep
a copy of it around anywhere convenient. We either need to re-expand it
or take the expanded version that was stored on the Task, which is what
is done here.
6 years ago
David Wilson a6995a5288 issue #338: refactor env handling into class and fix tests. 6 years ago
David Wilson 916e46621b issue #340: add connection delegation tests. 6 years ago
David Wilson 6dcd5f8998 issue #340: split up Connection._connect()
The logic was getting too busy.
6 years ago
David Wilson a1e653978b issue #340: connection delegation used wrong variable name.
When inventory name did not match remote_addr, it would attempt to SSH
to the inventory name.
6 years ago
David Wilson 370b98f960 ansible: tidy up connection.py.
- more docstrings.
- _wrap_or_none -> optional_secret()
6 years ago
David Wilson 9e572a7939 ansible: fix duplicate MuxProcess socket write.
The while: loop was necessary due to some cutpaste further on down the
file.
6 years ago
David Wilson b521f215fd ansible: handle >2.6 magic exceptions + sys.excepthook damage
Closes #332.
6 years ago
David Wilson 053c594d65 ansible: prevent logs spamming user console on exit.
Closes #331.
6 years ago
David Wilson 1f21a30e7f issue #251: ansible: watch for delegate_to during connection delegation.
This needs more work -- pretty certain that python_path and suchlike are
coming from the wrong place. Possibly we need another config_from_..()
specialized for delegate_to.
6 years ago
David Wilson 6c4b01642c ansible: don't crash when adhoc tries to run a missing module.
ansible-playbook prints a separate error during parsing stage, adhoc
performs no such check.
6 years ago
David Wilson 34a9f67115 issue #339: whoops, actually wire up new connection method. 6 years ago
David Wilson 1473f49505 ansible: emulate /etc/environment reloading behaviour of vanilla.
This change is relatively incomplete -- ideally we could snapshot
os.environ and /etc/environment at startup and respect key deletions
too, but that's a lot more work. Wait for a bug report instead.

Closes #338.
6 years ago
David Wilson 4077182fb2 ansible: plugins were missing absolute_import. 6 years ago
David Wilson 5c573f7fcb ansible: insert short sleep when MITOGEN_PROFILING active.
Hacky, but works fine.
6 years ago
David Wilson d62e6e2a7f ansible: serialize calls to ModuleDepService.
Concurrent calls to ModuleDepService would cause significant wasted
work, as potentially all pool threads run the same uncached module dep
scan.

Without:
         3243581 function calls (3233009 primitive calls) in 4770.672 seconds

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     2523    0.011    0.000   39.849    0.016 services.py:409(scan)

With:
         2801561 function calls (2800042 primitive calls) in 5166.843 seconds

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     2506    0.009    0.000    1.967    0.001 services.py:411(scan)

Ignore timing variance due to problems with the test job.
6 years ago
David Wilson 3e0de9790c issue #324: fix Python 3 fallout for custom module_utils.
Also enable at last one of its tests.
6 years ago
David Wilson f4a66194e4 ansible: Py3.x fixes for Ansible PR #41749 workaround. 6 years ago
David Wilson 6c03b83748 issue #291: don't attempt mitogen import until sys.path modified.
Given an extracted download of mitogen-2.2.tar.gz, with strategy_plugins
pointing into it, if an old version of the package was pip-installed,
then the old pip-installed package would be imported and override
whatever came from the tarball.

Instead, modify sys.path before attempting any import. This still isn't
perfect, but it's better.
6 years ago
David Wilson f8b3441431 ansible: work around Ansible PR #41749 6 years ago
David Wilson bfe9f81d0b ansible: fix RPC time logging.
Rendering call arguemtns was broken for non-positional arguments.
6 years ago
David Wilson 71b4294888 issue #291: 2.6 compat fix. 6 years ago
David Wilson b44b823c4a ansible: make _remote_expand_user() pay attention to sudoable=.. 6 years ago
David Wilson a8e4dcc98d issue #301: correct remote_tmp evaluation context.
Vanilla Ansible expands remote_tmp variables in the context of the login
account, not any become_user account.
6 years ago
David Wilson de0d526487 issue #291: restore behaviour of invoking binaries via /bin/sh
This ensures failed task output matches vanilla Ansible exactly (as it
did before starting #291).
6 years ago
David Wilson c5ea7c45a1 comments/docs: correct mitogen.master.Context -> mitogen.parent.Context. 6 years ago
David Wilson 2c74eac19a issue #291: more Ansible-compatible script invocation
When running any kind of script, rewrite the hashbang like Ansible does,
but subsequently ignore it and explicitly use a fragment of shell from
the ansible_*_interpreter variable to call the interpreter, just like
Ansible does.

This fixes hashbangs containing '/usr/bin/env A=1 bash' on Linux, where
putting that into a hashbang line results in an infinite loop.
6 years ago
David Wilson 17dda781c0 issue #317: ansible: fix log filtering in several cases
* mitogen/ansible_mitogen should only generate ERROR-level logs in
  log_path unless -vvv is enabled.
* Targets were accidentally configured to always have DEBUG set, causing
  many log messages to be sent on the wire even though they would be
  filtered in the master.

Closes #317.
6 years ago