Commit Graph

481 Commits (6b180a4091f9155d8a653d48479edc5fd5f6e633)

Author SHA1 Message Date
David Wilson f4cf67f0bd issue #615: remove meaningless test
It has been dead code since at least 2015
5 years ago
David Wilson e02be89879 issue #625: ignore SIGINT within MuxProcess
Without this, MuxProcess will start dying too early, before Ansible /
TaskQueueManager.cleanup() has a chance to wait on worker processes.
That would allow WorkerProcess to see ECONNREFUSED from the MuxProcess
socket much more easily.
5 years ago
David Wilson 67759371f9 issue #615: ensure 4GB max_message_size is configured for task workers.
This 4GB limit was already set for MuxProcess and inherited by all
descendents including the context running on the target host, but it was
not applied to the WorkerProcess router.

That explains why the error from the ticket is being raised by the
router within the WorkerProcess rather than the router on the original
target.
5 years ago
David Wilson 151b490890 issue #615: fetch_file() might be called with AnsibleUnicode. 5 years ago
David Wilson 03d2bc6c59 issue #615: redirect 'fetch' action to 'mitogen_fetch'. 5 years ago
David Wilson 52c8ed7715 issue #615: extricate slurp brainwrong from mitogen_fetch 5 years ago
David Wilson 069285a588 issue #615: ansible: import Ansible fetch.py action plug-in
From ansible/ansible#9773a1f2896a914d237cb9926e3b5cdc0f004d1a
5 years ago
David Wilson 8dfb3966df issue #558, #582: preserve remote tmpdir if caller did not supply one
The undocumented 'tmp' parameter controls whether _execute_module()
would delete anything on 2.3, so mimic that. This means
_execute_remove_stat() calls will not blow away the temp directory,
which broke the unarchive plugin.
5 years ago
David Wilson 3b63da670f Fix up another handful of LGTM errors. 5 years ago
David Wilson f78a5f08c6 issue #605: ansible: share a sem_t instead of a pthread_mutex_t
The previous version quite reliably causes worker deadlocks within 10
minutes running:

    # 100 times:
    - import_playbook: integration/async/runner_one_job.yml
    # 100 times:
    - import_playbook: integration/module_utils/adjacent_to_playbook.yml

via .ci/soak/mitogen.sh with PLAYBOOK= set to the above playbook.

Attaching to the worker with gdb reveals it in an instruction
immediately following a futex() call, which likely returned EINTR due to
attaching gdb. Examining the pthread_mutex_t state reveals it to be
completely unlocked.

pthread_mutex_t on Linux should have zero trouble living in shmem, so
it's not clear how this deadlock is happening. Meanwhile POSIX
semaphores are explicitly designed for cross-process use and have a
completely different internal implementation, so try those instead. 1
hour of soaking reveals no deadlock.

This is about avoiding managing a lockable temporary file on disk to
contain our counter, and somehow communicating a reference to it into
subprocesses (despite the subprocess module closing inherited fds, etc),
somehow deleting it reliably at exit, and somehow avoiding concurrent
Ansible runs stepping on the same file. For now ctypes is still less
pain.

A final possibility would be to abandon a shared counter and instead
pick a CPU based on the hash of e.g. the new child's process ID. That
would likely balance equally well, and might be worth exploring when
making this code work on BSD.
5 years ago
David Wilson 5af6c9b26f issue #615: use FileService for target->controll file transfers 5 years ago
David Wilson 6f12980611 [linear2] merge fallout: re-enable _send_module_forwards(). 5 years ago
David Wilson 5298e87548 Split out and make readable more log messages across both packages 5 years ago
David Wilson 0f23a90d50 ansible: log affinity assignments 5 years ago
David Wilson 4f051a38a7 ansible: improve docstring 5 years ago
David Wilson 5811909c8d [linear2] simplify _listener_for_name() 5 years ago
David Wilson c68dbdd569 ansible: stop relying on SIGTERM to shut down service pool
It's no longer necessary, since connection attempts are no longer truly
blocking. When CTRL+C is hit in the top-level process, broker will begin
shutdown, which will cancel all pending connection attempts, causing
pool threads to wake. The pool can't block during shutdown anymore.
5 years ago
David Wilson f4ca926b21 ansible: cleanup various docstrings 5 years ago
David Wilson edde251d58 issue #549: ansible: reduce risk by capping RLIM_INFINITY 5 years ago
David Wilson d408caccf5 issue #573: guard against a forked top-level Ansible process
See comment.
5 years ago
David Wilson 3ceac2c9ed [linear2] simplify ClassicWorkerModel and fix repeat initialization
"self.initialized = False" slipped in a few days ago, on second thoughts
that flag is not needed at all, by simply rearranging ClassicWorkerModel
to have a regular constructor.

This hierarchy is still squishy, it needs more love. Remaining
MuxProcess class attributes should eliminated.
5 years ago
David Wilson 395b03a77d issue #549: fix setrlimit() crash and hard-wire OS X default
OS X advertised unlimited, but really it means kern.maxfilesperproc.
5 years ago
David Wilson 33bceb6eb4 issue #602: recover task_vars for synchronize and meta: reset_connection 5 years ago
David Wilson 6b4bcf4fe0 ansible: remove cutpasted docstring 5 years ago
David Wilson 619f4dee07 [linear2] merge fallout: restore optimization from #491 / 7b129e857 5 years ago
David Wilson e4321f81a0 issue #600: /etc/environment may be non-ASCII in an unknown encoding 5 years ago
David Wilson 75d179e4b9 remove unused imports flagged by lgtm 5 years ago
David Wilson c80fddd487 [linear2]: merge fallout flaggged by LGTM 5 years ago
David Wilson eeb7150f24 issue #549: increase open file limit automatically if possible
While catching every possible case where "open file limit exceeded" is
not possible, we can at least increase the soft limit to the available
hard limit without any user effort.

Do this in Ansible top-level process, even though we probably only need
it in the MuxProcess. It seems there is no reason this could hurt
5 years ago
David Wilson acab26d796 ansible: improve process.py docs 5 years ago
David Wilson 4dfbe82e76 tests: hide ugly error during Ansible tests 5 years ago
David Wilson 108015aa22 ansible: gracefully handle failure to connect to MuxProcess
It's possible to hit an ugly exception during early CTRL+C
5 years ago
David Wilson bf1f3682aa ansible: pin per-CPU muxes to their corresponding CPU
This slightly breaks the old scheme, in that CPU 1 may now end up with a
mux and the top-level process pinned to it.
5 years ago
David Wilson dc9f4e89e6 ansible: reap mux processes on shut down
Previously we exitted without calling waitpid(), which meant the
top-level process struct rusage did not reflect the resource usage
consumed by the multiplexer processes.

Existing benchmarks are made using perf so this never created a problem,
but it could be confusing to others using the "time" command, and also
allows logging the final exit status of the process.
5 years ago
David Wilson 136dee1fb4 [linear2] more merge fallout, fix Connection._mitogen_reset(mode=) 5 years ago
David Wilson a9755d4ad0 [linear2] update mitogen_get_stack for new _build_stack() return value 5 years ago
David Wilson 1fca0b7a94 [linear2] fix MuxProcess test fixture and some merge fallout 5 years ago
David Wilson 0f63ca4c68 Make setting affinity optional. 5 years ago
David Wilson 9035884c77 ansible: abstract worker process model.
Move all details of broker/router setup out of connection.py, instead
deferring it to a WorkerModel class exported by process.py via
get_worker_model(). The running strategy can override the configured
worker model via _get_worker_model().

ClassicWorkerModel is installed by default, which implements the
extension's existing process model.

Add optional support for the third party setproctitle module, so
children have pretty names in ps output.

Add optional support for per-CPU multiplexers to classic runs.
5 years ago
David Wilson 402dba4197 module_finder: pass raw file to compile()
Newer Ansibles have e.g. UTF-8 present in apt.py.
5 years ago
David Wilson 1aceacf89e [stream-refactor] replace old detach_popen() reference 5 years ago
David Wilson 300f8b2ff9 ansible: fixturize creation of MuxProcess
This relies on the previous commit resetting global variables.

Update clean_shutdown() to handle duplicate calls, due to tests
repeatedly installing it.
5 years ago
David Wilson 26b6333787 [stream-refactor] fix unix.Listener construction 5 years ago
Jordan Webb 1a02a86331
Add buildah transport 5 years ago
David Wilson 7ae926b325 ansible: prevent tempfile.mkstemp() leaks.
This avoids a leak present in Ansible 2.7.0..current HEAD, and all
similar leaks.

See ansible/ansible#57327.
6 years ago
David Wilson 3620fce071 issue #593: expose configurables for SSH keepalive and increase the default 6 years ago
David Wilson 0b7fd3f290 issue #591: ansible: restore CWD prior to AnsibleModule initialization. 6 years ago
David Wilson 4f23f0bec1 issue #590: update comment to indicate the hack is permanent 6 years ago
David Wilson 1a92995a24 issue #590: include nasty workaround for sys.modules junk 6 years ago
David Wilson 92b4724010 issue #587: consistent become_exe() behaviour for older Ansibles. 6 years ago