Commit Graph

2863 Commits (67759371f9d2b85c922a973d3e31166cd207a5d3)
 

Author SHA1 Message Date
David Wilson 67759371f9 issue #615: ensure 4GB max_message_size is configured for task workers.
This 4GB limit was already set for MuxProcess and inherited by all
descendents including the context running on the target host, but it was
not applied to the WorkerProcess router.

That explains why the error from the ticket is being raised by the
router within the WorkerProcess rather than the router on the original
target.
5 years ago
David Wilson 3c8c11b360 issue #615: update Changelog. 5 years ago
David Wilson 3f5ff17c8c issue #615: route a dead message to recipients when no reply is expected 5 years ago
David Wilson 151b490890 issue #615: fetch_file() might be called with AnsibleUnicode. 5 years ago
David Wilson 03d2bc6c59 issue #615: redirect 'fetch' action to 'mitogen_fetch'. 5 years ago
David Wilson 52c8ed7715 issue #615: extricate slurp brainwrong from mitogen_fetch 5 years ago
David Wilson 069285a588 issue #615: ansible: import Ansible fetch.py action plug-in
From ansible/ansible#9773a1f2896a914d237cb9926e3b5cdc0f004d1a
5 years ago
David Wilson 98832f3b64 issue #533: include object identity of Stream in repr()
At least one of the causes of the #533 error appears to be that streams
with the same name exist
5 years ago
David Wilson db8f0db5e7 docs: lots more changelog 5 years ago
David Wilson 341c453eaa issue #595: add buildah to docs and changelog. 5 years ago
David Wilson e0d9b8d1e1 docs: a few more internals.rst additions 5 years ago
David Wilson db37000dd5 ci: update to Ansible 2.8.3 5 years ago
David Wilson bc275b2526 tests: another random string changed in 2.8.3 5 years ago
David Wilson 7e0c2fd1af tests: fix sudo_flags_failure for Ansible 2.8.3 5 years ago
David Wilson fa8755085a ci: fix procps command line format warning 5 years ago
David Wilson 866438aec6 Whoops, merge together lgtm.yml and .lgtm.yml
Also add ansible_mitogen/compat.
5 years ago
David Wilson d9cc577a6c issue #440: log Python version during bootstrap. 5 years ago
David Wilson 49796e0c39 docs: update changelog 5 years ago
David Wilson 39f5ecb3c8 Merge remote-tracking branch 'origin/dmw'
* origin/dmw:
  issue #558: disable test on OSX to cope with boundless mediocrity
  issue #558, #582: preserve remote tmpdir if caller did not supply one
5 years ago
David Wilson 206a8d4aeb issue #558: disable test on OSX to cope with boundless mediocrity 5 years ago
David Wilson 8dfb3966df issue #558, #582: preserve remote tmpdir if caller did not supply one
The undocumented 'tmp' parameter controls whether _execute_module()
would delete anything on 2.3, so mimic that. This means
_execute_remove_stat() calls will not blow away the temp directory,
which broke the unarchive plugin.
5 years ago
David Wilson 41d8a8a258 Merge remote-tracking branch 'origin/dmw'
* origin/dmw:
  issue #613: must await 'exit' and 'disconnect' in wait=False test
  Import LGTM config to disable some stuff
  Fix up another handful of LGTM errors.
  tests: work around AnsibleModule.run_command() race.
  docs: mention another __main__ safeguard
  docs: tweaks
  formatting error
  docs: make Sphinx install soft fail on Python 2.
  issue #598: allow disabling preempt in terraform
  issue #598: update Changelog.
5 years ago
David Wilson 0c1d882547 issue #613: must await 'exit' and 'disconnect' in wait=False test 5 years ago
David Wilson 6af337c3d3 Import LGTM config to disable some stuff
- ignore mitogen/compat/**
- switch off unreachable code check
- switch off try/finally vs. with
- switch off mixed import/import-from
5 years ago
David Wilson 3b63da670f Fix up another handful of LGTM errors. 5 years ago
David Wilson 4b9b1ca24d tests: work around AnsibleModule.run_command() race.
See https://github.com/ansible/ansible/issues/51393
5 years ago
David Wilson e12f391106 docs: mention another __main__ safeguard 5 years ago
David Wilson 1d41adb346 docs: tweaks 5 years ago
David Wilson 9cb187c2c4 formatting error 5 years ago
David Wilson 9b9fe57ea8 docs: make Sphinx install soft fail on Python 2. 5 years ago
David Wilson 9b45872246 issue #598: allow disabling preempt in terraform 5 years ago
David Wilson c89f6cbab6 issue #598: update Changelog. 5 years ago
David Wilson 74834c845f Merge remote-tracking branch 'origin/dmw'
* origin/dmw:
  issue #605: update Changelog.
  issue #605: ansible: share a sem_t instead of a pthread_mutex_t
  issue #613: add tests for all the weird shutdown methods
  Add mitogen.core.now() and use it everywhere; closes #614.
  docs: move decorator docs into core.py and use autodecorator
  preamble_size: make it work on Python 3.
  docs: upgrade Sphinx to 2.1.2, require Python 3 to build docs.
  docs: fix Sphinx warnings, add LogHandler, more docstrings
  docs: tidy up some Changelog text
5 years ago
David Wilson 240dc84d94 issue #605: update Changelog. 5 years ago
David Wilson f78a5f08c6 issue #605: ansible: share a sem_t instead of a pthread_mutex_t
The previous version quite reliably causes worker deadlocks within 10
minutes running:

    # 100 times:
    - import_playbook: integration/async/runner_one_job.yml
    # 100 times:
    - import_playbook: integration/module_utils/adjacent_to_playbook.yml

via .ci/soak/mitogen.sh with PLAYBOOK= set to the above playbook.

Attaching to the worker with gdb reveals it in an instruction
immediately following a futex() call, which likely returned EINTR due to
attaching gdb. Examining the pthread_mutex_t state reveals it to be
completely unlocked.

pthread_mutex_t on Linux should have zero trouble living in shmem, so
it's not clear how this deadlock is happening. Meanwhile POSIX
semaphores are explicitly designed for cross-process use and have a
completely different internal implementation, so try those instead. 1
hour of soaking reveals no deadlock.

This is about avoiding managing a lockable temporary file on disk to
contain our counter, and somehow communicating a reference to it into
subprocesses (despite the subprocess module closing inherited fds, etc),
somehow deleting it reliably at exit, and somehow avoiding concurrent
Ansible runs stepping on the same file. For now ctypes is still less
pain.

A final possibility would be to abandon a shared counter and instead
pick a CPU based on the hash of e.g. the new child's process ID. That
would likely balance equally well, and might be worth exploring when
making this code work on BSD.
5 years ago
David Wilson 4fa760cd21 issue #613: add tests for all the weird shutdown methods 5 years ago
David Wilson 57012e0f72 Add mitogen.core.now() and use it everywhere; closes #614. 5 years ago
David Wilson 379dca90b9 docs: move decorator docs into core.py and use autodecorator 5 years ago
David Wilson 284dda53e8 preamble_size: make it work on Python 3. 5 years ago
David Wilson a91a8bf19c docs: upgrade Sphinx to 2.1.2, require Python 3 to build docs. 5 years ago
David Wilson 93e8d5dfcc docs: fix Sphinx warnings, add LogHandler, more docstrings 5 years ago
David Wilson 1d943388b7 docs: tidy up some Changelog text 5 years ago
David Wilson 0b9c96482b Merge remote-tracking branch 'origin/dmw'
* origin/dmw:
  issue #615: fix up FileService tests for new logic
  issue #615: another Py3x fix.
  issue #615: Py3x fix.
  issue #615: update Changelog.
  issue #615: use FileService for target->controll file transfers
5 years ago
David Wilson 7d4ae6cec4 issue #615: fix up FileService tests for new logic
Can't perform authorization test in the same process so easily any more
since it checks is_privileged
5 years ago
David Wilson 588859423a issue #615: another Py3x fix. 5 years ago
David Wilson 9e1e1ba015 issue #615: Py3x fix. 5 years ago
David Wilson c464bb5346 issue #615: update Changelog. 5 years ago
David Wilson 5af6c9b26f issue #615: use FileService for target->controll file transfers 5 years ago
David Wilson ceddc5cee2 Merge remote-tracking branch 'origin/dmw'
* origin/dmw:
  issue #482: another Py3 fix
  ci: try removing exclude: to make Azure jobs work again
  compat: fix Py2.4 SyntaxError
  issue #482: remove 'ssh' from checked processes
  ci: Py3 fix
  issue #279: add one more test for max_message_size
  issue #482: ci: add stray process checks to all jobs
  tests: fix format string error
  core: MitogenProtocol.is_privileged was not set in children
  issue #482: tests: fail DockerMixin tests if stray processes exist
  docs: update Changelog.
  issue #586: update Changelog.
  docs: update Changelog.
  [security] core: undirectional routing wasn't respected in some cases
  docs: tidy up Select.all()
  issue #612: update Changelog.
5 years ago
David Wilson 8bac1cf368 issue #482: another Py3 fix 5 years ago