archive
/
mitogen
mirror of https://github.com/mitogen-hq/mitogen.git
1
0
Fork 0
Code Releases Activity

Merge remote-tracking branch 'origin/v028'

Browse Source 
* origin/v028:
  Bump version for release.
  docs: update Changelog for 0.2.8.
  issue #627: add test and tweak Reaper behaviour.
  docs: lots more changelog concision
  docs: changelog concision
  docs: more changelog tweaks
  docs: reorder chapters
  docs: versionless <title>
  docs: update supported Ansible version, mention unsupported features
  docs: changelog fixes/tweaks
  issue #590: update Changelog.
new-serialization
David Wilson 5 years ago
parent a3f4eb69bb c82112a34b
commit 2b05f2209a
7 changed files with 180 additions and 103 deletions
Show Stats Download Patch File Download Diff File

8
docs/ansible_detailed.rst
Unescape Escape View File

@ -140,7 +140,7 @@ Testimonials
Noteworthy Differences Noteworthy Differences
---------------------- ----------------------
* Ansible 2.3-2.7 are supported along with Python 2.6, 2.7, 3.6 and 3.7. Verify * Ansible 2.3-2.8 are supported along with Python 2.6, 2.7, 3.6 and 3.7. Verify
your installation is running one of these versions by checking ``ansible your installation is running one of these versions by checking ``ansible
--version`` output. --version`` output.
@ -164,6 +164,12 @@ Noteworthy Differences
- initech_app - initech_app
- y2k_fix - y2k_fix
* Ansible 2.8 `interpreter discovery
<https://docs.ansible.com/ansible/latest/reference_appendices/interpreter_discovery.html>`_
and `become plugins
<https://docs.ansible.com/ansible/latest/plugins/become.html>`_ are not yet
supported.
* The ``doas``, ``su`` and ``sudo`` become methods are available. File bugs to * The ``doas``, ``su`` and ``sudo`` become methods are available. File bugs to
register interest in more. register interest in more.

203
docs/changelog.rst
Unescape Escape View File

@ -15,61 +15,65 @@ Release Notes
</style> </style>
v0.2.8 (unreleased) v0.2.9 (unreleased)
------------------- -------------------
To avail of fixes in an unreleased version, please download a ZIP file To avail of fixes in an unreleased version, please download a ZIP file
`directly from GitHub <https://github.com/dw/mitogen/>`_. `directly from GitHub <https://github.com/dw/mitogen/>`_.
*(no changes)*
v0.2.8 (2019-08-18)
-------------------
This release includes Ansible 2.8 and SELinux support, fixes for two deadlocks,
and major internal design overhauls in preparation for future functionality.
Enhancements Enhancements
~~~~~~~~~~~~ ~~~~~~~~~~~~
* :gh:issue:`556`, * :gh:issue:`556`,
:gh:issue:`587`: Ansible 2.8 is partially :gh:issue:`587`: Ansible 2.8 is supported. `Become plugins
supported. `Become plugins
<https://docs.ansible.com/ansible/latest/plugins/become.html>`_ and <https://docs.ansible.com/ansible/latest/plugins/become.html>`_ and
`interpreter discovery `interpreter discovery
<https://docs.ansible.com/ansible/latest/reference_appendices/interpreter_discovery.html>`_ <https://docs.ansible.com/ansible/latest/reference_appendices/interpreter_discovery.html>`_
are not yet handled. are not yet handled.
* :gh:issue:`419`, :gh:issue:`470`, file descriptor usage during large runs is * :gh:issue:`419`, :gh:issue:`470`, file descriptor usage is approximately
halved, as it is no longer necessary to manage read and write sides halved, as it is no longer necessary to separately manage read and write
distinctly in order to work around a design problem. sides to work around a design problem.
* :gh:issue:`419`: almost all connection setup happens on one thread, reducing * :gh:issue:`419`: setup for all connections happens almost entirely on one
contention and context switching early in a run. thread, reducing contention and context switching early in a run.
* :gh:issue:`419`: Connection setup is better pipelined, eliminating some * :gh:issue:`419`: Connection setup is better pipelined, eliminating some
network round-trips. Most infrastructure is in place to support future network round-trips. Most infrastructure is in place to support future
removal of the final round-trips between a target fully booting and receiving removal of the final round-trips between a target booting and receiving
function calls. function calls.
* :gh:pull:`595`: the :meth:`~mitogen.parent.Router.buildah` connection method * :gh:pull:`595`: the :meth:`~mitogen.parent.Router.buildah` connection method
is available to manipulate `Buildah <https://buildah.io/>`_ containers, and is available to manipulate `Buildah <https://buildah.io/>`_ containers, and
is exposed to Ansible as the :ans:conn:`buildah`. is exposed to Ansible as the :ans:conn:`buildah`.
* :gh:issue:`615`: the ``mitogen_fetch`` * :gh:issue:`615`: a modified :ans:mod:`fetch` implements streaming transfer
action is included, and the standard Ansible :ans:mod:`fetch` is redirected even when ``become`` is active, avoiding excess CPU usage and memory spikes,
to it. This implements streaming file transfer in every case, including when and improving performance. A copy of two 512 MiB files drops from 47 seconds
``become`` is active, preventing excessive CPU usage and memory spikes, and to 7 seconds, with peak memory usage dropping from 10.7 GiB to 64.8 MiB.
significantly improving throughput. A copy of 2 files of 512 MiB each drops
from 47 seconds to just under 7 seconds, with peak memory usage dropping from
10.7 GiB to 64.8 MiB.
* `Operon <https://networkgenomics.com/operon/>`_ no longer requires a custom * `Operon <https://networkgenomics.com/operon/>`_ no longer requires a custom
installation, both Operon and Ansible are supported by a unified release. library installation, both Ansible and Operon are supported by a single
Mitogen release.
* The ``MITOGEN_CPU_COUNT`` environment variable shards the connection * The ``MITOGEN_CPU_COUNT`` variable shards the connection multiplexer into
multiplexer into per-CPU workers. This may improve throughput for runs per-CPU workers. This may improve throughput for large runs involving file
involving large file transfers, and is required for future in-process SSH transfer, and is required for future functionality. One multiplexer starts by
support. One multiplexer starts by default, to match existing behaviour. default, to match existing behaviour.
* :gh:commit:`d6faff06`, * :gh:commit:`d6faff06`, :gh:commit:`807cbef9`, :gh:commit:`e93762b3`,
:gh:commit:`807cbef9`, :gh:commit:`50bfe4c7`: locking is avoided on hot paths, and some locks are
:gh:commit:`e93762b3`, released before waking a thread that must immediately acquire the same lock.
:gh:commit:`50bfe4c7`: locking is
avoided on hot paths, and some locks are released earlier, before waking a
thread that must immediately take the same lock.
Mitogen for Ansible Mitogen for Ansible
@ -80,46 +84,41 @@ Mitogen for Ansible
* :gh:issue:`410`: Uses of :linux:man7:`unix` sockets are replaced with * :gh:issue:`410`: Uses of :linux:man7:`unix` sockets are replaced with
traditional :linux:man7:`pipe` pairs when SELinux is detected, to work around traditional :linux:man7:`pipe` pairs when SELinux is detected, to work around
a broken heuristic in popular SELinux policies that prevents inheriting a broken heuristic in common SELinux policies that prevents inheriting
:linux:man7:`unix` sockets across privilege domains. :linux:man7:`unix` sockets across privilege domains.
* `#467 <httpe://github.com/dw/mitogen/issues/467>`_: an incompatibility * `#467 <httpe://github.com/dw/mitogen/issues/467>`_: an incompatibility
running Mitogen under Molecule was resolved. running Mitogen under `Molecule
<https://molecule.readthedocs.io/en/stable/>`_ was resolved.
* :gh:issue:`547`, :gh:issue:`598`: fix a serious deadlock
possible while initializing the service pool of any child, such as during * :gh:issue:`547`, :gh:issue:`598`: fix a deadlock during initialization of
connection, ``async`` tasks, tasks using custom :mod:`module_utils`, connections, ``async`` tasks, tasks using custom :mod:`module_utils`,
``mitogen_task_isolation: fork`` modules, and those present on an internal ``mitogen_task_isolation: fork`` modules, and modules present on an internal
blacklist of misbehaving modules. blacklist. This would manifest as a timeout or hang, was easily hit, had been
present since 0.2.0, and likely impacted many users.
This deadlock is relatively easy hit, has been present since 0.2.0, and is
likely to have impacted many users. For new connections it could manifest as * :gh:issue:`549`: the open file limit is increased to the permitted hard
a *Connection timed out* error, for forked tasks it could manifest as a limit. It is common for distributions to ship with a higher hard limit than
timeout or an apparent hang. the default soft limit, allowing *"too many open files"* errors to be avoided
more often in large runs without user intervention.
* :gh:issue:`549`: the open file descriptor limit for the Ansible process is
increased to the available hard limit. It is common for distributions to ship * :gh:issue:`558`, :gh:issue:`582`: on Ansible 2.3 a directory was
with a much higher hard limit than their default soft limit, allowing *"too
many open files"* errors to be avoided more often in large runs without user
configuration.
* :gh:issue:`558`, :gh:issue:`582`: on Ansible 2.3 a remote directory was
unconditionally deleted after the first module belonging to an action plug-in unconditionally deleted after the first module belonging to an action plug-in
had executed, causing the :ans:mod:`unarchive` module to fail. had executed, causing the :ans:mod:`unarchive` to fail.
* :gh:issue:`578`: the extension could crash while rendering an error message, * :gh:issue:`578`: the extension could crash while rendering an error due to an
due to an incorrect format string. incorrect format string.
* :gh:issue:`590`: the importer can handle modules that replace themselves in * :gh:issue:`590`: the importer can handle modules that replace themselves in
:data:`sys.modules` during import. :data:`sys.modules` with completely unrelated modules during import, as in
the case of Ansible 2.8 :mod:`ansible.module_utils.distro`.
* :gh:issue:`591`: the target's current working directory is restored to a * :gh:issue:`591`: the working directory is reset between tasks to ensure
known-existent directory between tasks to ensure :func:`os.getcwd` will not :func:`os.getcwd` cannot fail, in the same way :class:`AnsibleModule`
fail when called, in the same way that :class:`AnsibleModule` restores it resets it during initialization. However this restore happens before the
during initialization. However this restore happens before the module ever module executes, ensuring code that calls :func:`os.getcwd` prior to
executes, ensuring any code that calls :func:`os.getcwd` prior to
:class:`AnsibleModule` initialization, such as the Ansible 2.7 :class:`AnsibleModule` initialization, such as the Ansible 2.7
:ans:mod:`pip`, cannot fail due to the behavior of a prior task. :ans:mod:`pip`, cannot fail due to the actions of a prior task.
* :gh:issue:`593`: the SSH connection method exposes * :gh:issue:`593`: the SSH connection method exposes
``mitogen_ssh_keepalive_interval`` and ``mitogen_ssh_keepalive_count`` ``mitogen_ssh_keepalive_interval`` and ``mitogen_ssh_keepalive_count``
@ -131,32 +130,47 @@ Mitogen for Ansible
encoding. encoding.
* :gh:issue:`602`: connection configuration is more accurately inferred for * :gh:issue:`602`: connection configuration is more accurately inferred for
`meta: reset_connection`, the `synchronize` module, and for any action :ans:mod:`meta: reset_connection <meta>` the :ans:mod:`synchronize`, and for
plug-ins that establish additional connections. any action plug-ins that establish additional connections.
* :gh:issue:`598`, :gh:issue:`605`: fix a deadlock managing a shared counter * :gh:issue:`598`, :gh:issue:`605`: fix a deadlock managing a shared counter
used for load balancing. used for load balancing, present since 0.2.4.
* :gh:issue:`615`: streaming file transfer is implemented for ``fetch`` and * :gh:issue:`615`: streaming is implemented for the :ans:mod:`fetch` and other
other actions that transfer files from the target to the controller. actions that transfer files from targets to the controller. Previously files
Previously the file was sent in one message, requiring it to fit in RAM and delivered were sent in one message, requiring them to fit in RAM and be
be smaller than the internal message size limit. smaller than an internal message size sanity check. Transfers from controller
to targets have been streaming since 0.2.0.
* :gh:commit:`7ae926b3`: the Ansible :ans:mod:`lineinfile` began leaking * :gh:commit:`7ae926b3`: the :ans:mod:`lineinfile` leaks writable temporary
writable temporary file descriptors since Ansible 2.7.0. When file descriptors since Ansible 2.7.0. When :ans:mod:`~lineinfile` created or
:ans:mod:`~lineinfile` was used to create or modify a script, and that script modified a script, and that script was later executed, the execution could
was later executed, the execution could fail with "*text file busy*" due to fail with "*text file busy*". Temporary descriptors are now tracked and
the leaked descriptor. Temporary descriptors are now tracked and cleaned up cleaned up on exit for all modules.
on exit for all modules.
Core Library Core Library
~~~~~~~~~~~~ ~~~~~~~~~~~~
* Log readability is improving, and many :func:`repr` strings are more * Log readability is improving and many :func:`repr` strings are more
descriptive. The old pseudo-function-call format is slowly migrating to descriptive. The old pseudo-function-call format is migrating to
human-readable output where possible. For example, readable output where possible. For example, *"Stream(ssh:123).connect()"*
*"Stream(ssh:123).connect()"* might be written *"connecting to ssh:123"*. might be written *"connecting to ssh:123"*.
* In preparation for reducing default log output, many messages are delivered
to per-component loggers, including messages originating from children,
enabling :mod:`logging` aggregation to function as designed. An importer
message like::
12:00:00 D mitogen.ctx.remotehost mitogen: loading module "foo"
Might instead be logged to the ``mitogen.importer.[remotehost]`` logger::
12:00:00 D mitogen.importer.[remotehost] loading module "foo"
Allowing a filter or handler for ``mitogen.importer`` to select that logger
in every process. This introduces a small risk of leaking memory in
long-lived programs, as logger objects are internally persistent.
* :func:`bytearray` was removed from the list of supported serialization types. * :func:`bytearray` was removed from the list of supported serialization types.
It was never portable between Python versions, unused, and never made much It was never portable between Python versions, unused, and never made much
@ -168,27 +182,24 @@ Core Library
asynchronous context. asynchronous context.
* :gh:issue:`419`: the internal * :gh:issue:`419`: the internal
:class:`~mitogen.core.Stream` has been refactored into 7 new classes, :class:`~mitogen.core.Stream` has been refactored into many new classes,
modularizing protocol behaviour, output buffering, line-oriented input modularizing protocol behaviour, output buffering, line-oriented input
parsing, option handling and connection management. Connection setup is parsing, option handling and connection management. Connection setup is
internally asynchronous, laying almost all the groundwork needed for fully internally asynchronous, laying most groundwork for fully asynchronous
asynchronous connect, proxied Ansible become plug-ins, and integrating connect, proxied Ansible become plug-ins, and in-process SSH.
`libssh <https://www.libssh.org/>`_.
* :gh:issue:`169`, * :gh:issue:`169`,
:gh:issue:`419`: zombie subprocess reaping :gh:issue:`419`: zombie subprocess reaping
has vastly improved, by using timers to efficiently poll for a slow child to has vastly improved, by using timers to efficiently poll for a child to exit,
finish exiting, and delaying broker shutdown while any subprocess remains. and delaying shutdown while any subprocess remains. Polling avoids
Polling avoids relying on process-global configuration such as a `SIGCHLD` process-global configuration such as a `SIGCHLD` handler, or
handler, or :func:`signal.set_wakeup_fd` available in modern Python. :func:`signal.set_wakeup_fd` available in modern Python.
* :gh:issue:`256`, * :gh:issue:`256`, :gh:issue:`419`: most :func:`os.dup` use was eliminated,
:gh:issue:`419`: most :func:`os.dup` use along with most manual file descriptor management. Descriptors are trapped in
was eliminated, along with almost all manual file descriptor management. :func:`os.fdopen` objects at creation, ensuring a leaked object will close
Descriptors are trapped in :func:`os.fdopen` objects at creation, ensuring a itself, and ensuring every descriptor is fused to a `closed` flag, preventing
leaked object will close itself, and ensuring every descriptor is fused to a historical bugs where a double close could destroy unrelated descriptors.
`closed` flag, preventing historical bugs where a double close could destroy
descriptors belonging to unrelated streams.
* :gh:issue:`533`: routing accounts for * :gh:issue:`533`: routing accounts for
a race between a parent (or cousin) sending a message to a child via an a race between a parent (or cousin) sending a message to a child via an
@ -218,13 +229,13 @@ Core Library
deliver a message for some reason other than the sender cannot or should not deliver a message for some reason other than the sender cannot or should not
reach the recipient, and no reply-to address is present on the message, reach the recipient, and no reply-to address is present on the message,
instead send a :ref:`dead message <IS_DEAD>` to the original recipient. This instead send a :ref:`dead message <IS_DEAD>` to the original recipient. This
ensures a descriptive messages is delivered to a thread sleeping on the reply ensures a descriptive message is delivered to a thread sleeping on the reply
to a function call, where the reply might be dropped due to exceeding the to a function call, where the reply might be dropped due to exceeding the
maximum configured message size. maximum configured message size.
* :gh:issue:`624`: the number of threads used for a child's auto-started thread * :gh:issue:`624`: the number of threads used for a child's automatically
pool has been reduced from 16 to 2. This may drop to 1 in future, and become initialized service thread pool has been reduced from 16 to 2. This may drop
configurable via a :class:`Router` option. to 1 in future, and become configurable via a :class:`Router` option.
* :gh:commit:`a5536c35`: avoid quadratic * :gh:commit:`a5536c35`: avoid quadratic
buffer management when logging lines received from a child's redirected buffer management when logging lines received from a child's redirected
@ -264,6 +275,7 @@ bug reports, testing, features and fixes in this release contributed by
`Florent Dutheil <https://github.com/fdutheil>`_, `Florent Dutheil <https://github.com/fdutheil>`_,
`James Hogarth <https://github.com/hogarthj>`_, `James Hogarth <https://github.com/hogarthj>`_,
`Jordan Webb <https://github.com/jordemort>`_, `Jordan Webb <https://github.com/jordemort>`_,
`Julian Andres Klode <https://github.com/julian-klode>`_,
`Marc Hartmayer <https://github.com/marc1006>`_, `Marc Hartmayer <https://github.com/marc1006>`_,
`Nigel Metheringham <https://github.com/nigelm>`_, `Nigel Metheringham <https://github.com/nigelm>`_,
`Orion Poplawski <https://github.com/opoplawski>`_, `Orion Poplawski <https://github.com/opoplawski>`_,
@ -271,6 +283,7 @@ bug reports, testing, features and fixes in this release contributed by
`Stefane Fermigier <https://github.com/sfermigier>`_, `Stefane Fermigier <https://github.com/sfermigier>`_,
`Szabó Dániel Ernő <https://github.com/r3ap3rpy>`_, `Szabó Dániel Ernő <https://github.com/r3ap3rpy>`_,
`Ulrich Schreiner <https://github.com/ulrichSchreiner>`_, `Ulrich Schreiner <https://github.com/ulrichSchreiner>`_,
`Vincent S. Cojot <https://github.com/ElCoyote27>`_,
`yen <https://github.com/antigenius0910>`_, `yen <https://github.com/antigenius0910>`_,
`Yuki Nishida <https://github.com/yuki-nishida-exa>`_, `Yuki Nishida <https://github.com/yuki-nishida-exa>`_,
`@alexhexabeam <https://github.com/alexhexabeam>`_, `@alexhexabeam <https://github.com/alexhexabeam>`_,

8
docs/conf.py
Unescape Escape View File

@ -10,6 +10,10 @@ author = u'Network Genomics'
copyright = u'2019, Network Genomics' copyright = u'2019, Network Genomics'
exclude_patterns = ['_build', '.venv'] exclude_patterns = ['_build', '.venv']
extensions = ['sphinx.ext.autodoc', 'sphinx.ext.intersphinx', 'sphinxcontrib.programoutput', 'domainrefs'] extensions = ['sphinx.ext.autodoc', 'sphinx.ext.intersphinx', 'sphinxcontrib.programoutput', 'domainrefs']
# get rid of version from <title>, it messes with piwik
html_title = 'Mitogen Documentation'
html_show_copyright = False html_show_copyright = False
html_show_sourcelink = False html_show_sourcelink = False
html_show_sphinx = False html_show_sphinx = False
@ -51,11 +55,11 @@ domainrefs = {
'url': 'https://github.com/dw/mitogen/pull/%s', 'url': 'https://github.com/dw/mitogen/pull/%s',
}, },
'ans:mod': { 'ans:mod': {
'text': '%s Module', 'text': '%s module',
'url': 'https://docs.ansible.com/ansible/latest/modules/%s_module.html', 'url': 'https://docs.ansible.com/ansible/latest/modules/%s_module.html',
}, },
'ans:conn': { 'ans:conn': {
'text': '%s Connection Plug-in', 'text': '%s connection plug-in',
'url': 'https://docs.ansible.com/ansible/latest/plugins/connection/%s.html', 'url': 'https://docs.ansible.com/ansible/latest/plugins/connection/%s.html',
}, },
'freebsd:man2': { 'freebsd:man2': {

4
docs/toc.rst
Unescape Escape View File

@ -7,11 +7,11 @@ Table Of Contents
index index
Mitogen for Ansible <ansible_detailed> Mitogen for Ansible <ansible_detailed>
contributors
changelog changelog
contributors
howitworks howitworks
getting_started
api api
getting_started
examples examples
internals internals

2
mitogen/__init__.py
Unescape Escape View File

@ -35,7 +35,7 @@ be expected. On the slave, it is built dynamically during startup.
#: Library version as a tuple. #: Library version as a tuple.
__version__ = (0, 2, 7) __version__ = (0, 2, 8)
#: This is :data:`False` in slave contexts. Previously it was used to prevent #: This is :data:`False` in slave contexts. Previously it was used to prevent

4
mitogen/parent.py
Unescape Escape View File

@ -2607,9 +2607,9 @@ class Reaper(object):
if not self.kill: if not self.kill:
pass pass
elif self._tries == 1: elif self._tries == 2:
self._signal_child(signal.SIGTERM) self._signal_child(signal.SIGTERM)
elif self._tries == 5: # roughly 4 seconds elif self._tries == 6: # roughly 4 seconds
self._signal_child(signal.SIGKILL) self._signal_child(signal.SIGKILL)

54
tests/reaper_test.py
Unescape Escape View File

@ -0,0 +1,54 @@
import signal
import unittest2
import testlib
import mock
import mitogen.parent
class ReaperTest(testlib.TestCase):
@mock.patch('os.kill')
def test_calc_delay(self, kill):
broker = mock.Mock()
proc = mock.Mock()
proc.poll.return_value = None
reaper = mitogen.parent.Reaper(broker, proc, True, True)
self.assertEquals(50, int(1000 * reaper._calc_delay(0)))
self.assertEquals(86, int(1000 * reaper._calc_delay(1)))
self.assertEquals(147, int(1000 * reaper._calc_delay(2)))
self.assertEquals(254, int(1000 * reaper._calc_delay(3)))
self.assertEquals(437, int(1000 * reaper._calc_delay(4)))
self.assertEquals(752, int(1000 * reaper._calc_delay(5)))
self.assertEquals(1294, int(1000 * reaper._calc_delay(6)))
@mock.patch('os.kill')
def test_reap_calls(self, kill):
broker = mock.Mock()
proc = mock.Mock()
proc.poll.return_value = None
reaper = mitogen.parent.Reaper(broker, proc, True, True)
reaper.reap()
self.assertEquals(0, kill.call_count)
reaper.reap()
self.assertEquals(1, kill.call_count)
reaper.reap()
reaper.reap()
reaper.reap()
self.assertEquals(1, kill.call_count)
reaper.reap()
self.assertEquals(2, kill.call_count)
self.assertEquals(kill.mock_calls, [
mock.call(proc.pid, signal.SIGTERM),
mock.call(proc.pid, signal.SIGKILL),
])
if __name__ == '__main__':
unittest2.main()
Write Preview
Loading…
Cancel
Save