From 7b046c9fbae04ca126814734636c833c2457b7ee Mon Sep 17 00:00:00 2001 From: David Wilson Date: Sun, 10 Feb 2019 01:39:27 +0000 Subject: [PATCH 1/9] docs: fix changelog syntax/order/"20KB" --- docs/changelog.rst | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/changelog.rst b/docs/changelog.rst index 15ff67d7..9148226f 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -455,6 +455,15 @@ Core Library import :mod:`__main__` on Python 3.4 and newer due to a breaking change in the :mod:`pkgutil` API. The program's main script is now handled specially. +* `#523 `_: the test suite didn't + generate a code coverage report if any test failed. + +* `#524 `_: Python 3.6+ emitted a + :class:`DeprecationWarning` for :func:`mitogen.utils.run_with_router`. + +* `#529 `_: Code coverage of the + test suite was not measured across all Python versions. + * `16ca111e `_: handle OpenSSH 7.5 permission denied prompts when ``~/.ssh/config`` rewrites are present. @@ -475,18 +484,9 @@ Core Library a failure. * `57b652ed `_: a stray import - meant an extra roundtrip and ~20KiB of data was wasted for any context that + meant an extra roundtrip and ~4KiB of data was wasted for any context that imported :mod:`mitogen.parent`. -* `#523 ` : the test suite didn't - generate a code coverage report if any test failed. - -* `#524 ` : Python 3.6+ emitted a - :class:`DeprecationWarning` for :func:`mitogen.utils.run_with_router`. - -* `#529 ` : Code coverage of the - test suite was not measured across all Python versions. - Thanks! ~~~~~~~ From 4f6c57b6a88ef96874b003d436d711b3c8e44c74 Mon Sep 17 00:00:00 2001 From: David Wilson Date: Sun, 10 Feb 2019 02:20:13 +0000 Subject: [PATCH 2/9] docs: lose "approaching stability" language, we're pretty good now --- docs/ansible.rst | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/ansible.rst b/docs/ansible.rst index e45a9f7a..17354755 100644 --- a/docs/ansible.rst +++ b/docs/ansible.rst @@ -10,9 +10,8 @@ Mitogen, replacing embedded shell invocations with pure-Python equivalents invoked via highly efficient remote procedure calls to persistent interpreters tunnelled over SSH. No changes are required to target hosts. -The extension is approaching stability and real-world usage is encouraged. `Bug -reports`_ are welcome: Ansible is huge, and only wide testing will ensure -soundness. +The extension is stable and real-world use is encouraged. `Bug reports`_ are +welcome: Ansible is huge, and only wide testing will ensure soundness. .. _Ansible: https://www.ansible.com/ From 32161f3df524fe7ccb962b55ef547d9afc0461d8 Mon Sep 17 00:00:00 2001 From: David Wilson Date: Sun, 10 Feb 2019 02:28:41 +0000 Subject: [PATCH 3/9] docs: update thanks --- docs/changelog.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/changelog.rst b/docs/changelog.rst index 9148226f..60a84685 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -508,6 +508,7 @@ bug reports, testing, features and fixes in this release contributed by `Johan Beisser `_, `Jonathan Rosser `_, `Josh Smift `_, +`Kevin Carter `_, `Mehdi `_, `Michael DeHaan `_, `Michal Medvecky `_, From 9c7d93885dbe413eea2d983c4292833d6b2a163a Mon Sep 17 00:00:00 2001 From: David Wilson Date: Sun, 10 Feb 2019 02:37:00 +0000 Subject: [PATCH 4/9] .ci: add README.md. --- .ci/README.md | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100644 .ci/README.md diff --git a/.ci/README.md b/.ci/README.md new file mode 100644 index 00000000..d99d8e6b --- /dev/null +++ b/.ci/README.md @@ -0,0 +1,35 @@ + +# `.ci` + +This directory contains scripts for Travis CI and (more or less) Azure +Pipelines, but they will also happily run on any Debian-like machine. + +The scripts are usually split into `_install` and `_test` steps. The `_install` +step will damage your machine, the `_test` step will just run the tests the way +CI runs them. + +There is a common library, `ci_lib.py`, which just centralized a bunch of +random macros and also environment parsing. + +Some of the scripts allow you to pass extra flags through to the component +under test, e.g. `../../.ci/ansible_tests.py -vvv` will run with verbose. + +Hack these scripts until your heart is content. There is no pride to be found +here, just necessity. + +### Environment Variables + +* `VER`: Ansible version the `_install` script should install. Default changes + over time. +* `TARGET_COUNT`: number of targets for `debops_` run. Defaults to 2. +* `DISTRO`: the `mitogen_` tests need a target Docker container distro. This + name comes from the Docker Hub `mitogen` user, i.e. `mitogen/$DISTRO-test` +* `DISTROS`: the `ansible_` tests can run against multiple targets + simultaneously, which speeds things up. This is a space-separated list of + DISTRO names, but additionally, supports: + * `debian-py3`: when generating Ansible inventory file, set + `ansible_python_interpreter` to `python3`, i.e. run a test where the + target interpreter is Python 3. + * `debian*16`: generate 16 Docker containers running Debian. Also works + with -py3. + From 6dba55624eb83a996fe7a2bccf0bf6f36a788365 Mon Sep 17 00:00:00 2001 From: David Wilson Date: Sun, 10 Feb 2019 02:40:14 +0000 Subject: [PATCH 5/9] .ci: add verbiage for run_batches() too. --- .ci/README.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/.ci/README.md b/.ci/README.md index d99d8e6b..1b9f9dfa 100644 --- a/.ci/README.md +++ b/.ci/README.md @@ -17,6 +17,15 @@ under test, e.g. `../../.ci/ansible_tests.py -vvv` will run with verbose. Hack these scripts until your heart is content. There is no pride to be found here, just necessity. + +### `ci_lib.run_batches()` + +There are some weird looking functions to extract more paralellism from the +build. The above function takes lists of strings, arranging for the strings in +each list to run in order, but for the lists to run in parallel. That's great +for doing `setup.py install` while pulling a Docker container, for example. + + ### Environment Variables * `VER`: Ansible version the `_install` script should install. Default changes From 92b1648729bf63f3631e0756a92dd43bb1d9a782 Mon Sep 17 00:00:00 2001 From: David Wilson Date: Sun, 10 Feb 2019 14:13:22 +0000 Subject: [PATCH 6/9] Add a few more important modules to preamble_size.py. --- preamble_size.py | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/preamble_size.py b/preamble_size.py index bf3b5950..ead5af85 100644 --- a/preamble_size.py +++ b/preamble_size.py @@ -16,6 +16,9 @@ import mitogen.service import mitogen.ssh import mitogen.sudo +import ansible_mitogen.runner +import ansible_mitogen.target + router = mitogen.master.Router() context = mitogen.parent.Context(router, 0) stream = mitogen.ssh.Stream(router, 0, max_message_size=0, hostname='foo') @@ -31,7 +34,7 @@ if '--dump' in sys.argv: print( - ' ' + ' ' ' ' ' Original ' ' ' @@ -42,6 +45,9 @@ print( for mod in ( mitogen.parent, + mitogen.fork, + ansible_mitogen.target, + ansible_mitogen.runner, mitogen.ssh, mitogen.sudo, mitogen.select, @@ -56,7 +62,7 @@ for mod in ( compressed = zlib.compress(minimized, 9) compressed_size = len(compressed) print( - '%-15s' + '%-25s' ' ' '%5i %4.1fKiB' ' ' From b263e01867b10caf39685d02388cd82620f7103a Mon Sep 17 00:00:00 2001 From: David Wilson Date: Sun, 10 Feb 2019 19:19:03 +0000 Subject: [PATCH 7/9] issue #481: avoid crash if disconnect occurs during forward_modules() --- mitogen/master.py | 5 +++++ tests/responder_test.py | 13 +++++++++++++ 2 files changed, 18 insertions(+) diff --git a/mitogen/master.py b/mitogen/master.py index 427bf731..257fb81b 100644 --- a/mitogen/master.py +++ b/mitogen/master.py @@ -922,6 +922,11 @@ class ModuleResponder(object): fullname, _, _ = str_rpartition(fullname, u'.') stream = self._router.stream_by_id(context.context_id) + if stream is None: + LOG.debug('%r: dropping forward of %s to no longer existent ' + '%r', self, path[0], context) + return + for fullname in reversed(path): self._send_module_and_related(stream, fullname) self._send_forward_module(stream, context, fullname) diff --git a/tests/responder_test.py b/tests/responder_test.py index 15f21976..da4f22d7 100644 --- a/tests/responder_test.py +++ b/tests/responder_test.py @@ -167,6 +167,19 @@ class BrokenModulesTest(testlib.TestCase): class ForwardTest(testlib.RouterMixin, testlib.TestCase): + def test_forward_to_nonexistent_context(self): + nonexistent = mitogen.core.Context(self.router, 123) + capture = testlib.LogCapturer() + capture.start() + self.broker.defer_sync(lambda: + self.router.responder.forward_modules( + nonexistent, + ['mitogen.core'] + ) + ) + s = capture.stop() + self.assertTrue('dropping forward of' in s) + def test_stats(self): # Forwarding stats broken because forwarding is broken. See #469. c1 = self.router.local() From 78ec634dab898a1106a2da7b6ebe22ca68f366c6 Mon Sep 17 00:00:00 2001 From: David Wilson Date: Sun, 10 Feb 2019 19:44:46 +0000 Subject: [PATCH 8/9] issue #481: core: preserve stderr TTY FD if one is present. Since 802de6a8d585fbc24434a993aa0e2bba02920ce1, sudo on CentOS 5 had begun failing due to a TTY FD leak in the parent process being fixed. The old versions of sudo doesn't hang around after starting a child -- they exec the privilege-escalated child process on top of themselves, meaning no spare copy of the TTY FD is kept alive by sudo. When the child starts up, it replaces stdio with IoLoggers, including the inherited stderr FD connected to DiagLogStream/the slave PTY. When the last process closes a slave PTY, the kernel sends SIGHUP to any processes still having it as the controlling TTY. Therefore we must either ignore SIGHUP until the first stage has been waited on (since the first stage also preserve the FD), or dup the inherited TTY FD and keep it around forever. Wasting one FD seems less annoying than modifying process signals for all potential library users, so that is the approach taken here. --- mitogen/core.py | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/mitogen/core.py b/mitogen/core.py index 3694342c..a48e13ed 100644 --- a/mitogen/core.py +++ b/mitogen/core.py @@ -3295,6 +3295,18 @@ class ExternalContext(object): os.close(fd) def _setup_stdio(self): + # #481: when stderr is a TTY due to being started via + # tty_create_child()/hybrid_tty_create_child(), and some privilege + # escalation tool like prehistoric versions of sudo exec this process + # over the top of itself, there is nothing left to keep the slave PTY + # open after we replace our stdio. Therefore if stderr is a TTY, keep + # around a permanent dup() to avoid receiving SIGHUP. + try: + if os.isatty(2): + self.reserve_tty_fd = os.dup(2) + set_cloexec(self.reserve_tty_fd) + except OSError: + pass # When sys.stdout was opened by the runtime, overwriting it will not # close FD 1. However when forking from a child that previously used # fdopen(), overwriting it /will/ close FD 1. So we must swallow the From b12539f99b42c4accfff881eb68ee011ce129dd3 Mon Sep 17 00:00:00 2001 From: David Wilson Date: Sun, 10 Feb 2019 19:53:00 +0000 Subject: [PATCH 9/9] docs: update Changelog; closes #481 --- docs/changelog.rst | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/changelog.rst b/docs/changelog.rst index 60a84685..bcca6088 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -455,6 +455,12 @@ Core Library import :mod:`__main__` on Python 3.4 and newer due to a breaking change in the :mod:`pkgutil` API. The program's main script is now handled specially. +* `#481 `_: the version of `sudo` + that shipped with CentOS 5 replaced itself with the program to be executed, + and therefore did not hold any child PTY open on our behalf. The child + context is updated to preserve any PTY FD in order to avoid the kernel + sending `SIGHUP` early during startup. + * `#523 `_: the test suite didn't generate a code coverage report if any test failed.