Commit Graph

2258 Commits (d763570ca2a23b7d21c241aff47316f3f1a36049)
 

Author SHA1 Message Date
David Wilson d763570ca2 Merge remote-tracking branch 'origin/dmw'
* origin/dmw:
  ci: fix incorrect partition/rpartition from 8a4caea84f
  issue #260: hide force-disconnect messages.
  issue #498: fix shutdown crash
  issue #260: avoid start_transmit()/on_transmit()/stop_transmit()
  core: ensure broker profiling output reaches disk
  master: keep is_stdlib_path() result as negative cache entry
  ci: Allow DISTROS="debian*32" variable, and KEEP=1
6 years ago
David Wilson 49a8745a45 ci: fix incorrect partition/rpartition from 8a4caea84f 6 years ago
David Wilson 696cee57dd issue #260: hide force-disconnect messages.
Parent is always force-disconnected, and now it shows up quite visibly
on every exit.
6 years ago
David Wilson 26c6c6d048 issue #498: fix shutdown crash
Traceback (most recent call last):
      File "<stdin>", line 2707, in _invoke
      File "<stdin>", line 2480, in _on_del_route
    NameError: global name 'target_id' is not defined
6 years ago
David Wilson a18a083c94 issue #260: avoid start_transmit()/on_transmit()/stop_transmit()
Previous transmit sequence was:
        Router._async_route -> Stream._send -> Broker._start_transmit ->
        Broker.loop -> Stream.on_transmit -> socket.write ->
        Broker.stop_transmit

New sequence, when socket buffer can hold message is:
        Router._async_route -> Stream._send -> socket.write

bench/roundtrip.py
        Before: 240 usec
        after: 178 usec

Stat before:
       5088.276050      task-clock (msec)         #    0.997 CPUs utilized
           185,568      context-switches          #    0.036 M/sec
                 0      cpu-migrations            #    0.000 K/sec
            18,923      page-faults               #    0.004 M/sec
    13,063,871,501      cycles                    #    2.567 GHz
    12,834,579,684      instructions              #    0.98  insn per cycle
     2,669,820,684      branches                  #  524.700 M/sec
       107,296,033      branch-misses             #    4.02% of all branches

       5.105018296 seconds time elapsed

       2.350970000 seconds user
       0.345497000 seconds sys

Stat after:
       4019.208047      task-clock (msec)         #    0.998 CPUs utilized
           249,471      context-switches          #    0.062 M/sec
                 0      cpu-migrations            #    0.000 K/sec
            20,990      page-faults               #    0.005 M/sec
    10,312,535,979      cycles                    #    2.566 GHz
    11,586,365,995      instructions              #    1.12  insn per cycle
     2,392,933,370      branches                  #  595.374 M/sec
        75,432,205      branch-misses             #    3.15% of all branches

       4.028763347 seconds time elapsed

       3.367051000 seconds user
       0.652962000 seconds sys
6 years ago
David Wilson b161408598 Merge commit 'fe74577'
* commit 'fe74577':
  Use develop mode in tox
  issue #429: fix sudo regression.
  misc: rename to scripts. tab completion!!
  core: Latch._wake improvements
  issue #498: prevent crash on double 'disconnect' signal.
  issue #413: don't double-propagate DEL_ROUTE to parent.
  issue #498: wrap Router dict mutations in a lock
  issue #429: enable en_US locale to unbreak debops test.
  issue #499: fix another mind-numbingly stupid vanilla inconsistency
  issue #497: do our best to cope with crap upstream code
  ssh: fix test to match updated log format.
  issue #429: update Changelog.
  issue #429: update Changelog.
  issue #429: teach sudo about every know i18n password string.
  issue #429: install i18n-related bits in test images.
  ssh: tidy up logs and stream names.
  tests: ensure file is closed in connection_test.
  gcloud: small updates
  tests: give ansible/gcloud/ its own requirements file.
6 years ago
David Wilson d4a0b70e15 core: ensure broker profiling output reaches disk
Profiler hasn't been used much since the hard exit was added.
6 years ago
David Wilson dc4b27c6bf master: keep is_stdlib_path() result as negative cache entry
On 32x Docker run of issue_140__thread_pileup.yml

Before:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  1218500    1.716    0.000    7.325    0.000 /home/dmw/src/mitogen/mitogen/master.py:118(is_stdlib_path)

After:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      166    0.000    0.000    0.001    0.000 /home/dmw/src/mitogen/mitogen/master.py:123(is_stdlib_path)
6 years ago
David Wilson 8a4caea84f ci: Allow DISTROS="debian*32" variable, and KEEP=1 6 years ago
David Wilson fe745779ba Use develop mode in tox 6 years ago
David Wilson fca5513610 issue #429: fix sudo regression. 6 years ago
David Wilson ae8173a29f misc: rename to scripts. tab completion!! 6 years ago
David Wilson 0c0d34241b core: Latch._wake improvements
os.write() can fail with EINTR due to signals, so wrap it in
io_op(). Closes #483.

Masking EBADF looks like it is/was almost certainly papering over a bug,
remove it and suffer the bug reports. Closes #495.
6 years ago
David Wilson 407294cd79 issue #498: prevent crash on double 'disconnect' signal.
Fixes:

ERROR! [pid 1096] 23:31:48.363215 E mitogen: _broker_main() crashed
Traceback (most recent call last):
  File "/home/dmw/src/mitogen/mitogen/core.py", line 2917, in _broker_main
    self._loop_once()
  File "/home/dmw/src/mitogen/mitogen/core.py", line 2875, in _loop_once
    self._call(side.stream, func)
  File "/home/dmw/src/mitogen/mitogen/core.py", line 2860, in _call
    stream.on_disconnect(self)
  File "/home/dmw/src/mitogen/mitogen/parent.py", line 1161, in on_disconnect
    super(Stream, self).on_disconnect(broker)
  File "/home/dmw/src/mitogen/mitogen/core.py", line 1534, in on_disconnect
    fire(self, 'disconnect')
  File "/home/dmw/src/mitogen/mitogen/core.py", line 390, in fire
    func(*args, **kwargs)
  File "/home/dmw/src/mitogen/mitogen/parent.py", line 1794, in <lambda>
    func=lambda: self._on_stream_disconnect(stream),
  File "/home/dmw/src/mitogen/mitogen/parent.py", line 1810, in _on_stream_disconnect
    routes = self._routes_by_stream.pop(stream)
KeyError: mitogen.ssh.Stream('ssh.localhost:2236')
6 years ago
David Wilson 9aa845669c issue #413: don't double-propagate DEL_ROUTE to parent.
propagate_up() sends ADD_ROUTE and DEL_ROUTE

propagate_down() sends only DEL_ROUTE, but didn't bother checking if
up() had sent it already.

Fixes:

     ERROR! [pid 41060] 17:55:30.739159 E mitogen.ctx.ssh.localhost:
     mitogen: RouteMonitor(): received DEL_ROUTE for 6081 from
     mitogen.fork.Stream(u'fork.41142'), expected
     mitogen.core.Stream('parent')
6 years ago
David Wilson e2478dcb9f issue #498: wrap Router dict mutations in a lock 6 years ago
David Wilson 60fe3fd6f5 issue #429: enable en_US locale to unbreak debops test. 6 years ago
David Wilson 7531af3ee0 issue #499: fix another mind-numbingly stupid vanilla inconsistency 6 years ago
David Wilson a40946297f issue #497: do our best to cope with crap upstream code 6 years ago
David Wilson 2fdbd0cfcd ssh: fix test to match updated log format. 6 years ago
David Wilson 77e7cadd22 issue #429: update Changelog. 6 years ago
David Wilson cd1e5e0138 issue #429: update Changelog. 6 years ago
David Wilson 6e9f8e829e issue #429: teach sudo about every know i18n password string. 6 years ago
David Wilson 960e505f07 issue #429: install i18n-related bits in test images. 6 years ago
David Wilson ec789513dc ssh: tidy up logs and stream names. 6 years ago
David Wilson eb93f82d05 tests: ensure file is closed in connection_test. 6 years ago
David Wilson 73979043ad gcloud: small updates 6 years ago
David Wilson 59068ca955 tests: give ansible/gcloud/ its own requirements file. 6 years ago
David Wilson fba06b0bde Merge remote-tracking branch 'origin/issue499'
* origin/issue499:
  issue #499: another totally moronic implementation difference
  issue #499: disable new test on vanilla.
  docs: update Changelog; closes #499.
  issue #499: respect C.BECOME_ALLOW_SAME_USER.
6 years ago
David Wilson 9df314f9c5 issue #499: another totally moronic implementation difference 6 years ago
David Wilson 53794469a0 issue #499: disable new test on vanilla. 6 years ago
David Wilson 60e9596e7d docs: update Changelog; closes #499. 6 years ago
David Wilson a1121c5a84 issue #499: respect C.BECOME_ALLOW_SAME_USER. 6 years ago
David Wilson 16a2de65c4 issue #493: another Py3.x fix. 6 years ago
David Wilson 2a70b3d5f4 issue #493: Py3.x fix. 6 years ago
David Wilson 9f7db66f67 tox: add py37 target. 6 years ago
David Wilson 3158c4ae9d docs: update Changelog; closes #493. 6 years ago
David Wilson bc84d1e950 issue #493: less CPU-intensive cookie format. 6 years ago
David Wilson 14d393765b issue #488: 2.4-compatible syntax in unix.py. 6 years ago
David Wilson b254051416 ansible: add test to ensure UNIX socket is cleaned on exit; closes #488. 6 years ago
David Wilson 628e8f4466 Bump PyYaML to a 3.7-compatible version. 6 years ago
David Wilson be6ab52fe1 issue #488: fix shutdown damage caused in 6ca2677de5
os._exit() subverted calm shutdown, meaning unix.Listener never had a
chance to cleanup its socket.

Move unix.Listener socket cleanup into its class so it is automatic
during shutdown, rather than cutpasted for each consumer.

Disable the watcher thread in the MuxProcess, it is useless.

Add .sock extension to /tmp/mitogen_unix_*, so we can write a test.
6 years ago
David Wilson 104fa463af Merge remote-tracking branch 'origin/issue479'
* origin/issue479:
  issue #490: log mitogen.unix server-side accept.
  issue #490: have Side._on_fork() empty _fork_refs
  issue #490: prevent double close() destroying unrelated Connection.
  docs: update Changelog; closes #479.
  issue #479: ModuleFinder special case for __main__ on Py3.x.
6 years ago
David Wilson b370076689 issue #490: log mitogen.unix server-side accept. 6 years ago
David Wilson 7dae88f0f5 issue #490: have Side._on_fork() empty _fork_refs
This is mostly to avoid ugly debugging that depends on the state of GC.
Discard sides from _fork_refs after they have been closed.
6 years ago
David Wilson 38a553d42d issue #490: prevent double close() destroying unrelated Connection. 6 years ago
David Wilson bf676aacfe docs: update Changelog; closes #479. 6 years ago
David Wilson 3435f24e8d issue #479: ModuleFinder special case for __main__ on Py3.x. 6 years ago
David Wilson 245dd9e166 Merge remote-tracking branch 'origin/dmw'
* origin/dmw: (135 commits)
  tests: just disable the test.
  tests: hopefully fix this dumb test for the final time
  docs: update Changelog; closes #477.
  issue #477: use MITOGEN_INVENTORY_FILE everywhere.
  issue #477: hacksmash weird 2.3 inventory_file var issue.
  issue #477: travis.yml typo.
  issue #477: fix sudo_args selection.
  issue #477: one more conditional test.
  issue #477: enable Ansible 2.3.3 CI.
  issue #477: some more conditional tests.
  docs: update Changelog.
  issue #477 / ansible: avoid a race in async job startup.
  issue #477: use assert_equal for nicer debug.
  issue #477: fix source of become_flags on 2.3.
  issue #477: add Connection.homedir test.
  core: docstring tidyups.
  core: ensure early debug messages are logged correctly.
  core: log disconnection reason.
  issue #477: target.file_exists() wrapper.
  issue #477: introduce subprocess isolation.
  ansible: docstring fixes.
  issue #477: paper over Ansible 2.3 flag handling difference
  issue #477: update forking_correct_parent for subprocess isolation
  issue #477: shlex.split() in 2.4 required bytes input.
  issue #477: get rid of perl JSON module requirement.
  issue #477: Ansible 2.3 did not support gather_facts min subset.
  issue #477: CentOS 5 image requires perl installed too.
  issue #477: missing stub-su.py from 137f5fa6c5
  issue #477: 2.4-compatible syntax.
  issue #477: clearing glibc caches is not possible on Py2.4.
  parent: --with-pydebug bootstrap could fail due to corrupted stream
  issue #477: install simplejson for vanilla tests.
  docs: update Changelog.
  ansible: synchronize module needs '.docker_cmd' attr for Docker plugin.
  issue #477: add basic su_test and Py2.4 polyfill.
  issue #477: import updated Python build scripts
  ci: don't use the TTY->pipe hack except on Travis where it's needed.
  WIP first run of py24 CI
  issue #477: initial Python 2.4.6 build for CI.
  issue #477: enable git-lfs for tests/data/*.tar.bz2.
  issue #477: import build script for Python 2.4.6.
  issue #477: add mitogen_py24 CI test type.
  issue #477: disable Django parts of module_finder_test on 2.4.
  issue #477: clean up globals after unix_test.
  issue #477: remove unused pytest bits from importer_test.
  issue #477: remove fork use from unix_test.
  parent: don't kill child when profiling=True
  issue #485: import new throuhgput bench
  issue #477: more fork removal
  issue #477: Py2.4 startswith() did not support tuples.
  issue #477: util/fakessh/two_three_compat fixes.
  issue #477: call_function_test fixes for 2.4.
  issue #477: promote setup_gil() to mitogen.utils
  issue #477: fix lxc_test any polyfill import.
  issue #477: stop using fork in responder_test.
  issue #477: stop using fork in service_test.
  issue #477: Python<2.5 ioctl() request parameter was signed.
  issue #477: stop using fork() in parent_test, compatible enumerate().
  issue #477: Popen.terminate() polyfill for Py2.4.
  issue #477: stop using .fork() in router_test, one small 2.4 fix.
  issue #477: document master.Router.max_message_size.
  issue #477: old Py zlib did not include extended exception text.
  issue #477: stop using router.fork() in receiver_test
  issue #477: any() polyfill for lxc_test.
  issue #477: replace type(e) -> __class__ for an exception
  issue #477: old Mock does not throw side_effect exceptions from a list
  issue #477: 2.4 stat() returned int timestamps not float.
  issue #477: set().union(a, b, ..) unsupported on Py2.4.
  issue #477: Logger.log(extra=) unsupported on Py2.4.
  issue #477: fix another Threading.getName() call.
  issue #477: %f date format requires Py2.6 or newer.
  issue #477: make mitogen.fork unsupported on Py<2.6.
  issue #477: Py2.4 dep scanner bytecode difference
  Drop 'alpha' trove classifier
  issue #477: fix another str/bytes mixup.
  issue #477: blacklist 'thread' module to avoid roundtrip on 2.x->3.x
  issue #477: fix 3.x failure in new target.set_file_mode() function.
  issue #477: fix 3.x failure in new target.set_file_mode() function.
  issue #477: fix 2 runner tests on Ansible 2.7.
  issue #477: fix 3.x test regressions.
  issue #477: fix new KwargsTest on Python 3.x.
  issue #477: ModuleFinder now returns Unicode module names.
  issue #477: Python3 does not have Pickler.dispatch.
  issue #477: ModuleFinder test fixes.
  issue #477: Ansible 2.3 compatible regression/all.yml.
  issue #477: Ansible 2.3 requires placeholder module for assert_equals
  issue #477: build a CentOS 5/Py2.4 container + playbook compat fixes.
  issue #477: use PY24 constant rather than explicit test.
  issue #477: backport mitogen.master to Python 2.4.
  issue #477: parent: make iter_read() log disconnect reason.
  issue #477: backport ansible_mitogen.runner to 2.4.
  issue #477: backport various test modules to Python 2.4.
  issue #477: backport ansible_mitogen/target.py to Python2.4
  issue #477: add all() polyfill to custom_python_detect_environmnet
  issue #477: polyfill partition() use in mitogen.parent.
  issue #477: polyfill partition() use in mitogen.service.
  issue #477: polyfill partition() use in mitogen.ssh.
  issue #477: vendorize the last 2.4-compatible simplejson
  issue #477: _update_linecache() must append newlines.
  issue #415, #477: Poller must handle POLLHUP too.
  ...
6 years ago
David Wilson a7da1b048b tests: just disable the test. 6 years ago