Commit Graph

2730 Commits (e87e41e69ebbe214994dddd2798e0b7d52203964)
 

Author SHA1 Message Date
David Wilson add357a029 os_fork: yet more doc tidyup 6 years ago
David Wilson 0a66ca72ef os_fork: more doc tweaks 6 years ago
David Wilson 5dc0bd6f8d os_fork: clean up docs 6 years ago
David Wilson 021cfda112 .ci: import soak scripts. 6 years ago
David Wilson ad7185444d .ci: allow containers for different jobs to run simultaneously 6 years ago
David Wilson c413d53144 os_fork: python 3 fixes and tests. 6 years ago
David Wilson 45e66541e0 Merge remote-tracking branch 'origin/dmw'
* origin/dmw:
  issue #535: activate Corker on 2.4 in master too.
6 years ago
David Wilson 18b984a0b4 issue #535: activate Corker on 2.4 in master too. 6 years ago
David Wilson 509590530b Merge remote-tracking branch 'origin/dmw'
* origin/dmw:
  issue #535: update Changelog.
  issue #535: wire mitogen.os_fork into Broker and Pool.
  issue #535: parent: add create_socketpair(size=..) parameter.
  issue #535: introduce mitogen.os_fork module and Corker class.
  issue #535: docs: update Changelog
  issue #535: service: support Pool.defer() like Broker.defer()
  issue #535: core: unicode.encode() may take importer lock on 2.x
  issue #535: docs: fix up Select doc
  issue #535: docs: update Changelog.
  issue #535: core/select: support selecting from Latches.
6 years ago
David Wilson ebbf5ab62c issue #535: update Changelog. 6 years ago
David Wilson 06e52ca89f issue #535: wire mitogen.os_fork into Broker and Pool. 6 years ago
David Wilson c1d73e1f4f issue #535: parent: add create_socketpair(size=..) parameter. 6 years ago
David Wilson 63f4864b21 issue #535: introduce mitogen.os_fork module and Corker class. 6 years ago
David Wilson c119ae8f66 issue #535: docs: update Changelog 6 years ago
David Wilson 514d35fd10 issue #535: service: support Pool.defer() like Broker.defer() 6 years ago
David Wilson eb9ec26622 issue #535: core: unicode.encode() may take importer lock on 2.x
Found on Python 2.4, where import happens immediately following connect.

- Main thread executes import statement, triggers request to parent
- Broker thread attempts to deliver request via Router
- Router discovers parent has disconnected, prepares a dead message
- .dead() calls unicode.encode() to format reason string
- .encode() attemptsto import a codec module
- deadlock

----

(gdb) pystack
/usr/local/python2.4.6/lib/python2.4/encodings/__init__.py (69): search_function
<stdin> (733): dead
<stdin> (2717): _maybe_send_dead
<stdin> (2724): _invoke
<stdin> (2749): _async_route
<stdin> (1635): _receive_one
<stdin> (1603): _internal_receive
<stdin> (1613): on_receive
<stdin> (2931): _call
<stdin> (2942): _loop_once
<stdin> (2988): _do_broker_main
<stdin> (545): _profile_hook
<stdin> (3007): _broker_main
/usr/local/python2.4.6/lib/python2.4/threading.py (420): run
/usr/local/python2.4.6/lib/python2.4/threading.py (424): __bootstrap
6 years ago
David Wilson 72862f0bb9 issue #535: docs: fix up Select doc 6 years ago
David Wilson 28aa8b3b27 issue #535: docs: update Changelog. 6 years ago
David Wilson b3f592acee issue #535: core/select: support selecting from Latches. 6 years ago
David Wilson 1397c0eec3 Merge remote-tracking branch 'origin/dmw'
* origin/dmw:
  core: increase cookie field lengths to 64-bit; closes #545.
  tests: ensure serialization restrictions are in effect
  tests/bench: set process affinity in throughput.py.
  docs: update copyright year.
  docs: update Changelog.
  core: Make Latch.put(obj=) optional.
6 years ago
David Wilson 7d0480e8bd core: increase cookie field lengths to 64-bit; closes #545. 6 years ago
David Wilson e517810e5a tests: ensure serialization restrictions are in effect 6 years ago
David Wilson ffdf31edd7 tests/bench: set process affinity in throughput.py. 6 years ago
David Wilson c0db283ac7 docs: update copyright year. 6 years ago
David Wilson 14e6c6e49e docs: update Changelog. 6 years ago
David Wilson ca63c26e01 core: Make Latch.put(obj=) optional. 6 years ago
David Wilson 06c116257f Merge remote-tracking branch 'origin/dmw'
* origin/dmw:
  docs: change 'unreleased' Changelog format and add a hint.
6 years ago
David Wilson d865fb797d docs: change 'unreleased' Changelog format and add a hint. 6 years ago
David Wilson 97399477a5 Merge remote-tracking branch 'origin/azure3'
* origin/azure3:
  docs: update Changelog; closes #542.
  issue #542: return of select poller, new selection logic
  issue #542: .ci: move some tests to Azure and enable Mac job.
  ansible: create stub __init__.py for sdist.
6 years ago
David Wilson 2cde51ea63 docs: update Changelog; closes #542. 6 years ago
David Wilson 9bcd2ec56c issue #542: return of select poller, new selection logic 6 years ago
David Wilson 0aa4c9d8fc issue #542: .ci: move some tests to Azure and enable Mac job. 6 years ago
David Wilson 458a4faa97 ansible: create stub __init__.py for sdist.
This went into 0.2.5 sdist tarball but it's not checked in.
6 years ago
David Wilson 2654ab470d Merge remote-tracking branch 'origin/dmw'
* origin/dmw:
  issue #537: disable just the trivial LinuxPolicyTest on Travis.
  docs: update Changelog; closes #537.
  ansible: refactor affinity class and add abstract tests.
  Bump version for release.
6 years ago
David Wilson 5ed445c4aa issue #537: disable just the trivial LinuxPolicyTest on Travis. 6 years ago
David Wilson 45f915f392 docs: update Changelog; closes #537. 6 years ago
David Wilson 8f9c67daf1 ansible: refactor affinity class and add abstract tests. 6 years ago
David Wilson e010667230 Bump version for release. 6 years ago
David Wilson 24dd64a998 Merge remote-tracking branch 'origin/dmw'
* origin/dmw:
  docs: update Changelog.
  core: serialize calls to _service_stub_main().
  docs: update Changelog; closes #532.
  issue #532: PushFileService race.
  docs: more concise Changelog.
  issue #541: changelog typos.
  ansible: quiesce boto logger; closes #541.
6 years ago
David Wilson 6d7bd7e3b9 docs: update Changelog. 6 years ago
David Wilson 2a8567b432 core: serialize calls to _service_stub_main().
See comment.
6 years ago
David Wilson cf8ecf19b7 docs: update Changelog; closes #532. 6 years ago
David Wilson d4c0250083 issue #532: PushFileService race.
There has always been a race in PushFileService since given a parent
asked to forward modules to two children via some intermediary:

    interm = router.local()
    c1 = router.local(via=interm)
    c2 = router.local(via=interm)

    service.propagate_to(c1, 'foo/bar.py')
    service.propagate_to(c2, 'foo/bar.py')

Two calls will be emitted to 'interm':

    PushFileService.store_and_forward(c1, 'foo/bar.py', [blob])
    PushFileService.store(c2, 'foo/bar.py')

Which will be processed in-order up to the point where service pool
threads in 'interm' are woken to process the message.

While it is guaranteed store_and_forward() will be processed first, no
guarantee existed that its assigned pool thread would wake and take
_lock first, thus it was possible for forward() to win the race, and for
a request to arrive to forward a file that had not been placed in local
cache yet.

Here we get rid of SerializedInvoker entirely, as it is partially to
blame for hiding the race: SerializedInvoker can only ensure no two
messages are processed simultaneously, it cannot ensure the messages are
processed in their intended order.

Instead, teach forward() that it may be called before
store_and_forward(), and if that is the case, to place the forward
request on to _waiters alongside any local threads blocked in get().
6 years ago
David Wilson 1b4eb06f72 docs: more concise Changelog. 6 years ago
David Wilson b317b66317 issue #541: changelog typos. 6 years ago
David Wilson 0f30808234 ansible: quiesce boto logger; closes #541. 6 years ago
David Wilson 6727402526 Merge remote-tracking branch 'origin/dmw'
* origin/dmw:
  docs: update Changelog.
  tests/ansible: Spec.port() test & mitogen_via= fix.
  Update copyright year everywhere.
  tests/ansible: Spec.become_pass() test.
  docs: remove top "Table of Contents" link
  docs: remove a little more top margin wastage
  tests/ansible: Spec.become_user() test.
  docs: update Changelog; closes #539.
  issue #539: disable logger propagation.
6 years ago
David Wilson c39ee9b7fe docs: update Changelog. 6 years ago
David Wilson 7fd0d34910 tests/ansible: Spec.port() test & mitogen_via= fix.
ansible_ssh_port was not respected.
6 years ago
David Wilson 1f77d24bec Update copyright year everywhere. 6 years ago