Commit Graph

2730 Commits (e87e41e69ebbe214994dddd2798e0b7d52203964)
 

Author SHA1 Message Date
David Wilson de65790a74 docs: note fd usage has halved 5 years ago
David Wilson ded00d25c9 docs: more stream-refactor work 5 years ago
David Wilson 4cd32cbcba docs: update Changelog for stream-refactor. 5 years ago
David Wilson d35bca3f15 docs: Add lineinfile bug to changelog. 5 years ago
David Wilson 1fca0b7a94 [linear2] fix MuxProcess test fixture and some merge fallout 5 years ago
David Wilson e93762b3db service: avoid taking another lock in the usual case 5 years ago
David Wilson 50bfe4c746 service: don't acquire lock when pool already initialized 5 years ago
David Wilson f4709b1dc2 profiler: marginal improvements 5 years ago
David Wilson 3b585b841e core: ensure 'exit' signal fires even on Broker crash. 5 years ago
David Wilson d6faff06c1 core: wake Waker outside of lock.
Given:

- Broker asleep in poll()
- thread B calling Latch.put()

Previously,

- B takes lock,
- B wakes socket by dropping GIL and writing to it
- Broker wakes from poll(), acquires GIL only to find Latch._lock is held
- Broker drops GIL, sleeps on futex() for _lock
- B wakes, acquires GIL, releases _lock
- Broker wakes from futex(), acquires lock

Now,

- B takes lock, updates state, releases lock
- B wakes socket by droppping GIL and writing to it
- Broker wakes from poll(), acquires GIL and _lock
- Everyone lives happily ever after.
5 years ago
David Wilson 807cbef9ca core: wake Latch outside of lock.
Given:

- thread A asleep in Latch._get_sleep()
- thread B calling Latch.put()

Previously,

- B takes lock,
- B wakes socket by dropping GIL and writing to it
- A wakes from poll(), acquires GIL only to find Latch._lock is held
- A drops GIL, sleeps on futex() for _lock
- B wakes, acquires GIL, releases _lock
- A wakes from futex(), acquires lock

Now,

- B takes lock, updates state, releases lock
- B wakes socket by droppping GIL and writing to it
- A wakes from poll(), acquires GIL and _lock
- Everyone lives happily ever after.
5 years ago
David Wilson 7e51a93231 core: remove old blocking call guard, it's in the wrong place
It should have been in Receiver.get(). Placing it here prevents
*_async() method calls from broker thread.
5 years ago
David Wilson 0f63ca4c68 Make setting affinity optional. 5 years ago
David Wilson 9035884c77 ansible: abstract worker process model.
Move all details of broker/router setup out of connection.py, instead
deferring it to a WorkerModel class exported by process.py via
get_worker_model(). The running strategy can override the configured
worker model via _get_worker_model().

ClassicWorkerModel is installed by default, which implements the
extension's existing process model.

Add optional support for the third party setproctitle module, so
children have pretty names in ps output.

Add optional support for per-CPU multiplexers to classic runs.
5 years ago
David Wilson 6b8a7cbcc4 [stream-refactor] parent: fix crash on graceful shutdown
Now it's possible for stream.protocol to not refer to MitogenProtocol,
move the signal handler to a MitogenProtocol subclass instead.

Fixes a crash where CTRL+C during child bootstrap would print
AttributeError.
5 years ago
David Wilson 2ccdeeeb87 parent: tidy up create_socketpair() 5 years ago
David Wilson c0513425ca core: more concise Side.repr. 5 years ago
David Wilson a5619a62bf Merge remote-tracking branch 'origin/stream-refactor'
* origin/stream-refactor:
  [stream-refactor] Py3.x test fixes
  [stream-refactor] mark py24 as allow-fail
  [stream-refactor] Debian Docker container image initctl
  [stream-refactor] replace cutpaste with Stream.accept() in mitogen.unix
  [stream-refactor] fix flake8 errors
  [stream-refactor] fix testlib assertion format string
  [stream-refactor] make mitogen-fuse work on Linux
  [stream-refactor] repair preamble_size.py again
  [stream-refactor] don't abort Connection until all buffers are empty
  Normalize docstring formatting
  [stream-refactor] fix LogHandler.uncork() race
  [stream-refactor] BufferedWriter must disconenct Stream, not Protocol
  [stream-refactor] statically link doas binary using musl
  [stream-refactor] stop writing to /tmp/foo in fd_check.py.
  [stream-refactor] yet another 2.4 issue in create_child_test
  [stream-refactor] fix Py2.4 failure by implementing missing Timer method
  [stream-refactor] allow up to 30 seconds to connect in unix_test
  [stream-refactor] mark setns module as requiring Python >2.4
  [stream-refactor] another 2.4 fix for create_child_test
  .travis.yml: Add reverse shell spawn for Travis too
  core: better Side attribute docstrings
  [stream-refactor] remove one more getuser() usage
  [stream-refactor] allow doas_test to succeed on CentOS
  Pin idna==2.7 when running on Python<2.7.
  [stream-refactor] Py2.4 compat fix for iter_split_test.
  [stream-refactor] add descriptive task names to _container_prep
  [stream-refactor] 3.x socket.send() requires bytes
  [stream-refactor] fix 2.4 syntax error.
  [stream-refactor] avoid os.wait3() for Py2.4.
  Allow specifying -vvv to debops_tests.
  [stream-refactor] send MITO002 earlier
  module_finder: pass raw file to compile()
  [stream-refactor] merge stdout+stderr when reporting EofError
  [stream-refactor] fix crash in detach() / during async/multiple_items_loop.yml
  [stream-refactor] fix crash in runner/forking_active.yml
  [stream-refactor] replace old detach_popen() reference
  ansible: fixturize creation of MuxProcess
  unix: ensure mitogen.context_id is reset when client disconnects
  [stream-refactor] make syntax 2.4 compatible
  [stream-refactor] make trusty our Travis dist.
  [stream-refactor] fix su_test failure (issue #363)
  [stream-refactor] more readable log string format
  [stream-refactor] dont doubly log last partial line
  [stream-refactor] import fd_check.py used by create_child_test
  [stream-refactor] port mitogen.buildah, added to master since work began
  [stream-refactor] fix unix.Listener construction
  [stream-refactor] fix crash when no stderr present.
  [stream-refactor] fix Process constructor invocation
  Add tests/ansible/.*.pid to gitignore (for ansible_mitogen/process.py)
  Add extra/ to gitignore
  import release-notes script.
  [stream-refactor] repaired rest of create_child_test.
  [stream-refactor] rename Process attrs, fix up more create_child_test
  [stream-refactor] import incomplete create_child_test
  issue #482: tests: check for zombie process after test.
  issue #363: add test.
  tests: clean up old-style SSH exception catch
  issue #271: add mitogen__permdenied user to Docker image.
  ssh: fix issue #271 regression due to refactor, add test.
  Refactor Stream, introduce quasi-asynchronous connect, much more
  core: teach iter_split() to break on callback returning False.
  issue #507: log fatal errors to syslog.
  testlib: have LogCapturer.raw() return unicode on 2.x.
  core/master: docstring, repr, and debug log message cleanups
  parent: remove unused Timer parameter.
  tests: jail_test fixes.
  parent: docstring improvements, cfmakeraw() regression.
  core: introduce Protocol, DelimitedProtocol and BufferedWriter.
  core: introduce mitogen.core.pipe()
  tests/bench: import ssh-roundtrip.py.
  tests: note location of related tests.
  tests: add real test for doas.
  tests: install OpenBSD doas port in Debian image.
  tests: add setns_test that works if password localhost sudo works.
  Import minimal jail_test.
  core: move message encoding to Message.pack(), add+refactor tests.
  master: expect forwarded logs to be in UTF-8.
  tests: add some UTF-8 to ssh_login_banner to encourage breakage.
  core: bootstrap FD management improvements
  core: pending timers should keep broker alive.
  core: more succinct iter_split().
  core: replace UTF8_CODEC with encodings.utf_8.encode() function.
  docs: remove bytearray from supported types list.
  core: docstring style cleanups, dead code.
  testlib: disable lsof warnings due to Docker crap
  parent: discard cancelled events in TimerList.get_timeout().
  core: split out iter_split() for use in parent.py.
  parent: various style cleanups, remove unused function.
  issue #170: add TimerList docstrings.
  core: eliminate some quadratric behaviour from IoLogger
  issue #170: update Changelog; closes #170.
  issue #170: add timers to internals.rst.
  issue #170: implement timers.
5 years ago
David Wilson 6da991fae9 [stream-refactor] Py3.x test fixes 5 years ago
David Wilson 14f8f00d4d [stream-refactor] mark py24 as allow-fail
This needs a day or two's worth of soaking to fix all the remaining nits
5 years ago
David Wilson 87440ec6f7 [stream-refactor] Debian Docker container image initctl 5 years ago
David Wilson f45d8eae66 [stream-refactor] replace cutpaste with Stream.accept() in mitogen.unix 5 years ago
David Wilson 1843f183a3 [stream-refactor] fix flake8 errors 5 years ago
David Wilson 0e6de532de [stream-refactor] fix testlib assertion format string 5 years ago
David Wilson 6a106f03ff [stream-refactor] make mitogen-fuse work on Linux 5 years ago
David Wilson fc57c1d1a0 [stream-refactor] repair preamble_size.py again 5 years ago
David Wilson c02358698b [stream-refactor] don't abort Connection until all buffers are empty 5 years ago
David Wilson 93342ba60c Normalize docstring formatting 5 years ago
David Wilson 4e6aadc40a [stream-refactor] fix LogHandler.uncork() race
During early initialization under hackbench, it is possible for Broker
to be in LogHandler._send() while the main thread has already destroyed
_buffer. So we must synchronize them, but only while the handler is
corked.
5 years ago
David Wilson 90c989ee59 [stream-refactor] BufferedWriter must disconenct Stream, not Protocol
Fix a race where if Stream.on_receive() detects disconnect, it calls
Stream.on_disconnect(), which fires Stream 'disconnect' event, whereas
if BufferedWriter.on_transmit() detects disconnect, it called
Protocol.on_disconnect(), which did not fire the Stream 'disconnect'
event.

Since mitogen.parent listens on Stream's 'disconnect' event to reap
children, this was causing a very difficult to trigger test failure.

Triggered after <1000 runs on a Xeon E5530 with hyperthreading using
hackbench running at the same priority:

    $ hackbench -s 1048576 -l 100000000000 -g 4
5 years ago
David Wilson 5f7ab220cb [stream-refactor] statically link doas binary using musl
So it can run on CentOS 5
5 years ago
David Wilson 462c4ff59f [stream-refactor] stop writing to /tmp/foo in fd_check.py. 5 years ago
David Wilson cd0a557602 [stream-refactor] yet another 2.4 issue in create_child_test 5 years ago
David Wilson 65e31f63fe [stream-refactor] fix Py2.4 failure by implementing missing Timer method 5 years ago
David Wilson 54987100b2 [stream-refactor] allow up to 30 seconds to connect in unix_test
It reliably fails when running on a (intentionally) heavily loaded
machine
5 years ago
David Wilson db9066fbfb [stream-refactor] mark setns module as requiring Python >2.4 5 years ago
David Wilson 856dfcebcd [stream-refactor] another 2.4 fix for create_child_test 5 years ago
David Wilson 054643783c .travis.yml: Add reverse shell spawn for Travis too 5 years ago
David Wilson 11ae6f3873 core: better Side attribute docstrings 5 years ago
David Wilson bed5931194 [stream-refactor] remove one more getuser() usage 5 years ago
David Wilson 2f950b3bda [stream-refactor] allow doas_test to succeed on CentOS
Unlike on Debian, some environment variables that tickle
getpass.getuser() are being inherited. So use getuid() instead.

Also install the doas binary on CentOS. CI was changed (I believe) to
shrink the configuration matrix, and now these tests run on CentOS too.
5 years ago
David Wilson 2ba3973bc5 Pin idna==2.7 when running on Python<2.7. 5 years ago
David Wilson 07f3179e58 [stream-refactor] Py2.4 compat fix for iter_split_test. 5 years ago
David Wilson f0065d76d8 [stream-refactor] add descriptive task names to _container_prep 5 years ago
David Wilson fdf3484a2a [stream-refactor] 3.x socket.send() requires bytes 5 years ago
David Wilson c09bbdc2f9 [stream-refactor] fix 2.4 syntax error. 5 years ago
David Wilson bd80d4b0af [stream-refactor] avoid os.wait3() for Py2.4. 5 years ago
David Wilson 321dac3046 Allow specifying -vvv to debops_tests. 5 years ago
David Wilson b1379e6f45 [stream-refactor] send MITO002 earlier
Prevents 2.4 bootstrap from attempting to fetch os_fork too early.

Connection(None).connect(): pid:25098 stdin:81 stdout:81 stderr:79
ssh.localhost:2201: (partial): mitogen__has_sudo_nopw@localhost's password:
ssh.localhost:2201: (password prompt): mitogen__has_sudo_nopw@localhost's password:
ssh.localhost:2201: (unrecognized): mitogen__has_sudo_nopw@localhost's password:
BootstrapProtocol(ssh.localhost:2201): first stage started succcessfully
BootstrapProtocol(ssh.localhost:2201): first stage received bootstrap
ssh.localhost:2201: (partial): MIdmitogen.os_fork
ssh.localhost:2201: (unrecognized partial): MIdmitogen.os_fork
ssh.localhost:2201: failing connection due to TimeoutError(u'Failed to setup connection after 10.00 seconds',)
5 years ago
David Wilson 402dba4197 module_finder: pass raw file to compile()
Newer Ansibles have e.g. UTF-8 present in apt.py.
5 years ago