Commit Graph

32 Commits (master)

Author SHA1 Message Date
David Wilson d6329f3446 Merge devel/290 @ 79b979ec8544ef5d8620c64068d4a42fabf50415 5 years ago
David Wilson d75c9cffc3 docs: add domainrefs plugin to make link aliases everywhere \o/
PATENT PENDING
5 years ago
David Wilson 7379144a12 Stop using mitogen root logger in more modules, remove unused loggers 5 years ago
David Wilson 1fca0b7a94 [linear2] fix MuxProcess test fixture and some merge fallout 5 years ago
David Wilson fdf3484a2a [stream-refactor] 3.x socket.send() requires bytes 5 years ago
David Wilson 4b0870aa6e [stream-refactor] fix Process constructor invocation 5 years ago
David Wilson 8d1b01d8ef Refactor Stream, introduce quasi-asynchronous connect, much more
Split Stream into many, many classes

  * mitogen.parent.Connection: Handles connection setup logic only.
    * Maintain references to stdout and stderr streams.
    * Manages TimerList timer to cancel connection attempt after
      deadline
    * Blocking setup code replaced by async equivalents running on the
      broker

  * mitogen.parent.Options: Tracks connection-specific options. This
    keeps the connection class small, but more importantly, it is
    generic to the future desire to build and execute command lines
    without starting a full connection.

  * mitogen.core.Protocol: Handles program behaviour relating to events
    on a stream. Protocol performs no IO of its own, instead deferring
    it to Stream and Side. This makes testing much easier, and means
    libssh can reimplement Stream and Side to reuse MitogenProtocol

  * mitogen.core.MitogenProtocol: Guts of the old Mitogen stream
    implementtion

  * mitogen.core.BufferedWriter: Guts of the old Mitogen buffered
    transmit implementation, made generic

  * mitogen.core.DelineatedProtocol: Guts of the old IoLogger, knows how
    to split up input and pass it on to a
    on_line_received()/on_partial_line_received() callback.

  * mitogen.parent.BootstrapProtocol: Asynchronous equivalent of the old
    blocking connect code. Waits for various prompts (MITO001 etc) and
    writes the bootstrap using a BufferedWriter. On success, switches
    the stream to MitogenProtocol.

  * mitogen.core.Message: move encoding parts of MitogenProtocol out to
    Message (where it belongs) and write a bunch of new tests for
    pickling.

  * The bizarre Stream.construct() is gone now, Option.__init__ is its
    own constructor. Should fix many LGTM errors.

* Update all connection methods:  Every connection method is updated to
  use async logic, defining protocols as required to handle interactive
  prompts like in SSH or su. Add new real integration tests for at least
  doas and su.

* Eliminate manual fd management: File descriptors are trapped in file
  objects at their point of origin, and Side is updated to use file
  objects rather than raw descriptors. This eliminates a whole class of
  bugs where unrelated FDs could be closed by the wrong component. Now
  an FD's open/closed status is fused to it everywhere in the library.

* Halve file descriptor usage: now FD open/close state is tracked by
  its file object, we don't need to duplicate FDs everywhere so that
  receive/transmit side can be closed independently. Instead both sides
  back on to the same file object. Closes #26, Closes #470.

* Remove most uses of dup/dup2: Closes #256. File descriptors are
  trapped in a common file object and shared among classes. The
  remaining few uses for dup/dup2 are as close to minimal as possible.

* Introduce mitogen.parent.Process: uniform interface for subprocesses
  created either via mitogen.fork or the subprocess module. Remove all
  the crap where we steal a pid from subprocess guts. Now we use
  subprocess to manage its processes as it should be. Closes #169 by
  using the new Timers facility to poll for a slow-to-exit subprocess.

* Fix su password race: Closes #363. DelineatedProtocol naturally
  retries partially received lines, preventing the cause of the original
  race.

* Delete old blocking IO utility functions
  iter_read()/write_all()/discard_until().

Closes #26
Closes #147
Closes #169
Closes #256
Closes #363
Closes #419
Closes #470
5 years ago
David Wilson 2a8567b432 core: serialize calls to _service_stub_main().
See comment.
5 years ago
David Wilson 1f77d24bec Update copyright year everywhere. 5 years ago
David Wilson 0e193c223c issue #508: master: minify all Mitogen/ansible_mitogen sources.
Minify-safe files are marked with a magical "# !mitogen: minify_safe"
comment anywhere in the file, which activates the minifier. The result
is naturally cached by ModuleResponder, therefore lru_cache is gone too.

Given:

    import os, mitogen
    @mitogen.main()
    def main(router):
        c = router.ssh(hostname='k3')
        c.call(os.getpid)
        router.sudo(via=c)

SSH footprint drops from 56.2 KiB to 42.75 KiB (-23.9%)
Ansible "shell: hostname" drops 149.26 KiB to 117.42 KiB (-21.3%)
5 years ago
David Wilson 6a2f88d6a3 issue #477: make mitogen.fork unsupported on Py<2.6. 5 years ago
David Wilson 3767d56187 issue #477: Python <2.5 did not have combined try/finally/except. 5 years ago
David Wilson 1119f24071 issue #414: reinitialize service pool lock on fork 5 years ago
David Wilson 802de6a8d5 issue #406: clean up DiagLogStream handling and connect() failure.
When Stream.connect() fails, have it just use on_disconnect(). Now there
is a single disconnect cleanup path.

Remove cutpasted DiagLogStream setup/destruction, and move it into the
base class (temporarily), and only manage the lifetime of its underlying
FD via Side.close().  This cures another EBADF failure.
6 years ago
David Wilson af2ded663d fork: public on_fork() function.
Generally useful, and needed for ongoing Ansible work.
6 years ago
David Wilson 410016ff47 Initial Python 3.x port work.
* ansible: use unicode_literals everywhere since it only needs to be
  compatible back to 2.6.
* compat/collections.py: delete this entirely and rip out the parts of
  functools that require it.
* Introduce serializable Kwargs dict subclass that translates keys to
  Unicode on instantiation.
* enable_debug_logging() must set _v/_vv globals.
* cStringIO does not exist in 3.x.
* Treat IOLogger and LogForwarder input as latin-1.
* Avoid ResourceWarnings in first stage by explicitly closing fps.
* Fix preamble_size.py syntax errors.
6 years ago
David Wilson 23b2a545cf fork: avoid another logging deadlock at startup.
The very first task /must/ be clearing out logging locks, since
_at_fork() functions call LOG.debug() via Side.close(). Additionally,
the root logger is not included in loggerDict, so we must specify it
explicitly.
6 years ago
David Wilson e118963b30 issue #254: fork: take care not to rely on FD numbers. 6 years ago
David Wilson b0ce6eecd7 fork: support on_start= argument. 6 years ago
David Wilson 00edf0d66d core: have ExternalContext accept a config dict rather than kwargs.
The parameter lists had gotten out of control.
6 years ago
David Wilson 356647bef4 issue #132: initial unidirectional routing mode. 6 years ago
David Wilson 7f1060f54a issue #186: initial version of subtree detachment. 6 years ago
David Wilson e43c6c531b Mostly implement hybrid TTY/socket mode for sudo and SSH.
Presently there is still no mechanism to add :attr:`tty_stream` to the
multiplexer after connection is successful, but for now it's not
expected that anything will be logged to it anyway.

Closes #148.
6 years ago
David Wilson c4bef102fe issue #16: Python 2.4-3.x compatible exception handling. 6 years ago
David Wilson bbb0f1bbd8 issue #155: fix double-fork behaviour and test it this time. 6 years ago
David Wilson 1ff27ada49 Add maximum message size checks. Closes #151. 6 years ago
David Wilson e1af2db4ae issue #155: handle crash in the forking child better.
This code path is probably only necessary during development, but it
prevents tracebacks (etc.) getting written over the Stream socket, which
naturally causes corruption.

Instead keep whatever the parent has for stderr, manually write a
traceback there and hard exit.
6 years ago
David Wilson 4903052f42 fork: reseed Python/SSL PRNGs
Mac's SSL seems to have a pthread_atfork handler or similar that does
this for us, no clue if Linux is the same.
6 years ago
David Wilson 52d980ad58 issue #155: fork: nop out get_boot_command, it's become quite expensive
-16% reduction in fork cost.
6 years ago
David Wilson 2ea65420d0 issue #155: fork.py v2, now with full resource cleanup 6 years ago
David Wilson f6c24ab615 issue #155: don't inherit TLS state in mitogen.fork
This is a partial fix to a general problem: deciding which bits of state
to keep from the parent, and which to clear out. When forking from a
heavily threaded process, there will be 2x$n_threads fds just sitting
around doing nothing, due to Latch use in the parent.

We can't just close all nonstandard fds post-fork, since user code may
be expecting some FDs to be preserved.
6 years ago
David Wilson 1a8ac9f4d1 issue #155: introduce mitogen.fork / Router.fork() 6 years ago