Commit Graph

232 Commits (3e2abf36d5a31fd88b0b1846d10d10d5e06db0c9)

Author SHA1 Message Date
David Wilson 6aad45748b docs: typo 7 years ago
David Wilson 9f94fb78c8 issue #159: make LRU size configurable. 7 years ago
David Wilson a96969ee29 docs: concision 7 years ago
David Wilson cc980569a3 issue #159: initial context LRU implementation
Now Connection.close() *must* be called in the worker, to ensure the
reference count for a context drops correctly.

Remove 'discriminator' for now, I'm not using it for testing any more
and it complicated this code.

This code is a car crash, it needs rewritten again. Ideally some/most of
this behaviour could live on services.DeduplicatingService somehow, but
I couldn't come up with a sensible design.
7 years ago
David Wilson 38c0ad1eea core: don't deregister Router handles until Broker exit.
Lots of "invalid handle: ..., 102" messages started appearing during
exit recently because ordering changed slightly, and local handles were
sent _DEAD even though the broker loop was still progressing through
shutdown.

The "shutdown" event is too early to close handles: it is the start of
the grace period where streams and downstream contexts can finish up any
work and deliver buffered data, including FORWARD_LOG messages that
haven't arrived yet.

So instead,

- move the _DEAD logic to the "exit" event,
- get rid of Context.on_shutdown() entirely, it's been unused for over
  a month,
- get rid of the "crash" event, since it always fires prior to "exit",
  and its only use was to send _DEAD to local handles, which now happens
  during exit anyway.
7 years ago
David Wilson 0e839fe63c docs: override insane list of fallback fonts
WHY ARE THE DOCS RENDERING IN ITALIC HOW LONG HAS IT BEEN LIKE THIS
ARGAHGR
7 years ago
David Wilson e9f94e1bbb docs: tidy up big list of bullets. 7 years ago
David Wilson 43ecbe350c docs: rearrange more ansible risks 7 years ago
David Wilson 19a418cc6f docs: document mitogen_task_isolation. 7 years ago
David Wilson ba74f8f743 docs: remove Ansible risk 7 years ago
David Wilson 3613162bc0 ansible: enable forking when requested and for async jobs.
Closes #105.
References #155.

mitogen/service.py:
    Refactor services to support individually exposed methods with
    different security policies for each method.

    - @mitogen.service.expose() to expose a method and set its policy
    - @mitogen.service.arg_spec() to validate input.
    - Require basic service message format to be a tuple of
      `(method, kwargs)`, where kwargs is always a dict.
    - Update DeduplicatingService to match the new scheme.

ansible_mitogen/connection.py:
    - Rename 'method' to 'method_name' to disambiguate it from the
      service.call()'s method= argument.

ansible_mitogen/planner.py:
    - Generate an ID for every job, sync or not, and fetch job results
      from JobResultService rather than via the initiating function
      call's return value.
    - Planner subclasses now get to select whether their Runner should
      run in a forked process. The base implementation requests this if
      the 'mitogen_isolation_mode=fork' task variable is present.

ansible_mitogen/runner.py:
    Teach runners to deliver their result via JobResultService executing
    in their indirect parent mux process.

ansible_mitogen/plugins/actions/mitogen_async_status.py:
    Split the implementation up into methods, and more compatibly
    emulate Ansible's existing output.

ansible_mitogen/process.py:
    Mux processes now host JobResultService.

ansible_mitogen/services.py:
    Update existing services to the new mitogen.service scheme, and
    implement JobResultService:

    * listen() method for synchronous jobs. planner.invoke() registers a
      Sender with the service prior to invoking the job, then sleeps
      waiting for the service to write the job result to the
      corresponding Receiver.

    * Non-blocking get() method for implementing mitogen_async_status
      action.

    * Child-accessible push() method for delivering task results.

ansible_mitogen/target.py:
    New helpers for spawning a virginal subprocess on startup, from
    which asynchronous and mitogen_task_isolation=fork jobs are forked.
    Necessary to avoid a task inheriting potentially
    polluted/monkey-patched parent environment, since remaining jobs
    continue to run in the original child process.

docs/ansible.rst:
    Add/merge/remove some behaviours/risks.

tests/ansible/integration:
    New tests for forking/async.
7 years ago
David Wilson a643f13ebe issue #106: docs: tidyup. 7 years ago
David Wilson 432ebbca89 issue #106: docs: initial docs for how modules execute. 7 years ago
David Wilson b595314619 docs: fix intensely annoying _prefix, 2 years later. 7 years ago
David Wilson 6aeb4e9f05 issue #164: precisely emulate Ansible's stdio behaviour.
* Use identical logic to select when stdout/stderr are merged, so
  'stdout', 'stdout_lines', 'stderr', 'stderr_lines' contain the same
  output before/after the extension.

* When stdout/stderr are merged, synthesize carriage returns just like
  the TTY layer.

* Mimic the SSH connection multiplexing message on stderr. Not really
  for user code, but so compare_output_test.sh needs fewer fixups.
7 years ago
David Wilson e0381606af Ensure remote_tmp is respected everywhere.
Logic is still somewhat different from Ansible: we don't have to care
about sudo/non-sudo cases, etc.
7 years ago
David Wilson 38311336e1 docs: link to Ansible video demo 7 years ago
David Wilson 380ef7376d ansible: Add support for free strategy. 7 years ago
David Wilson 8425b196e7 docs: merge duplicate risks 7 years ago
David Wilson 6dcefd631a issue #106: docs: remove built-in only limitation :> 7 years ago
David Wilson 6958b8ff09 docs: More getting started. 7 years ago
David Wilson 8674ec42dd docs: add new risk 7 years ago
David Wilson bbb0f1bbd8 issue #155: fix double-fork behaviour and test it this time. 7 years ago
David Wilson 6670cba41c Introduce handler policy functions; closes #138.
Now you can specify a function to add_handler() that authenticates the
message header, with has_parent_authority() and is_immediate_child()
built in.
7 years ago
David Wilson 80a97fbc9b core: Rename Sender.put() to Sender.send().
Been annoying me for months.
7 years ago
David Wilson b48d63f33b docs: add to_sender() and update serialization notes 7 years ago
David Wilson 8676c40674 core: make _start_transmit / _stop_transmit async-only
For now at least, these APIs are always used in an asynchronous context,
so stop using the defer mechanism.
7 years ago
David Wilson c33bc22f21 docs: fix reading time tracking :( 7 years ago
David Wilson 4903052f42 fork: reseed Python/SSL PRNGs
Mac's SSL seems to have a pthread_atfork handler or similar that does
this for us, no clue if Linux is the same.
7 years ago
David Wilson 7f4368db87 docs: pad out signal handlers note 7 years ago
David Wilson d68833757b docs: slightly bikeshed last testimonial 7 years ago
dw d23a6174a2
Merge pull request #167 from danielcompton/patch-1
Add testimonial from Clojars
7 years ago
David Wilson 0dbb63ae3d docs: typo 7 years ago
David Wilson b254373cd7 docs: add local task serialization risk 7 years ago
Daniel Compton 871b853b59
Add testimonial from Clojars 7 years ago
David Wilson 2813d1a968 docs: update 7 years ago
David Wilson 5579ac936b issue #155: docs: more fork() liability warnings 7 years ago
David Wilson f457f54758 docs: fix formatting 7 years ago
David Wilson d370290687 docs: one more warning 7 years ago
David Wilson 03e08e25fd docs: put huge liability notice in fork() docs. 7 years ago
David Wilson 48351a1889 issue #155: parent: support Context.shutdown(), reap children on exit.
This permits graceful shutdown of individual contexts, without tearing
down everything.

Update mitogen.parent.Stream to also wait for the child to exit, to
prevent the buildup of zombie processes. This introduces a blocking wait
for process exit on the Broker thread, let's see if we can get away with
it. Chances are reasonable that it'll cause needless hangs on heavily
loaded machines.
7 years ago
David Wilson 6a74edce6b issue #155: parent: move master.Context into parent.
The Context and Router APIs for constructing children and making
function calls should be available in every parent context, as user code
wants to have access to the same API.
7 years ago
David Wilson 447353ecb8 docs: ansible.rst: note multi-host perf isn't great right now 7 years ago
David Wilson 20780820a6 docs: typo 7 years ago
David Wilson 75b9e1d71e issue #155: docs: document behaviour of forked children 7 years ago
David Wilson 110fdf24cd docs: add mitogen.fork.Stream to internals.rst 7 years ago
David Wilson 1a8ac9f4d1 issue #155: introduce mitogen.fork / Router.fork() 7 years ago
David Wilson 27175e3126 docs: tidy up signals.rst a little. 7 years ago
David Wilson 732a610246 docs: add beginnings of section on func decorators 7 years ago
David Wilson 2abe87472c issue #162: docs: begin documenting mitogen.service 7 years ago