Commit Graph

843 Commits (9f94fb78c858cda6defb3fec1d1e72f108610873)
 

Author SHA1 Message Date
David Wilson 9f94fb78c8 issue #159: make LRU size configurable. 7 years ago
David Wilson a96969ee29 docs: concision 7 years ago
David Wilson cc980569a3 issue #159: initial context LRU implementation
Now Connection.close() *must* be called in the worker, to ensure the
reference count for a context drops correctly.

Remove 'discriminator' for now, I'm not using it for testing any more
and it complicated this code.

This code is a car crash, it needs rewritten again. Ideally some/most of
this behaviour could live on services.DeduplicatingService somehow, but
I couldn't come up with a sensible design.
7 years ago
David Wilson 6394226722 issue #159: verify no more than 20 tasks exist on the remote 7 years ago
David Wilson 633cd888a0 issue #182: test for previous commit. 7 years ago
David Wilson e119058d9b issue #182: ensure connection is reset during with_items.
Elements of a with_items loop reuse one WorkerProcess to execute every
iteration, requiring us to reset Connection's idea of the connection on
each iteration, otherwise the tasks will erroneously execute in the
wrong context.
7 years ago
David Wilson 9acca37050 issue #159: lots more test users. 7 years ago
David Wilson 4c8ec131f9 issue #16: initial smorgasbord of 3.x fixes. 7 years ago
David Wilson c4bef102fe issue #16: Python 2.4-3.x compatible exception handling. 7 years ago
David Wilson 8889708f24 core: blacklist Jython org.* by default too.
1 silly roundtrip.
7 years ago
David Wilson e5b784ed32 parent: reduce cutpaste
Unclear whether exec() is better or worse than __import__(), but at
least the semantics are correct.
7 years ago
David Wilson 38c0ad1eea core: don't deregister Router handles until Broker exit.
Lots of "invalid handle: ..., 102" messages started appearing during
exit recently because ordering changed slightly, and local handles were
sent _DEAD even though the broker loop was still progressing through
shutdown.

The "shutdown" event is too early to close handles: it is the start of
the grace period where streams and downstream contexts can finish up any
work and deliver buffered data, including FORWARD_LOG messages that
haven't arrived yet.

So instead,

- move the _DEAD logic to the "exit" event,
- get rid of Context.on_shutdown() entirely, it's been unused for over
  a month,
- get rid of the "crash" event, since it always fires prior to "exit",
  and its only use was to send _DEAD to local handles, which now happens
  during exit anyway.
7 years ago
David Wilson 813d139d48 Import v2.7.11 tokenize.py for use on older Pythons; closes #189.
It's worth note that 2.7.10 shipped with Sierra, managed to not notice
this due to using a Homebrew 2.7.14.
7 years ago
David Wilson 0c803141a7 tests: fix filename. 7 years ago
David Wilson ec720eab1a Merge commit 'refs/pull/178/head' of github.com:dw/mitogen 7 years ago
David Wilson 0e839fe63c docs: override insane list of fallback fonts
WHY ARE THE DOCS RENDERING IN ITALIC HOW LONG HAS IT BEEN LIKE THIS
ARGAHGR
7 years ago
David Wilson 3682ac6e29 fork: ensure importer handle is installed on the new router. 7 years ago
David Wilson e9f94e1bbb docs: tidy up big list of bullets. 7 years ago
David Wilson 20de3489de ansible: tidy up annoying syntax 7 years ago
David Wilson c14f6c98d1 ansible: import osx_setup.yml. 7 years ago
David Wilson 98ee3e177a ansible: tests for sudo behaviour; closes #143. 7 years ago
David Wilson 4a61527774 ansible: more docstrings and call_async() 7 years ago
David Wilson 6a4ce84c6b ansible: more docstring fixes. 7 years ago
David Wilson 135b3738ba ansible: don't wait on FileService response
Any (unlikely) execption will show up in the debug logs.
7 years ago
David Wilson 70a735f23a ansible: tidy up service.py docstrings. 7 years ago
David Wilson 79b75aabae ansilbe: remove unused class variables 7 years ago
David Wilson 43ecbe350c docs: rearrange more ansible risks 7 years ago
David Wilson 19a418cc6f docs: document mitogen_task_isolation. 7 years ago
David Wilson ba74f8f743 docs: remove Ansible risk 7 years ago
David Wilson 3613162bc0 ansible: enable forking when requested and for async jobs.
Closes #105.
References #155.

mitogen/service.py:
    Refactor services to support individually exposed methods with
    different security policies for each method.

    - @mitogen.service.expose() to expose a method and set its policy
    - @mitogen.service.arg_spec() to validate input.
    - Require basic service message format to be a tuple of
      `(method, kwargs)`, where kwargs is always a dict.
    - Update DeduplicatingService to match the new scheme.

ansible_mitogen/connection.py:
    - Rename 'method' to 'method_name' to disambiguate it from the
      service.call()'s method= argument.

ansible_mitogen/planner.py:
    - Generate an ID for every job, sync or not, and fetch job results
      from JobResultService rather than via the initiating function
      call's return value.
    - Planner subclasses now get to select whether their Runner should
      run in a forked process. The base implementation requests this if
      the 'mitogen_isolation_mode=fork' task variable is present.

ansible_mitogen/runner.py:
    Teach runners to deliver their result via JobResultService executing
    in their indirect parent mux process.

ansible_mitogen/plugins/actions/mitogen_async_status.py:
    Split the implementation up into methods, and more compatibly
    emulate Ansible's existing output.

ansible_mitogen/process.py:
    Mux processes now host JobResultService.

ansible_mitogen/services.py:
    Update existing services to the new mitogen.service scheme, and
    implement JobResultService:

    * listen() method for synchronous jobs. planner.invoke() registers a
      Sender with the service prior to invoking the job, then sleeps
      waiting for the service to write the job result to the
      corresponding Receiver.

    * Non-blocking get() method for implementing mitogen_async_status
      action.

    * Child-accessible push() method for delivering task results.

ansible_mitogen/target.py:
    New helpers for spawning a virginal subprocess on startup, from
    which asynchronous and mitogen_task_isolation=fork jobs are forked.
    Necessary to avoid a task inheriting potentially
    polluted/monkey-patched parent environment, since remaining jobs
    continue to run in the original child process.

docs/ansible.rst:
    Add/merge/remove some behaviours/risks.

tests/ansible/integration:
    New tests for forking/async.
7 years ago
David Wilson 71057c78f9 ansible: rename helpers.py to target.py, to reflect its purpose 7 years ago
Alex Willmer 1bc32adad4 Issue #160: Cache minimize_source()
Before

```
python tests/bench/local.py
0 120.245933533
1 119.300842285
2 118.355035782
3 123.839855194
```

After

```
python tests/bench/local.py
0 66.3640499115
1 65.9508705139
2 72.4799633026
3 65.7958984375
```
7 years ago
Alex Willmer 0b46b90c62 Issue #160: Apply 2.x workarounds to lru_cache backport
all() was introduced in Python 2.5
with: blocks arrived in Python 2.6
nonlocal and default utf-8 source encoding were added in Python 3.x
7 years ago
Alex Willmer fc0fb732a3 Issue #160: Vendorize namedtuple and lru_cache
namedtuple from
https://hg.python.org/cpython/file/3f79ecab5e52/Lib/collections.py
(Python 2.6)

lru_cache from
937ac1fe06/Lib/functools.py
Python 3.5
7 years ago
Alex Willmer dc3f5730a2 Merge branch 'master' into eye-of-the-token-its-the-thrill-of-the-light 7 years ago
David Wilson d9e4781d11 ansible: rename helpers.py to target.py, to reflect its purpose 7 years ago
David Wilson a643f13ebe issue #106: docs: tidyup. 7 years ago
David Wilson 432ebbca89 issue #106: docs: initial docs for how modules execute. 7 years ago
David Wilson b595314619 docs: fix intensely annoying _prefix, 2 years later. 7 years ago
David Wilson b247c320d2 issue #164: rename tests for clarity 7 years ago
David Wilson f655be1455 ssh: fix password prompt check when running with -vvv
Can only happen by hacking -vvv into ssh.py at present, but that will
probably be exposed via a constructor parameter in future.
7 years ago
David Wilson bc4a6b39bf issue #164: teach debops_tests.sh to use SSH
Login with a non-privileged account over SSH rather than just jumping
straight in as root via Docker.
7 years ago
David Wilson cd098ef158 issue #183: re-raise StreamError in calling context.
This allows catching just StreamError regardless of via=None or
via=<context>. Deserves a more general solution, but it's easy to fix up
later.
7 years ago
David Wilson 998a1209cc issue #183: make PasswordErrors subclass of StreamError. 7 years ago
David Wilson 35fdd97f9a issue #164: utility to print Docker hostname for use from shell scripts. 7 years ago
David Wilson 7fd88868a6 ansible: raise AnsibleConnectionFailure on connection failure; closes #183
Before:

    $ ANSIBLE_STRATEGY=mitogen ansible -i derp, derp -m setup
    An exception occurred during task execution. To see the full traceback, use -vvv. The error was:     (''.join(bits)[-300:],)
    derp | FAILED! => {
        "msg": "Unexpected failure during module execution.",
        "stdout": ""
    }

After:

    $ ANSIBLE_STRATEGY=mitogen ansible -i derp, derp -m setup
    derp | UNREACHABLE! => {
        "changed": false,
        "msg": "EOF on stream; last 300 bytes received: 'ssh: Could not resolve hostname derp: nodename nor servname provided, or not known\\r\\n'",
        "unreachable": true
    }
7 years ago
David Wilson b9d4ec57b3 issue #164: some more ActionMixin tests. 7 years ago
David Wilson 4a823c7a27 issue #164: missing cast() for _remote_file_exists(). 7 years ago
David Wilson 29288b236b issue #164: import run_ansible_playbook.sh. 7 years ago
David Wilson db894478f8 issue #164: make become_flags work without FOO=2 env var. 7 years ago