Commit Graph

33 Commits (master)

Author SHA1 Message Date
Alex Willmer d7979c3597 mitogen: Raise TypeError on `mitogen.utils.cast(custom_str)` failures
If casting a string fails then raise a TypeError. This is potentially an API
breaking change; chosen as the lesser evil vs. allowing silent errors.

`cast()` relies on `bytes(obj)` & `str(obj)` returning the respective
supertype. That's no longer the case for `AnsibleUnsafeBytes` &
`AnsibleUnsafeText`; since fixes/mitigations for  CVE-2023-5764.

fixes #1046, refs #977

See also
- https://github.com/advisories/GHSA-7j69-qfc3-2fq9
- https://github.com/ansible/ansible/pull/82293
2 months ago
Alex Willmer e101cc4f44 mitogen.utils: Preserve docstring of functions decorated @with_router
Co-authored-by: Rezart Qelibari <gast-kontakt+mitogen@astzweig.de>

Replaces #837
Fixes #836
2 years ago
Alex Willmer 18c89de5a9 Remove unused module imports 2 years ago
David Wilson 7379144a12 Stop using mitogen root logger in more modules, remove unused loggers 5 years ago
David Wilson 1f77d24bec Update copyright year everywhere. 5 years ago
David Wilson 0e193c223c issue #508: master: minify all Mitogen/ansible_mitogen sources.
Minify-safe files are marked with a magical "# !mitogen: minify_safe"
comment anywhere in the file, which activates the minifier. The result
is naturally cached by ModuleResponder, therefore lru_cache is gone too.

Given:

    import os, mitogen
    @mitogen.main()
    def main(router):
        c = router.ssh(hostname='k3')
        c.call(os.getpid)
        router.sudo(via=c)

SSH footprint drops from 56.2 KiB to 42.75 KiB (-23.9%)
Ansible "shell: hostname" drops 149.26 KiB to 117.42 KiB (-21.3%)
5 years ago
David Wilson c6d5aa29ba ansible: new multiplexer/workers configuration
Following on from 152effc26c9a5918cb7ead7a97fe7fa7f81b6764,

* Pin mux to CPU 0
* Pin top-level CPU 1
* Pin workers sequentially to CPU 2..n

Nets 19.5% improvement on issue_140__thread_pileup.yml when targetting
64 Docker containers on the same 8 core/16 thread machine.

Before (prior to last scheme, no affinity at all):

    2294528.731458      task-clock (msec)         #    6.443 CPUs utilized
        10,429,745      context-switches          #    0.005 M/sec
         2,049,618      cpu-migrations            #    0.893 K/sec
         8,258,952      page-faults               #    0.004 M/sec
 5,532,719,253,824      cycles                    #    2.411 GHz                      (83.35%)
 3,267,471,616,230      instructions              #    0.59  insn per cycle
                                                  #    1.22  stalled cycles per insn  (83.35%)
   662,006,455,943      branches                  #  288.515 M/sec                    (83.33%)
    39,453,895,977      branch-misses             #    5.96% of all branches          (83.37%)

     356.148064576 seconds time elapsed

After:

    2226463.958975      task-clock (msec)         #    7.784 CPUs utilized
         9,831,466      context-switches          #    0.004 M/sec
           180,065      cpu-migrations            #    0.081 K/sec
         5,082,278      page-faults               #    0.002 M/sec
 5,592,548,587,259      cycles                    #    2.512 GHz                      (83.35%)
 3,135,038,855,414      instructions              #    0.56  insn per cycle
                                                  #    1.32  stalled cycles per insn  (83.32%)
   636,397,509,232      branches                  #  285.833 M/sec                    (83.30%)
    39,135,441,790      branch-misses             #    6.15% of all branches          (83.35%)

     286.036681644 seconds time elapsed
5 years ago
David Wilson e77048ec2d utils: pad out reset_affinity() and integrate with detach_popen() 5 years ago
David Wilson b0e3807b16 utils: import reset_affinity() function. 5 years ago
Alex Willmer 0d725f9cd8 Fix DeprecationWarning in mitogen.utils.run_with_router()
```
14:20:32.186270 W py.warnings:
/home/alex/src/mitogen/mitogen/utils.py:152: DeprecationWarning: invalid
escape sequence \*
  """
```
5 years ago
David Wilson dd30a907ce issue #477: promote setup_gil() to mitogen.utils
This is since ansible_mitogen/process.py is 2.6-only, and I want to use
setup_gil() in 2.4 code.
5 years ago
David Wilson bd81b6fd6e issue #477: %f date format requires Py2.6 or newer. 5 years ago
David Wilson a7ee23719a issue #388: move a ton of documentation back into the source 6 years ago
David Wilson caa648b1ea utils: handle duplicate log_to_file() calls. 6 years ago
David Wilson 6cbf34d5fe tests: another smattering of 3.x type/API fixes. 6 years ago
David Wilson 0eb77b5f7c utils: always enable microsecond logging.
It's too useful, and the logs are fairly out of control already, may as
well just capture everything in the first pass.
6 years ago
David Wilson 410016ff47 Initial Python 3.x port work.
* ansible: use unicode_literals everywhere since it only needs to be
  compatible back to 2.6.
* compat/collections.py: delete this entirely and rip out the parts of
  functools that require it.
* Introduce serializable Kwargs dict subclass that translates keys to
  Unicode on instantiation.
* enable_debug_logging() must set _v/_vv globals.
* cStringIO does not exist in 3.x.
* Treat IOLogger and LogForwarder input as latin-1.
* Avoid ResourceWarnings in first stage by explicitly closing fps.
* Fix preamble_size.py syntax errors.
6 years ago
David Wilson 2fbe1f1b54 Get integration tests running under 2.6.
Closes #270
Closes #273
6 years ago
David Wilson cecef992b0 issue #218: core: add Secret and Blob types. 6 years ago
David Wilson 8fc1eac6ae utils: combine MITOGEN_LOG_LEVEL and MITOGEN_LOG_IO.
Saves lots of readline fiddling.
6 years ago
David Wilson 7c88e4d013 Move _DEAD into header, autogenerate dead messages
This change blocks off 2 common scenarios where a race condition is
upgraded to a hang, when the library could internally do better.

* Since we don't know whether the receiver of a `reply_to` is expecting
  a raw or pickled message, and since in the case of a raw reply, there
  is no way to signal "dead" to the receiver, override the reply_to
  field to explicitly mark a message as dead using a special handle.

  This replaces the serialized _DEAD sentinel value with a slightly
  neater interface, in the form of the reserved IS_DEAD handle, and
  enables an important subsequent change: when a context cannot route a
  message, it can send a generic 'dead' reply back towards the message
  source, ensuring any sleeping thread is woken with ChannelError.

  The use of this field could potentially be extended later on if
  additional flags are needed, but for now this seems to suffice.

* Teach Router._invoke() to reply with a dead message when it receives a
  message for an invalid local handle.

* Teach Router._async_route() to reply with a dead message when it
  receives an unroutable message.
6 years ago
David Wilson 7ab1af043e ansible: redirect logging into display 'framework'. closes #111 6 years ago
David Wilson ff617824a1 ansible: fix some flake8 errors
* Unused imports
* Undefined names in helpers.py
* Copyright header wrapping
6 years ago
David Wilson b221eaaa10 ansible: log call timings 6 years ago
David Wilson 91116810a1 ansible: delete utils.py and promote cast() to mitogen.utils 6 years ago
David Wilson 18eaf14dca ansible: migrate logging variables into utils. 6 years ago
David Wilson 09eb3fd9ba utils: support log_to_file(usec=True) 6 years ago
David Wilson 326886832e Add license text everywhere. 6 years ago
David Wilson 6a10ab605e docs: move mitogen.utils docs to Sphinx. 6 years ago
Alex Willmer d706b7d6b8 Update lingering references to with_broker & run_with_broker
A previous commit renamed run_with_broker() and with_broker() to
run_with_router() and with_router() respctively. Some references were
missed.
6 years ago
David Wilson b827ee1bc7 Delete mitogen.utils.log_to_tmp() 6 years ago
David Wilson 750e9fab24 Make log_to_file accept string level rather than logging package constant
Saves an import, pain in the ass to type all the time.
6 years ago
David Wilson e7ff6259a3 Initial commit. 6 years ago