Merge remote-tracking branch 'origin/dmw'
issue510d2eb01ftests: pin idna to last supporting 2.6-compat version.b90889ctests: pin idna to last supporting 2.6-compat version.9da6e6atests: don't call Router.shutdown() twice.b2b7e7btests: file_service_test fixes04755c3issue #426: tighten up PushFileService types.8fa3c74issue #426: RouterMonitor format incorrect for 3->2 forward.d8b9634issue #426: PushFileService missing to_text() call.1e9f344issue #426: big hack so reset_connection has task_var access18bfde5issue #444: update Changelog.835beadtests: allow running scripts from any subdir.81c93e1ci: remove duplicate /usr/bin/time callca9ae45issue #426: TemporaryEnvironment must coerce to Unicode.4bc0d0eissue #426: apply_mode_spec() must handle bytes.a8921bbtests: fix scaling in fork_histogram374a361docs: try to fix CSS difference between local and rtfd861be2edocs: wrap text around logosb084d83docs: fit SVG viewbox to ansible logo98d06e2docs: delete shame.rst to make room for new chapters.5f3244adocs: import pcaps (using LFS) to regenerate charts.6936b93tests: import fork_histogram.py.5a96d13issue #426: fix all.yml sorting, one more delegate_tobd82fa1issue #426: fix low_level_execute_command.yml breakage.a6e6bc4issue #426: to_text filter.d15f533Turn on Travis build notifcations, but send them to IRC.9d87f03issue #426: disable Ansible smart transport.49d37bfissue #426: remove LANG and LC_ALL during tests.a6e6fd1issue #426: more 2->3 test fixes.67f710fissue #426: use delegate_to in fixup_perms2 and copy.ymla67a436docs: add #374 to Changelog.2b229a6docs: add thanks entry.
David Wilson
6 years ago
commit
9a305acdda
File diff suppressed because one or more lines are too long
|
Before Width: | Height: | Size: 8.0 KiB After Width: | Height: | Size: 8.0 KiB |
@ -0,0 +1,3 @@
|
||||
*.pcapng filter=lfs diff=lfs merge=lfs -text
|
||||
run_hostname_100_times_mito.pcap.gz filter=lfs diff=lfs merge=lfs -text
|
||||
run_hostname_100_times_vanilla.pcap.gz filter=lfs diff=lfs merge=lfs -text
|
||||
@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:6d9b4d4ff263003bd16e44c265783e7c1deff19950e453e3adeb8a6ab5052081
|
||||
size 175120
|
||||
@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:7a993832501b7948a38c2e8ced3467d1cb04279a57ddd6afc735ed96ec509c08
|
||||
size 7623337
|
||||
@ -1,90 +0,0 @@
|
||||
|
||||
Importer Wall Of Shame
|
||||
----------------------
|
||||
|
||||
The following modules and packages violate protocol or best practice in some way:
|
||||
|
||||
* They run magic during ``__init.py__`` that makes life hard for Mitogen.
|
||||
Executing code during module import is always bad, and Mitogen is a concrete
|
||||
benchmark for why it's bad.
|
||||
|
||||
* They install crap in :py:data:`sys.modules` that completely ignore or
|
||||
partially implement the protocols laid out in PEP-302.
|
||||
|
||||
* They "vendor" a third party package, either incompletely, using hacks visible
|
||||
through the runtime's standard interfaces, or with ancient versions of code
|
||||
that in turn mess with :py:data:`sys.modules` in some horrible way.
|
||||
|
||||
Bugs will probably be filed for these in time, but it does not address the huge
|
||||
installed base of existing old software versions, so hacks are needed anyway.
|
||||
|
||||
|
||||
``pbr``
|
||||
=======
|
||||
|
||||
It claims to use ``pkg_resources`` to read version information
|
||||
(``_get_version_from_pkg_metadata()``), which would result in PEP-302 being
|
||||
reused and everything just working wonderfully, but instead it actually does
|
||||
direct filesystem access.
|
||||
|
||||
**What could it do instead?**
|
||||
|
||||
* ``pkg_resources.resource_stream()``
|
||||
|
||||
**What Mitogen is forced to do**
|
||||
|
||||
When it sees ``pbr`` being loaded, it smodges the process environment with a
|
||||
``PBR_VERSION`` variable to override any attempt to auto-detect the version.
|
||||
This will probably break code I haven't seen yet.
|
||||
|
||||
|
||||
``pkg_resources``
|
||||
=================
|
||||
|
||||
Anything that imports ``pkg_resources`` will eventually cause ``pkg_resources``
|
||||
to try and import and scan ``__main__`` for its ``__requires__`` attribute
|
||||
(``pkg_resources/__init__.py::_build_master()``). This breaks any app that is
|
||||
not expecting its ``__main__`` to suddenly be sucked over a network and
|
||||
injected into a remote process, like py.test.
|
||||
|
||||
A future version of Mitogen might have a more general hack that doesn't import
|
||||
the master's ``__main__`` as ``__main__`` in the slave, avoiding all kinds of
|
||||
issues like these.
|
||||
|
||||
**What could it do instead?**
|
||||
|
||||
* Explicit is better than implicit: wait until the magical behaviour is
|
||||
explicitly requested (i.e. an API call).
|
||||
|
||||
* Use ``get("__main__")`` on :py:data:`sys.modules` rather than ``import``, but
|
||||
this method isn't general enough, it only really helps tools like Mitogen.
|
||||
|
||||
**What Mitogen is forced to do**
|
||||
|
||||
Examine the stack during every attempt to import ``__main__`` and check if the
|
||||
requestee module is named ``pkg_resources``, if so then refuse the import.
|
||||
|
||||
|
||||
``six``
|
||||
=======
|
||||
|
||||
The ``six`` module makes some effort to conform to PEP-302, but it is missing
|
||||
several critical pieces, e.g. the ``__loader__`` attribute. This not only
|
||||
breaks the Python standard library tooling (such as the :py:mod:`inspect`
|
||||
module), but also Mitogen. Newer versions of ``six`` improve things somewhat,
|
||||
but there are still outstanding issues preventing Mitogen from working with
|
||||
``six``.
|
||||
|
||||
This package is sufficiently popular that it must eventually be supported. See
|
||||
`here for an example issue`_.
|
||||
|
||||
.. _here for an example issue: https://github.com/dw/mitogen/issues/31
|
||||
|
||||
**What could it do instead?**
|
||||
|
||||
* Any custom hacks installed into :py:data:`sys.modules` should support the
|
||||
protocols laid out in PEP-302.
|
||||
|
||||
**What Mitogen is forced to do**
|
||||
|
||||
Vendored versions of ``six`` currently don't work at all.
|
||||
@ -0,0 +1,92 @@
|
||||
|
||||
# Monkey-patch os.fork() to produce a latency histogram on run completion.
|
||||
# Requires 'hdrhsitograms' PyPI module.
|
||||
|
||||
from __future__ import print_function
|
||||
|
||||
import os
|
||||
import resource
|
||||
import sys
|
||||
import time
|
||||
|
||||
import ansible.plugins.callback
|
||||
import hdrh.histogram
|
||||
|
||||
|
||||
def get_fault_count(who=resource.RUSAGE_CHILDREN):
|
||||
ru = resource.getrusage(who)
|
||||
return ru.ru_minflt + ru.ru_majflt
|
||||
|
||||
|
||||
class CallbackModule(ansible.plugins.callback.CallbackBase):
|
||||
hist = None
|
||||
|
||||
def v2_playbook_on_start(self, playbook):
|
||||
if self.hist is not None:
|
||||
return
|
||||
|
||||
self.hist = hdrh.histogram.HdrHistogram(1, int(1e6*60), 3)
|
||||
self.fork_latency_sum_usec = 0.0
|
||||
self.install()
|
||||
|
||||
def install(self):
|
||||
self.faults_at_start = get_fault_count(resource.RUSAGE_SELF)
|
||||
self.run_start_time = time.time()
|
||||
self.real_fork = os.fork
|
||||
os.fork = self.my_fork
|
||||
|
||||
self_fault_usec = 1.113664156753052
|
||||
child_fault_usec = 4.734975610975617
|
||||
|
||||
dummy_heap_size = int(os.environ.get('FORK_STATS_FAKE_HEAP_MB', '0'))
|
||||
dummy_heap = 'x' * (dummy_heap_size * 1048576)
|
||||
|
||||
def my_fork(self):
|
||||
# doesnt count last child, oh well
|
||||
now_faults = get_fault_count()
|
||||
t0 = time.time()
|
||||
try:
|
||||
return self.real_fork()
|
||||
finally:
|
||||
latency_usec = (1e6 * (time.time() - t0))
|
||||
self.fork_latency_sum_usec += latency_usec
|
||||
self.hist.record_value(latency_usec)
|
||||
|
||||
def playbook_on_stats(self, stats):
|
||||
self_faults = get_fault_count(resource.RUSAGE_SELF) - self.faults_at_start
|
||||
child_faults = get_fault_count()
|
||||
run_duration_sec = time.time() - self.run_start_time
|
||||
fault_wastage_usec = (
|
||||
((self.self_fault_usec * self_faults) +
|
||||
(self.child_fault_usec * child_faults))
|
||||
)
|
||||
fork_wastage = self.hist.get_total_count()
|
||||
all_wastage_usec = ((2*self.fork_latency_sum_usec) + fault_wastage_usec)
|
||||
|
||||
print('--- Fork statistics ---')
|
||||
print('Post-boot run duration: %.02f ms, %d total forks' % (
|
||||
1000 * run_duration_sec,
|
||||
self.hist.get_total_count(),
|
||||
))
|
||||
print('Self faults during boot: %d, post-boot: %d, avg %d/child' % (
|
||||
self.faults_at_start,
|
||||
self_faults,
|
||||
self_faults / self.hist.get_total_count(),
|
||||
))
|
||||
print('Total child faults: %d, avg %d/child' % (
|
||||
child_faults,
|
||||
child_faults / self.hist.get_total_count(),
|
||||
))
|
||||
print('Est. wastage on faults: %d ms, forks+faults+waits: %d ms (%.2f%%)' % (
|
||||
fault_wastage_usec / 1000,
|
||||
all_wastage_usec / 1000,
|
||||
100 * (all_wastage_usec / (run_duration_sec * 1e6)),
|
||||
))
|
||||
print('99th%% fork latency: %.03f msec, max %d new tasks/sec' % (
|
||||
self.hist.get_value_at_percentile(99) / 1000.0,
|
||||
1e6 / self.hist.get_value_at_percentile(99),
|
||||
))
|
||||
|
||||
self.hist.output_percentile_distribution(sys.stdout, 1000)
|
||||
print('--- End fork statistics ---')
|
||||
print()
|
||||
@ -0,0 +1,27 @@
|
||||
|
||||
from ansible.module_utils._text import to_text
|
||||
|
||||
|
||||
try:
|
||||
Unicode = unicode
|
||||
except:
|
||||
Unicode = str
|
||||
|
||||
|
||||
def to_text(s):
|
||||
"""
|
||||
Ensure the str or unicode `s` is unicode, and strip away any subclass. Also
|
||||
works on lists.
|
||||
"""
|
||||
if isinstance(s, list):
|
||||
return [to_text(ss) for ss in s]
|
||||
if not isinstance(s, Unicode):
|
||||
s = to_text(s)
|
||||
return Unicode(s)
|
||||
|
||||
|
||||
class FilterModule(object):
|
||||
def filters(self):
|
||||
return {
|
||||
'to_text': to_text,
|
||||
}
|
||||
Loading…
Reference in New Issue