macOS 11 is not longer an available runner on Azure Devops. The minimum is now
macOS 12. This runner does not have Python 2.7 installed, so running them
would require a custom install - which I'm declaring too much effort for too
little gain.
refs #1090
`mitogen.parent.POLLER_LIGHTWEIGHT` will normally be `PollPoller`, falling
back to `EpollPoller`, `KqueuePoller`, or `Poller`.
Fixes#957
Co-authored-by: Luca Berruti <nadirio@gmail.com>
Co-authored-by: Philippe Kueck <bqobccy6ejnq2bqvmebqiwqha4cs4@protected32.unixadm.org>
Follwing fixes in Ansible 7-9 for CVE-2023-5764 cating `AnsibleUnsafeBytes` &
`AnsibleUnsafeText` to `bytes()` or `str()` requires special handling. The
handling is Ansible specific, so it shouldn't go in the mitogen package but
rather the ansible_mitogen package.
`ansible_mitogen.utils.unsafe.cast()` is most like `mitogen.utils.cast()`.
During development it began as `ansible_mitogen.utils.unsafe.unwrap_var()`,
closer to an inverse of `ansible.utils.unsafe_procy.wrap_var()`. Future
enhancements may move in this direction.
refs #977, refs #1046
See also
- https://github.com/advisories/GHSA-7j69-qfc3-2fq9
- https://github.com/ansible/ansible/pull/82293
- https://github.com/mitogen-hq/mitogen/wiki/AnsibleUnsafe-notes
If casting a string fails then raise a TypeError. This is potentially an API
breaking change; chosen as the lesser evil vs. allowing silent errors.
`cast()` relies on `bytes(obj)` & `str(obj)` returning the respective
supertype. That's no longer the case for `AnsibleUnsafeBytes` &
`AnsibleUnsafeText`; since fixes/mitigations for CVE-2023-5764.
fixes#1046, refs #977
See also
- https://github.com/advisories/GHSA-7j69-qfc3-2fq9
- https://github.com/ansible/ansible/pull/82293
Most of the necessary changes were made in recent PEP 451 commits. This bumps
the CI jobs, and declares the support. Test dependendancies are bumped to
latest supportted/available versions.
refs #1033
importlib.machinery.ModuleSpec and find_spec() were introduced in Python 3.4
under PEP 451. They replace the find_module() API of PEP 302, which was
deprecated from Python 3.4. They were removed in Python 3.12 along with the
imp module.
This change adds support for the PEP 451 APIs. Mitogen should no longer import
imp on Python versions that support ModuleSpec. Tests have been added to cover
the new APIs.
CI jobs have been added to cover Python 3.x on macOS.
Refs #1033
Co-authored-by: Witold Baryluk <witold.baryluk@gmail.com>
faulthandler is a stdlib module in Python 3.3+. For a long time a PyPI package
of the same name was available for earlier Python releases. That package has
since been removed from PyPI, and the source respoitory archived. So we should
not rely on it.
fixes#983 refs #970
Refs #925#969
I'm not 100% confident that merely removing this is the full fix,
without substituting something else. I am sure keeping it would be
the greater of two evils. __del__() should be avoided on general
principal, and it's associated with multiple intermittant CI
failures, plus multiple user reported issues.
This reapplies an earlier change, when this plugin was first introduced to
Mitogen. The plugin was updated to fix
[DEPRECATION WARNING]: The '_remote_checksum()' method is deprecated.
I've elected to short-circuit the if statemtn logic, rather than
deleting/unindenting, to make the code delta much smaller. This should make it
easier to maintain/update.
Fixes#915
Co-authored-by: Stefano Rivera <stefano@rivera.za.net>
When the requested module (e.g. ansible.module_utils.distro)
- is provided by another module *e.g. distro)
- that itself was a package (e.g. distro 1.7.0)
At runtime
- ansible/module_utils/distro/__init__.py executes
- if https://pypi.org/project/distro/ is present, it's loaded as
ansible.module_utils.distro
- otherwise ansible/module_utils/distro/_distro.py is loaded
ParentEnumerationMethod would wrongly use whatever was in
sys.modules['ansible.module_utils.distro]. Instead we should ascend to
the first parent that has fullname == sys.modules[fullname].__name__.
Then descend to the appropriate .py file on disk.
This bug didn't show up before because until distro 1.7.0 (Feb 2022) the
top-level distro module was a module (distro.py) not a package
(distro/__init__.py)
fixes#906
Also
- Simplifies adding support for additional Ansible versions
- Unifies Python package versioning in CI and local test environments
- Matches Python versions tested, with those declared in setup.py
- Expands targets covered by automated Ansible tests to
- centos6, centos8
- debian9, debian11
- ubuntu1604, ubuntu2004
This reverse shell was historically used to debug CI jobs interactively.
It is not used anymore, and may be causing jobs to hang, then timeout.
There is no reason to keep it, and removing it simplifies CI jobs.
Additionally it has been reported as flagged by security scanners, so
removing it makes Mitogen easier to package/adopt.
fixes#847
The Travis Linux distribution must be upgraded because the Trusty
(Ubuntu 14.04) image does not have Python 3.9. Xenial (Ubuntu 16.04) is
the earliest version that offers Python 3.9.
I have not chosen a later release, in order to aid restoration of Python
2.4 - 2.6 tests.
* origin/master: (277 commits)
Fix DjangoMixin test imports for setuptools >= 50.0
Add ansible.legacy.setup to be fixed on py3.5
code cleanup + adds 0.2.10 + 0.3.0 changelog
adding clarifying comments
fix py3.5.1-3.5.3 setup import error for Ansible 2.10
tests: Fix AttributeError in callback plugins used by test suite
code review changes, using when statements and adding trailing comma
ssh: Match newer ssh host key prompt that accepts the fingerprint
🎉 no more warnings, only load specific collection subdirs instead of top-level collection path (ie no ansible_collections/google, only ansible_collections/google/cloud, etc)
ansible 2.10 no longer has a at the end of the error msg... 🤦
skip vanilla Ansible 2.10 hanging task if not is_mitogen
vanilla ansible is now running but is really slow; bump timeout
try vanilla ansible 2.10 on Mac
travis is having trouble running vanilla Ansible so migrating to Azure
disable debops since it breaks with ansible 2.10
install all required debops extras for ansible
netaddr needs to be on the Ansible controller, not in target nodes
forgot to update apt cache
turn off host key checking with ad-hoc python-netaddr install and add back in debops command line
don't need to ci_lib run setting up python-netaddr
need to specify strategy plugin for ansible ad-hoc
need python-netaddr in docker target containers for debops
adding hopefully new-style import that works for Ansible 2.10
make sure to apt-get update first before install
apt needs sudo
disable python <= 2.6 tests
install missing python-netaddr for debops
revert missing interpreter change, it breaks with Mitogen and without Mitogen, something else might be causing new-style detection to not work
oops, broke new-style missing interpreter detection. Regex should match now
fix custom_python_new_style_missing_interpreter, looks like Ansible 2.10 changed how new-style module detection works
add workaround for TravisCI 4MB log limit job termination
fix regression in Darwin 19 (OSX 10.15+) ansible python interpreter detection
something broke with Mac 10.14 with dscl, before trying a hack see if OS upgrade works
don't run sshpass install through run
azure tests don't like sshpass v1.06 so pegging to 1.05
fix Error: Calling Non-checksummed download of sshpass formula file from an arbitrary URL is disabled
result length is 3 in Azure, 4 on local Mac
fixed ansible_become_pass test, looks like regression on Ansible's end
localhost_ansible tests now pass, adding -vvv to ansible_tests to get more debug info there
fixed issue of switching between mitogen and non-mitogen strategies
fix yml parsing
oops, yml file can't be empty
ignore another flaky test that works locally
fix ansible version check error
fix runner_one_job ansible version comparison
oops, 0664 not 0666
fix fixup_perms2() test
default copy perms look like 0644 now based on ansible source and docs
missed a format call var
remove ansible from github tag install setup in test config files
add support for ansible_collections site-package (from pip ansible==2.10.0 install) + switch to ansible 2.10.0 rather than github tag
remove debugging
remove synchronize fail test for azure
ignore synchronize for now, made ticket
try and get some visibility into test failures
fix venv install
see if sys.path is being loaded properly on azure
print didn't work because verbosity, throw valueerror to see
more debugging, synchronize is being weird on azure
python3 needs python3-venv
tests are in a bad state...somehow both apt and brew can exist on azure using a linux job with an ubuntu vm image???
need to group all python install commands together
python3 tests are broken...
cffi super old, try and update it
try a different psycopg2 package as well
need to install psycopg2-binary in the created venv
fix 'struct _is' error hopefully
brew is missing postgresql
awesome, /usr/local/bin/python2.7 already exists
missed a format
wrong letter 🤦 what am I doing
missed a )
missed a ,
clean up azure python version used
print what's being ran in tests
try running ansible_mitogen 2.10 tests with python3
check sys.path issue
add back in ansible tests but don't run synchronize
turn off failing Ansible-only tests for now, also raising errors to see what Azure is gonna do with collections
removed duplicate install and added debug dump of collection loading to see what tests are doing
ansible.posix.synchronize isn't being loaded in tests but is locally, reducing v count to get around azure devops scroll bug
hopefully this also fails the same way
any amount of v is too much v, even when viewing tests in raw log file mode
add missing collections 🤦
verify collection is working as expected
can't replicate but think it's because synchronize is now a collection
2 v freezes things...this is impossible to debug
figure out what synchronize is now
put future import in wrong place
3 v is too much v for azure devops to render
add some debugging info, was able to run the failed synchronize test locally just fine using test framework, not sure what's going on
test cleanup and trying to replicate synchronize fails
warnings silenced, see if can put back in vvv
try and suppress mode warning clogging up logs
logs too verbose, unable to load test page
run tests with verbose logging
perhaps a modern debops version will work
travis pip is 9 from what the logs say
remove ansible 2.4-specific test
fix fixup_perms2 default file mode
...
CALLBACK_VERSION et al are documented as required in
https://docs.ansible.com/ansible/2.10/dev_guide/developing_plugins.html#callback-plugins.
The need for document_fragment is noted in
cfa8075537/lib/ansible/plugins/callback/default.py (L28-L32)Fixes#758
This addresses the following error, seen while running
`ansible_tests.py`.
```
TASK [Gathering Facts gather_timeout=10, gather_subset=['all']]
****************
task path:
/home/alex/src/mitogen/tests/ansible/regression/issue_109__target_has_old_ansible_installed.yml:4
[WARNING]: Failure using method (v2_runner_on_start) in callback plugin
(<ansible.plugins.callback.nice_stdout.CallbackModule object at
0x7f76b3dad090>): 'show_per_host_start'
Callback Exception:
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/executor/task_queue_manager.py",
line 372, in send_callback
method(*new_args, **kwargs)
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/plugins/callback/default.py",
line 240, in v2_runner_on_start
if self.get_option('show_per_host_start'):
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/plugins/callback/__init__.py",
line 91, in get_option
return self._plugin_options[k]
Callback Exception:
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/executor/task_queue_manager.py",
line 372, in send_callback
method(*new_args, **kwargs)
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/plugins/callback/default.py",
line 240, in v2_runner_on_start
if self.get_option('show_per_host_start'):
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/plugins/callback/__init__.py",
line 91, in get_option
return self._plugin_options[k]
[task 339882] 00:00:08.172036 D ansible_mitogen.affinity: CPU mask for
WorkerProcess: 0x000004
Callback Exception:
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/executor/task_queue_manager.py",
line 372, in send_callback
method(*new_args, **kwargs)
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/plugins/callback/default.py",
line 240, in v2_runner_on_start
if self.get_option('show_per_host_start'):
File
"/home/alex/src/mitogen/.tox/py27-ansible2.10/lib/python2.7/site-packages/ansible/plugins/callback/__init__.py",
line 91, in get_option
return self._plugin_options[k]
```
Looking at textedit window before closing, I notice the Mitogen time was
taken from /usr/bin/time while the Ansible time was taken from the
profile callback plugin.
Ansible has no blocking services running, or really any service that
would have an outsized benefit from multiple IO waiters. Probably we
only need 1, but let's start with 2 just in case.
Alex Willmer