* add optional module_utils import support
Treat core and collections module_utils imports nested within any Python block statement (eg, `try`, `if`) as optional. This allows Ansible modules to implement runtime fallback behavior for missing module_utils (eg from a newer version of ansible-core), where previously, the module payload builder would always fail when unable to locate a module_util (regardless of any runtime behavior the module may implement).
* sanity test fixes
ci_complete
* module compat for py3.8+ controller
* replaced internal usages of selinux bindings with internal ctypes binding (allows basic selinux operations from any Python interpreter), plus tests
* added new respawn_module API to allow modules to import Python packages that are only available under a well-known interpreter, plus tests
* added respawn logic to modules that need Python libs from a specific system interpreter (apt, apt_repository, dnf, yum)
minimize internal HAVE_SELINUX usage
spurious junk
pep8
* pylint fixes
* add RHEL8 Python 3.8 testing
* more pylint
* import sanity
* unit tests
* changelog update
* fix a bunch of stuff
* tweak changelog
* fix setup_rpm_repo on EL8
* misc sanity/test fixes
* misc feedback tweaks
* fix import fallback in test module
* fix selinux MU test
* fix dnf tests to avoid python-dependent test packages
* add trailing LFs to aliases
* fix yum tests to avoid test package with Python deps
* hack create_repo for EL6 to create noarch package
* Support docker and k8s action groups for moved modules in community.docker and community.kubevirt.
* Also support k8s action group for community.okd.
* Also add kubernetes.core.
* Adjust PR #.
* Fix changelog fragment.
* Remove community.okd.
* Revert "Remove community.okd."
This reverts commit 812b5aa6e2.
* speed up modify_module
* Remove debugging
* ci_complete
* Simplify generic_visit, alias to visit, eliminate some attr lookups
* ci_complete
* Add changelog fragment
Change:
- Fix a UnicodeDecodeError in executor.module_common that could get
triggered with -vvvvv.
Test Plan:
- `ansible-test integration --docker centos7 module_utils -vvvvv`
This would show the error previously, and no loner does after this
patch.
Signed-off-by: Rick Elrod <rick@elrod.me>
* Only allow groups which were hardcoded in module_defaults.yml
only load action groups from the collection if module_defaults contains a potential group for the action
* Fix tests using modules that override those whitelisted in lib/ansible/config/module_defaults.yml
Third party modules should not be using group/ - use the action name instead
* add externalized module_defaults tests
add the missing group and collections
ci_complete
Co-authored-by: Matt Davis <mrd@redhat.com>
* changelog
ci_complete
* Fix import in tests
ci_complete
* Update with requested changes
ci_complete
* don't traceback since we don't validate the contents of module_defaults
ci_complete
Co-authored-by: Matt Davis <mrd@redhat.com>
Change:
On OpenBSD when using pipelining, we do not set cwd which results in a
permissions fatal. Ensure that `''` - cwd - is not in `sys.path`.
Test Plan:
Tested against local OpenBSD VM
Tickets:
Fixes#69320
Signed-off-by: Rick Elrod <rick@elrod.me>
* `meta/` directory in collections
* runtime metadata for redirection/deprecation/removal of plugin loads
* a compatibility layer to keep existing content working on ansible-base + collections
* a Python import redirection layer to keep collections-hosted (and otherwise moved) content importable by things that don't know better
* supported Ansible version validation on collection loads
* Address compat issue for collection loading on py26
* Move import_module shim to utils for compat across the codebase
* Enable collection tests on py2.6
* Update changelog fragment
* Simplify code using sys.moduls
* Move compat to module_utils/compat/importlib
* Add back errantly deleted newline
* Remove hack comment
Co-Authored-By: Matt Clay <matt@mystile.com>
Co-authored-by: Matt Clay <matt@mystile.com>
* Set alter_sys=True instead of False to address backwards incompat
* ci_complete
* Add integration test
* ci_complete
* sanity
* ci_complete
* Changelog fragment
* Update import test and validate-modules to match
* Added coverage collection for PowerShell - ci_complete ci_coverage
* uncomment out coverage uploader call
* Generate XML for PowerShell coverage
* Use whitelist to exclude coverage run on non content plugins
* Remove uneeded ignore entry
* Try to reduce diff in cover.py
* Fix up coverage report package - ci_complete ci_coverage
ansible-test only passes files which have the .py suffix for sanity
tests on python files. This change will allow sanity tests to run on
the Python files in hacking/
* Rename test-module to test-module.py
* Symlink test-module for backwards compat since end users may be using
test-module
* Fix test-module sanity errors that are now triggered
* Rename ansible_profile to ansible-profile.py
* Rename build-ansible
* fix module defaults
- corrected precedence (specific module > group)
- made into reusable function
- use from gather_facts/service/package to match 'actual module used'
* Support using importlib on py>=3 to avoid imp deprecation
* Add changelog fragment
* importlib coverage for py3
* Ansiballz execute should use importlib too
* recursive module_utils finder should utilize importlib too
* don't be dumb
* Fix up units
* Clean up tests
* Prefer importlib.util in plugin loader when available
* insert the module into sys.modules
* 3 before 2 for consistency
* ci_complete
* Address importlib.util.find_spec returning None
* module_utils fixes in collections
* fixed Windows module_utils in collections
* fixed more Python module_utils cases (from X import module)
* "medium style" Ansiballz modules now work properly with collections (ie, non-replacer but also not using basic.py)
* added more tests
* split Windows/POSIX exec
* sanity
* basic plugin loading working (with many hacks)
* task collections working
* play/block-level collection module/action working
* implement PEP302 loader
* implicit package support (no need for __init.py__ in collections)
* provides future options for secure loading of content that shouldn't execute inside controller (eg, actively ignore __init__.py on content/module paths)
* provide hook for synthetic collection setup (eg ansible.core pseudo-collection for specifying built-in plugins without legacy path, etc)
* synthetic package support
* ansible.core.plugins mapping works, others don't
* synthetic collections working for modules/actions
* fix direct-load legacy
* change base package name to ansible_collections
* note
* collection role loading
* expand paths from installed content root vars
* feature complete?
* rename ansible.core to ansible.builtin
* and various sanity fixes
* sanity tweaks
* unittest fixes
* less grabby error handler on has_plugin
* probably need to replace with a or harden callers
* fix win_ping test
* disable module test with explicit file extension; might be able to support in some scenarios, but can't see any other tests that verify that behavior...
* fix unicode conversion issues on py2
* attempt to keep things working-ish on py2.6
* python2.6 test fun round 2
* rename dirs/configs to "collections"
* add wrapper dir for content-adjacent
* fix pythoncheck to use localhost
* unicode tweaks, native/bytes string prefixing
* rename COLLECTION_PATHS to COLLECTIONS_PATHS
* switch to pathspec
* path handling cleanup
* change expensive `all` back to or chain
* unused import cleanup
* quotes tweak
* use wrapped iter/len in Jinja proxy
* var name expansion
* comment seemingly overcomplicated playbook_paths resolution
* drop unnecessary conditional nesting
* eliminate extraneous local
* zap superfluous validation function
* use slice for rolespec NS assembly
* misc naming/unicode fixes
* collection callback loader asks if valid FQ name instead of just '.'
* switch collection role resolution behavior to be internally `text` as much as possible
* misc fixmes
* to_native in exception constructor
* (slightly) detangle tuple accumulation mess in module_utils __init__ walker
* more misc fixmes
* tighten up action dispatch, add unqualified action test
* rename Collection mixin to CollectionSearch
* (attempt to) avoid potential confusion/conflict with builtin collections, etc
* stale fixmes
* tighten up pluginloader collections determination
* sanity test fixes
* ditch regex escape
* clarify comment
* update default collections paths config entry
* use PATH format instead of list
* skip integration tests on Python 2.6
ci_complete
* Python interpreter discovery
* No longer blindly default to only `/usr/bin/python`
* `ansible_python_interpreter` defaults to `auto_legacy`, which will discover the platform Python interpreter on some platforms (but still favor `/usr/bin/python` if present for backward compatibility). Use `auto` to always use the discovered interpreter, append `_silent` to either value to suppress warnings.
* includes new doc utility method `get_versioned_doclink` to generate a major.minor versioned doclink against docs.ansible.com (or some other config-overridden URL)
* docs revisions for python interpreter discovery
(cherry picked from commit 5b53c0012ab7212304c28fdd24cb33fd8ff755c2)
* verify output on some distros, cleanup
User module can contain Indentation errors or syntax errors.
Handle AST exceptions rather than showing traceback while importing such module.
Fixes: #21707
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Added basic equivalent to PowerShell modules
* changes based on latest review
* Added tests
* ignore sanity test due to how tests are set up
* Changes to work with PSCore
* Added documentation and change updated more modules
* Add some speed optimisations to AddType
* fix some issues in the doc changes
* doc changes
When python is compiled in debug mode, or certain command line flags are
passed, python issues helpful warnings to let users know of files opened
by not closed.
This can be fairly spammy on an ansible run. This change reduces the
number of such warnings by a factor of 10.
* win_exec: refactor PS exec runner
* more changes for PSCore compatibility
* made some changes based on the recent review
* split up module exec scripts for smaller payload
* removed C# module support to focus on just error msg improvement
* cleaned up c# test classifier code
* win_script: add support for become and centralise exec wrapper builder
* satisfying the pep8 gods
* do not scan for module dependencies when running as a script
The AnsiballZ optimization which only uses one pyton interpreter
currently monkeypatches the arguments into a global argument in module_utils
so we need to always include basic.py. In the future we should fix this
so that it monkeypatches its own file. That way we won't need to always
include basic.py
Now that we don't need to worry about python-2.4 and 2.5, we can make
some improvements to the way AnsiballZ handles modules.
* Change AnsiballZ wrapper to use import to invoke the module
We need the module to think of itself as a script because it could be
coded as:
main()
or as:
if __name__ == '__main__':
main()
Or even as:
if __name__ == '__main__':
random_function_name()
A script will invoke all of those. Prior to this change, we invoked
a second Python interpreter on the module so that it really was
a script. However, this means that we have to run python twice (once
for the AnsiballZ wrapper and once for the module). This change makes
the module think that it is a script (because __name__ in the module ==
'__main__') but it's actually being invoked by us importing the module
code.
There's three ways we've come up to do this.
* The most elegant is to use zipimporter and tell the import mechanism
that the module being loaded is __main__:
* 5959f11c9d/lib/ansible/executor/module_common.py (L175)
* zipimporter is nice because we do not have to extract the module from
the zip file and save it to the disk when we do that. The import
machinery does it all for us.
* The drawback is that modules do not have a __file__ which points
to a real file when they do this. Modules could be using __file__
to for a variety of reasons, most of those probably have
replacements (the most common one is to find a writable directory
for temporary files. AnsibleModule.tmpdir should be used instead)
We can monkeypatch __file__ in fom AnsibleModule initialization
but that's kind of gross. There's no way I can see to do this
from the wrapper.
* Next, there's imp.load_module():
* https://github.com/abadger/ansible/blob/340edf7489/lib/ansible/executor/module_common.py#L151
* imp has the nice property of allowing us to set __name__ to
__main__ without changing the name of the file itself
* We also don't have to do anything special to set __file__ for
backwards compatibility (although the reason for that is the
drawback):
* Its drawback is that it requires the file to exist on disk so we
have to explicitly extract it from the zipfile and save it to
a temporary file
* The last choice is to use exec to execute the module:
* https://github.com/abadger/ansible/blob/f47a4ccc76/lib/ansible/executor/module_common.py#L175
* The code we would have to maintain for this looks pretty clean.
In the wrapper we create a ModuleType, set __file__ on it, read
the module's contents in from the zip file and then exec it.
* Drawbacks: We still have to explicitly extract the file's contents
from the zip archive instead of letting python's import mechanism
handle it.
* Exec also has hidden performance issues and breaks certain
assumptions that modules could be making about their own code:
http://lucumr.pocoo.org/2011/2/1/exec-in-python/
Our plan is to use imp.load_module() for now, deprecate the use of
__file__ in modules, and switch to zipimport once the deprecation
period for __file__ is over (without monkeypatching a fake __file__ in
via AnsibleModule).
* Rename the name of the AnsiBallZ wrapped module
This makes it obvious that the wrapped module isn't the module file that
we distribute. It's part of trying to mitigate the fact that the module
is now named __main)).py in tracebacks.
* Shield all wrapper symbols inside of a function
With the new import code, all symbols in the wrapper become visible in
the module. To mitigate the chance of collisions, move most symbols
into a toplevel function. The only symbols left in the global namespace
are now _ANSIBALLZ_WRAPPER and _ansiballz_main.
revised porting guide entry
Integrate code coverage collection into AnsiballZ.
ci_coverage
ci_complete
* tolerate windows line endings when loading windows module utils. Helpful for old custom windows modules.
* add test modules to demonstrate win line ending module load behaviour.
* attempt to fix sanity check failures
* pep8 fix
* explict skip of test modules from shebang check (core modules must still have expected unix style line endings)
* switch to rstrip() following core team meeting feedback
* module_common: set required parameter templar
Fix the following error (related to b455901):
$ ./hacking/test-module -m ./lib/ansible/modules/system/ping.py -I ansible_python_interpreter=/usr/bin/python
Traceback (most recent call last):
File "./hacking/test-module", line 268, in <module>
main()
File "./hacking/test-module", line 249, in main
(modfile, modname, module_style) = boilerplate_module(options.module_path, options.module_args, interpreters, options.check, options.filename)
File "./hacking/test-module", line 152, in boilerplate_module
task_vars=task_vars
File "ansible/lib/ansible/executor/module_common.py", line 910, in modify_module
environment=environment)
File "ansible/lib/ansible/executor/module_common.py", line 736, in _find_module_utils
shebang, interpreter = _get_shebang(u'/usr/bin/python', task_vars, templar)
File "ansible/lib/ansible/executor/module_common.py", line 452, in _get_shebang
interpreter = templar.template(task_vars[interpreter_config].strip())
AttributeError: 'NoneType' object has no attribute 'template'
* module_common.modify_module: templar is required
* Fix types when evaluating interpreter. Fixes#36536
* Rename variables that contain bytes to b_*
* Get rid of to_text() and to_bytes() calls that do nothing (because the
data is already the proper type)
* windows: add #AnsibleRequires to set whether a module requires module or a specific version
* fix up pep8 issues
* changed psversion to use the actual ps Requires -Version syntax
* missed the check on #Requires -Version
* fix #Requires module extensions
* Ansible Config part2
- made dump_me nicer, added note this is not prod
- moved internal key removal function to vars
- carry tracebacks in errors we can now show tracebacks for plugins on vvv
- show inventory plugin tracebacks on vvv
- minor fixes to cg groups plugin
- draft config from plugin docs
- made search path warning 'saner' (top level dirs only)
- correctly display config entries and others
- removed unneeded code
- commented out some conn plugin specific from base.yml
- also deprecated sudo/su
- updated ssh conn docs
- shared get option method for connection plugins
- note about needing eval for defaults
- tailored yaml ext
- updated strategy entry
- for connection pliugins, options load on plugin load
- allow for long types in definitions
- better display in ansible-doc
- cleaned up/updated source docs and base.yml
- added many descriptions
- deprecated include toggles as include is
- draft backwards compat get_config
- fixes to ansible-config, added --only-changed
- some code reoorg
- small license headers
- show default in doc type
- pushed module utils details to 5vs
- work w/o config file
- PEPE ATE!
- moved loader to it's own file
- fixed rhn_register test
- fixed boto requirement in make tests
- I ate Pepe
- fixed dynamic eval of defaults
- better doc code
skip ipaddr filter tests when missing netaddr
removed devnull string from config
better becoem resolution
* killed extra space with extreeme prejudice
cause its an affront against all that is holy that 2 spaces touch each other!
shippable timing out on some images, but merging as it passes most
The AnsiBallZ wrapper is transferred to the remote machine with
a filename similar to the Ansible-module it runs. For modules like copy
and tempfile, this can end up conflicting with stdlib modules on the
remote machine depending on how python is setup there. We have a little
bit of code in the wrapper to deal with this by removing the path that
the ansible module resides in from sys.path.
On MacOSX, that code was having a problem. The path the module ends up
in included a symlinked directory so we were looking for a path in
sys.path but we had to look for the unsymlinked path instead.
Fix that by using os.path.realpath() instead of os.path.abspath()
Matt Davis