Commit Graph

27 Commits (56d142350d62fb674e1c6874b3ace2cf5cb933a7)

Author SHA1 Message Date
Matt Martz 56d142350d
Add support for importlib.resources (#78915)
* Add support for importlib.resources

* Remove the importlib.resources imports

* return the correct data

* Some code comments, and re-order for consistency

* Disallow traversing packages below an individual collection

* Add a traversable class for namespaces

* Re-use variable

* Utilize itertools.chain.from_iterable

Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>

* Simplify logic to check for packages from ansible loaders

Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>

* Just a generator expression, instead of a generator

* docstrings

* Add comment about find_spec for our namespaces

* Add some initial unit tests for importlib.resources

* normalize

* Utilize importlib.resources for listing collections

* collections_path is already in config, just use config

* install uses a different default for collections_path

* Remove unused import

* Remove duplicate __truediv__

* Bring back TraversableResources

* Apply some small suggestions from code review

Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
Co-authored-by: Matt Davis <6775756+nitzmahone@users.noreply.github.com>

* Remove cross contamination between plugin loader code and CLI code

* Remove unused import

Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
Co-authored-by: Matt Davis <6775756+nitzmahone@users.noreply.github.com>
1 year ago
Matt Martz 884244f1b2
Python 3.9 min for controller (#77566) 2 years ago
Brian Coca b439e41a91
expand ansible-doc coverage (#74963)
* Expand ansible-doc to tests/filters and fix existing issues

  enable filter/test docs if in single file or companion yaml
  add docs for several filters/tests plugins
  allow .yml companion for docs for other plugins, must be colocated
  verify plugins are valid (not modules, cannot)
  fix 'per collection' filtering
  limit old style deprecation (_ prefix) to builtin/legacy
  start move to pathlib for saner path handling
  moved some funcitons, kept backwards compat shims with deprecation notice

  Co-authored-by: Abhijeet Kasurde <akasurde@redhat.com>
  Co-authored-by: Felix Fontein <felix@fontein.de>
  Co-authored-by: Sandra McCann <samccann@redhat.com>
2 years ago
Matt Clay 27923aad7e Add inline type hint ignores. 2 years ago
Matt Clay 4867ac217b Code cleanup for type hinting issues. 2 years ago
Matt Clay 0c4c18bc04 Type hint fixes. 2 years ago
Matt Clay d31730d781
Fixes for type hinting issues. (#77170) 2 years ago
Abhijeet Kasurde ea34fdbfe2
Misc typo fixes in plugins directory (#76566) 2 years ago
Sloane Hertel ed6581e4db
check finder type before passing path (#76448)
* check finder type before passing path

ci_complete

* Reduce nesting

* Test find_module does not cause a traceback with Python 3 FileFinder

* Update lib/ansible/utils/collection_loader/_collection_finder.py
2 years ago
Sloane Hertel b50f16db91
Update collection loader for Python 3.10 (#76225)
* Implement find_spec and exec_module to remove reliance on deprecated methods in the collection loader

ci_complete

* Move module execution to exec_module

Remove extra sys.modules handling

Use default module initialization by returning None from loader.create_module

Refactor

ci_complete

* Remove ansible-test's copy of the collection loader

ci_complete

* Fix metaclass for Python 2.x

ci_complete

* Fix Py2/Py3 syntax compatibility

* Refactor

ci_complete

* update collection_loader comments

ci_complete

* simplify find_module

ci_complete

* Fix Py2 compatibility - don't get loader from nonexistent spec

Remove unnecessary PY3 checking

* Refactor common code in load_module and exec_module

ci_complete

* tidy diff

ci_complete

* Include collection_loader in target paths for 'compile' sanity test

* add changelog

* Add "return None" instead of doing it implicitly

Remove get_filename

short-circuit exec_module if it's a redirect

ci_complete
2 years ago
Matt Clay 99a79e1969
ansible-test - Move code from _data to _util. (#75495)
* Update paths to match relocated files.
* Update ansible-test symlink paths.
* Update path classification.
* Update MANIFEST.in
* Update sanity test context paths.
* Update sanity ignores.
* Update shebang sanity test.
* Update configure-remoting-ps1 sanity test.
* Update BOTMETA.yml to reflect new paths.
* Update paths in collection loader comments.
* Update Makefile for ansible-test.
* Update docs.
* Add changelog fragment.
3 years ago
Sloane Hertel 3b861abce1
add action_groups support to collections (#74039)
* Canonicalize module_defaults actions and action_groups pre-fork and cache them on the play

* Call get_action_args_with_defaults with the resolved FQCN plugin and don't pass the redirect list

* Add validation for action_group metadata and a toggle to disable the warnings

* Handle groups recursively referring to each other

* Remove special-casing for non-fqcn actions in module_defaults groups

* Error for actions and groups in module_defaults that can't be resolved

* Error for fully templated module_defaults

* Add integration tests for action_groups

* Changelog
3 years ago
Matt Clay baa371e7b5 Add comments to keep collection loader in sync. 3 years ago
Brian Coca 6418f368e3
Fix pb subs collapse (#74301)
* fix subdir parsing for plays

  fixes #74283

  Co-authored-by: Nikolaos Kakouros <nkak@kth.se>
3 years ago
Ikko Ashimine b70a42f693
fix typo in _collection_finder.py (#74218) 3 years ago
Alexei Znamensky 920b68f5f2
Fixed/improved regular expresssion for collection names (#73577)
* added changelog fragment
* added a couple of tests to coll name validation
3 years ago
Sviatoslav Sydorenko f327e65d11
Use iskeyword and str.isidentifier for "is FQCN" (#73279)
* Use valid FQCN in test_verbosity_arguments

* Use iskeyword and str.isidentifier for "is FQCN"
3 years ago
Brian Coca d22804c4fb
saner path dir management (#72648)
* saner path dir management

   fixes #72628

   ensure we always store paths w/o a_c

Co-authored-by: Sloane Hertel <19572925+s-hertel@users.noreply.github.com>
3 years ago
Brian Coca 8eab113cb1
show keyword documentation in ansible-doc (#72476)
* show keyword documentation in ansible-doc

Co-authored-by: Felix Fontein <felix@fontein.de>
4 years ago
Brian Coca e05c62547b
run playbook from collections (#67435)
* fixes for collection playbooks

 - add fqcn invocation, also handles extensions
 - bring import_playbook into new normal
 - avoid adding collection playbooks to adjacency
 - deal with extensions or lack of em
 - added tests
 - fix bugs with multiple playbooks supplied
 - unicode/bytes/native fixes
 - fix modulenotfound for < py3.7
 - tests
4 years ago
Matt Davis e813b0151c
fix coverage output from synthetic packages (#71727)
* fix coverage output from synthetic packages

* synthetic packages (eg, implicit collection packages without `__init__.py`) were always created at runtime with empty string source, which was compiled to a code object and exec'd during the package load. When run with code coverage, it created a bogus coverage entry (since the `__synthetic__`-suffixed `__file__` entry didn't exist on disk).
* modified collection loader `get_code` to preserve the distinction between `None` (eg synthetic package) and empty string (eg empty `__init__.py`) values from `get_source`, and to return `None` when the source is `None`. This allows the package loader to skip `exec`ing things that truly have no source file on disk, thus not creating bogus coverage entries, while preserving behavior and coverage reporting for empty package inits that actually exist.

* add unit test
4 years ago
Matt Davis c616e54a6e
refactor Python module_utils locator (#70610)
* refactor Python module_utils locator

* no longer recursive
* embed special-case module code internally
* share common code between collections/not cases
* fixes #70134
* properly support subpackage redirection
* adds support for FQCN redirect targets used by migration (expands to FQ Python name)
* add tests

* add changelog
4 years ago
Sloane Hertel 51f6d129cb
support hard coded module_defaults.yml groups for collections (#69919)
* Only allow groups which were hardcoded in module_defaults.yml

only load action groups from the collection if module_defaults contains a potential group for the action

* Fix tests using modules that override those whitelisted in lib/ansible/config/module_defaults.yml

Third party modules should not be using group/ - use the action name instead

* add externalized module_defaults tests

add the missing group and collections

ci_complete

Co-authored-by: Matt Davis <mrd@redhat.com>

* changelog

ci_complete

* Fix import in tests

ci_complete

* Update with requested changes

ci_complete

* don't traceback since we don't validate the contents of module_defaults

ci_complete

Co-authored-by: Matt Davis <mrd@redhat.com>
4 years ago
Matt Davis 984216f52e
various deprecation, display, warning, error fixes for collections redirection (#69822)
* various deprecation, display, warning, error fixes

* Update lib/ansible/utils/display.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* Update lib/ansible/utils/display.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* Update lib/ansible/utils/display.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* cleanup, test fixes

* add collection name to deprecated() calls

* clean up redirect entries from uncommitted tests

* fix dep warning/error header text to match previous

Co-authored-by: Felix Fontein <felix@fontein.de>
4 years ago
Matt Davis d79b23910a
fix sys.modules cleanup and blacklist behavior (#69898)
* fix sys.modules cleanup and blacklist behavior

* fix map-as-generator py2/py3 issue
* clear path_importer_cache between runs

* sanity fix

* don't be stupid with moving target generators
4 years ago
Matt Davis fa81cc6a0c
fix delegated loading when path is not a directory (#69713)
* find_module can't pop ImportError- we need to just translate to `None` since this is a normal condition with files on sys.path (eg `/usr/lib/python36.zip`)
* added test
4 years ago
Matt Davis f7dfa817ae
collection routing (#67684)
* `meta/` directory in collections
* runtime metadata for redirection/deprecation/removal of plugin loads
* a compatibility layer to keep existing content working on ansible-base + collections
* a Python import redirection layer to keep collections-hosted (and otherwise moved) content importable by things that don't know better
* supported Ansible version validation on collection loads
4 years ago