Commit Graph

19 Commits (4e55b936132c0363e23708cfbbe5e5f53e2b5cdf)

Author SHA1 Message Date
Matt Davis c616e54a6e
refactor Python module_utils locator (#70610)
* refactor Python module_utils locator

* no longer recursive
* embed special-case module code internally
* share common code between collections/not cases
* fixes #70134
* properly support subpackage redirection
* adds support for FQCN redirect targets used by migration (expands to FQ Python name)
* add tests

* add changelog
4 years ago
Bob Weinand f200487414
Fix filedescriptor out of range in select() when running commands (#65058)
* Fix filedescriptor out of range in select() when running commands

* Simplify the run_command() code

Now that we're using selectors in run_command(), we can simplify some of
the code.

* Use fileobj.read() instead of os.read()
* No longer use get_buffer_size() as we can just slurp all of the data
  instead.

Also use a simpler conditional check of whether the selector map is
empty

Co-authored-by: Toshio Kuratomi <a.badger@gmail.com>
5 years ago
Sam Doran 3461c682c3
Add mechanism for storing warnings and deprecations outside of AnsibleModule (#58993)
* Move warn() and deprecate() methods out of basic.py
* Use _global_warnings and _global_deprications and create accessor functions
    - This lays the foundation for future functions being moved outside of AnsibleModule
       that need an interface to warnings and deprecations without modifying them.
* Add unit tests for new warn and deprecate functions
5 years ago
Matt Clay f510d59943
Support relative imports in AnsiballZ. (#61196) 5 years ago
Sam Doran 2d266fbc87
Fix unit tests to work with pytest >= 5.0 (#60246)
pytest made a change in the way the message from ExceptionInfo objects was returned.

https://docs.pytest.org/en/latest/changelog.html#pytest-5-0-0-2019-06-28
5 years ago
Matt Martz 2732cde031
Support using importlib on py>=3 to avoid imp deprecation (#54883)
* Support using importlib on py>=3 to avoid imp deprecation

* Add changelog fragment

* importlib coverage for py3

* Ansiballz execute should use importlib too

* recursive module_utils finder should utilize importlib too

* don't be dumb

* Fix up units

* Clean up tests

* Prefer importlib.util in plugin loader when available

* insert the module into sys.modules

* 3 before 2 for consistency

* ci_complete

* Address importlib.util.find_spec returning None
6 years ago
Sam Doran ff88bd82b5
Move type checking methods out of basic.py and add unit tests (#53687)
* Move check_type_str() out of basic.py

* Move check_type_list() out of basic.py

* Move safe_eval() out of basic.py

* Move check_type_dict() out of basic.py

* Move json importing code to common location

* Move check_type_bool() out of basic.py

* Move _check_type_int() out of basic.py

* Move _check_type_float() out of basic.py

* Move _check_type_path() out of basic.py

* Move _check_type_raw() out of basic.py

* Move _check_type_bytes() out of basic.py

* Move _check_type_bits() out of basic.py

* Create text.formatters.py

Move human_to_bytes, bytes_to_human, and _lenient_lowercase out of basic.py into text.formatters.py
Change references in modules to point to function at new location

* Move _check_type_jsonarg() out of basic.py

* Rename json related functions and put them in common.text.converters

Move formatters.py to common.text.formatters.py and update references in modules.

* Rework check_type_str()

Add allow_conversion option to make the function more self-contained.
Move the messaging back to basic.py since those error messages are more relevant to using this function in the context of AnsibleModule and not when using the function in isolation.

* Add unit tests for type checking functions

* Change _lenient_lowercase to lenient_lowercase per feedback
6 years ago
Sam Doran 43a44e6f35
Move utility functions out of basic.py (#51715)
Move the following methods to lib/anisble/module_utils/common/validation.py:

- _count_terms()
- _check_mutually_exclusive()
- _check_required_one_of()
- _check_required_together()
- _check_required_by()
- _check_required_arguments()
- _check_required_if
- fail_on_missing_params() --> create check_missing_parameters()
6 years ago
Sam Doran aba4bed803
Move _handle_no_log_values() out of basic.py (#48628)
* Rename method and make private
* Use is_iterable, combine transformations
* Remove unused return_values from network modules
* Improve docstrings in new functions
* Add new PASS_VAR
* Add unit tests for list_no_log_values
* Fix unit tests for Python 2.6
6 years ago
Sam Doran 2a98faee2b
Move _handle_aliases() out of basic.py (#48578)
Refinements:
- return legal_inputs and update class properties
- remove redundant arguments from method and handle in caller
- add better exception types to method

* Add unit tests for handle_aliases
6 years ago
Abhijeet Kasurde bc6cd13874
Handle exception raised in recursive_finder API (#49590)
User module can contain Indentation errors or syntax errors.
Handle AST exceptions rather than showing traceback while importing such module.

Fixes: #21707

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
6 years ago
Toshio Kuratomi 5844c8c7f0 Cleanups to the common.sys_info API
* Move get_all_subclasses out of sys_info as it is unrelated to system
  information.
* get_all_subclasses now returns a set() instead of a list.
* Don't port get_platform to sys_info as it is deprecated.  Code using
  the common API should just use platform.system() directly.
* Rename load_platform_subclass() to get_platform_subclass and do not
  instantiate the rturned class.
* Test the compat shims in module_utils/basic.py separately from the new
  API in module_utils/common/sys_info.py and module_utils/common/_utils.py
6 years ago
Toshio Kuratomi 61b1daa65f Port from plaform.dist to ansible.module_utils.distro.linux_distribution
ci_complete
6 years ago
Andreas Calminder 876b637208 move some of basic into common (#48078)
* move file functions into common.file

* move *_PERM_BITS and mark as private (_*_PERM_BITS)

* move get_{platform, distribution, distribution_version} get_all_subclasses and load_platform_subclass into common.sys_info

* forgot get_distribution_version, properly rename get_all_subclasses

* add common/sys_info.py to recursive finder test

* update module paths in test_platform_distribution.py

* update docstrings, _get_all_subclasses -> get_all_subclasses

* forgot to update names

* remove trailing whitespace
6 years ago
Toshio Kuratomi 8ec973b453 Fix unittests for always adding basic to the AnsiBallZ zips 6 years ago
Toshio Kuratomi 9f7b0dfc30 Remove automatic use of system six
* Enable the pylint no-name-in-module check.  Checks that identifiers in
  imports actually exist.  When we do this, we also have to ignore
  _MovedItems used in our bundled six.  This means pylint won't check
  for bad imports below ansible.module_utils.six.moves but that's
  something that pylint punts on with a system copy of six so this is
  still an improvement.
* Remove automatic use of system six.  The exec in the six code which
  tried to use a system library if available destroyed pylint's ability
  to check for imports of identifiers which did not exist (the
  no-name-in-module check).  That test is important enough that we
  should sacrifice the bundling detection in favour of the test.
  Distributions that want to unbundle six can replace the bundled six in
  ansible/module_utils/six/__init__.py to unbundle.  however, be aware
  that six is tricky to unbundle.  They may want to base their efforts
  off the code we were using:

  2fff690caa/lib/ansible/module_utils/six/__init__.py

* Update tests for new location of bundled six Several code-smell tests
  whitelist the bundled six library.  Update the path to the library so
  that they work.

* Also check for basestring in modules as the enabled pylint tests will
  also point out basestring usage for us.
7 years ago
Dag Wieers 4efec414e7 test/: PEP8 compliancy (#24803)
* test/: PEP8 compliancy

- Make PEP8 compliant

* Python3 chokes on casting int to bytes (#24952)

But if we tell the formatter that the var is a number, it works
8 years ago
Toshio Kuratomi 2fff690caa Update module_utils.six to latest (#22855)
* Update module_utils.six to latest

We've been held back on the version of six we could use on the module
side to 1.4.x because of python-2.4 compatibility.  Now that our minimum
is Python-2.6, we can update to the latest version of six in
module_utils and get rid of the second copy in lib/ansible/compat.
8 years ago
Toshio Kuratomi 5c38f3cea2 Combine jimi-c and bcoca's ideas and work on hooking module-utils into PluginLoader.
This version just gets the relevant paths from PluginLoader and then
uses the existing imp.find_plugin() calls in the AnsiballZ code to load
the proper module_utils.

Modify PluginLoader to optionally omit subdirectories (module_utils
needs to operate on top level dirs, not on subdirs because it has
a hierarchical namespace whereas all other plugins use a flat
namespace).

Rename snippet* variables to module_utils*

Add a small number of unittests for recursive_finder

Add a larger number of integration tests to demonstrate that
module_utils is working.

Whitelist module-style shebang in test target library dirs

Prefix module_data variable with b_ to be clear that it holds bytes data
8 years ago