* Allow to specify collection_name separately for deprecation.
* Use new functionality in Ansible.
* Use new functionality in tests.
* Update tagging/untagging functions.
* Update pylint deprecated sanity test.
* Update validate-modules. Missing are basic checks for version_added (validate semantic version format for collections).
* Improve version validation. Re-add version_added validation.
* Make sure collection names are added to return docs before schema validation.
* Extra checks to avoid crashes on bad data.
* Make C# module utils code work, and update/extend tests.
* Add changelog fragment.
* Stop extracting collection name from potentially tagged versions/dates.
* Simplify C# code.
* Update Windows modules docs.
* Forgot semicolons.
To avoid issues with Flatcar Container Linux being unable to be found,
detect Flatcar distro name especially for hostname, just like CoreOS
Container Linux was supported.
See also https://github.com/ansible/ansible/issues/69516
* Track collection for version_added.
Validate *all* version numbers in validate-modules.
For tagged version numbers (i.e. version_added), consider source collection to chose validation.
* Make tagging/untagging functions more flexible.
* Tag all versions in doc fragments.
* Tag all deprecation versions issued by code.
* Make Display.deprecated() understand tagged versions.
* Extend validation to enforce tagged version numbers.
* Tag versions in tests.
* Lint and fix test.
* Mention collection name in collection loader's deprecation/removal messages.
* Fix error IDs.
* Handle tagged dates in Display.deprecated().
* Also require that removed_at_date and deprecated_aliases.date are tagged.
* Also automatically tag/untag removed_at_date; fix sanity module removal version check.
* Improve error message when invalid version number is used (like '2.14' in collections).
* Allow to deprecate options and aliases by date instead of only by version.
* Update display.deprecate().
* Adjust behavior to conform to tested behavior, extend tests, and improve C# style.
* Parse date and fail on invalid date.
This is mainly to make sure that people start using invalid dates, and we eventually have a mess to clean up.
* C# code: improve validation and update/extend tests.
* Make sure that deprecate() is not called with both date and version.
* Forgot to remove no longer necessary formatting.
* Adjust order of warnings in C# code.
* Adjust unrelated test.
* Fix grammar (and make that test pass).
* Don't parse date, and adjust message to be same as in #67684.
* Sanity tests: disable date in past test.
* Validate-modules: validate ISO 8601 date format.
* Validate-modules: switch schema declaration for deprecated_aliases to improve error messages for invalid dates.
* Use DateTime instead of string for date deprecation.
* Validate that date in deprecated_aliases is actually a DateTime.
* Fix tests.
* Fix rebasing error.
* Adjust error codes for pylint, and add removed_at_date and deprecated_aliases.date checks to validate-modules.
* Make deprecation date in the past error codes optional.
* Make sure not both version and date are specified for AnsibleModule.deprecate() calls.
* Stop using Python 3.7+ API.
* Make sure errors are actually reported. Re-add 'ansible-' prefix.
* Avoid crashing when 'name' isn't there.
* Linting.
* Update lib/ansible/module_utils/csharp/Ansible.Basic.cs
Co-authored-by: Jordan Borean <jborean93@gmail.com>
* Adjust test to latest change.
* Prefer date over version if both end up in Display.deprecated().
Co-authored-by: Jordan Borean <jborean93@gmail.com>
* Fix filedescriptor out of range in select() when running commands
* Simplify the run_command() code
Now that we're using selectors in run_command(), we can simplify some of
the code.
* Use fileobj.read() instead of os.read()
* No longer use get_buffer_size() as we can just slurp all of the data
instead.
Also use a simpler conditional check of whether the selector map is
empty
Co-authored-by: Toshio Kuratomi <a.badger@gmail.com>
fial_json() requires a message be given to it to inform the end user of
why the module failed. Prior to this commit, the message had to be a
keyword argument:
module.fail_json(msg='Failed due to error')
Since this is a required parameter, this commit allows the message to be
given as a positional argument instead:
module.fail_json('Failed due to an error')
* Move warn() and deprecate() methods out of basic.py
* Use _global_warnings and _global_deprications and create accessor functions
- This lays the foundation for future functions being moved outside of AnsibleModule
that need an interface to warnings and deprecations without modifying them.
* Add unit tests for new warn and deprecate functions
On Python 2, leave all fds open since there is no mechanism to close specific fds with subprocess.Popen() on Python 2
Add unit tests.
Co-authored-by: Matt Martz <matt@sivel.net>
As AnsibleModule._log_invocation is currently implemented, any parameter
with a name that matches PASSWORD_MATCH triggers the no_log warning as a
precaution against parameters that may contain sensitive data, but have not
been marked as sensitive by the module author.
This patch would allow module authors to explicitly mark the aforementioned
parameters as not sensitive thereby bypassing an erroneous warning message,
while still catching parameters which have not been marked at all by the
author.
Adds tests for various no_log states including True, False, and None (as
extracted by AnsibleModule._log_invocation) when applied to an argument with
a name that matches PASSWORD_MATCH.
Fixes: #49465#64656
* Print warning when both an option and its alias is specified.
* Improve output.
* Put warnings into self._warnings directly, resp. use self.warn() when handling subspecs.
* Add changelog.
* Add unit test.
* Move check_type_str() out of basic.py
* Move check_type_list() out of basic.py
* Move safe_eval() out of basic.py
* Move check_type_dict() out of basic.py
* Move json importing code to common location
* Move check_type_bool() out of basic.py
* Move _check_type_int() out of basic.py
* Move _check_type_float() out of basic.py
* Move _check_type_path() out of basic.py
* Move _check_type_raw() out of basic.py
* Move _check_type_bytes() out of basic.py
* Move _check_type_bits() out of basic.py
* Create text.formatters.py
Move human_to_bytes, bytes_to_human, and _lenient_lowercase out of basic.py into text.formatters.py
Change references in modules to point to function at new location
* Move _check_type_jsonarg() out of basic.py
* Rename json related functions and put them in common.text.converters
Move formatters.py to common.text.formatters.py and update references in modules.
* Rework check_type_str()
Add allow_conversion option to make the function more self-contained.
Move the messaging back to basic.py since those error messages are more relevant to using this function in the context of AnsibleModule and not when using the function in isolation.
* Add unit tests for type checking functions
* Change _lenient_lowercase to lenient_lowercase per feedback
* Rename method and make private
* Use is_iterable, combine transformations
* Remove unused return_values from network modules
* Improve docstrings in new functions
* Add new PASS_VAR
* Add unit tests for list_no_log_values
* Fix unit tests for Python 2.6
Refinements:
- return legal_inputs and update class properties
- remove redundant arguments from method and handle in caller
- add better exception types to method
* Add unit tests for handle_aliases
* Add support for elements validation in argspec
Fixes#48473
* Add support to validate the elements value in argspec
when type is `list`
* Fix unit test failures
* Add unit test for elements validation
* Fix CI failures
* Fix review comments
* Fix unit test and CI failures after rebase
* Introduce new "required_by' argument_spec option
This PR introduces a new **required_by** argument_spec option which allows you to say *"if parameter A is set, parameter B and C are required as well"*.
- The difference with **required_if** is that it can only add dependencies if a parameter is set to a specific value, not when it is just defined.
- The difference with **required_together** is that it has a commutative property, so: *"Parameter A and B are required together, if one of them has been defined"*.
As an example, we need this for the complex options that the xml module provides. One of the issues we often see is that users are not using the correct combination of options, and then are surprised that the module does not perform the requested action(s).
This would be solved by adding the correct dependencies, and mutual exclusives. For us this is important to get this shipped together with the new xml module in Ansible v2.4. (This is related to bugfix https://github.com/ansible/ansible/pull/28657)
```python
module = AnsibleModule(
argument_spec=dict(
path=dict(type='path', aliases=['dest', 'file']),
xmlstring=dict(type='str'),
xpath=dict(type='str'),
namespaces=dict(type='dict', default={}),
state=dict(type='str', default='present', choices=['absent',
'present'], aliases=['ensure']),
value=dict(type='raw'),
attribute=dict(type='raw'),
add_children=dict(type='list'),
set_children=dict(type='list'),
count=dict(type='bool', default=False),
print_match=dict(type='bool', default=False),
pretty_print=dict(type='bool', default=False),
content=dict(type='str', choices=['attribute', 'text']),
input_type=dict(type='str', default='yaml', choices=['xml',
'yaml']),
backup=dict(type='bool', default=False),
),
supports_check_mode=True,
required_by=dict(
add_children=['xpath'],
attribute=['value', 'xpath'],
content=['xpath'],
set_children=['xpath'],
value=['xpath'],
),
required_if=[
['count', True, ['xpath']],
['print_match', True, ['xpath']],
],
required_one_of=[
['path', 'xmlstring'],
['add_children', 'content', 'count', 'pretty_print', 'print_match', 'set_children', 'value'],
],
mutually_exclusive=[
['add_children', 'content', 'count', 'print_match','set_children', 'value'],
['path', 'xmlstring'],
],
)
```
* Rebase and fix conflict
* Add modules that use required_by functionality
* Update required_by schema
* Fix rebase issue
* Move get_all_subclasses out of sys_info as it is unrelated to system
information.
* get_all_subclasses now returns a set() instead of a list.
* Don't port get_platform to sys_info as it is deprecated. Code using
the common API should just use platform.system() directly.
* Rename load_platform_subclass() to get_platform_subclass and do not
instantiate the rturned class.
* Test the compat shims in module_utils/basic.py separately from the new
API in module_utils/common/sys_info.py and module_utils/common/_utils.py
* Move ansible.compat.tests to test/units/compat/.
* Fix unit test references to ansible.compat.tests.
* Move builtins compat to separate file.
* Fix classification of test/units/compat/ dir.
* Remove use of simplejson throughout code base. Fixes#42761
* Address failing tests
* Remove simplejson from contrib and other outlying files
* Add changelog fragment for simplejson removal
Now that we don't need to worry about python-2.4 and 2.5, we can make
some improvements to the way AnsiballZ handles modules.
* Change AnsiballZ wrapper to use import to invoke the module
We need the module to think of itself as a script because it could be
coded as:
main()
or as:
if __name__ == '__main__':
main()
Or even as:
if __name__ == '__main__':
random_function_name()
A script will invoke all of those. Prior to this change, we invoked
a second Python interpreter on the module so that it really was
a script. However, this means that we have to run python twice (once
for the AnsiballZ wrapper and once for the module). This change makes
the module think that it is a script (because __name__ in the module ==
'__main__') but it's actually being invoked by us importing the module
code.
There's three ways we've come up to do this.
* The most elegant is to use zipimporter and tell the import mechanism
that the module being loaded is __main__:
* 5959f11c9d/lib/ansible/executor/module_common.py (L175)
* zipimporter is nice because we do not have to extract the module from
the zip file and save it to the disk when we do that. The import
machinery does it all for us.
* The drawback is that modules do not have a __file__ which points
to a real file when they do this. Modules could be using __file__
to for a variety of reasons, most of those probably have
replacements (the most common one is to find a writable directory
for temporary files. AnsibleModule.tmpdir should be used instead)
We can monkeypatch __file__ in fom AnsibleModule initialization
but that's kind of gross. There's no way I can see to do this
from the wrapper.
* Next, there's imp.load_module():
* https://github.com/abadger/ansible/blob/340edf7489/lib/ansible/executor/module_common.py#L151
* imp has the nice property of allowing us to set __name__ to
__main__ without changing the name of the file itself
* We also don't have to do anything special to set __file__ for
backwards compatibility (although the reason for that is the
drawback):
* Its drawback is that it requires the file to exist on disk so we
have to explicitly extract it from the zipfile and save it to
a temporary file
* The last choice is to use exec to execute the module:
* https://github.com/abadger/ansible/blob/f47a4ccc76/lib/ansible/executor/module_common.py#L175
* The code we would have to maintain for this looks pretty clean.
In the wrapper we create a ModuleType, set __file__ on it, read
the module's contents in from the zip file and then exec it.
* Drawbacks: We still have to explicitly extract the file's contents
from the zip archive instead of letting python's import mechanism
handle it.
* Exec also has hidden performance issues and breaks certain
assumptions that modules could be making about their own code:
http://lucumr.pocoo.org/2011/2/1/exec-in-python/
Our plan is to use imp.load_module() for now, deprecate the use of
__file__ in modules, and switch to zipimport once the deprecation
period for __file__ is over (without monkeypatching a fake __file__ in
via AnsibleModule).
* Rename the name of the AnsiBallZ wrapped module
This makes it obvious that the wrapped module isn't the module file that
we distribute. It's part of trying to mitigate the fact that the module
is now named __main)).py in tracebacks.
* Shield all wrapper symbols inside of a function
With the new import code, all symbols in the wrapper become visible in
the module. To mitigate the chance of collisions, move most symbols
into a toplevel function. The only symbols left in the global namespace
are now _ANSIBALLZ_WRAPPER and _ansiballz_main.
revised porting guide entry
Integrate code coverage collection into AnsiballZ.
ci_coverage
ci_complete
d7df072b96 changed how we call
journal.send() from positional arguments to keyword arguments. So we
need to update the test to check for the arguments it was called with in
the keyword args, not in the positional args.
* Fix tmpdir on non root become
- also avoid exception if tmpdir and remote_tmp are None
- give 'None' on deescalation so tempfile will fallback to it's default behaviour
and use system dirs
- fix issue with bad tempdir (not existing/not createable/not writeable)
i.e nobody and ~/.ansible/tmp
- added tests for blockfile case
* Revert "Temporarily revert c119d54"
This reverts commit 5c614a59a6.
* changes based on PR feedback and changelog fragment
* changes based on the review
* Fix tmpdir when makedirs failed so we just use the system tmp
* Let missing remote_tmp fail
If remote_tmp is missing then there's something more basic wrong in the
communication from the controller to the module-side. It's better to
be alerted in this case than to silently ignore it.
jborean and I have independently checked what happens if the user sets
ansible_remote_tmp to empty string and !!null and both cases work fine.
(null is turned into a default value controller-side. empty string
triggers the warning because it is probably not a directory that the
become user is able to use).
* create module tmpdir based on remote_tmp
* Source remote_tmp from controller if possible
* Fixed sanity test and not use lambda
* Added expansion of env vars to the remote tmp
* Fixed sanity issues
* Added note around shell remote_tmp option
* Changed fallback tmp dir to ~/.ansible/tmp to make shell defaults
* Allow subspec defaults to be processed when the parent argument is not supplied
* Allow this to be configurable via apply_defaults on the parent
* Document attributes of arguments in argument_spec
* Switch manageiq_connection to use apply_defaults
* add choices to api_version in argument_spec
* basic: allow one or more when param list having choices
* add unit tests
* optimize a bit
* re-add get_exception import
* a number of existing modules expect to be able to get it from basic.py
* basic.py: add mock to os.path.exists
* set_*_if_different: if check_mode enabled & file missing: set changed to True
Fixes#32676
Thanks to mscherer and Spredzy for the distributed triplet programming
session!
Split the one monolithic test for basic.py into several files
* Split test_basic.py along categories.
This is preliminary to get a handle on things. Eventually we may want
to further split it so each file is only testing a single function.
* Cleanup unused imports from splitting test_basic.py
* Port atomic_move test to pytest.
Working on getting rid of need to maintain procenv
* Split a test of symbolic_mode_to_octal to follow unittest best practices
Each test should only invoke the function under test once
* Port test_argument_spec to pytest.
* Fix suboptions failure
* Porting tests to pytest
* Achievement Get: No longer need mock/generator.py
* Now done via pytest's parametrization
* Port safe_eval to pytest
* Port text tests to pytest
* Port test_set_mode_if_different to pytest
* Change conftest AnsibleModule fixtures to be more flexible
* Move the AnsibleModules fixtures to module_utils/conftest.py for sharing
* Testing the argspec code requires:
* injecting both the argspec and the arguments.
* Patching the arguments into sys.stdin at a different level
* More porting to obsolete mock/procenv.py
* Port run_command to pytest
* Port known_hosts tests to pytest
* Port safe_eval to pytest
* Port test_distribution_version.py to pytest
* Port test_log to pytest
* Port test__log_invocation to pytest
* Remove unneeded import of procenv in test_postgresql
* Port test_pip to pytest style
* As part of this, create a pytest ansiblemodule fixture in
modules/conftest.py. This is slightly different than the
approach taken in module_utils because here we need to override the
AnsibleModule that the modules will inherit from instead of one that
we're instantiating ourselves.
* Fixup usage of parametrization in test_deprecate_warn
* Check that the pip module failed in our test
Matt Clay