* Begin using ArgumentSpecValidator in AnsibleModule
* Add check parameters to ArgumentSpecValidator
Add additional parameters for specifying required and mutually exclusive parameters.
Add code to the .validate() method that runs these additional checks.
* Make errors related to unsupported parameters match existing behavior
Update the punctuation in the message slightly to make it more readable.
Add a property to ArgumentSpecValidator to hold valid parameter names.
* Set default values after performining checks
* FIx sanity test failure
* Use correct parameters when checking sub options
* Use a dict when iterating over check functions
Referencing by key names makes things a bit more readable IMO.
* Fix bug in comparison for sub options evaluation
* Add options_context to check functions
This allows the parent parameter to be added the the error message if a validation
error occurs in a sub option.
* Fix bug in apply_defaults behavior of sub spec validation
* Accept options_conext in get_unsupported_parameters()
If options_context is supplied, a tuple of parent key names of unsupported parameter will be
created. This allows the full "path" to the unsupported parameter to be reported.
* Build path to the unsupported parameter for error messages.
* Remove unused import
* Update recursive finder test
* Skip if running in check mode
This was done in the _check_arguments() method. That was moved to a function that has no
way of calling fail_json(), so it must be done outside of validation.
This is a silght change in behavior, but I believe the correct one.
Previously, only unsupported parameters would cause a failure. All other checks would not be executed
if the modlue did not support check mode. This would hide validation failures in check mode.
* The great purge
Remove all methods related to argument spec validation from AnsibleModule
* Keep _name and kind in the caller and out of the validator
This seems a bit awkward since this means the caller could end up with {name} and {kind} in
the error message if they don't run the messages through the .format() method
with name and kind parameters.
* Double moustaches work
I wasn't sure if they get stripped or not. Looks like they do. Neat trick.
* Add changelog
* Update unsupported parameter test
The error message changed to include name and kind.
* Remove unused import
* Add better documentation for ArgumentSpecValidator class
* Fix example
* Few more docs fixes
* Mark required and mutually exclusive attributes as private
* Mark validate functions as private
* Reorganize functions in validation.py
* Remove unused imports in basic.py related to argument spec validation
* Create errors is module_utils
We have errors in lib/ansible/errors/ but those cannot be used by modules.
* Update recursive finder test
* Move errors to file rather than __init__.py
* Change ArgumentSpecValidator.validate() interface
Raise AnsibleValidationErrorMultiple on validation error which contains all AnsibleValidationError
exceptions for validation failures.
Return the validated parameters if validation is successful rather than True/False.
Update docs and tests.
* Get attribute in loop so that the attribute name can also be used as a parameter
* Shorten line
* Update calling code in AnsibleModule for new validator interface
* Update calling code in validate_argument_spec based in new validation interface
* Base custom exception class off of Exception
* Call the __init__ method of the base Exception class to populate args
* Ensure no_log values are always updated
* Make custom exceptions more hierarchical
This redefines AnsibleError from lib/ansible/errors with a different signature since that cannot
be used by modules. This may be a bad idea. Maybe lib/ansible/errors should be moved to
module_utils, or AnsibleError defined in this commit should use the same signature as the original.
* Just go back to basing off Exception
* Return ValidationResult object on successful validation
Create a ValidationResult class.
Return a ValidationResult from ArgumentSpecValidator.validate() when validation is successful.
Update class and method docs.
Update unit tests based on interface change.
* Make it easier to get error objects from AnsibleValidationResultMultiple
This makes the interface cleaner when getting individual error objects contained in a single
AnsibleValidationResultMultiple instance.
* Define custom exception for each type of validation failure
These errors indicate where a validation error occured. Currently they are empty but could
contain specific data for each exception type in the future.
* Update tests based on (yet another) interface change
* Mark several more functions as private
These are all doing rather "internal" things. The ArgumentSpecValidator class is the preferred
public interface.
* Move warnings and deprecations to result object
Rather than calling deprecate() and warn() directly, store them on the result object so the
caller can decide what to do with them.
* Use subclass for module arg spec validation
The subclass uses global warning and deprecations feature
* Fix up docs
* Remove legal_inputs munging from _handle_aliases()
This is done in AnsibleModule by the _set_internal_properties() method. It only makes sense
to do that for an AnsibleModule instance (it should update the parameters before performing
validation) and shouldn't be done by the validator.
Create a private function just for getting legal inputs since that is done in a couple of places.
It may make sense store that on the ValidationResult object.
* Increase test coverage
* Remove unnecessary conditional
ci_complete
* Mark warnings and deprecations as private in the ValidationResult
They can be made public once we come up with a way to make them more generally useful,
probably by creating cusom objects to store the data in more structure way.
* Mark valid_parameter_names as private and populate it during initialization
* Use a global for storing the list of additonal checks to perform
This list is used by the main validate method as well as the sub spec validation.
Ansible can gather distribution facts for older Amazon Linux
with /etc/os-release data.
Fixes: #73946
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* finish migrating ssh plugin to config system
fixes#72739fixes#57220
* fix connection detection in reset
* correct options for connection meta reset
Co-authored-by: David Shrewsbury <Shrews@users.noreply.github.com>
* Add more scenarios to basic valid testing
* Update invalid tests
* Fix test for Python 2
* Condense data
* Add tests for missing required and invalid-elements
* Update aliases tests
* Add invalid scenarios for aliases
* Add tests for _add_error() method
* Fix sanity test failure
* Manage the in-memory cache in _call_galaxy but let the caller set the file cache after getting paginated results
* Add a test for caching successful and not caching unsuccessful paginated results
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
Only add data when there is data to add
also avoid clobbering existing data with empty file
fixes#45843
* remove redundant code, update comments
* fix mock dataloader, original does not return None
* added test
* module compat for py3.8+ controller
* replaced internal usages of selinux bindings with internal ctypes binding (allows basic selinux operations from any Python interpreter), plus tests
* added new respawn_module API to allow modules to import Python packages that are only available under a well-known interpreter, plus tests
* added respawn logic to modules that need Python libs from a specific system interpreter (apt, apt_repository, dnf, yum)
minimize internal HAVE_SELINUX usage
spurious junk
pep8
* pylint fixes
* add RHEL8 Python 3.8 testing
* more pylint
* import sanity
* unit tests
* changelog update
* fix a bunch of stuff
* tweak changelog
* fix setup_rpm_repo on EL8
* misc sanity/test fixes
* misc feedback tweaks
* fix import fallback in test module
* fix selinux MU test
* fix dnf tests to avoid python-dependent test packages
* add trailing LFs to aliases
* fix yum tests to avoid test package with Python deps
* hack create_repo for EL6 to create noarch package
* Include all collections in single json object / yaml document
* Add tests
* For galaxy list yaml/json output, use dictionary of dictionaries instead of list
* Add tests for listing single collection in yaml / output format
* --output -> --format
* Add explicit test for listing collection in human format
* Fix bug where empty json object was emitted + add test
Change:
- Remove check that states that only Fedora can be an OSTree
distribution.
- This allows us to correctly return "atomic_container" as the pkg_mgr
fact for RHEL for Edge, Fedora/RHEL/CentOS Atomic Host, etc.
Test Plan:
- Created local RHEL for Edge image and tested against it.
- Tested against regular RHEL 8 and still got `dnf` as expected.
- Tested against RHEL 7 Atomic Host and got `atomic_container` now.
- New unit tests.
Tickets:
- Fixes#73084
Signed-off-by: Rick Elrod <rick@elrod.me>
The `UserFactCollector` queries the user login name via
`getpass.getuser()` and looks up the corresponding entry
in the password database.
The login name may differ from the actual user name,
eg. if the `LOGNAME` env variable is set. The lookup
fails in this case. Added a fallback in this case that
tries to get the entry via the user ID.
PR #72591
This change:
* Adds an artifacts manager that abstracts away extracting the
metadata from artifacts, downloading and caching them in a
temporary location.
* Adds `resolvelib` to direct ansible-core dependencies[0].
* Implements a `resolvelib`-based dependency resolver for
`collection` subcommands that replaces the legacy
in-house code.
This is a dependency resolution library that pip 20.3+ uses
by default. It's now integrated for use for the collection
dependency resolution in ansible-galaxy CLI.
* Refactors of the `ansible-galaxy collection` CLI.
In particular, it:
- reimplements most of the `download`, `install`, `list` and
`verify` subcommands from scratch;
- reuses helper bits previously moved out into external modules;
- replaces the old in-house resolver with a more clear
implementation based on the resolvelib library[0][1][2].
* Adds a multi Galaxy API proxy layer that abstracts accessing the
version and dependencies via API or local artifacts manager.
* Makes `GalaxyAPI` instances sortable.
* Adds string representation methods to `GalaxyAPI`.
* Adds dev representation to `GalaxyAPI`.
* Removes unnecessary integration and unit tests.
* Aligns the tests with the new expectations.
* Adds more tests, integration ones in particular.
[0]: https://pypi.org/p/resolvelib
[1]: https://github.com/sarugaku/resolvelib
[2]: https://pradyunsg.me/blog/2020/03/27/pip-resolver-testing
Co-Authored-By: Jordan Borean <jborean93@gmail.com>
Co-Authored-By: Matt Clay <matt@mystile.com>
Co-Authored-By: Sam Doran <sdoran@redhat.com>
Co-Authored-By: Sloane Hertel <shertel@redhat.com>
Co-Authored-By: Sviatoslav Sydorenko <webknjaz@redhat.com>
Signed-Off-By: Sviatoslav Sydorenko <webknjaz@redhat.com>
Use correct ssh executable and options in all cases on connection plugin
* Also nicer naming/comments
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
The `command` module does not return stdout & sterr when calling
a non existing executable or an unknown exception arises. This fix
lets the module return empty byte strings in those cases.
* Fix YAML error message when error is at the end of the file
If a YAML file fails to load due to a syntax error in a file, or there is an error in the last line of a
file, PyYAML reports the last line number of the file as the index where the error occurred.
When reading the file lines, we use that index to the get the relevant line. If the index value is out
of range, the relevant line is lost for error reporting.
Subtract one from the index value to avoid the IndexError in this specific scenario. It is possible
to still get an IndexError, which will be handled as it is currently.
* Update existing tests and add new tests
Change:
- On CentOS Stream, make distribution_release be "Stream"
- On CentOS Core, it continues to be "Core"
- Implement custom distribution file parser for CentOS, so we can look
for "CentOS Linux" and "CentOS Stream"
- Two new fixtures introduced (CentOS Linux 8.1 and CentOS Stream 8)
- Removed two dicts from `Distribution` class that were seemingly not
used anywhere.
Test Plan:
- ci_complete
- New test fixtures
Tickets:
- Fixes#73027
Signed-off-by: Rick Elrod <rick@elrod.me>
The test currently only expects the literal `ansible` followed
by a semver in the first output line of `ansible --version`.
When running from within a git checkout, additional information
like the currently checkout branch are attached, which lets
the test fail. This commit allows arbitrary information to
follow the semver.
`test_prepare_multipart` fails in non Debian environments since
Debian installations map the file ending `.key` to the MIME type
`application/pgp-keys`, which is not IANA conformant. This commit
explicitly sets the corresponding file type to
`application/octet-stream` and adjusts the expected serialized
result.
Change:
- The FreeBSD release can contain -RC or -PRERELEASE in addition to
-RELEASE, -STABLE, or -CURRENT.
Test Plan:
- Added new fixed from an RC version of TrueNAS which uses a -PRERELEASE
version of FreeBSD.
Tickets:
- Fixes#72331
Signed-off-by: Rick Elrod <rick@elrod.me>
On python3 sys.stdin is an encoded file object that does not support
reading raw binary data. Use the supplied buffer object to do so.
Signed-off-by: Sven Wegener <sven.wegener@inovex.de>
Co-authored-by: Sven Wegener <sven.wegener@inovex.de>
* file: add symlink is in a sticky directory tests
* file: handle symlink in a sticky directory
Co-Authored-By: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
* Add changelog and fix unit test
The builtins import was removed since it was unused, but it is now needed.
* Move _syslog_facitily to __init__
No good reason it should not be set for each object
* Move internal property setting to private method
* Create check_arguments() function
* Remove unused import
* Rename function to better match its behavior
Change the behavior to return a set, either empty or populated, with unsupported keys.
Accept legal_inputs as optional which will not required calling handle_aliases before calling
get_unsupported_parameters().
* Add changelog
* Rework function behavior and documentation
I realized I missed the original intent of this method when moving it to a function. It
is meant to compared the parameter keys to legal inputs always, not compare
parameter keys to argument spec keys, even though the argument spec keys should
be a subset of legal inputs.
* Add tests
* Fix typo.
* Set internal properties when handling suboptions
* Added caching mechanism for Galaxy API requests
* Add cache options and split up code
* Added unit tests
* Fix sanity test
* Use modified date and fix up caching for explicit servers
* Make sure credentials are not in cached server name
* Added test for getting updated cache version
* Changes from review
* Reorder comment postition
* Add comment unit test
* Fix unit test
* Fix unit test
* Add changelog
* Add paramaters which would be problematic without this fix
* Fix typo
* Fix unit test
* Fix unit test
* added timeout options to adhoc and console
* added test
* fix typosesz
* fix conflict
* task_timeout
* fix timeout option, added extra vars to console
* actually use right cli switch .. DUH!
* added timeout to include 'valid' but ignored keys
* fix default
* fixes per review
* fixes for collection playbooks
- add fqcn invocation, also handles extensions
- bring import_playbook into new normal
- avoid adding collection playbooks to adjacency
- deal with extensions or lack of em
- added tests
- fix bugs with multiple playbooks supplied
- unicode/bytes/native fixes
- fix modulenotfound for < py3.7
- tests
* GitHub is removing the underlying API used to implement the `login` command. Since the general consensus seems to be that relatively nobody currently uses this command (in favor of explicit token passing), support was simply removed for interactive login. If a future need arises, this command should be reimplemented via OAuth Device Auth Grants.
* login or role login commands now produce a fatal error with a descriptive message
* updated 2.10 and 2.11 porting guide entries
* remove dead code/config, update messages and porting guides
Change:
- When a plugin defines `type: str` on a parameter, treat more kinds of
input as a string instead of whatever it is parsed as.
Test Plan:
- New unit tests
- CI
Signed-off-by: Rick Elrod <rick@elrod.me>
* Add support for GSSAPI/Kerberos to urls.py
* Test out changes with the latest test container
* Get remote hosts working
* Fix up httptester_krb5_password reader
* Fix tests for opensuse and macOS
* Hopefully last lot of testing changes
* Dont do CBT on macOS
* Fixes from review
Change:
- podman > 2 && < 2.2 does not support "images --format {{json .}}"
- podman also now outputs images JSON differently than docker
- Work around both of the above.
Test Plan:
- Tested with podman 2.0.6 in Fedora 31.
Signed-off-by: Rick Elrod <rick@elrod.me>
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
Change:
- Use `sysctl -n` for openbsd uptime information
- Allow `get_sysctl()` to account for multi-line sysctl settings
- Add unit tests for `get_sysctl()`
Test Plan:
- New unit tests
Tickets:
- Fixes#71968
- Refs #72025
- Refs #72067
Signed-off-by: Rick Elrod <rick@elrod.me>
Co-authored-by: Brian Coca <brian.coca+git@gmail.com>
Co-authored-by: Brian Coca <bcoca@ansible.com>
b6b238a fixed the SLES4SAP detection, which was at this time ok.
Sadly Suse changed with SLES 15 the /etc/os-release file, so the above
change will no longer work.
This commit updates the SLES4SAP detection regarding
https://www.suse.com/support/kb/doc/?id=000019341.
The symlink realpath is matched with endswith, because in SLES 12+ the
link target is SLES_SAP.prod, but in SLES 11 the link target is
SUSE_SLES_SAP.prod.
* Let get_file_attributes() work without `lsattr -v`
Change:
- module_utils's get_file_attributes() expects `lsattr -v` to work, but
in some cases, it may not.
- The function now takes an optional include_version bool parameter,
which removes this expectation.
- Places where we call get_file_attributes() without using the 'version'
it returns, we now call it with include_version=False.
Test Plan:
- New unit tests
Signed-off-by: Rick Elrod <rick@elrod.me>
* fix coverage output from synthetic packages
* synthetic packages (eg, implicit collection packages without `__init__.py`) were always created at runtime with empty string source, which was compiled to a code object and exec'd during the package load. When run with code coverage, it created a bogus coverage entry (since the `__synthetic__`-suffixed `__file__` entry didn't exist on disk).
* modified collection loader `get_code` to preserve the distinction between `None` (eg synthetic package) and empty string (eg empty `__init__.py`) values from `get_source`, and to return `None` when the source is `None`. This allows the package loader to skip `exec`ing things that truly have no source file on disk, thus not creating bogus coverage entries, while preserving behavior and coverage reporting for empty package inits that actually exist.
* add unit test
test_dict_undefined_values is a duplicate of test_dict_undefined_values_is_defined.
It seems originally test_dict_undefined_values was supposed
to do what test_dict_undefined_values_bare does so it is safe to remove.
* Validate salt when using crypt. Respect salt_size in password lookup. Repair salt for bcrypt. Fixes#71107. Fixes#53750. Fixes#36129.
* Handle algorithms we don't know about, and make sure to return the salt
* Account for old passlib
* Add tests for salt constraints
* Add changelog fragment
* Add test for #36129
* Revert "Change default file permissions so they are not world readable (#70221)"
This reverts commit 5260527c4a.
* Revert "Fix warning for new default permissions when mode is not specified (#70976)"
This reverts commit dc79528cc6.
* Fix tty_ify bugs and refactor
* Move tty_ify() and supporting attributes to the DocCLI class as that's
the only thing using it.
* Add unittest for the code.
* Fix a bug where the substitution macros can be detected when they are
a part of another word.
* Add support for L(), R(), and HORIZONTALLINE which were added to the
website docs many years ago.
* Update test/units/cli/test_doc.py
Co-authored-by: Matt Clay <matt@mystile.com>
Co-authored-by: Matt Clay <matt@mystile.com>
Change:
- Use `chmod +a` in the fallback chain to allow MacOS to use ACLs to
allow an unprivileged user to become an unprivileged user.
Test Plan:
- CI, new tests
Tickets:
- Fixes#70648
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Previously CachePluginAdjudicator#flush only removed entries from the
cache backend that it knew about by using them earlier. Now it calls
the underlying plugin's flush() method.
Test Plan:
- New unit tests
Tickets:
- Fixes#68770
Signed-off-by: Rick Elrod <rick@elrod.me>
The iso8601_micro and iso8601 facts incorrectly called now.utcnow(), resulting
in a new timestamp at the time it was called, not a conversion of the previously
stored timestamp.
Correct this by capturing the UTC timestamp once then calculating the local
time using the UTC offset of the current system.
* Use time.time() for getting the current time
* Convert from that stored epoch timestamp to local and UTC times
* Used existing timestamp for epoch time
* Add unit tests that validate the formate of the return value rather than an exact value since mocking time and timezone is non-trivial
Change:
- Clarify that not passing `--tags` will cause `ansible_run_tags` to
default to `["all"]`.
- Add some extra coverage around `ansible_run_tags`
Test Plan:
- New integration and unit tests
Tickets:
- Fixes#69619
Signed-off-by: Rick Elrod <rick@elrod.me>
* Change default file permissions so they are not world readable
CVE-2020-1736
Set the default permissions for files we create with atomic_move() to 0o0660. Track
which files we create that did not exist and warn if the module supports 'mode'
and it was not specified and the module did not call set_mode_if_different(). This allows the user to take action and specify a mode rather than using the defaults.
A code audit is needed to find all instances of modules that call atomic_move()
but do not call set_mode_if_different(). The findings need to be documented in
a changelog since we are not warning. Warning in those instances would be frustrating
to the user since they have no way to change the module code.
- use a set for storing list of created files
- just check the argument spac and params rather than using another property
- improve the warning message to include the default permissions
Change:
- Refactoring to make it harder to get wrong and easier to read.
- Generalize become_unprivileged tests and fix some that never worked
but also never failed.
Test Plan:
- CI, new units/integration tests
Signed-off-by: Rick Elrod <rick@elrod.me>
A couple of years ago Slackware -current began using a plus (“+”) at the end of the distribution version string to indicate a future version work-in-progress.
Rearrange distribution_files unit tests to easily support more tests
- add conftest with common fixtures
- use parametrize for testing multiple scenarios
* Add changelog
* Add unit tests for Slackware distribution parsing
* Use correct fixtures for Slackware
Data comes from /etc/slackware-version
Co-authored-by: Sam Doran <sdoran@redhat.com>
Co-authored-by: <Eduard Rozenberg <eduardr@pobox.com>>
- ensure we preserve the typeerror part of the exception so loop defereed error handling
can postpone those caused by undefined variables until the when check is done.
- fix tests to comply with the 'new normal'
- human_to_bytes and others can issue TypeError not only on 'non string'
but also bad string that is not convertable.
Co-authored-by: Sloane Hertel <shertel@redhat.com>
Co-authored-by: Sloane Hertel <shertel@redhat.com>
CIFS can be mounted using backward slash as well in /etc/fstab like
\\Windows\share /data/ cifs credentials=/root/.creds 0 0
Handle this condition while gather mount information in Linux.
Fixes: #48813
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
Sam Doran