PR #72591
This change:
* Adds an artifacts manager that abstracts away extracting the
metadata from artifacts, downloading and caching them in a
temporary location.
* Adds `resolvelib` to direct ansible-core dependencies[0].
* Implements a `resolvelib`-based dependency resolver for
`collection` subcommands that replaces the legacy
in-house code.
This is a dependency resolution library that pip 20.3+ uses
by default. It's now integrated for use for the collection
dependency resolution in ansible-galaxy CLI.
* Refactors of the `ansible-galaxy collection` CLI.
In particular, it:
- reimplements most of the `download`, `install`, `list` and
`verify` subcommands from scratch;
- reuses helper bits previously moved out into external modules;
- replaces the old in-house resolver with a more clear
implementation based on the resolvelib library[0][1][2].
* Adds a multi Galaxy API proxy layer that abstracts accessing the
version and dependencies via API or local artifacts manager.
* Makes `GalaxyAPI` instances sortable.
* Adds string representation methods to `GalaxyAPI`.
* Adds dev representation to `GalaxyAPI`.
* Removes unnecessary integration and unit tests.
* Aligns the tests with the new expectations.
* Adds more tests, integration ones in particular.
[0]: https://pypi.org/p/resolvelib
[1]: https://github.com/sarugaku/resolvelib
[2]: https://pradyunsg.me/blog/2020/03/27/pip-resolver-testing
Co-Authored-By: Jordan Borean <jborean93@gmail.com>
Co-Authored-By: Matt Clay <matt@mystile.com>
Co-Authored-By: Sam Doran <sdoran@redhat.com>
Co-Authored-By: Sloane Hertel <shertel@redhat.com>
Co-Authored-By: Sviatoslav Sydorenko <webknjaz@redhat.com>
Signed-Off-By: Sviatoslav Sydorenko <webknjaz@redhat.com>
While getting policy name in get_chain_policy API,
module does not require any additional parameters except chain
Enabling flag in get_chain_policy API call fixes this.
Fixes: #68612
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
Use correct ssh executable and options in all cases on connection plugin
* Also nicer naming/comments
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
Ability to add local variables into AnsibleJ2Vars was added in
18a9eff11f to fix#6653. Local variables
are added using ``AnsibleJ2Vars.add_locals()`` method when creating a
new context - typically when including/importing a template with
context. For that use case local template variables created using
``set`` should override variables from higher contexts - either from the
play or any parent template, or both; Jinja behaves the same way.
Also removes AnsibleJ2Vars.extras instance variable which is not used.
Also adds missing test for #6653.
Fixes#72262Fixes#72615
ci_complete
The `command` module does not return stdout & sterr when calling
a non existing executable or an unknown exception arises. This fix
lets the module return empty byte strings in those cases.
* Consolidate filters/tests handling into JinjaPluginIntercept
ci_complete
* Postpone loading all ansible plugins
* Do we need to create an overlay?
ci_complete
* Typo
ci_complete
* Add FIXME
* conditional.py: use public Environment.parse() method
* Remove remaining occurrences of shared_loader_obj being passed to Templar
* __UNROLLED__ not needed with this change anymore
* Incorrect rebase at some point?
When calling compile(), the filename argument should be either a real file
name or a string. According to Python docs, suggested one is '<string>'.
Keep the current behaviour (encapsulate the actual expression),
but enclose it into angle brackets.
Signed-off-by: Oldřich Jedlička <oldium.pro@gmail.com>
* Fix YAML error message when error is at the end of the file
If a YAML file fails to load due to a syntax error in a file, or there is an error in the last line of a
file, PyYAML reports the last line number of the file as the index where the error occurred.
When reading the file lines, we use that index to the get the relevant line. If the index value is out
of range, the relevant line is lost for error reporting.
Subtract one from the index value to avoid the IndexError in this specific scenario. It is possible
to still get an IndexError, which will be handled as it is currently.
* Update existing tests and add new tests
* Allow restricting config values to enumerated list
* dont document internal entries
* also ignore private defs for ansible-config
* remove invalid value from tests
* added porting entry
When the pause module is run in the background and seconds parameter is provided,
do not warn.
* Add tests
* Fix existing tests
The test wasn't failing when it should have.
Change:
- On CentOS Stream, make distribution_release be "Stream"
- On CentOS Core, it continues to be "Core"
- Implement custom distribution file parser for CentOS, so we can look
for "CentOS Linux" and "CentOS Stream"
- Two new fixtures introduced (CentOS Linux 8.1 and CentOS Stream 8)
- Removed two dicts from `Distribution` class that were seemingly not
used anywhere.
Test Plan:
- ci_complete
- New test fixtures
Tickets:
- Fixes#73027
Signed-off-by: Rick Elrod <rick@elrod.me>
Do the right thing on Linux when password lock and a password hash are provided by writing
out the password hash prepended by the appropriate lock string rather than using -U and -L.
This is the correct way to set and lock the account in one command.
On BSD, run separate commands as appropriate since locking and setting the password cannot
be done in a single action.
FreeBSD requires running several commands to get the account in the desired state. As a result,
the rc, output, and error from all commands need to be combined and evaluated so an accurate
and complete summary can be given at the end of module execution.
* Improve integration tests to cover this scenario.
* Break up user integration tests into smaller files
* Properly lock account when creating a new account and password is supplied
* Simplify rc collection in FreeBSD class
Since the _handle_lock() method was added, the rc would be set to None, which could make
task change reporting incorrect. My first attempt to solve this used a set and was a bit too
complicated. Simplify it my comparing the rc from _handle_lock() and the current value of rc.
* Improve the Linux password hash and locking behavior
If password lock and hash are provided, set the hash and lock the account by using a password
hash since -L cannot be used with -p.
* Ensure -U and -L are not combined with -p since they are mutually exclusive to usermod.
* Clarify password_lock behavior.
The old example failed with:
fatal: [localhost]: FAILED! => {"msg": "to count backwards make stride negative"}
I fixed this and placed a start parameter to consolidate with the rest of the examples provided.
Co-authored-by: Jorge Rua <jorua@redhat.com>
Change:
- Internally, use dnf.subject.Subject#get_best_query for state: absent
- Add a bunch of tests for removing packages, given a bunch of different
pkg specs (nv, nvr, nvra, wildcard, etc.)
Test Plan:
- New tests
- Local experiments with DNF API via PDB.
Tickets:
- Fixes#72809
Signed-off-by: Rick Elrod <rick@elrod.me>
* Replace the use of the function string.replace() with the method str.replace()
because that's what works for both Python 2 and 3.
* Cleanup the unused string import.
Added a changelog fragment.
* The documentation for os.write() seems a bit iffy, but in Python 3 we
definitely cannot pass it a string and we need to encode it into
bytes.
The Python documentation at
https://docs.python.org/3/howto/pyporting.html#text-versus-binary-data
says that this code will work in Python 2 as well.
Co-authored-by: Lloyd Parkes <lloyd@must-have-coffee.gen.nz>
Change:
- The FreeBSD release can contain -RC or -PRERELEASE in addition to
-RELEASE, -STABLE, or -CURRENT.
Test Plan:
- Added new fixed from an RC version of TrueNAS which uses a -PRERELEASE
version of FreeBSD.
Tickets:
- Fixes#72331
Signed-off-by: Rick Elrod <rick@elrod.me>
* arg_spec - move type checking lookup method to a function
* Change get_wanted_type name and behavior
Change the name to get_validator to bette describe what it is doing.
Change the interface to always return a value. This lines up with the behavior of get_*
functions always returning something or None and check_* functions raising an
Exception if something went wrong during the check.
* Add param to check_type_str()
Not meant to be a long term fix, but gets tests passing. More work is needed to figure
out how to solve this cleanly.
* Remove private attribute mapping types to validator
Since the function that needs it has moved to parameters.py, there is no need to have it as
a attribute of AnsibleModule.
Update tests that were referencing the private attribute.
* Use private method for 'str' type
To avoid having to put the string conversion warning behavior in the check_type_str() method,
use the private _check_type_str() method for 'str' type.
Import CHECK_ARGUMENT_TYPES_DISPATCHER for backwards compalitibility and store it as
a private attribute.
Revert changes to support plugins that are referencing serf._CHECK_ARGUMENT_TYPES_DISPATCHER.
* Add changelog
* Change function name to better reflect its... function
* Change dict name to better reflect its contents
CHECK_ARGUMENT_TYPES_DISPATCHER --> DEFAULT_TYPE_VALIDATORS
* Fix changelog
* systemd - do not overwrite unit name when searching
PR #72702 introduced a bug that changed the unit name when splitting it up for the purpose
of searching for the unit. This only happens on unit file templates on systems that have a 5.8
or newer kernel and a version of systemd that does not contain a bugfix that causes systmed
to fail to parse dbus.
* Use facts rather than a manual probe to determine if systmed is present
* Remov unnecessary block
* Use vars files instead of set_fact
* Add tests for using a templated unit file
* Update changelog fragment
* Use template to get correct path to sleep binary
* import_playbook - change additional params to deprecation
I incorrectly recommended this be set as a warning when it should have been a deprecation.
* Fix deprecation sanity test to not required a collection name when not inside a collection
* Change role argspec file used by ansible-doc
This changes the file used for role argument specs from meta/argument_specs.yml
to meta/main.yml. The argument specs are expected to be in that file under the
top-level entry of `argument_spec`.
* Switch to argument_specs
* Update yum/dnf module docs to include version comp
Fixes#61234
Signed-off-by: Adam Miller <admiller@redhat.com>
* fix up docs syntax to sanity tests are happy
Signed-off-by: Adam Miller <admiller@redhat.com>
* ansible-galaxy collection list and verify now utilize collections in site-packages.
This is a short term fix for #70147. The long term fix needs to handle
install (but that discussion is also bound up in how upgrade is going to
work and where things can get installed so it's deferred for 2.11.)
* Add test for ansible-galaxy collection list with site-packages
* Fix sanity issue
Co-authored-by: David Moreau Simard <moi@dmsimard.com>
Co-authored-by: Jordan Borean <jborean93@gmail.com>
On python3 sys.stdin is an encoded file object that does not support
reading raw binary data. Use the supplied buffer object to do so.
Signed-off-by: Sven Wegener <sven.wegener@inovex.de>
Co-authored-by: Sven Wegener <sven.wegener@inovex.de>
This should allow users to extract specific files from an archive as
desired.
Fixes#16130, #27081.
* Rebase and make a few minor changes
* Add changelog
* Improve tests
- move to separate tasks file
- change assertions to check for exactly one file
- use remote_tmp_dir for output dir
* Make exclude and include mutually exclusive
* Don't remove files needed by other tasks
* Fix sanity tests
* Improve feature documentation
* Skip tests that use map() on CentOS 6
* Use fnmatch on include for zip archives
This matches the behavior of exclude
Co-authored-by: Sam Doran <sdoran@redhat.com>
* Remove space before comma in '... bla , use ...'
* 'why' is inserted in the middle of a sentence, between two commas.
* Make deprecations from base.yml show source ansible-core.
* Add changelog fragment.
* Improve some more 'why's.
* Add PR URL to fragment.
* file: add symlink is in a sticky directory tests
* file: handle symlink in a sticky directory
Co-Authored-By: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
* Add changelog and fix unit test
The builtins import was removed since it was unused, but it is now needed.
This is in reference to the issue: https://github.com/ansible/ansible/issues/41130 (Cannot decrypt with copy module when remote_src: yes)
Autodecryption of the vaulted files does not work when we want to copy files with remote_src=yes.
Co-authored-by: ziaurrehman72 <ziaurrheman72@yahoo.com>
* Move _syslog_facitily to __init__
No good reason it should not be set for each object
* Move internal property setting to private method
* Create check_arguments() function
* Remove unused import
* Rename function to better match its behavior
Change the behavior to return a set, either empty or populated, with unsupported keys.
Accept legal_inputs as optional which will not required calling handle_aliases before calling
get_unsupported_parameters().
* Add changelog
* Rework function behavior and documentation
I realized I missed the original intent of this method when moving it to a function. It
is meant to compared the parameter keys to legal inputs always, not compare
parameter keys to argument spec keys, even though the argument spec keys should
be a subset of legal inputs.
* Add tests
* Fix typo.
* Set internal properties when handling suboptions
* Support listing roles in text and JSON
* Change tests for unfrack'd playbook_dir var
These tests were using '/tmp' for testing the setting of the playbook_dir
var. Now that we unfrack that var, MacOS will change this to '/private/tmp'
causing the tests to fail. We can choose a path that does not exist (since
unfrack does not validate existence) so that we can guarantee unfracking
will not change the value.
Change:
- Initial set of changes for renaming to ansible-core
- Includes changelog fragment changes from base -> core
- Does NOT include docs changes
- Modifies detection stuff in setup.py to support ansible<2.9 and ansible-base
Test Plan:
- ci_complete
* Fix type of mode field in copy module
* Remove module type checking from ignore.txt sanity checks
* Remove mode type
Co-authored-by: Matt Martz <matt@sivel.net>
* Fix super annoying Python 2.6 multiprocessing.Queue stack trace in CI
A bug exists in Python 2.6 that sometimes raises an exception during interpreter shutdown. We
encounter this frequently in our CI since we run tests on CentOS 6 as the control node, which
has Python 2.6.6 with this bug.
This PR adds a very minor sleep only on Python 2.6 which gets around this issue. I did lot of testing
using a standalon script I found that easily duplicated the issue to find the minimum sleep value
needed to avoid this issue.
CPython issue: https://bugs.python.org/issue4106
Fix in CPython: https://hg.python.org/cpython/rev/d316315a8781
* Use correct attribute
* Consolidate logic for determining whether or not session is interactive
into a single function, is_interactive()
* Increase test coverage
I wasn't able to find a good way of simulating running a backgrounded test with CI since the
whole test is essentially run not in a TTY, which is similar enough to cause the new is_interactive()
function to always return false.
* add return documentation for unarchive, add fragment for changelog, apply suggestions from code review
Co-authored-by: Amin Vakil <info@aminvakil.com>
* Clarify collection paths in docs, fix rst syntax, rephrase docs about using collection from folder/local git
Co-authored-by: Felix Fontein <felix@fontein.de>
* Added caching mechanism for Galaxy API requests
* Add cache options and split up code
* Added unit tests
* Fix sanity test
* Use modified date and fix up caching for explicit servers
* Make sure credentials are not in cached server name
* Added test for getting updated cache version
* Changes from review
* Reorder comment postition
* Add comment unit test
* Fix unit test
* Fix unit test
* Add changelog
* Add paramaters which would be problematic without this fix
* Fix typo
* Fix unit test
* Fix unit test
* Emit warning when running on the controller with a Python older than 3.8
* Add spaces
Co-authored-by: Matt Clay <matt@mystile.com>
* and more spaces
Co-authored-by: Matt Clay <matt@mystile.com>
* s/Py/Python/
Co-authored-by: Matt Clay <matt@mystile.com>
* Add note to Control node requirements about Py3.8 requirement
* Add collection_name to deprecated call
* more spaces
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
* Clarify that we are only packaging for py3.8+
Co-authored-by: Matt Clay <matt@mystile.com>
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
* Fix 'role_name : tast_name' notation if task contains role name
* Add tests for notifying handler names which contain the role name
Co-authored-by: Thomas Wouters <thomaswouters@gmail.com>
* Support docker and k8s action groups for moved modules in community.docker and community.kubevirt.
* Also support k8s action group for community.okd.
* Also add kubernetes.core.
* Adjust PR #.
* Fix changelog fragment.
* Remove community.okd.
* Revert "Remove community.okd."
This reverts commit 812b5aa6e2.
Change:
- Docs: Add note that security/bugfix apply to dependencies too, like
the dnf command.
- dnf: security/bugfix only makes sense for updates, so limit the
package query sack to available updates.
- tests: Limit tests to our known-good test packages, so that RHEL
packages marked security/bugfix without similarly marked dependencies
don't fail our tests.
Test Plan:
- Tested with `dnf upgrade-minimal --bugfix` and reproduced the same
error currently seen in CI, showing that we are consistent with what
dnf does.
Tickets:
- Likely fixes#72316
Signed-off-by: Rick Elrod <rick@elrod.me>
* Indent Jinja2 directives.
* Show version_added and deprecations for ini settings and environment variables.
* Make default conditional, add choices. Copied from #55474.
* Add Ansible variables. Copied from #55474.
* Turn ini keys and environment variables into lists. Copied from #55474.
* Improve formatting. Copied from #55474.
* hint in doc of git module to abort on missing http password
improves #69489
* fix syntax in doc of git module
Co-authored-by: Matt Martz <matt@sivel.net>
Co-authored-by: Matt Martz <matt@sivel.net>
* ansible-doc: export has_action when --json is used.
* Remove docuri and now_data, which were not used resp. ignored in format_plugin_doc and the functions it calls anyway.
* Add function _combine_plugin_doc.
* added timeout options to adhoc and console
* added test
* fix typosesz
* fix conflict
* task_timeout
* fix timeout option, added extra vars to console
* actually use right cli switch .. DUH!
* added timeout to include 'valid' but ignored keys
* fix default
* fixes per review
* fixes for collection playbooks
- add fqcn invocation, also handles extensions
- bring import_playbook into new normal
- avoid adding collection playbooks to adjacency
- deal with extensions or lack of em
- added tests
- fix bugs with multiple playbooks supplied
- unicode/bytes/native fixes
- fix modulenotfound for < py3.7
- tests
* Add DarwinStrategy class and integration tests
macOS has three seprate hostname params that need to be set. One of those params, LocalHostName, has more stringent requirements than the other two, which accept special characters and spaces. Create a method to scrub the hostname to ensure it works well with the system requirements.
* Update documentation
* Account for virtualization type returned on Azure Pipelines
* Do not be dependent on order of self.name_types
Use the scrubbed name when the name type is LocalHostName
A bug existed in systemd 245 that did not properly handle unknown kernel
capabilities gracefully. This resulted in incomplete output when querying
for the service status. It is possible to get service status by other means.
This PR works around this issue by getting service status using other commands
in the event of a failure due to this bug.
* GitHub is removing the underlying API used to implement the `login` command. Since the general consensus seems to be that relatively nobody currently uses this command (in favor of explicit token passing), support was simply removed for interactive login. If a future need arises, this command should be reimplemented via OAuth Device Auth Grants.
* login or role login commands now produce a fatal error with a descriptive message
* updated 2.10 and 2.11 porting guide entries
* remove dead code/config, update messages and porting guides
* Fix the reversed order of return values
AnsibleModule.run_command returns a tuple of return code, stdout and stderr.
The module main function of the user module expects user.create_user to
return a tuple of return code, stdout and stderr.
Fix the locations where stdout and stderr got reversed.
* Fix another missed occurance
* Add changelog
* Fix further reversed order of return values occurrences
Followup to 72c3ba62c8e2606a6a35d51075fba29ce5faf34e:
Fix further reversed order of return values occurrences.
Change:
- containerd is now recognized as container tech
Test Plan:
- Tested with Docker inside Github Actions in a private repo:
"ansible_virtualization_role": "guest",
"ansible_virtualization_tech_guest": [
"containerd",
"container",
"VirtualPC"
],
"ansible_virtualization_tech_host": [],
"ansible_virtualization_type": "containerd",
Tickets:
- Fixes#66304 because this is what docker containers show up as
in Github Actions.
Signed-off-by: Rick Elrod <rick@elrod.me>
* Ensure we call action_loader.get with collection_list. Fixes#72170
* Add tests and changelog
* Remove grep, do assertion in playbook. ci_complete
* Skip old jinja2 versions
* ci_complete
* dedupe
make collections whitelist follow normal flow
* fixes missing set_options call and adhoc and stdout processing rules
* avoid dupes
* fixed to handle redirects
* also updated tests with new and more accurate skip message
* fix callback tests for envs with cowsay installed
* lots MOAR comments on why the code is as it is, some todos to refactor in future
Co-authored-by: Matt Davis <nitzmahone@users.noreply.github.com>
Change:
- When a plugin defines `type: str` on a parameter, treat more kinds of
input as a string instead of whatever it is parsed as.
Test Plan:
- New unit tests
- CI
Signed-off-by: Rick Elrod <rick@elrod.me>
* Add support for GSSAPI/Kerberos to urls.py
* Test out changes with the latest test container
* Get remote hosts working
* Fix up httptester_krb5_password reader
* Fix tests for opensuse and macOS
* Hopefully last lot of testing changes
* Dont do CBT on macOS
* Fixes from review
* Fix various sanity errors in plugins.
* Revert callback_type -> type transform.
* Undo paramiko_ssh 'connection' change, since this discrepancy is hardcoded in the plugin loader.
* Standardize on name and type (for callbacks).
* Fix existing author entries.
* Add 'Unknown (!UNKNOWN)' as author when author is missing.
* These are actually parsed as integers in the code.
* Revert "Add 'Unknown (!UNKNOWN)' as author when author is missing."
This reverts commit 29d1438aca.
Change:
- No longer fail due to old Fedora/RHEL and our failure to try to cast
gids to integers before trying to pass them to getgrgid() before
trying to use them.
- Add tests for user/mode for various unarchive formats.
Test Plan:
- New integration tests, ran against centos6 container
Tickets:
- Fixes#71903
Change:
- Make KubeVirt be identified as such, and with a guest role.
- This is a breaking change, since playbooks could be relying on the
incorrect behavior previously.
Test Plan:
- Issue author
Tickets:
- Fixes#72001
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Use `sysctl -n` for openbsd uptime information
- Allow `get_sysctl()` to account for multi-line sysctl settings
- Add unit tests for `get_sysctl()`
Test Plan:
- New unit tests
Tickets:
- Fixes#71968
- Refs #72025
- Refs #72067
Signed-off-by: Rick Elrod <rick@elrod.me>
Co-authored-by: Brian Coca <brian.coca+git@gmail.com>
Co-authored-by: Brian Coca <bcoca@ansible.com>
The luseradd / lusermod commands do not support the -e option. Set
the expiry time in this case via lchage after the user was
created / modified.
Fixes: #71942
In Python3 math.floor returns an integer whereas Python2 returns a float.
Hence always convert the result of math.floor to an int to ensure that
lexpires is an integer.
Move local expires tests in a separate file and import the tasks to the
main.yml to keep main.yml smaller.
Module package is actually a proxy to a native package manager
modules (internally it delegates to the module that matches the
results of ansible_pkg_mgr).
Fixes#69371
b6b238a fixed the SLES4SAP detection, which was at this time ok.
Sadly Suse changed with SLES 15 the /etc/os-release file, so the above
change will no longer work.
This commit updates the SLES4SAP detection regarding
https://www.suse.com/support/kb/doc/?id=000019341.
The symlink realpath is matched with endswith, because in SLES 12+ the
link target is SLES_SAP.prod, but in SLES 11 the link target is
SUSE_SLES_SAP.prod.
This change preserves how all the external imports refer to this code
while allowing us to start cutting the spaghetti into more easily
maintainable pieces.
This is a start of the upcoming refactoring effort destined to
eliminate tight coupling, implicit data manipulation, god objects,
abstraction leaks and other code smells.
Essentially, `ansible.galaxy.collection` is going to be a package that
holds parts of the collection management code spread across loosely
coupled modules.
PR: #71931
* Let get_file_attributes() work without `lsattr -v`
Change:
- module_utils's get_file_attributes() expects `lsattr -v` to work, but
in some cases, it may not.
- The function now takes an optional include_version bool parameter,
which removes this expectation.
- Places where we call get_file_attributes() without using the 'version'
it returns, we now call it with include_version=False.
Test Plan:
- New unit tests
Signed-off-by: Rick Elrod <rick@elrod.me>
* Reorder test/sanity/ignore.txt to better see changes
* Remove extra empty line causing pass fail
* Apply suggestions made by Akasurde
* Minor fix in package_facts and remove two unnecessary ignores
* Fix subversion based on a suggestion made by felixfontein
* Apply suggestions made by felixfontein and Andersson007
* Fix subversion.py as suggested by felixfontein
* Minor reformatting in yum_repository description
* Reformat changelog
* Add key to apt_key, add deprecated changelog
* Add PR url to changelog
* Ignore paramater-type-not-in-doc in favour of adding key back to apt_key
* Fix apt_key
* Remove undocumented-paramater from apt_key ignore
* Ignore doc-choices-do-not-match-spec in package_facts
* Fix package_facts
* Fix filter option in setup module
Change:
- In some cases (always with free strategy, sometimes with linear), the
default callback would not show the task banner for include_tasks.
- This only affects the include_tasks task itself, not the tasks in the
included file.
Test Plan:
- Updated default callback tests
Tickets:
- Fixes#71277
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Previously when `security: true` and `bugfix: true` were both given,
only security updates would get applied. Filters now accumulate so
that both get applied in this case.
Test Plan:
- New integration tests for both check_mode and not. These tests make
use of a contrived yum repository which is stored in S3.
Tickets:
- Fixes#70854
Signed-off-by: Rick Elrod <rick@elrod.me>
Co-authored-by: Matt Martz <matt@sivel.net>
Co-authored-by: Matt Martz <matt@sivel.net>
* fix coverage output from synthetic packages
* synthetic packages (eg, implicit collection packages without `__init__.py`) were always created at runtime with empty string source, which was compiled to a code object and exec'd during the package load. When run with code coverage, it created a bogus coverage entry (since the `__synthetic__`-suffixed `__file__` entry didn't exist on disk).
* modified collection loader `get_code` to preserve the distinction between `None` (eg synthetic package) and empty string (eg empty `__init__.py`) values from `get_source`, and to return `None` when the source is `None`. This allows the package loader to skip `exec`ing things that truly have no source file on disk, thus not creating bogus coverage entries, while preserving behavior and coverage reporting for empty package inits that actually exist.
* add unit test
The setup module can now filter out multiple pattern by providing a list
to the filter parameter instead of just a string. Single string sill
works. Previous behaviour remains.
(cherry picked from commit b5c36dac483fdd74d6c570d77cc8f3e396720366)
* Use InventoryManager for inventory_hostnames lookup. Fixes#17268
* Add test for using patterns
* s/it's/its/
* Allow bypassing parse_sources in InventoryManager
* Remove unneeded method call
Change:
- By default the dnf API does not gpg-verify packages. This is a feature
that is executed in its CLI code. It never made it into Ansible's
usage of the API, so packages were previously not verified.
- This fixes CVE-2020-14365.
Test Plan:
- New integration tests
Signed-off-by: Rick Elrod <rick@elrod.me>
Remove "mode: preserve" option from documentation in doc fragments "FILE_COMMON_ARGUMENTS", as it was incorrectly included in the documentation for the `lineinfile`, `unarchive` and other file-related modules.
The `copy` and `templates` modules documentation remains untouched and still contain "mode: preserve", as intended.
* Validate salt when using crypt. Respect salt_size in password lookup. Repair salt for bcrypt. Fixes#71107. Fixes#53750. Fixes#36129.
* Handle algorithms we don't know about, and make sure to return the salt
* Account for old passlib
* Add tests for salt constraints
* Add changelog fragment
* Add test for #36129
Some parameters for systemd are ignored, add a note about such parameters in documentation.
Fixes: #23144
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* added configurable dir for tree callback
- allows usage in playbooks
- also made errors more specific if bad dir is supplied
- ensure we expand paths
* docfix
This script is mostly unmaintained and relies on the deprecated and
unmaintained `boto` library. Featuring it prominently in the docs
leads to many new users using it instead of the supported `aws_ec2`
inventory plugin.
Change:
- Now sends meta tasks to the task start callback
- Lets callback plugins opt-in to receiving implicit tasks
Test Plan:
- New integration tests
Tickets:
- Indirectly fixes#71007 by allowing custom callbacks with this data
Signed-off-by: Rick Elrod <rick@elrod.me>
skip_missing parameter in subelements lookup plugin is accepted from
inside the dictionary.
Fixes: #38182
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* check run state of current block only
* Add changelog and test
* Add test for issue 29047
* Fix for both tests
* blerg
* Change test messages
* fix tests
* Add multi-level block in rescue test case
* Add recursive rescue check and multi-level test
* Should probably run the new test
* ci_complete
* Merge new tests
* ci_complete
This would be a partial solution for #69364 in that the SHASUMS file can be downloaded and gpg verified but then used from the downloaded location to verify the get_url's file.
* Make checksum url parsing more explicit
Use urlsplit to test if the checksum string has a (currently tested and) supported url scheme.
* Fix whitespace
* Changelog fragment
* Added tests
* Fix typo in test setup
The message generated by systemctl has been updated in 9321e23c40, which requires a corresponding change in the systemd module.
In addition, this fixes the module when the SYSTEMD_OFFLINE environment variable is set.
* POC for supporting callback events that come from the worker
* linting fixes. ci_complete
* fix up units. ci_complete
* Try moving the sentinel put higher. ci_complete
* safeguards. ci_complete
* Move queue killing to terminate
* LINTING. ci_complete
* Subclass Queue, to add helper send_callback method
* Just use _final_q instead of adding another queue and thread
* Revert a few changes
* Add helper for inserting a TaskResult into the _final_q
* Add changelog fragment
* Address rebase issue
* ci_complete
* Add test to assert async poll callback from fork
* Don't use full path
* ci_complete
* Use _results_lock as a context manager
* Add new generic lock decorator, and use it with send_callback
* Revert "Change default file permissions so they are not world readable (#70221)"
This reverts commit 5260527c4a.
* Revert "Fix warning for new default permissions when mode is not specified (#70976)"
This reverts commit dc79528cc6.
Password lookup can be used to generate random string
with desired length. This is useful in various scenarios.
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Ensure -k is set to delegated hosts without a pass
* Fix up some broken tests
* Update task_executor.py
one possible fix, the other is updating winrm to normalize on 'password' like the other connection plugins
* Add alias for winrm and fix incorrect assumption
* Make sure aliases are used for keyword options
* Conditionally run test if sshpass is present, fix sanity
Co-authored-by: Brian Coca <bcoca@users.noreply.github.com>
Change:
- Allow systems to declare multiple virt techs. For example if a system
is both a docker container, but virtualized on KVM, show both. If a
system is set up to run virtualbox and KVM VMs, show both.
- This is done by introducing new facts keys:
- virtualization_tech_guest
- virtualization_tech_host
- Backwards compatibility is preserved by keeping track of the previous
return-points and refusing to update those keys after we would have
returned, but now returning them at the end, so that the new keys can
accumulate their data.
Test Plan:
- Local
- CI
Tickets:
- Refs #66304
- Refs #17151
- Refs #17058
- Probably others
Signed-off-by: Rick Elrod <rick@elrod.me>
* Fix tty_ify bugs and refactor
* Move tty_ify() and supporting attributes to the DocCLI class as that's
the only thing using it.
* Add unittest for the code.
* Fix a bug where the substitution macros can be detected when they are
a part of another word.
* Add support for L(), R(), and HORIZONTALLINE which were added to the
website docs many years ago.
* Update test/units/cli/test_doc.py
Co-authored-by: Matt Clay <matt@mystile.com>
Co-authored-by: Matt Clay <matt@mystile.com>
* add jinja2 global to reserved names
also allow expansion by additional context provided from caller
fixes#41955
Co-authored-by: Abhijeet Kasurde <akasurde@redhat.com>
Change:
- Use `chmod +a` in the fallback chain to allow MacOS to use ACLs to
allow an unprivileged user to become an unprivileged user.
Test Plan:
- CI, new tests
Tickets:
- Fixes#70648
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Add a parameter `read_whole_file` which allows for reading the whole
file when doing a `contains` regex search.
- This allows for (for example) matching a pattern at the very end of
a file.
Test Plan:
- New integration tests
Tickets:
- Fixes#63378
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Previously, `cache_timeout=0` was seen as falsey and not passed along
to cache plugins as an option. Now only "nulley" values are not sent.
Test Plan:
- New integration tests
Tickets:
- Fixes#70702
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Previously CachePluginAdjudicator#flush only removed entries from the
cache backend that it knew about by using them earlier. Now it calls
the underlying plugin's flush() method.
Test Plan:
- New unit tests
Tickets:
- Fixes#68770
Signed-off-by: Rick Elrod <rick@elrod.me>
* Strip spaces in module names in explicit actions
Change:
- When an action is called like "action: copy foo=bar", strip spaces
around the action name.
- This allows "action: copy foo=bar" to work as expected.
Test Plan:
- New integration tests
Tickets:
- Fixes#62136
Signed-off-by: Rick Elrod <rick@elrod.me>
Add integration tests for unarchiving as unprivileged user
Break tasks into separate files for easier reading and maintenance
Create a user by specifying a default group of 'staff' for macOS.
The user module does not actually remove the user directory on macOS,
so explicitly remove it.
Put the removal tasks in an always block to ensure they always run
Co-authored-by: Philip Douglass <philip.douglass@amadeus.com>
Co-authored-by: Sam Doran <sdoran@redhat.com>
The iso8601_micro and iso8601 facts incorrectly called now.utcnow(), resulting
in a new timestamp at the time it was called, not a conversion of the previously
stored timestamp.
Correct this by capturing the UTC timestamp once then calculating the local
time using the UTC offset of the current system.
* Use time.time() for getting the current time
* Convert from that stored epoch timestamp to local and UTC times
* Used existing timestamp for epoch time
* Add unit tests that validate the formate of the return value rather than an exact value since mocking time and timezone is non-trivial
Follow up to #70221
Related to #67794
CVE-2020-1736
When set_mode_if_different() is called with mode of 'None', ensure we issue
a warning about the change in default permissions.
Add integration tests to ensure the warning works properly.
* Fix tests
- actually use custom module 🤦♂️
- verify file permission on created files
- use remote_tmp_dir so we're ready for split controller
- improve test module so we can skip the call to set_fs_attributes_if_different()
- fix tests for CentOS 6
Change:
- Clarify that not passing `--tags` will cause `ansible_run_tags` to
default to `["all"]`.
- Add some extra coverage around `ansible_run_tags`
Test Plan:
- New integration and unit tests
Tickets:
- Fixes#69619
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Allow older git to verify tags again
- Enable verification tests everywhere, even if most of them only work
on newer git. Some of them work on older git and they test the --raw
parameter.
Test Plan:
- Re-enabled subset of git tests
Tickets:
- Fixes#64469
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Previously, we only showed that something would have changed, not what
would have changed. This allows us to show what will chang as well.
Test Plan:
- Local RHEL8 VM
- New integration tests
Tickets:
- Fixes#66132
Signed-off-by: Rick Elrod <rick@elrod.me>
* dnf: Add nobest option
* dnf: Fix indent, add nobest specifically to dnf not yum
* Add changelog for dnf: add nobest option
* dnf: Add nobest to yumdnf module argument_spec
* dnf: remove nobest from module paramaters in yumdnf.py
* dnf: Add test for nobest option
* dnf: Cleanup packages in nobest test at last
* dnf: Cleanup manually added repos in nobest test at last
* dnf: Remove dnf-plugins-core as well in nobest test
* dnf: Change nobest release version to 2.11
* Change changelog number according to change in PR number
* Change changelog number according to change in PR number
Change:
- On Linux, there are situations where a host might be a KVM host but
not have the kernel module enabled (it might be compiled in instead).
In these cases, /dev/kvm will still exist, and rather than reporting
NA, we should report that the host is a KVM host.
Test Plan:
- Local
Signed-off-by: Rick Elrod <rick@elrod.me>
* Change default file permissions so they are not world readable
CVE-2020-1736
Set the default permissions for files we create with atomic_move() to 0o0660. Track
which files we create that did not exist and warn if the module supports 'mode'
and it was not specified and the module did not call set_mode_if_different(). This allows the user to take action and specify a mode rather than using the defaults.
A code audit is needed to find all instances of modules that call atomic_move()
but do not call set_mode_if_different(). The findings need to be documented in
a changelog since we are not warning. Warning in those instances would be frustrating
to the user since they have no way to change the module code.
- use a set for storing list of created files
- just check the argument spac and params rather than using another property
- improve the warning message to include the default permissions
* ensure hostvars are available on delegation
* also inventory_hostname must point to current host and not delegated one
* fix get_connection since it was still mixing original host vars and delegated ones
* also return connection vars for delegation and non delegation alike
* add test to ensure we have expected usage when directly assigning for non delegated host
Some platform such as ESXi does not implement EpollSelector,
which is selected by DefaultSelector. Use SelectSelector which is
based upon 'select' implementation. This works perfectly with
a platform like VMware ESXi.
Fixes: #70238
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Encode/Decode files in UTF-8
* Use helper function in ansible
* Add an integration test
* Use emoji in test data.
* add changelog
* Also support non-ascii chars in filepath and add tests about this.
* Also use non-ascii chars in replaced text and ensure not to break cron syntax.
* rename self.existing to self.n_existing
* rename crontab.existing to crontab.n_existing
Change:
- Refactoring to make it harder to get wrong and easier to read.
- Generalize become_unprivileged tests and fix some that never worked
but also never failed.
Test Plan:
- CI, new units/integration tests
Signed-off-by: Rick Elrod <rick@elrod.me>
* Fix missing quoting for remote_tmp in second mkdir of shell module. Issue #69577
* adding changelog
* fixing typo in changelog entry
* adding test case
Adding test case written by bmillemayhias.
* using $HOME instead of ~
* fixing commit measage
* Update 69578-shell-remote_tmp-quoting.yaml
Co-authored-by: Brian Kohles <me@briankohles.com>
debconf module exposes sensitive information to logs, console.
Add a note to user about using no_log=True to hide such
information from console.
Fixes: #32386
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Followup to #70610: fix redirects to wrong collections
* gcp module_utils is in community.general.
* dellos10_* modules are in dellemc_networking.os10, not dellemc.os10.
* intersight_facts was renamed to intersight_info in cisco.intersight in version 1.0.7.
* Fix more dellos10 redirects.
A couple of years ago Slackware -current began using a plus (“+”) at the end of the distribution version string to indicate a future version work-in-progress.
Rearrange distribution_files unit tests to easily support more tests
- add conftest with common fixtures
- use parametrize for testing multiple scenarios
* Add changelog
* Add unit tests for Slackware distribution parsing
* Use correct fixtures for Slackware
Data comes from /etc/slackware-version
Co-authored-by: Sam Doran <sdoran@redhat.com>
Co-authored-by: <Eduard Rozenberg <eduardr@pobox.com>>
pipe lookup plugin uses Popen with shell=True intentionally.
This is considered a security issue if user input is not validated.
Updated docs to reflect this information for the user. Also, added
Bandit B602 documentation link for further reading.
Fixes: #70159
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
Change:
- Add integration tests for various cases
- Fix wrong use of "its" in an exception thrown in varnames when it
throws an AnsibleError, given a term of the wrong type.
Test Plan:
- new tests, CI
Tickets:
- Fixes#70546
Signed-off-by: Rick Elrod <rick@elrod.me>
- ensure we preserve the typeerror part of the exception so loop defereed error handling
can postpone those caused by undefined variables until the when check is done.
- fix tests to comply with the 'new normal'
- human_to_bytes and others can issue TypeError not only on 'non string'
but also bad string that is not convertable.
Co-authored-by: Sloane Hertel <shertel@redhat.com>
Co-authored-by: Sloane Hertel <shertel@redhat.com>
Change:
- Use parse_kv() for parsing in the csvfile lookup plugin. This allows
us to handle multi-word search keys and filenames. Previously, the
plugin split on space and so none of these things worked as expected.
- Add integration tests for csvfile, testing a plethora of weird cases.
Test Plan:
- New integration tests, CI
Tickets:
- Fixes#70545
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Move hostnamectl check out of GenericStrategy because it was incorrect
for everything except the SystemdStrategy which is where it belongs.
- Add some initial tests for the hostname module, though we are limited
by the fact that we can't do much testing with it in containers.
Test Plan:
- new hostname integration tests
Signed-off-by: Rick Elrod <rick@elrod.me>
Co-authored-by: Sam Doran <sdoran@redhat.com>
Change:
- Followup to #70504. We need to not pass the 'warn' parameter from the
action plugin either, unless it's True. Otherwise, even though it
defaults to false, we always show the deprecation.
Test Plan:
- Local
Signed-off-by: Rick Elrod <rick@elrod.me>
* speed up modify_module
* Remove debugging
* ci_complete
* Simplify generic_visit, alias to visit, eliminate some attr lookups
* ci_complete
* Add changelog fragment
Change:
- The command warnings feature which suggests that users use modules
instead of certain commands is now deprecated. Its `warn` paramater
and `COMMAND_WARNINGS` configuration options are also deprecated.
Their use will become an error in version 2.13.
Test Plan:
- CI
Signed-off-by: Rick Elrod <rick@elrod.me>
Co-authored-by: Sam Doran <sdoran@redhat.com>
Change:
Our handling of NetBSD virtualization facts led to facts that were just
plain incorrect. One example is reporting Xen even when the system is
running on something completely different (like KVM).
As stated by the reporter of #69352, NetBSD has a better sysctl setting
to use for this information, machdep.hypervisor.
This PR does the following:
- Try to use machdep.hypervisor sysctl value if the other sysctl values
we check don't end up with enough information to be useful
- Only look for /dev/xencons and assume Xen if nothing else works
(Really this should probably return 'unknown' since the file exists on
non-Xen systems and is not very useful).
- Add a few more patterns (Xen matches and also Hyper-V) to
VirtualSysctlDetectionMixin#detect_virt_product.
This change is slightly breaking:
- If the first two attempts at using sysctl worked before,
(machdep.dmi.system-product and machdep.dmi.system-vendor), they will
continue to work.
- For cases when those values didn't work, previously the existence of
/dev/xencons was checked, and if found, we reported 'xen' (even on
non-Xen systems when the file existed). After this PR, we try the
machdep.hypervisor sysctl key before still falling back to
/dev/xencons. This means that in some cases, we might go from
(wrongly) saying "xen" to giving a more accurate value such as "kvm"
or "Hyper-V".
Test Plan:
- Tested with local NetBSD VM and got 'kvm' instead of 'xen' back.
Tickets:
- Fixes#69352
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- module_utils.basic.is_special_selinux_path() used a string ==
bytestring comparison which returned False and made Ansible think that
certain filesystems aren't, in fact, special-cased, when they should
be. Ensure both sides of the == are bytestrings.
Test Plan:
- Added `copy` integration tests for this case.
Tickets:
- Fixes#70244
Signed-off-by: Rick Elrod <rick@elrod.me>
* ansible-doc man formatter: do not crash when description isn't there.
* Change to report a better error message when description is not there.
* Add test.
When user uses home directory in --playbook-dir option
of ansible-inventory command, it warns user about this.
This PR suppress the warning message for user's home directory usage
in ansible-inventory command.
Fixes: #65262
Signed-off-by: Satyajit Bulage <sbulage@redhat.com>
The ssh_args were sometimes not correctly applied to the connection
when using _play_context. Use get_option() instead to ensure the
correct ssh_args are always applied.
Change:
- Update bundled six to 1.13 (last with py2.6 support)
- Make it pass lint
- Fix check to allow skipping over compat __init__.py files we authored
- Fix check to allow files that can't be updated for some reason
Test Plan:
- ansible-test sanity --docker
- CI
Signed-off-by: Rick Elrod <rick@elrod.me>
* Use the first galaxy server supporting v1 for roles. Fixes#65440
* Add changelog fragment
* This is best effort, fall back to original behavior if something bad happens
* misc collection metadata fixes
* parse collection meta with libyaml if available
* require only Mapping for validation
* add explanatory text for _meta_yml_to_dict
* ignore custom pylint rule
* this code shouldn't import a bunch of stuff from ansible, since it's run under the import sanity test
Change:
- In certain situations, such as when the input string contains null
bytes (\0), syslog.syslog will throw a TypeError. Handle that and
fail_json instead.
Test Plan:
- New test
- ansible-test --docker centos[68] (for py2 and py3 respectively)
Tickets:
- Refs #70269
Signed-off-by: Rick Elrod <rick@elrod.me>
CIFS can be mounted using backward slash as well in /etc/fstab like
\\Windows\share /data/ cifs credentials=/root/.creds 0 0
Handle this condition while gather mount information in Linux.
Fixes: #48813
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Update documentation as per sanity tests
* Added example about custom SSH port in example section
Fixes: #29236
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Ensure not to remove existing packages while installing apt packages.
* Make all lines shorter than 160 characters
* Allow removing packages only when upgrading.
* Add integration tests
Change:
- Fix a UnicodeDecodeError in executor.module_common that could get
triggered with -vvvvv.
Test Plan:
- `ansible-test integration --docker centos7 module_utils -vvvvv`
This would show the error previously, and no loner does after this
patch.
Signed-off-by: Rick Elrod <rick@elrod.me>
In some usecases, we want to be able to clone a single branch
of a repository, without using --depth (which implies --single-branch).
* Use branch name when available
- update description of parameter
- consolidate branch or tag checking for easy reuse
* Add changelog
* Use static task imports rather than dynamic includes
* Add integration tests for single_branch
* Account for older versions of git
* Minor tweak to warnings
Co-authored-by: Laurent Coustet <laurent.coustet@clarisys.fr>
Co-authored-by: Sam Doran <sdoran@redhat.com>
* [Bugfix] curses.setupterm() error
When run playbook in celery task, curses.setupterm() will be failed
```
File "<frozen importlib._bootstrap>", line 675, in _load
File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
File "/Users/guang/.virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/action/pause.py", line 45, in <module>
curses.setupterm()
TypeError: argument must be an int, or have a fileno() method.
```
* Add changelog
Co-authored-by: Sam Doran <sdoran@redhat.com>
Handle colon appearing in filename while parsing the mimetype and charset
using file command.
Fixes: #70256
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
Change:
- On Python 3.9, `crypt.crypt` will throw instead of returning `None`
when the algorithm isn't supported. So we catch that and handle it
the same way we handled the algorithm not being supported on 3.8: by
throwing AnsibleError.
Test Plan:
- CI for <=3.8.
- Local for 3.9b3:
ansible -m debug -a "msg=\"{{ 'changeme' | password_hash('bcrypt') }}\"" localhost
Before:
localhost | FAILED! => {
"msg": "Unexpected failure during module execution.",
"stdout": ""
}
After:
localhost | FAILED! => {
"msg": "crypt.crypt does not support 'bcrypt' algorithm"
}
Tickets:
- Fixes#69930
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Remove _get_item() alias as it has been deprecated
- Update tests
- Remove relevant sanity curtailment
- Add changelog
Test Plan:
CI, grep
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
Removes with_* loop squashing and tests for 2.11
Test Plan:
CI, and grepped for with_items in package manager integration targets.
There might be some test cases in collections which need to stop testing
this behavior.
Signed-off-by: Rick Elrod <rick@elrod.me>
* Test galaxy cli against pulp
* linting fix
* Renames and small fixes
* Better handling for resetting pulp
* Clean up some things, add a comment
* I can't spell
* Bump fallaxy, use alternate pulp image
* Only reset pulp when we're are executing against pulp
* Update for updated pulp container
* Update some comments with correct URLs and typos
* Linting fix
* Pin pulp-fedora31 to a digest
* Address review comments for documentation
* try to load unqualified plugins from whitelist
* necessary for backcompat loading of unqualified collectionized callback plugins redirected from <= 2.9 core
* also added de-duping from actual loaded name
* add tests
* add warning test
* group test script entries by topic
* shorten warning text grep because wrapping is dumb
* fix adhoc callback loading behavior
* collections pass over whitelist wasn't respecting `_run_additional_callbacks`
* adds regression tests for same
* avoid `grep -L` in tests since it breaks the world
* Clarify blockinfile docs for insertafter/insertbefore
It's not clear from the docs that these options take effect
only when no marker lines are found in the document.
* Add changelog fragment
* document return values of file module
* Clarify whether path or dest input is used as dest output in file module
Co-authored-by: Alicia Cozine <879121+acozine@users.noreply.github.com>
* builtin downstream vendoring support
* allows downstream packagers to install packages to `ansible/_vendor` that will automatically be added to head of sys.path during `ansible` package load
* tests
* sort conflicting package names in warning text
* sanity fixes
* skip unnecessary comparison
If we fail to find a member when extracting a directory, try adding a trailing
slash to the member name. In certain cases, the member in the tarfile will
contain a trailing slash but the file name in FILES.json will never contain
the trailing slash.
If unable to find the member, handle the KeyError and print a nicer error.
Also check if a directory exists before creating it since it may have been
extracted from the archive.
Fixes#70009
* Add unit tests
* Use loop for trying to get members
When using 'remote_src: yes' and 'mode: preserve', the code handling
the file modes has to be handled on the remote node because it's
the one that has access to the source files. This means that the
copy module itself must handle this, rather than the copy action
plugin (which is where all that logic exists). The copy module
handles this when we copy a single file over. But when it is a
directory as the src parameter value, the mode of the files
beneath it are not considered. Subdirectories are copied with
shutil.copytree() which will preserve permissions automatically.
Individual files are copied with shutil.copyfile() which does NOT
preserve permissions. We need to add some calls to shutil.copymode()
to correct that.
Note: This *always* retains individial file permissions. Specifying
a 'mode' other than 'preserve' when giving a source directory for
the 'src' param does not make sense so will be ignored in that case
only.
Fixes#69783
* Add changelog and test
* config: singular ANSIBLE_COLLECTIONS_PATH
Every other *_PATH setting in ansible is singular, and the traditional
$PATH variable is also singular despite containing a list of
directories. Let's be consistent both internally and with POSIX
tradition.
* update all ANSIBLE_COLLECTIONS_PATHS env references to be singular
* deprecate plural ANSIBLE_COLLECTIONS_PATHS setting
* Tag return value docs if they are a dict (and not str/None).
* Try to parse return docs as YAML.
* Properly dump return values in ansible-doc.
* Adjust plugin formatter.
* Add changelog fragment.
* Don't add 'default' for return values.
* Fix plugin_formatter.
* Only try to parse return docs if they are still a string.
* Add tests.
* Warn if RETURN cannot be parsed.
* Adjust tests. Also test for warning.
* if -> elif (otherwise EXAMPLE will be parsed too).
* Always parse return documentation, and fail if it is invalid YAML.
* Polishing.
* Mostly re-enable ansible-doc tests.
Listing from the local collection seems to be somewhat broken. I assume this
is why the test was disabled.
* Lint and make tests work with Python 2.
* Keep FQCNs in plugins (not modules), i.e. restore previous state.
* Support removed_at_date in ansible-doc
Signed-off-by: NilashishC <nilashishchakraborty8@gmail.com>
Changes:
* ansible-doc does not support `removed_at_date` and assumes that
deprecated dict will either have `removed_in` or `version`. This
results in ansible-doc (and hence "sanity --test=ansible-doc")
failing for modules having only `removed_at_date`.
* This patch adds support for `removed_at_date` and also gives it
precedence over `removed_in` or `version`.
* Add tests and changelog
Signed-off-by: NilashishC <nilashishchakraborty8@gmail.com>
* ansible-galaxy - Fix role info when role is not installed
Only report the role not found if in offline mode, otherwise query the galaxy API
to get role information.
Fixes#69867
* Improve error message when role is not found in Ansible Galaxy
* Extra whitespace around imported playbook filename are stripped.
* Corrected call for display.deprecated
* Changed warning type & added test
* Added auto verification of raised warning
* More accurate warning message
This prevents PlayIterator having to go through empty blocks
that were created in filter_tagged_tasks. This should
be a performance improvement for playbooks that mostly skip
tasks with tags.
ci_complete
* Only allow groups which were hardcoded in module_defaults.yml
only load action groups from the collection if module_defaults contains a potential group for the action
* Fix tests using modules that override those whitelisted in lib/ansible/config/module_defaults.yml
Third party modules should not be using group/ - use the action name instead
* add externalized module_defaults tests
add the missing group and collections
ci_complete
Co-authored-by: Matt Davis <mrd@redhat.com>
* changelog
ci_complete
* Fix import in tests
ci_complete
* Update with requested changes
ci_complete
* don't traceback since we don't validate the contents of module_defaults
ci_complete
Co-authored-by: Matt Davis <mrd@redhat.com>
* Allow to specify collection_name separately for deprecation.
* Use new functionality in Ansible.
* Use new functionality in tests.
* Update tagging/untagging functions.
* Update pylint deprecated sanity test.
* Update validate-modules. Missing are basic checks for version_added (validate semantic version format for collections).
* Improve version validation. Re-add version_added validation.
* Make sure collection names are added to return docs before schema validation.
* Extra checks to avoid crashes on bad data.
* Make C# module utils code work, and update/extend tests.
* Add changelog fragment.
* Stop extracting collection name from potentially tagged versions/dates.
* Simplify C# code.
* Update Windows modules docs.
* Forgot semicolons.
* Make AnsibleVaultEncryptedUnicode work more like a string. Fixes#24425
* Remove debugging
* Wrap some things
* Reduce diff
* data should always result in text
* add tests
* Don't just copy and paste, kids
* Add eq and ne back
* Go full UserString copy/paste
* Various version related fixes
* Remove trailing newline
* py2v3
* Add a test that can evaluate whether a variable is vault encrypted
* map was introduces in jinja2 2.7
* moar jinja
* type fix
Co-Authored-By: Sam Doran <sdoran@redhat.com>
* Remove duplicate __hash__
* Fix typo
* Add changelog fragment
* ci_complete
Co-authored-by: Sam Doran <sdoran@redhat.com>
* Auto unroll generators produced by jinja filters
* Unroll for native in finalize
* Fix indentation
Co-authored-by: Sam Doran <sdoran@redhat.com>
* Add changelog fragment
* ci_complete
* Always unroll regardless of jinja2
* ci_complete
Co-authored-by: Sam Doran <sdoran@redhat.com>
* various deprecation, display, warning, error fixes
* Update lib/ansible/utils/display.py
Co-authored-by: Felix Fontein <felix@fontein.de>
* Update lib/ansible/utils/display.py
Co-authored-by: Felix Fontein <felix@fontein.de>
* Update lib/ansible/utils/display.py
Co-authored-by: Felix Fontein <felix@fontein.de>
* cleanup, test fixes
* add collection name to deprecated() calls
* clean up redirect entries from uncommitted tests
* fix dep warning/error header text to match previous
Co-authored-by: Felix Fontein <felix@fontein.de>
Change:
- The `add_host` action now shows an accurate change status.
Test Plan:
- Added a plethora of integration tests.
Tickets:
Fixes#69881
Signed-off-by: Rick Elrod <rick@elrod.me>
* starting metadata sunset
- purged metadata from any requirements
- fix indent in generic handler for yaml content (whey metadata display was off)
- make more resilient against bad formed docs
- removed all metadata from docs template
- remove metadata from schemas
- removed mdata tests and from unrelated tests
Co-authored-by: Felix Fontein <felix@fontein.de>
Co-authored-by: Rick Elrod <rick@elrod.me>
Change:
- This was removed in 2014 in 122a7021bc.
- The option still exists and is enabled by default and can lead to user
confusion when people aren't expecting packages (or updated
dependencies for it) to get installed and they do.
- Add the option documentation back with a few notes to make it clear what
is happening.
Test Plan:
N/A, no code change, just documentation
Tickets:
- Refs #69497
Signed-off-by: Rick Elrod <rick@elrod.me>
* Fix sanity errors
Signed-off-by: Rick Elrod <rick@elrod.me>
* Use fqcr from command module invocation using shell module
Fixes https://github.com/ansible/ansible/issues/69788
Use fully qualified collection reference while invoking
command module from shell module
* Fox review comment
In the case of a free style strategy, it is possible to end up with
multiple hosts trying to include from the same role, however the tasks
being included may be different with the use of tasks_from. Previously
if you had two hosts that were included the same role when the
process_include_results function tries to determine if a included needs
to be run on a specific host, it would end up merging two different
tasks into which ever one was processed first.
This change updates the equality check to also check if the task uuid
associated with the IncludedFile is the same. The previous check only
checked if the task's parent uuid was the same. This breaks down when
both includes have the same parent.
- hosts: all
strategy: free
gather_facts: false
tasks:
- include_role:
name: random_sleep
- block:
- name: set a fact (1)
include_role:
name: set_a_fact
tasks_from: fact1.yml
- name: set a fact (2)
include_role:
name: set_a_fact
tasks_from: fact2.yml
- name: include didn't run
fail:
msg: >
set_a_fact didn't run
fact1: {{ fact1 | default('not defined')}}
fact2: {{ fact2 | default('not defined') }}"
when: (fact1 is not defined or fact2 is not defined)
Closes#69521
To avoid issues with Flatcar Container Linux being unable to be found,
detect Flatcar distro name especially for hostname, just like CoreOS
Container Linux was supported.
See also https://github.com/ansible/ansible/issues/69516
ModuleNotFoundError is a subclass of ImportError but only exists in
Python 3.6 or newer. Instead of doing hacks to be able to catch that on
older Pythons, just always only catch ImportError, which will also catch
ModuleNotFoundError on Python 3.6 or later
* do not return the body even if it failed
* add some tests for this and rebase
* import test task
* ignore_errors when fails
Co-authored-by: Jack Zhang <jack.zhang@aspiraconnect.com>
Change:
On OpenBSD when using pipelining, we do not set cwd which results in a
permissions fatal. Ensure that `''` - cwd - is not in `sys.path`.
Test Plan:
Tested against local OpenBSD VM
Tickets:
Fixes#69320
Signed-off-by: Rick Elrod <rick@elrod.me>
* Enable installing collections from git repositories
* Add tests for installing individual and multiple collections from git repositories
* Test to make sure recursive dependencies with different syntax are deduplicated
* Add documentation
* add a changelog
* Skip Python 2.6
* Only fail if no collections are located in a git repository
Add support for a 'type' key for collections in requirement.yml files.
Update the changelog and document the supported keys and allowed values for the type.
Add a note that the collection(s) in the repo must contain a galaxy.yml
* Add a warning about embedding credentials in SCM URLs
* Update with review suggestions
* suppress sanity compile failure for Python 2.6
Change:
Rather than only using config, have base connection plugins fall back to
play_context.
Test Plan:
- Tested ansible-connection logic against an IOS device
- Tested -k against a VM
- CI
Signed-off-by: Rick Elrod <rick@elrod.me>
* routing.yml: update collections
routing.yml had gotten out of sync since `migrate.py` was run.
Update based on where files are today (more updates are needed)
Also validate routing.yml's schema
* Update lib/ansible/config/routing.yml
Co-authored-by: Andrew Klychkov <aaklychkov@mail.ru>
* rename scripts
* docker_service
* updated ansible_builtin_runtime schema
* Correct schema for plugin_router and import_redirect
* Correct schema for plugin_router and import_redirect
* validation moved to #69742
* netap, slxos, checkpoint
* test moved to separate PR
* even more
Co-authored-by: Andrew Klychkov <aaklychkov@mail.ru>
* Track collection for version_added.
Validate *all* version numbers in validate-modules.
For tagged version numbers (i.e. version_added), consider source collection to chose validation.
* Make tagging/untagging functions more flexible.
* Tag all versions in doc fragments.
* Tag all deprecation versions issued by code.
* Make Display.deprecated() understand tagged versions.
* Extend validation to enforce tagged version numbers.
* Tag versions in tests.
* Lint and fix test.
* Mention collection name in collection loader's deprecation/removal messages.
* Fix error IDs.
* Handle tagged dates in Display.deprecated().
* Also require that removed_at_date and deprecated_aliases.date are tagged.
* Also automatically tag/untag removed_at_date; fix sanity module removal version check.
* Improve error message when invalid version number is used (like '2.14' in collections).
* Allow to deprecate module by date in documentation.
* Make sure deprecation date/version match between module docs and meta/runtime.yml.
* Unrelated fix: don't compare deprecated module version to Ansible's version in collection.
* Allow documentation's removal version to be something else than fixed list of Ansible versions for collections.
* Linting.
* Allow to deprecate plugin options by date.
* Add changelog fragment for deprecation by date (also covers #68177).
* Fix example format in several Ansible modules
* Update lib/ansible/modules/fail.py
* Update lib/ansible/modules/pip.py
Co-authored-by: Felix Fontein <felix@fontein.de>
Gets rid of the unknown field names in the return data.
Allows the plugin return docs to format under the new docs pipeline.
* Expect that package_facts will pass return-syntax-error now.
PR #66461 introduced a regression that resulted in an in correct block in the file
if the block to be inserted did not end with a line separator. Fix this bug and add
tests to cover this scenario.
Fixes#64966
* Fix "TypeError: splitlines() takes no keyword arguments" on Python2.7
* Add changelog fragment
* Don't use `grep -P` for BSD/macOS compatibility
* Fix sanity checks complaining about test fixtures with mixed line endings
* Update changelogs/fragments/66461-blockinfile_preserve_line_endings.yaml
Change:
Rather than connection plugins directly accessing play_context and pulling
the password from there, have them pull it from the config system, and
have TaskExecutor store it there for now.
Internally, it still routes through play_context for now, but this is
the first step away from that.
Test Plan:
- Local test with `ansible -c ssh`
- grep -R play_context.pass lib/ansible/plugins/connection/
- CI
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
Allows the user to configure sshpass (1.06+) to look for a different
substring than the default "assword" that it comes with.
Test Plan:
Set a custom ssh password prompt on a VM with PAM and tried connecting to
it. Without `ansible_sshpass_prompt` set in inventory: experienced hang.
With `ansible_sshpass_prompt` in inventory: connected successfully.
Tried setting `ansible_sshpass_prompt` with an older `sshpass` in PATH
and got a loud error, as expected.
Tickets:
Fixes#34722, fixes#54743, refs #11565.
Signed-off-by: Rick Elrod <rick@elrod.me>
* Allow to deprecate options and aliases by date instead of only by version.
* Update display.deprecate().
* Adjust behavior to conform to tested behavior, extend tests, and improve C# style.
* Parse date and fail on invalid date.
This is mainly to make sure that people start using invalid dates, and we eventually have a mess to clean up.
* C# code: improve validation and update/extend tests.
* Make sure that deprecate() is not called with both date and version.
* Forgot to remove no longer necessary formatting.
* Adjust order of warnings in C# code.
* Adjust unrelated test.
* Fix grammar (and make that test pass).
* Don't parse date, and adjust message to be same as in #67684.
* Sanity tests: disable date in past test.
* Validate-modules: validate ISO 8601 date format.
* Validate-modules: switch schema declaration for deprecated_aliases to improve error messages for invalid dates.
* Use DateTime instead of string for date deprecation.
* Validate that date in deprecated_aliases is actually a DateTime.
* Fix tests.
* Fix rebasing error.
* Adjust error codes for pylint, and add removed_at_date and deprecated_aliases.date checks to validate-modules.
* Make deprecation date in the past error codes optional.
* Make sure not both version and date are specified for AnsibleModule.deprecate() calls.
* Stop using Python 3.7+ API.
* Make sure errors are actually reported. Re-add 'ansible-' prefix.
* Avoid crashing when 'name' isn't there.
* Linting.
* Update lib/ansible/module_utils/csharp/Ansible.Basic.cs
Co-authored-by: Jordan Borean <jborean93@gmail.com>
* Adjust test to latest change.
* Prefer date over version if both end up in Display.deprecated().
Co-authored-by: Jordan Borean <jborean93@gmail.com>
* find_module can't pop ImportError- we need to just translate to `None` since this is a normal condition with files on sys.path (eg `/usr/lib/python36.zip`)
* added test
* [yum] Make package removal confirmation strict
Change:
After removing packages, the yum module does a final check to ensure the
packages are really installed. The check would include packages that
were RPM `Provides:` values of another package.
This means that, for example, if a third-party kernel RPM spec had
`Provides: kernel` in it, removing the stock kernel would be successful
but the check to see if it was really removed would fail and cause
Ansible to report a failure.
Test Plan:
Tested on local CentOS 7 VM with kernel from elrepo which is known to
`Provides: kernel`.
Tickets:
Fixes#69237
Refs #35672
Refs #40723
Signed-off-by: Rick Elrod <rick@elrod.me>
* `meta/` directory in collections
* runtime metadata for redirection/deprecation/removal of plugin loads
* a compatibility layer to keep existing content working on ansible-base + collections
* a Python import redirection layer to keep collections-hosted (and otherwise moved) content importable by things that don't know better
* supported Ansible version validation on collection loads
* Fix `ansible -K` become_pass regression
Change:
- This fixes a breaking change introduced in
2165f9ac40
Test Plan:
- Local VM for now, with plans to add an integration test for -K going
forward.
Tickets:
Refs #69244
* fix delegated interpeter
* allow returning fact if it is 'the right host'
* added note for future fix/efficiency
as it stands we rerun discovery for the delegated host
unless its saving facts to itself
* fixed test lacking delegate_to mock
When mixed with the free strategy (or any custom strategy that does not behave in
a lock-step manner), the linear methodology of _wait_on_handler_results may cause
race conditions with regular task result processing if the strategy uses
_process_pending_results directly. This patch addresses that by splitting the queues
used for results and adding a flag to _process_pending_results to determine which
queue to check.
Fixes#69457
* Add multipart/form-data functionality
* Fix some linting issues
* Fix error message
* Allow filename to be provided with content
* Add integration test
* Update examples
* General improvements to multipart handling
* Use prepare_multipart for galaxy collection publish
* Properly account for py2 vs py3, ensuring no max header length
* Address test assumptions
* Add unit tests
* Add changelog
* Ensure to use CRLF instead of NL
* Ignore line-endings in fixture
* Consolidate code, add comment
* Bump fallaxy container version
* ci_complete
This doesn't seem to be used by config although it came in with the
config PR. It's not displayed in docs. Removing it as this is the only
place where it's present.
Since it doesn't make sense to have extra copies of the _info modules in community.general only so that the _facts
symlinks are not in google.cloud, let's simply redirect the _facts modules to their google.cloud _info counterpart.
See also ansible-collections/community.general#351.
* ansible-galaxy - remove warning during collection install
If existing collections do not contain a MANIFEST.json, which is common
for collections under development that were not installed from Ansible
Galaxy, fall back to inspecting galaxy.yml rather than displaying a
warning.
A warning will still be displayed in neither a MANIFEST.json nor
galaxy.yml are present.
* Update unit tests
* Unify ansible-galaxy install -r
* Minor nit fixes for docs
* Re-align warnings
* Fix up integration test
* Fix up test where no roles/collections were in file
This fact reflects the number of usable vcpus (which might be different
from ansible_processor_vcpus, e.g., in containers with limits). See
also #51504.
* Add fixture data and update unit tests
Co-authored-by: Sam Doran <sdoran@redhat.com>
* fix delegation vars usage and reporting
- just pass delegated host vars + task vars to plugins
and avoid poluting with original host vars
- updated tests
* Add a better error for "ansible-galaxy verify" if the MANIFEST.json has been deleted from the installed collection or if the collection hasn't been installed via normal means
* Fix unit tests for the remote collection
If there's something wrong with the local collection's version it will fail before the remote collection is sought
* Add a test for the new error msg
* Prevent the duplicate warning
Mock the new isfile call where needed
* Update lib/ansible/galaxy/collection.py
Co-Authored-By: Martin Krizek <martin.krizek@gmail.com>
Co-authored-by: Martin Krizek <martin.krizek@gmail.com>
* Fix filedescriptor out of range in select() when running commands
* Simplify the run_command() code
Now that we're using selectors in run_command(), we can simplify some of
the code.
* Use fileobj.read() instead of os.read()
* No longer use get_buffer_size() as we can just slurp all of the data
instead.
Also use a simpler conditional check of whether the selector map is
empty
Co-authored-by: Toshio Kuratomi <a.badger@gmail.com>
* only show_vars when showing vars
avoid processing function params that can be very expensive
and might not be used at all in called function.
fixes#69357
* Update changelogs/fragments/69357_optimize_inventory_graph_wo_vars.yml
Co-authored-by: Sloane Hertel <shertel@redhat.com>
Co-authored-by: Sloane Hertel <shertel@redhat.com>
The galaxy lib knew about the cli args in context. This shouldn't be
the case as it makes it hard to use the lib in other contexts. Moved
the context knowledge into cli/galaxy.py.
Trying to build docs for Ansible-2.10 and ran into error #69475
in ansible-doc. To work around that I need to get a newer cyberark
collection which fixes the issues. Unfortunately, it looks like the
cyberark team has renamed their collection from bizdev to pas is not
responding to issues on github. I'm putting in this PR for them to
get this updated before 2.10 so that the modules will work out of the
box.
Since Ansible 2.9.8, if the fileglob plugin is passed a path containing
a subdirectory of a non-existent directory, it will fail. For example:
lookup('fileglob', '/'): ok
lookup('fileglob', '/foo'): (non-existent): ok
lookup('fileglob', '/foo/bar'): (non-existent): FAIL
The exact error depends on Python 2 or 3, but here is the error on
Python 2:
AttributeError: 'NoneType' object has no attribute 'endswith'
And on Python 3:
TypeError: expected str, bytes or os.PathLike object, not NoneType
This change fixes the issue by skipping paths that are falsey before
passing them to os.path.join().
Fixes: #69450
* Fix galaxy publish sha256 value format.
The multipart/form content used for the body
of the POST to /api/automation-hub/v3/collections
was missing a newline before the line with the value
of the sha256.
automation-hub/galaxy/django skips the field entirely in
that case and automation-hub code will use None for default
to indicate that no sha256 is provided (an available option).
Fixesansible/galaxy-dev#246
* Add changelog fragment
Co-authored-by: Matt Martz <matt@sivel.net>
Change:
- Remove some no-op code
- Split up a somewhat complex line into two lines
- Nuke an incorrect comment
Test Plan:
CI
Signed-off-by: Rick Elrod <rick@elrod.me>
$HOME variable does work with command module. This may be a bug. $HOSTNAME never works. Changing the example to use $HOSTNAME in the Synopsis of command module documentation page.
Left hand side slicing is confusing and slower but maybe more memory
efficient in some circumstances. There is one case where it adds to
code safety: when it's used to substitute a different list in place of a
slice of the original list and the original list could have been bound
to a different variable in some other code. (The most likely case of
this is when it's a global variable and some other code might import
that variable name).
Because of the confusion factor we think it should only be used for the
safety case or where it's been benchmarked and shown to have some sort
of documentatble improvement. At the moment, only one piece of code
falls into those categories so this PR removes all the other instances
of left hand side slicing.
* use orderdict for yaml dictionaries
* clog
* SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
* allow user to toggle odict off
* removed config, since requored to load config
* remove unused import
* Update changelogs/fragments/yaml_orderd_mappings.yml
Co-authored-by: Matt Clay <matt@mystile.com>
* Update lib/ansible/parsing/yaml/objects.py
Co-authored-by: Matt Clay <matt@mystile.com>
* Update lib/ansible/parsing/yaml/objects.py
Co-authored-by: Matt Clay <matt@mystile.com>
Co-authored-by: Matt Clay <matt@mystile.com>
PopOS is a Debian based OS distribution, added support to detect
ansible_os_family as 'debian' instead of 'Pop!_OS'
Fixes: #69286
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Metadata defaults were not being set if only a few fields were missing.
* ansible-doc with no documentation and no status in metadata should
return empty, just like if there was no documentation and no metadata
at all.
* Fix support for Kali Linux detection
* Add test for Kali Linux detection
* Improve path matching with "in list"
Co-Authored-By: Abhijeet Kasurde <akasurde@redhat.com>
Co-authored-by: Abhijeet Kasurde <akasurde@redhat.com>
* Address FIXME's in sysctl.py
* Added changelog fragment
* Updated check
* Update conditions
* if not instead of is None
* Restore and delete FIXME comments
* Do not pass file mode during recursive copy on symlink files.
The 'file' module cannot deal with mode=preserve. Do not pass that
mode to the module when 'preserve' is used.
* Fix changelog fragment filename
Add the ability to pass allowerasing to alter the dnf transaction
behavior.
Fixes#24161
This is effectively a port of the original pull request from
poettler-ric who has since abandoned the PR
https://github.com/ansible/ansible/pull/34111
Signed-off-by: Adam Miller <admiller@redhat.com>
Change:
Extend the logic for custom error handling in the dnf module, so that on
newer DNF (such as DNF that ships with modern Fedora 31 container
images, and ships with RHEL 8.2) we report errors consistently with
older DNF.
Test Plan:
Ran dnf integration tests against an old Fedora 31 container image and a
brand new Fedora 32 container image; tess passed on both.
Signed-off-by: Rick Elrod <rick@elrod.me>
* Make sure collection is a list if a str is given
* Call field validation early on collections
Because we are doing work on modifying the collections value before
it is actually validated, we can validate it ourselves early to make
sure the user supplies either a string or list. Dicts are not valid.
The new validation allows us to simplify the _ensure_default_collection()
function. And since the field is now static, we no longer need to specify
a default for it, which also allows us to simplify the function. Since
the default is now removed, we can also remove the sanity/ignore.txt entry
for collectionsearch.py.
New unit tests are added (and the existing one modified) that allow us to
make sure that we throw a parser error if a user specifies something other
than a string or list for the collections value everywhere it can be specified.
* Revert removing the collection default
The default is actually used, so restore it.
* Fix unit tests in test_helpers.py affected by early collection validation
The call to daemonize() in sysvinit.py was missing the module parameter included in the function definition in service.py.
This pull request simply adds that parameter, as the module is used for error handling in daemonize().
* service_facts: fix for systemd 245
Since systemd 245, `systemctl list-unit-files` comes with a new column
"VENDOR PRESET" [1] and breaks the service_facts module:
$ ansible localhost -m service_facts
localhost | FAILED! => {
"changed": false,
"msg": "Malformed output discovered from systemd list-unit-files: auditd.service disabled disabled "
}
This patch drops the third column to make it work with old and new
systemd. With the new slice operation, IndexError instead of ValueError
is raised if the output contains less than 2 columns.
Test plan: running `ansible-test integration -v service_facts` on
up-to-date Arch Linux
[1] https://github.com/systemd/systemd/pull/14445
* add changelog
Signed-off-by: Rick Elrod <rick@elrod.me>
Co-authored-by: Rick Elrod <rick@elrod.me>
Currently if virtualenv_command has arguments, then the
search for the binary in the path does not work so the
user has to specify the full path to it.
To allow arguments to be used without having to specify
the path to the binary, we split the module argument into
the command and anything after the first space.
This makes using this module argument more flexible and
user friendly.
Fixes: #52275
Change:
Rather than hardcoding .pyo and .pyc, filter on all BLACKLIST_EXTS in
the non-legacy logic of PluginLoader (_find_fq_plugin). The two harcoded
extensions are part of BLACKLIST_EXTS already and this simply adds the
rest of the blacklisted extensions to the check.
In addition, check .endswith() instead of an exact match of the suffix,
like everywhere else that uses BLACKLIST_EXTS. This allows for
blacklisting, for example, emacs's backup files which can appear after
any extension, leading to things like `foo.py~`.
Test Plan:
Ran `ansible-playbook` against a collection where a `foo.py~` module was
getting executed instead of `foo.py` which also appeared in the same
directory. `foo.py~` is no longer executed.
Tickets:
Fixes#22268
Refs #27235
Signed-off-by: Rick Elrod <rick@elrod.me>
* Allow a collection role to call a standalone role by default. Fixes#69101
* tweaked changelog text
* Guard against NoneType
Co-authored-by: Matt Davis <nitzmahone@users.noreply.github.com>
* update ActionBase._low_level_execute_command to honor executable
* adding changelog fragment
* renaming changelog fragment to .yml
* noop change to bump shippable
* adding raw_executable integration test
* copying aliases from raw
* removing blank lines
* skipping aix and freebsd
* noop to bump shippable
* moving tests to raw/
* removing become_method: sudo ; it doesn't work on AIX
* removing trailing blank line
* forcing become_method: su to try to get AIX to work
Co-authored-by: Rob Wagner <rob.wagner@sas.com>
This patch fixes the issue where nested Block copies were created from
incorrect Block object. This resulted in nested Blocks data like ``name``
or ``_uuid`` to contain values from the Block the filter_tagged_tasks
method was called on.
The chapter that described more information about become to an unprivileged user has moved. This commit points to an existing chapter
Relates to #69021
With https://github.com/pallets/jinja/pull/1190 merged our short-circuit
is no longer valid (has it ever been?) as now data like ' True ' may go
through our ansible_native_concat function as opposed to going through
intermediate call to Jinja2's native_concat before. Now we need to always
send data through literal_eval to ensure native types are returned.
* ansible-galaxy - fix listing specific role
If the role was not in the first search path, it was reported as not found
* Properly display role description
Default to description to top level description, falling back to the description from within galaxy_info
* Display proper message when a role does not exist
* Add integration tests
* Use context manager
* BSD and macOS ruining all the fun
* also consolidated temp dir name generation, added pid for more 'uniqness'
* generalize error message
* added notes about remote expansion
CVE-2020-1733
fixes#67791
* Force collections to be static
Templating of collection names does not work at all. Force them to
be static so that a warning is generated for the user.
* Add collectionsearch unit test and fix for reviews
New unit test validates the new _load_collections() code and moves
the new check to the end of the method.
* Change unit test to pytest
* Adjust unit test to use capsys instead of monkeypatch
* Fix pep8 error
* Add changelog fragment
Closes#68704
* subversion module - provide password securely with svn command line option --password-from-stdin when possible, and provide a warning otherwise.
* Update lib/ansible/modules/source_control/subversion.py.
* Add a test.
Co-authored-by: Sam Doran <sdoran@redhat.com>
gcp_kms_encrypt and gcp_kms_decrypt located in
google.cloud collection and not in community.general collection
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
On OSes where we fall back to dmidecode (freebsd and linux currently),
add some missing fields that dmidecode provides when it is able.
Closes#59317.
Signed-off-by: Rick Elrod <rick@elrod.me>
* fixed fetch traversal from slurp
* ignore slurp result for dest
* fixed naming when source is relative
* fixed bug in local connection plugin
* added tests with fake slurp
* moved existing role tests into runme.sh
* normalized on action excepts
* moved dest transform down to when needed
* added is_subpath check
* fixed bug in local connection
fixes#67793
CVE-2019-3828
* Allow tasks to notify a fqcn handler name
* Add tests. Fixes#68181
* Add changelog fragment
* Add test to ensure handlers are deduped properly with fqcn, role, and just handler names
* Add some docs about new special vars
fial_json() requires a message be given to it to inform the end user of
why the module failed. Prior to this commit, the message had to be a
keyword argument:
module.fail_json(msg='Failed due to error')
Since this is a required parameter, this commit allows the message to be
given as a positional argument instead:
module.fail_json('Failed due to an error')
* Always set the discovered interpreter on the delegated host. Fixes#63180
* Make code a little more generic
* Move code into a function
* Implement some changes based on reviews
* Add changelog fragment
* when possible, use filedescriptors from mkstemp to avoid race
* when using path strings, ensure we are always creating the file
CVE-2020-1740
Fixes#67798
Co-authored-by: samdoran
The last task in a play should now properly report code coverage.
This change should also eliminate empty coverage files, as well as incomplete coverage files resulting from early worker termination.
* Fix test_check_mutually_exclusive exception-checking
Asserting inside of the `with` context of `pytest.raises`
doesn't actually have any effect. So we move the assert
out, using the exception that gets placed into the scope
after we leave the context, and ensure that it actually gets
checked.
This is also what the pytest documentation says to do:
https://docs.pytest.org/en/latest/assert.html#assertions-about-expected-exceptions
Signed-off-by: Rick Elrod <rick@elrod.me>
* Add some tests for check_required_together
Signed-off-by: Rick Elrod <rick@elrod.me>
* use to_native instead of str, for consistency
Signed-off-by: Rick Elrod <rick@elrod.me>
* Add newlines for pep8
Signed-off-by: Rick Elrod <rick@elrod.me>
* Add tests for check_required_arguments
Signed-off-by: Rick Elrod <rick@elrod.me>
* Sort missing keys in error message, since hashes are unsorted and this can be random
Signed-off-by: Rick Elrod <rick@elrod.me>
* Add changelog entry
Signed-off-by: Rick Elrod <rick@elrod.me>
* Allow custom inventory plugins and cache plugins
If _load_name is not set correctly the cache plugin can't load the documentation (which is also the arg spec)
Fix the existing inventory plugin in the collections tests
Add integration tests for using a cache plugin in a collection
* Set the attribute on the instance instead of the class
Deprecate importing custom CacheModules directly - they should use the cache_loader
* fix vault tmpe file handling
* use local temp dir instead of system temp
* ensure each worker clears dataloader temp files
* added test for dangling temp files
* added notes to data loader
CVE-2020-10685
should fix following when setting windows path in ansible.cfg:
{"ansible_facts": {}, "changed": false, "failed_modules": {"setup": {"changed": false, "failed": true, "msg": "Get-AnsibleParam: Parameter 'fact_path' has an invalid path '/home/myname/ansible/test/C:\\users\\vagrant\\facts.d\\' specified."}}, "msg": "The following modules failed to execute: setup\n"}
Sviatoslav Sydorenko