Use correct ssh executable and options in all cases on connection plugin
* Also nicer naming/comments
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
Ability to add local variables into AnsibleJ2Vars was added in
18a9eff11f to fix#6653. Local variables
are added using ``AnsibleJ2Vars.add_locals()`` method when creating a
new context - typically when including/importing a template with
context. For that use case local template variables created using
``set`` should override variables from higher contexts - either from the
play or any parent template, or both; Jinja behaves the same way.
Also removes AnsibleJ2Vars.extras instance variable which is not used.
Also adds missing test for #6653.
Fixes#72262Fixes#72615
ci_complete
The `command` module does not return stdout & sterr when calling
a non existing executable or an unknown exception arises. This fix
lets the module return empty byte strings in those cases.
Change:
- Add Ubuntu 20.04 to CI now that venv is default instead of virtualenv in ansible-test.
Test Plan:
- CI
Tickets:
- Fixes#69203
Signed-off-by: Rick Elrod <rick@elrod.me>
The init script for the test container writes additional lines to
the .bashrc of the user. This was done via a `cat` multiline
instruction, which is implemented internally by writing a
temporary file to TMPDIR (/tmp in this case) first. Docker fails
to provide /tmp just after creation, which results in a race
condition that rarely makes the init fail. Changed the `cat`
statement to multiple `echo`s.
Co-authored-by: Matt Clay <matt@mystile.com>
When calling compile(), the filename argument should be either a real file
name or a string. According to Python docs, suggested one is '<string>'.
Keep the current behaviour (encapsulate the actual expression),
but enclose it into angle brackets.
Signed-off-by: Oldřich Jedlička <oldium.pro@gmail.com>
* Fix YAML error message when error is at the end of the file
If a YAML file fails to load due to a syntax error in a file, or there is an error in the last line of a
file, PyYAML reports the last line number of the file as the index where the error occurred.
When reading the file lines, we use that index to the get the relevant line. If the index value is out
of range, the relevant line is lost for error reporting.
Subtract one from the index value to avoid the IndexError in this specific scenario. It is possible
to still get an IndexError, which will be handled as it is currently.
* Update existing tests and add new tests
* Allow restricting config values to enumerated list
* dont document internal entries
* also ignore private defs for ansible-config
* remove invalid value from tests
* added porting entry
When the pause module is run in the background and seconds parameter is provided,
do not warn.
* Add tests
* Fix existing tests
The test wasn't failing when it should have.
Change:
- On CentOS Stream, make distribution_release be "Stream"
- On CentOS Core, it continues to be "Core"
- Implement custom distribution file parser for CentOS, so we can look
for "CentOS Linux" and "CentOS Stream"
- Two new fixtures introduced (CentOS Linux 8.1 and CentOS Stream 8)
- Removed two dicts from `Distribution` class that were seemingly not
used anywhere.
Test Plan:
- ci_complete
- New test fixtures
Tickets:
- Fixes#73027
Signed-off-by: Rick Elrod <rick@elrod.me>
* Remove unused code in ansible-test.
* Remove obsolete endpoint logic from ansible-test.
* Remove obsolete region selection in ansible-test.
* Remove obsolete port logic in ansible-test.
* Clean up ansible-test remote providers.
Do the right thing on Linux when password lock and a password hash are provided by writing
out the password hash prepended by the appropriate lock string rather than using -U and -L.
This is the correct way to set and lock the account in one command.
On BSD, run separate commands as appropriate since locking and setting the password cannot
be done in a single action.
FreeBSD requires running several commands to get the account in the desired state. As a result,
the rc, output, and error from all commands need to be combined and evaluated so an accurate
and complete summary can be given at the end of module execution.
* Improve integration tests to cover this scenario.
* Break up user integration tests into smaller files
* Properly lock account when creating a new account and password is supplied
* Simplify rc collection in FreeBSD class
Since the _handle_lock() method was added, the rc would be set to None, which could make
task change reporting incorrect. My first attempt to solve this used a set and was a bit too
complicated. Simplify it my comparing the rc from _handle_lock() and the current value of rc.
* Improve the Linux password hash and locking behavior
If password lock and hash are provided, set the hash and lock the account by using a password
hash since -L cannot be used with -p.
* Ensure -U and -L are not combined with -p since they are mutually exclusive to usermod.
* Clarify password_lock behavior.
Tests should use the Python built-in ``venv`` module on Python 3 instead of the standalone ``virtualenv`` module.
On Python 2 the ``virtualenv`` module continues to be the only option.
The version installed is either the OS packaged version or the last release to support Python 2, which is version 16.7.10.
Change:
- Internally, use dnf.subject.Subject#get_best_query for state: absent
- Add a bunch of tests for removing packages, given a bunch of different
pkg specs (nv, nvr, nvra, wildcard, etc.)
Test Plan:
- New tests
- Local experiments with DNF API via PDB.
Tickets:
- Fixes#72809
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Changes to make F33 work with current tests
- Add F33 to docker.txt
- Add F33 to AZP config
Test Plan:
- ci_complete
Signed-off-by: Rick Elrod <rick@elrod.me>
* Replace the use of the function string.replace() with the method str.replace()
because that's what works for both Python 2 and 3.
* Cleanup the unused string import.
Added a changelog fragment.
* The documentation for os.write() seems a bit iffy, but in Python 3 we
definitely cannot pass it a string and we need to encode it into
bytes.
The Python documentation at
https://docs.python.org/3/howto/pyporting.html#text-versus-binary-data
says that this code will work in Python 2 as well.
Co-authored-by: Lloyd Parkes <lloyd@must-have-coffee.gen.nz>
Change:
- The FreeBSD release can contain -RC or -PRERELEASE in addition to
-RELEASE, -STABLE, or -CURRENT.
Test Plan:
- Added new fixed from an RC version of TrueNAS which uses a -PRERELEASE
version of FreeBSD.
Tickets:
- Fixes#72331
Signed-off-by: Rick Elrod <rick@elrod.me>
* ExecutionPolicy default value is RemoteSigned
##### SUMMARY
In Windows setup steps, settings `ExecutionPolicy` is restored to value `Restricted`. But actually default value for Windows server is not `Restricted` but `RemoteSigned` ([source](https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_execution_policies?view=powershell-7.1#default)).
I have got issue with `win_dsc` because `ExecutionPolicy` was `Restricted`.
##### ISSUE TYPE
- Docs Pull Request
+label: docsite_pr
* Add changelog fragment
* Apply suggestions from code review
Co-authored-by: Sandra McCann <samccann@redhat.com>
* arg_spec - move type checking lookup method to a function
* Change get_wanted_type name and behavior
Change the name to get_validator to bette describe what it is doing.
Change the interface to always return a value. This lines up with the behavior of get_*
functions always returning something or None and check_* functions raising an
Exception if something went wrong during the check.
* Add param to check_type_str()
Not meant to be a long term fix, but gets tests passing. More work is needed to figure
out how to solve this cleanly.
* Remove private attribute mapping types to validator
Since the function that needs it has moved to parameters.py, there is no need to have it as
a attribute of AnsibleModule.
Update tests that were referencing the private attribute.
* Use private method for 'str' type
To avoid having to put the string conversion warning behavior in the check_type_str() method,
use the private _check_type_str() method for 'str' type.
Import CHECK_ARGUMENT_TYPES_DISPATCHER for backwards compalitibility and store it as
a private attribute.
Revert changes to support plugins that are referencing serf._CHECK_ARGUMENT_TYPES_DISPATCHER.
* Add changelog
* Change function name to better reflect its... function
* Change dict name to better reflect its contents
CHECK_ARGUMENT_TYPES_DISPATCHER --> DEFAULT_TYPE_VALIDATORS
* Fix changelog
* systemd - do not overwrite unit name when searching
PR #72702 introduced a bug that changed the unit name when splitting it up for the purpose
of searching for the unit. This only happens on unit file templates on systems that have a 5.8
or newer kernel and a version of systemd that does not contain a bugfix that causes systmed
to fail to parse dbus.
* Use facts rather than a manual probe to determine if systmed is present
* Remov unnecessary block
* Use vars files instead of set_fact
* Add tests for using a templated unit file
* Update changelog fragment
* Use template to get correct path to sleep binary
* import_playbook - change additional params to deprecation
I incorrectly recommended this be set as a warning when it should have been a deprecation.
* Fix deprecation sanity test to not required a collection name when not inside a collection
Also pin virtualenv to 16.7.10 for older Mac OS X systems. This was the version being installed
anway with the previous constraint (<20).
On systems with Python 3, now prefer venv over virtualenv. Test to see if venv is functional since
some systems have a non-functional venv installation (such as Debian).
* Update yum/dnf module docs to include version comp
Fixes#61234
Signed-off-by: Adam Miller <admiller@redhat.com>
* fix up docs syntax to sanity tests are happy
Signed-off-by: Adam Miller <admiller@redhat.com>
* ansible-galaxy collection list and verify now utilize collections in site-packages.
This is a short term fix for #70147. The long term fix needs to handle
install (but that discussion is also bound up in how upgrade is going to
work and where things can get installed so it's deferred for 2.11.)
* Add test for ansible-galaxy collection list with site-packages
* Fix sanity issue
Co-authored-by: David Moreau Simard <moi@dmsimard.com>
Co-authored-by: Jordan Borean <jborean93@gmail.com>
On python3 sys.stdin is an encoded file object that does not support
reading raw binary data. Use the supplied buffer object to do so.
Signed-off-by: Sven Wegener <sven.wegener@inovex.de>
Co-authored-by: Sven Wegener <sven.wegener@inovex.de>
This should allow users to extract specific files from an archive as
desired.
Fixes#16130, #27081.
* Rebase and make a few minor changes
* Add changelog
* Improve tests
- move to separate tasks file
- change assertions to check for exactly one file
- use remote_tmp_dir for output dir
* Make exclude and include mutually exclusive
* Don't remove files needed by other tasks
* Fix sanity tests
* Improve feature documentation
* Skip tests that use map() on CentOS 6
* Use fnmatch on include for zip archives
This matches the behavior of exclude
Co-authored-by: Sam Doran <sdoran@redhat.com>
* Remove space before comma in '... bla , use ...'
* 'why' is inserted in the middle of a sentence, between two commas.
* Make deprecations from base.yml show source ansible-core.
* Add changelog fragment.
* Improve some more 'why's.
* Add PR URL to fragment.
* file: add symlink is in a sticky directory tests
* file: handle symlink in a sticky directory
Co-Authored-By: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
* Add changelog and fix unit test
The builtins import was removed since it was unused, but it is now needed.
The recently released version of cffi fails to install on systems with an older version of gcc. In
our case, this in the CentOS 6 test image. There is a fix but it has not yet been released.
https://foss.heptapod.net/pypy/cffi/-/issues/480
* Move _syslog_facitily to __init__
No good reason it should not be set for each object
* Move internal property setting to private method
* Create check_arguments() function
* Remove unused import
* Rename function to better match its behavior
Change the behavior to return a set, either empty or populated, with unsupported keys.
Accept legal_inputs as optional which will not required calling handle_aliases before calling
get_unsupported_parameters().
* Add changelog
* Rework function behavior and documentation
I realized I missed the original intent of this method when moving it to a function. It
is meant to compared the parameter keys to legal inputs always, not compare
parameter keys to argument spec keys, even though the argument spec keys should
be a subset of legal inputs.
* Add tests
* Fix typo.
* Set internal properties when handling suboptions
We could not find reason for the docutils install so trying out removing it.
Also bumping to latest version of ansible-runner
Skip all python2 versions because next ansible-runner drops it
Co-authored-by: Elijah DeLee <kdelee@redhat.com>
Change:
- Initial set of changes for renaming to ansible-core
- Includes changelog fragment changes from base -> core
- Does NOT include docs changes
- Modifies detection stuff in setup.py to support ansible<2.9 and ansible-base
Test Plan:
- ci_complete
* Fix super annoying Python 2.6 multiprocessing.Queue stack trace in CI
A bug exists in Python 2.6 that sometimes raises an exception during interpreter shutdown. We
encounter this frequently in our CI since we run tests on CentOS 6 as the control node, which
has Python 2.6.6 with this bug.
This PR adds a very minor sleep only on Python 2.6 which gets around this issue. I did lot of testing
using a standalon script I found that easily duplicated the issue to find the minimum sleep value
needed to avoid this issue.
CPython issue: https://bugs.python.org/issue4106
Fix in CPython: https://hg.python.org/cpython/rev/d316315a8781
* Use correct attribute
* Consolidate logic for determining whether or not session is interactive
into a single function, is_interactive()
* Increase test coverage
I wasn't able to find a good way of simulating running a backgrounded test with CI since the
whole test is essentially run not in a TTY, which is similar enough to cause the new is_interactive()
function to always return false.
* add return documentation for unarchive, add fragment for changelog, apply suggestions from code review
Co-authored-by: Amin Vakil <info@aminvakil.com>
* Added caching mechanism for Galaxy API requests
* Add cache options and split up code
* Added unit tests
* Fix sanity test
* Use modified date and fix up caching for explicit servers
* Make sure credentials are not in cached server name
* Added test for getting updated cache version
* Changes from review
Change:
- Bump default, ansible-base, distro containers
- We do NOT add fedora33 yet, because it doesn't work right on Shippable
due to an old kernel. This will be added post-AZP.
Test Plan:
- CI
Signed-off-by: Rick Elrod <rick@elrod.me>
* Reorder comment postition
* Add comment unit test
* Fix unit test
* Fix unit test
* Add changelog
* Add paramaters which would be problematic without this fix
* Fix typo
* Fix unit test
* Fix unit test
* Emit warning when running on the controller with a Python older than 3.8
* Add spaces
Co-authored-by: Matt Clay <matt@mystile.com>
* and more spaces
Co-authored-by: Matt Clay <matt@mystile.com>
* s/Py/Python/
Co-authored-by: Matt Clay <matt@mystile.com>
* Add note to Control node requirements about Py3.8 requirement
* Add collection_name to deprecated call
* more spaces
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
* Clarify that we are only packaging for py3.8+
Co-authored-by: Matt Clay <matt@mystile.com>
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
* Fix 'role_name : tast_name' notation if task contains role name
* Add tests for notifying handler names which contain the role name
Co-authored-by: Thomas Wouters <thomaswouters@gmail.com>
* Support docker and k8s action groups for moved modules in community.docker and community.kubevirt.
* Also support k8s action group for community.okd.
* Also add kubernetes.core.
* Adjust PR #.
* Fix changelog fragment.
* Remove community.okd.
* Revert "Remove community.okd."
This reverts commit 812b5aa6e2.
* ansible-doc: export has_action when --json is used.
* Remove docuri and now_data, which were not used resp. ignored in format_plugin_doc and the functions it calls anyway.
* Add function _combine_plugin_doc.
* added timeout options to adhoc and console
* added test
* fix typosesz
* fix conflict
* task_timeout
* fix timeout option, added extra vars to console
* actually use right cli switch .. DUH!
* added timeout to include 'valid' but ignored keys
* fix default
* fixes per review
* Add DarwinStrategy class and integration tests
macOS has three seprate hostname params that need to be set. One of those params, LocalHostName, has more stringent requirements than the other two, which accept special characters and spaces. Create a method to scrub the hostname to ensure it works well with the system requirements.
* Update documentation
* Account for virtualization type returned on Azure Pipelines
* Do not be dependent on order of self.name_types
Use the scrubbed name when the name type is LocalHostName
A bug existed in systemd 245 that did not properly handle unknown kernel
capabilities gracefully. This resulted in incomplete output when querying
for the service status. It is possible to get service status by other means.
This PR works around this issue by getting service status using other commands
in the event of a failure due to this bug.
Change:
- Cryptography 3.2 drops support for OpenSSL 1.0.2. Some of our CI
infrastructure still uses this version (FreeBSD, namely). For now,
just add a constraint to use old cryptography.
Test Plan:
- CI
Signed-off-by: Rick Elrod <rick@elrod.me>
* GitHub is removing the underlying API used to implement the `login` command. Since the general consensus seems to be that relatively nobody currently uses this command (in favor of explicit token passing), support was simply removed for interactive login. If a future need arises, this command should be reimplemented via OAuth Device Auth Grants.
* login or role login commands now produce a fatal error with a descriptive message
* updated 2.10 and 2.11 porting guide entries
* remove dead code/config, update messages and porting guides
* Fix ansible-test docker container detection.
* Attach test containers to the correct network.
* Do not assume `localhost` for accesing Docker.
* Look for containers on current network.
* Always map /var/run/docker.sock into containers.
This fixes issues when using a remote Docker host.
* Support container IP lookup from networks list.
* Fix container network attachment.
* Remove redundant container detection messages.
* Limit DOCKER_HOST parsing to TCP.
* Restore docker socket existence check.
The check is skipped if the docker hostname is not localhost.
* Correct changelog entry.
* Fix the reversed order of return values
AnsibleModule.run_command returns a tuple of return code, stdout and stderr.
The module main function of the user module expects user.create_user to
return a tuple of return code, stdout and stderr.
Fix the locations where stdout and stderr got reversed.
* Fix another missed occurance
* Add changelog
* Fix further reversed order of return values occurrences
Followup to 72c3ba62c8e2606a6a35d51075fba29ce5faf34e:
Fix further reversed order of return values occurrences.
Change:
- containerd is now recognized as container tech
Test Plan:
- Tested with Docker inside Github Actions in a private repo:
"ansible_virtualization_role": "guest",
"ansible_virtualization_tech_guest": [
"containerd",
"container",
"VirtualPC"
],
"ansible_virtualization_tech_host": [],
"ansible_virtualization_type": "containerd",
Tickets:
- Fixes#66304 because this is what docker containers show up as
in Github Actions.
Signed-off-by: Rick Elrod <rick@elrod.me>
* Ensure we call action_loader.get with collection_list. Fixes#72170
* Add tests and changelog
* Remove grep, do assertion in playbook. ci_complete
* Skip old jinja2 versions
* ci_complete
* dedupe
make collections whitelist follow normal flow
* fixes missing set_options call and adhoc and stdout processing rules
* avoid dupes
* fixed to handle redirects
* also updated tests with new and more accurate skip message
* fix callback tests for envs with cowsay installed
* lots MOAR comments on why the code is as it is, some todos to refactor in future
Co-authored-by: Matt Davis <nitzmahone@users.noreply.github.com>
Change:
- When a plugin defines `type: str` on a parameter, treat more kinds of
input as a string instead of whatever it is parsed as.
Test Plan:
- New unit tests
- CI
Signed-off-by: Rick Elrod <rick@elrod.me>
* Add support for GSSAPI/Kerberos to urls.py
* Test out changes with the latest test container
* Get remote hosts working
* Fix up httptester_krb5_password reader
* Fix tests for opensuse and macOS
* Hopefully last lot of testing changes
* Dont do CBT on macOS
* Fixes from review
Change:
- No longer fail due to old Fedora/RHEL and our failure to try to cast
gids to integers before trying to pass them to getgrgid() before
trying to use them.
- Add tests for user/mode for various unarchive formats.
Test Plan:
- New integration tests, ran against centos6 container
Tickets:
- Fixes#71903
Change:
- Make KubeVirt be identified as such, and with a guest role.
- This is a breaking change, since playbooks could be relying on the
incorrect behavior previously.
Test Plan:
- Issue author
Tickets:
- Fixes#72001
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- podman > 2 && < 2.2 does not support "images --format {{json .}}"
- podman also now outputs images JSON differently than docker
- Work around both of the above.
Test Plan:
- Tested with podman 2.0.6 in Fedora 31.
Signed-off-by: Rick Elrod <rick@elrod.me>
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
Change:
- Use `sysctl -n` for openbsd uptime information
- Allow `get_sysctl()` to account for multi-line sysctl settings
- Add unit tests for `get_sysctl()`
Test Plan:
- New unit tests
Tickets:
- Fixes#71968
- Refs #72025
- Refs #72067
Signed-off-by: Rick Elrod <rick@elrod.me>
Co-authored-by: Brian Coca <brian.coca+git@gmail.com>
Co-authored-by: Brian Coca <bcoca@ansible.com>
The luseradd / lusermod commands do not support the -e option. Set
the expiry time in this case via lchage after the user was
created / modified.
Fixes: #71942
In Python3 math.floor returns an integer whereas Python2 returns a float.
Hence always convert the result of math.floor to an int to ensure that
lexpires is an integer.
Move local expires tests in a separate file and import the tasks to the
main.yml to keep main.yml smaller.
b6b238a fixed the SLES4SAP detection, which was at this time ok.
Sadly Suse changed with SLES 15 the /etc/os-release file, so the above
change will no longer work.
This commit updates the SLES4SAP detection regarding
https://www.suse.com/support/kb/doc/?id=000019341.
The symlink realpath is matched with endswith, because in SLES 12+ the
link target is SLES_SAP.prod, but in SLES 11 the link target is
SUSE_SLES_SAP.prod.
* Validate removal versions.
* Validate that removal collection versions and version_added collection versions conform to semver spec.
* Validate removal version numbers in meta/runtime.yml.
* Stricter validation for isodates (f.ex. YYYY-M-D is not allowed).
* Improve error reporting.
* Validate removal collection versions.
Co-authored-by: Andrew Klychkov <aaklychkov@mail.ru>
* Let get_file_attributes() work without `lsattr -v`
Change:
- module_utils's get_file_attributes() expects `lsattr -v` to work, but
in some cases, it may not.
- The function now takes an optional include_version bool parameter,
which removes this expectation.
- Places where we call get_file_attributes() without using the 'version'
it returns, we now call it with include_version=False.
Test Plan:
- New unit tests
Signed-off-by: Rick Elrod <rick@elrod.me>
* Reorder test/sanity/ignore.txt to better see changes
* Remove extra empty line causing pass fail
* Apply suggestions made by Akasurde
* Minor fix in package_facts and remove two unnecessary ignores
* Fix subversion based on a suggestion made by felixfontein
* Apply suggestions made by felixfontein and Andersson007
* Fix subversion.py as suggested by felixfontein
* Minor reformatting in yum_repository description
* Reformat changelog
* Add key to apt_key, add deprecated changelog
* Add PR url to changelog
* Ignore paramater-type-not-in-doc in favour of adding key back to apt_key
* Fix apt_key
* Remove undocumented-paramater from apt_key ignore
* Ignore doc-choices-do-not-match-spec in package_facts
* Fix package_facts
* Fix filter option in setup module
Change:
- In some cases (always with free strategy, sometimes with linear), the
default callback would not show the task banner for include_tasks.
- This only affects the include_tasks task itself, not the tasks in the
included file.
Test Plan:
- Updated default callback tests
Tickets:
- Fixes#71277
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Previously when `security: true` and `bugfix: true` were both given,
only security updates would get applied. Filters now accumulate so
that both get applied in this case.
Test Plan:
- New integration tests for both check_mode and not. These tests make
use of a contrived yum repository which is stored in S3.
Tickets:
- Fixes#70854
Signed-off-by: Rick Elrod <rick@elrod.me>
Co-authored-by: Matt Martz <matt@sivel.net>
Co-authored-by: Matt Martz <matt@sivel.net>
* fix coverage output from synthetic packages
* synthetic packages (eg, implicit collection packages without `__init__.py`) were always created at runtime with empty string source, which was compiled to a code object and exec'd during the package load. When run with code coverage, it created a bogus coverage entry (since the `__synthetic__`-suffixed `__file__` entry didn't exist on disk).
* modified collection loader `get_code` to preserve the distinction between `None` (eg synthetic package) and empty string (eg empty `__init__.py`) values from `get_source`, and to return `None` when the source is `None`. This allows the package loader to skip `exec`ing things that truly have no source file on disk, thus not creating bogus coverage entries, while preserving behavior and coverage reporting for empty package inits that actually exist.
* add unit test
On some systems (tested with official Debian Buster-based Python 3.6
docker image), setting the LC_ALL environment variable to en_US.UTF-8
will trigger Python into switching its preferred encoding to ASCII.
If any python process tries to read a non-ASCII file in this scenario,
it will terminate with an error.
And this is exactly what happens to pytest when it tries to load its
configuration that ansible-test supplies because the configuration
contains an em dash.
In order to bypass this issue, we replaced the em dash with a regular
dash and things started working again.
An alternative solution would be to replace the en_US.UTF-8 locale with
something safer, but unfortunately, the en_US.UTF-8 is probably as safe
as it gets.
The setup module can now filter out multiple pattern by providing a list
to the filter parameter instead of just a string. Single string sill
works. Previous behaviour remains.
(cherry picked from commit b5c36dac483fdd74d6c570d77cc8f3e396720366)
* Use InventoryManager for inventory_hostnames lookup. Fixes#17268
* Add test for using patterns
* s/it's/its/
* Allow bypassing parse_sources in InventoryManager
* Remove unneeded method call
Change:
- By default the dnf API does not gpg-verify packages. This is a feature
that is executed in its CLI code. It never made it into Ansible's
usage of the API, so packages were previously not verified.
- This fixes CVE-2020-14365.
Test Plan:
- New integration tests
Signed-off-by: Rick Elrod <rick@elrod.me>
* Validate salt when using crypt. Respect salt_size in password lookup. Repair salt for bcrypt. Fixes#71107. Fixes#53750. Fixes#36129.
* Handle algorithms we don't know about, and make sure to return the salt
* Account for old passlib
* Add tests for salt constraints
* Add changelog fragment
* Add test for #36129
* added configurable dir for tree callback
- allows usage in playbooks
- also made errors more specific if bad dir is supplied
- ensure we expand paths
* docfix
Change:
- Now sends meta tasks to the task start callback
- Lets callback plugins opt-in to receiving implicit tasks
Test Plan:
- New integration tests
Tickets:
- Indirectly fixes#71007 by allowing custom callbacks with this data
Signed-off-by: Rick Elrod <rick@elrod.me>
skip_missing parameter in subelements lookup plugin is accepted from
inside the dictionary.
Fixes: #38182
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* check run state of current block only
* Add changelog and test
* Add test for issue 29047
* Fix for both tests
* blerg
* Change test messages
* fix tests
* Add multi-level block in rescue test case
* Add recursive rescue check and multi-level test
* Should probably run the new test
* ci_complete
* Merge new tests
* ci_complete
* Don't do conflict check on sdist and egg_info. Fixes#71279
* Add changelog
* adjust changelog PR link
Co-authored-by: Rick Elrod <rick@elrod.me>
* add warning about bdist_wheel
Co-authored-by: Rick Elrod <rick@elrod.me>
This would be a partial solution for #69364 in that the SHASUMS file can be downloaded and gpg verified but then used from the downloaded location to verify the get_url's file.
* Make checksum url parsing more explicit
Use urlsplit to test if the checksum string has a (currently tested and) supported url scheme.
* Fix whitespace
* Changelog fragment
* Added tests
* Fix typo in test setup
The message generated by systemctl has been updated in 9321e23c40, which requires a corresponding change in the systemd module.
In addition, this fixes the module when the SYSTEMD_OFFLINE environment variable is set.
* POC for supporting callback events that come from the worker
* linting fixes. ci_complete
* fix up units. ci_complete
* Try moving the sentinel put higher. ci_complete
* safeguards. ci_complete
* Move queue killing to terminate
* LINTING. ci_complete
* Subclass Queue, to add helper send_callback method
* Just use _final_q instead of adding another queue and thread
* Revert a few changes
* Add helper for inserting a TaskResult into the _final_q
* Add changelog fragment
* Address rebase issue
* ci_complete
* Add test to assert async poll callback from fork
* Don't use full path
* ci_complete
* Use _results_lock as a context manager
* Add new generic lock decorator, and use it with send_callback
* Revert "Change default file permissions so they are not world readable (#70221)"
This reverts commit 5260527c4a.
* Revert "Fix warning for new default permissions when mode is not specified (#70976)"
This reverts commit dc79528cc6.
* Ensure -k is set to delegated hosts without a pass
* Fix up some broken tests
* Update task_executor.py
one possible fix, the other is updating winrm to normalize on 'password' like the other connection plugins
* Add alias for winrm and fix incorrect assumption
* Make sure aliases are used for keyword options
* Conditionally run test if sshpass is present, fix sanity
Co-authored-by: Brian Coca <bcoca@users.noreply.github.com>
Change:
- Allow systems to declare multiple virt techs. For example if a system
is both a docker container, but virtualized on KVM, show both. If a
system is set up to run virtualbox and KVM VMs, show both.
- This is done by introducing new facts keys:
- virtualization_tech_guest
- virtualization_tech_host
- Backwards compatibility is preserved by keeping track of the previous
return-points and refusing to update those keys after we would have
returned, but now returning them at the end, so that the new keys can
accumulate their data.
Test Plan:
- Local
- CI
Tickets:
- Refs #66304
- Refs #17151
- Refs #17058
- Probably others
Signed-off-by: Rick Elrod <rick@elrod.me>
* Fix tty_ify bugs and refactor
* Move tty_ify() and supporting attributes to the DocCLI class as that's
the only thing using it.
* Add unittest for the code.
* Fix a bug where the substitution macros can be detected when they are
a part of another word.
* Add support for L(), R(), and HORIZONTALLINE which were added to the
website docs many years ago.
* Update test/units/cli/test_doc.py
Co-authored-by: Matt Clay <matt@mystile.com>
Co-authored-by: Matt Clay <matt@mystile.com>
* add jinja2 global to reserved names
also allow expansion by additional context provided from caller
fixes#41955
Co-authored-by: Abhijeet Kasurde <akasurde@redhat.com>
Change:
- Use `chmod +a` in the fallback chain to allow MacOS to use ACLs to
allow an unprivileged user to become an unprivileged user.
Test Plan:
- CI, new tests
Tickets:
- Fixes#70648
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Add a parameter `read_whole_file` which allows for reading the whole
file when doing a `contains` regex search.
- This allows for (for example) matching a pattern at the very end of
a file.
Test Plan:
- New integration tests
Tickets:
- Fixes#63378
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Previously CachePluginAdjudicator#flush only removed entries from the
cache backend that it knew about by using them earlier. Now it calls
the underlying plugin's flush() method.
Test Plan:
- New unit tests
Tickets:
- Fixes#68770
Signed-off-by: Rick Elrod <rick@elrod.me>
* Strip spaces in module names in explicit actions
Change:
- When an action is called like "action: copy foo=bar", strip spaces
around the action name.
- This allows "action: copy foo=bar" to work as expected.
Test Plan:
- New integration tests
Tickets:
- Fixes#62136
Signed-off-by: Rick Elrod <rick@elrod.me>
Add integration tests for unarchiving as unprivileged user
Break tasks into separate files for easier reading and maintenance
Create a user by specifying a default group of 'staff' for macOS.
The user module does not actually remove the user directory on macOS,
so explicitly remove it.
Put the removal tasks in an always block to ensure they always run
Co-authored-by: Philip Douglass <philip.douglass@amadeus.com>
Co-authored-by: Sam Doran <sdoran@redhat.com>
The iso8601_micro and iso8601 facts incorrectly called now.utcnow(), resulting
in a new timestamp at the time it was called, not a conversion of the previously
stored timestamp.
Correct this by capturing the UTC timestamp once then calculating the local
time using the UTC offset of the current system.
* Use time.time() for getting the current time
* Convert from that stored epoch timestamp to local and UTC times
* Used existing timestamp for epoch time
* Add unit tests that validate the formate of the return value rather than an exact value since mocking time and timezone is non-trivial
Follow up to #70221
Related to #67794
CVE-2020-1736
When set_mode_if_different() is called with mode of 'None', ensure we issue
a warning about the change in default permissions.
Add integration tests to ensure the warning works properly.
* Fix tests
- actually use custom module 🤦♂️
- verify file permission on created files
- use remote_tmp_dir so we're ready for split controller
- improve test module so we can skip the call to set_fs_attributes_if_different()
- fix tests for CentOS 6
Change:
- Allow older git to verify tags again
- Enable verification tests everywhere, even if most of them only work
on newer git. Some of them work on older git and they test the --raw
parameter.
Test Plan:
- Re-enabled subset of git tests
Tickets:
- Fixes#64469
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Previously, we only showed that something would have changed, not what
would have changed. This allows us to show what will chang as well.
Test Plan:
- Local RHEL8 VM
- New integration tests
Tickets:
- Fixes#66132
Signed-off-by: Rick Elrod <rick@elrod.me>
* Also validate top-level version_added.
* Fix error code.
* Produce same version_added validation error in schema than in code (and stop returning it twice).
* Return correct error codes for invalid version_added for options and return values.
* Add changelog.
* Fix forgotten closing braket.
* Accept 'historical' for some top-level version_added.
* dnf: Add nobest option
* dnf: Fix indent, add nobest specifically to dnf not yum
* Add changelog for dnf: add nobest option
* dnf: Add nobest to yumdnf module argument_spec
* dnf: remove nobest from module paramaters in yumdnf.py
* dnf: Add test for nobest option
* dnf: Cleanup packages in nobest test at last
* dnf: Cleanup manually added repos in nobest test at last
* dnf: Remove dnf-plugins-core as well in nobest test
* dnf: Change nobest release version to 2.11
* Change changelog number according to change in PR number
* Change changelog number according to change in PR number
Change:
- On Linux, there are situations where a host might be a KVM host but
not have the kernel module enabled (it might be compiled in instead).
In these cases, /dev/kvm will still exist, and rather than reporting
NA, we should report that the host is a KVM host.
Test Plan:
- Local
Signed-off-by: Rick Elrod <rick@elrod.me>
* Change default file permissions so they are not world readable
CVE-2020-1736
Set the default permissions for files we create with atomic_move() to 0o0660. Track
which files we create that did not exist and warn if the module supports 'mode'
and it was not specified and the module did not call set_mode_if_different(). This allows the user to take action and specify a mode rather than using the defaults.
A code audit is needed to find all instances of modules that call atomic_move()
but do not call set_mode_if_different(). The findings need to be documented in
a changelog since we are not warning. Warning in those instances would be frustrating
to the user since they have no way to change the module code.
- use a set for storing list of created files
- just check the argument spac and params rather than using another property
- improve the warning message to include the default permissions
* ensure hostvars are available on delegation
* also inventory_hostname must point to current host and not delegated one
* fix get_connection since it was still mixing original host vars and delegated ones
* also return connection vars for delegation and non delegation alike
* add test to ensure we have expected usage when directly assigning for non delegated host
Some platform such as ESXi does not implement EpollSelector,
which is selected by DefaultSelector. Use SelectSelector which is
based upon 'select' implementation. This works perfectly with
a platform like VMware ESXi.
Fixes: #70238
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Encode/Decode files in UTF-8
* Use helper function in ansible
* Add an integration test
* Use emoji in test data.
* add changelog
* Also support non-ascii chars in filepath and add tests about this.
* Also use non-ascii chars in replaced text and ensure not to break cron syntax.
* rename self.existing to self.n_existing
* rename crontab.existing to crontab.n_existing
Change:
- Refactoring to make it harder to get wrong and easier to read.
- Generalize become_unprivileged tests and fix some that never worked
but also never failed.
Test Plan:
- CI, new units/integration tests
Signed-off-by: Rick Elrod <rick@elrod.me>
* Fix missing quoting for remote_tmp in second mkdir of shell module. Issue #69577
* adding changelog
* fixing typo in changelog entry
* adding test case
Adding test case written by bmillemayhias.
* using $HOME instead of ~
* fixing commit measage
* Update 69578-shell-remote_tmp-quoting.yaml
Co-authored-by: Brian Kohles <me@briankohles.com>
debconf module exposes sensitive information to logs, console.
Add a note to user about using no_log=True to hide such
information from console.
Fixes: #32386
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
A couple of years ago Slackware -current began using a plus (“+”) at the end of the distribution version string to indicate a future version work-in-progress.
Rearrange distribution_files unit tests to easily support more tests
- add conftest with common fixtures
- use parametrize for testing multiple scenarios
* Add changelog
* Add unit tests for Slackware distribution parsing
* Use correct fixtures for Slackware
Data comes from /etc/slackware-version
Co-authored-by: Sam Doran <sdoran@redhat.com>
Co-authored-by: <Eduard Rozenberg <eduardr@pobox.com>>
A recent updated to psutil, which is a dependency of ansible-runner, fails
to install on older versions of pip.
Commit with the breaking change:
135628639b
pipe lookup plugin uses Popen with shell=True intentionally.
This is considered a security issue if user input is not validated.
Updated docs to reflect this information for the user. Also, added
Bandit B602 documentation link for further reading.
Fixes: #70159
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
Change:
- Add integration tests for various cases
- Fix wrong use of "its" in an exception thrown in varnames when it
throws an AnsibleError, given a term of the wrong type.
Test Plan:
- new tests, CI
Tickets:
- Fixes#70546
Signed-off-by: Rick Elrod <rick@elrod.me>
- ensure we preserve the typeerror part of the exception so loop defereed error handling
can postpone those caused by undefined variables until the when check is done.
- fix tests to comply with the 'new normal'
- human_to_bytes and others can issue TypeError not only on 'non string'
but also bad string that is not convertable.
Co-authored-by: Sloane Hertel <shertel@redhat.com>
Co-authored-by: Sloane Hertel <shertel@redhat.com>
Change:
- Use parse_kv() for parsing in the csvfile lookup plugin. This allows
us to handle multi-word search keys and filenames. Previously, the
plugin split on space and so none of these things worked as expected.
- Add integration tests for csvfile, testing a plethora of weird cases.
Test Plan:
- New integration tests, CI
Tickets:
- Fixes#70545
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Followup to #70504. We need to not pass the 'warn' parameter from the
action plugin either, unless it's True. Otherwise, even though it
defaults to false, we always show the deprecation.
Test Plan:
- Local
Signed-off-by: Rick Elrod <rick@elrod.me>
* speed up modify_module
* Remove debugging
* ci_complete
* Simplify generic_visit, alias to visit, eliminate some attr lookups
* ci_complete
* Add changelog fragment
* Fix ansible-test error in community.aws
* Add changelog entry for fix
* Change check from None to string_types
* Update changelogs/fragments/70507-validate-null-author.yaml
clarify wording "or a list of strings"
Co-authored-by: Felix Fontein <felix@fontein.de>
* Update test/lib/ansible_test/_data/sanity/validate-modules/validate_modules/schema.py
clarify wording - single string or not specified valid
Co-authored-by: Felix Fontein <felix@fontein.de>
* Do not fail but return None when given outside list
Co-authored-by: Felix Fontein <felix@fontein.de>
Change:
- The command warnings feature which suggests that users use modules
instead of certain commands is now deprecated. Its `warn` paramater
and `COMMAND_WARNINGS` configuration options are also deprecated.
Their use will become an error in version 2.13.
Test Plan:
- CI
Signed-off-by: Rick Elrod <rick@elrod.me>
Co-authored-by: Sam Doran <sdoran@redhat.com>
Change:
Our handling of NetBSD virtualization facts led to facts that were just
plain incorrect. One example is reporting Xen even when the system is
running on something completely different (like KVM).
As stated by the reporter of #69352, NetBSD has a better sysctl setting
to use for this information, machdep.hypervisor.
This PR does the following:
- Try to use machdep.hypervisor sysctl value if the other sysctl values
we check don't end up with enough information to be useful
- Only look for /dev/xencons and assume Xen if nothing else works
(Really this should probably return 'unknown' since the file exists on
non-Xen systems and is not very useful).
- Add a few more patterns (Xen matches and also Hyper-V) to
VirtualSysctlDetectionMixin#detect_virt_product.
This change is slightly breaking:
- If the first two attempts at using sysctl worked before,
(machdep.dmi.system-product and machdep.dmi.system-vendor), they will
continue to work.
- For cases when those values didn't work, previously the existence of
/dev/xencons was checked, and if found, we reported 'xen' (even on
non-Xen systems when the file existed). After this PR, we try the
machdep.hypervisor sysctl key before still falling back to
/dev/xencons. This means that in some cases, we might go from
(wrongly) saying "xen" to giving a more accurate value such as "kvm"
or "Hyper-V".
Test Plan:
- Tested with local NetBSD VM and got 'kvm' instead of 'xen' back.
Tickets:
- Fixes#69352
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- module_utils.basic.is_special_selinux_path() used a string ==
bytestring comparison which returned False and made Ansible think that
certain filesystems aren't, in fact, special-cased, when they should
be. Ensure both sides of the == are bytestrings.
Test Plan:
- Added `copy` integration tests for this case.
Tickets:
- Fixes#70244
Signed-off-by: Rick Elrod <rick@elrod.me>
* ansible-doc man formatter: do not crash when description isn't there.
* Change to report a better error message when description is not there.
* Add test.
When user uses home directory in --playbook-dir option
of ansible-inventory command, it warns user about this.
This PR suppress the warning message for user's home directory usage
in ansible-inventory command.
Fixes: #65262
Signed-off-by: Satyajit Bulage <sbulage@redhat.com>
The ssh_args were sometimes not correctly applied to the connection
when using _play_context. Use get_option() instead to ensure the
correct ssh_args are always applied.
* Use the first galaxy server supporting v1 for roles. Fixes#65440
* Add changelog fragment
* This is best effort, fall back to original behavior if something bad happens
* misc collection metadata fixes
* parse collection meta with libyaml if available
* require only Mapping for validation
* add explanatory text for _meta_yml_to_dict
* ignore custom pylint rule
* this code shouldn't import a bunch of stuff from ansible, since it's run under the import sanity test
Change:
- In certain situations, such as when the input string contains null
bytes (\0), syslog.syslog will throw a TypeError. Handle that and
fail_json instead.
Test Plan:
- New test
- ansible-test --docker centos[68] (for py2 and py3 respectively)
Tickets:
- Refs #70269
Signed-off-by: Rick Elrod <rick@elrod.me>
* ansible-test - do not validate blacklisted ps modules
* Update changelogs/fragments/validate-modules-ps-doc-blacklist.yaml
Co-authored-by: Matt Clay <matt@mystile.com>
Co-authored-by: Matt Clay <matt@mystile.com>
* Ensure not to remove existing packages while installing apt packages.
* Make all lines shorter than 160 characters
* Allow removing packages only when upgrading.
* Add integration tests
In some usecases, we want to be able to clone a single branch
of a repository, without using --depth (which implies --single-branch).
* Use branch name when available
- update description of parameter
- consolidate branch or tag checking for easy reuse
* Add changelog
* Use static task imports rather than dynamic includes
* Add integration tests for single_branch
* Account for older versions of git
* Minor tweak to warnings
Co-authored-by: Laurent Coustet <laurent.coustet@clarisys.fr>
Co-authored-by: Sam Doran <sdoran@redhat.com>
* [Bugfix] curses.setupterm() error
When run playbook in celery task, curses.setupterm() will be failed
```
File "<frozen importlib._bootstrap>", line 675, in _load
File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
File "/Users/guang/.virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/action/pause.py", line 45, in <module>
curses.setupterm()
TypeError: argument must be an int, or have a fileno() method.
```
* Add changelog
Co-authored-by: Sam Doran <sdoran@redhat.com>
Handle colon appearing in filename while parsing the mimetype and charset
using file command.
Fixes: #70256
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
Change:
- On Python 3.9, `crypt.crypt` will throw instead of returning `None`
when the algorithm isn't supported. So we catch that and handle it
the same way we handled the algorithm not being supported on 3.8: by
throwing AnsibleError.
Test Plan:
- CI for <=3.8.
- Local for 3.9b3:
ansible -m debug -a "msg=\"{{ 'changeme' | password_hash('bcrypt') }}\"" localhost
Before:
localhost | FAILED! => {
"msg": "Unexpected failure during module execution.",
"stdout": ""
}
After:
localhost | FAILED! => {
"msg": "crypt.crypt does not support 'bcrypt' algorithm"
}
Tickets:
- Fixes#69930
Signed-off-by: Rick Elrod <rick@elrod.me>
The `packaging` and `pyparsing` packages are now installed by `ansible-test` during provisioning of RHEL instances to match the downstream vendored versions.
Change:
- Remove _get_item() alias as it has been deprecated
- Update tests
- Remove relevant sanity curtailment
- Add changelog
Test Plan:
CI, grep
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
Removes with_* loop squashing and tests for 2.11
Test Plan:
CI, and grepped for with_items in package manager integration targets.
There might be some test cases in collections which need to stop testing
this behavior.
Signed-off-by: Rick Elrod <rick@elrod.me>
* Clarify blockinfile docs for insertafter/insertbefore
It's not clear from the docs that these options take effect
only when no marker lines are found in the document.
* Add changelog fragment
The upcoming pyparsing 3 release will require Python 3.5 or later, see:
https://github.com/pypa/packaging/issues/313
Unfortunately pip 8.x and earlier versions do not support python version requirements, which is why this constraint is needed.
* builtin downstream vendoring support
* allows downstream packagers to install packages to `ansible/_vendor` that will automatically be added to head of sys.path during `ansible` package load
* tests
* sort conflicting package names in warning text
* sanity fixes
* skip unnecessary comparison
* Validate ansible-base & collection's runtime.yml
Add new test `runtime-metadata`
* Schema validation of file
* Error if a a legacy meta/routing.yml exist in a collection
* removal_date OR removal_version
* Add tombstone validation.
* Allow both ISO 8601 date strings and datetime.date objects (from YAML dates).
* Address review comments.
* Add metadata to test collection.
* Add requirements file.
Co-authored-by: Felix Fontein <felix@fontein.de>
Co-authored-by: Matt Clay <matt@mystile.com>
* Add mccabe complexity testing
* Make mccabe complexity an optional error
* Add mccabe to new sanity pylint requirements
* Add a changelog fragment.
Co-authored-by: Matt Clay <matt@mystile.com>
When using 'remote_src: yes' and 'mode: preserve', the code handling
the file modes has to be handled on the remote node because it's
the one that has access to the source files. This means that the
copy module itself must handle this, rather than the copy action
plugin (which is where all that logic exists). The copy module
handles this when we copy a single file over. But when it is a
directory as the src parameter value, the mode of the files
beneath it are not considered. Subdirectories are copied with
shutil.copytree() which will preserve permissions automatically.
Individual files are copied with shutil.copyfile() which does NOT
preserve permissions. We need to add some calls to shutil.copymode()
to correct that.
Note: This *always* retains individial file permissions. Specifying
a 'mode' other than 'preserve' when giving a source directory for
the 'src' param does not make sense so will be ignored in that case
only.
Fixes#69783
* Add changelog and test
* config: singular ANSIBLE_COLLECTIONS_PATH
Every other *_PATH setting in ansible is singular, and the traditional
$PATH variable is also singular despite containing a list of
directories. Let's be consistent both internally and with POSIX
tradition.
* update all ANSIBLE_COLLECTIONS_PATHS env references to be singular
* deprecate plural ANSIBLE_COLLECTIONS_PATHS setting
Replace the ansible-base changelog linting and generation tool with antsibull-changelog and make it available for linting collections. Previously changelog linting was limited to ansible-base.
* Tag return value docs if they are a dict (and not str/None).
* Try to parse return docs as YAML.
* Properly dump return values in ansible-doc.
* Adjust plugin formatter.
* Add changelog fragment.
* Don't add 'default' for return values.
* Fix plugin_formatter.
* Only try to parse return docs if they are still a string.
* Add tests.
* Warn if RETURN cannot be parsed.
* Adjust tests. Also test for warning.
* if -> elif (otherwise EXAMPLE will be parsed too).
* Always parse return documentation, and fail if it is invalid YAML.
* Polishing.
* Mostly re-enable ansible-doc tests.
Listing from the local collection seems to be somewhat broken. I assume this
is why the test was disabled.
* Lint and make tests work with Python 2.
* Keep FQCNs in plugins (not modules), i.e. restore previous state.
* use security_fix category in changelogs for CVEs
* these fragments do not say CVE but are security fixes
Co-authored-by: Alicia Cozine <acozine@users.noreply.github.com>
* Support removed_at_date in ansible-doc
Signed-off-by: NilashishC <nilashishchakraborty8@gmail.com>
Changes:
* ansible-doc does not support `removed_at_date` and assumes that
deprecated dict will either have `removed_in` or `version`. This
results in ansible-doc (and hence "sanity --test=ansible-doc")
failing for modules having only `removed_at_date`.
* This patch adds support for `removed_at_date` and also gives it
precedence over `removed_in` or `version`.
* Add tests and changelog
Signed-off-by: NilashishC <nilashishchakraborty8@gmail.com>
This prevents PlayIterator having to go through empty blocks
that were created in filter_tagged_tasks. This should
be a performance improvement for playbooks that mostly skip
tasks with tags.
ci_complete
* Move ansible-test completion code.
* Fix a few type hints.
* Change docker completion based on context.
Collections now use version 2.0 of the default-test-container.
This is an updated version of the earlier 1.x default-test-container with ansible-base and cloud specific requirements removed.
Testing of ansible-base now uses version 1.0 of a new ansible-base-test-container.
This container is similar to the earlier 1.x default-test-container, but with unnecessary duplication of requirements across Python versions removed.
Collections which have tests that depend on requirements no longer present in the default test container should specify them in their test requirements files:
* tests/integration/requirements.txt
* tests/unit/requirements.txt
* Bump test container versions
Co-authored-by: Jordan Borean <jborean93@gmail.com>
* Only allow groups which were hardcoded in module_defaults.yml
only load action groups from the collection if module_defaults contains a potential group for the action
* Fix tests using modules that override those whitelisted in lib/ansible/config/module_defaults.yml
Third party modules should not be using group/ - use the action name instead
* add externalized module_defaults tests
add the missing group and collections
ci_complete
Co-authored-by: Matt Davis <mrd@redhat.com>
* changelog
ci_complete
* Fix import in tests
ci_complete
* Update with requested changes
ci_complete
* don't traceback since we don't validate the contents of module_defaults
ci_complete
Co-authored-by: Matt Davis <mrd@redhat.com>
* Allow to specify collection_name separately for deprecation.
* Use new functionality in Ansible.
* Use new functionality in tests.
* Update tagging/untagging functions.
* Update pylint deprecated sanity test.
* Update validate-modules. Missing are basic checks for version_added (validate semantic version format for collections).
* Improve version validation. Re-add version_added validation.
* Make sure collection names are added to return docs before schema validation.
* Extra checks to avoid crashes on bad data.
* Make C# module utils code work, and update/extend tests.
* Add changelog fragment.
* Stop extracting collection name from potentially tagged versions/dates.
* Simplify C# code.
* Update Windows modules docs.
* Forgot semicolons.
* Split out sanity test requirements.
* Run each --venv test separately.
This provides verification that the requirements for each test are properly specified.
* Use a separate requirements file per sanity test.
* Skip setuptools/cryptography setup for sanity.
* Eliminate pyyaml missing warning.
* Eliminate more pip noise.
* Fix conflicting generate_pip_install commands.
* Add changelog fragment.
* Make AnsibleVaultEncryptedUnicode work more like a string. Fixes#24425
* Remove debugging
* Wrap some things
* Reduce diff
* data should always result in text
* add tests
* Don't just copy and paste, kids
* Add eq and ne back
* Go full UserString copy/paste
* Various version related fixes
* Remove trailing newline
* py2v3
* Add a test that can evaluate whether a variable is vault encrypted
* map was introduces in jinja2 2.7
* moar jinja
* type fix
Co-Authored-By: Sam Doran <sdoran@redhat.com>
* Remove duplicate __hash__
* Fix typo
* Add changelog fragment
* ci_complete
Co-authored-by: Sam Doran <sdoran@redhat.com>
* Auto unroll generators produced by jinja filters
* Unroll for native in finalize
* Fix indentation
Co-authored-by: Sam Doran <sdoran@redhat.com>
* Add changelog fragment
* ci_complete
* Always unroll regardless of jinja2
* ci_complete
Co-authored-by: Sam Doran <sdoran@redhat.com>
Change:
- The `add_host` action now shows an accurate change status.
Test Plan:
- Added a plethora of integration tests.
Tickets:
Fixes#69881
Signed-off-by: Rick Elrod <rick@elrod.me>
* Use fqcr from command module invocation using shell module
Fixes https://github.com/ansible/ansible/issues/69788
Use fully qualified collection reference while invoking
command module from shell module
* Fox review comment
In the case of a free style strategy, it is possible to end up with
multiple hosts trying to include from the same role, however the tasks
being included may be different with the use of tasks_from. Previously
if you had two hosts that were included the same role when the
process_include_results function tries to determine if a included needs
to be run on a specific host, it would end up merging two different
tasks into which ever one was processed first.
This change updates the equality check to also check if the task uuid
associated with the IncludedFile is the same. The previous check only
checked if the task's parent uuid was the same. This breaks down when
both includes have the same parent.
- hosts: all
strategy: free
gather_facts: false
tasks:
- include_role:
name: random_sleep
- block:
- name: set a fact (1)
include_role:
name: set_a_fact
tasks_from: fact1.yml
- name: set a fact (2)
include_role:
name: set_a_fact
tasks_from: fact2.yml
- name: include didn't run
fail:
msg: >
set_a_fact didn't run
fact1: {{ fact1 | default('not defined')}}
fact2: {{ fact2 | default('not defined') }}"
when: (fact1 is not defined or fact2 is not defined)
Closes#69521
To avoid issues with Flatcar Container Linux being unable to be found,
detect Flatcar distro name especially for hostname, just like CoreOS
Container Linux was supported.
See also https://github.com/ansible/ansible/issues/69516
* do not return the body even if it failed
* add some tests for this and rebase
* import test task
* ignore_errors when fails
Co-authored-by: Jack Zhang <jack.zhang@aspiraconnect.com>
Change:
On OpenBSD when using pipelining, we do not set cwd which results in a
permissions fatal. Ensure that `''` - cwd - is not in `sys.path`.
Test Plan:
Tested against local OpenBSD VM
Tickets:
Fixes#69320
Signed-off-by: Rick Elrod <rick@elrod.me>
* Enable installing collections from git repositories
* Add tests for installing individual and multiple collections from git repositories
* Test to make sure recursive dependencies with different syntax are deduplicated
* Add documentation
* add a changelog
* Skip Python 2.6
* Only fail if no collections are located in a git repository
Add support for a 'type' key for collections in requirement.yml files.
Update the changelog and document the supported keys and allowed values for the type.
Add a note that the collection(s) in the repo must contain a galaxy.yml
* Add a warning about embedding credentials in SCM URLs
* Update with review suggestions
* suppress sanity compile failure for Python 2.6
* Track collection for version_added.
Validate *all* version numbers in validate-modules.
For tagged version numbers (i.e. version_added), consider source collection to chose validation.
* Make tagging/untagging functions more flexible.
* Tag all versions in doc fragments.
* Tag all deprecation versions issued by code.
* Make Display.deprecated() understand tagged versions.
* Extend validation to enforce tagged version numbers.
* Tag versions in tests.
* Lint and fix test.
* Mention collection name in collection loader's deprecation/removal messages.
* Fix error IDs.
* Handle tagged dates in Display.deprecated().
* Also require that removed_at_date and deprecated_aliases.date are tagged.
* Also automatically tag/untag removed_at_date; fix sanity module removal version check.
* Improve error message when invalid version number is used (like '2.14' in collections).
* Allow to deprecate module by date in documentation.
* Make sure deprecation date/version match between module docs and meta/runtime.yml.
* Unrelated fix: don't compare deprecated module version to Ansible's version in collection.
* Allow documentation's removal version to be something else than fixed list of Ansible versions for collections.
* Linting.
* Allow to deprecate plugin options by date.
* Add changelog fragment for deprecation by date (also covers #68177).
PR #66461 introduced a regression that resulted in an in correct block in the file
if the block to be inserted did not end with a line separator. Fix this bug and add
tests to cover this scenario.
Fixes#64966
* Fix "TypeError: splitlines() takes no keyword arguments" on Python2.7
* Add changelog fragment
* Don't use `grep -P` for BSD/macOS compatibility
* Fix sanity checks complaining about test fixtures with mixed line endings
* Update changelogs/fragments/66461-blockinfile_preserve_line_endings.yaml
Change:
Allows the user to configure sshpass (1.06+) to look for a different
substring than the default "assword" that it comes with.
Test Plan:
Set a custom ssh password prompt on a VM with PAM and tried connecting to
it. Without `ansible_sshpass_prompt` set in inventory: experienced hang.
With `ansible_sshpass_prompt` in inventory: connected successfully.
Tried setting `ansible_sshpass_prompt` with an older `sshpass` in PATH
and got a loud error, as expected.
Tickets:
Fixes#34722, fixes#54743, refs #11565.
Signed-off-by: Rick Elrod <rick@elrod.me>
* `meta/` directory in collections
* runtime metadata for redirection/deprecation/removal of plugin loads
* a compatibility layer to keep existing content working on ansible-base + collections
* a Python import redirection layer to keep collections-hosted (and otherwise moved) content importable by things that don't know better
* supported Ansible version validation on collection loads
* fix delegated interpeter
* allow returning fact if it is 'the right host'
* added note for future fix/efficiency
as it stands we rerun discovery for the delegated host
unless its saving facts to itself
* fixed test lacking delegate_to mock
When mixed with the free strategy (or any custom strategy that does not behave in
a lock-step manner), the linear methodology of _wait_on_handler_results may cause
race conditions with regular task result processing if the strategy uses
_process_pending_results directly. This patch addresses that by splitting the queues
used for results and adding a flag to _process_pending_results to determine which
queue to check.
Fixes#69457
* Add multipart/form-data functionality
* Fix some linting issues
* Fix error message
* Allow filename to be provided with content
* Add integration test
* Update examples
* General improvements to multipart handling
* Use prepare_multipart for galaxy collection publish
* Properly account for py2 vs py3, ensuring no max header length
* Address test assumptions
* Add unit tests
* Add changelog
* Ensure to use CRLF instead of NL
* Ignore line-endings in fixture
* Consolidate code, add comment
* Bump fallaxy container version
* ci_complete
* ansible-galaxy - remove warning during collection install
If existing collections do not contain a MANIFEST.json, which is common
for collections under development that were not installed from Ansible
Galaxy, fall back to inspecting galaxy.yml rather than displaying a
warning.
A warning will still be displayed in neither a MANIFEST.json nor
galaxy.yml are present.
* Update unit tests
* Unify ansible-galaxy install -r
* Minor nit fixes for docs
* Re-align warnings
* Fix up integration test
* Fix up test where no roles/collections were in file
This fact reflects the number of usable vcpus (which might be different
from ansible_processor_vcpus, e.g., in containers with limits). See
also #51504.
* Add fixture data and update unit tests
Co-authored-by: Sam Doran <sdoran@redhat.com>
The updated container includes fewer requirements now that the collection migration has completed.
Collections which encounter test issues with this new container should update their test requirements files to include the necessary requirements.
* Fix filedescriptor out of range in select() when running commands
* Simplify the run_command() code
Now that we're using selectors in run_command(), we can simplify some of
the code.
* Use fileobj.read() instead of os.read()
* No longer use get_buffer_size() as we can just slurp all of the data
instead.
Also use a simpler conditional check of whether the selector map is
empty
Co-authored-by: Toshio Kuratomi <a.badger@gmail.com>
* only show_vars when showing vars
avoid processing function params that can be very expensive
and might not be used at all in called function.
fixes#69357
* Update changelogs/fragments/69357_optimize_inventory_graph_wo_vars.yml
Co-authored-by: Sloane Hertel <shertel@redhat.com>
Co-authored-by: Sloane Hertel <shertel@redhat.com>
Since Ansible 2.9.8, if the fileglob plugin is passed a path containing
a subdirectory of a non-existent directory, it will fail. For example:
lookup('fileglob', '/'): ok
lookup('fileglob', '/foo'): (non-existent): ok
lookup('fileglob', '/foo/bar'): (non-existent): FAIL
The exact error depends on Python 2 or 3, but here is the error on
Python 2:
AttributeError: 'NoneType' object has no attribute 'endswith'
And on Python 3:
TypeError: expected str, bytes or os.PathLike object, not NoneType
This change fixes the issue by skipping paths that are falsey before
passing them to os.path.join().
Fixes: #69450
* Fix galaxy publish sha256 value format.
The multipart/form content used for the body
of the POST to /api/automation-hub/v3/collections
was missing a newline before the line with the value
of the sha256.
automation-hub/galaxy/django skips the field entirely in
that case and automation-hub code will use None for default
to indicate that no sha256 is provided (an available option).
Fixesansible/galaxy-dev#246
* Add changelog fragment
Co-authored-by: Matt Martz <matt@sivel.net>
Change:
Adds Fedora 32 to shippable and alters tests slightly for new Fedora.
Test Plan:
CI
Tickets:
Fixes#69230
Co-authored-by: Matt Clay <matt@mystile.com>
* validate-modules: deprecated modules in collections
In Collections a module is marked as deprecated via meta/routing.yml
Use this file, rather than the leading `_` as part of the deprecated
test.
* Correct variable
* review comments
* indentation
* Read routing.yml only once
* pep8
* Apply suggestions from code review
Co-authored-by: Matt Clay <matt@mystile.com>
* review: remove duplicated conditional
Co-authored-by: Matt Clay <matt@mystile.com>
* use orderdict for yaml dictionaries
* clog
* SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
* allow user to toggle odict off
* removed config, since requored to load config
* remove unused import
* Update changelogs/fragments/yaml_orderd_mappings.yml
Co-authored-by: Matt Clay <matt@mystile.com>
* Update lib/ansible/parsing/yaml/objects.py
Co-authored-by: Matt Clay <matt@mystile.com>
* Update lib/ansible/parsing/yaml/objects.py
Co-authored-by: Matt Clay <matt@mystile.com>
Co-authored-by: Matt Clay <matt@mystile.com>
PopOS is a Debian based OS distribution, added support to detect
ansible_os_family as 'debian' instead of 'Pop!_OS'
Fixes: #69286
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Metadata defaults were not being set if only a few fields were missing.
* ansible-doc with no documentation and no status in metadata should
return empty, just like if there was no documentation and no metadata
at all.
* Address FIXME's in sysctl.py
* Added changelog fragment
* Updated check
* Update conditions
* if not instead of is None
* Restore and delete FIXME comments
* Do not pass file mode during recursive copy on symlink files.
The 'file' module cannot deal with mode=preserve. Do not pass that
mode to the module when 'preserve' is used.
* Fix changelog fragment filename
Add the ability to pass allowerasing to alter the dnf transaction
behavior.
Fixes#24161
This is effectively a port of the original pull request from
poettler-ric who has since abandoned the PR
https://github.com/ansible/ansible/pull/34111
Signed-off-by: Adam Miller <admiller@redhat.com>
Change:
Extend the logic for custom error handling in the dnf module, so that on
newer DNF (such as DNF that ships with modern Fedora 31 container
images, and ships with RHEL 8.2) we report errors consistently with
older DNF.
Test Plan:
Ran dnf integration tests against an old Fedora 31 container image and a
brand new Fedora 32 container image; tess passed on both.
Signed-off-by: Rick Elrod <rick@elrod.me>
With inventory script migrated to their respective collection,
redirect links in documentation to their respective collection
location.
Fixes: #69139
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Make sure collection is a list if a str is given
* Call field validation early on collections
Because we are doing work on modifying the collections value before
it is actually validated, we can validate it ourselves early to make
sure the user supplies either a string or list. Dicts are not valid.
The new validation allows us to simplify the _ensure_default_collection()
function. And since the field is now static, we no longer need to specify
a default for it, which also allows us to simplify the function. Since
the default is now removed, we can also remove the sanity/ignore.txt entry
for collectionsearch.py.
New unit tests are added (and the existing one modified) that allow us to
make sure that we throw a parser error if a user specifies something other
than a string or list for the collections value everywhere it can be specified.
* Revert removing the collection default
The default is actually used, so restore it.
* Fix unit tests in test_helpers.py affected by early collection validation
The call to daemonize() in sysvinit.py was missing the module parameter included in the function definition in service.py.
This pull request simply adds that parameter, as the module is used for error handling in daemonize().
* service_facts: fix for systemd 245
Since systemd 245, `systemctl list-unit-files` comes with a new column
"VENDOR PRESET" [1] and breaks the service_facts module:
$ ansible localhost -m service_facts
localhost | FAILED! => {
"changed": false,
"msg": "Malformed output discovered from systemd list-unit-files: auditd.service disabled disabled "
}
This patch drops the third column to make it work with old and new
systemd. With the new slice operation, IndexError instead of ValueError
is raised if the output contains less than 2 columns.
Test plan: running `ansible-test integration -v service_facts` on
up-to-date Arch Linux
[1] https://github.com/systemd/systemd/pull/14445
* add changelog
Signed-off-by: Rick Elrod <rick@elrod.me>
Co-authored-by: Rick Elrod <rick@elrod.me>
Openstack dynamic inventory script is moved from community.general to
openstack.cloud, adjust the bot meta accordingly.
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
Currently if virtualenv_command has arguments, then the
search for the binary in the path does not work so the
user has to specify the full path to it.
To allow arguments to be used without having to specify
the path to the binary, we split the module argument into
the command and anything after the first space.
This makes using this module argument more flexible and
user friendly.
Fixes: #52275
Change:
Rather than hardcoding .pyo and .pyc, filter on all BLACKLIST_EXTS in
the non-legacy logic of PluginLoader (_find_fq_plugin). The two harcoded
extensions are part of BLACKLIST_EXTS already and this simply adds the
rest of the blacklisted extensions to the check.
In addition, check .endswith() instead of an exact match of the suffix,
like everywhere else that uses BLACKLIST_EXTS. This allows for
blacklisting, for example, emacs's backup files which can appear after
any extension, leading to things like `foo.py~`.
Test Plan:
Ran `ansible-playbook` against a collection where a `foo.py~` module was
getting executed instead of `foo.py` which also appeared in the same
directory. `foo.py~` is no longer executed.
Tickets:
Fixes#22268
Refs #27235
Signed-off-by: Rick Elrod <rick@elrod.me>
* Allow a collection role to call a standalone role by default. Fixes#69101
* tweaked changelog text
* Guard against NoneType
Co-authored-by: Matt Davis <nitzmahone@users.noreply.github.com>
Brian Coca