In order to test this piece of functionality, we create a dedicated
collection that contains files that should be ignored when building
the collection tarball.
After the collection is built, we make sure the produced tarball does
not contain any files that should be ignored.
Change:
- By default the dnf API does not gpg-verify packages. This is a feature
that is executed in its CLI code. It never made it into Ansible's
usage of the API, so packages were previously not verified.
- This fixes CVE-2020-14365.
Test Plan:
- New integration tests
Signed-off-by: Rick Elrod <rick@elrod.me>
test_dict_undefined_values is a duplicate of test_dict_undefined_values_is_defined.
It seems originally test_dict_undefined_values was supposed
to do what test_dict_undefined_values_bare does so it is safe to remove.
* Validate salt when using crypt. Respect salt_size in password lookup. Repair salt for bcrypt. Fixes#71107. Fixes#53750. Fixes#36129.
* Handle algorithms we don't know about, and make sure to return the salt
* Account for old passlib
* Add tests for salt constraints
* Add changelog fragment
* Add test for #36129
Change:
- Now sends meta tasks to the task start callback
- Lets callback plugins opt-in to receiving implicit tasks
Test Plan:
- New integration tests
Tickets:
- Indirectly fixes#71007 by allowing custom callbacks with this data
Signed-off-by: Rick Elrod <rick@elrod.me>
* check run state of current block only
* Add changelog and test
* Add test for issue 29047
* Fix for both tests
* blerg
* Change test messages
* fix tests
* Add multi-level block in rescue test case
* Add recursive rescue check and multi-level test
* Should probably run the new test
* ci_complete
* Merge new tests
* ci_complete
This would be a partial solution for #69364 in that the SHASUMS file can be downloaded and gpg verified but then used from the downloaded location to verify the get_url's file.
* Make checksum url parsing more explicit
Use urlsplit to test if the checksum string has a (currently tested and) supported url scheme.
* Fix whitespace
* Changelog fragment
* Added tests
* Fix typo in test setup
The message generated by systemctl has been updated in 9321e23c40, which requires a corresponding change in the systemd module.
In addition, this fixes the module when the SYSTEMD_OFFLINE environment variable is set.
* POC for supporting callback events that come from the worker
* linting fixes. ci_complete
* fix up units. ci_complete
* Try moving the sentinel put higher. ci_complete
* safeguards. ci_complete
* Move queue killing to terminate
* LINTING. ci_complete
* Subclass Queue, to add helper send_callback method
* Just use _final_q instead of adding another queue and thread
* Revert a few changes
* Add helper for inserting a TaskResult into the _final_q
* Add changelog fragment
* Address rebase issue
* ci_complete
* Add test to assert async poll callback from fork
* Don't use full path
* ci_complete
* Use _results_lock as a context manager
* Add new generic lock decorator, and use it with send_callback
* Revert "Change default file permissions so they are not world readable (#70221)"
This reverts commit 5260527c4a.
* Revert "Fix warning for new default permissions when mode is not specified (#70976)"
This reverts commit dc79528cc6.
* Ensure -k is set to delegated hosts without a pass
* Fix up some broken tests
* Update task_executor.py
one possible fix, the other is updating winrm to normalize on 'password' like the other connection plugins
* Add alias for winrm and fix incorrect assumption
* Make sure aliases are used for keyword options
* Conditionally run test if sshpass is present, fix sanity
Co-authored-by: Brian Coca <bcoca@users.noreply.github.com>
* Start of alpine testing
* More updates
* Add forgotten file
* remove debug
* Add alpine3
* equal
* group 4
* group 4
* group 5
* Try to decrease test length
* libuser only available in testing
* Remove debug
* Make loops target work on hosts without gnu date
* Enable alpine testing
* ci_complete
* Don't specify uid for creating test user
* ci_complete
* Re-sort docker completion
* use newer container image
* ci_complete
* fix indentation
Co-authored-by: Matt Clay <matt@mystile.com>
Co-authored-by: Matt Clay <matt@mystile.com>
* Fix tty_ify bugs and refactor
* Move tty_ify() and supporting attributes to the DocCLI class as that's
the only thing using it.
* Add unittest for the code.
* Fix a bug where the substitution macros can be detected when they are
a part of another word.
* Add support for L(), R(), and HORIZONTALLINE which were added to the
website docs many years ago.
* Update test/units/cli/test_doc.py
Co-authored-by: Matt Clay <matt@mystile.com>
Co-authored-by: Matt Clay <matt@mystile.com>
* add jinja2 global to reserved names
also allow expansion by additional context provided from caller
fixes#41955
Co-authored-by: Abhijeet Kasurde <akasurde@redhat.com>
Change:
- Use `chmod +a` in the fallback chain to allow MacOS to use ACLs to
allow an unprivileged user to become an unprivileged user.
Test Plan:
- CI, new tests
Tickets:
- Fixes#70648
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Adds some intentional coverage around PluginLoader for cases that
incidental_azure_rm_resource covered.
- Specifically, modules starting with an underscore, and starting with
an underscore but a symlink.
Test Plan:
- CI
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Add hardware_facts test target which manually sets up some LVM devices
and tests facts against them.
Test Plan:
- New integration tests
Tickets:
- Refs #71041 and #71042 both of which I discovered during this
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Add a parameter `read_whole_file` which allows for reading the whole
file when doing a `contains` regex search.
- This allows for (for example) matching a pattern at the very end of
a file.
Test Plan:
- New integration tests
Tickets:
- Fixes#63378
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Previously, `cache_timeout=0` was seen as falsey and not passed along
to cache plugins as an option. Now only "nulley" values are not sent.
Test Plan:
- New integration tests
Tickets:
- Fixes#70702
Signed-off-by: Rick Elrod <rick@elrod.me>
Changes to sanity and unit tests now trigger the ansible-test self-test integration tests.
No changelog entry since this only affects tests for ansible itself and not collections.
Change:
- Previously CachePluginAdjudicator#flush only removed entries from the
cache backend that it knew about by using them earlier. Now it calls
the underlying plugin's flush() method.
Test Plan:
- New unit tests
Tickets:
- Fixes#68770
Signed-off-by: Rick Elrod <rick@elrod.me>
There are links from the scenario guides to collections but collections
docs aren't built in testing or locally when we're on the devel branch.
Due to that we need to make sure those references resolve to the
production docsite. We can use intersphinx to make sure that happens.
* The test for binary files wasn't reading enough of the file.
Checking for null bytes in the first 1024 bytes failed to diagnose the
ansible_2_10.inv file as binary
* Strip spaces in module names in explicit actions
Change:
- When an action is called like "action: copy foo=bar", strip spaces
around the action name.
- This allows "action: copy foo=bar" to work as expected.
Test Plan:
- New integration tests
Tickets:
- Fixes#62136
Signed-off-by: Rick Elrod <rick@elrod.me>
* Incase of network integration test for connection local
test the paramiko_ssh auth_timeout is the value of timeout
under defaults section which is 10 seconds.
* For slower connection 10sec timout value result in authentication
timeout error hence increase the timeout value to 90 seconds
Add integration tests for unarchiving as unprivileged user
Break tasks into separate files for easier reading and maintenance
Create a user by specifying a default group of 'staff' for macOS.
The user module does not actually remove the user directory on macOS,
so explicitly remove it.
Put the removal tasks in an always block to ensure they always run
Co-authored-by: Philip Douglass <philip.douglass@amadeus.com>
Co-authored-by: Sam Doran <sdoran@redhat.com>
The iso8601_micro and iso8601 facts incorrectly called now.utcnow(), resulting
in a new timestamp at the time it was called, not a conversion of the previously
stored timestamp.
Correct this by capturing the UTC timestamp once then calculating the local
time using the UTC offset of the current system.
* Use time.time() for getting the current time
* Convert from that stored epoch timestamp to local and UTC times
* Used existing timestamp for epoch time
* Add unit tests that validate the formate of the return value rather than an exact value since mocking time and timezone is non-trivial
Follow up to #70221
Related to #67794
CVE-2020-1736
When set_mode_if_different() is called with mode of 'None', ensure we issue
a warning about the change in default permissions.
Add integration tests to ensure the warning works properly.
* Fix tests
- actually use custom module 🤦♂️
- verify file permission on created files
- use remote_tmp_dir so we're ready for split controller
- improve test module so we can skip the call to set_fs_attributes_if_different()
- fix tests for CentOS 6
Change:
- Clarify that not passing `--tags` will cause `ansible_run_tags` to
default to `["all"]`.
- Add some extra coverage around `ansible_run_tags`
Test Plan:
- New integration and unit tests
Tickets:
- Fixes#69619
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Allow older git to verify tags again
- Enable verification tests everywhere, even if most of them only work
on newer git. Some of them work on older git and they test the --raw
parameter.
Test Plan:
- Re-enabled subset of git tests
Tickets:
- Fixes#64469
Signed-off-by: Rick Elrod <rick@elrod.me>
* Fix verbosity for var_templating test
Display the subtests
* fix specificity and set ANSIBLE_BECOME_ALLOW_SAME_USER
Co-authored-by: Matt Martz <matt@sivel.net>
Co-authored-by: Matt Martz <matt@sivel.net>
Change:
- Previously, we only showed that something would have changed, not what
would have changed. This allows us to show what will chang as well.
Test Plan:
- Local RHEL8 VM
- New integration tests
Tickets:
- Fixes#66132
Signed-off-by: Rick Elrod <rick@elrod.me>
* Also validate top-level version_added.
* Fix error code.
* Produce same version_added validation error in schema than in code (and stop returning it twice).
* Return correct error codes for invalid version_added for options and return values.
* Add changelog.
* Fix forgotten closing braket.
* Accept 'historical' for some top-level version_added.
* dnf: Add nobest option
* dnf: Fix indent, add nobest specifically to dnf not yum
* Add changelog for dnf: add nobest option
* dnf: Add nobest to yumdnf module argument_spec
* dnf: remove nobest from module paramaters in yumdnf.py
* dnf: Add test for nobest option
* dnf: Cleanup packages in nobest test at last
* dnf: Cleanup manually added repos in nobest test at last
* dnf: Remove dnf-plugins-core as well in nobest test
* dnf: Change nobest release version to 2.11
* Change changelog number according to change in PR number
* Change changelog number according to change in PR number
* Change default file permissions so they are not world readable
CVE-2020-1736
Set the default permissions for files we create with atomic_move() to 0o0660. Track
which files we create that did not exist and warn if the module supports 'mode'
and it was not specified and the module did not call set_mode_if_different(). This allows the user to take action and specify a mode rather than using the defaults.
A code audit is needed to find all instances of modules that call atomic_move()
but do not call set_mode_if_different(). The findings need to be documented in
a changelog since we are not warning. Warning in those instances would be frustrating
to the user since they have no way to change the module code.
- use a set for storing list of created files
- just check the argument spac and params rather than using another property
- improve the warning message to include the default permissions
* ensure hostvars are available on delegation
* also inventory_hostname must point to current host and not delegated one
* fix get_connection since it was still mixing original host vars and delegated ones
* also return connection vars for delegation and non delegation alike
* add test to ensure we have expected usage when directly assigning for non delegated host
* Encode/Decode files in UTF-8
* Use helper function in ansible
* Add an integration test
* Use emoji in test data.
* add changelog
* Also support non-ascii chars in filepath and add tests about this.
* Also use non-ascii chars in replaced text and ensure not to break cron syntax.
* rename self.existing to self.n_existing
* rename crontab.existing to crontab.n_existing
Change:
- Refactoring to make it harder to get wrong and easier to read.
- Generalize become_unprivileged tests and fix some that never worked
but also never failed.
Test Plan:
- CI, new units/integration tests
Signed-off-by: Rick Elrod <rick@elrod.me>
* Fix missing quoting for remote_tmp in second mkdir of shell module. Issue #69577
* adding changelog
* fixing typo in changelog entry
* adding test case
Adding test case written by bmillemayhias.
* using $HOME instead of ~
* fixing commit measage
* Update 69578-shell-remote_tmp-quoting.yaml
Co-authored-by: Brian Kohles <me@briankohles.com>
A couple of years ago Slackware -current began using a plus (“+”) at the end of the distribution version string to indicate a future version work-in-progress.
Rearrange distribution_files unit tests to easily support more tests
- add conftest with common fixtures
- use parametrize for testing multiple scenarios
* Add changelog
* Add unit tests for Slackware distribution parsing
* Use correct fixtures for Slackware
Data comes from /etc/slackware-version
Co-authored-by: Sam Doran <sdoran@redhat.com>
Co-authored-by: <Eduard Rozenberg <eduardr@pobox.com>>
* Build documentation for Ansible-2.10 (formerly known as ACD).
Builds plugin docs from collections whose source is on galaxy
The new command downloads collections from galaxy, then finds the
plugins inside of them to get the documentation for those plugins.
* Update the python syntax checks
* docs builds can now require python 3.6+.
* Move plugin formatter code out to an external tool, antsibull-docs.
Collection owners want to be able to extract docs for their own
websites as well.
* The jinja2 filters, tests, and other support code have moved to antsibull
* Remove document_plugins as that has now been integrated into antsibull-docs
* Cleanup and bugfix to other build script code:
* The Commands class needed to have its metaclass set for abstractmethod
to work correctly
* Fix lint issues in some command plugins
* Add the docs/docsite/rst/collections to .gitignore as
everything in that directory will be generated so we don't want any of
it saved in the git repository
* gitignore the build dir and remove edit docs link on module pages
* Add docs/rst/collections as a directory to remove on make clean
* Split the collections docs from the main docs
* remove version and edit on github
* remove version banner for just collections
* clarify examples need collection keyword defined
* Remove references to plugin documentation locations that no longer exist.
* Perhaps the pages in plugins/*.rst should be deprecated
altogether and their content moved?
* If not, perhaps we want to rephrase and link into the collection
documentation?
* Or perhaps we want to link to the plugins which are present in
collections/ansible/builtin?
* Remove PYTHONPATH from the build-ansible calls
One of the design goals of the build-ansible.py script was for it to
automatically set its library path to include the checkout of ansible
and the library of code to implement itself. Because it automatically
includes the checkout of ansible, we don't need to set PYTHONPATH in
the Makefile any longer.
* Create a command to only build ansible-base plugin docs
* When building docs for devel, only build the ansible-base docs for
now. This is because antsibull needs support for building a "devel
tree" of docs. This can be changed once that is implemented
* When building docs for the sanity tests, only build the ansible-base
plugin docs for now. Those are the docs which are in this repo so
that seems appropriate for now.
A recent updated to psutil, which is a dependency of ansible-runner, fails
to install on older versions of pip.
Commit with the breaking change:
135628639b
* Get m_u.facts.utils coverage up to 100%
Change:
- Add tests to 'gathering_facts' integration target to get
module_utils.facts.utils coverage up to 100%.
- This also clears incidental coverage from incidental_selinux.
Test Plan:
- CI
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Test async_wrapper when the module it runs has stderr output
Test Plan:
- CI
- Looked at coverage report and saw green for a few lines that weren't
previously green.
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Add integration tests for various cases
- Fix wrong use of "its" in an exception thrown in varnames when it
throws an AnsibleError, given a term of the wrong type.
Test Plan:
- new tests, CI
Tickets:
- Fixes#70546
Signed-off-by: Rick Elrod <rick@elrod.me>
- ensure we preserve the typeerror part of the exception so loop defereed error handling
can postpone those caused by undefined variables until the when check is done.
- fix tests to comply with the 'new normal'
- human_to_bytes and others can issue TypeError not only on 'non string'
but also bad string that is not convertable.
Co-authored-by: Sloane Hertel <shertel@redhat.com>
Co-authored-by: Sloane Hertel <shertel@redhat.com>
Change:
- Use parse_kv() for parsing in the csvfile lookup plugin. This allows
us to handle multi-word search keys and filenames. Previously, the
plugin split on space and so none of these things worked as expected.
- Add integration tests for csvfile, testing a plethora of weird cases.
Test Plan:
- New integration tests, CI
Tickets:
- Fixes#70545
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Move hostnamectl check out of GenericStrategy because it was incorrect
for everything except the SystemdStrategy which is where it belongs.
- Add some initial tests for the hostname module, though we are limited
by the fact that we can't do much testing with it in containers.
Test Plan:
- new hostname integration tests
Signed-off-by: Rick Elrod <rick@elrod.me>
Co-authored-by: Sam Doran <sdoran@redhat.com>
* Fix ansible-test error in community.aws
* Add changelog entry for fix
* Change check from None to string_types
* Update changelogs/fragments/70507-validate-null-author.yaml
clarify wording "or a list of strings"
Co-authored-by: Felix Fontein <felix@fontein.de>
* Update test/lib/ansible_test/_data/sanity/validate-modules/validate_modules/schema.py
clarify wording - single string or not specified valid
Co-authored-by: Felix Fontein <felix@fontein.de>
* Do not fail but return None when given outside list
Co-authored-by: Felix Fontein <felix@fontein.de>
Change:
- The command warnings feature which suggests that users use modules
instead of certain commands is now deprecated. Its `warn` paramater
and `COMMAND_WARNINGS` configuration options are also deprecated.
Their use will become an error in version 2.13.
Test Plan:
- CI
Signed-off-by: Rick Elrod <rick@elrod.me>
Co-authored-by: Sam Doran <sdoran@redhat.com>
Change:
- module_utils.basic.is_special_selinux_path() used a string ==
bytestring comparison which returned False and made Ansible think that
certain filesystems aren't, in fact, special-cased, when they should
be. Ensure both sides of the == are bytestrings.
Test Plan:
- Added `copy` integration tests for this case.
Tickets:
- Fixes#70244
Signed-off-by: Rick Elrod <rick@elrod.me>
* ansible-doc man formatter: do not crash when description isn't there.
* Change to report a better error message when description is not there.
* Add test.
Change:
- Update bundled six to 1.13 (last with py2.6 support)
- Make it pass lint
- Fix check to allow skipping over compat __init__.py files we authored
- Fix check to allow files that can't be updated for some reason
Test Plan:
- ansible-test sanity --docker
- CI
Signed-off-by: Rick Elrod <rick@elrod.me>
* Test against galaxy_ng
* Switch container image
* Remove redundant |default
* Re-enable
* Update image
* Update wording
* Don't use pulp as the container name
Change:
- In certain situations, such as when the input string contains null
bytes (\0), syslog.syslog will throw a TypeError. Handle that and
fail_json instead.
Test Plan:
- New test
- ansible-test --docker centos[68] (for py2 and py3 respectively)
Tickets:
- Refs #70269
Signed-off-by: Rick Elrod <rick@elrod.me>
* ansible-test - do not validate blacklisted ps modules
* Update changelogs/fragments/validate-modules-ps-doc-blacklist.yaml
Co-authored-by: Matt Clay <matt@mystile.com>
Co-authored-by: Matt Clay <matt@mystile.com>
CIFS can be mounted using backward slash as well in /etc/fstab like
\\Windows\share /data/ cifs credentials=/root/.creds 0 0
Handle this condition while gather mount information in Linux.
Fixes: #48813
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Update documentation as per sanity tests
* Added example about custom SSH port in example section
Fixes: #29236
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Ensure not to remove existing packages while installing apt packages.
* Make all lines shorter than 160 characters
* Allow removing packages only when upgrading.
* Add integration tests
Change:
- Fix a UnicodeDecodeError in executor.module_common that could get
triggered with -vvvvv.
Test Plan:
- `ansible-test integration --docker centos7 module_utils -vvvvv`
This would show the error previously, and no loner does after this
patch.
Signed-off-by: Rick Elrod <rick@elrod.me>
In some usecases, we want to be able to clone a single branch
of a repository, without using --depth (which implies --single-branch).
* Use branch name when available
- update description of parameter
- consolidate branch or tag checking for easy reuse
* Add changelog
* Use static task imports rather than dynamic includes
* Add integration tests for single_branch
* Account for older versions of git
* Minor tweak to warnings
Co-authored-by: Laurent Coustet <laurent.coustet@clarisys.fr>
Co-authored-by: Sam Doran <sdoran@redhat.com>
Handle colon appearing in filename while parsing the mimetype and charset
using file command.
Fixes: #70256
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
The `packaging` and `pyparsing` packages are now installed by `ansible-test` during provisioning of RHEL instances to match the downstream vendored versions.
Change:
- Remove _get_item() alias as it has been deprecated
- Update tests
- Remove relevant sanity curtailment
- Add changelog
Test Plan:
CI, grep
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
Removes with_* loop squashing and tests for 2.11
Test Plan:
CI, and grepped for with_items in package manager integration targets.
There might be some test cases in collections which need to stop testing
this behavior.
Signed-off-by: Rick Elrod <rick@elrod.me>
* Disable too-many-ancestors pylint check.
This check was only enabled for ansible-test code.
In some cases this check counts 1 more ancestor in python 3.7+ than under python 3.6, making the check inconsistent and unsuitable for use in ansible-test.
* Disable additional pylint checks:
- import-error
- no-name-in-module
These checks were only enabled for ansible-test code.
These checks provide inconsistent results since they are dependent on available imports, which vary between environments.
It may be practical to enable these checks in the future if changes are made to ensure a consistent test environment for pylint.
* Test galaxy cli against pulp
* linting fix
* Renames and small fixes
* Better handling for resetting pulp
* Clean up some things, add a comment
* I can't spell
* Bump fallaxy, use alternate pulp image
* Only reset pulp when we're are executing against pulp
* Update for updated pulp container
* Update some comments with correct URLs and typos
* Linting fix
* Pin pulp-fedora31 to a digest
* Address review comments for documentation
The current author line wants to match a github author id. But
some people, including the OpenStack project, do not use github,
and additionally do not claim individual ownership but instead
group ownership.
Since there are already a couple of hard-coded examples in the
regex, just add one more. Alternately we could come up with some
mechanism to indicate that the author is purposely not listing
a github id, but that seems a bit heavywight.
* try to load unqualified plugins from whitelist
* necessary for backcompat loading of unqualified collectionized callback plugins redirected from <= 2.9 core
* also added de-duping from actual loaded name
* add tests
* add warning test
* group test script entries by topic
* shorten warning text grep because wrapping is dumb
* fix adhoc callback loading behavior
* collections pass over whitelist wasn't respecting `_run_additional_callbacks`
* adds regression tests for same
* avoid `grep -L` in tests since it breaks the world
The upcoming pyparsing 3 release will require Python 3.5 or later, see:
https://github.com/pypa/packaging/issues/313
Unfortunately pip 8.x and earlier versions do not support python version requirements, which is why this constraint is needed.
* builtin downstream vendoring support
* allows downstream packagers to install packages to `ansible/_vendor` that will automatically be added to head of sys.path during `ansible` package load
* tests
* sort conflicting package names in warning text
* sanity fixes
* skip unnecessary comparison
If we fail to find a member when extracting a directory, try adding a trailing
slash to the member name. In certain cases, the member in the tarfile will
contain a trailing slash but the file name in FILES.json will never contain
the trailing slash.
If unable to find the member, handle the KeyError and print a nicer error.
Also check if a directory exists before creating it since it may have been
extracted from the archive.
Fixes#70009
* Add unit tests
* Use loop for trying to get members
Matt Clay