* Change default file permissions so they are not world readable
CVE-2020-1736
Set the default permissions for files we create with atomic_move() to 0o0660. Track
which files we create that did not exist and warn if the module supports 'mode'
and it was not specified and the module did not call set_mode_if_different(). This allows the user to take action and specify a mode rather than using the defaults.
A code audit is needed to find all instances of modules that call atomic_move()
but do not call set_mode_if_different(). The findings need to be documented in
a changelog since we are not warning. Warning in those instances would be frustrating
to the user since they have no way to change the module code.
- use a set for storing list of created files
- just check the argument spac and params rather than using another property
- improve the warning message to include the default permissions
* ensure hostvars are available on delegation
* also inventory_hostname must point to current host and not delegated one
* fix get_connection since it was still mixing original host vars and delegated ones
* also return connection vars for delegation and non delegation alike
* add test to ensure we have expected usage when directly assigning for non delegated host
* Encode/Decode files in UTF-8
* Use helper function in ansible
* Add an integration test
* Use emoji in test data.
* add changelog
* Also support non-ascii chars in filepath and add tests about this.
* Also use non-ascii chars in replaced text and ensure not to break cron syntax.
* rename self.existing to self.n_existing
* rename crontab.existing to crontab.n_existing
Change:
- Refactoring to make it harder to get wrong and easier to read.
- Generalize become_unprivileged tests and fix some that never worked
but also never failed.
Test Plan:
- CI, new units/integration tests
Signed-off-by: Rick Elrod <rick@elrod.me>
* Fix missing quoting for remote_tmp in second mkdir of shell module. Issue #69577
* adding changelog
* fixing typo in changelog entry
* adding test case
Adding test case written by bmillemayhias.
* using $HOME instead of ~
* fixing commit measage
* Update 69578-shell-remote_tmp-quoting.yaml
Co-authored-by: Brian Kohles <me@briankohles.com>
A couple of years ago Slackware -current began using a plus (“+”) at the end of the distribution version string to indicate a future version work-in-progress.
Rearrange distribution_files unit tests to easily support more tests
- add conftest with common fixtures
- use parametrize for testing multiple scenarios
* Add changelog
* Add unit tests for Slackware distribution parsing
* Use correct fixtures for Slackware
Data comes from /etc/slackware-version
Co-authored-by: Sam Doran <sdoran@redhat.com>
Co-authored-by: <Eduard Rozenberg <eduardr@pobox.com>>
* Build documentation for Ansible-2.10 (formerly known as ACD).
Builds plugin docs from collections whose source is on galaxy
The new command downloads collections from galaxy, then finds the
plugins inside of them to get the documentation for those plugins.
* Update the python syntax checks
* docs builds can now require python 3.6+.
* Move plugin formatter code out to an external tool, antsibull-docs.
Collection owners want to be able to extract docs for their own
websites as well.
* The jinja2 filters, tests, and other support code have moved to antsibull
* Remove document_plugins as that has now been integrated into antsibull-docs
* Cleanup and bugfix to other build script code:
* The Commands class needed to have its metaclass set for abstractmethod
to work correctly
* Fix lint issues in some command plugins
* Add the docs/docsite/rst/collections to .gitignore as
everything in that directory will be generated so we don't want any of
it saved in the git repository
* gitignore the build dir and remove edit docs link on module pages
* Add docs/rst/collections as a directory to remove on make clean
* Split the collections docs from the main docs
* remove version and edit on github
* remove version banner for just collections
* clarify examples need collection keyword defined
* Remove references to plugin documentation locations that no longer exist.
* Perhaps the pages in plugins/*.rst should be deprecated
altogether and their content moved?
* If not, perhaps we want to rephrase and link into the collection
documentation?
* Or perhaps we want to link to the plugins which are present in
collections/ansible/builtin?
* Remove PYTHONPATH from the build-ansible calls
One of the design goals of the build-ansible.py script was for it to
automatically set its library path to include the checkout of ansible
and the library of code to implement itself. Because it automatically
includes the checkout of ansible, we don't need to set PYTHONPATH in
the Makefile any longer.
* Create a command to only build ansible-base plugin docs
* When building docs for devel, only build the ansible-base docs for
now. This is because antsibull needs support for building a "devel
tree" of docs. This can be changed once that is implemented
* When building docs for the sanity tests, only build the ansible-base
plugin docs for now. Those are the docs which are in this repo so
that seems appropriate for now.
A recent updated to psutil, which is a dependency of ansible-runner, fails
to install on older versions of pip.
Commit with the breaking change:
135628639b
* Get m_u.facts.utils coverage up to 100%
Change:
- Add tests to 'gathering_facts' integration target to get
module_utils.facts.utils coverage up to 100%.
- This also clears incidental coverage from incidental_selinux.
Test Plan:
- CI
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Test async_wrapper when the module it runs has stderr output
Test Plan:
- CI
- Looked at coverage report and saw green for a few lines that weren't
previously green.
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Add integration tests for various cases
- Fix wrong use of "its" in an exception thrown in varnames when it
throws an AnsibleError, given a term of the wrong type.
Test Plan:
- new tests, CI
Tickets:
- Fixes#70546
Signed-off-by: Rick Elrod <rick@elrod.me>
- ensure we preserve the typeerror part of the exception so loop defereed error handling
can postpone those caused by undefined variables until the when check is done.
- fix tests to comply with the 'new normal'
- human_to_bytes and others can issue TypeError not only on 'non string'
but also bad string that is not convertable.
Co-authored-by: Sloane Hertel <shertel@redhat.com>
Co-authored-by: Sloane Hertel <shertel@redhat.com>
Change:
- Use parse_kv() for parsing in the csvfile lookup plugin. This allows
us to handle multi-word search keys and filenames. Previously, the
plugin split on space and so none of these things worked as expected.
- Add integration tests for csvfile, testing a plethora of weird cases.
Test Plan:
- New integration tests, CI
Tickets:
- Fixes#70545
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Move hostnamectl check out of GenericStrategy because it was incorrect
for everything except the SystemdStrategy which is where it belongs.
- Add some initial tests for the hostname module, though we are limited
by the fact that we can't do much testing with it in containers.
Test Plan:
- new hostname integration tests
Signed-off-by: Rick Elrod <rick@elrod.me>
Co-authored-by: Sam Doran <sdoran@redhat.com>
* Fix ansible-test error in community.aws
* Add changelog entry for fix
* Change check from None to string_types
* Update changelogs/fragments/70507-validate-null-author.yaml
clarify wording "or a list of strings"
Co-authored-by: Felix Fontein <felix@fontein.de>
* Update test/lib/ansible_test/_data/sanity/validate-modules/validate_modules/schema.py
clarify wording - single string or not specified valid
Co-authored-by: Felix Fontein <felix@fontein.de>
* Do not fail but return None when given outside list
Co-authored-by: Felix Fontein <felix@fontein.de>
Change:
- The command warnings feature which suggests that users use modules
instead of certain commands is now deprecated. Its `warn` paramater
and `COMMAND_WARNINGS` configuration options are also deprecated.
Their use will become an error in version 2.13.
Test Plan:
- CI
Signed-off-by: Rick Elrod <rick@elrod.me>
Co-authored-by: Sam Doran <sdoran@redhat.com>
Change:
- module_utils.basic.is_special_selinux_path() used a string ==
bytestring comparison which returned False and made Ansible think that
certain filesystems aren't, in fact, special-cased, when they should
be. Ensure both sides of the == are bytestrings.
Test Plan:
- Added `copy` integration tests for this case.
Tickets:
- Fixes#70244
Signed-off-by: Rick Elrod <rick@elrod.me>
* ansible-doc man formatter: do not crash when description isn't there.
* Change to report a better error message when description is not there.
* Add test.
Change:
- Update bundled six to 1.13 (last with py2.6 support)
- Make it pass lint
- Fix check to allow skipping over compat __init__.py files we authored
- Fix check to allow files that can't be updated for some reason
Test Plan:
- ansible-test sanity --docker
- CI
Signed-off-by: Rick Elrod <rick@elrod.me>
* Test against galaxy_ng
* Switch container image
* Remove redundant |default
* Re-enable
* Update image
* Update wording
* Don't use pulp as the container name
Change:
- In certain situations, such as when the input string contains null
bytes (\0), syslog.syslog will throw a TypeError. Handle that and
fail_json instead.
Test Plan:
- New test
- ansible-test --docker centos[68] (for py2 and py3 respectively)
Tickets:
- Refs #70269
Signed-off-by: Rick Elrod <rick@elrod.me>
* ansible-test - do not validate blacklisted ps modules
* Update changelogs/fragments/validate-modules-ps-doc-blacklist.yaml
Co-authored-by: Matt Clay <matt@mystile.com>
Co-authored-by: Matt Clay <matt@mystile.com>
CIFS can be mounted using backward slash as well in /etc/fstab like
\\Windows\share /data/ cifs credentials=/root/.creds 0 0
Handle this condition while gather mount information in Linux.
Fixes: #48813
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Update documentation as per sanity tests
* Added example about custom SSH port in example section
Fixes: #29236
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Ensure not to remove existing packages while installing apt packages.
* Make all lines shorter than 160 characters
* Allow removing packages only when upgrading.
* Add integration tests
Change:
- Fix a UnicodeDecodeError in executor.module_common that could get
triggered with -vvvvv.
Test Plan:
- `ansible-test integration --docker centos7 module_utils -vvvvv`
This would show the error previously, and no loner does after this
patch.
Signed-off-by: Rick Elrod <rick@elrod.me>
In some usecases, we want to be able to clone a single branch
of a repository, without using --depth (which implies --single-branch).
* Use branch name when available
- update description of parameter
- consolidate branch or tag checking for easy reuse
* Add changelog
* Use static task imports rather than dynamic includes
* Add integration tests for single_branch
* Account for older versions of git
* Minor tweak to warnings
Co-authored-by: Laurent Coustet <laurent.coustet@clarisys.fr>
Co-authored-by: Sam Doran <sdoran@redhat.com>
Handle colon appearing in filename while parsing the mimetype and charset
using file command.
Fixes: #70256
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
The `packaging` and `pyparsing` packages are now installed by `ansible-test` during provisioning of RHEL instances to match the downstream vendored versions.
Change:
- Remove _get_item() alias as it has been deprecated
- Update tests
- Remove relevant sanity curtailment
- Add changelog
Test Plan:
CI, grep
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
Removes with_* loop squashing and tests for 2.11
Test Plan:
CI, and grepped for with_items in package manager integration targets.
There might be some test cases in collections which need to stop testing
this behavior.
Signed-off-by: Rick Elrod <rick@elrod.me>
* Disable too-many-ancestors pylint check.
This check was only enabled for ansible-test code.
In some cases this check counts 1 more ancestor in python 3.7+ than under python 3.6, making the check inconsistent and unsuitable for use in ansible-test.
* Disable additional pylint checks:
- import-error
- no-name-in-module
These checks were only enabled for ansible-test code.
These checks provide inconsistent results since they are dependent on available imports, which vary between environments.
It may be practical to enable these checks in the future if changes are made to ensure a consistent test environment for pylint.
* Test galaxy cli against pulp
* linting fix
* Renames and small fixes
* Better handling for resetting pulp
* Clean up some things, add a comment
* I can't spell
* Bump fallaxy, use alternate pulp image
* Only reset pulp when we're are executing against pulp
* Update for updated pulp container
* Update some comments with correct URLs and typos
* Linting fix
* Pin pulp-fedora31 to a digest
* Address review comments for documentation
The current author line wants to match a github author id. But
some people, including the OpenStack project, do not use github,
and additionally do not claim individual ownership but instead
group ownership.
Since there are already a couple of hard-coded examples in the
regex, just add one more. Alternately we could come up with some
mechanism to indicate that the author is purposely not listing
a github id, but that seems a bit heavywight.
* try to load unqualified plugins from whitelist
* necessary for backcompat loading of unqualified collectionized callback plugins redirected from <= 2.9 core
* also added de-duping from actual loaded name
* add tests
* add warning test
* group test script entries by topic
* shorten warning text grep because wrapping is dumb
* fix adhoc callback loading behavior
* collections pass over whitelist wasn't respecting `_run_additional_callbacks`
* adds regression tests for same
* avoid `grep -L` in tests since it breaks the world
The upcoming pyparsing 3 release will require Python 3.5 or later, see:
https://github.com/pypa/packaging/issues/313
Unfortunately pip 8.x and earlier versions do not support python version requirements, which is why this constraint is needed.
* builtin downstream vendoring support
* allows downstream packagers to install packages to `ansible/_vendor` that will automatically be added to head of sys.path during `ansible` package load
* tests
* sort conflicting package names in warning text
* sanity fixes
* skip unnecessary comparison
If we fail to find a member when extracting a directory, try adding a trailing
slash to the member name. In certain cases, the member in the tarfile will
contain a trailing slash but the file name in FILES.json will never contain
the trailing slash.
If unable to find the member, handle the KeyError and print a nicer error.
Also check if a directory exists before creating it since it may have been
extracted from the archive.
Fixes#70009
* Add unit tests
* Use loop for trying to get members
* Validate ansible-base & collection's runtime.yml
Add new test `runtime-metadata`
* Schema validation of file
* Error if a a legacy meta/routing.yml exist in a collection
* removal_date OR removal_version
* Add tombstone validation.
* Allow both ISO 8601 date strings and datetime.date objects (from YAML dates).
* Address review comments.
* Add metadata to test collection.
* Add requirements file.
Co-authored-by: Felix Fontein <felix@fontein.de>
Co-authored-by: Matt Clay <matt@mystile.com>
* Add mccabe complexity testing
* Make mccabe complexity an optional error
* Add mccabe to new sanity pylint requirements
* Add a changelog fragment.
Co-authored-by: Matt Clay <matt@mystile.com>
When using 'remote_src: yes' and 'mode: preserve', the code handling
the file modes has to be handled on the remote node because it's
the one that has access to the source files. This means that the
copy module itself must handle this, rather than the copy action
plugin (which is where all that logic exists). The copy module
handles this when we copy a single file over. But when it is a
directory as the src parameter value, the mode of the files
beneath it are not considered. Subdirectories are copied with
shutil.copytree() which will preserve permissions automatically.
Individual files are copied with shutil.copyfile() which does NOT
preserve permissions. We need to add some calls to shutil.copymode()
to correct that.
Note: This *always* retains individial file permissions. Specifying
a 'mode' other than 'preserve' when giving a source directory for
the 'src' param does not make sense so will be ignored in that case
only.
Fixes#69783
* Add changelog and test
* config: singular ANSIBLE_COLLECTIONS_PATH
Every other *_PATH setting in ansible is singular, and the traditional
$PATH variable is also singular despite containing a list of
directories. Let's be consistent both internally and with POSIX
tradition.
* update all ANSIBLE_COLLECTIONS_PATHS env references to be singular
* deprecate plural ANSIBLE_COLLECTIONS_PATHS setting
Replace the ansible-base changelog linting and generation tool with antsibull-changelog and make it available for linting collections. Previously changelog linting was limited to ansible-base.
* Tag return value docs if they are a dict (and not str/None).
* Try to parse return docs as YAML.
* Properly dump return values in ansible-doc.
* Adjust plugin formatter.
* Add changelog fragment.
* Don't add 'default' for return values.
* Fix plugin_formatter.
* Only try to parse return docs if they are still a string.
* Add tests.
* Warn if RETURN cannot be parsed.
* Adjust tests. Also test for warning.
* if -> elif (otherwise EXAMPLE will be parsed too).
* Always parse return documentation, and fail if it is invalid YAML.
* Polishing.
* Mostly re-enable ansible-doc tests.
Listing from the local collection seems to be somewhat broken. I assume this
is why the test was disabled.
* Lint and make tests work with Python 2.
* Keep FQCNs in plugins (not modules), i.e. restore previous state.
* Support removed_at_date in ansible-doc
Signed-off-by: NilashishC <nilashishchakraborty8@gmail.com>
Changes:
* ansible-doc does not support `removed_at_date` and assumes that
deprecated dict will either have `removed_in` or `version`. This
results in ansible-doc (and hence "sanity --test=ansible-doc")
failing for modules having only `removed_at_date`.
* This patch adds support for `removed_at_date` and also gives it
precedence over `removed_in` or `version`.
* Add tests and changelog
Signed-off-by: NilashishC <nilashishchakraborty8@gmail.com>
Ensure the vcenter provider initialize the `module_defaults` of all
the vmware modules, not just `vmware_guest`.
The VMware CI relies on this for the authentication of the different
VMware modules.
The commit adjust `incidental_vmware_prepare_tests/tasks/init_vcsim.yml`.
The test-suite uses a copy of `vmware_guest` that is not in the
`group/vmware` group. As a result, we need to manually pass the
authentification parameter.
Some code-smell sanity tests for ansible-base use subprocess to invoke ansible commands.
Intercept these commands to make sure the correct script and python version are used.
* ansible-galaxy - Fix role info when role is not installed
Only report the role not found if in offline mode, otherwise query the galaxy API
to get role information.
Fixes#69867
* Improve error message when role is not found in Ansible Galaxy
* Extra whitespace around imported playbook filename are stripped.
* Corrected call for display.deprecated
* Changed warning type & added test
* Added auto verification of raised warning
* More accurate warning message
* Move ansible-test completion code.
* Fix a few type hints.
* Change docker completion based on context.
Collections now use version 2.0 of the default-test-container.
This is an updated version of the earlier 1.x default-test-container with ansible-base and cloud specific requirements removed.
Testing of ansible-base now uses version 1.0 of a new ansible-base-test-container.
This container is similar to the earlier 1.x default-test-container, but with unnecessary duplication of requirements across Python versions removed.
Collections which have tests that depend on requirements no longer present in the default test container should specify them in their test requirements files:
* tests/integration/requirements.txt
* tests/unit/requirements.txt
* Bump test container versions
Co-authored-by: Jordan Borean <jborean93@gmail.com>
* Only allow groups which were hardcoded in module_defaults.yml
only load action groups from the collection if module_defaults contains a potential group for the action
* Fix tests using modules that override those whitelisted in lib/ansible/config/module_defaults.yml
Third party modules should not be using group/ - use the action name instead
* add externalized module_defaults tests
add the missing group and collections
ci_complete
Co-authored-by: Matt Davis <mrd@redhat.com>
* changelog
ci_complete
* Fix import in tests
ci_complete
* Update with requested changes
ci_complete
* don't traceback since we don't validate the contents of module_defaults
ci_complete
Co-authored-by: Matt Davis <mrd@redhat.com>
* Allow to specify collection_name separately for deprecation.
* Use new functionality in Ansible.
* Use new functionality in tests.
* Update tagging/untagging functions.
* Update pylint deprecated sanity test.
* Update validate-modules. Missing are basic checks for version_added (validate semantic version format for collections).
* Improve version validation. Re-add version_added validation.
* Make sure collection names are added to return docs before schema validation.
* Extra checks to avoid crashes on bad data.
* Make C# module utils code work, and update/extend tests.
* Add changelog fragment.
* Stop extracting collection name from potentially tagged versions/dates.
* Simplify C# code.
* Update Windows modules docs.
* Forgot semicolons.
* Split out sanity test requirements.
* Run each --venv test separately.
This provides verification that the requirements for each test are properly specified.
* Use a separate requirements file per sanity test.
* Skip setuptools/cryptography setup for sanity.
* Eliminate pyyaml missing warning.
* Eliminate more pip noise.
* Fix conflicting generate_pip_install commands.
* Add changelog fragment.
* Make AnsibleVaultEncryptedUnicode work more like a string. Fixes#24425
* Remove debugging
* Wrap some things
* Reduce diff
* data should always result in text
* add tests
* Don't just copy and paste, kids
* Add eq and ne back
* Go full UserString copy/paste
* Various version related fixes
* Remove trailing newline
* py2v3
* Add a test that can evaluate whether a variable is vault encrypted
* map was introduces in jinja2 2.7
* moar jinja
* type fix
Co-Authored-By: Sam Doran <sdoran@redhat.com>
* Remove duplicate __hash__
* Fix typo
* Add changelog fragment
* ci_complete
Co-authored-by: Sam Doran <sdoran@redhat.com>
Matt Davis