* fix coverage output from synthetic packages
* synthetic packages (eg, implicit collection packages without `__init__.py`) were always created at runtime with empty string source, which was compiled to a code object and exec'd during the package load. When run with code coverage, it created a bogus coverage entry (since the `__synthetic__`-suffixed `__file__` entry didn't exist on disk).
* modified collection loader `get_code` to preserve the distinction between `None` (eg synthetic package) and empty string (eg empty `__init__.py`) values from `get_source`, and to return `None` when the source is `None`. This allows the package loader to skip `exec`ing things that truly have no source file on disk, thus not creating bogus coverage entries, while preserving behavior and coverage reporting for empty package inits that actually exist.
* add unit test
On some systems (tested with official Debian Buster-based Python 3.6
docker image), setting the LC_ALL environment variable to en_US.UTF-8
will trigger Python into switching its preferred encoding to ASCII.
If any python process tries to read a non-ASCII file in this scenario,
it will terminate with an error.
And this is exactly what happens to pytest when it tries to load its
configuration that ansible-test supplies because the configuration
contains an em dash.
In order to bypass this issue, we replaced the em dash with a regular
dash and things started working again.
An alternative solution would be to replace the en_US.UTF-8 locale with
something safer, but unfortunately, the en_US.UTF-8 is probably as safe
as it gets.
The setup module can now filter out multiple pattern by providing a list
to the filter parameter instead of just a string. Single string sill
works. Previous behaviour remains.
(cherry picked from commit b5c36dac483fdd74d6c570d77cc8f3e396720366)
* Use InventoryManager for inventory_hostnames lookup. Fixes#17268
* Add test for using patterns
* s/it's/its/
* Allow bypassing parse_sources in InventoryManager
* Remove unneeded method call
Change:
- By default the dnf API does not gpg-verify packages. This is a feature
that is executed in its CLI code. It never made it into Ansible's
usage of the API, so packages were previously not verified.
- This fixes CVE-2020-14365.
Test Plan:
- New integration tests
Signed-off-by: Rick Elrod <rick@elrod.me>
* Validate salt when using crypt. Respect salt_size in password lookup. Repair salt for bcrypt. Fixes#71107. Fixes#53750. Fixes#36129.
* Handle algorithms we don't know about, and make sure to return the salt
* Account for old passlib
* Add tests for salt constraints
* Add changelog fragment
* Add test for #36129
* added configurable dir for tree callback
- allows usage in playbooks
- also made errors more specific if bad dir is supplied
- ensure we expand paths
* docfix
Change:
- Now sends meta tasks to the task start callback
- Lets callback plugins opt-in to receiving implicit tasks
Test Plan:
- New integration tests
Tickets:
- Indirectly fixes#71007 by allowing custom callbacks with this data
Signed-off-by: Rick Elrod <rick@elrod.me>
skip_missing parameter in subelements lookup plugin is accepted from
inside the dictionary.
Fixes: #38182
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* check run state of current block only
* Add changelog and test
* Add test for issue 29047
* Fix for both tests
* blerg
* Change test messages
* fix tests
* Add multi-level block in rescue test case
* Add recursive rescue check and multi-level test
* Should probably run the new test
* ci_complete
* Merge new tests
* ci_complete
* Don't do conflict check on sdist and egg_info. Fixes#71279
* Add changelog
* adjust changelog PR link
Co-authored-by: Rick Elrod <rick@elrod.me>
* add warning about bdist_wheel
Co-authored-by: Rick Elrod <rick@elrod.me>
This would be a partial solution for #69364 in that the SHASUMS file can be downloaded and gpg verified but then used from the downloaded location to verify the get_url's file.
* Make checksum url parsing more explicit
Use urlsplit to test if the checksum string has a (currently tested and) supported url scheme.
* Fix whitespace
* Changelog fragment
* Added tests
* Fix typo in test setup
The message generated by systemctl has been updated in 9321e23c40, which requires a corresponding change in the systemd module.
In addition, this fixes the module when the SYSTEMD_OFFLINE environment variable is set.
* POC for supporting callback events that come from the worker
* linting fixes. ci_complete
* fix up units. ci_complete
* Try moving the sentinel put higher. ci_complete
* safeguards. ci_complete
* Move queue killing to terminate
* LINTING. ci_complete
* Subclass Queue, to add helper send_callback method
* Just use _final_q instead of adding another queue and thread
* Revert a few changes
* Add helper for inserting a TaskResult into the _final_q
* Add changelog fragment
* Address rebase issue
* ci_complete
* Add test to assert async poll callback from fork
* Don't use full path
* ci_complete
* Use _results_lock as a context manager
* Add new generic lock decorator, and use it with send_callback
* Revert "Change default file permissions so they are not world readable (#70221)"
This reverts commit 5260527c4a.
* Revert "Fix warning for new default permissions when mode is not specified (#70976)"
This reverts commit dc79528cc6.
* Ensure -k is set to delegated hosts without a pass
* Fix up some broken tests
* Update task_executor.py
one possible fix, the other is updating winrm to normalize on 'password' like the other connection plugins
* Add alias for winrm and fix incorrect assumption
* Make sure aliases are used for keyword options
* Conditionally run test if sshpass is present, fix sanity
Co-authored-by: Brian Coca <bcoca@users.noreply.github.com>
Change:
- Allow systems to declare multiple virt techs. For example if a system
is both a docker container, but virtualized on KVM, show both. If a
system is set up to run virtualbox and KVM VMs, show both.
- This is done by introducing new facts keys:
- virtualization_tech_guest
- virtualization_tech_host
- Backwards compatibility is preserved by keeping track of the previous
return-points and refusing to update those keys after we would have
returned, but now returning them at the end, so that the new keys can
accumulate their data.
Test Plan:
- Local
- CI
Tickets:
- Refs #66304
- Refs #17151
- Refs #17058
- Probably others
Signed-off-by: Rick Elrod <rick@elrod.me>
* Fix tty_ify bugs and refactor
* Move tty_ify() and supporting attributes to the DocCLI class as that's
the only thing using it.
* Add unittest for the code.
* Fix a bug where the substitution macros can be detected when they are
a part of another word.
* Add support for L(), R(), and HORIZONTALLINE which were added to the
website docs many years ago.
* Update test/units/cli/test_doc.py
Co-authored-by: Matt Clay <matt@mystile.com>
Co-authored-by: Matt Clay <matt@mystile.com>
* add jinja2 global to reserved names
also allow expansion by additional context provided from caller
fixes#41955
Co-authored-by: Abhijeet Kasurde <akasurde@redhat.com>
Change:
- Use `chmod +a` in the fallback chain to allow MacOS to use ACLs to
allow an unprivileged user to become an unprivileged user.
Test Plan:
- CI, new tests
Tickets:
- Fixes#70648
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Add a parameter `read_whole_file` which allows for reading the whole
file when doing a `contains` regex search.
- This allows for (for example) matching a pattern at the very end of
a file.
Test Plan:
- New integration tests
Tickets:
- Fixes#63378
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Previously CachePluginAdjudicator#flush only removed entries from the
cache backend that it knew about by using them earlier. Now it calls
the underlying plugin's flush() method.
Test Plan:
- New unit tests
Tickets:
- Fixes#68770
Signed-off-by: Rick Elrod <rick@elrod.me>
* Strip spaces in module names in explicit actions
Change:
- When an action is called like "action: copy foo=bar", strip spaces
around the action name.
- This allows "action: copy foo=bar" to work as expected.
Test Plan:
- New integration tests
Tickets:
- Fixes#62136
Signed-off-by: Rick Elrod <rick@elrod.me>
Add integration tests for unarchiving as unprivileged user
Break tasks into separate files for easier reading and maintenance
Create a user by specifying a default group of 'staff' for macOS.
The user module does not actually remove the user directory on macOS,
so explicitly remove it.
Put the removal tasks in an always block to ensure they always run
Co-authored-by: Philip Douglass <philip.douglass@amadeus.com>
Co-authored-by: Sam Doran <sdoran@redhat.com>
The iso8601_micro and iso8601 facts incorrectly called now.utcnow(), resulting
in a new timestamp at the time it was called, not a conversion of the previously
stored timestamp.
Correct this by capturing the UTC timestamp once then calculating the local
time using the UTC offset of the current system.
* Use time.time() for getting the current time
* Convert from that stored epoch timestamp to local and UTC times
* Used existing timestamp for epoch time
* Add unit tests that validate the formate of the return value rather than an exact value since mocking time and timezone is non-trivial
Follow up to #70221
Related to #67794
CVE-2020-1736
When set_mode_if_different() is called with mode of 'None', ensure we issue
a warning about the change in default permissions.
Add integration tests to ensure the warning works properly.
* Fix tests
- actually use custom module 🤦♂️
- verify file permission on created files
- use remote_tmp_dir so we're ready for split controller
- improve test module so we can skip the call to set_fs_attributes_if_different()
- fix tests for CentOS 6
Change:
- Allow older git to verify tags again
- Enable verification tests everywhere, even if most of them only work
on newer git. Some of them work on older git and they test the --raw
parameter.
Test Plan:
- Re-enabled subset of git tests
Tickets:
- Fixes#64469
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Previously, we only showed that something would have changed, not what
would have changed. This allows us to show what will chang as well.
Test Plan:
- Local RHEL8 VM
- New integration tests
Tickets:
- Fixes#66132
Signed-off-by: Rick Elrod <rick@elrod.me>
* Also validate top-level version_added.
* Fix error code.
* Produce same version_added validation error in schema than in code (and stop returning it twice).
* Return correct error codes for invalid version_added for options and return values.
* Add changelog.
* Fix forgotten closing braket.
* Accept 'historical' for some top-level version_added.
* dnf: Add nobest option
* dnf: Fix indent, add nobest specifically to dnf not yum
* Add changelog for dnf: add nobest option
* dnf: Add nobest to yumdnf module argument_spec
* dnf: remove nobest from module paramaters in yumdnf.py
* dnf: Add test for nobest option
* dnf: Cleanup packages in nobest test at last
* dnf: Cleanup manually added repos in nobest test at last
* dnf: Remove dnf-plugins-core as well in nobest test
* dnf: Change nobest release version to 2.11
* Change changelog number according to change in PR number
* Change changelog number according to change in PR number
Change:
- On Linux, there are situations where a host might be a KVM host but
not have the kernel module enabled (it might be compiled in instead).
In these cases, /dev/kvm will still exist, and rather than reporting
NA, we should report that the host is a KVM host.
Test Plan:
- Local
Signed-off-by: Rick Elrod <rick@elrod.me>
* Change default file permissions so they are not world readable
CVE-2020-1736
Set the default permissions for files we create with atomic_move() to 0o0660. Track
which files we create that did not exist and warn if the module supports 'mode'
and it was not specified and the module did not call set_mode_if_different(). This allows the user to take action and specify a mode rather than using the defaults.
A code audit is needed to find all instances of modules that call atomic_move()
but do not call set_mode_if_different(). The findings need to be documented in
a changelog since we are not warning. Warning in those instances would be frustrating
to the user since they have no way to change the module code.
- use a set for storing list of created files
- just check the argument spac and params rather than using another property
- improve the warning message to include the default permissions
* ensure hostvars are available on delegation
* also inventory_hostname must point to current host and not delegated one
* fix get_connection since it was still mixing original host vars and delegated ones
* also return connection vars for delegation and non delegation alike
* add test to ensure we have expected usage when directly assigning for non delegated host
Some platform such as ESXi does not implement EpollSelector,
which is selected by DefaultSelector. Use SelectSelector which is
based upon 'select' implementation. This works perfectly with
a platform like VMware ESXi.
Fixes: #70238
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Encode/Decode files in UTF-8
* Use helper function in ansible
* Add an integration test
* Use emoji in test data.
* add changelog
* Also support non-ascii chars in filepath and add tests about this.
* Also use non-ascii chars in replaced text and ensure not to break cron syntax.
* rename self.existing to self.n_existing
* rename crontab.existing to crontab.n_existing
Change:
- Refactoring to make it harder to get wrong and easier to read.
- Generalize become_unprivileged tests and fix some that never worked
but also never failed.
Test Plan:
- CI, new units/integration tests
Signed-off-by: Rick Elrod <rick@elrod.me>
* Fix missing quoting for remote_tmp in second mkdir of shell module. Issue #69577
* adding changelog
* fixing typo in changelog entry
* adding test case
Adding test case written by bmillemayhias.
* using $HOME instead of ~
* fixing commit measage
* Update 69578-shell-remote_tmp-quoting.yaml
Co-authored-by: Brian Kohles <me@briankohles.com>
debconf module exposes sensitive information to logs, console.
Add a note to user about using no_log=True to hide such
information from console.
Fixes: #32386
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
A couple of years ago Slackware -current began using a plus (“+”) at the end of the distribution version string to indicate a future version work-in-progress.
Rearrange distribution_files unit tests to easily support more tests
- add conftest with common fixtures
- use parametrize for testing multiple scenarios
* Add changelog
* Add unit tests for Slackware distribution parsing
* Use correct fixtures for Slackware
Data comes from /etc/slackware-version
Co-authored-by: Sam Doran <sdoran@redhat.com>
Co-authored-by: <Eduard Rozenberg <eduardr@pobox.com>>
A recent updated to psutil, which is a dependency of ansible-runner, fails
to install on older versions of pip.
Commit with the breaking change:
135628639b
pipe lookup plugin uses Popen with shell=True intentionally.
This is considered a security issue if user input is not validated.
Updated docs to reflect this information for the user. Also, added
Bandit B602 documentation link for further reading.
Fixes: #70159
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
Change:
- Add integration tests for various cases
- Fix wrong use of "its" in an exception thrown in varnames when it
throws an AnsibleError, given a term of the wrong type.
Test Plan:
- new tests, CI
Tickets:
- Fixes#70546
Signed-off-by: Rick Elrod <rick@elrod.me>
- ensure we preserve the typeerror part of the exception so loop defereed error handling
can postpone those caused by undefined variables until the when check is done.
- fix tests to comply with the 'new normal'
- human_to_bytes and others can issue TypeError not only on 'non string'
but also bad string that is not convertable.
Co-authored-by: Sloane Hertel <shertel@redhat.com>
Co-authored-by: Sloane Hertel <shertel@redhat.com>
Change:
- Use parse_kv() for parsing in the csvfile lookup plugin. This allows
us to handle multi-word search keys and filenames. Previously, the
plugin split on space and so none of these things worked as expected.
- Add integration tests for csvfile, testing a plethora of weird cases.
Test Plan:
- New integration tests, CI
Tickets:
- Fixes#70545
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Followup to #70504. We need to not pass the 'warn' parameter from the
action plugin either, unless it's True. Otherwise, even though it
defaults to false, we always show the deprecation.
Test Plan:
- Local
Signed-off-by: Rick Elrod <rick@elrod.me>
* speed up modify_module
* Remove debugging
* ci_complete
* Simplify generic_visit, alias to visit, eliminate some attr lookups
* ci_complete
* Add changelog fragment
* Fix ansible-test error in community.aws
* Add changelog entry for fix
* Change check from None to string_types
* Update changelogs/fragments/70507-validate-null-author.yaml
clarify wording "or a list of strings"
Co-authored-by: Felix Fontein <felix@fontein.de>
* Update test/lib/ansible_test/_data/sanity/validate-modules/validate_modules/schema.py
clarify wording - single string or not specified valid
Co-authored-by: Felix Fontein <felix@fontein.de>
* Do not fail but return None when given outside list
Co-authored-by: Felix Fontein <felix@fontein.de>
Change:
- The command warnings feature which suggests that users use modules
instead of certain commands is now deprecated. Its `warn` paramater
and `COMMAND_WARNINGS` configuration options are also deprecated.
Their use will become an error in version 2.13.
Test Plan:
- CI
Signed-off-by: Rick Elrod <rick@elrod.me>
Co-authored-by: Sam Doran <sdoran@redhat.com>
Change:
Our handling of NetBSD virtualization facts led to facts that were just
plain incorrect. One example is reporting Xen even when the system is
running on something completely different (like KVM).
As stated by the reporter of #69352, NetBSD has a better sysctl setting
to use for this information, machdep.hypervisor.
This PR does the following:
- Try to use machdep.hypervisor sysctl value if the other sysctl values
we check don't end up with enough information to be useful
- Only look for /dev/xencons and assume Xen if nothing else works
(Really this should probably return 'unknown' since the file exists on
non-Xen systems and is not very useful).
- Add a few more patterns (Xen matches and also Hyper-V) to
VirtualSysctlDetectionMixin#detect_virt_product.
This change is slightly breaking:
- If the first two attempts at using sysctl worked before,
(machdep.dmi.system-product and machdep.dmi.system-vendor), they will
continue to work.
- For cases when those values didn't work, previously the existence of
/dev/xencons was checked, and if found, we reported 'xen' (even on
non-Xen systems when the file existed). After this PR, we try the
machdep.hypervisor sysctl key before still falling back to
/dev/xencons. This means that in some cases, we might go from
(wrongly) saying "xen" to giving a more accurate value such as "kvm"
or "Hyper-V".
Test Plan:
- Tested with local NetBSD VM and got 'kvm' instead of 'xen' back.
Tickets:
- Fixes#69352
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- module_utils.basic.is_special_selinux_path() used a string ==
bytestring comparison which returned False and made Ansible think that
certain filesystems aren't, in fact, special-cased, when they should
be. Ensure both sides of the == are bytestrings.
Test Plan:
- Added `copy` integration tests for this case.
Tickets:
- Fixes#70244
Signed-off-by: Rick Elrod <rick@elrod.me>
* ansible-doc man formatter: do not crash when description isn't there.
* Change to report a better error message when description is not there.
* Add test.
When user uses home directory in --playbook-dir option
of ansible-inventory command, it warns user about this.
This PR suppress the warning message for user's home directory usage
in ansible-inventory command.
Fixes: #65262
Signed-off-by: Satyajit Bulage <sbulage@redhat.com>
The ssh_args were sometimes not correctly applied to the connection
when using _play_context. Use get_option() instead to ensure the
correct ssh_args are always applied.
* Use the first galaxy server supporting v1 for roles. Fixes#65440
* Add changelog fragment
* This is best effort, fall back to original behavior if something bad happens
* misc collection metadata fixes
* parse collection meta with libyaml if available
* require only Mapping for validation
* add explanatory text for _meta_yml_to_dict
* ignore custom pylint rule
* this code shouldn't import a bunch of stuff from ansible, since it's run under the import sanity test
Change:
- In certain situations, such as when the input string contains null
bytes (\0), syslog.syslog will throw a TypeError. Handle that and
fail_json instead.
Test Plan:
- New test
- ansible-test --docker centos[68] (for py2 and py3 respectively)
Tickets:
- Refs #70269
Signed-off-by: Rick Elrod <rick@elrod.me>
* ansible-test - do not validate blacklisted ps modules
* Update changelogs/fragments/validate-modules-ps-doc-blacklist.yaml
Co-authored-by: Matt Clay <matt@mystile.com>
Co-authored-by: Matt Clay <matt@mystile.com>
* Ensure not to remove existing packages while installing apt packages.
* Make all lines shorter than 160 characters
* Allow removing packages only when upgrading.
* Add integration tests
In some usecases, we want to be able to clone a single branch
of a repository, without using --depth (which implies --single-branch).
* Use branch name when available
- update description of parameter
- consolidate branch or tag checking for easy reuse
* Add changelog
* Use static task imports rather than dynamic includes
* Add integration tests for single_branch
* Account for older versions of git
* Minor tweak to warnings
Co-authored-by: Laurent Coustet <laurent.coustet@clarisys.fr>
Co-authored-by: Sam Doran <sdoran@redhat.com>
* [Bugfix] curses.setupterm() error
When run playbook in celery task, curses.setupterm() will be failed
```
File "<frozen importlib._bootstrap>", line 675, in _load
File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
File "/Users/guang/.virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/action/pause.py", line 45, in <module>
curses.setupterm()
TypeError: argument must be an int, or have a fileno() method.
```
* Add changelog
Co-authored-by: Sam Doran <sdoran@redhat.com>
Handle colon appearing in filename while parsing the mimetype and charset
using file command.
Fixes: #70256
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
Change:
- On Python 3.9, `crypt.crypt` will throw instead of returning `None`
when the algorithm isn't supported. So we catch that and handle it
the same way we handled the algorithm not being supported on 3.8: by
throwing AnsibleError.
Test Plan:
- CI for <=3.8.
- Local for 3.9b3:
ansible -m debug -a "msg=\"{{ 'changeme' | password_hash('bcrypt') }}\"" localhost
Before:
localhost | FAILED! => {
"msg": "Unexpected failure during module execution.",
"stdout": ""
}
After:
localhost | FAILED! => {
"msg": "crypt.crypt does not support 'bcrypt' algorithm"
}
Tickets:
- Fixes#69930
Signed-off-by: Rick Elrod <rick@elrod.me>
The `packaging` and `pyparsing` packages are now installed by `ansible-test` during provisioning of RHEL instances to match the downstream vendored versions.
Change:
- Remove _get_item() alias as it has been deprecated
- Update tests
- Remove relevant sanity curtailment
- Add changelog
Test Plan:
CI, grep
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
Removes with_* loop squashing and tests for 2.11
Test Plan:
CI, and grepped for with_items in package manager integration targets.
There might be some test cases in collections which need to stop testing
this behavior.
Signed-off-by: Rick Elrod <rick@elrod.me>
* Clarify blockinfile docs for insertafter/insertbefore
It's not clear from the docs that these options take effect
only when no marker lines are found in the document.
* Add changelog fragment
The upcoming pyparsing 3 release will require Python 3.5 or later, see:
https://github.com/pypa/packaging/issues/313
Unfortunately pip 8.x and earlier versions do not support python version requirements, which is why this constraint is needed.
* builtin downstream vendoring support
* allows downstream packagers to install packages to `ansible/_vendor` that will automatically be added to head of sys.path during `ansible` package load
* tests
* sort conflicting package names in warning text
* sanity fixes
* skip unnecessary comparison
* Validate ansible-base & collection's runtime.yml
Add new test `runtime-metadata`
* Schema validation of file
* Error if a a legacy meta/routing.yml exist in a collection
* removal_date OR removal_version
* Add tombstone validation.
* Allow both ISO 8601 date strings and datetime.date objects (from YAML dates).
* Address review comments.
* Add metadata to test collection.
* Add requirements file.
Co-authored-by: Felix Fontein <felix@fontein.de>
Co-authored-by: Matt Clay <matt@mystile.com>
* Add mccabe complexity testing
* Make mccabe complexity an optional error
* Add mccabe to new sanity pylint requirements
* Add a changelog fragment.
Co-authored-by: Matt Clay <matt@mystile.com>
When using 'remote_src: yes' and 'mode: preserve', the code handling
the file modes has to be handled on the remote node because it's
the one that has access to the source files. This means that the
copy module itself must handle this, rather than the copy action
plugin (which is where all that logic exists). The copy module
handles this when we copy a single file over. But when it is a
directory as the src parameter value, the mode of the files
beneath it are not considered. Subdirectories are copied with
shutil.copytree() which will preserve permissions automatically.
Individual files are copied with shutil.copyfile() which does NOT
preserve permissions. We need to add some calls to shutil.copymode()
to correct that.
Note: This *always* retains individial file permissions. Specifying
a 'mode' other than 'preserve' when giving a source directory for
the 'src' param does not make sense so will be ignored in that case
only.
Fixes#69783
* Add changelog and test
* config: singular ANSIBLE_COLLECTIONS_PATH
Every other *_PATH setting in ansible is singular, and the traditional
$PATH variable is also singular despite containing a list of
directories. Let's be consistent both internally and with POSIX
tradition.
* update all ANSIBLE_COLLECTIONS_PATHS env references to be singular
* deprecate plural ANSIBLE_COLLECTIONS_PATHS setting
Replace the ansible-base changelog linting and generation tool with antsibull-changelog and make it available for linting collections. Previously changelog linting was limited to ansible-base.
* Tag return value docs if they are a dict (and not str/None).
* Try to parse return docs as YAML.
* Properly dump return values in ansible-doc.
* Adjust plugin formatter.
* Add changelog fragment.
* Don't add 'default' for return values.
* Fix plugin_formatter.
* Only try to parse return docs if they are still a string.
* Add tests.
* Warn if RETURN cannot be parsed.
* Adjust tests. Also test for warning.
* if -> elif (otherwise EXAMPLE will be parsed too).
* Always parse return documentation, and fail if it is invalid YAML.
* Polishing.
* Mostly re-enable ansible-doc tests.
Listing from the local collection seems to be somewhat broken. I assume this
is why the test was disabled.
* Lint and make tests work with Python 2.
* Keep FQCNs in plugins (not modules), i.e. restore previous state.
* use security_fix category in changelogs for CVEs
* these fragments do not say CVE but are security fixes
Co-authored-by: Alicia Cozine <acozine@users.noreply.github.com>
* Support removed_at_date in ansible-doc
Signed-off-by: NilashishC <nilashishchakraborty8@gmail.com>
Changes:
* ansible-doc does not support `removed_at_date` and assumes that
deprecated dict will either have `removed_in` or `version`. This
results in ansible-doc (and hence "sanity --test=ansible-doc")
failing for modules having only `removed_at_date`.
* This patch adds support for `removed_at_date` and also gives it
precedence over `removed_in` or `version`.
* Add tests and changelog
Signed-off-by: NilashishC <nilashishchakraborty8@gmail.com>
This prevents PlayIterator having to go through empty blocks
that were created in filter_tagged_tasks. This should
be a performance improvement for playbooks that mostly skip
tasks with tags.
ci_complete
* Move ansible-test completion code.
* Fix a few type hints.
* Change docker completion based on context.
Collections now use version 2.0 of the default-test-container.
This is an updated version of the earlier 1.x default-test-container with ansible-base and cloud specific requirements removed.
Testing of ansible-base now uses version 1.0 of a new ansible-base-test-container.
This container is similar to the earlier 1.x default-test-container, but with unnecessary duplication of requirements across Python versions removed.
Collections which have tests that depend on requirements no longer present in the default test container should specify them in their test requirements files:
* tests/integration/requirements.txt
* tests/unit/requirements.txt
* Bump test container versions
Co-authored-by: Jordan Borean <jborean93@gmail.com>
* Only allow groups which were hardcoded in module_defaults.yml
only load action groups from the collection if module_defaults contains a potential group for the action
* Fix tests using modules that override those whitelisted in lib/ansible/config/module_defaults.yml
Third party modules should not be using group/ - use the action name instead
* add externalized module_defaults tests
add the missing group and collections
ci_complete
Co-authored-by: Matt Davis <mrd@redhat.com>
* changelog
ci_complete
* Fix import in tests
ci_complete
* Update with requested changes
ci_complete
* don't traceback since we don't validate the contents of module_defaults
ci_complete
Co-authored-by: Matt Davis <mrd@redhat.com>
* Allow to specify collection_name separately for deprecation.
* Use new functionality in Ansible.
* Use new functionality in tests.
* Update tagging/untagging functions.
* Update pylint deprecated sanity test.
* Update validate-modules. Missing are basic checks for version_added (validate semantic version format for collections).
* Improve version validation. Re-add version_added validation.
* Make sure collection names are added to return docs before schema validation.
* Extra checks to avoid crashes on bad data.
* Make C# module utils code work, and update/extend tests.
* Add changelog fragment.
* Stop extracting collection name from potentially tagged versions/dates.
* Simplify C# code.
* Update Windows modules docs.
* Forgot semicolons.
* Split out sanity test requirements.
* Run each --venv test separately.
This provides verification that the requirements for each test are properly specified.
* Use a separate requirements file per sanity test.
* Skip setuptools/cryptography setup for sanity.
* Eliminate pyyaml missing warning.
* Eliminate more pip noise.
* Fix conflicting generate_pip_install commands.
* Add changelog fragment.
* Make AnsibleVaultEncryptedUnicode work more like a string. Fixes#24425
* Remove debugging
* Wrap some things
* Reduce diff
* data should always result in text
* add tests
* Don't just copy and paste, kids
* Add eq and ne back
* Go full UserString copy/paste
* Various version related fixes
* Remove trailing newline
* py2v3
* Add a test that can evaluate whether a variable is vault encrypted
* map was introduces in jinja2 2.7
* moar jinja
* type fix
Co-Authored-By: Sam Doran <sdoran@redhat.com>
* Remove duplicate __hash__
* Fix typo
* Add changelog fragment
* ci_complete
Co-authored-by: Sam Doran <sdoran@redhat.com>
* Auto unroll generators produced by jinja filters
* Unroll for native in finalize
* Fix indentation
Co-authored-by: Sam Doran <sdoran@redhat.com>
* Add changelog fragment
* ci_complete
* Always unroll regardless of jinja2
* ci_complete
Co-authored-by: Sam Doran <sdoran@redhat.com>
Change:
- The `add_host` action now shows an accurate change status.
Test Plan:
- Added a plethora of integration tests.
Tickets:
Fixes#69881
Signed-off-by: Rick Elrod <rick@elrod.me>
* Use fqcr from command module invocation using shell module
Fixes https://github.com/ansible/ansible/issues/69788
Use fully qualified collection reference while invoking
command module from shell module
* Fox review comment
In the case of a free style strategy, it is possible to end up with
multiple hosts trying to include from the same role, however the tasks
being included may be different with the use of tasks_from. Previously
if you had two hosts that were included the same role when the
process_include_results function tries to determine if a included needs
to be run on a specific host, it would end up merging two different
tasks into which ever one was processed first.
This change updates the equality check to also check if the task uuid
associated with the IncludedFile is the same. The previous check only
checked if the task's parent uuid was the same. This breaks down when
both includes have the same parent.
- hosts: all
strategy: free
gather_facts: false
tasks:
- include_role:
name: random_sleep
- block:
- name: set a fact (1)
include_role:
name: set_a_fact
tasks_from: fact1.yml
- name: set a fact (2)
include_role:
name: set_a_fact
tasks_from: fact2.yml
- name: include didn't run
fail:
msg: >
set_a_fact didn't run
fact1: {{ fact1 | default('not defined')}}
fact2: {{ fact2 | default('not defined') }}"
when: (fact1 is not defined or fact2 is not defined)
Closes#69521
To avoid issues with Flatcar Container Linux being unable to be found,
detect Flatcar distro name especially for hostname, just like CoreOS
Container Linux was supported.
See also https://github.com/ansible/ansible/issues/69516
* do not return the body even if it failed
* add some tests for this and rebase
* import test task
* ignore_errors when fails
Co-authored-by: Jack Zhang <jack.zhang@aspiraconnect.com>
Change:
On OpenBSD when using pipelining, we do not set cwd which results in a
permissions fatal. Ensure that `''` - cwd - is not in `sys.path`.
Test Plan:
Tested against local OpenBSD VM
Tickets:
Fixes#69320
Signed-off-by: Rick Elrod <rick@elrod.me>
* Enable installing collections from git repositories
* Add tests for installing individual and multiple collections from git repositories
* Test to make sure recursive dependencies with different syntax are deduplicated
* Add documentation
* add a changelog
* Skip Python 2.6
* Only fail if no collections are located in a git repository
Add support for a 'type' key for collections in requirement.yml files.
Update the changelog and document the supported keys and allowed values for the type.
Add a note that the collection(s) in the repo must contain a galaxy.yml
* Add a warning about embedding credentials in SCM URLs
* Update with review suggestions
* suppress sanity compile failure for Python 2.6
* Track collection for version_added.
Validate *all* version numbers in validate-modules.
For tagged version numbers (i.e. version_added), consider source collection to chose validation.
* Make tagging/untagging functions more flexible.
* Tag all versions in doc fragments.
* Tag all deprecation versions issued by code.
* Make Display.deprecated() understand tagged versions.
* Extend validation to enforce tagged version numbers.
* Tag versions in tests.
* Lint and fix test.
* Mention collection name in collection loader's deprecation/removal messages.
* Fix error IDs.
* Handle tagged dates in Display.deprecated().
* Also require that removed_at_date and deprecated_aliases.date are tagged.
* Also automatically tag/untag removed_at_date; fix sanity module removal version check.
* Improve error message when invalid version number is used (like '2.14' in collections).
* Allow to deprecate module by date in documentation.
* Make sure deprecation date/version match between module docs and meta/runtime.yml.
* Unrelated fix: don't compare deprecated module version to Ansible's version in collection.
* Allow documentation's removal version to be something else than fixed list of Ansible versions for collections.
* Linting.
* Allow to deprecate plugin options by date.
* Add changelog fragment for deprecation by date (also covers #68177).
PR #66461 introduced a regression that resulted in an in correct block in the file
if the block to be inserted did not end with a line separator. Fix this bug and add
tests to cover this scenario.
Fixes#64966
* Fix "TypeError: splitlines() takes no keyword arguments" on Python2.7
* Add changelog fragment
* Don't use `grep -P` for BSD/macOS compatibility
* Fix sanity checks complaining about test fixtures with mixed line endings
* Update changelogs/fragments/66461-blockinfile_preserve_line_endings.yaml
Change:
Allows the user to configure sshpass (1.06+) to look for a different
substring than the default "assword" that it comes with.
Test Plan:
Set a custom ssh password prompt on a VM with PAM and tried connecting to
it. Without `ansible_sshpass_prompt` set in inventory: experienced hang.
With `ansible_sshpass_prompt` in inventory: connected successfully.
Tried setting `ansible_sshpass_prompt` with an older `sshpass` in PATH
and got a loud error, as expected.
Tickets:
Fixes#34722, fixes#54743, refs #11565.
Signed-off-by: Rick Elrod <rick@elrod.me>
* `meta/` directory in collections
* runtime metadata for redirection/deprecation/removal of plugin loads
* a compatibility layer to keep existing content working on ansible-base + collections
* a Python import redirection layer to keep collections-hosted (and otherwise moved) content importable by things that don't know better
* supported Ansible version validation on collection loads
* fix delegated interpeter
* allow returning fact if it is 'the right host'
* added note for future fix/efficiency
as it stands we rerun discovery for the delegated host
unless its saving facts to itself
* fixed test lacking delegate_to mock
When mixed with the free strategy (or any custom strategy that does not behave in
a lock-step manner), the linear methodology of _wait_on_handler_results may cause
race conditions with regular task result processing if the strategy uses
_process_pending_results directly. This patch addresses that by splitting the queues
used for results and adding a flag to _process_pending_results to determine which
queue to check.
Fixes#69457
* Add multipart/form-data functionality
* Fix some linting issues
* Fix error message
* Allow filename to be provided with content
* Add integration test
* Update examples
* General improvements to multipart handling
* Use prepare_multipart for galaxy collection publish
* Properly account for py2 vs py3, ensuring no max header length
* Address test assumptions
* Add unit tests
* Add changelog
* Ensure to use CRLF instead of NL
* Ignore line-endings in fixture
* Consolidate code, add comment
* Bump fallaxy container version
* ci_complete
* ansible-galaxy - remove warning during collection install
If existing collections do not contain a MANIFEST.json, which is common
for collections under development that were not installed from Ansible
Galaxy, fall back to inspecting galaxy.yml rather than displaying a
warning.
A warning will still be displayed in neither a MANIFEST.json nor
galaxy.yml are present.
* Update unit tests
* Unify ansible-galaxy install -r
* Minor nit fixes for docs
* Re-align warnings
* Fix up integration test
* Fix up test where no roles/collections were in file
This fact reflects the number of usable vcpus (which might be different
from ansible_processor_vcpus, e.g., in containers with limits). See
also #51504.
* Add fixture data and update unit tests
Co-authored-by: Sam Doran <sdoran@redhat.com>
The updated container includes fewer requirements now that the collection migration has completed.
Collections which encounter test issues with this new container should update their test requirements files to include the necessary requirements.
* Fix filedescriptor out of range in select() when running commands
* Simplify the run_command() code
Now that we're using selectors in run_command(), we can simplify some of
the code.
* Use fileobj.read() instead of os.read()
* No longer use get_buffer_size() as we can just slurp all of the data
instead.
Also use a simpler conditional check of whether the selector map is
empty
Co-authored-by: Toshio Kuratomi <a.badger@gmail.com>
* only show_vars when showing vars
avoid processing function params that can be very expensive
and might not be used at all in called function.
fixes#69357
* Update changelogs/fragments/69357_optimize_inventory_graph_wo_vars.yml
Co-authored-by: Sloane Hertel <shertel@redhat.com>
Co-authored-by: Sloane Hertel <shertel@redhat.com>
Since Ansible 2.9.8, if the fileglob plugin is passed a path containing
a subdirectory of a non-existent directory, it will fail. For example:
lookup('fileglob', '/'): ok
lookup('fileglob', '/foo'): (non-existent): ok
lookup('fileglob', '/foo/bar'): (non-existent): FAIL
The exact error depends on Python 2 or 3, but here is the error on
Python 2:
AttributeError: 'NoneType' object has no attribute 'endswith'
And on Python 3:
TypeError: expected str, bytes or os.PathLike object, not NoneType
This change fixes the issue by skipping paths that are falsey before
passing them to os.path.join().
Fixes: #69450
* Fix galaxy publish sha256 value format.
The multipart/form content used for the body
of the POST to /api/automation-hub/v3/collections
was missing a newline before the line with the value
of the sha256.
automation-hub/galaxy/django skips the field entirely in
that case and automation-hub code will use None for default
to indicate that no sha256 is provided (an available option).
Fixesansible/galaxy-dev#246
* Add changelog fragment
Co-authored-by: Matt Martz <matt@sivel.net>
Change:
Adds Fedora 32 to shippable and alters tests slightly for new Fedora.
Test Plan:
CI
Tickets:
Fixes#69230
Co-authored-by: Matt Clay <matt@mystile.com>
* validate-modules: deprecated modules in collections
In Collections a module is marked as deprecated via meta/routing.yml
Use this file, rather than the leading `_` as part of the deprecated
test.
* Correct variable
* review comments
* indentation
* Read routing.yml only once
* pep8
* Apply suggestions from code review
Co-authored-by: Matt Clay <matt@mystile.com>
* review: remove duplicated conditional
Co-authored-by: Matt Clay <matt@mystile.com>
* use orderdict for yaml dictionaries
* clog
* SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
* allow user to toggle odict off
* removed config, since requored to load config
* remove unused import
* Update changelogs/fragments/yaml_orderd_mappings.yml
Co-authored-by: Matt Clay <matt@mystile.com>
* Update lib/ansible/parsing/yaml/objects.py
Co-authored-by: Matt Clay <matt@mystile.com>
* Update lib/ansible/parsing/yaml/objects.py
Co-authored-by: Matt Clay <matt@mystile.com>
Co-authored-by: Matt Clay <matt@mystile.com>
Sloane Hertel