This allows collections to specify requirements and constraints for packages that ansible-test has requirements or constraints for.
(cherry picked from commit 5f76bd2af7)
Co-authored-by: Matt Clay <matt@mystile.com>
* Make Azure Pipelines resource_prefix lowercase.
* Make classification of CI files consistent.
* Update package-data sanity test for AZP.
(cherry picked from commit 92b66e3e31)
Co-authored-by: Matt Clay <mclay@redhat.com>
* updates intersphinx references for docs links (#71921)
* DOCS: updates intersphinx references for docs links
* TESTS: Raise the number of bytes scanned to determine if a file is binary. The newest ansible-2.10.inv file has its first null byte at position 2261. 4096 is still a cheap chunksize to read so it still makes sense to raise this.
Co-authored-by: Alicia Cozine <acozine@users.noreply.github.com>
Co-authored-by: Toshio Kuratomi <a.badger@gmail.com>
(cherry picked from commit 27826827e9)
* adds changelog for already-merged PR (#71947)
* adds changelogs/fragments/71921-raise-bytes-for-binary-test.yml
Co-authored-by: Matt Clay <matt@mystile.com>
(cherry picked from commit 91b0dfb659)
Use of this new version is experimental, so it is not enabled in CI yet.
(cherry picked from commit 2bbcbe99fd)
Co-authored-by: Matt Clay <matt@mystile.com>
* Updates to docs build for new antsibull CLI
* --ansible-base-cache renamed to --ansible-base-source
* _acd_version in the .deps file has been renamed to _ansible_version
(cherry picked from commit 22d2c97b81)
* Fix expr regex for MacOSX compat
MacOSX seems to want bare `+` whereas GNU expr wants escaped `+` (`\+`)
to mean match one or more. Use `\{1,\}` instead which will match one or
more on both MaxOSX and GNU-using systems.
Fixes#71053
(cherry picked from commit 99cac0b135)
* Force an upgrade to a newer version of antsibull
(cherry picked from commit d816a5966e)
* Bump antsibull-changelog version.
* Flag all dotfiles, except .keep and .gitkeep.
* Enable ignoring other fragment extensions.
(cherry picked from commit fbfc0f99eb)
There are links from the scenario guides to collections but collections
docs aren't built in testing or locally when we're on the devel branch.
Due to that we need to make sure those references resolve to the
production docsite. We can use intersphinx to make sure that happens.
* The test for binary files wasn't reading enough of the file.
Checking for null bytes in the first 1024 bytes failed to diagnose the
ansible_2_10.inv file as binary
(cherry picked from commit e10902d744)
Co-authored-by: Toshio Kuratomi <a.badger@gmail.com>
Changes to sanity and unit tests now trigger the ansible-test self-test integration tests.
No changelog entry since this only affects tests for ansible itself and not collections.
(cherry picked from commit b53e7f8720)
Co-authored-by: Matt Clay <matt@mystile.com>
* Misc typo fixes (#71089)
(cherry picked from commit 504ef607f3)
* Add some documentation for the format of meta/runtime.yml (#71035)
* Document the format of meta/runtime.yml
* Document multiple Ansible versions
Clarify difference between deprecation and tombstone fields
* add note
(cherry picked from commit a9eb8b0488)
* add note to uninstall older versions of ansible for pip (#71023)
* add note to uninstall older versions of ansible for pip
* combine with the other PR
(cherry picked from commit 72d3d44163)
* VMware: Inventory scenario guide for hostnames (#71055)
Added a scenario guide for ``hostnames`` parameter
for vmware_vm_inventory.
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
(cherry picked from commit 0055673c70)
* Document string tests a bit more (#71049)
- Explain how `regex` differs from `match` and `search`.
- Document `multiline` and `ignorecase`.
Signed-off-by: Rick Elrod <rick@elrod.me>
(cherry picked from commit 701c638757)
* docs: Add a note about package requirements for fact gathering (#70796)
Fixes: #26148
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
(cherry picked from commit a6725d6e2a)
* added note about fakeroot (#71018)
see #70895
(cherry picked from commit 11a31e99e6)
* Update documentation of httpapi's handle_httperror method for clarity (#70991)
(cherry picked from commit a0523e5b8a)
* DOCS: add 2.10 collections roadmap (#70975)
* draft of 2.10 collections roadmap
* incorporates feedback from felixfontein
* gundalow and samccann feedback, fix link
Co-authored-by: Alicia Cozine <acozine@users.noreply.github.com>
(cherry picked from commit 9879da8e23)
* updates changelog types; some updates for easier translation (#71027)
Co-authored-by: Alicia Cozine <acozine@users.noreply.github.com>
Co-authored-by: Felix Fontein <felix@fontein.de>
(cherry picked from commit 4f4436c124)
* Document common return values with examples (#71046)
* adding return value examples
* shift to console code blocks
* cleaning up whitespace and shortening invocation example
* reordering diff section
(cherry picked from commit 864573a38d)
* Update intro_getting_started.rst (#71039)
Added two additional learning resources in the See also: section- forgot closing backticks
(cherry picked from commit 9850915bd6)
* Guide users to use ansible-runner (#71063)
Update the docs to guide users to use `ansible-runner` instead of using Python API directly. In many use cases, executing Ansible playbooks are sufficient. In those use cases, `ansible-runner` is easier and much stable to use comparing with Python API, but there is no mention of it.
(cherry picked from commit 0c855dc70b)
* Porting guides for ansible-base 2.10 and ansible 2.10 (#70891)
* Fix changelog link title.
* Rename Ansible 2.10 and 2.11 porting guides to Ansible-base porting guides.
* Add stub for automatically generated 2.10 porting guide.
* Move things that should not be in the ansible-base porting guide to the ansible porting guide.
* Apply changes to base porting guides.
* Add remark that ansible-base is mainly for developers.
* Ansible Base -> Ansible-base
* Fix link in base porting guide.
* Add generated porting guide.
* Use same header signs as antsibull-changelog's RST builder.
* Update generated porting guide.
(cherry picked from commit 61b36c6f30)
* Update network platform guides with FQCN (#70699)
* fqcn all the docs things!
(cherry picked from commit 54bee7152b)
* Document how to upgrade to ansible with pip (#70768)
Fixes#70348
(cherry picked from commit 5019335660)
* document how to migrate between collections (#70243)
* document how to migrate between collections
* Apply suggestions from code review
Co-authored-by: John R Barker <john@johnrbarker.com>
Co-authored-by: Felix Fontein <felix@fontein.de>
(cherry picked from commit 58145dff9c)
* remove github link from plugins (#70951)
(cherry picked from commit e28b20d729)
* Add latest rc from ansible-base (#70974)
* Add latest rc from ansible-base
(cherry picked from commit d62dffafb3)
* Document to_json will convert to ASCII strings by default (#70954)
... as reported in issue #68702
(cherry picked from commit 8c48366f1c)
* Update the porting guide for ansible-2.10.0a8 (#71141)
(cherry picked from commit 0a9638ce4b)
Co-authored-by: Abhijeet Kasurde <akasurde@redhat.com>
Co-authored-by: Sloane Hertel <shertel@redhat.com>
Co-authored-by: Rick Elrod <rick@elrod.me>
Co-authored-by: Brian Coca <bcoca@users.noreply.github.com>
Co-authored-by: Nathaniel Case <ncase@redhat.com>
Co-authored-by: Alicia Cozine <879121+acozine@users.noreply.github.com>
Co-authored-by: Terciero <terciero@users.noreply.github.com>
Co-authored-by: Brendon O'Sullivan <49501251+bjosullivan@users.noreply.github.com>
Co-authored-by: EthanHur <ethan0311@gmail.com>
Co-authored-by: Felix Fontein <felix@fontein.de>
Co-authored-by: Baptiste Mille-Mathias <baptiste.millemathias@gmail.com>
Co-authored-by: Toshio Kuratomi <a.badger@gmail.com>
* Also validate top-level version_added.
* Fix error code.
* Produce same version_added validation error in schema than in code (and stop returning it twice).
* Return correct error codes for invalid version_added for options and return values.
* Add changelog.
* Fix forgotten closing braket.
* Accept 'historical' for some top-level version_added.
(cherry picked from commit 7e2cc7db12)
* Fix ansible-test error in community.aws
* Add changelog entry for fix
* Change check from None to string_types
* Update changelogs/fragments/70507-validate-null-author.yaml
clarify wording "or a list of strings"
Co-authored-by: Felix Fontein <felix@fontein.de>
* Update test/lib/ansible_test/_data/sanity/validate-modules/validate_modules/schema.py
clarify wording - single string or not specified valid
Co-authored-by: Felix Fontein <felix@fontein.de>
* Do not fail but return None when given outside list
Co-authored-by: Felix Fontein <felix@fontein.de>
(cherry picked from commit b0d9deeae3)
Co-authored-by: Alan Rominger <arominge@redhat.com>
Parse errors from libyaml now compensate for the offset of the documentation within a module.
(cherry picked from commit 1a0d8a51cd)
Co-authored-by: Matt Clay <matt@mystile.com>
* ansible-test - do not validate blacklisted ps modules
* Update changelogs/fragments/validate-modules-ps-doc-blacklist.yaml
Co-authored-by: Matt Clay <matt@mystile.com>
Co-authored-by: Matt Clay <matt@mystile.com>
(cherry picked from commit 40ce448657)
The `packaging` and `pyparsing` packages are now installed by `ansible-test` during provisioning of RHEL instances to match the downstream vendored versions.
(cherry picked from commit 70c59423fc)
Co-authored-by: Matt Clay <matt@mystile.com>
The current author line wants to match a github author id. But
some people, including the OpenStack project, do not use github,
and additionally do not claim individual ownership but instead
group ownership.
Since there are already a couple of hard-coded examples in the
regex, just add one more. Alternately we could come up with some
mechanism to indicate that the author is purposely not listing
a github id, but that seems a bit heavywight.
The upcoming pyparsing 3 release will require Python 3.5 or later, see:
https://github.com/pypa/packaging/issues/313
Unfortunately pip 8.x and earlier versions do not support python version requirements, which is why this constraint is needed.
* Validate ansible-base & collection's runtime.yml
Add new test `runtime-metadata`
* Schema validation of file
* Error if a a legacy meta/routing.yml exist in a collection
* removal_date OR removal_version
* Add tombstone validation.
* Allow both ISO 8601 date strings and datetime.date objects (from YAML dates).
* Address review comments.
* Add metadata to test collection.
* Add requirements file.
Co-authored-by: Felix Fontein <felix@fontein.de>
Co-authored-by: Matt Clay <matt@mystile.com>
* Add mccabe complexity testing
* Make mccabe complexity an optional error
* Add mccabe to new sanity pylint requirements
* Add a changelog fragment.
Co-authored-by: Matt Clay <matt@mystile.com>
* config: singular ANSIBLE_COLLECTIONS_PATH
Every other *_PATH setting in ansible is singular, and the traditional
$PATH variable is also singular despite containing a list of
directories. Let's be consistent both internally and with POSIX
tradition.
* update all ANSIBLE_COLLECTIONS_PATHS env references to be singular
* deprecate plural ANSIBLE_COLLECTIONS_PATHS setting
Replace the ansible-base changelog linting and generation tool with antsibull-changelog and make it available for linting collections. Previously changelog linting was limited to ansible-base.
Ensure the vcenter provider initialize the `module_defaults` of all
the vmware modules, not just `vmware_guest`.
The VMware CI relies on this for the authentication of the different
VMware modules.
The commit adjust `incidental_vmware_prepare_tests/tasks/init_vcsim.yml`.
The test-suite uses a copy of `vmware_guest` that is not in the
`group/vmware` group. As a result, we need to manually pass the
authentification parameter.
Some code-smell sanity tests for ansible-base use subprocess to invoke ansible commands.
Intercept these commands to make sure the correct script and python version are used.
* Move ansible-test completion code.
* Fix a few type hints.
* Change docker completion based on context.
Collections now use version 2.0 of the default-test-container.
This is an updated version of the earlier 1.x default-test-container with ansible-base and cloud specific requirements removed.
Testing of ansible-base now uses version 1.0 of a new ansible-base-test-container.
This container is similar to the earlier 1.x default-test-container, but with unnecessary duplication of requirements across Python versions removed.
Collections which have tests that depend on requirements no longer present in the default test container should specify them in their test requirements files:
* tests/integration/requirements.txt
* tests/unit/requirements.txt
* Bump test container versions
Co-authored-by: Jordan Borean <jborean93@gmail.com>
* Only allow groups which were hardcoded in module_defaults.yml
only load action groups from the collection if module_defaults contains a potential group for the action
* Fix tests using modules that override those whitelisted in lib/ansible/config/module_defaults.yml
Third party modules should not be using group/ - use the action name instead
* add externalized module_defaults tests
add the missing group and collections
ci_complete
Co-authored-by: Matt Davis <mrd@redhat.com>
* changelog
ci_complete
* Fix import in tests
ci_complete
* Update with requested changes
ci_complete
* don't traceback since we don't validate the contents of module_defaults
ci_complete
Co-authored-by: Matt Davis <mrd@redhat.com>
* Allow to specify collection_name separately for deprecation.
* Use new functionality in Ansible.
* Use new functionality in tests.
* Update tagging/untagging functions.
* Update pylint deprecated sanity test.
* Update validate-modules. Missing are basic checks for version_added (validate semantic version format for collections).
* Improve version validation. Re-add version_added validation.
* Make sure collection names are added to return docs before schema validation.
* Extra checks to avoid crashes on bad data.
* Make C# module utils code work, and update/extend tests.
* Add changelog fragment.
* Stop extracting collection name from potentially tagged versions/dates.
* Simplify C# code.
* Update Windows modules docs.
* Forgot semicolons.
* Split out sanity test requirements.
* Run each --venv test separately.
This provides verification that the requirements for each test are properly specified.
* Use a separate requirements file per sanity test.
* Skip setuptools/cryptography setup for sanity.
* Eliminate pyyaml missing warning.
* Eliminate more pip noise.
* Fix conflicting generate_pip_install commands.
* Add changelog fragment.
* starting metadata sunset
- purged metadata from any requirements
- fix indent in generic handler for yaml content (whey metadata display was off)
- make more resilient against bad formed docs
- removed all metadata from docs template
- remove metadata from schemas
- removed mdata tests and from unrelated tests
Co-authored-by: Felix Fontein <felix@fontein.de>
Co-authored-by: Rick Elrod <rick@elrod.me>
* Work around strange behavior of StrictVersion and SemanticVersion constructors that they accept an falsy value.
* Do not accept empty strings as versions.
* Track collection for version_added.
Validate *all* version numbers in validate-modules.
For tagged version numbers (i.e. version_added), consider source collection to chose validation.
* Make tagging/untagging functions more flexible.
* Tag all versions in doc fragments.
* Tag all deprecation versions issued by code.
* Make Display.deprecated() understand tagged versions.
* Extend validation to enforce tagged version numbers.
* Tag versions in tests.
* Lint and fix test.
* Mention collection name in collection loader's deprecation/removal messages.
* Fix error IDs.
* Handle tagged dates in Display.deprecated().
* Also require that removed_at_date and deprecated_aliases.date are tagged.
* Also automatically tag/untag removed_at_date; fix sanity module removal version check.
* Improve error message when invalid version number is used (like '2.14' in collections).
* Allow to deprecate module by date in documentation.
* Make sure deprecation date/version match between module docs and meta/runtime.yml.
* Unrelated fix: don't compare deprecated module version to Ansible's version in collection.
* Allow documentation's removal version to be something else than fixed list of Ansible versions for collections.
* Linting.
* Allow to deprecate plugin options by date.
* Add changelog fragment for deprecation by date (also covers #68177).
* Allow to deprecate options and aliases by date instead of only by version.
* Update display.deprecate().
* Adjust behavior to conform to tested behavior, extend tests, and improve C# style.
* Parse date and fail on invalid date.
This is mainly to make sure that people start using invalid dates, and we eventually have a mess to clean up.
* C# code: improve validation and update/extend tests.
* Make sure that deprecate() is not called with both date and version.
* Forgot to remove no longer necessary formatting.
* Adjust order of warnings in C# code.
* Adjust unrelated test.
* Fix grammar (and make that test pass).
* Don't parse date, and adjust message to be same as in #67684.
* Sanity tests: disable date in past test.
* Validate-modules: validate ISO 8601 date format.
* Validate-modules: switch schema declaration for deprecated_aliases to improve error messages for invalid dates.
* Use DateTime instead of string for date deprecation.
* Validate that date in deprecated_aliases is actually a DateTime.
* Fix tests.
* Fix rebasing error.
* Adjust error codes for pylint, and add removed_at_date and deprecated_aliases.date checks to validate-modules.
* Make deprecation date in the past error codes optional.
* Make sure not both version and date are specified for AnsibleModule.deprecate() calls.
* Stop using Python 3.7+ API.
* Make sure errors are actually reported. Re-add 'ansible-' prefix.
* Avoid crashing when 'name' isn't there.
* Linting.
* Update lib/ansible/module_utils/csharp/Ansible.Basic.cs
Co-authored-by: Jordan Borean <jborean93@gmail.com>
* Adjust test to latest change.
* Prefer date over version if both end up in Display.deprecated().
Co-authored-by: Jordan Borean <jborean93@gmail.com>
* `meta/` directory in collections
* runtime metadata for redirection/deprecation/removal of plugin loads
* a compatibility layer to keep existing content working on ansible-base + collections
* a Python import redirection layer to keep collections-hosted (and otherwise moved) content importable by things that don't know better
* supported Ansible version validation on collection loads
* Add multipart/form-data functionality
* Fix some linting issues
* Fix error message
* Allow filename to be provided with content
* Add integration test
* Update examples
* General improvements to multipart handling
* Use prepare_multipart for galaxy collection publish
* Properly account for py2 vs py3, ensuring no max header length
* Address test assumptions
* Add unit tests
* Add changelog
* Ensure to use CRLF instead of NL
* Ignore line-endings in fixture
* Consolidate code, add comment
* Bump fallaxy container version
* ci_complete
The updated container includes fewer requirements now that the collection migration has completed.
Collections which encounter test issues with this new container should update their test requirements files to include the necessary requirements.
Change:
Adds Fedora 32 to shippable and alters tests slightly for new Fedora.
Test Plan:
CI
Tickets:
Fixes#69230
Co-authored-by: Matt Clay <matt@mystile.com>
* validate-modules: deprecated modules in collections
In Collections a module is marked as deprecated via meta/routing.yml
Use this file, rather than the leading `_` as part of the deprecated
test.
* Correct variable
* review comments
* indentation
* Read routing.yml only once
* pep8
* Apply suggestions from code review
Co-authored-by: Matt Clay <matt@mystile.com>
* review: remove duplicated conditional
Co-authored-by: Matt Clay <matt@mystile.com>
Left hand side slicing is confusing and slower but maybe more memory
efficient in some circumstances. There is one case where it adds to
code safety: when it's used to substitute a different list in place of a
slice of the original list and the original list could have been bound
to a different variable in some other code. (The most likely case of
this is when it's a global variable and some other code might import
that variable name).
Because of the confusion factor we think it should only be used for the
safety case or where it's been benchmarked and shown to have some sort
of documentatble improvement. At the moment, only one piece of code
falls into those categories so this PR removes all the other instances
of left hand side slicing.
* Testing: Add CentOS Linux On Power platform
* Add arch designation to remotes.
This avoids overloading the provider with the arch.
Also add a changelog entry.
Co-authored-by: Matt Clay <matt@mystile.com>
* Don't trigger full CI run for changes to changelogs/ and docs/ in collections.
* Add changelog fragment.
* Update changelogs/fragments/68550-ansible-test-docs-changelogs.yml
Co-Authored-By: Matt Clay <matt@mystile.com>
Co-authored-by: Matt Clay <matt@mystile.com>
* Update docker.txt to use the OpenSUSE 15.1 container image
Signed-off-by: Rick Elrod <rick@elrod.me>
* handle installing mysql on suse
Signed-off-by: Rick Elrod <rick@elrod.me>
* add changelog fragment
Signed-off-by: Rick Elrod <rick@elrod.me>
* Update changelogs/fragments/ansible-test-opensuse-15.1.yml
Co-Authored-By: Matt Clay <matt@mystile.com>
Co-authored-by: Matt Clay <matt@mystile.com>
* Update tests to use RHEL 7.8.
Keeping support for RHEL 7.6 since collections are still using it.
* Fix tests for RHEL 7.7+ due to extras repo name change.
Matt Clay