PR #72591
This change:
* Adds an artifacts manager that abstracts away extracting the
metadata from artifacts, downloading and caching them in a
temporary location.
* Adds `resolvelib` to direct ansible-core dependencies[0].
* Implements a `resolvelib`-based dependency resolver for
`collection` subcommands that replaces the legacy
in-house code.
This is a dependency resolution library that pip 20.3+ uses
by default. It's now integrated for use for the collection
dependency resolution in ansible-galaxy CLI.
* Refactors of the `ansible-galaxy collection` CLI.
In particular, it:
- reimplements most of the `download`, `install`, `list` and
`verify` subcommands from scratch;
- reuses helper bits previously moved out into external modules;
- replaces the old in-house resolver with a more clear
implementation based on the resolvelib library[0][1][2].
* Adds a multi Galaxy API proxy layer that abstracts accessing the
version and dependencies via API or local artifacts manager.
* Makes `GalaxyAPI` instances sortable.
* Adds string representation methods to `GalaxyAPI`.
* Adds dev representation to `GalaxyAPI`.
* Removes unnecessary integration and unit tests.
* Aligns the tests with the new expectations.
* Adds more tests, integration ones in particular.
[0]: https://pypi.org/p/resolvelib
[1]: https://github.com/sarugaku/resolvelib
[2]: https://pradyunsg.me/blog/2020/03/27/pip-resolver-testing
Co-Authored-By: Jordan Borean <jborean93@gmail.com>
Co-Authored-By: Matt Clay <matt@mystile.com>
Co-Authored-By: Sam Doran <sdoran@redhat.com>
Co-Authored-By: Sloane Hertel <shertel@redhat.com>
Co-Authored-By: Sviatoslav Sydorenko <webknjaz@redhat.com>
Signed-Off-By: Sviatoslav Sydorenko <webknjaz@redhat.com>
* Fix type of mode field in copy module
* Remove module type checking from ignore.txt sanity checks
* Remove mode type
Co-authored-by: Matt Martz <matt@sivel.net>
This change preserves how all the external imports refer to this code
while allowing us to start cutting the spaghetti into more easily
maintainable pieces.
This is a start of the upcoming refactoring effort destined to
eliminate tight coupling, implicit data manipulation, god objects,
abstraction leaks and other code smells.
Essentially, `ansible.galaxy.collection` is going to be a package that
holds parts of the collection management code spread across loosely
coupled modules.
PR: #71931
* Reorder test/sanity/ignore.txt to better see changes
* Remove extra empty line causing pass fail
* Apply suggestions made by Akasurde
* Minor fix in package_facts and remove two unnecessary ignores
* Fix subversion based on a suggestion made by felixfontein
* Apply suggestions made by felixfontein and Andersson007
* Fix subversion.py as suggested by felixfontein
* Minor reformatting in yum_repository description
* Reformat changelog
* Add key to apt_key, add deprecated changelog
* Add PR url to changelog
* Ignore paramater-type-not-in-doc in favour of adding key back to apt_key
* Fix apt_key
* Remove undocumented-paramater from apt_key ignore
* Ignore doc-choices-do-not-match-spec in package_facts
* Fix package_facts
* Fix filter option in setup module
The setup module can now filter out multiple pattern by providing a list
to the filter parameter instead of just a string. Single string sill
works. Previous behaviour remains.
(cherry picked from commit b5c36dac483fdd74d6c570d77cc8f3e396720366)
* Build documentation for Ansible-2.10 (formerly known as ACD).
Builds plugin docs from collections whose source is on galaxy
The new command downloads collections from galaxy, then finds the
plugins inside of them to get the documentation for those plugins.
* Update the python syntax checks
* docs builds can now require python 3.6+.
* Move plugin formatter code out to an external tool, antsibull-docs.
Collection owners want to be able to extract docs for their own
websites as well.
* The jinja2 filters, tests, and other support code have moved to antsibull
* Remove document_plugins as that has now been integrated into antsibull-docs
* Cleanup and bugfix to other build script code:
* The Commands class needed to have its metaclass set for abstractmethod
to work correctly
* Fix lint issues in some command plugins
* Add the docs/docsite/rst/collections to .gitignore as
everything in that directory will be generated so we don't want any of
it saved in the git repository
* gitignore the build dir and remove edit docs link on module pages
* Add docs/rst/collections as a directory to remove on make clean
* Split the collections docs from the main docs
* remove version and edit on github
* remove version banner for just collections
* clarify examples need collection keyword defined
* Remove references to plugin documentation locations that no longer exist.
* Perhaps the pages in plugins/*.rst should be deprecated
altogether and their content moved?
* If not, perhaps we want to rephrase and link into the collection
documentation?
* Or perhaps we want to link to the plugins which are present in
collections/ansible/builtin?
* Remove PYTHONPATH from the build-ansible calls
One of the design goals of the build-ansible.py script was for it to
automatically set its library path to include the checkout of ansible
and the library of code to implement itself. Because it automatically
includes the checkout of ansible, we don't need to set PYTHONPATH in
the Makefile any longer.
* Create a command to only build ansible-base plugin docs
* When building docs for devel, only build the ansible-base docs for
now. This is because antsibull needs support for building a "devel
tree" of docs. This can be changed once that is implemented
* When building docs for the sanity tests, only build the ansible-base
plugin docs for now. Those are the docs which are in this repo so
that seems appropriate for now.
Change:
- Use parse_kv() for parsing in the csvfile lookup plugin. This allows
us to handle multi-word search keys and filenames. Previously, the
plugin split on space and so none of these things worked as expected.
- Add integration tests for csvfile, testing a plethora of weird cases.
Test Plan:
- New integration tests, CI
Tickets:
- Fixes#70545
Signed-off-by: Rick Elrod <rick@elrod.me>
* Update documentation as per sanity tests
* Added example about custom SSH port in example section
Fixes: #29236
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
Change:
- Remove _get_item() alias as it has been deprecated
- Update tests
- Remove relevant sanity curtailment
- Add changelog
Test Plan:
CI, grep
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
Removes with_* loop squashing and tests for 2.11
Test Plan:
CI, and grepped for with_items in package manager integration targets.
There might be some test cases in collections which need to stop testing
this behavior.
Signed-off-by: Rick Elrod <rick@elrod.me>
* Only allow groups which were hardcoded in module_defaults.yml
only load action groups from the collection if module_defaults contains a potential group for the action
* Fix tests using modules that override those whitelisted in lib/ansible/config/module_defaults.yml
Third party modules should not be using group/ - use the action name instead
* add externalized module_defaults tests
add the missing group and collections
ci_complete
Co-authored-by: Matt Davis <mrd@redhat.com>
* changelog
ci_complete
* Fix import in tests
ci_complete
* Update with requested changes
ci_complete
* don't traceback since we don't validate the contents of module_defaults
ci_complete
Co-authored-by: Matt Davis <mrd@redhat.com>
Change:
- This was removed in 2014 in 122a7021bc.
- The option still exists and is enabled by default and can lead to user
confusion when people aren't expecting packages (or updated
dependencies for it) to get installed and they do.
- Add the option documentation back with a few notes to make it clear what
is happening.
Test Plan:
N/A, no code change, just documentation
Tickets:
- Refs #69497
Signed-off-by: Rick Elrod <rick@elrod.me>
* Fix sanity errors
Signed-off-by: Rick Elrod <rick@elrod.me>
* Enable installing collections from git repositories
* Add tests for installing individual and multiple collections from git repositories
* Test to make sure recursive dependencies with different syntax are deduplicated
* Add documentation
* add a changelog
* Skip Python 2.6
* Only fail if no collections are located in a git repository
Add support for a 'type' key for collections in requirement.yml files.
Update the changelog and document the supported keys and allowed values for the type.
Add a note that the collection(s) in the repo must contain a galaxy.yml
* Add a warning about embedding credentials in SCM URLs
* Update with review suggestions
* suppress sanity compile failure for Python 2.6
Gets rid of the unknown field names in the return data.
Allows the plugin return docs to format under the new docs pipeline.
* Expect that package_facts will pass return-syntax-error now.
* `meta/` directory in collections
* runtime metadata for redirection/deprecation/removal of plugin loads
* a compatibility layer to keep existing content working on ansible-base + collections
* a Python import redirection layer to keep collections-hosted (and otherwise moved) content importable by things that don't know better
* supported Ansible version validation on collection loads
* Add multipart/form-data functionality
* Fix some linting issues
* Fix error message
* Allow filename to be provided with content
* Add integration test
* Update examples
* General improvements to multipart handling
* Use prepare_multipart for galaxy collection publish
* Properly account for py2 vs py3, ensuring no max header length
* Address test assumptions
* Add unit tests
* Add changelog
* Ensure to use CRLF instead of NL
* Ignore line-endings in fixture
* Consolidate code, add comment
* Bump fallaxy container version
* ci_complete
* Fix filedescriptor out of range in select() when running commands
* Simplify the run_command() code
Now that we're using selectors in run_command(), we can simplify some of
the code.
* Use fileobj.read() instead of os.read()
* No longer use get_buffer_size() as we can just slurp all of the data
instead.
Also use a simpler conditional check of whether the selector map is
empty
Co-authored-by: Toshio Kuratomi <a.badger@gmail.com>
Change:
Rather than hardcoding .pyo and .pyc, filter on all BLACKLIST_EXTS in
the non-legacy logic of PluginLoader (_find_fq_plugin). The two harcoded
extensions are part of BLACKLIST_EXTS already and this simply adds the
rest of the blacklisted extensions to the check.
In addition, check .endswith() instead of an exact match of the suffix,
like everywhere else that uses BLACKLIST_EXTS. This allows for
blacklisting, for example, emacs's backup files which can appear after
any extension, leading to things like `foo.py~`.
Test Plan:
Ran `ansible-playbook` against a collection where a `foo.py~` module was
getting executed instead of `foo.py` which also appeared in the same
directory. `foo.py~` is no longer executed.
Tickets:
Fixes#22268
Refs #27235
Signed-off-by: Rick Elrod <rick@elrod.me>
With https://github.com/pallets/jinja/pull/1190 merged our short-circuit
is no longer valid (has it ever been?) as now data like ' True ' may go
through our ansible_native_concat function as opposed to going through
intermediate call to Jinja2's native_concat before. Now we need to always
send data through literal_eval to ensure native types are returned.
* Rename `tests` test to match plugin type.
* Rename `test_infra` test to avoid confusion.
This test target is not a test for test plugins.
* Rename `vars_prompt` test to avoid confusion.
* Update sanity ignores.
Sviatoslav Sydorenko