* Make sure paths are correct when building collection files manifest
This commit makes sure the path of the files part of
the collection build manifest are correct.
This commit uses os.path.commonprefix instead of
dealing with strings.
Signed-off-by: Carlos Camacho <ccamacho@redhat.com>
Bugfix Pull Request
Fixes: #81618
* Revert the change note type to `minor_changes`
* Clarify the change note with user-oriented details
---------
Signed-off-by: Carlos Camacho <ccamacho@redhat.com>
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
* User-provided collection type might differ from collection
source. Cross-check the type before proceeding
Fixes: #79463
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Only install collections which can't be satisfied by a collection in any of the configured paths.
* Improve warning for unexpected collection install path
Fix warning when path is configured, but is a pip-managed path
Normalize the path before validating to fix warning consistency
* Improve coverage of validate-modules unit tests
* Remove unused galaxy unit test code
* Fix galaxy unit test teardown logic
* Improve coverage of galaxy unit test code
* Improve coverage of galaxy unit tests
* Remove unused code in galaxy API tests
* Remove unused galaxy collection unit test code
* Improve coverage of galaxy collection unit tests
* Remove unused galaxy unit test code
* Add support for importlib.resources
* Remove the importlib.resources imports
* return the correct data
* Some code comments, and re-order for consistency
* Disallow traversing packages below an individual collection
* Add a traversable class for namespaces
* Re-use variable
* Utilize itertools.chain.from_iterable
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
* Simplify logic to check for packages from ansible loaders
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
* Just a generator expression, instead of a generator
* docstrings
* Add comment about find_spec for our namespaces
* Add some initial unit tests for importlib.resources
* normalize
* Utilize importlib.resources for listing collections
* collections_path is already in config, just use config
* install uses a different default for collections_path
* Remove unused import
* Remove duplicate __truediv__
* Bring back TraversableResources
* Apply some small suggestions from code review
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
Co-authored-by: Matt Davis <6775756+nitzmahone@users.noreply.github.com>
* Remove cross contamination between plugin loader code and CLI code
* Remove unused import
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
Co-authored-by: Matt Davis <6775756+nitzmahone@users.noreply.github.com>
Fix ignoring certs when downloading tarballs
Fix ignoring certs when downloading a collection from a specific source that isn't in the configured servers list
* galaxy: Add license_file to manifest directives
* ag collection build: Test license handling
This adds tests to ensure that
- REUSE licensing files: .reuse/dep5, LICENSES/*, anyfile.license
- galaxy.yml license_file
are always included in the manifest.
* Add --offline option to 'ansible-galaxy collection install' to prevent querying distribution servers
This allows installing/upgrading individual tarfiles to have dependency resolution.
Previously needed to be done manually with --no-deps or else all collections and dependencies needed to be included in the requirements.
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
* Rethread pr/70185 through the dependency resolver
Hang optional metadata toggle on the ConcreteArtifactsManager instead of threading it through whole list codepath
Don't error while listing collections if a collection's metadata is missing keys required for building a collection.
Give an informative warning if metadata has been badly formatted.
Co-authored-by: Sam Doran <sdoran@redhat.com>
* Fix KeyError for ansible-galaxy when caching paginated responses from v3
* changelog
* generate responses in loop for test
Co-authored-by: Matt Martz <matt@sivel.net>
* ansible-galaxy configurable timeouts
- also fixed issues with precedence,
so --ignore-certs now overrides config
- made galaxy_timeout generic setting,
if set, it becomes default for server configs,
but now specific servers can override
- updated tests or added notes (some tests ignore/override precedence)
Co-authored-by: Sloane Hertel <19572925+s-hertel@users.noreply.github.com>
* replace hardcoded '~/.ansible' to C.ANSIBLE_HOME
* rename previously existing env ANSIBLE_HOME in env-setup script
* modify cache dir monkeypatching in galaxy api unit tests
* update "version_added" to 2.14 for ANSIBLE_HOME
* fix description of collections with proper use of ANSIBLE_HOME
Co-authored-by: htol <github@h-tol.net>
Co-authored-by: stefanwascoding <stefan@syntaxhelden.de>
* Add a toggle to control the number of signatures required to verify the authenticity of a collection
* Make the default number of required valid signatures 1
* Add option to make signature verification strict and fail if there are no valid signatures (e.g. "+1")
* Use a regex to validate --required-valid-signature-count
* Add a toggle to limit the gpg status codes that are considered a failure
* Update documentation and changelog
* Add unit and integration tests for the new options
* Fixes#77146
Fix using user-provided signatures when running 'ansible-galaxy collection verify ns.coll --offline'
Add a test for a user-provided signature when running ansible-galaxy collection verify with --offline
Fix displaying overall gpg failure without extra verbosity
Add a test for displaying gpg failure without verbosity
Improve documentation to be more clear that signature verification only currently applies to collections directly sourced from Galaxy servers
* ansible-galaxy collection install|verify:
- Support verifying the origin of the MANIFEST.json when the Galaxy server has provided signatures.
- Allow supplemental signatures to use during verification on the CLI/requirements file.
* ansible-galaxy collection install:
- Support disabling signature verification. This silences the warning provided by ansible-galaxy if the Galaxy server provided signatures it cannot use because no keyring is configured.
- Store Galaxy server metadata alongside installed collections for provenance. This is used by 'ansible-galaxy collection verify --offline'.
* Add unit tests for method that gets signatures from a Galaxy server
* Add integration tests for user-provided signature sources
- Test CLI option combinations
- Test installing collections with valid/invalid signature sources
- Test disabling GPG verification when installing collections
- Test verifying collections with valid/invalid signature sources
* Make signature verification advisory-by-default if signatures are provided by the Galaxy server
- Make the default keyring None
- Warn if the keyring is None but the Galaxy server provided signatures
- Error if the keyring is None but the user supplied signatures
- Error if the keyring is not None but is invalid
* changelog
* add ansible-galaxy user documentation for new options
Co-authored-by: Matt Martz <matt@sivel.net>
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
Co-authored-by: Martin Krizek <martin.krizek@gmail.com>
Co-authored-by: Sandra McCann <samccann@redhat.com>
Co-authored-by: Andy Mott <amott@redhat.com>
Co-authored-by: John R Barker <john@johnrbarker.com>
* ansible-galaxy - fix the --ignore-certs flag for the implicit galaxy server
* changelog
* Add a test without the server config
* Fix respecting --ignore-certs for individual --server URLs also
* Update changelogs/fragments/76735-ansible-galaxy-fix-ignore-certs.yaml
* download role from api response location
* include changelog fragment
* add unit test for role download url
Co-authored-by: Sam Doran <sdoran@redhat.com>
* ansible-galaxy: Clone git collections using shallow clones
This ensures the collection obtained via git url is a result of a
shallow git clone (git clone --depth=1). The git history of the
collection is not used by ansible, and as such, cloning the entire
history of the repo is unnecessary.
Signed-off-by: Tomas Babej <tomas@tbabej.com>
* ansible-galaxy: Only perform shallow clones on non-versioned git urls
In general, the version can be anything we can checkout (branch, tag,
but also a commit hash). In particular for commit hashes we cannot
perform a shallow clone.
Err on the safe side and only perform shallow clones for non-versioned
git urls (cloning HEAD).
* galaxy-install: Make shallow cloning compatible with older Python versions
Signed-off-by: Tomas Babej <tomas@tbabej.com>
* Pass args as a tuple
Add tests for the git command created from different repo and version combinations
* changelog
Co-authored-by: Tomas Babej <tomas@tbabej.com>
* Enable ansible-galaxy to specify client id override with Keycloak Token
* Specify ability to provide override of client_id
* Test client_id can be configured for individual servers
* Add issue link to changelog
* Document client_id as a config option and add an example
Co-authored-by: s-hertel <19572925+s-hertel@users.noreply.github.com>
* Get available collection versions with page_size=100 for v2 and limit=100 for v3
* Update unit tests for larger page sizes
* Add a generic retry decorator in module_utils/api.py that accepts an Iterable of delays and a callable to determine if an exception inheriting from Exception should be retried
* Use the new decorator to handle Galaxy API rate limiting
* Add unit tests for new retry decorator
* Preserve the decorated function's metadata with functools.wraps
Co-authored-by: Matt Martz <matt@sivel.net>
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
* --offline allows in-place verify for installed collections with manifests
* manifest hash, collection name, version, and path are now always displayed
* test updates
* Manage the in-memory cache in _call_galaxy but let the caller set the file cache after getting paginated results
* Add a test for caching successful and not caching unsuccessful paginated results
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
PR #72591
This change:
* Adds an artifacts manager that abstracts away extracting the
metadata from artifacts, downloading and caching them in a
temporary location.
* Adds `resolvelib` to direct ansible-core dependencies[0].
* Implements a `resolvelib`-based dependency resolver for
`collection` subcommands that replaces the legacy
in-house code.
This is a dependency resolution library that pip 20.3+ uses
by default. It's now integrated for use for the collection
dependency resolution in ansible-galaxy CLI.
* Refactors of the `ansible-galaxy collection` CLI.
In particular, it:
- reimplements most of the `download`, `install`, `list` and
`verify` subcommands from scratch;
- reuses helper bits previously moved out into external modules;
- replaces the old in-house resolver with a more clear
implementation based on the resolvelib library[0][1][2].
* Adds a multi Galaxy API proxy layer that abstracts accessing the
version and dependencies via API or local artifacts manager.
* Makes `GalaxyAPI` instances sortable.
* Adds string representation methods to `GalaxyAPI`.
* Adds dev representation to `GalaxyAPI`.
* Removes unnecessary integration and unit tests.
* Aligns the tests with the new expectations.
* Adds more tests, integration ones in particular.
[0]: https://pypi.org/p/resolvelib
[1]: https://github.com/sarugaku/resolvelib
[2]: https://pradyunsg.me/blog/2020/03/27/pip-resolver-testing
Co-Authored-By: Jordan Borean <jborean93@gmail.com>
Co-Authored-By: Matt Clay <matt@mystile.com>
Co-Authored-By: Sam Doran <sdoran@redhat.com>
Co-Authored-By: Sloane Hertel <shertel@redhat.com>
Co-Authored-By: Sviatoslav Sydorenko <webknjaz@redhat.com>
Signed-Off-By: Sviatoslav Sydorenko <webknjaz@redhat.com>
* Added caching mechanism for Galaxy API requests
* Add cache options and split up code
* Added unit tests
* Fix sanity test
* Use modified date and fix up caching for explicit servers
* Make sure credentials are not in cached server name
* Added test for getting updated cache version
* Changes from review
* GitHub is removing the underlying API used to implement the `login` command. Since the general consensus seems to be that relatively nobody currently uses this command (in favor of explicit token passing), support was simply removed for interactive login. If a future need arises, this command should be reimplemented via OAuth Device Auth Grants.
* login or role login commands now produce a fatal error with a descriptive message
* updated 2.10 and 2.11 porting guide entries
* remove dead code/config, update messages and porting guides
* Enable installing collections from git repositories
* Add tests for installing individual and multiple collections from git repositories
* Test to make sure recursive dependencies with different syntax are deduplicated
* Add documentation
* add a changelog
* Skip Python 2.6
* Only fail if no collections are located in a git repository
Add support for a 'type' key for collections in requirement.yml files.
Update the changelog and document the supported keys and allowed values for the type.
Add a note that the collection(s) in the repo must contain a galaxy.yml
* Add a warning about embedding credentials in SCM URLs
* Update with review suggestions
* suppress sanity compile failure for Python 2.6
Matt Clay