Commit Graph

1211 Commits (a9bd8b6cff7504df0b3d270d04cbd0009e3ae4b1)

Author SHA1 Message Date
Thomas Sjögren 57359d0174
use passlib default if `password_hash` option isn't set (#75043)
* add changelog fragment
* password_hash|length == 120
* mention the new default bcrypt format in the porting guide
3 years ago
Matt Clay 4ea8d9a782
ansible-test - split controller/target testing (#75605) 3 years ago
Pilou 3d7f2a1366
Fix an exception when passlib library is used with a wrapped algo (#75527)
* Test a passlib wrapped algo with a password lookup

* Fix error when passlib is used with a wrapped algo

The exception was:

     An unhandled exception occurred while running the lookup plugin 'password'.
     Error was a <class 'TypeError'>, original message: issubclass() arg 1 must be a class

and can be reproduced using the following command:

    ansible localhost -i localhost, -mdebug -amsg="{{ lookup('password', '/dev/null encrypt=ldap_sha512_crypt') }}"

The concerned algo are: bsd_nthash, django_argon2, django_bcrypt, ldap_bcrypt,
ldap_bsdi_crypt, ldap_des_crypt, ldap_hex_md5, ldap_hex_sha1, ldap_md5_crypt,
ldap_pbkdf2_sha1, ldap_pbkdf2_sha256, ldap_pbkdf2_sha512, ldap_sha1_crypt,
ldap_sha256_crypt, ldap_sha512_crypt, roundup_plaintext
3 years ago
Matt Clay 99a79e1969
ansible-test - Move code from _data to _util. (#75495)
* Update paths to match relocated files.
* Update ansible-test symlink paths.
* Update path classification.
* Update MANIFEST.in
* Update sanity test context paths.
* Update sanity ignores.
* Update shebang sanity test.
* Update configure-remoting-ps1 sanity test.
* Update BOTMETA.yml to reflect new paths.
* Update paths in collection loader comments.
* Update Makefile for ansible-test.
* Update docs.
* Add changelog fragment.
3 years ago
Matt Clay ce6d8a143c
Code cleanup to prepare for pylint update. (#75475)
* user - Remove unused code.
* Replace deprecated abstractproperty decorator.
* Fix __all__ to be a tuple.
* Use a generator in subelements lookup.
* Use from import in basic.py
* Add changelog fragment.
* Fix selinux unit test.
3 years ago
Felix Fontein fae299708f
plugin attributes: make validate-modules strict again, improve version_added support (#74602)
* Add basic schema for attributes.

* Support attributes for version_added_collection adding mechanism.

* Add changelog fragment.

* Support new attribute properties from #74331.

* Require additional properties strictly.
3 years ago
Sloane Hertel 3b861abce1
add action_groups support to collections (#74039)
* Canonicalize module_defaults actions and action_groups pre-fork and cache them on the play

* Call get_action_args_with_defaults with the resolved FQCN plugin and don't pass the redirect list

* Add validation for action_group metadata and a toggle to disable the warnings

* Handle groups recursively referring to each other

* Remove special-casing for non-fqcn actions in module_defaults groups

* Error for actions and groups in module_defaults that can't be resolved

* Error for fully templated module_defaults

* Add integration tests for action_groups

* Changelog
3 years ago
Junegunn Choi 8e755707b9
Add 'hash_behaviour' option to include_vars module (#72944) 3 years ago
s-hamann e2658801f6
Add seed parameter to password lookup (#69775) 3 years ago
Abhijeet Kasurde 1bd7dcf339
encrypt: add new paramter 'ident' (#74595)
Add a new parameter `ident` for specifying version of BCrypt
algorithm. This parameter is only valid for `blowfish` hash type.
3 years ago
Matt Martz 8d1cf7f266
Vendor `distutils.version` (#74644)
* Vendor distutils.version

* Fix import order. ci_complete

* remove distutils warning filter

* Don't remove warnings filter from importer

* ci_complete

* Add pylint config for preventing distutils.version

* Add changelog fragment
3 years ago
Matt Clay baa371e7b5 Add comments to keep collection loader in sync. 3 years ago
Matt Martz bf7d4ce260
Abstract away libyaml details into yaml utils (#72607)
* Add yaml utils file and use it

* Linting fix and missing import

* Abstract a few more details

* Parser imports

* Don't use CSafeDumper in AnsibleDumper

* Move and use convert_yaml_objects_to_native when libyaml is present

* yaml_load and yaml_dump, instead of safe_*

* re-use HAS_LIBYAML from utils.yaml

* add changelog fragment

* Address recent changes

* Use representer instead of recursive type converter

* Restore needed import

* move yaml utils to module_utils

* Properly guard imports

* Update from_yaml(_all)? to handle text wrappers with CSafeLoader

* Use yaml utils for legacy_collection_loader

* Add HAS_YAML, and ignore pylint issue

* oops

* GPL->BSD
3 years ago
Brian Coca 6418f368e3
Fix pb subs collapse (#74301)
* fix subdir parsing for plays

  fixes #74283

  Co-authored-by: Nikolaos Kakouros <nkak@kth.se>
3 years ago
Brian Coca 4494ef3a9d
defend against bad or missing crypt (#74304)
* defend against bad or missing crypt

  fixes #74279
3 years ago
Brian Coca 07939b04f3
[2.12] 'attributes' (#73707)
* wip 'attributes'

* added version added tests

* syntzx

* not bile

* correztlys merges

* moved desc to frag

* simpler as dict

* unused

* clog

* Update lib/ansible/utils/plugin_docs.py

Co-authored-by: Jacob Floyd <cognifloyd@gmail.com>

* unnoted

* added action plugins

* longer list

* add sttri schema

* huh?

* itsdict

* dictit

* yolo

* gnore for now

* moar attribs

* allow extras

* positive

* added loop, documented 'imports'

* support is now none/partial/full

* import_playbook is outside host loop

Co-authored-by: Jacob Floyd <cognifloyd@gmail.com>
3 years ago
Ikko Ashimine b70a42f693
fix typo in _collection_finder.py (#74218) 3 years ago
Brian Coca 935528e22e
finish migrating ssh plugin to config system (#73708)
* finish migrating ssh plugin to config system

  fixes #72739
  fixes #57220

* fix connection detection in reset
* correct options for connection meta reset

Co-authored-by: David Shrewsbury <Shrews@users.noreply.github.com>
3 years ago
Matt Martz ba3f84883f
Reduce complexity of Templar._lookup slightly (#73277)
* A little more complexity reduction
* restore logic
* Readability
* Add ran check back
* Add clog
3 years ago
Alexei Znamensky 920b68f5f2
Fixed/improved regular expresssion for collection names (#73577)
* added changelog fragment
* added a couple of tests to coll name validation
3 years ago
Sviatoslav Sydorenko f327e65d11
Use iskeyword and str.isidentifier for "is FQCN" (#73279)
* Use valid FQCN in test_verbosity_arguments

* Use iskeyword and str.isidentifier for "is FQCN"
3 years ago
Brian Coca 1202dd000f
Allow restricting config values to enumerated list (#73162)
* Allow restricting config values to enumerated list
* dont document internal entries
* also ignore private defs for ansible-config
* remove invalid value from tests
* added porting entry
3 years ago
Brian Coca d22804c4fb
saner path dir management (#72648)
* saner path dir management

   fixes #72628

   ensure we always store paths w/o a_c

Co-authored-by: Sloane Hertel <19572925+s-hertel@users.noreply.github.com>
4 years ago
Rick Elrod 6894ae7d1d
Rename to ansible-core (#72594)
Change:
- Initial set of changes for renaming to ansible-core
- Includes changelog fragment changes from base -> core
- Does NOT include docs changes
- Modifies detection stuff in setup.py to support ansible<2.9 and ansible-base

Test Plan:
- ci_complete
4 years ago
Brian Coca 8eab113cb1
show keyword documentation in ansible-doc (#72476)
* show keyword documentation in ansible-doc

Co-authored-by: Felix Fontein <felix@fontein.de>
4 years ago
Felix Fontein da60525610
Fix missing ansible.builtin FQCNs in hardcoded action names (#71824)
* Make sure hard-coded action names also check for FQCN.
* Use _add_internal_fqcn() to avoid hardcoded lists and typoes.
4 years ago
Brian Coca 4b673484f0
rethink wording (#70028)
* rethink wording
* removed unrequired requirement
* fix tests
* fixed versions
Co-authored-by: Sloane Hertel <shertel@redhat.com>
4 years ago
Brian Coca e05c62547b
run playbook from collections (#67435)
* fixes for collection playbooks

 - add fqcn invocation, also handles extensions
 - bring import_playbook into new normal
 - avoid adding collection playbooks to adjacency
 - deal with extensions or lack of em
 - added tests
 - fix bugs with multiple playbooks supplied
 - unicode/bytes/native fixes
 - fix modulenotfound for < py3.7
 - tests
4 years ago
Felix Fontein 1107aace1b
Add deprecation collection name to plugin options (#71735)
* Add changelog fragment.

* Use correct field that's expected by lib/ansible/cli/__init__.py..

* Add basic unit tests.
4 years ago
Matt Davis e813b0151c
fix coverage output from synthetic packages (#71727)
* fix coverage output from synthetic packages

* synthetic packages (eg, implicit collection packages without `__init__.py`) were always created at runtime with empty string source, which was compiled to a code object and exec'd during the package load. When run with code coverage, it created a bogus coverage entry (since the `__synthetic__`-suffixed `__file__` entry didn't exist on disk).
* modified collection loader `get_code` to preserve the distinction between `None` (eg synthetic package) and empty string (eg empty `__init__.py`) values from `get_source`, and to return `None` when the source is `None`. This allows the package loader to skip `exec`ing things that truly have no source file on disk, thus not creating bogus coverage entries, while preserving behavior and coverage reporting for empty package inits that actually exist.

* add unit test
4 years ago
Martin Krizek a3b954e5c9
Force template module to use non-native Jinja2 (#68560)
Fixes #46169
4 years ago
Matt Martz fdf5dd02b3
Updates for password hashing (#71120)
* Validate salt when using crypt. Respect salt_size in password lookup. Repair salt for bcrypt. Fixes #71107. Fixes #53750. Fixes #36129.

* Handle algorithms we don't know about, and make sure to return the salt

* Account for old passlib

* Add tests for salt constraints

* Add changelog fragment

* Add test for #36129
4 years ago
Matt Martz 5821128995
Allow callbacks from forks (#70501)
* POC for supporting callback events that come from the worker

* linting fixes. ci_complete

* fix up units. ci_complete

* Try moving the sentinel put higher. ci_complete

* safeguards. ci_complete

* Move queue killing to terminate

* LINTING. ci_complete

* Subclass Queue, to add helper send_callback method

* Just use _final_q instead of adding another queue and thread

* Revert a few changes

* Add helper for inserting a TaskResult into the _final_q

* Add changelog fragment

* Address rebase issue

* ci_complete

* Add test to assert async poll callback from fork

* Don't use full path

* ci_complete

* Use _results_lock as a context manager

* Add new generic lock decorator, and use it with send_callback
4 years ago
Abhijeet Kasurde 504ef607f3
Misc typo fixes (#71089) 4 years ago
Brian Coca 1223ce656a
reset logging to INFO (#70878)
- due to CVE-2019-14846
 - also added comments and test to avoid 'oportunistic' reversion
4 years ago
Matt Davis c616e54a6e
refactor Python module_utils locator (#70610)
* refactor Python module_utils locator

* no longer recursive
* embed special-case module code internally
* share common code between collections/not cases
* fixes #70134
* properly support subpackage redirection
* adds support for FQCN redirect targets used by migration (expands to FQ Python name)
* add tests

* add changelog
4 years ago
Matt Davis b9e38e8b55
misc collection metadata fixes (#70403)
* misc collection metadata fixes

* parse collection meta with libyaml if available
* require only Mapping for validation
* add explanatory text for _meta_yml_to_dict

* ignore custom pylint rule

* this code shouldn't import a bunch of stuff from ansible, since it's run under the import sanity test
4 years ago
Felix Fontein 689cfd1983
Top-level deprecation of plugin did not get collection_name added when deprecating by version (#70344)
* Top-level deprecation of plugin did not get collection_name added when deprecating by version.

* Add changelog fragment.
4 years ago
Pavel Březina 45e0f74702
display: use stdout for column width (#70199)
stdout may differ from stdin so it should be used to determine the column
width, especially since it is the target file descriptor.
4 years ago
Rick Elrod 2531d54880
Throw AnsibleError instead of OSError, py3.9 crypt (#70246)
Change:
- On Python 3.9, `crypt.crypt` will throw instead of returning `None`
  when the algorithm isn't supported. So we catch that and handle it
  the same way we handled the algorithm not being supported on 3.8: by
  throwing AnsibleError.

Test Plan:
- CI for <=3.8.
- Local for 3.9b3:
  ansible -m debug -a "msg=\"{{ 'changeme' | password_hash('bcrypt') }}\"" localhost

  Before:
  localhost | FAILED! => {
      "msg": "Unexpected failure during module execution.",
      "stdout": ""
  }

  After:
  localhost | FAILED! => {
      "msg": "crypt.crypt does not support 'bcrypt' algorithm"
  }

Tickets:
- Fixes #69930

Signed-off-by: Rick Elrod <rick@elrod.me>
4 years ago
Matt Martz 1fedb95e4b
Use libc wcwidth to calculate print width in display (#66214)
* Use libc wcwidth to calculate print width in display. Fixes #63105

* Remove errantly added blank lines

* Fixes

* Move setlocale, adjust tests to work around py2 oddity with characters following null

* Don't change cli stub

* emojis

* Remove to_text call

* Special accounting for deletions

* Add initialization function, expand tests, ensure fallback to len

* get_text_width requires text, ensure banner deals with it

* Handle setlocale errors

* Move variable decrement

* Remove unused import
4 years ago
Felix Fontein 8d93ba9120
Plugin/module docs: parse return values, add collection names in them (version_added_collection), and format them nicely in ansible-doc (#69796)
* Tag return value docs if they are a dict (and not str/None).

* Try to parse return docs as YAML.

* Properly dump return values in ansible-doc.

* Adjust plugin formatter.

* Add changelog fragment.

* Don't add 'default' for return values.

* Fix plugin_formatter.

* Only try to parse return docs if they are still a string.

* Add tests.

* Warn if RETURN cannot be parsed.

* Adjust tests. Also test for warning.

* if -> elif (otherwise EXAMPLE will be parsed too).

* Always parse return documentation, and fail if it is invalid YAML.

* Polishing.

* Mostly re-enable ansible-doc tests.

Listing from the local collection seems to be somewhat broken. I assume this
is why the test was disabled.

* Lint and make tests work with Python 2.

* Keep FQCNs in plugins (not modules), i.e. restore previous state.
4 years ago
Sloane Hertel 51f6d129cb
support hard coded module_defaults.yml groups for collections (#69919)
* Only allow groups which were hardcoded in module_defaults.yml

only load action groups from the collection if module_defaults contains a potential group for the action

* Fix tests using modules that override those whitelisted in lib/ansible/config/module_defaults.yml

Third party modules should not be using group/ - use the action name instead

* add externalized module_defaults tests

add the missing group and collections

ci_complete

Co-authored-by: Matt Davis <mrd@redhat.com>

* changelog

ci_complete

* Fix import in tests

ci_complete

* Update with requested changes

ci_complete

* don't traceback since we don't validate the contents of module_defaults

ci_complete

Co-authored-by: Matt Davis <mrd@redhat.com>
4 years ago
Felix Fontein a862ff2d43
Deprecation revisited (#69926)
* Allow to specify collection_name separately for deprecation.

* Use new functionality in Ansible.

* Use new functionality in tests.

* Update tagging/untagging functions.

* Update pylint deprecated sanity test.

* Update validate-modules. Missing are basic checks for version_added (validate semantic version format for collections).

* Improve version validation. Re-add version_added validation.

* Make sure collection names are added to return docs before schema validation.

* Extra checks to avoid crashes on bad data.

* Make C# module utils code work, and update/extend tests.

* Add changelog fragment.

* Stop extracting collection name from potentially tagged versions/dates.

* Simplify C# code.

* Update Windows modules docs.

* Forgot semicolons.
4 years ago
Brian Coca a64418c2b3
avoid constant checking controlpersist (#69910)
* avoid constant checking controlpersist
4 years ago
Matt Davis 984216f52e
various deprecation, display, warning, error fixes for collections redirection (#69822)
* various deprecation, display, warning, error fixes

* Update lib/ansible/utils/display.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* Update lib/ansible/utils/display.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* Update lib/ansible/utils/display.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* cleanup, test fixes

* add collection name to deprecated() calls

* clean up redirect entries from uncommitted tests

* fix dep warning/error header text to match previous

Co-authored-by: Felix Fontein <felix@fontein.de>
4 years ago
Matt Davis d79b23910a
fix sys.modules cleanup and blacklist behavior (#69898)
* fix sys.modules cleanup and blacklist behavior

* fix map-as-generator py2/py3 issue
* clear path_importer_cache between runs

* sanity fix

* don't be stupid with moving target generators
4 years ago
Sloane Hertel e40889e711
Add support to install collections from git repositories (#69154)
* Enable installing collections from git repositories

* Add tests for installing individual and multiple collections from git repositories

* Test to make sure recursive dependencies with different syntax are deduplicated

* Add documentation

* add a changelog

* Skip Python 2.6

* Only fail if no collections are located in a git repository

Add support for a 'type' key for collections in requirement.yml files.
Update the changelog and document the supported keys and allowed values for the type.

Add a note that the collection(s) in the repo must contain a galaxy.yml

* Add a warning about embedding credentials in SCM URLs

* Update with review suggestions

* suppress sanity compile failure for Python 2.6
4 years ago
Felix Fontein 40f21dfd3c
Version source tagging (automatic and manual) for version_added and deprecation versions (#69680)
* Track collection for version_added.
Validate *all* version numbers in validate-modules.
For tagged version numbers (i.e. version_added), consider source collection to chose validation.

* Make tagging/untagging functions more flexible.

* Tag all versions in doc fragments.

* Tag all deprecation versions issued by code.

* Make Display.deprecated() understand tagged versions.

* Extend validation to enforce tagged version numbers.

* Tag versions in tests.

* Lint and fix test.

* Mention collection name in collection loader's deprecation/removal messages.

* Fix error IDs.

* Handle tagged dates in Display.deprecated().

* Also require that removed_at_date and deprecated_aliases.date are tagged.

* Also automatically tag/untag removed_at_date; fix sanity module removal version check.

* Improve error message when invalid version number is used (like '2.14' in collections).
4 years ago
Felix Fontein ea04e0048d
Allow to deprecate options and aliases by date (#68177)
* Allow to deprecate options and aliases by date instead of only by version.

* Update display.deprecate().

* Adjust behavior to conform to tested behavior, extend tests, and improve C# style.

* Parse date and fail on invalid date.

This is mainly to make sure that people start using invalid dates, and we eventually have a mess to clean up.

* C# code: improve validation and update/extend tests.

* Make sure that deprecate() is not called with both date and version.

* Forgot to remove no longer necessary formatting.

* Adjust order of warnings in C# code.

* Adjust unrelated test.

* Fix grammar (and make that test pass).

* Don't parse date, and adjust message to be same as in #67684.

* Sanity tests: disable date in past test.

* Validate-modules: validate ISO 8601 date format.

* Validate-modules: switch schema declaration for deprecated_aliases to improve error messages for invalid dates.

* Use DateTime instead of string for date deprecation.

* Validate that date in deprecated_aliases is actually a DateTime.

* Fix tests.

* Fix rebasing error.

* Adjust error codes for pylint, and add removed_at_date and deprecated_aliases.date checks to validate-modules.

* Make deprecation date in the past error codes optional.

* Make sure not both version and date are specified for AnsibleModule.deprecate() calls.

* Stop using Python 3.7+ API.

* Make sure errors are actually reported. Re-add 'ansible-' prefix.

* Avoid crashing when 'name' isn't there.

* Linting.

* Update lib/ansible/module_utils/csharp/Ansible.Basic.cs

Co-authored-by: Jordan Borean <jborean93@gmail.com>

* Adjust test to latest change.

* Prefer date over version if both end up in Display.deprecated().

Co-authored-by: Jordan Borean <jborean93@gmail.com>
4 years ago