Change:
Removes with_* loop squashing and tests for 2.11
Test Plan:
CI, and grepped for with_items in package manager integration targets.
There might be some test cases in collections which need to stop testing
this behavior.
Signed-off-by: Rick Elrod <rick@elrod.me>
* config: singular ANSIBLE_COLLECTIONS_PATH
Every other *_PATH setting in ansible is singular, and the traditional
$PATH variable is also singular despite containing a list of
directories. Let's be consistent both internally and with POSIX
tradition.
* update all ANSIBLE_COLLECTIONS_PATHS env references to be singular
* deprecate plural ANSIBLE_COLLECTIONS_PATHS setting
* Only allow groups which were hardcoded in module_defaults.yml
only load action groups from the collection if module_defaults contains a potential group for the action
* Fix tests using modules that override those whitelisted in lib/ansible/config/module_defaults.yml
Third party modules should not be using group/ - use the action name instead
* add externalized module_defaults tests
add the missing group and collections
ci_complete
Co-authored-by: Matt Davis <mrd@redhat.com>
* changelog
ci_complete
* Fix import in tests
ci_complete
* Update with requested changes
ci_complete
* don't traceback since we don't validate the contents of module_defaults
ci_complete
Co-authored-by: Matt Davis <mrd@redhat.com>
* Allow to specify collection_name separately for deprecation.
* Use new functionality in Ansible.
* Use new functionality in tests.
* Update tagging/untagging functions.
* Update pylint deprecated sanity test.
* Update validate-modules. Missing are basic checks for version_added (validate semantic version format for collections).
* Improve version validation. Re-add version_added validation.
* Make sure collection names are added to return docs before schema validation.
* Extra checks to avoid crashes on bad data.
* Make C# module utils code work, and update/extend tests.
* Add changelog fragment.
* Stop extracting collection name from potentially tagged versions/dates.
* Simplify C# code.
* Update Windows modules docs.
* Forgot semicolons.
* various deprecation, display, warning, error fixes
* Update lib/ansible/utils/display.py
Co-authored-by: Felix Fontein <felix@fontein.de>
* Update lib/ansible/utils/display.py
Co-authored-by: Felix Fontein <felix@fontein.de>
* Update lib/ansible/utils/display.py
Co-authored-by: Felix Fontein <felix@fontein.de>
* cleanup, test fixes
* add collection name to deprecated() calls
* clean up redirect entries from uncommitted tests
* fix dep warning/error header text to match previous
Co-authored-by: Felix Fontein <felix@fontein.de>
* Track collection for version_added.
Validate *all* version numbers in validate-modules.
For tagged version numbers (i.e. version_added), consider source collection to chose validation.
* Make tagging/untagging functions more flexible.
* Tag all versions in doc fragments.
* Tag all deprecation versions issued by code.
* Make Display.deprecated() understand tagged versions.
* Extend validation to enforce tagged version numbers.
* Tag versions in tests.
* Lint and fix test.
* Mention collection name in collection loader's deprecation/removal messages.
* Fix error IDs.
* Handle tagged dates in Display.deprecated().
* Also require that removed_at_date and deprecated_aliases.date are tagged.
* Also automatically tag/untag removed_at_date; fix sanity module removal version check.
* Improve error message when invalid version number is used (like '2.14' in collections).
Change:
Rather than connection plugins directly accessing play_context and pulling
the password from there, have them pull it from the config system, and
have TaskExecutor store it there for now.
Internally, it still routes through play_context for now, but this is
the first step away from that.
Test Plan:
- Local test with `ansible -c ssh`
- grep -R play_context.pass lib/ansible/plugins/connection/
- CI
Signed-off-by: Rick Elrod <rick@elrod.me>
* `meta/` directory in collections
* runtime metadata for redirection/deprecation/removal of plugin loads
* a compatibility layer to keep existing content working on ansible-base + collections
* a Python import redirection layer to keep collections-hosted (and otherwise moved) content importable by things that don't know better
* supported Ansible version validation on collection loads
* Fix `ansible -K` become_pass regression
Change:
- This fixes a breaking change introduced in
2165f9ac40
Test Plan:
- Local VM for now, with plans to add an integration test for -K going
forward.
Tickets:
Refs #69244
* fix delegation vars usage and reporting
- just pass delegated host vars + task vars to plugins
and avoid poluting with original host vars
- updated tests
* Try to load network action plugin from the same collection as the module
* Alter tests to match
Just make sure the action plugin is as qualified as the module it is paired with
* Use is_sequence, and Mapping throughout, add support for tuples. Fixes#65722
* Address tests
* Remove unused import
* Add changelog
* Add docstring for clarity
* Argh, linting fix
* Not chasing this rabbit
* wrap_var doesn't return a ref to the original item
* no ref tests
* Remove unused import
* Fix network action plugin load in collection
Fixes https://github.com/ansible/ansible/issues/65071
* Load network action plugin that matches the module
prefix name from list of collections.
* Update changelog
* Fix unit test
* Use correct var, move cleanup for async
* Add changelog and tests. Fixes#65393. Fixes#65277.
* Kill off all long running async tasks from listen_ports_facts
* Update task to work with older jinja2
* Fix cli context check for network_cli connection
Fixes#64575
* Check cli context for network_cli connection
at the start of new task run only.
* Pass task_uuid around to identify start of new task run
* Handle for local connection
* Properly load module_prefix for collections
Now that we are using collections for tasks, we need to properly split
the name and load the prefix properly.
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
* Add unit tests for task_executor
This commit adds missing unit tests for action handler in test_task_executor.
Signed-off-by: Daniel Mellado <dmellado@redhat.com>
* Remove UnsafeProxy
Move the work from UnsafeProxy to wrap_var and add support for bytes.
Where wrap_var is not needed, use AnsibleUnsafeBytes/AnsibleUnsafeText
directly.
Fixes#59606
* item is not always text
* Address issues from reviews
* ci_complete
* fix module defaults
- corrected precedence (specific module > group)
- made into reusable function
- use from gather_facts/service/package to match 'actual module used'
* Don't pollute include_variables. Fixes#51667. Fixes#54618.
* Rename include_variables to include_args, so we can make the distinction about what they are
* Track args and vars separately
* oops
* oops again
* linting fix
* Add test
* basic plugin loading working (with many hacks)
* task collections working
* play/block-level collection module/action working
* implement PEP302 loader
* implicit package support (no need for __init.py__ in collections)
* provides future options for secure loading of content that shouldn't execute inside controller (eg, actively ignore __init__.py on content/module paths)
* provide hook for synthetic collection setup (eg ansible.core pseudo-collection for specifying built-in plugins without legacy path, etc)
* synthetic package support
* ansible.core.plugins mapping works, others don't
* synthetic collections working for modules/actions
* fix direct-load legacy
* change base package name to ansible_collections
* note
* collection role loading
* expand paths from installed content root vars
* feature complete?
* rename ansible.core to ansible.builtin
* and various sanity fixes
* sanity tweaks
* unittest fixes
* less grabby error handler on has_plugin
* probably need to replace with a or harden callers
* fix win_ping test
* disable module test with explicit file extension; might be able to support in some scenarios, but can't see any other tests that verify that behavior...
* fix unicode conversion issues on py2
* attempt to keep things working-ish on py2.6
* python2.6 test fun round 2
* rename dirs/configs to "collections"
* add wrapper dir for content-adjacent
* fix pythoncheck to use localhost
* unicode tweaks, native/bytes string prefixing
* rename COLLECTION_PATHS to COLLECTIONS_PATHS
* switch to pathspec
* path handling cleanup
* change expensive `all` back to or chain
* unused import cleanup
* quotes tweak
* use wrapped iter/len in Jinja proxy
* var name expansion
* comment seemingly overcomplicated playbook_paths resolution
* drop unnecessary conditional nesting
* eliminate extraneous local
* zap superfluous validation function
* use slice for rolespec NS assembly
* misc naming/unicode fixes
* collection callback loader asks if valid FQ name instead of just '.'
* switch collection role resolution behavior to be internally `text` as much as possible
* misc fixmes
* to_native in exception constructor
* (slightly) detangle tuple accumulation mess in module_utils __init__ walker
* more misc fixmes
* tighten up action dispatch, add unqualified action test
* rename Collection mixin to CollectionSearch
* (attempt to) avoid potential confusion/conflict with builtin collections, etc
* stale fixmes
* tighten up pluginloader collections determination
* sanity test fixes
* ditch regex escape
* clarify comment
* update default collections paths config entry
* use PATH format instead of list
* skip integration tests on Python 2.6
ci_complete
* [WIP] become plugins
Move from hardcoded method to plugins for ease of use, expansion and overrides
- load into connection as it is going to be the main consumer
- play_context will also use to keep backwards compat API
- ensure shell is used to construct commands when needed
- migrate settings remove from base config in favor of plugin specific configs
- cleanup ansible-doc
- add become plugin docs
- remove deprecated sudo/su code and keywords
- adjust become options for cli
- set plugin options from context
- ensure config defs are avaialbe before instance
- refactored getting the shell plugin, fixed tests
- changed into regex as they were string matching, which does not work with random string generation
- explicitly set flags for play context tests
- moved plugin loading up front
- now loads for basedir also
- allow pyc/o for non m modules
- fixes to tests and some plugins
- migrate to play objects fro play_context
- simiplify gathering
- added utf8 headers
- moved option setting
- add fail msg to dzdo
- use tuple for multiple options on fail/missing
- fix relative plugin paths
- shift from play context to play
- all tasks already inherit this from play directly
- remove obsolete 'set play'
- correct environment handling
- add wrap_exe option to pfexec
- fix runas to noop
- fixed setting play context
- added password configs
- removed required false
- remove from doc building till they are ready
future development:
- deal with 'enable' and 'runas' which are not 'command wrappers' but 'state flags' and currently hardcoded in diff subsystems
* cleanup
remove callers to removed func
removed --sudo cli doc refs
remove runas become_exe
ensure keyerorr on plugin
also fix backwards compat, missing method is attributeerror, not ansible error
get remote_user consistently
ignore missing system_tmpdirs on plugin load
correct config precedence
add deprecation
fix networking imports
backwards compat for plugins using BECOME_METHODS
* Port become_plugins to context.CLIARGS
This is a work in progress:
* Stop passing options around everywhere as we can use context.CLIARGS
instead
* Refactor make_become_commands as asked for by alikins
* Typo in comment fix
* Stop loading values from the cli in more than one place
Both play and play_context were saving default values from the cli
arguments directly. This changes things so that the default values are
loaded into the play and then play_context takes them from there.
* Rename BECOME_PLUGIN_PATH to DEFAULT_BECOME_PLUGIN_PATH
As alikins said, all other plugin paths are named
DEFAULT_plugintype_PLUGIN_PATH. If we're going to rename these, that
should be done all at one time rather than piecemeal.
* One to throw away
This is a set of hacks to get setting FieldAttribute defaults to command
line args to work. It's not fully done yet.
After talking it over with sivel and jimi-c this should be done by
fixing FieldAttributeBase and _get_parent_attribute() calls to do the
right thing when there is a non-None default.
What we want to be able to do ideally is something like this:
class Base(FieldAttributeBase):
_check_mode = FieldAttribute([..] default=lambda: context.CLIARGS['check'])
class Play(Base):
# lambda so that we have a chance to parse the command line args
# before we get here. In the future we might be able to restructure
# this so that the cli parsing code runs before these classes are
# defined.
class Task(Base):
pass
And still have a playbook like this function:
---
- hosts:
tasks:
- command: whoami
check_mode: True
(The check_mode test that is added as a separate commit in this PR will
let you test variations on this case).
There's a few separate reasons that the code doesn't let us do this or
a non-ugly workaround for this as written right now. The fix that
jimi-c, sivel, and I talked about may let us do this or it may still
require a workaround (but less ugly) (having one class that has the
FieldAttributes with default values and one class that inherits from
that but just overrides the FieldAttributes which now have defaults)
* Revert "One to throw away"
This reverts commit 23aa883cbed11429ef1be2a2d0ed18f83a3b8064.
* Set FieldAttr defaults directly from CLIARGS
* Remove dead code
* Move timeout directly to PlayContext, it's never needed on Play
* just for backwards compat, add a static version of BECOME_METHODS to constants
* Make the become attr on the connection public, since it's used outside of the connection
* Logic fix
* Nuke connection testing if it supports specific become methods
* Remove unused vars
* Address rebase issues
* Fix path encoding issue
* Remove unused import
* Various cleanups
* Restore network_cli check in _low_level_execute_command
* type improvements for cliargs_deferred_get and swap shallowcopy to default to False
* minor cleanups
* Allow the su plugin to work, since it doesn't define a prompt the same way
* Fix up ksu become plugin
* Only set prompt if build_become_command was called
* Add helper to assist connection plugins in knowing they need to wait for a prompt
* Fix tests and code expectations
* Doc updates
* Various additional minor cleanups
* Make doas functional
* Don't change connection signature, load become plugin from TaskExecutor
* Remove unused imports
* Add comment about setting the become plugin on the playcontext
* Fix up tests for recent changes
* Support 'Password:' natively for the doas plugin
* Make default prompts raw
* wording cleanups. ci_complete
* Remove unrelated changes
* Address spelling mistake
* Restore removed test, and udpate to use new functionality
* Add changelog fragment
* Don't hard fail in set_attributes_from_cli on missing CLI keys
* Remove unrelated change to loader
* Remove internal deprecated FieldAttributes now
* Emit deprecation warnings now
* Revert "avoid x2 setting of set_fact when 'cacheable' (#50564)"
This reverts commit 207848f354.
* clarify clear_facts with set_fact cacheable
revert previous 'fix' as it will break playbooks by changing precedence
opted to leave current behaviour but document it on both plugins to mitigate confusion
fixes#50556
also fix grammer, add comment, remove unused e
* Connections can queue messages to be returned from ansible-connection
* Provide fallback for invalid display level
* Strip display from plugins
* Route messages through helper method to try to avoid improper appends
* Bare minimum rip out cliconf
* nxapi changeover
* Update documentation, move options
* Memoize device_info
* Gratuitous rename to underscore use of local api implementation
Fixup eos module_utils like nxos
* Streamline version and image scans
* Expose get_capabilities through module_utils
* Add load_config to module_utils
* Support rpcs using both args and kwargs
* Add get_config for nxos
* Add get_diff
* module context, pulled from nxapi
We could probably do this correctly later
* Fix eos issues
* Limit connection._sub_plugin to only one plugin
* Add info about loop based on jinja2 loop var
* ansible_loop
* Update test count
* Add extended loop_control that defines whether ansible_loop should be added
* Extended needs to be defaulted
* Revert "Update test count"
This reverts commit f1e93ee469825f4cdcd90fb28667d29aa088275c.
* Add docs about loop_control.extended
* Add revindex and revindex0
* Document ansible_loop in special vars
* Add changelog fragment
* Add tests, change items to allitems so that dot notation works, fix logic error with previtem
Add some extra information to an async failure message to reflect the
actually timeout value of the failure.
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
* Try to intuit proper plugins to send to ansible-connection
* Move sub-plugins to init so that vars will be populated in executor
* Fix connection unit tests
* win async: use async_dir for the async results file directory
* tried to unify POSIX and PowerShell async implementations of async_dir
* fix sanity issue
Extends `module_defaults` by adding a prefix to defaults `group/` which denotes a builtin list of modules. Initial groups are: `group/aws`, `group/azure`, and `group/gcp`
* Don't use copy.deepcopy in high workload areas, use deepishcopy. ci_complete
* Add tests
* Add changelog fragment
* rename to naive_deepcopy and add extra docs
* Rename to module_response_deepcopy and move to vars/clean
* restore task arg splatting
* reverts #41804
* supersedes #41295
* fixes#42192
* after lots of discussion amongst the core team, we decided to preserve this feature, clarify the runtime warnings/docs, and prioritize a path toward fixing the underlying behavior that causes this feature to be insecure (un-namespaced facts).
* update faq text
note that warning is disabled when inject_facts_as_vars is
* wordsmithing FAQ entry
* Change how data is sent to the persistent connection socket.
We can't rely on readline(), so send the size of the data first. We can
then read that many bytes from the stream on the recieving end.
* Set pty to noncanonical mode before sending
* Now that we send data length, we don't need a sentinel anymore
* Copy socket changes to persistent, too
* Use os.write instead of fdopen()ing and using that.
* Follow pickle with sha1sum of pickle
* Swap order of vars and init being passed to ansible-connection
* fix minor issues with debug and item labels
- no more `item=None`, we always have a label now
- debug should only show expected information, either msg= or the var in var=
- also fixed method name, deprecated misleading _get_item
* I seem to have forgotten the back half of tests
* Set http timeout from persistent_command_timeout
* Tweak URL generation and provide URL on error
* Push var_options to connection process
* Don't wait forever if coming from persistent
* Don't send the entire contents of variables to ansible-connection
* WIP Pull persistent connection parameters via get_option
* Fix pep8
* Add use_persistent_connection setting to paramiko_ssh plugin
* Add vars section to persistent_command_timeout setting and prevail provider values over config manager
* Use persistent_command_timeout on network_cli instead of timeout
* Fix unit tests
If we don't call loader to get network_cli, then _load_name is never
set and we get KeyError.
* Pull persistent_command_timeout via config manager for ios connection local
* Pull persistent_command_timeout via config manager on connection local
* task_executor.py: Raise 'conditional exception' in case of 'include_*'
Fixes#33632
Signed-off-by: Patrick Ringl <patrick_@freenet.de>
* Re-organize tests, add static include test with undefined var
* Remove unnecessary conditional
this change further enables the ansible-vcr project to collect fixtures per host in
situations where the host context is totally lost due to delegation.
* Keep some original pre-delegate values in connection options
include_vars and set_fact are already updating hostvars in strategy
no need to 're add again' with lower priority the same data.
fixes#37535, mostly by avoiding reprocessing and 'cleaning'
* Check that connection error msg are not unsafe
* Connection error messages are unsafe: wrap them
For example, in case of error, docker connection plugin returns exception
message containing Go template. These messages weren't tagged as unsafe
and were consequently rendered:
The conditional check 'result is failed' failed. The error was:
{
'msg': u'Docker version check ([\'/usr/bin/docker\', \'version\', \'--format\', "\'{{.Server.Version}}\'"]) failed: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.35/version: dial unix /var/run/docker.sock: connect: permission denied\n',
'failed': True
}:
template error while templating string: unexpected '.'.
String: Docker version check (['/usr/bin/docker', 'version', '--format', "'{{.Server.Version}}'"]) failed: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.35/version: dial unix /var/run/docker.sock: connect: permission denied
* fix loop_control templating
- properly template ALL properites for loop_control
- loop_control inherits from base, it should not, but needs validate/attribute functionality
fixes#24719
* Fix debug logs failing with persistent connection
Fixes#33047
* As debug logs are written on stdout, it interrupts
the communication between ansible-connection(background)
process and main process. To avoid this add a string similar
to exactly identify the response string.
* Remove unwanted code in ansible-connection
* Fix review comments
* Fix spurious log emitted on ansible-connection stdout issue
* ansible-connection which runs as a background process sends a
json string (contains response received from remote device)
to foreground ansible-playbook process over stdout.
* If in case debug flag is enabled the connection_loader api
invoked from ansible-connection `ssh = connection_loader.get('ssh', class_only=True)`
results in emitting debug logs on stdout. This spurious log
interfere with the actual response and results in failure while
reading json string in ansible-playbook process
* To avoid this save stdout of ansible-connection and redirect it string
buffer to accumulate all the logs emitted by core API's
* Add these logs in `result['messages']` which is send a json string after reinstating saved stdout
* Remove unwanted code in ansible-connection
* Fix review comment
* allow shells to have per host options, remote_tmp
added language to shell
removed module lang setting from general as plugins have it now
use get to avoid bad powershell plugin
more resilient tmp discovery, fall back to `pwd`
add shell to docs
fixed options for when frags are only options
added shell set ops in t_e and fixed option frags
normalize tmp dir usag4e
- pass tmpdir/tmp/temp options as env var to commands, making it default for tempfile
- adjusted ansiballz tmpdir
- default local tempfile usage to the configured local tmp
- set env temp in action
add options to powershell
shift temporary to internal envvar/params
ensure tempdir is set if we pass var
ensure basic and url use expected tempdir
ensure localhost uses local tmp
give /var/tmp priority, less perms issues
more consistent tempfile mgmt for ansiballz
made async_dir configurable
better action handling, allow for finally rm tmp
fixed tmp issue and no more tempdir in ballz
hostvarize world readable and admin users
always set shell tempdir
added comment to discourage use of exception/flow control
* Mostly revert expand_user as it's not quite working.
This was an additional feature anyhow.
Kept the use of pwd as a fallback but moved it to a second ssh
connection. This is not optimal but getting that to work in a single
ssh connection was part of the problem holding this up.
(cherry picked from commit 395b714120522f15e4c90a346f5e8e8d79213aca)
* fixed script and other action plugins
ensure tmpdir deletion
allow for connections that don't support new options (legacy, 3rd party)
fixed tests
* Fix persistent command timeout handling
We were using play context timeout on ansible-connect to set
the persistent command timeout handler. Thus, we were ignoring
the persistent_command_timeout setting.
Moreover, even by changing that on ansible-connection, were again
overriding it on cliconf send_command, since in a same process we can
just set a single alarm and cliconf send_command alarm setup is executed
after ansible-connection alarm setup.
* Remove alarm setting on cliconf send_command
The alarm is set regardless before it is executed by ansible-connection.
Setting an alarm again, overrides/disables the previous ones as a single
process can just have a single alarm set.
* Move the setting of persistent command timeout to network_cli
We do also use ansible-connection for connection local, so if a
user provides a timeout via provider that would be ignored if we
set the value on ansible-connection.
Moving that logic to network_cli plugin constructor makes both
connections to work.
* Remove debug statements
* Set the persistent command timeout on task_executor
We can't set the timeout on ansible-connection nor network_cli,
otherwise tasks using provider timeout won't work.
* Add integration test for missing attempts in until with failed/changed_when
* Extend tests to validate additional known behaviors of do-until loops
* Inject attempts into result earlier
The short-circuiting of include_role in TaskExecutor was using the _task object
in the result dictionary. This causes the playbook class to be serialized, which
is something we no longer do due to speed/size problems with nested objects.
Simply removing this should fix a lot of memory and speed problems with include_role.
Sloane Hertel