Commit Graph

187 Commits (c9245439dc46914af2832e73d9dd2cfa79e2c4eb)

Author SHA1 Message Date
Brian Coca 4aac0f5f18
allow configuring sftp/scp executables (#36648)
* allow configuring sftp/scp executables

fixes #36616


also removed dupe test
7 years ago
Matthew Stone 5b1a8ee1e8 Adding slx_command module and supporting module_utils. (#38235)
* Adding slx_command module and supporting module_utils.

This commit adds the slx_command module and tests as well as the
required slxos module_utils.

* Update copyright in header

* Adding missing module init

* Cleaning up shebangs/licensing.

* Incorporating feedback

Removing reference to `waitfor` alias in `slxos_command` module.
Adding `Extreme Networks` to `short_description` of `slxos_command` module.

* Adding cliconf tests

* Fixing 3.X tests

* Adding docstrings to test methods for slxos cliconf tests

* Adding slxos terminal tests

* Adding slxos module_utils tests

* Adding Extreme Networks team members to BOTMETA.yml
7 years ago
AlanCoding 153c9bd539 Reduce recursion within group methods
This offers an optimization that allows loading larger
inventories of various structure by improving the
scaling laws involved for adding hosts and groups.

The primary speed benefit is the elimination of duplicate
recusion from traversing converging paths.
7 years ago
Izuke e02c72d856 Fixes #32146 Fixes reconcile inventory to correctly remove hosts from ungrouped 7 years ago
Nathaniel Case 594840c1d6
Close & remove paramiko connection where appropriate (#37528)
* Close & remove paramiko connection where appropriate

* Update unit test
7 years ago
mikedlr 52f2057472 aws ssm parameter lookup - change to return Nones for missing variables (#36456)
* aws ssm parameter lookup - change to return Nones for missing variables

* aws ssm parameter lookup - fix error case message to dump response

* aws ssm parameter lookup - fix integration test cases
7 years ago
Brian Coca a79378fccb remove extra fields from debug output
fixes #35493

updated tests
7 years ago
Toshio Kuratomi 06f73ad578
Normalize usage of temp and tmp on tmp (#36221)
* Normalize usage of temp and tmp on tmp
* Rename system_tmps system_tmpdirs
* Add ANSIBLE_REMOTE_TMP spelling of environment variables
7 years ago
Brian Coca 8743b20c8e skip aws inventory test if no boto 7 years ago
mikedlr d31ded47fb Aws ssm multiple fixes (#35569)
* aws ssm parameter lookup test case - fails demonstrating no exception when parameter missing

* aws ssm parameter lookup - fail in case parameter doesn't exist

* aws ssm parameter lookup test case - failing case for nice return from path lookup

* aws ssm parameter lookup - convert incoming taglist to a key-value dictionary

* aws ssm parameter lookup - pep8 / style clean up

* aws_ssm lookup plugin rewrite for more standard interface

* aws_ssm module and lookup - introduce integration test and fix:

* aws_ssm module and lookup - error case integraton test and many PEP8 and other cleanups

* aws ssm parameter lookup - Various fixes in response to review + recursive fix & test

* aws ssm parameter lookup - more in response to review - shertel/abadger

* aws ssm parameter lookup unit test - move to mocker according to abadger

* aws ssm parameter lookup - integrate with new documentation fragment

* aws ssm parameter lookup - accept either aws_profile or boto_profile

* aws ssm parameter lookup - eliminate lookup document fragment until env vars are fixed later
7 years ago
Jason Vanderhoof 7c8e365dff Conjur Lookup Plugin (#34280)
* Imported lookup plugin from Role

* Plugin cleanup, including:
* Use existing Python YAML parsing
* Remove environment variables as connection options
* Added initial debugging information

* Reworked the lookup plugin using the Python Request library.  As it's available through Ansible, it makes communication with Conjur much more straight forward.

* Removed un-used libraries

* Fixed linting issues

* Standardized output on `format` and insure it works for 2.6, 2.7, and 3.x.

* Use quote_plus from the six library for improved python 2/3 behavior.

* Refactored identity & configuration to prefer user's file. This also includes a refactor to remove an un-needed dictionary merge method.

* Removed `requests` in favor of `ansible.module_utils.urls`.

* Refactored netrc loading to warn if host is not present.

* Tests and a refactor to support easier testing.

* Added reference to website

* Fixed two linting errors

* Fixed an extra line found by linting

* Updated file write to use binary to insure config files are written correctly

* Resolved linting issues

* Refactored config & identity loading to take advantage of plugin options

* Cleanup a bunch of small items caught by linting

* Removed extra line caught by linting

* Swapped in pytest and added some tests with mocked network responses

* Pushing to see if this approach works better...

* Refactored be open_url mocking based on feedback

* Fixed a couple linting issues & refactored mocking into each method to attempt to resolve a failing test

* Use a generic MagicMock for python 2.6

* Fixes doc typo

require -> required

* Use `type: path` in identity_file and config_file

Also removes `expanduser` calls below (which will now be called automatically on
paths.)

* Defines maintainers for conjur_variable plugin

* BOTMETA.yml:
** defines $team_cyberark_conjur as maintainers of Conjur Variable plugin
** adds myself and @jvanderhoof to that team

* Adds URLs to relevant documentation for Conjur Variable lookup plugin

* Clarifies "the server," "the machine" -> "controlling host"

The machine identity used is that of the Ansible controlling host, not any
server being provisioned or instructed. This documentation change aims to make
that relationship clear.

* Adds response code to exception message on authentication failure

* Enhances exception messages to specify the controlling host

These error messages are less likely to confuse a user as to which machine is
associated with the files, identities, and configurations being described.

* Adds ANSIBLE_METADATA for Conjur variable lookup plugin
7 years ago
Toshio Kuratomi 8c7bd8beb4 Set the tempdir on the shell mock object so that it is json serializable 7 years ago
Sloane Hertel b94198f9de [cloud] New inventory plugin: aws_ec2 (#34416)
aws_ec2 is an inventory plugin intended to replace uses of the `contrib/inventory/ec2.py`
inventory script. 

For advanced grouping, please use the `constructed` inventory plugin.
7 years ago
Matt Martz 29c1d5cb5d Ensure that the become password is written on py3 in the ssh connection plugin. Fixes #34727 7 years ago
Brian Coca bbd6b8bb42 Temporary (#31677)
* allow shells to have per host options, remote_tmp

added language to shell
removed module lang setting from general as  plugins have it now
use get to avoid bad powershell plugin
more resilient tmp discovery, fall back to `pwd`
add shell to docs
fixed options for when frags are only options
added shell set ops in t_e and fixed option frags
normalize tmp dir usag4e

- pass tmpdir/tmp/temp options as env var to commands, making it default for tempfile
- adjusted ansiballz tmpdir
- default local tempfile usage to the configured local tmp
- set env temp in action

add options to powershell
shift temporary to internal envvar/params
ensure tempdir is set if we pass var
ensure basic and url use expected tempdir
ensure localhost uses local tmp
give /var/tmp priority, less perms issues
more consistent tempfile mgmt for ansiballz
made async_dir configurable
better action handling, allow for finally rm tmp
fixed tmp issue and no more tempdir in ballz
hostvarize world readable and admin users
always set shell tempdir
added comment to discourage use of exception/flow control

* Mostly revert expand_user as it's not quite working.

This was an additional feature anyhow.

Kept the use of pwd as a fallback but moved it to a second ssh
connection.  This is not optimal but getting that to work in a single
ssh connection was part of the problem holding this up.

(cherry picked from commit 395b714120522f15e4c90a346f5e8e8d79213aca)

* fixed script and other action plugins

ensure tmpdir deletion
allow for connections that don't support new options (legacy, 3rd party)
fixed tests
7 years ago
James Cammarata b107e397cb
Cache tasks as they are queued instead of en masse (#34752)
* Cache tasks as they are queued instead of en masse

This also moves the task caching from the PlayIterator to the
StrategyBase class, where it makes more sense (and makes it easier
to not have to change the strategy class methods leading to an API
change).

Fixes #31673

* Cleaning up unit tests due to 502ca780
7 years ago
Matt Martz d1846425db
Provide a way to explicitly invoke the debugger (#34006)
* Provide a way to explicitly invoke the debugger with in the debug strategy

* Merge the debugger strategy into StrategyBase

* Fix some logic, pin to a single result

* Make redo also continue

* Make sure that if the debug closure doesn't need to process the result, that we still return it

* Fix failing tests for the strategy

* Clean up messages from debugger and exit code to match bin/ansible

* Move the FieldAttribute higher, to apply at different levels

* make debugger a string, expand logic

* Better host state rollbacks

* More explicit debugger prompt

* ENABLE_TASK_DEBUGGER should be boolean, and better docs

* No bare except, add pprint, alias h, vars to task_vars

* _validate_debugger can ignore non-string, that can be caught later

* Address issue if there were no previous tasks/state, and use the correct key

* Update docs for changes to the debugger

* Guard against a stat going negative through use of decrement

* Add a few notes about using the debugger on the free strategy

* Add changelog entry for task debugger

* Add a few versionadded indicators and a note about vars -> task_vars
7 years ago
Matt Martz aece0818c4
Don't cache duplicate plugin names (#34420)
* Don't cache duplicate plugin names. Fixes #33484

* Add tests for duplicate plugin filenames, to showcase what happens
7 years ago
Pierre-Louis Bonicoli 737920cd89 YAML inventory unit test: add tests
Relates: #33878
7 years ago
Pierre-Louis Bonicoli 35a4fe8546 YAML inventory unit test: add tests 7 years ago
Pierre-Louis Bonicoli 8dff0b091e Fix YAML inventory unit test 7 years ago
John R Barker cfb15b0272 Revert "YAML inventory unit test: fix test inventory format (#33828)" (#33869)
This reverts commit dfb2f346d8.

https://github.com/ansible/ansible/pull/33828
7 years ago
Pilou dfb2f346d8 YAML inventory unit test: fix test inventory format (#33828)
* Fix YAML inventory unit test

* YAML inventory unit test: add checks

* YAML: add hosts without any group to ungrouped
7 years ago
Toshio Kuratomi 155f36bbd8
Rekey on member (#33836)
* Change cast_list_to_dict to more generic rekey_on_member

cast_list_to_dict was taking an arbitrary data format in and returning
an arbitrary data format out.  Rework this to be a more generic function
which creates a dict of dicts based on a member of the dict.

Remove cast_dict_to_list since rekey_on_member handles the use cases we
know about and cast_dict_to_list suffers from the same problems as
cast_list_to_dict.  If this is still needed we could think about filters
we could add to do this in a short jinja2 pipeline.

* Fix bare excepts (bare excepts even catch sys.exit())
7 years ago
Adrian Likins a57d6a4206 Use pop in cb clean results (#33779)
* Fix _clean_result for debug callback with 1 char var names

The check in _clean_results was removing any keys that happened
to be one of the chars in ('invocation') instead of the string
'invocation'. This was meant to be a tuple but there was no comma
so the for iterated the string instead of the tuple.

Introduced in 9dba580204

Update unit test to catch this.

Fixes #33723

* Use .pop() to remove invocation from results dict

In base callback _clean_results, simplify the way the
'invocation' item is removed.

Add some more unit tests.
7 years ago
Ganesh Nalawade 95044d69c7
Fix unit test failure for netconf connection plugin (#33547) 7 years ago
Toshio Kuratomi e499bccbaa
Port some tests away from nose as examples (#33437)
We don't need to use both nose and pytest.  Once we get rid of all uses
of nose we can remove the extra dependency
7 years ago
Nathaniel Case a1da3dc997
Auth to become (#33319)
* terminal.on_(de)?authorize -> terminal.on_(un)?become

* Update network_cli

* Readd on_(de)?authorize as wrappers

* Update tests
7 years ago
Ken Celenza 748107d369 Type manipulation (#28446)
* add type manipulation

* update for py3

* add tests for list to dict

* Add dict to list tests

* Update tests to search for regex in response

* pep8 clean up

* update Exception for py3

* update test to be py2/3 compat

* update for py26 compat

* potential fix for py3 and py26

* potential fix for py3 and py26 take2

* add new line to kick off shippable

* remove cache file created

* fix filter name

* add space for shipable
7 years ago
James Cammarata d8ae4dfbf2 Adding aliases for field attributes and renaming async attribute (#33141)
* Adding aliases for field attributes and renaming async attribute

As of Python 3.7, the use of async raises an error, whereas before the use
of the reserved word was ignored. This adds an alias field for field attrs
so that both async and async_val (interally) work. This allows us to be
backwards-compatible with 3rd party plugins that may still reference Task.async,
but for the core engine to work on Py3.7+.

* Remove files fixed for 'async' usage from the python 3.7 skip list
7 years ago
jctanner 218987eac1
ANSIBLE_SSH_USETTY configuration option (#33148)
* Allow the user to circumvent adding -tt on ssh commands to help aid in
debugging ssh related problems.
* Move config to the plugin
* Set version_added
* Change yaml section to "connection"
* Fix ssh unit tests
7 years ago
Ganesh Nalawade 0ddf092ae3
Add new filter to parse xml output for network use cases (#31562)
* Add new filter to parse xml output for network use cases

Fixes #31026
*  Add parse_xml filter
*  Add documentation for parse_xml filter

* Edited for clarity.

* Fix review comment and add unit tests

* Fix unit test CI failure

* Fix CI issues

* Fix unit test failures

* Fix review comments

* More copy edits.
7 years ago
Brian Coca 23b1dbacaf
Config continued (#31024)
* included inventory and callback in new config

allow inventory to be configurable
updated connection options settings
also updated winrm to work with new configs
removed now obsolete set_host_overrides
added notes for future bcoca, current one is just punting, it's future's problem
updated docs per feedback
added remove group/host methods to inv data
moved fact cache from data to constructed
cleaner/better options
fix when vars are added
extended ignore list to config dicts
updated paramiko connection docs
removed options from base that paramiko already handles
left the look option as it is used by other plugin types
resolve delegation
updated cache doc options
fixed test_script
better fragment merge for options
fixed proxy command
restore ini for proxy
normalized options
moved pipelining to class
updates for host_key_checking
restructured mixins

* fix typo
7 years ago
Frederic Lepied 35f79370e1 mock ncclient import in test_connection.py (#32786) 7 years ago
Nathaniel Case 9c0275a879
Connection plugins network_cli and netconf (#32521)
* implements jsonrpc message passing for ansible-connection

* implements more generic mechanism for persistent connections
* starts persistent connection in task_executor if enabled and supported
* supports using network_cli as top level connection plugin
* enhances logging for persistent connection to stdout

* Update action plugins

* Fix Python3 RPC

* Fix Junos bytes<-->str issues

* supports using netconf as top level connection plugin

* Error message when running netconf on an unsupported platform
* Update tests

* Fix `authorize: yes` for `connection: local`

* Handle potentially JSON data in terminal

* Add clarifying detail if possible on ConnectionError
7 years ago
Brian Coca db749de5b8 namespace facts
updated action plugins to use new guranteed facts
updated tests to new data clean
added cases for ansible_local and some docstrings
7 years ago
Chris Meyers cf938e9992 tests for InventoryModule error conditions (#31381)
* tests for InventoryModule error conditions

* modified unicode in tests to ahear to Ansible best practices

* flake8 fixes
7 years ago
Brian Coca 01b6c7c9c6 better cleanup on task results display (#27175)
* better cleanup on task results display

callbacks get 'clean' copy of result objects
moved cleanup into result object itself
removed now redundant callback cleanup
moved no_log tests

* moved import as per feedback
7 years ago
Hervé Beraud db70eeb913 Feature/test inventory (#30707)
* [test] move inventory test to right path
* [feat] add unit test for yaml inventory plugin
7 years ago
Matt Clay dbf9634b1b Enable more pylint rules and fix reported issues. (#30543)
* Enable pylint invalid-encoded-data test.
* Enable pylint anomalous-unicode-escape-in-string test.
7 years ago
Toshio Kuratomi 75249e311e Fix set theory filters for python3
The set theory filters need to use lists rather than generators on python3.

Also add unit tests for most of the mathstuff filters

Fixes #26494
7 years ago
Nicolas Porcel ba420c47cd synchronize: Use private_key and ssh_user with delegate_to (Fixes #16215) (#21686) 7 years ago
Brian Coca f921369445 Ansible Config part2 (#27448)
* Ansible Config part2

- made dump_me nicer, added note this is not prod
- moved internal key removal function to vars
- carry tracebacks in errors we can now show tracebacks for plugins on vvv
- show inventory plugin tracebacks on vvv
- minor fixes to cg groups plugin
- draft config from plugin docs
- made search path warning 'saner' (top level dirs only)
- correctly display config entries and others
- removed unneeded code
- commented out some conn plugin specific from base.yml
- also deprecated sudo/su
- updated ssh conn docs
- shared get option method for connection plugins
- note about needing eval for defaults
- tailored yaml ext
- updated strategy entry
- for connection pliugins, options load on plugin load
- allow for long types in definitions
- better display in ansible-doc
- cleaned up/updated source docs and base.yml
- added many descriptions
- deprecated include toggles as include is
- draft backwards compat get_config
- fixes to ansible-config, added --only-changed
- some code reoorg
- small license headers
- show default in doc type
- pushed module utils details to 5vs
- work w/o config file
- PEPE ATE!
- moved loader to it's own file
- fixed rhn_register test
- fixed boto requirement in make tests
- I ate Pepe
- fixed dynamic eval of defaults
- better doc code

skip ipaddr filter tests when missing netaddr
removed devnull string from config
better becoem resolution

* killed extra space with extreeme prejudice

cause its an affront against all that is holy that 2 spaces touch each other!

shippable timing out on some images, but merging as it passes most
7 years ago
3onyc 554496c404 [passwordstore] Use builtin _random_password function instead of pwgen (#25843)
* [password] _random_password -> random_password and moved to util/encrypt.py
* [passwordstore] Use built-in random_password instead of pwgen utility
* [passwordstore] Add integration tests
7 years ago
Toshio Kuratomi 3edac559d3 the smart transport is broken by ssh retry code
1fe67f9 introduced retries to the ssh connection put file and fetch
file.  Unfortunately, that change broke the smart transport because it
started raising exceptions instead of returning from _run().  This
breakage is documented in #23711.

An attempt to fix it was made at #23717 but the first attempt was
objected to as needing to touch too much code.  The second attmept was
objected to as smart was forced to encapsulate retries (thus retrying
a sftp "rety" times before trying scp "retry" times and then finally
moving onto piped).  This third attempt has retries encapsulate smart.
So each sub-transport is tried once and if all three fail, another retry
attempt is made which tries each of the three again.

Fixes #23711
Fixes #23717
7 years ago
Toshio Kuratomi 802c9efaa3 Disable abstract-class-instantiated for smoketests
These tests are actually checking that the classes will error out if
they are still abstracted and instantiated
7 years ago
Gabor Lekeny 7eab802669 SSH fails with '"parsed": false' error message
Fixes: #15436
7 years ago
Ken Celenza 31b6ac896d Kc update ip filter (#26566)
* add first, last and next usable

* add usable ip filters

* add size usable, range usable and wildcard

* add ip prefix and netmask filter

* add network formatting and check if ip in subnet

* clean up order, add comments

* fix pep8

* update format by index

* clean up and updates from jmcgill298
7 years ago
Ganesh Nalawade 3bd8dbb53d Add support for netconf network discovery (#25435)
* Add support for netconf network discovery

* Fix unit test failure
7 years ago
Matt Davis 36ad934156 re-enable non-pipelined mode for Powershell (#25012)
* fixes #23986
* fixes 3rd-party Windows connection plugins that don't support pipelining (eg awsrun)
7 years ago