When using hostvars to get extra connection-specific vars for connection
plugins, use this raw lookup to avoid prematurely templating all of the
hostvar data (triggering unnecessary lookups).
Fixes#17024
* Add oVirt utility module
This patch add oVirt utility module, which contains helper functions,
for oVirt modules and also shared documentation fragment for oVirt.
* Adjust to Python 2.4
* Fixups
* Add support for poll interval and fixes
When using the Cli transport, if the session hung on a command and the
socket timed out, the config session would be left behind. This change
will allow the shell to try to get control back and remove the config
session, assuming the channel is still open.
fixesansible/ansible-modules-core#4945
* changed missing file error to warning for lookups
* changed plugins that expected exception
warning will still be displayed, they now work with None value
* Improve unit testing of 'password' lookup
The tests showed some UnicodeErrors for the
cases where the 'chars' param include unicode,
causing the 'getattr(string, c, c)' to fail.
So the candidate char generation code try/excepts
UnicodeErrors there now.
Some refactoring of the password.py module to make
it easier to test, and some new tests that cover more
of the password and salt generation.
* More refactoring and fixes.
* manual merge of text enc fixes from pr17475
* moving methods to module scope
* more refactoring
* A few more text encoding fixes/merges
* remove now unused code
* Add test cases and data for _gen_candidate_chars
* more test coverage for password lookup
* wip
* More text encoding fixes and test coverage
* cleanups
* reenable text_type assert
* Remove unneeded conditional in _random_password
* Add docstring for _gen_candidate_chars
* remove redundant to_text and list comphenesion
* Move set of 'chars' default in _random_password
on py2, C.DEFAULT_PASSWORD_CHARS is a regular str
type, so the assert here fails. Move setting the
default into the method and to_text(DEFAULT_PASSWORD_CHARS)
if it's needed.
* combine _random_password and _gen_password
* s/_create_password_file/_create_password_file_dir
* native strings for exception msgs
* move password to_text to _read_password_file
* move to_bytes(content) to _write_password_file
* add more test assertions about genned pw's
* Some cleanups to alikins and abadger's password lookup refactoring:
* Make DEFAULT_PASSWORD_CHARS into a text string in constants.py
- Move this into the nonconfigurable section of constants.
* Make utils.encrypt.do_encrypt() return a text string because all the
hashes in passlib should be returning ascii-only strings and they are
text strings in python3.
* Make the split up of functions more sane:
- Don't split such that conditionals have to occur in two separate functions.
- Don't go overboard: Good to split file system manipulation from parsing
but we don't need to do every file manipulation in a separate
function.
- Don't split so that creation of the password store happens in two
parts.
- Don't split in such a way that no decisions are made in run.
* Organize functions by when it gets called from run().
* Run all potential characters through the gen_candidate_chars function
because it does both normalization and validation.
* docstrings for functions
* Change when we store salt slightly. Store it whenever it was already
present in the file as well as when encrypt is requested. This will
head of potential idempotence bugs where a user has two playbook tasks
using the same password and in one they need it encrypted but in the
other they need it plaintext.
* Reorganize tests to follow the order of the functions so it's easier
to figure out if/where a function has been tested.
* Add tests for the functions that read and write the password file.
* Add tests of run() when the password has already been created.
* Test coverage currently at 100%
The Conditional instance will cause a stack trace if the provided conditional
does not map properly to the response. This fixes that issue so that the
Conditional instance will now raise a FailedConditionalError with the
conditional that caused the failure.
Modules *_command modules (and any other modules that create an instance
of Conditional) should be updated to catch the FailedConditionalError
exception.
This addresses a problem when *_config or *_template network modules are
being used in roles. The module will error with the above message. This
fixes that problem
fixedansible/ansible-modules-core#4840
* By default, ansible_distribution is not set on DragonFly systems,
preventing some distribution-specific tests from being written
* This commit fixes the issue by returning the quite logical value
of "DragonFly" when appropriate
If 'fact_caching=jsonfile' was configured, but
'fact_caching_connection' was not configured, jsonfile
would fail and ansible-playbook would exit with a traceback.
Fixes#17566
* Pass the absolute path to dirname when assigning basedir
If no path is specified when calling the playbook, os.path.dirname(playbook_path) returns ''
This will cause failure when creating the retry file.
Fixes#17456
* Updated to use os.pathdirname(os.path.abspath())
* Make is_encrypted_file handle both files opened in text and binary mode
On python3, by default files are opened in text mode. Since we know
the encoding of vault files (and especially the header which is the
first set of bytes) we can decide whether the file is an encrypted
vault file in either case.
* Fix is_encrypted_file not resetting the file position
* Update is_encrypted_file to check that all the data in the file is ascii
* For is_encrypted_file(), add start_pos and count parameters
This allows callers to specify reading vaulttext from the middle of
a file if necessary.
* Combine VaultLib.encrypt() and VaultLib.encrypt_bytestring()
* Change vault's is_encrypted() to take either text or byte strings and to return False if any part of the data is non-ascii.
* Remove unnecessary use of six.b
* Vault Cipher: mark a few methods as private.
* VaultAES256._is_equal throws a TypeError if given non byte strings
* Make VaultAES256 methods that don't need self staticmethods and classmethods
* Mark VaultAES and is_encrypted as deprecated
* Get rid of VaultFile (unused and feature implemented in a different way)
* Normalize variable and parameter names on plaintext, ciphertext, vaulttext
* Normalize variable and parameter names on "b_" prefix when dealing with bytes
* Test changes:
* Remove redundant tests( both checking the same byte string)
* Fix use of format string without format operator
* Enable vault editor tests on python3
* Initialize the vault_cipher for VaultAES256 testing in setUp()
* Make assertTrue and assertFalse take the actual method calls for
better error messages.
* Test that non-ascii byte strings compare correctly.
* Test that unicode strings and ints raise TypeError
* Test-specific:
* Removed test_methods_exist(). We only have one VaultLib so the
implementation is the assurance that the methods exist. (Can use an abc for
this if it changes).
* Add tests for both byte string and text string input where the API takes either.
* Convert "assert" to unittest assert functions or add a custom message where
that will make failures easier to debug.
* Move instantiating the VaultLib into setUp().
Later in the stack, further code will check and inform the user that var names must start with a letter
or underscore, so this fix only allows us to get to that previously existing policy.
Fixes#16008
When an inventory file looks executable (with a #!) but
isn't, the error message could be confusing. Especially
if the inventory file was named something like 'inventory'
or 'hosts'. Add some context and quote the filename.
This is based on https://github.com/ansible/ansible/pull/15758
While doing evil things with action plugins, I hit a code path in which
the mkdir here was failing due to lack of parent dir. Changing this to
makedirs made everything happy. Now, I'd obviously like to understand
why the parent dir exists in some places and not others - but I could
not find anywhere that C.DEFAULT_LOCAL_TMP is ensured to be created.
* Add support for no-expiration to jsonfile cache
* Let memcached cache use fact_caching_timeout=0
If fact_cache=memcached and fact_caching_timeout=0
memcached would hit a NameError on _expire_keys
Change linux fact gathering to correctly gather ansible_processor_count
and ansible_processor_vcpus on systems without vendor_id/model_name in
/proc/cpuinfo (for ex, ppc64/POWER)
* Added aws_retry decorator function with unit tests
* Restructured the code to be used with a base class.
This base class CloudRetry can be reused by any other cloud provider.
This decorator should be used in situations, where you need to implement
a backoff algorithm and want to retry based on the status code from the
exception.
* updated documentation
* fixed tabs
* added botocore and boto3 to requirements.txt
* removed cloud.py from py24 tests, as it depends on boto3
* fix relative imports
* updated test to be 2.6 compat
* updated method name from retry to backoff
* readded lxd
* Updated default backoff from 2 seconds to 1.1s.
This will be about a total of 48 seconds in 10 tries. This is
configurable.
* Fixes to the controller text model
* Change command line args to text type
* Make display replace undecodable bytes with replacement chars. This
is only a problem on pyhton3 where surrogates can enter into the msg
but sys.stdout doesn't know how to handle them.
* Remove a deprecated playbook syntax in unicode.yml
* Fix up run_cmd to change its parameters to byte string at appropriate times.
* Add a new config option to cache the check for controlpersist on the
control machine.
Fixes#15844
* Remove the option and make the behavior the default
* Make the check for controlpersist cache its status per-ssh executable
Trying to preserve the meaning of the examples. Not all occurrences in
`docsite/rst/playbooks_lookups.rst` have been changed for instance to
allow the unchanged examples to be used for testing.
Related to: #17479
The statvfs(3) manpage on Linux states that `f_blocks` is the "size of fs in `f_frsize` units". The manpages on Solaris and AIX state something similar.
With ext4 on Linux, I suspect that `f_bsize` and `f_frsize` are always identical, masking this error. On Solaris, the sizes differ for each of ufs, vxfs and zfs causing the `size_available` and `size_total` facts to be set incorrectly on this OS.
The fileglob lookup plugin only returns files, not directories.
This is to be expected, as a mixed list would not be very useful in with_fileglob.
However the fileglob filter does return anything glob.glob() returns.
This change fixes this, so that fileglob returns files (as the name indicates).
PS We could also offer a glob filter for thos that would need it ?
This relates to comments in issue #17136 and fixes confusion in #17269.
In the 'comment' filter, if the 'prefix' parameter is set as empty,
don't add an empty line before the comment. To get the previous
behaviour (empty line before comment), set the prefix to '\n'.
which got lost in recent big 'performance improvements' merge by @jimi-c.
I had made a previous PR to fix this, then @bcoca had committed an
improved fix. Now it's lost again.
cf: d2b3b2c03e (lost here)
cf: 25e9b5788b (previous fix)
Earlier PR #14849
Earlier issue #14843
Please note that jimi-c broke this last time as well ... seeing a
pattern here.
The diff returned from eos when the transport was set to eapi was as
a dict but is expected to be a str. This change extracts the diff string
from the dict object and returns it. The behavior is now consistent
between cli and eapi transports.
We couldn't copy to_unicode, to_bytes, to_str into module_utils because
of licensing. So once created it we had two sets of functions that did
the same things but had different implementations. To remedy that, this
change removes the ansible.utils.unicode versions of those functions.
There is an issue when piping cli commands through json but the output
is specified as either text or the output is none and the transport is
cli. The results would not be loaded properly for conditional
evaluation. This is similar to #17422
The caching of commands in CommandRunner is providing no useful feature
and causing problems. This removes the code and simply returns the
requested command results.
Some old remnants of code from the refactor of netcli was left over as
reported in #17408. This commit removes the old code as it isn't need
and in fact wasnt doing anything
Exception thrown when using cli transport in eos but piping the command
through json
* eos now checks for `| json` and automatically changes the output type
* adds back import of Command object
tested on EOS 4.15.4F
* Clean up EOS, IOS, IOS-XR, Junos, NX-OS, and OpenSwitch
* Cleanup net* files
* Re-add NetworkModule import to network module_utils files
This will trick modules into importing code from module_utils code, thus
including it in the final Ansiballz zipfile.
* Give asa a look over, too
* dynamic role_include
* more fixes for dynamic include roles
* set play yfrom iterator when dynamic
* changes from jimi-c
* avoid modules that break ad hoc
TODO: should really be a config
* add authorize() method to handle authorization
* move terminal commands to after authorization completed
* add save_config() method to handling writing config to disk
* fix minor issues with get_config
* adds action plugin asa_config
* Fix paramiko's exec_command() to return bytes on python3
* Run test_connection for python3 now too
* Fix atomic_move for problem in shippable's testing
* Python-2.4 needs to use b()
I can't figure out any reason that we'd need to use long explicitly here
as python implicitly moves from a C long int to python Long
automatically under the covers. My best guess is that it was originally
used so that the facts module would work on python-2.2 where the user
had to convert a number from int to long manually but python-2.4 is our
current baseline.
long isn't present on Python3 so now is a good time to remove this
cruft. (We had a workaround for Python3; this commit also removes the
workaround.)
for `VariableManager._get_magic_variables()`.
This saves a lot of time re-iterating the nearly always constant global
list of groups and their members.
Generate once and cache, and invalidate cache in case `add_host:` or
`group_by:` are used.
* Port set_*_if_different functions to python3
* Add surrogate_or_strict and surrogate_or_replace error handlers for
to_text, to_bytes, to_native
* Set default error handler to surrogate_or_replace
* Make use of the new error handlers in the already ported code
* Move the unittests for module_utils._text as they aren't in basic.py
* Cleanup around SEQUENCETYPE. On python2.6+ SEQUENCETYPE includes
strings so make sure code omits those explicitly if necessary
* Allow arg_spec aliases to be other sequence types
This feature also cleans up and extends the meta subsystem:
* Allows for some meta actions (noop, clear_facts, clear_host_errors,
and end_play) to operate on a per-host basis, meaning they can work
with the free strategy as expected.
* Allows for conditionals on meta tasks.
* Fixes a bug where (for the linear strategy) metas were not treated
as a run_once task, meaning every host in inventory would run the
meta task.
Fixes#1476
* adds squashing to objects, which allows them to be squashed down
to a final "view" before post_validate to avoid expensive evaluations
of parent attributes
Introduces the `inherit` param for FieldAttributes, which is now used
in BaseMeta when constructing the getter property to enhance performance
by reducing the amount of work the getter generally has to do.
* Use six instead of urllib2, for python 3 compat
* Open the certificate file using binary mode
On python3, os.write requires 'bytes'. Also avoid
using a too broad exception, since the issue was hard
to spot due to it.
* Do not add the header User-agent if not set
Python3 module do raise a exception if a header is
not a string-like object, and the default value is None.
The authorize method was calling run_commands() instead of execute(). This
fixes that problem so that authorize() calls are made direclty on the shell
object now
* fix setting cookie after successful login
* raise NotImplementedError if run_commands is called in Rest
* return header msg key if status is not 2xx
* add action plugin ops_config
* New features for include_vars
include_vars.py now allows you to include an entire directory and its nested directories of variable files.
Added Features..
* Ignore by default *.md, *.py, and *.pyc
* Ignore any list of files.
* Only include files nested by depth (default=unlimited)
* Match only files matching (valid regex)
* Sort files alphabetically and load in that order.
* Sort directories alphabetically and load in that order.
```
- include_vars: 'vars/all.yml'
- name: include all.yml
include_vars:
file: 'vars/all.yml'
- name: include all yml files in vars/all and all nested directories
include_vars:
dir: 'vars/all'
- name: include all yml files in vars/all and all nested directories and save the output in test.
include_vars:
dir: 'vars/all'
name: test
- name: include all yml files in vars/services
include_vars:
dir: 'vars/services'
depth: 1
- name: include only bastion.yml files
include_vars:
dir: 'vars'
files_matching: 'bastion.yml'
- name: include only all yml files exception bastion.yml
include_vars:
dir: 'vars'
ignore_files: 'bastion.yml'
```
* Added whitelist for file extensisions (yaml, yml, json)
* Removed unit tests in favor of integration tests
Working on the test suite, I tried to replace a call to sudo to a
call to su, and found out that I can't change user to 'nobody'
without changing the option become_flags in ansible.cfg
As this would be dependent on the user and the task, it make more sense
to push the setting there.
* Fix to_native call in selinux_context and selinux_default_context to
use the error handler correctly.
* Port set_mode_if_different to work on python3
* Port atomic_move to work on python3
* Fix check_password_prompt variable which wasn't renamed properly
* univention: add common code for univention corporate server modules
* univention: try import only univention specific libraries
* Code Review with @2-B, slight API changes and refactoring.
* Added module documentation overview, describing the provided functions
* Moved module-global objects into getter functions, so that we don't
need to import possibly-unavailable univention modules at the module level.
* Renamed some exports for improved consistency:
- module_name() -> module_by_name()
- orig_ldap -> ldap_module()
- ldap -> uldap()
Note that this introduces slight API changes from the outside. Instead of
directly accessing module properties, you now have module functions with the
same name. Examples:
- ansible.module_utils.univention.position_base_dn()
- ansible.module_utils.univention.config_registry()
- ansible.module_utils.univention.base_dn()
- ansible.module_utils.univention.config()
* module_utils univention: fix library
* move module_utils from univention to univention_umc, because python import univention fails if library is called univention
* univention_umc: fix intention
* univention: change common code to BSD-2-clause
* attempt #11 to role_include
* fixes from jimi-c
* do not override load_data, move all to load
* removed debugging
* implemented tasks_from parameter, must break cache
* fixed issue with cache and tasks_from
* make resolution of from_tasks prioritize literal
* avoid role dependency dedupe when include_role
* fixed role deps and handlers are now loaded
* simplified code, enabled k=v parsing
used example from jimi-c
* load role defaults for task when include_role
* fixed issue with from_Tasks overriding all subdirs
* corrected priority order of main candidates
* made tasks_from a more generic interface to roles
* fix block inheritance and handler order
* allow vars: clause into included role
* pull vars already processed vs from raw data
* fix from jimi-c blocks i broke
* added back append for dynamic includes
* only allow for basename in from parameter
* fix for docs when no default
* fixed notes
* added include_role to changelog
* Add OpenBSD virtualization facts.
Patch written by @jasperla.
Tested by various people on:
- virtualbox
- vmware esx(i) + fusion
- kvm (smartos + plain linux + a random cloud provider)
This patch is already present in the OpenBSD port of ansible.
* Rework diff to get rid of extra returns.
Requested by @bcoca.
While here, use four-space indentations of all code blocks.
* Set facts even if no match is found.
Discussed with @bcoca.
* Find sysctl via get_bin_path().
Requested by @bcoca.
* Fail if we do not find a sysctl binary.
* Do not fail if a sysctl binary is not found.
Just set empty fact values instead.
Requested by @bcoca.
There was general consensus that displaying every plugin load on -vvv
was *way* too noisy. This commit reformats the log message to be less
verbose, and drops it down to debugging-only level.
tempfile.NamedTemporaryFile keeps a file handle causing os.rename() to fail with windows based vboxfs: [Errno 26] Text file busy.
Changed NamedTemporaryFile to mkstemp() and added a finally block to unlink the temp file in each and every case.
AnsibleError is not imported in that file, and since that's
a parsing time issue, better raise AnsibleParserError like the
rest of the file.
Issue signaled on irc by gordon`
* run_command needed a bit of tweaking to its string handling of
arguments.
* The run_command change fixes the last bit of lineinfile so we can
enable its tests
This adds a new property to the Command object that is used to hold
modified command strings that could be different from the command used
to create the object. This allows for seamless switch between text and
json enabled commands.
To override a generic class that is subclassed based on platform, the
subclass must define platform and distribution.
The load_platform_subclass() calls the get_platform() and
get_distribution() methods to detect the platform and the distribution.
On Alpine Linux, get_distribution() method returns None and it is not
possible to have different implementations based on detected platform.
groups['x']|map('extract', hostvars, 'somevar') would break if any host
didn't have 'somevar' set. With this change, it will return Undefined
instead. This change permits |map('extract', …)|map('default', 42) to
set a default value in such cases.
This adds a cli transport, netcfg, and netcli implementations for working
with devices running Nokia SROS. There is also an update to netcfg
to support the sros config file format.
- Fix octal formatting of file mode in module response on py3.
- Convert file path to unicode in copy action.
- Enable file and copy module tests for py3 now that they pass.
Make some python3 fixes to make the unittests pass:
* galaxy imports
* dictionary iteration in role requirements
* swap_stdout helper for unittests
* Normalize to text string in a facts.py function
Fixes for these are either rewriting to get rid of the need for the
functions or using six.moves to get equivalent functions for both
python2 and python3
This completes the refactor of the iosxr 2.2 shared module. It also
includes the iosxr_config action plugin to be implemented by the
iosxr_config module for 2.2
ran task_executor through python-modernize and then made changes to the
code pointed out by it:
* Most places where we looped through dict.keys() changed to
for key in dict:
Using keys() in python2 creates a list() of keys. For iterating, we
can iterate over the dict itself and we'll be handed back each key.
In python3, doing it this way does not create a new list and thus is
more memory efficient.
* In one place, use:
for key in list(dict.keys()):
because we're deleting elements from the dictionary inside of the
loop. So we really do need to iterate over a separate list of the
keys to avoid modifying the dictionary that we're iterating over.
(Fixes Python3 bug)
* In one place, change the order of an if-elif-else tree so that the
most frequent cases are evaluated first. (Optimization)
Make !vault-encrypted create a AnsibleVaultUnicode
yaml object that can be used as a regular string object.
This allows a playbook to include a encrypted vault
blob for the value of a yaml variable. A 'secret_password'
variable can have it's value encrypted instead of having
to vault encrypt an entire vars file.
Add __ENCRYPTED__ to the vault yaml types so
template.Template can treat it similar
to __UNSAFE__ flags.
vault.VaultLib api changes:
- Split VaultLib.encrypt to encrypt and encrypt_bytestring
- VaultLib.encrypt() previously accepted the plaintext data
as either a byte string or a unicode string.
Doing the right thing based on the input type would fail
on py3 if given a arg of type 'bytes'. To simplify the
API, vaultlib.encrypt() now assumes input plaintext is a
py2 unicode or py3 str. It will encode to utf-8 then call
the new encrypt_bytestring(). The new methods are less
ambiguous.
- moved VaultLib.is_encrypted logic to vault module scope
and split to is_encrypted() and is_encrypted_file().
Add a test/unit/mock/yaml_helper.py
It has some helpers for testing parsing/yaml
Integration tests added as roles test_vault and test_vault_embedded
The 'import xmltodict' was causing import
errors when generating documentation. Since
xmltodict is a required but not stdlib module,
throw AnsibleError if unable to import.
Remove unused combine_vars.
Replace a use of 'stdin_iterator == None' with
idiomatic 'stdin_iterat is None'
Misc pep8 cleanups.
Make the plugin loading info displayed by callback plugins
match.
In debug mode (ANSIBLE_DEBUG=1 env), log all requests for
plugins including already cached plugins and class_only
requests.
Traceback (most recent call last):
File "/tmp/ansible_tpehdgt7/ansible_module_setup.py", line 134, in <module>
main()
File "/tmp/ansible_tpehdgt7/ansible_module_setup.py", line 124, in main
supports_check_mode = True,
File "/tmp/ansible_tpehdgt7/ansible_modlib.zip/ansible/module_utils/basic.py", line 696, in __init__
File "/tmp/ansible_tpehdgt7/ansible_modlib.zip/ansible/module_utils/basic.py", line 1670, in _log_invocation
File "/tmp/ansible_tpehdgt7/ansible_modlib.zip/ansible/module_utils/basic.py", line 469, in heuristic_log_sanitize
TypeError: 'str' does not support the buffer interface
This is enough to get minimal copy module working on python3
We have t omodify dataloader's path_dwim_relative_stack and everything
that calls it to use text paths instead of byte string paths
* Give native strings to selinux library functions.
SELinux takes pathnames as native strings. That means we need to
convert to bytes on python2 and convert to text on python3.
Fixes#17155
* Read kitchen documentation, make module_utils params more like kitchen API
* Remove none nonstring strategy and add strict
* Raise TypeError on invalid nonstring strategy
* Document to_native()
* Make unittests for testing module_utils.text
* Rm py2.7+ code in docker connection plugin
The docker connection plugin was using subprocess.check_output
which only exists in python 2.7 and later. Connection plugins
need to support python2.6 so this replaces it with Popen/communicate()
* Handle docker ver errors in docker connection
Add unit tests for DockerConnection
Fixes#16971
This commit updates the nxos transport shared plugins for
2.2. This includes updates to both Cli and Nxapi. This commit
also includes the nxos_config action plugin
* Cleanup basic.py code now that six is available
We had some hacks in basic.py to allow us python2 and python3
compatibility. Those can now be offloaded to the six library that we're
bundling.
* Cleanup basic.py code now that six is available
We had some hacks in basic.py to allow us python2 and python3
compatibility. Those can now be offloaded to the six library that we're
bundling.
Peter Sprygada