* Warn on tests used as filters
* Update docs, add aliases for tests that fit more gramatically with test syntax
* Fix rst formatting
* Add successful filter, alias of success
* Remove renamed_deprecation, it was overkill
* Make directory alias for is_dir
* Update tests to use proper jinja test syntax
* Update additional documentation, living outside of YAML files, to reflect proper jinja test syntax
* Add conversion script, porting guide updates, and changelog updates
* Update newly added uses of tests as filters
* No underscore variable
* Convert recent tests as filter changes to win_stat
* Fix some changes related to rebasing a few integration tests
* Make tests_as_filters_warning explicitly accept the name of the test, instead of inferring the name
* Add test for tests_as_filters_warning
* Update tests as filters in newly added/modified tests
* Address recent changes to several integration tests
* Address recent changes in cs_vpc
* Remove uses of assert in production code
* Fix assertion
* Add code smell test for assertions, currently limited to lib/ansible
* Fix assertion
* Add docs for no-assert
* Remove new assert from enos
* Fix assert in module_utils.connection
* Ansible Config part2
- made dump_me nicer, added note this is not prod
- moved internal key removal function to vars
- carry tracebacks in errors we can now show tracebacks for plugins on vvv
- show inventory plugin tracebacks on vvv
- minor fixes to cg groups plugin
- draft config from plugin docs
- made search path warning 'saner' (top level dirs only)
- correctly display config entries and others
- removed unneeded code
- commented out some conn plugin specific from base.yml
- also deprecated sudo/su
- updated ssh conn docs
- shared get option method for connection plugins
- note about needing eval for defaults
- tailored yaml ext
- updated strategy entry
- for connection pliugins, options load on plugin load
- allow for long types in definitions
- better display in ansible-doc
- cleaned up/updated source docs and base.yml
- added many descriptions
- deprecated include toggles as include is
- draft backwards compat get_config
- fixes to ansible-config, added --only-changed
- some code reoorg
- small license headers
- show default in doc type
- pushed module utils details to 5vs
- work w/o config file
- PEPE ATE!
- moved loader to it's own file
- fixed rhn_register test
- fixed boto requirement in make tests
- I ate Pepe
- fixed dynamic eval of defaults
- better doc code
skip ipaddr filter tests when missing netaddr
removed devnull string from config
better becoem resolution
* killed extra space with extreeme prejudice
cause its an affront against all that is holy that 2 spaces touch each other!
shippable timing out on some images, but merging as it passes most
* template: fix KeyError: 'undefined variable: 0
For compatibility with the Context.get_all() implementation
in jinja 2.9, make AnsibleJ2Vars implement collections.Mapping.
Also, make AnsibleJ2Template.newcontext() handle dict type
for the 'vars' parameter.
See: d67f0fd4cc
Fixes: https://github.com/ansible/ansible/issues/20494
* add units/template/test_vars
* intg tests for jinja-2.9 issues like 20494
test cases here are based on
https://github.com/ansible/ansible/issues/20494#issue-202108318
* show original exception for yaml (and other) errors
In places where we need to catch a yaml error and raise
an AnsibleError, add the orig yaml exc to the AnsibleError
via the orig_exc arg.
When the AnsibleError is displayed it will now include the
AnsibleError (AnsibleParserError for example) and the type
and message from the original yaml exception.
This provides more detail to the error messages related to
yaml errors.
This also improves errors from dataloader (for example,
previously if a wrong password was used for a vault encrypted
yaml file, the error was very vague and suggested yaml errors,
but now the message includes the original exception from vault
indicating the password was incorrect or missing).
Add a text note to playbook helper asserts. For playbook
syntax/layout errors that aren't yaml errors, but errors
indicating invalid data structures for a playbook/task/role/block,
we now include some info about where the assert was and
why it was raised.
In places we raise an AnsibleParserError in an except
clause, pass the original exception to AnsibleParserError via
orig_exc arg.
Make assorted error messages a little more specific (like
the playbook helper load methods)
* Revert "Include the original YAML error in syntax error messages"
This reverts commit 781bb44b02.
CVE-2017-7481
Lookup returns wrap the result in unsafe, however when used through the
standard templar engine, this does not result in the jinja2 environment being
marked as unsafe as a whole. This means the lookup result looses the unsafe
protection and may become simple unicode strings, which can result in bad
things being re-templated.
This also adds a global lookup param and cfg options for lookups to allow
unsafe returns, so users can force the previous (insecure) behavior.
template/__init__.py imported unsafe_proxy from vars which caused
vars/__init__.py to load. vars/__init__.py needed template/__init__.py
which caused issues. Loading unsafe_proxy from another location fixes
that.
* keep unsafe .. unsafe
fixes#23734, which was broken in previous fix that allowed non string types to be templated
use new 'is_template' function vs bastardizing others
refactored clean_data to allow for arbitrary data structures to clean
fixed/removed some tests
* deal with complex data for is_template
* typos
This change to the template action plugin make template use the
platform's native newline_sequence for Jinja.
We also added the option `newline_sequence` to change the newline
sequence using by Jinja if you need to use another newline sequence than
the platform default.
This was previously discussed in
https://github.com/ansible/ansible/issues/16255#issuecomment-278289414
And also relates to issue #21128
* Update module_utils.six to latest
We've been held back on the version of six we could use on the module
side to 1.4.x because of python-2.4 compatibility. Now that our minimum
is Python-2.6, we can update to the latest version of six in
module_utils and get rid of the second copy in lib/ansible/compat.
* Don't check for var._obj in template._clean_data
AnsibleUnsafe or other unsafe vars used to have a
'_obj' slot but no longer do. This was causing attribute
errors if a object was 'unsafe' but not a string.
Add tests for AnsibleUnsafe, lookups, and AnsibleContext
Per a change in jinja2 2.9, local variables no longer are prefixed
with l_, so this updates AnsibleJ2Vars to pull in all locals (while
excluding some) regardless of name.
Fixes#20063
(cherry picked from commit 4d49b317929b86e1fc1b0cbace825ff73b372dc7)
* Enable tests on python 3 for uri
* Added one more node type to SAFE_NODES into safe_eval module.
ast.USub represents unary operators. This is necessary for
parsing some unusual but still valid JSON files during testing
with Python 3.
* Have template action plugin call do_template
Avoids all the magic done for 'inline templating' for ansible plays.
renamed _do_template to do_template in templar to make externally accessible.
fixes#18192
* added backwards compat as per feedback
We couldn't copy to_unicode, to_bytes, to_str into module_utils because
of licensing. So once created it we had two sets of functions that did
the same things but had different implementations. To remedy that, this
change removes the ansible.utils.unicode versions of those functions.
* dynamic role_include
* more fixes for dynamic include roles
* set play yfrom iterator when dynamic
* changes from jimi-c
* avoid modules that break ad hoc
TODO: should really be a config
Make !vault-encrypted create a AnsibleVaultUnicode
yaml object that can be used as a regular string object.
This allows a playbook to include a encrypted vault
blob for the value of a yaml variable. A 'secret_password'
variable can have it's value encrypted instead of having
to vault encrypt an entire vars file.
Add __ENCRYPTED__ to the vault yaml types so
template.Template can treat it similar
to __UNSAFE__ flags.
vault.VaultLib api changes:
- Split VaultLib.encrypt to encrypt and encrypt_bytestring
- VaultLib.encrypt() previously accepted the plaintext data
as either a byte string or a unicode string.
Doing the right thing based on the input type would fail
on py3 if given a arg of type 'bytes'. To simplify the
API, vaultlib.encrypt() now assumes input plaintext is a
py2 unicode or py3 str. It will encode to utf-8 then call
the new encrypt_bytestring(). The new methods are less
ambiguous.
- moved VaultLib.is_encrypted logic to vault module scope
and split to is_encrypted() and is_encrypted_file().
Add a test/unit/mock/yaml_helper.py
It has some helpers for testing parsing/yaml
Integration tests added as roles test_vault and test_vault_embedded
now deprecation message appears with variable name in all spots where this occurs
debug's var= option is excluded as this is only place where bare variables shold actually
be accepted.
There was code to support set literals (on Python 2.7 and newer), but it
was buggy: SAFE_NODES.union() doesn't modify SAFE_NODES in place,
instead it returns a new set object that is then silently discarded.
I added a unit test and fixed the code. I also changed the version
check to use sys.version_tuple instead of a string comparison, for
consistency with the subsequent Python 3.4 version check that I added in
the previous commit.
Two things changed in Python 3.4:
- 'basestring' is no longer defined, so use six.string_types
- True/False are now special AST node types (NamedConstant) rather than
just names
(Good thing we had tests, or I wouldn't have noticed the 2nd thing!)
I found only one place where safe_eval() is called inside the ansible
codebase: in lib/template/__init__.py. The call to safe_eval(result,
...) is protected by result.startswith('...'), which means result cannot
possibly be a byte string on Python 3 (or startswith() would raise, so
six.string_types (which excludes byte strings on Python 3) is fine here.
Replace .iteritems() with six.iteritems() everywhere except in
module_utils (because there's no 'six' on the remote host). And except
in lib/ansible/galaxy/data/metadata_template.j2, because I'm not sure
six is available there.
In v1, a trailing newline was kept if the parameter was passed as key=value. If
the parameter was passed as yaml dict the trailing newline was
discarded. Since key-value and yaml dict were unified in v2 we have to
make a choice as to which behaviour we want. Decided that keeping trailing
newlines by default made the most sense.
Fixes#12200Fixes#12199
This change is similar to https://github.com/ansible/ansible/pull/10465
It extends the logic there to also support none types. Right now if you have
a '!!null' in yaml, and that var gets passed around, it will get converted to
a string.
eg. defaults/main.yml
```
ENABLE_AWESOME_FEATURE: !!null # Yaml Null
OTHER_CONFIG:
secret1: "so_secret"
secret2: "even_more_secret"
CONFIG:
hostname: "some_hostname"
features:
awesame_feature: "{{ ENABLE_AWESOME_FEATURE}}"
secrets: "{{ OTHER_CONFIG }}"
```
If you output `CONFIG` to json or yaml, the feature flag would get represented in the output
as a string instead of as a null, but secrets would get represented as a dictionary. This is
a mis-match in behaviour where some "types" are retained and others are not. This change
should fix the issue.
I also updated the template test to test for this and made the changes to v2.
Added a changelog entry specifically for the change from empty string to null as the default.
Made the null representation configurable.
It still defaults to the python NoneType but can be overriden to be an emptystring by updating
the DEFAULT_NULL_REPRESENTATION config.
Fixes bugs related to creating Templar() objects on the fly, where
the shared loader objects (serialized to TaskExecutor) aren't used
so information loaded into plugin loaders after forking is lost.
Fixes#11815
This is unsafe and we debated re-adding it to the v2/2.0 codebase,
however it is a common-enough feature that we will simply mark it
as deprecated for now and remove it at some point in the future.
Fixes#11718
- now obeys global undefined var setting and allows override (mostly for with_ )
- moved environment instanciation to init instead of each template call
- removed hardcoded template token matching and now use actually configured tokens, now it won't break if someone changes default configs in ansible.cfg
- made reenetrant template calls now pass the same data it got, dictionary and lists were loosing existing and new params
- moved fail_on_undeinfed parameter to template call, as it should only realky be set to false on specific templates and not globally
- added overrides, which will allow template to implement jinja2 header override features
- added filter list to overrides to disallow possibly insecure ones, TODO: check if this is still needed as facts should not be templated anymore
- TODO: actually implement jinja2 header overrides
Matt Martz