diff --git a/docs/docsite/rst/user_guide/complex_data_manipulation.rst b/docs/docsite/rst/user_guide/complex_data_manipulation.rst new file mode 100644 index 00000000000..b9900b50105 --- /dev/null +++ b/docs/docsite/rst/user_guide/complex_data_manipulation.rst @@ -0,0 +1,243 @@ +.. _complex_data_manipulation: + +Data manipulation +######################### + +In many cases, you need to do some complex operation with your variables, while Ansible is not recommended as a data processing/manipulation tool, you can use the existing Jinja2 templating in conjunction with the many added Ansible filters, lookups and tests to do some very complex transformations. + +Let's start with a quick definition of each type of plugin: + - lookups: Mainly used to query 'external data', in Ansible these were the primary part of loops using the ``with_`` construct, but they can be used independently to return data for processing. They normally return a list due to their primary function in loops as mentioned previously. Used with the ``lookup`` or ``query`` Jinja2 operators. + - filters: used to change/transform data, used with the ``|`` Jinja2 operator. + - tests: used to validate data, used with the ``is`` Jinja2 operator. + +.. _note: + * Some tests and filters are provided directly by Jinja2, so their availability depends on the Jinja2 version, not Ansible. + +.. _for_loops_or_list_comprehensions: + +Loops and list comprehensions +============================= + +Most programming languages have loops (``for``, ``while``, etc.) and list comprehensions to do transformations on lists including lists of objects. Jinja2 has a few filters that provide this functionality: ``map``, ``select``, ``reject``, ``selectattr``, ``rejectattr``. + +- map: this is a basic for loop that just allows you to change every item in a list, using the 'attribute' keyword you can do the transformation based on attributes of the list elements. +- select/reject: this is a for loop with a condition, that allows you to create a subset of a list that matches (or not) based on the result of the condition. +- selectattr/rejectattr: very similar to the above but it uses a specific attribute of the list elements for the conditional statement. + + +.. _keys_from_dict_matching_list: + +Extract keys from a dictionary matching elements from a list +------------------------------------------------------------ + +The Python equivalent code would be: + +.. code-block:: python + + chains = [1, 2] + for chain in chains: + for config in chains_config[chain]['configs']: + print(config['type']) + +There are several ways to do it in Ansible, this is just one example: + +.. code-block:: YAML+Jinja + :emphasize-lines: 3 + :caption: Way to extract matching keys from a list of dictionaries + + tasks: + - name: Show extracted list of keys from a list of dictionaries + debug: msg="{{ chains | map('extract', chains_config) | map(attribute='configs') | flatten | map(attribute='type') | flatten }}" + vars: + chains: [1, 2] + chains_config: + 1: + foo: bar + configs: + - type: routed + version: 0.1 + - type: bridged + version: 0.2 + 2: + foo: baz + configs: + - type: routed + version: 1.0 + - type: bridged + version: 1.1 + + +.. code-block:: ansible-output + :caption: Results of debug task, a list with the extracted keys + + ok: [localhost] => { + "msg": [ + "routed", + "bridged", + "routed", + "bridged" + ] + } + + +.. _find_mount_point: + +Find mount point +---------------- + +In this case, we want to find the mount point for a given path across our machines, since we already collect mount facts, we can use the following: + +.. code-block:: YAML+Jinja + :caption: Use selectattr to filter mounts into list I can then sort and select the last from + :emphasize-lines: 7 + + - hosts: all + gather_facts: True + vars: + path: /var/lib/cache + tasks: + - name: The mount point for {{path}}, found using the Ansible mount facts, [-1] is the same as the 'last' filter + debug: msg="{{(ansible_facts.mounts | selectattr('mount', 'in', path) | list | sort(attribute='mount'))[-1]['mount']}}" + + + +Omit elements from a list +------------------------- + +The special ``omit`` variable ONLY works with module options, but we can still use it in other ways as an identifier to tailor a list of elements: + +.. code-block:: YAML+Jinja + :caption: Inline list filtering when feeding a module option + :emphasize-lines: 3, 7 + + - name: enable a list of Windows features, by name + set_fact: + win_feature_list: "{{ namestuff | reject('equalto', omit) | list }}" + vars: + namestuff: + - "{{ (fs_installed_smb_v1 | default(False)) | ternary(omit, 'FS-SMB1') }}" + - "foo" + - "bar" + + +Another way is to avoid adding elements to the list in the first place, so you can just use it directly: + +.. code-block:: YAML+Jinja + :caption: Using set_fact in a loop to increment a list conditionally + :emphasize-lines: 3, 4, 6 + + - name: build unique list with some items conditionally omittted + set_fact: + namestuff: ' {{ (namestuff | default([])) | union([item]) }}' + when: item != omit + loop: + - "{{ (fs_installed_smb_v1 | default(False)) | ternary(omit, 'FS-SMB1') }}" + - "foo" + - "bar" + + +.. _complex_type_transfomations: + +Complex Type transformations +============================= + +Jinja provides filters for simple data type transformations (``int``, ``bool``, etc), but when you want to transform data structures things are not as easy. +You can use loops and list comprehensions as shown above to help, also other filters and lookups can be chained and leveraged to achieve more complex transformations. + + +.. _create_dictionary_from_list: + +Create dictionary from list +--------------------------- + +In most languages it is easy to create a dictionary (a.k.a. map/associative array/hash etc.) from a list of pairs, in Ansible there are a couple of ways to do it and the best one for you might depend on the source of your data. + + +These example produces ``{"a": "b", "c": "d"}`` + +.. code-block:: YAML+Jinja + :caption: Simple list to dict by assuming the list is [key, value , key, value, ...] + + vars: + single_list: [ 'a', 'b', 'c', 'd' ] + mydict: "{{ dict(single_list) | slice(2) | list }}" + + +.. code-block:: YAML+Jinja + :caption: It is simpler when we have a list of pairs: + + vars: + list_of_pairs: [ ['a', 'b'], ['c', 'd'] ] + mydict: "{{ dict(list_of_pairs) }}" + +Both end up being the same thing, with the ``slice(2) | list`` transforming ``single_list`` to the same structure as ``list_of_pairs``. + + + +A bit more complex, using ``set_fact`` and a ``loop`` to create/update a dictionary with key value pairs from 2 lists: + +.. code-block:: YAML+Jinja + :caption: Using set_fact to create a dictionary from a set of lists + :emphasize-lines: 3, 4 + + - name: Uses 'combine' to update the dictionary and 'zip' to make pairs of both lists + set_fact: + mydict: "{{ mydict | default({}) | combine({item[0]: item[1]}) }}" + loop: "{{ (keys | zip(values)) | list }}" + vars: + keys: + - foo + - var + - bar + values: + - a + - b + - c + +This results in ``{"foo": "a", "var": "b", "bar": "c"}``. + + +You can even combine these simple examples with other filters and lookups to create a dictionary dynamically by matching patterns to variable names: + +.. code-block:: YAML+Jinja + :caption: Using 'vars' to define dictionary from a set of lists without needing a task + + vars: + myvarnames: "{{ q('varnames', '^my') }}" + mydict: "{{ dict(myvarnames | zip(q('vars', *myvarnames))) }}" + +A quick explanation, since there is a lot to unpack from these two lines: + + - The ``varnames`` lookup returns a list of variables that match "begin with ``my``". + - Then feeding the list from the previous step into the ``vars`` lookup to get the list of values. + The ``*`` is used to 'dereference the list' (a pythonism that works in Jinja), otherwise it would take the list as a single argument. + - Both lists get passed to the ``zip`` filter to pair them off into a unified list (key, value, key2, value2, ...). + - The dict function then takes this 'list of pairs' to create the dictionary. + + +An example on how to use facts to find a host's data that meets condition X: + + +.. code-block:: YAML+Jinja + + vars: + uptime_of_host_most_recently_rebooted: "{{ansible_play_hosts_all | map('extract', hostvars, 'ansible_uptime_seconds') | sort | first}}" + + +Using an example from @zoradache on reddit, to show the 'uptime in days/hours/minutes' (assumes facts where gathered). +https://www.reddit.com/r/ansible/comments/gj5a93/trying_to_get_uptime_from_seconds/fqj2qr3/ + +.. code-block:: YAML+Jinja + + - debug: + msg: Timedelta {{ now() - now().fromtimestamp(now(fmt='%s') | int - ansible_uptime_seconds) }} + + +.. seealso:: + + :doc:`playbooks_filters` + Jinja2 filters included with Ansible + :doc:`playbooks_tests` + Jinja2 tests included with Ansible + `Jinja2 Docs `_ + Jinja2 documentation, includes lists for core filters and tests diff --git a/docs/docsite/rst/user_guide/playbooks_advanced_syntax.rst b/docs/docsite/rst/user_guide/playbooks_advanced_syntax.rst index 8ab71aa87b5..2edf36d39d6 100644 --- a/docs/docsite/rst/user_guide/playbooks_advanced_syntax.rst +++ b/docs/docsite/rst/user_guide/playbooks_advanced_syntax.rst @@ -104,6 +104,8 @@ You've anchored the value of ``version`` with the ``&my_version`` anchor, and re :ref:`playbooks_variables` All about variables + :doc:`complex_data_manipulation` + Doing complex data manipulation in Ansible `User Mailing List `_ Have a question? Stop by the google group! `irc.freenode.net `_ diff --git a/docs/docsite/rst/user_guide/playbooks_special_topics.rst b/docs/docsite/rst/user_guide/playbooks_special_topics.rst index a1646413d7d..0ef1f53086e 100644 --- a/docs/docsite/rst/user_guide/playbooks_special_topics.rst +++ b/docs/docsite/rst/user_guide/playbooks_special_topics.rst @@ -16,6 +16,7 @@ As you write more playbooks and roles, you might have some special use cases. Fo playbooks_environment playbooks_error_handling playbooks_advanced_syntax + complex_data_manipulation ../plugins/plugins playbooks_prompts playbooks_tags