Deprecate non lists lookups (#77875)

changelogs/fragments/77788-deprecate-non-lists-lookups.yml

@@ -0,0 +1,2 @@
+deprecated_features:
+  - Deprecate ability of lookup plugins to return arbitrary data. Lookup plugins must return lists, failing to do so will be an error in 2.18. (https://github.com/ansible/ansible/issues/77788)

docs/docsite/rst/dev_guide/developing_plugins.rst

@@ -344,7 +344,7 @@ Lookup plugins
 
 Lookup plugins pull in data from external data stores. Lookup plugins can be used within playbooks both for looping --- playbook language constructs like ``with_fileglob`` and ``with_items`` are implemented via lookup plugins --- and to return values into a variable or parameter.
 
-Lookup plugins are very flexible, allowing you to retrieve and return any type of data. When writing lookup plugins, always return data of a consistent type that can be easily consumed in a playbook. Avoid parameters that change the returned data type. If there is a need to return a single value sometimes and a complex dictionary other times, write two different lookup plugins.
+Lookup plugins are expected to return lists, even if just a single element.
 
 Ansible includes many :ref:`filters <playbooks_filters>` which can be used to manipulate the data returned by a lookup plugin. Sometimes it makes sense to do the filtering inside the lookup plugin, other times it is better to return results that can be filtered in the playbook. Keep in mind how the data will be referenced when determining the appropriate level of filtering to be done inside the lookup plugin.
 

docs/docsite/rst/plugins/lookup.rst

@@ -39,7 +39,7 @@ You can use lookup plugins anywhere you can use templating in Ansible: in a play
   vars:
     file_contents: "{{ lookup('file', 'path/to/file.txt') }}"
 
-Lookups are an integral part of loops. Wherever you see ``with_``, the part after the underscore is the name of a lookup. For this reason, most lookups output lists and take lists as input; for example, ``with_items`` uses the :ref:`items <items_lookup>` lookup::
+Lookups are an integral part of loops. Wherever you see ``with_``, the part after the underscore is the name of a lookup. For this reason, lookups are expected to output lists; for example, ``with_items`` uses the :ref:`items <items_lookup>` lookup::
 
   tasks:
     - name: count to 3

lib/ansible/template/__init__.py

@@ -988,6 +988,14 @@ class Templar:
                 raise AnsibleError(to_native(msg), orig_exc=e)
             return [] if wantlist else None
 
+        if not is_sequence(ran):
+            display.deprecated(
+                f'The lookup plugin \'{name}\' was expected to return a list, got \'{type(ran)}\' instead. '
+                f'The lookup plugin \'{name}\' needs to be changed to return a list. '
+                'This will be an error in Ansible 2.18',
+                version='2.18'
+            )
+
         if ran and allow_unsafe is False:
             if self.cur_context:
                 self.cur_context.unsafe = True
@@ -1013,9 +1021,10 @@ class Templar:
                     ran = wrap_var(ran[0])
                 else:
                     ran = wrap_var(ran)
-
             except KeyError:
-                # Lookup Plugin returned a dict.  Return comma-separated string.
+                # Lookup Plugin returned a dict.  Return comma-separated string of keys
+                # for backwards compat.
+                # FIXME this can be removed when support for non-list return types is removed.
                 # See https://github.com/ansible/ansible/pull/77789
                 ran = wrap_var(",".join(ran))