Role arg spec validation implementation (#73152)

* Initial import of modified version of alikins' code
* Add unit testing for new Role methods
* Fix validate_arg_spec module for sanity test. Add test_include_role_fails.yml integration test from orig PR.
* Add testing of suboptions
* Use new ArgumentSpecValidator class instead of AnsibleModule
* fix for roles with no tasks, use FQ name of new plugin
* Add role dep warning

changelogs/fragments/73152-role-arg-spec.yaml

@@ -0,0 +1,4 @@
+major_changes:
+  - Support for role argument specification validation at role execution time.
+    When a role contains an argument spec, an implicit validation task is inserted
+    at the start of role execution.

docs/docsite/rst/user_guide/playbooks_reuse_roles.rst

@@ -256,6 +256,118 @@ You can pass other keywords, including variables and tags, when importing roles:
 
 When you add a tag to an ``import_role`` statement, Ansible applies the tag to `all` tasks within the role. See :ref:`tag_inheritance` for details.
 
+Role Argument Validation
+========================
+
+Beginning with version 2.11, you may choose to enable role argument validation based on an argument
+specification defined in the role ``meta/main.yml`` file. When this argument specification is defined,
+a new task is inserted at the beginning of role execution that will validate the parameters supplied
+for the role against the specification. If the parameters fail validation, the role will fail execution.
+
+Specification Format
+--------------------
+
+The role argument specification must be defined in a top-level ``argument_specs`` block within the
+role ``meta/main.yml`` file. All fields are lower-case.
+
+:entry-point-name:
+
+    * The name of the role entry point.
+    * This should be ``main`` in the case of an unspecified entry point.
+    * This will be the base name of the tasks file to execute, with no ``.yml`` or ``.yaml`` file extension.
+
+    :short_description:
+
+        * A short, one-line description of the entry point.
+        * The ``short_description`` is displayed by ``ansible-doc -t role -l``.
+
+    :description:
+
+        * A longer description that may contain multiple lines.
+
+    :author:
+
+        * Name of the entry point authors.
+        * Use a multi-line list if there is more than one author.
+
+    :options:
+
+        * Options are often called "parameters" or "arguments". This section defines those options.
+        * For each role option (argument), you may include:
+
+        :option-name:
+
+           * The name of the option/argument.
+
+        :description:
+
+           * Detailed explanation of what this option does. It should be written in full sentences.
+
+        :type:
+
+           * The data type of the option. Default is ``str``.
+           * If an option is of type ``list``, ``elements`` should be specified.
+
+        :required:
+
+           * Only needed if ``true``.
+           * If missing, the option is not required.
+
+        :default:
+
+           * If ``required`` is false/missing, ``default`` may be specified (assumed 'null' if missing).
+           * Ensure that the default value in the docs matches the default value in the code. The actual
+             default for the role variable will always come from ``defaults/main.yml``.
+           * The default field must not be listed as part of the description, unless it requires additional information or conditions.
+           * If the option is a boolean value, you can use any of the boolean values recognized by Ansible:
+             (such as true/false or yes/no).  Choose the one that reads better in the context of the option.
+
+        :choices:
+
+           * List of option values.
+           * Should be absent if empty.
+
+        :elements:
+
+           * Specifies the data type for list elements when type is ``list``.
+
+        :suboptions:
+
+           * If this option takes a dict or list of dicts, you can define the structure here.
+
+Sample Specification
+--------------------
+
+.. code-block:: yaml
+
+ # roles/myapp/meta/main.yml
+ ---
+ argument_specs:
+   # roles/myapp/tasks/main.yml entry point
+   main:
+     short_description: The main entry point for the myapp role.
+     options:
+       myapp_int:
+         type: "int"
+         required: false
+         default: 42
+         description: "The integer value, defaulting to 42."
+
+       myapp_str:
+         type: "str"
+         required: true
+         description: "The string value"
+
+   # roles/maypp/tasks/alternate.yml entry point
+   alternate:
+     short_description: The alternate entry point for the myapp role.
+     options:
+       myapp_int:
+         type: "int"
+         required: false
+         default: 1024
+         description: "The integer value, defaulting to 1024."
+
 .. _run_role_twice:
 
 Running a role multiple times in one playbook

lib/ansible/modules/validate_argument_spec.py

@@ -0,0 +1,63 @@
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+
+# Copyright 2021 Red Hat
+# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
+
+from __future__ import absolute_import, division, print_function
+__metaclass__ = type
+
+
+DOCUMENTATION = r'''
+---
+module: validate_argument_spec
+short_description: Validate role argument specs.
+description:
+     - This module validates role arguments with a defined argument specification.
+version_added: "2.11"
+options:
+  argument_spec:
+    description:
+        - A dictionary like AnsibleModule argument_spec
+    required: true
+  provided_arguments:
+    description:
+        - A dictionary of the arguments that will be validated according to argument_spec
+author:
+    - Ansible Core Team
+'''
+
+EXAMPLES = r'''
+'''
+
+RETURN = r'''
+argument_errors:
+  description: A list of arg validation errors.
+  returned: failure
+  type: list
+  elements: str
+  sample:
+    - "error message 1"
+    - "error message 2"
+
+argument_spec_data:
+  description: A dict of the data from the 'argument_spec' arg.
+  returned: failure
+  type: dict
+  sample:
+    some_arg:
+      type: "str"
+    some_other_arg:
+      type: "int"
+      required: true
+
+validate_args_context:
+  description: A dict of info about where validate_args_spec was used
+  type: dict
+  returned: always
+  sample:
+    name: my_role
+    type: role
+    path: /home/user/roles/my_role/
+    argument_spec_name: main
+'''

lib/ansible/playbook/role/__init__.py

@@ -255,6 +255,9 @@ class Role(Base, Conditional, Taggable, CollectionSearch):
                 self.collections.append(default_append_collection)
 
         task_data = self._load_role_yaml('tasks', main=self._from_files.get('tasks'))
+
+        task_data = self._prepend_validation_task(task_data)
+
         if task_data:
             try:
                 self._task_blocks = load_list_of_blocks(task_data, play=self._play, role=self, loader=self._loader, variable_manager=self._variable_manager)
@@ -271,6 +274,67 @@ class Role(Base, Conditional, Taggable, CollectionSearch):
                 raise AnsibleParserError("The handlers/main.yml file for role '%s' must contain a list of tasks" % self._role_name,
                                          obj=handler_data, orig_exc=e)
 
+    def _prepend_validation_task(self, task_data):
+        '''Insert a role validation task if we have a role argument spec.
+
+        This method will prepend a validation task to the front of the role task
+        list to perform argument spec validation before any other tasks, if an arg spec
+        exists for the entry point. Entry point defaults to `main`.
+
+        :param task_data: List of tasks loaded from the role.
+
+        :returns: The (possibly modified) task list.
+        '''
+        if self._metadata.argument_specs:
+            if self._dependencies:
+                display.warning("Dependent roles will run before roles with argument specs even if validation fails.")
+
+            # Determine the role entry point so we can retrieve the correct argument spec.
+            # This comes from the `tasks_from` value to include_role or import_role.
+            entrypoint = self._from_files.get('tasks', 'main')
+            entrypoint_arg_spec = self._metadata.argument_specs.get(entrypoint)
+
+            if entrypoint_arg_spec:
+                validation_task = self._create_validation_task(entrypoint_arg_spec, entrypoint)
+
+                # Prepend our validate_argument_spec action to happen before any tasks provided by the role.
+                # 'any tasks' can and does include 0 or None tasks, in which cases we create a list of tasks and add our
+                # validate_argument_spec task
+                if not task_data:
+                    task_data = []
+                task_data.insert(0, validation_task)
+        return task_data
+
+    def _create_validation_task(self, argument_spec, entrypoint_name):
+        '''Create a new task data structure that uses the validate_argument_spec action plugin.
+
+        :param argument_spec: The arg spec definition for a particular role entry point.
+            This will be the entire arg spec for the entry point as read from the input file.
+        :param entrypoint_name: The name of the role entry point associated with the
+            supplied `argument_spec`.
+        '''
+
+        # If the arg spec provides a short description, use it to flesh out the validation task name
+        task_name = "Validating arguments against arg spec '%s'" % entrypoint_name
+        if 'short_description' in argument_spec:
+            task_name = task_name + ' - ' + argument_spec['short_description']
+
+        return {
+            'action': {
+                'module': 'ansible.builtin.validate_argument_spec',
+                # Pass only the 'options' portion of the arg spec to the module.
+                'argument_spec': argument_spec.get('options', {}),
+                'provided_arguments': self._role_params,
+                'validate_args_context': {
+                    'type': 'role',
+                    'name': self._role_name,
+                    'argument_spec_name': entrypoint_name,
+                    'path': self._role_path
+                },
+            },
+            'name': task_name,
+        }
+
     def _load_role_yaml(self, subdir, main=None, allow_dir=False):
         '''
         Find and load role YAML files and return data found.

lib/ansible/plugins/action/validate_argument_spec.py

@@ -0,0 +1,97 @@
+# Copyright 2021 Red Hat
+# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
+
+from __future__ import (absolute_import, division, print_function)
+__metaclass__ = type
+
+from ansible.errors import AnsibleError
+from ansible.plugins.action import ActionBase
+from ansible.module_utils.six import iteritems, string_types
+from ansible.module_utils.common.arg_spec import ArgumentSpecValidator
+
+
+class ActionModule(ActionBase):
+    ''' Validate an arg spec'''
+
+    TRANSFERS_FILES = False
+
+    def get_args_from_task_vars(self, argument_spec, task_vars):
+        '''
+        Get any arguments that may come from `task_vars`.
+
+        Expand templated variables so we can validate the actual values.
+
+        :param argument_spec: A dict of the argument spec.
+        :param task_vars: A dict of task variables.
+
+        :returns: A dict of values that can be validated against the arg spec.
+        '''
+        args = {}
+
+        for argument_name, argument_attrs in iteritems(argument_spec):
+            if argument_name in task_vars:
+                if isinstance(task_vars[argument_name], string_types):
+                    value = self._templar.do_template(task_vars[argument_name])
+                    if value:
+                        args[argument_name] = value
+                else:
+                    args[argument_name] = task_vars[argument_name]
+        return args
+
+    def run(self, tmp=None, task_vars=None):
+        '''
+        Validate an argument specification against a provided set of data.
+
+        The `validate_argument_spec` module expects to receive the arguments:
+            - argument_spec: A dict whose keys are the valid argument names, and
+                  whose values are dicts of the argument attributes (type, etc).
+            - provided_arguments: A dict whose keys are the argument names, and
+                  whose values are the argument value.
+
+        :param tmp: Deprecated. Do not use.
+        :param task_vars: A dict of task variables.
+        :return: An action result dict, including a 'argument_errors' key with a
+            list of validation errors found.
+        '''
+        if task_vars is None:
+            task_vars = dict()
+
+        result = super(ActionModule, self).run(tmp, task_vars)
+        del tmp  # tmp no longer has any effect
+
+        # This action can be called from anywhere, so pass in some info about what it is
+        # validating args for so the error results make some sense
+        result['validate_args_context'] = self._task.args.get('validate_args_context', {})
+
+        if 'argument_spec' not in self._task.args:
+            raise AnsibleError('"argument_spec" arg is required in args: %s' % self._task.args)
+
+        # Get the task var called argument_spec. This will contain the arg spec
+        # data dict (for the proper entry point for a role).
+        argument_spec_data = self._task.args.get('argument_spec')
+
+        # the values that were passed in and will be checked against argument_spec
+        provided_arguments = self._task.args.get('provided_arguments', {})
+
+        if not isinstance(argument_spec_data, dict):
+            raise AnsibleError('Incorrect type for argument_spec, expected dict and got %s' % type(argument_spec_data))
+
+        if not isinstance(provided_arguments, dict):
+            raise AnsibleError('Incorrect type for provided_arguments, expected dict and got %s' % type(provided_arguments))
+
+        args_from_vars = self.get_args_from_task_vars(argument_spec_data, task_vars)
+        provided_arguments.update(args_from_vars)
+
+        validator = ArgumentSpecValidator(argument_spec_data, provided_arguments)
+
+        if not validator.validate():
+            result['failed'] = True
+            result['msg'] = 'Validation of arguments failed:\n%s' % '\n'.join(validator.error_messages)
+            result['argument_spec_data'] = argument_spec_data
+            result['argument_errors'] = validator.error_messages
+            return result
+
+        result['changed'] = False
+        result['msg'] = 'The arg spec validation passed'
+
+        return result

test/integration/targets/roles_arg_spec/aliases

@@ -0,0 +1 @@
+shippable/posix/group5

test/integration/targets/roles_arg_spec/roles/a/meta/main.yml

@@ -0,0 +1,17 @@
+argument_specs:
+    main:
+        short_description: Main entry point for role A.
+        options:
+            a_str:
+                type: "str"
+                required: true
+
+    alternate:
+        short_description: Alternate entry point for role A.
+        options:
+            a_int:
+                type: "int"
+                required: true
+
+    no_spec_entrypoint:
+        short_description: An entry point with no spec

test/integration/targets/roles_arg_spec/roles/a/tasks/alternate.yml

@@ -0,0 +1,3 @@
+---
+- debug:
+    msg: "Role A (alternate) with {{ a_int }}"

test/integration/targets/roles_arg_spec/roles/a/tasks/main.yml

@@ -0,0 +1,3 @@
+---
+- debug:
+    msg: "Role A with {{ a_str }}"

test/integration/targets/roles_arg_spec/roles/a/tasks/no_spec_entrypoint.yml

@@ -0,0 +1,3 @@
+---
+- debug:
+    msg: "Role A no_spec_entrypoint"

test/integration/targets/roles_arg_spec/roles/b/meta/main.yml

@@ -0,0 +1,13 @@
+argument_specs:
+    main:
+        short_description: Main entry point for role B.
+        options:
+            b_str:
+                type: "str"
+                required: true
+            b_int:
+                type: "int"
+                required: true
+            b_bool:
+                type: "bool"
+                required: true

test/integration/targets/roles_arg_spec/roles/b/tasks/main.yml

@@ -0,0 +1,9 @@
+---
+- debug:
+    msg: "Role B"
+- debug:
+    var: b_str
+- debug:
+    var: b_int
+- debug:
+    var: b_bool

test/integration/targets/roles_arg_spec/roles/c/meta/main.yml

@@ -0,0 +1,7 @@
+argument_specs:
+    main:
+        short_description: Main entry point for role C.
+        options:
+            c_int:
+                type: "int"
+                required: true

test/integration/targets/roles_arg_spec/roles/c/tasks/main.yml

@@ -0,0 +1,12 @@
+---
+- debug:
+    msg: "Role C that includes Role A with var {{ c_int }}"
+
+- name: "Role C import_role A with a_str {{ a_str }}"
+  import_role:
+    name: a
+
+- name: "Role C include_role A with a_int {{ a_int }}"
+  include_role:
+    name: a
+    tasks_from: "alternate"

test/integration/targets/roles_arg_spec/roles/role_with_no_tasks/meta/main.yml

@@ -0,0 +1,7 @@
+argument_specs:
+    main:
+        short_description: Main entry point for role role_with_no_tasks.
+        options:
+            a_str:
+                type: "str"
+                required: true

test/integration/targets/roles_arg_spec/roles/test1/defaults/main.yml

@@ -0,0 +1,3 @@
+---
+# defaults file for test1
+test1_var1: 'THE_TEST1_VAR1_DEFAULT_VALUE'

test/integration/targets/roles_arg_spec/roles/test1/meta/main.yml

@@ -0,0 +1,107 @@
+---
+argument_specs:
+  main:
+    short_description: "EXPECTED FAILURE Validate the argument spec for the 'test1' role"
+    options:
+     test1_choices:
+       required: false
+       # required: true
+       choices:
+         - "this paddle game"
+         - "the astray"
+         - "this remote control"
+         - "the chair"
+       type: "str"
+       default: "this paddle game"
+     tidy_expected:
+       # required: false
+       # default: none
+       type: "list"
+     test1_var1:
+       # required: true
+       default: "THIS IS THE DEFAULT SURVEY ANSWER FOR test1_survey_test1_var1"
+       type: "str"
+     test1_var2:
+       required: false
+       default: "This IS THE DEFAULT fake band name / test1_var2 answer from survey_spec.yml"
+       type: "str"
+     bust_some_stuff:
+       # required: false
+       type: "int"
+     some_choices:
+       choices:
+         - "choice1"
+         - "choice2"
+       required: false
+       type: "str"
+     some_str:
+       type: "str"
+     some_list:
+       type: "list"
+       elements: "float"
+     some_dict:
+       type: "dict"
+     some_bool:
+       type: "bool"
+     some_int:
+       type: "int"
+     some_float:
+       type: "float"
+     some_path:
+       type: "path"
+     some_raw:
+       type: "raw"
+     some_jsonarg:
+       type: "jsonarg"
+       required: true
+     some_json:
+       type: "json"
+       required: true
+     some_bytes:
+       type: "bytes"
+     some_bits:
+       type: "bits"
+     some_str_aliases:
+       type: "str"
+       aliases:
+         - "some_str_nicknames"
+         - "some_str_akas"
+         - "some_str_formerly_known_as"
+     some_dict_options:
+       type: "dict"
+       options:
+         some_second_level:
+           type: "bool"
+           default: true
+     some_str_removed_in:
+       type: "str"
+       removed_in: 2.10
+     some_tmp_path:
+       type: "path"
+     multi_level_option:
+       type: "dict"
+       options:
+         second_level:
+           type: "dict"
+           options:
+             third_level:
+               type: "int"
+               required: true
+
+  other:
+    short_description: "test1_simple_preset_arg_spec_other"
+    description: "A simpler set of required args for other tasks"
+    options:
+      test1_var1:
+        default: "This the default value for the other set of arg specs for  test1 test1_var1"
+        type: "str"
+
+  test1_other:
+    description: "test1_other for role_that_includes_role"
+    options:
+     some_test1_other_arg:
+       default: "The some_test1_other_arg default value"
+       type: str
+     some_required_str:
+       type: str
+       required: true

test/integration/targets/roles_arg_spec/roles/test1/tasks/main.yml

@@ -0,0 +1,11 @@
+---
+# tasks file for test1
+- name: debug for task1 show test1_var1
+  debug:
+     var: test1_var1
+  tags: ["runme"]
+
+- name: debug for task1 show test1_var2
+  debug:
+     var: test1_var2
+  tags: ["runme"]

test/integration/targets/roles_arg_spec/roles/test1/tasks/other.yml

@@ -0,0 +1,11 @@
+---
+# "other" tasks file for test1
+- name: other tasks debug for task1 show test1_var1
+  debug:
+     var: test1_var1
+  tags: ["runme"]
+
+- name: other tasks debug for task1 show test1_var2
+  debug:
+     var: test1_var2
+  tags: ["runme"]

test/integration/targets/roles_arg_spec/roles/test1/tasks/test1_other.yml

@@ -0,0 +1,11 @@
+---
+# "test1_other" tasks file for test1
+- name: "test1_other BLIPPY test1_other tasks debug for task1 show test1_var1"
+  debug:
+     var: test1_var1
+  tags: ["runme"]
+
+- name: "BLIPPY FOO test1_other tasks debug for task1 show test1_var2"
+  debug:
+     var: test1_var2
+  tags: ["runme"]

test/integration/targets/roles_arg_spec/roles/test1/vars/main.yml

@@ -0,0 +1,4 @@
+---
+# vars file for test1
+test1_var1: 'THE_TEST1_VAR1_VARS_VALUE'
+test1_var2: 'THE_TEST1_VAR2_VARS_VALUE'

test/integration/targets/roles_arg_spec/roles/test1/vars/other.yml

@@ -0,0 +1,4 @@
+---
+# vars file for test1
+test1_var1: 'other_THE_TEST1_VAR1_VARS_VALUE'
+test1_var2: 'other_THE_TEST1_VAR2_VARS_VALUE'

test/integration/targets/roles_arg_spec/runme.sh

@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+
+set -eux
+
+# Various simple role scenarios
+ansible-playbook test.yml -i ../../inventory "$@"
+
+# More complex role test
+ansible-playbook test_complex_role_fails.yml -i ../../inventory "$@"
+
+# Test play level role will fail
+set +e
+ansible-playbook test_play_level_role_fails.yml -i ../../inventory "$@"
+test $? -ne 0
+set -e

test/integration/targets/roles_arg_spec/test.yml

@@ -0,0 +1,249 @@
+---
+- hosts: localhost
+  gather_facts: false
+  roles:
+      - { role: a, a_str: "roles" }
+
+  vars:
+    INT_VALUE: 42
+
+  tasks:
+
+      - name: "Valid simple role usage with include_role"
+        include_role:
+          name: a
+        vars:
+          a_str: "include_role"
+
+      - name: "Valid simple role usage with import_role"
+        import_role:
+          name: a
+        vars:
+            a_str: "import_role"
+
+      - name: "Valid role usage (more args)"
+        include_role:
+          name: b
+        vars:
+          b_str: "xyz"
+          b_int: 5
+          b_bool: true
+
+      - name: "Valid simple role usage with include_role of different entry point"
+        include_role:
+          name: a
+          tasks_from: "alternate"
+        vars:
+          a_int: 256
+
+      - name: "Valid simple role usage with import_role of different entry point"
+        import_role:
+          name: a
+          tasks_from: "alternate"
+        vars:
+          a_int: 512
+
+      - name: "Valid simple role usage with a templated value"
+        import_role:
+          name: a
+        vars:
+          a_int: "{{ INT_VALUE }}"
+
+      - name: "Call role entry point that is defined, but has no spec data"
+        import_role:
+          name: a
+          tasks_from: "no_spec_entrypoint"
+
+- name: "New play to reset vars: Test include_role fails"
+  hosts: localhost
+  gather_facts: false
+  vars:
+    expected_returned_spec:
+      b_bool:
+        required: true
+        type: "bool"
+      b_int:
+        required: true
+        type: "int"
+      b_str:
+        required: true
+        type: "str"
+
+  tasks:
+      - block:
+        - name: "EXPECTED FAILURE: Invalid role usage"
+          include_role:
+            name: b
+          vars:
+            b_bool: 7
+
+        - fail:
+            msg: "Should not get here"
+
+        rescue:
+          - debug:
+              var: ansible_failed_result
+
+          - name: "Validate failure"
+            assert:
+              that:
+                - ansible_failed_task.name == "Validating arguments against arg spec 'main' - Main entry point for role B."
+                - ansible_failed_result.argument_errors | length == 2
+                - "'missing required arguments: b_int, b_str' in ansible_failed_result.argument_errors"
+                - ansible_failed_result.validate_args_context.argument_spec_name == "main"
+                - ansible_failed_result.validate_args_context.name == "b"
+                - ansible_failed_result.validate_args_context.type == "role"
+                - "ansible_failed_result.validate_args_context.path is search('roles_arg_spec/roles/b')"
+                - ansible_failed_result.argument_spec_data == expected_returned_spec
+
+
+- name: "New play to reset vars: Test import_role fails"
+  hosts: localhost
+  gather_facts: false
+  vars:
+    expected_returned_spec:
+      b_bool:
+        required: true
+        type: "bool"
+      b_int:
+        required: true
+        type: "int"
+      b_str:
+        required: true
+        type: "str"
+
+  tasks:
+      - block:
+        - name: "EXPECTED FAILURE: Invalid role usage"
+          import_role:
+            name: b
+          vars:
+            b_bool: 7
+
+        - fail:
+            msg: "Should not get here"
+
+        rescue:
+          - debug:
+              var: ansible_failed_result
+
+          - name: "Validate failure"
+            assert:
+              that:
+                - ansible_failed_task.name == "Validating arguments against arg spec 'main' - Main entry point for role B."
+                - ansible_failed_result.argument_errors | length == 2
+                - "'missing required arguments: b_int, b_str' in ansible_failed_result.argument_errors"
+                - ansible_failed_result.validate_args_context.argument_spec_name == "main"
+                - ansible_failed_result.validate_args_context.name == "b"
+                - ansible_failed_result.validate_args_context.type == "role"
+                - "ansible_failed_result.validate_args_context.path is search('roles_arg_spec/roles/b')"
+                - ansible_failed_result.argument_spec_data == expected_returned_spec
+
+
+- name: "New play to reset vars: Test nested role including/importing role succeeds"
+  hosts: localhost
+  gather_facts: false
+  vars:
+    c_int: 1
+    a_str: "some string"
+    a_int: 42
+  tasks:
+      - name: "Test import_role of role C"
+        import_role:
+          name: c
+
+      - name: "Test include_role of role C"
+        include_role:
+          name: c
+
+
+- name: "New play to reset vars: Test nested role including/importing role fails"
+  hosts: localhost
+  gather_facts: false
+  vars:
+    main_expected_returned_spec:
+      a_str:
+        required: true
+        type: "str"
+    alternate_expected_returned_spec:
+      a_int:
+        required: true
+        type: "int"
+
+  tasks:
+      - block:
+        - name: "EXPECTED FAILURE: Test import_role of role C (missing a_str)"
+          import_role:
+            name: c
+          vars:
+            c_int: 100
+
+        - fail:
+            msg: "Should not get here"
+
+        rescue:
+          - debug:
+              var: ansible_failed_result
+          - name: "Validate import_role failure"
+            assert:
+              that:
+                # NOTE: a bug here that prevents us from getting ansible_failed_task
+                - ansible_failed_result.argument_errors | length == 1
+                - "'missing required arguments: a_str' in ansible_failed_result.argument_errors"
+                - ansible_failed_result.validate_args_context.argument_spec_name == "main"
+                - ansible_failed_result.validate_args_context.name == "a"
+                - ansible_failed_result.validate_args_context.type == "role"
+                - "ansible_failed_result.validate_args_context.path is search('roles_arg_spec/roles/a')"
+                - ansible_failed_result.argument_spec_data == main_expected_returned_spec
+
+      - block:
+        - name: "EXPECTED FAILURE: Test include_role of role C (missing a_int from `alternate` entry point)"
+          include_role:
+            name: c
+          vars:
+            c_int: 200
+            a_str: "some string"
+
+        - fail:
+            msg: "Should not get here"
+
+        rescue:
+          - debug:
+              var: ansible_failed_result
+          - name: "Validate include_role failure"
+            assert:
+              that:
+                # NOTE: a bug here that prevents us from getting ansible_failed_task
+                - ansible_failed_result.argument_errors | length == 1
+                - "'missing required arguments: a_int' in ansible_failed_result.argument_errors"
+                - ansible_failed_result.validate_args_context.argument_spec_name == "alternate"
+                - ansible_failed_result.validate_args_context.name == "a"
+                - ansible_failed_result.validate_args_context.type == "role"
+                - "ansible_failed_result.validate_args_context.path is search('roles_arg_spec/roles/a')"
+                - ansible_failed_result.argument_spec_data == alternate_expected_returned_spec
+
+- name: "New play to reset vars: Test role with no tasks can fail"
+  hosts: localhost
+  gather_facts: false
+  tasks:
+      - block:
+        - name: "EXPECTED FAILURE: Test import_role of role role_with_no_tasks (missing a_str)"
+          import_role:
+            name: role_with_no_tasks
+
+        - fail:
+            msg: "Should not get here"
+
+        rescue:
+          - debug:
+              var: ansible_failed_result
+          - name: "Validate import_role failure"
+            assert:
+              that:
+                # NOTE: a bug here that prevents us from getting ansible_failed_task
+                - ansible_failed_result.argument_errors | length == 1
+                - "'missing required arguments: a_str' in ansible_failed_result.argument_errors"
+                - ansible_failed_result.validate_args_context.argument_spec_name == "main"
+                - ansible_failed_result.validate_args_context.name == "role_with_no_tasks"
+                - ansible_failed_result.validate_args_context.type == "role"
+                - "ansible_failed_result.validate_args_context.path is search('roles_arg_spec/roles/role_with_no_tasks')"

test/integration/targets/roles_arg_spec/test_complex_role_fails.yml

@@ -0,0 +1,150 @@
+---
+- name: "Running include_role test1"
+  hosts: localhost
+  gather_facts: false
+  vars:
+      unicode_type_match: "<type 'unicode'>"
+      string_type_match: "<type 'str'>"
+      float_type_match: "<type 'float'>"
+      unicode_class_match: "<class 'unicode'>"
+      string_class_match: "<class 'str'>"
+      bytes_class_match: "<class 'bytes'>"
+      float_class_match: "<class 'float'>"
+      expected:
+          test1_1:
+              argument_errors: [
+                  "argument 'tidy_expected' is of type <class 'ansible.parsing.yaml.objects.AnsibleMapping'> and we were unable to convert to list: <class 'ansible.parsing.yaml.objects.AnsibleMapping'> cannot be converted to a list",
+                  "argument 'bust_some_stuff' is of type <class 'str'> and we were unable to convert to int: <class 'str'> cannot be converted to an int",
+                  "argument 'some_list' is of type <class 'ansible.parsing.yaml.objects.AnsibleMapping'> and we were unable to convert to list: <class 'ansible.parsing.yaml.objects.AnsibleMapping'> cannot be converted to a list",
+                  "argument 'some_dict' is of type <class 'ansible.parsing.yaml.objects.AnsibleSequence'> and we were unable to convert to dict: <class 'ansible.parsing.yaml.objects.AnsibleSequence'> cannot be converted to a dict",
+                  "argument 'some_int' is of type <class 'float'> and we were unable to convert to int: <class 'float'> cannot be converted to an int",
+                  "argument 'some_float' is of type <class 'str'> and we were unable to convert to float: <class 'str'> cannot be converted to a float",
+                  "argument 'some_bytes' is of type <class 'bytes'> and we were unable to convert to bytes: <class 'bytes'> cannot be converted to a Byte value",
+                  "argument 'some_bits' is of type <class 'str'> and we were unable to convert to bits: <class 'str'> cannot be converted to a Bit value",
+                  "value of test1_choices must be one of: this paddle game, the astray, this remote control, the chair, got: My dog",
+                  "value of some_choices must be one of: choice1, choice2, got: choice4",
+                  "argument 'some_second_level' is of type <class 'ansible.parsing.yaml.objects.AnsibleUnicode'> found in 'some_dict_options'. and we were unable to convert to bool: The value 'not-a-bool' is not a valid boolean.  ",
+                  "argument 'third_level' is of type <class 'ansible.parsing.yaml.objects.AnsibleUnicode'> found in 'multi_level_option -> second_level'. and we were unable to convert to int: <class 'ansible.parsing.yaml.objects.AnsibleUnicode'> cannot be converted to an int"
+                  ]
+
+  tasks:
+      # This test play requires jinja >= 2.7
+      - name: get the jinja2 version
+        shell: python -c 'import jinja2; print(jinja2.__version__)'
+        register: jinja2_version
+        delegate_to: localhost
+        changed_when: false
+
+      - debug:
+          msg: "Jinja version: {{ jinja2_version.stdout }}"
+
+      - name: include_role test1 since it has a arg_spec.yml
+        block:
+            - include_role:
+                name: test1
+              vars:
+                tidy_expected:
+                    some_key: some_value
+                test1_var1: 37.4
+                test1_choices: "My dog"
+                bust_some_stuff: "some_string_that_is_not_an_int"
+                some_choices: "choice4"
+                some_str: 37.5
+                some_list: {'a': false}
+                some_dict:
+                    - "foo"
+                    - "bar"
+                some_int: 37.
+                some_float: "notafloatisit"
+                some_path: "anything_is_a_valid_path"
+                some_raw: {"anything_can_be": "a_raw_type"}
+                # not sure what would be an invalid jsonarg
+                # some_jsonarg: "not sure what this does yet"
+                some_json: |
+                    '{[1, 3, 3] 345345|45v<#!}'
+                some_jsonarg: |
+                    {"foo": [1, 3, 3]}
+                # not sure we can load binary in safe_load
+                some_bytes: !!binary |
+                    R0lGODlhDAAMAIQAAP//9/X17unp5WZmZgAAAOfn515eXvPz7Y6OjuDg4J+fn5
+                    OTk6enp56enmlpaWNjY6Ojo4SEhP/++f/++f/++f/++f/++f/++f/++f/++f/+
+                    +f/++f/++f/++f/++f/++SH+Dk1hZGUgd2l0aCBHSU1QACwAAAAADAAMAAAFLC
+                    AgjoEwnuNAFOhpEMTRiggcz4BNJHrv/zCFcLiwMWYNG84BwwEeECcgggoBADs=
+                some_bits: "foo"
+                # some_str_nicknames: []
+                # some_str_akas: {}
+                some_str_removed_in: "foo"
+                some_dict_options:
+                  some_second_level: "not-a-bool"
+                multi_level_option:
+                  second_level:
+                    third_level: "should_be_int"
+
+            - fail:
+                msg: "Should not get here"
+
+        rescue:
+            - debug:
+                var: ansible_failed_result
+
+            - name: replace py version specific types with generic names so tests work on py2 and py3
+              set_fact:
+                  # We want to compare if the actual failure messages and the expected failure messages
+                  # are the same. But to compare and do set differences, we have to handle some
+                  # differences between py2/py3.
+                  # The validation failure messages include python type and class reprs, which are
+                  # different between py2 and py3. For ex, "<type 'str'>" vs "<class 'str'>". Plus
+                  # the usual py2/py3 unicode/str/bytes type shenanigans. The 'THE_FLOAT_REPR' is
+                  # because py3 quotes the value in the error while py2 does not, so we just ignore
+                  # the rest of the line.
+                  actual_generic: "{{ ansible_failed_result.argument_errors|
+                                      map('replace', unicode_type_match, 'STR')|
+                                      map('replace', string_type_match, 'STR')|
+                                      map('replace', float_type_match, 'FLOAT')|
+                                      map('replace', unicode_class_match, 'STR')|
+                                      map('replace', string_class_match, 'STR')|
+                                      map('replace', bytes_class_match, 'STR')|
+                                      map('replace', float_class_match, 'FLOAT')|
+                                      map('regex_replace', '''float:.*$''', 'THE_FLOAT_REPR')|
+                                      map('regex_replace', 'Valid booleans include.*$', '')|
+                                      list }}"
+                  expected_generic: "{{ expected.test1_1.argument_errors|
+                                        map('replace', unicode_type_match, 'STR')|
+                                        map('replace', string_type_match, 'STR')|
+                                        map('replace', float_type_match, 'FLOAT')|
+                                        map('replace', unicode_class_match, 'STR')|
+                                        map('replace', string_class_match, 'STR')|
+                                        map('replace', bytes_class_match, 'STR')|
+                                        map('replace', float_class_match, 'FLOAT')|
+                                        map('regex_replace', '''float:.*$''', 'THE_FLOAT_REPR')|
+                                        map('regex_replace', 'Valid booleans include.*$', '')|
+                                        list }}"
+
+            - name: figure out the difference between expected and actual validate_argument_spec failures
+              set_fact:
+                  actual_not_in_expected: "{{ actual_generic| difference(expected_generic) | sort() }}"
+                  expected_not_in_actual: "{{ expected_generic | difference(actual_generic) | sort() }}"
+
+            - name: assert that all actual validate_argument_spec failures were in expected
+              assert:
+                  that:
+                      - actual_not_in_expected | length == 0
+                  msg: "Actual validate_argument_spec failures that were not expected: {{ actual_not_in_expected }}"
+
+            - name: assert that all expected validate_argument_spec failures were in expected
+              assert:
+                  that:
+                      - expected_not_in_actual | length == 0
+                  msg: "Expected validate_argument_spec failures that were not in actual results: {{ expected_not_in_actual }}"
+
+            - name: assert that `validate_args_context` return value has what we expect
+              assert:
+                that:
+                  - ansible_failed_result.validate_args_context.argument_spec_name == "main"
+                  - ansible_failed_result.validate_args_context.name == "test1"
+                  - ansible_failed_result.validate_args_context.type == "role"
+                  - "ansible_failed_result.validate_args_context.path is search('roles_arg_spec/roles/test1')"
+
+        # skip this task if jinja isnt >= 2.7, aka centos6
+        when:
+            - jinja2_version.stdout is version('2.7', '>=')

test/integration/targets/roles_arg_spec/test_play_level_role_fails.yml

@@ -0,0 +1,5 @@
+---
+- hosts: localhost
+  gather_facts: false
+  roles:
+      - { role: a, invalid_str: "roles" }