validate-modules: don't fail on invalid YAML (#79682)

* validate-modules: don't fail on invalid YAML

When validate-modules encounters invalid YAML (e.g. in the EXAMPLES
section), it tries to reformat the exception to include the line number
in the Python file instead of the line number of the embedded YAML
document. However, PyYAML doesn't allow modification of the Mark object
(anymore) which leads to a new exception being raised, instead of
reporting the original exception.

As the original exception is not needed in other places anymore, we
don't have to modify it at all and can just compute the right line
number when reporting the error via ansible-test.

Fixes: #75837

* Add test for invalid module doc YAML syntax.

Co-authored-by: Matt Clay <matt@mystile.com>

changelogs/fragments/75837-validate-modules-invalid-yaml.yml

@@ -0,0 +1,2 @@
+bugfixes:
+  - ansible-test sanity - correctly report invalid YAML in validate-modules (https://github.com/ansible/ansible/issues/75837).

test/integration/targets/ansible-test-sanity-validate-modules/ansible_collections/ns/col/plugins/modules/invalid_yaml_syntax.py

@@ -0,0 +1,27 @@
+#!/usr/bin/python
+# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
+
+from __future__ import absolute_import, division, print_function
+__metaclass__ = type
+
+DOCUMENTATION = '''
+- key: "value"wrong
+'''
+
+EXAMPLES = '''
+- key: "value"wrong
+'''
+
+RETURN = '''
+- key: "value"wrong
+'''
+
+from ansible.module_utils.basic import AnsibleModule
+
+
+def main():
+    AnsibleModule(argument_spec=dict())
+
+
+if __name__ == '__main__':
+    main()

test/integration/targets/ansible-test-sanity-validate-modules/expected.txt

@@ -0,0 +1,5 @@
+plugins/modules/invalid_yaml_syntax.py:0:0: deprecation-mismatch: "meta/runtime.yml" and DOCUMENTATION.deprecation do not agree.
+plugins/modules/invalid_yaml_syntax.py:0:0: missing-documentation: No DOCUMENTATION provided
+plugins/modules/invalid_yaml_syntax.py:8:15: documentation-syntax-error: DOCUMENTATION is not valid YAML
+plugins/modules/invalid_yaml_syntax.py:12:15: invalid-examples: EXAMPLES is not valid YAML
+plugins/modules/invalid_yaml_syntax.py:16:15: return-syntax-error: RETURN is not valid YAML

test/integration/targets/ansible-test-sanity-validate-modules/runme.sh

@@ -4,7 +4,9 @@ source ../collection/setup.sh
 
 set -eux
 
-ansible-test sanity --test validate-modules --color --truncate 0 "${@}"
+ansible-test sanity --test validate-modules --color --truncate 0 --failure-ok --lint "${@}" 1> actual-stdout.txt 2> actual-stderr.txt
+diff -u "${TEST_DIR}/expected.txt" actual-stdout.txt
+grep -f "${TEST_DIR}/expected.txt" actual-stderr.txt
 
 cd ../ps_only
 

test/lib/ansible_test/_util/controller/sanity/validate-modules/validate_modules/utils.py

@@ -154,11 +154,9 @@ def parse_yaml(value, lineno, module, name, load_all=False, ansible_loader=False
         if load_all:
             data = list(data)
     except yaml.MarkedYAMLError as e:
-        e.problem_mark.line += lineno - 1
-        e.problem_mark.name = '%s.%s' % (module, name)
         errors.append({
             'msg': '%s is not valid YAML' % name,
-            'line': e.problem_mark.line + 1,
+            'line': e.problem_mark.line + lineno,
             'column': e.problem_mark.column + 1
         })
         traces.append(e)