Change interpreter discovery defaults to silently prefer Python 3 (#75049)

Push /usr/bin/python to almost the bottom of the barrel. This makes the strategy to prefer
specific versions of the "mystery meat" version.

* Change INTERPRETER_PYTHON default to 'auto'
    Change description to match change in behavior.
    Change deprecation message to a warning.

* Update docs
* Add porting guide entry
* Update unit tests
* Update integration test
* Allow INTERPRETER_PYTHON_FALLBACK to be configure using a variable
* Prefer platform-python above other Python 2 interpreters
* Add Python 3.10 to the list of interpreters

changelogs/fragments/interepreter-discovery-prefer-python3.yml

@@ -0,0 +1,3 @@
+minor_changes:
+  - interpreter discovery - prefer Python 3 over Python 2
+  - interpreter discovery - allow the default list of ``INTERPRETER_PYTHON_FALLBACK`` to be changed using a variable

docs/docsite/rst/porting_guides/porting_guide_core_2.12.rst

@@ -21,6 +21,15 @@ Playbook
 
 No notable changes
 
+Python Interpreter Discovery
+============================
+
+The default value of ``INTERPRETER_PYTHON_FALLBACK`` changed to ``auto``. The list of Python interpreters in ``INTERPRETER_PYTHON_FALLBACK`` changed to prefer Python 3 over Python 2. The combination of these two changes means the new default behavior is to quietly prefer Python 3 over Python 2 on remote hosts. Previously a deprecation warning was issued in situations where interpreter discovery would have used Python 3 but the interpreter was set to ``/usr/bin/python``.
+
+``INTERPRETER_PYTHON_FALLBACK`` can be changed from the default list of interpreters by setting the ``ansible_interpreter_python_fallback`` variable.
+
+See :ref:`interpreter discovery documentation <interpreter_discovery>` for more details.
+
 
 Command Line
 ============

docs/docsite/rst/reference_appendices/interpreter_discovery.rst

@@ -15,7 +15,7 @@ To control the discovery behavior:
 
 Use one of the following values:
 
-auto_legacy : (default in 2.8)
+auto_legacy :
   Detects the target OS platform, distribution, and version, then consults a
   table listing the correct Python interpreter and path for each
   platform/distribution/version. If an entry is found, and ``/usr/bin/python`` is absent, uses the discovered interpreter (and path). If an entry
@@ -30,7 +30,7 @@ auto_legacy : (default in 2.8)
   paths and uses the first one found; also issues a warning that future
   installation of another Python interpreter could alter the one chosen.
 
-auto : (future default in 2.12)
+auto : (default in 2.12)
   Detects the target OS platform, distribution, and version, then consults a
   table listing the correct Python interpreter and path for each
   platform/distribution/version. If an entry is found, uses the discovered

lib/ansible/config/base.yml

@@ -422,7 +422,7 @@ DEFAULT_ALLOW_UNSAFE_LOOKUPS:
     - "When enabled, this option allows lookup plugins (whether used in variables as ``{{lookup('foo')}}`` or as a loop as with_foo)
       to return data that is not marked 'unsafe'."
     - By default, such data is marked as unsafe to prevent the templating engine from evaluating any jinja2 templating language,
-      as this could represent a security risk.  This option is provided to allow for backwards-compatibility,
+      as this could represent a security risk. This option is provided to allow for backward compatibility,
       however users should first consider adding allow_unsafe=True to any lookups which may be expected to contain data which may be run
       through the templating engine late
   env: []
@@ -1433,7 +1433,7 @@ HOST_PATTERN_MISMATCH:
   version_added: "2.8"
 INTERPRETER_PYTHON:
   name: Python interpreter path (or automatic discovery behavior) used for module execution
-  default: auto_legacy
+  default: auto
   env: [{name: ANSIBLE_PYTHON_INTERPRETER}]
   ini:
   - {key: interpreter_python, section: defaults}
@@ -1442,14 +1442,13 @@ INTERPRETER_PYTHON:
   version_added: "2.8"
   description:
   - Path to the Python interpreter to be used for module execution on remote targets, or an automatic discovery mode.
-    Supported discovery modes are ``auto``, ``auto_silent``, and ``auto_legacy`` (the default). All discovery modes
-    employ a lookup table to use the included system Python (on distributions known to include one), falling back to a
-    fixed ordered list of well-known Python interpreter locations if a platform-specific default is not available. The
-    fallback behavior will issue a warning that the interpreter should be set explicitly (since interpreters installed
-    later may change which one is used). This warning behavior can be disabled by setting ``auto_silent``. The default
-    value of ``auto_legacy`` provides all the same behavior, but for backwards-compatibility with older Ansible releases
-    that always defaulted to ``/usr/bin/python``, will use that interpreter if present (and issue a warning that the
-    default behavior will change to that of ``auto`` in a future Ansible release.
+    Supported discovery modes are ``auto`` (the default), ``auto_silent``, ``auto_legacy``, and ``auto_legacy_silent``.
+    All discovery modes employ a lookup table to use the included system Python (on distributions known to include one),
+    falling back to a fixed ordered list of well-known Python interpreter locations if a platform-specific default is not
+    available. The fallback behavior will issue a warning that the interpreter should be set explicitly (since interpreters
+    installed later may change which one is used). This warning behavior can be disabled by setting ``auto_silent`` or
+    ``auto_legacy_silent``. The value of ``auto_legacy`` provides all the same behavior, but for backwards-compatibility
+    with older Ansible releases that always defaulted to ``/usr/bin/python``, will use that interpreter if present.
 INTERPRETER_PYTHON_DISTRO_MAP:
   name: Mapping of known included platform pythons for various Linux distros
   default:
@@ -1474,18 +1473,21 @@ INTERPRETER_PYTHON_DISTRO_MAP:
 INTERPRETER_PYTHON_FALLBACK:
   name: Ordered list of Python interpreters to check for in discovery
   default:
-  - /usr/bin/python
+  - python3.10
   - python3.9
   - python3.8
   - python3.7
   - python3.6
   - python3.5
+  - /usr/bin/python3
+  - /usr/libexec/platform-python
   - python2.7
   - python2.6
-  - /usr/libexec/platform-python
-  - /usr/bin/python3
+  - /usr/bin/python
   - python
-  # FUTURE: add inventory override once we're sure it can't be abused by a rogue target
+  vars:
+    - name: ansible_interpreter_python_fallback
+  type: list
   version_added: "2.8"
 TRANSFORM_INVALID_GROUP_CHARS:
   name: Transform invalid characters in group names

lib/ansible/executor/interpreter_discovery.py

@@ -116,16 +116,13 @@ def discover_interpreter(action, interpreter_name, discovery_mode, task_vars):
         # provide a transition period for hosts that were using /usr/bin/python previously (but shouldn't have been)
         if is_auto_legacy:
             if platform_interpreter != u'/usr/bin/python' and u'/usr/bin/python' in found_interpreters:
-                # FIXME: support comments in sivel's deprecation scanner so we can get reminded on this
                 if not is_silent:
-                    action._discovery_deprecation_warnings.append(dict(
-                        msg=u"Distribution {0} {1} on host {2} should use {3}, but is using "
-                            u"/usr/bin/python for backward compatibility with prior Ansible releases. "
-                            u"A future Ansible release will default to using the discovered platform "
-                            u"python for this host. See {4} for more information"
-                            .format(distro, version, host, platform_interpreter,
-                                    get_versioned_doclink('reference_appendices/interpreter_discovery.html')),
-                        version='2.12'))
+                    action._discovery_warnings.append(
+                        u"Distribution {0} {1} on host {2} should use {3}, but is using "
+                        u"/usr/bin/python for backward compatibility with prior Ansible releases. "
+                        u"See {4} for more information"
+                        .format(distro, version, host, platform_interpreter,
+                                get_versioned_doclink('reference_appendices/interpreter_discovery.html')))
                 return u'/usr/bin/python'
 
         if platform_interpreter not in found_interpreters:

test/integration/targets/interpreter_discovery_python/tasks/main.yml

@@ -76,10 +76,10 @@
     ping:
     register: legacy
 
-  - name: check for dep warning (only on platforms where auto result is not /usr/bin/python and legacy is)
+  - name: check for warning (only on platforms where auto result is not /usr/bin/python and legacy is)
     assert:
       that:
-      - legacy.deprecations | default([]) | length > 0
+      - legacy.warnings | default([]) | length > 0
     # only check for a dep warning if legacy returned /usr/bin/python and auto didn't
     when: legacy.ansible_facts.discovered_interpreter_python == '/usr/bin/python' and
           auto_out.ansible_facts.discovered_interpreter_python != '/usr/bin/python'

test/units/executor/test_interpreter_discovery.py

@@ -29,10 +29,9 @@ def test_discovery_interpreter_linux_auto_legacy():
 
     assert actual == u'/usr/bin/python'
     assert len(mock_action.method_calls) == 3
-    assert mock_action.method_calls[2][0] == '_discovery_deprecation_warnings.append'
+    assert mock_action.method_calls[2][0] == '_discovery_warnings.append'
     assert u'Distribution Ubuntu 16.04 on host host-fóöbär should use /usr/bin/python3, but is using /usr/bin/python' \
-           u' for backward compatibility' in mock_action.method_calls[2][1][0]['msg']
-    assert mock_action.method_calls[2][1][0]['version'] == '2.12'
+           u' for backward compatibility' in mock_action.method_calls[2][1][0]
 
 
 def test_discovery_interpreter_linux_auto_legacy_silent():