=========================== Porting Modules to Python 3 =========================== Ansible modules are not the usual Python-3 porting exercise. There are two factors that make it harder to port them than most code: 1. Many modules need to run on Python-2.4 in addition to Python-3. 2. A lot of mocking has to go into unittesting a Python-3 module. So it's harder to test that your porting has fixed everything or to make sure that later commits haven't regressed. Which version of Python-3.x and which version of Python-2.x are our minimums? ============================================================================= The short answer is Python-3.4 and Python-2.4 but please read on for more information. For Python-3 we are currently using Python-3.4 as a minimum. However, no long term supported Linux distributions currently ship with Python-3. When that occurs, we will probably take that as our minimum Python-3 version rather than Python-3.4. Thus far, Python-3 has been adding small changes that make it more compatible with Python-2 in its newer versions (For instance, Python-3.5 added the ability to use percent-formatted byte strings.) so it should be more pleasant to use a newer version of Python-3 if it's available. At some point this will change but we'll just have to cross that bridge when we get to it. For Python-2 the default is for modules to run on Python-2.4. This allows users with older distributions that are stuck on Python-2.4 to manage their machines. Modules are allowed to drop support for Python-2.4 when one of their dependent libraries require a higher version of python. This is not an invitation to add unnecessary dependent libraries in order to force your module to be usable only with a newer version of Python. Instead it is an acknowledgment that some libraries (for instance, boto3 and docker-py) will only function with newer Python. .. note:: When will we drop support for Python-2.4? The only long term supported distro that we know of with Python-2.4 is RHEL5 (and its rebuilds like CentOS5) which is supported until April of 2017. We will likely end our support for Python-2.4 in modules in an Ansible release around that time. We know of no long term supported distributions with Python-2.5 so the new minimum Python-2 version will likely be Python-2.6. This will let us take advantage of the forwards-compat features of Python-2.6 so porting and maintainance of Python-2/Python-3 code will be easier after that. Supporting only Python-2 or only Python-3 ========================================= Sometimes a module's dependent libraries only run on Python-2 or only run on Python-3. We do not yet have a strategy for these modules but we'll need to come up with one. I see three possibilities: 1. We treat these libraries like any other libraries that may not be installed on the system. When we import them we check if the import was successful. If so, then we continue. If not we return an error about the library being missing. Users will have to find out that the library is unavailable on their version of Python either by searching for the library on their own or reading the requirements section in :command:`ansible-doc`. 2. The shebang line is the only metadata that Ansible extracts from a module so we may end up using that to specify what we mean. Something like ``#!/usr/bin/python`` means the module will run on both Python-2 and Python-3, ``#!/usr/bin/python2`` means the module will only run on Python-2, and ``#!/usr/bin/python3`` means the module will only run on Python-3. Ansible's code will need to be modified to accommodate this. For :command:`python2`, if ``ansible_python2_interpreter`` is not set, it will have to fallback to `` ansible_python_interpreter`` and if that's not set, fallback to ``/usr/bin/python``. For :command:`python3`, Ansible will have to first try ``ansible_python3_interpreter`` and then fallback to ``/usr/bin/python3`` as normal. 3. We add a way for Ansible to retrieve metadata about modules. The metadata will include the version of Python that is required. Methods 2 and 3 will both require that we modify modules or otherwise add this additional information somewhere. 2 needs only a little code changes in executor/module_common.py to parse. 3 will require a lot of work. This is probably not worthwhile if this is the only change but could be worthwhile if there's other things as well. 1 requires that we port all modules to work with python3 syntax but only the code path to get to the library import being attempted and then a fail_json() being called because the libraries are unavailable needs to actually work. Tips, tricks, and idioms to adopt ================================= Exceptions ---------- In code which already needs Python-2.6+ (For instance, because a library it depends on only runs on Python >= 2.6) it is okay to port directly to the new exception catching syntax:: try: a = 2/0 except ValueError as e: module.fail_json(msg="Tried to divide by zero!") For modules which also run on Python-2.4, we have to use an uglier construction to make this work under both Python-2.4 and Python-3:: from ansible.module_utils.pycompat24 import get_exception [...] try: a = 2/0 except ValueError: e = get_exception() module.fail_json(msg="Tried to divide by zero!") Octal numbers ------------- In Python-2.4, octal literals are specified as ``0755``. In Python-3, that is invalid and octals must be specified as ``0o755``. To bridge this gap, modules should create their octals like this:: # Can't use 0755 on Python-3 and can't use 0o755 on Python-2.4 EXECUTABLE_PERMS = int('0755', 8) Bundled six ----------- The third-party python-six library exists to help projects create code that runs on both Python-2 and Python-3. Ansible includes version 1.4.1 in module_utils so that other modules can use it without requiring that it is installed on the remote system. To make use of it, import it like this:: from ansible.module_utils import six .. note:: Why version 1.4.1? six-1.4.1 is the last version of python-six to support Python-2.4. As long as Ansible modules need to run on Python-2.4 we won't be able to update the bundled copy of six. Compile Test ------------ We have travis compiling all modules with various versions of Python to check that the modules conform to the syntax at those versions. When you've ported a module so that its syntax works with Python-3, we need to modify .travis.yml so that the module is included in the syntax check. Here's the relevant section of .travis.yml:: script: [...] - python3.4 -m compileall -fq system/ping.py - python3.5 -m compileall -fq system/ping.py At the moment this is a whitelist. Just add your newly ported module to that line. Eventually, not compiling on Python-3 will be the exception. When that occurs, we will move to a blacklist for listing which modules do not compile under Python-3.