@ -160,7 +160,6 @@ Ansible conventions offer a predictable user interface across all modules, playb
* Implement declarative operations (not CRUD) so the user can ignore existing state and focus on final state. For example, use ``started/stopped``, ``present/absent``.
* Implement declarative operations (not CRUD) so the user can ignore existing state and focus on final state. For example, use ``started/stopped``, ``present/absent``.
* Strive for a consistent final state (aka idempotency). If running your module twice in a row against the same system would result in two different states, see if you can redesign or rewrite to achieve consistent final state. If you can't, document the behavior and the reasons for it.
* Strive for a consistent final state (aka idempotency). If running your module twice in a row against the same system would result in two different states, see if you can redesign or rewrite to achieve consistent final state. If you can't, document the behavior and the reasons for it.
* Provide consistent return values within the standard Ansible return structure, even if NA/None are used for keys normally returned under other options.
* Provide consistent return values within the standard Ansible return structure, even if NA/None are used for keys normally returned under other options.
* Follow additional guidelines that apply to families of modules if applicable. For example, AWS modules should follow the :ref:`Amazon development checklist <AWS_module_development>`.
The Ansible AWS collection (on `Galaxy <https://galaxy.ansible.com/community/aws>`_, source code `repository <https://github.com/ansible-collections/community.aws>`_) is maintained by the Ansible AWS Working Group. For further information see the `AWS working group community page <https://github.com/ansible/community/wiki/aws>`_. If you are planning to contribute AWS modules to Ansible then getting in touch with the working group is a good way to start, especially because a similar module may already be under development.
This guide has moved to :ref:`ansible_collections.amazon.aws.docsite.dev_guide_intro`.
..contents::
:local:
Requirements
============
Python Compatibility
--------------------
AWS content in Ansible 2.9 and 1.x collection releases supported Python 2.7 and newer.
Starting with the 2.0 releases of both collections, Python 2.7 support will be ended in accordance with AWS' `end of Python 2.7 support <https://aws.amazon.com/blogs/developer/announcing-end-of-support-for-python-2-7-in-aws-sdk-for-python-and-aws-cli-v1/>`_. Contributions to both collections that target the 2.0 or later collection releases can be written to support Python 3.6+ syntax.
SDK Version Support
-------------------
Starting with the 2.0 releases of both collections, it is generally the policy to support the versions of botocore and boto3 that were released 12 months prior to the most recent major collection release, following semantic versioning (for example, 2.0.0, 3.0.0).
Features and functionality that require newer versions of the SDK can be contributed provided they are noted in the module documentation:
..code-block:: yaml
DOCUMENTATION = '''
---
module: ec2_vol
options:
throughput:
description:
- Volume throughput in MB/s.
- This parameter is only valid for gp3 volumes.
- Valid range is from 125 to 1000.
- Requires at least botocore version 1.19.27.
type: int
version_added: 1.4.0
And handled using the ``botocore_at_least`` helper method:
..code-block:: python
if module.params.get('throughput'):
if not module.botocore_at_least("1.19.27"):
module.fail_json(msg="botocore >= 1.19.27 is required to set the throughput for a volume")
Maintaining existing modules
============================
Fixing bugs
-----------
Bug fixes to code that relies on boto will still be accepted. When possible,
the code should be ported to use boto3.
Adding new features
-------------------
Try to keep backward compatibility with relatively recent versions of boto3. That means that if you
want to implement some functionality that uses a new feature of boto3, it should only fail if that
feature actually needs to be run, with a message stating the missing feature and minimum required
version of boto3.
Use feature testing (for example, ``hasattr('boto3.module', 'shiny_new_method')``) to check whether boto3
supports a feature rather than version checking. For example, from the ``ec2`` module:
module.fail_json(msg="instance_profile_name parameter requires boto version 2.5.0 or higher")
Migrating to boto3
------------------
Prior to Ansible 2.0, modules were written in either boto3 or boto. We are
still porting some modules to boto3. Modules that still require boto should be ported to use boto3 rather than using both libraries (boto and boto3). We would like to remove the boto dependency from all modules.
Porting code to AnsibleAWSModule
---------------------------------
Some old AWS modules use the generic ``AnsibleModule`` as a base rather than the more efficient ``AnsibleAWSModule``. To port an old module to ``AnsibleAWSModule``, change:
..code-block:: python
from ansible.module_utils.basic import AnsibleModule
...
module = AnsibleModule(...)
to:
..code-block:: python
from ansible_collections.amazon.aws.plugins.module_utils.core import AnsibleAWSModule
...
module = AnsibleAWSModule(...)
Few other changes are required. AnsibleAWSModule
does not inherit methods from AnsibleModule by default, but most useful methods
are included. If you do find an issue, please raise a bug report.
When porting, keep in mind that AnsibleAWSModule also will add the default ec2
argument spec by default. In pre-port modules, you should see common arguments
There are two :ref:`common documentation fragments <module_docs_fragments>`
that should be included into almost all AWS modules:
* ``aws`` - contains the common boto connection parameters
* ``ec2`` - contains the common region parameter required for many AWS modules
These fragments should be used rather than re-documenting these properties to ensure consistency
and that the more esoteric connection options are documented. For example:
..code-block:: python
DOCUMENTATION = '''
module: my_module
# some lines omitted here
requirements: [ 'botocore', 'boto3' ]
extends_documentation_fragment:
- amazon.aws.aws
- amazon.aws.ec2
'''
Handling exceptions
===================
You should wrap any boto3 or botocore call in a try block. If an exception is thrown, then there
are a number of possibilities for handling it.
* Catch the general ``ClientError`` or look for a specific error code with
``is_boto3_error_code``.
* Use ``aws_module.fail_json_aws()`` to report the module failure in a standard way
* Retry using AWSRetry
* Use ``fail_json()`` to report the failure without using ``ansible_collections.amazon.aws.plugins.module_utils.core``
* Do something custom in the case where you know how to handle the exception
For more information on botocore exception handling see the `botocore error documentation <https://botocore.readthedocs.io/en/latest/client_upgrades.html#error-handling>`_.
Using is_boto3_error_code
-------------------------
To use ``ansible_collections.amazon.aws.plugins.module_utils.core.is_boto3_error_code`` to catch a single
AWS error code, call it in place of ``ClientError`` in your except clauses. In
this case, *only* the ``InvalidGroup.NotFound`` error code will be caught here,
and any other error will be raised for handling elsewhere in the program.
..code-block:: python
try:
info = connection.describe_security_groups(**kwargs)
# Compare the user submitted policy to the current policy ignoring order
if compare_policies(user_policy, current_policy):
# Update the policy
aws_object.set_policy(user_policy)
else:
# Nothing to do
pass
Dealing with tags
=================
AWS has a concept of resource tags. Usually the boto3 API has separate calls for tagging and
untagging a resource. For example, the ec2 API has a create_tags and delete_tags call.
It is common practice in Ansible AWS modules to have a `purge_tags` parameter that defaults to
true.
The `purge_tags` parameter means that existing tags will be deleted if they are not specified by
the Ansible task.
There is a helper function `compare_aws_tags` to ease dealing with tags. It can compare two dicts
and return the tags to set and the tags to delete. See the Helper function section below for more
detail.
Helper functions
================
Along with the connection functions in Ansible ec2.py module_utils, there are some other useful
functions detailed below.
camel_dict_to_snake_dict
------------------------
boto3 returns results in a dict. The keys of the dict are in CamelCase format. In keeping with
Ansible format, this function will convert the keys to snake_case.
``camel_dict_to_snake_dict`` takes an optional parameter called ``ignore_list`` which is a list of
keys not to convert (this is usually useful for the ``tags`` dict, whose child keys should remain with
case preserved)
Another optional parameter is ``reversible``. By default, ``HTTPEndpoint`` is converted to ``http_endpoint``,
which would then be converted by ``snake_dict_to_camel_dict`` to ``HttpEndpoint``.
Passing ``reversible=True`` converts HTTPEndpoint to ``h_t_t_p_endpoint`` which converts back to ``HTTPEndpoint``.
snake_dict_to_camel_dict
------------------------
`snake_dict_to_camel_dict` converts snake cased keys to camel case. By default, because it was
first introduced for ECS purposes, this converts to dromedaryCase. An optional
parameter called `capitalize_first`, which defaults to `False`, can be used to convert to CamelCase.
ansible_dict_to_boto3_filter_list
---------------------------------
Converts a an Ansible list of filters to a boto3 friendly list of dicts. This is useful for any
boto3 `_facts` modules.
boto_exception
--------------
Pass an exception returned from boto or boto3, and this function will consistently get the message from the exception.
Deprecated: use `AnsibleAWSModule`'s `fail_json_aws` instead.
boto3_tag_list_to_ansible_dict
------------------------------
Converts a boto3 tag list to an Ansible dict. Boto3 returns tags as a list of dicts containing keys
called 'Key' and 'Value' by default. This key names can be overridden when calling the function.
For example, if you have already camel_cased your list of tags you may want to pass lowercase key
names instead, in other words, 'key' and 'value'.
This function converts the list in to a single dict where the dict key is the tag key and the dict
value is the tag value.
ansible_dict_to_boto3_tag_list
------------------------------
Opposite of above. Converts an Ansible dict to a boto3 tag list of dicts. You can again override
the key names used if 'Key' and 'Value' is not suitable.
get_ec2_security_group_ids_from_names
-------------------------------------
Pass this function a list of security group names or combination of security group names and IDs
and this function will return a list of IDs. You should also pass the VPC ID if known because
security group names are not necessarily unique across VPCs.
compare_policies
----------------
Pass two dicts of policies to check if there are any meaningful differences and returns true
if there are. This recursively sorts the dicts and makes them hashable before comparison.
This method should be used any time policies are being compared so that a change in order
doesn't result in unnecessary changes.
compare_aws_tags
----------------
Pass two dicts of tags and an optional purge parameter and this function will return a dict
containing key pairs you need to modify and a list of tag key names that you need to remove. Purge
is True by default. If purge is False then any existing tags will not be modified.
This function is useful when using boto3 'add_tags' and 'remove_tags' functions. Be sure to use the
other helper function `boto3_tag_list_to_ansible_dict` to get an appropriate tag dict before
calling this function. Since the AWS APIs are not uniform (for example, EC2 is different from Lambda) this will work
without modification for some (Lambda) and others may need modification before using these values
(such as EC2, with requires the tags to unset to be in the form `[{'Key': key1}, {'Key': key2}]`).
Integration Tests for AWS Modules
=================================
All new AWS modules should include integration tests to ensure that any changes in AWS APIs that
affect the module are detected. At a minimum this should cover the key API calls and check the
documented return values are present in the module result.
For general information on running the integration tests see the :ref:`Integration Tests page of the
Module Development Guide <testing_integration>`, especially the section on configuration for cloud tests.
The integration tests for your module should be added in `test/integration/targets/MODULE_NAME`.
You must also have a aliases file in `test/integration/targets/MODULE_NAME/aliases`. This file serves
two purposes. First indicates it's in an AWS test causing the test framework to make AWS credentials
available during the test run. Second putting the test in a test group causing it to be run in the
continuous integration build.
Tests for new modules should be added to the same group as existing AWS tests. In general just copy
an existing aliases file such as the `aws_s3 tests aliases file <https://github.com/ansible-collections/amazon.aws/blob/master/tests/integration/targets/aws_s3/aliases>`_.
AWS Credentials for Integration Tests
-------------------------------------
The testing framework handles running the test with appropriate AWS credentials, these are made available
to your test in the following variables:
* `aws_region`
* `aws_access_key`
* `aws_secret_key`
* `security_token`
So all invocations of AWS modules in the test should set these parameters. To avoid duplicating these
for every call, it's preferable to use :ref:`module_defaults <module_defaults>`. For example:
..code-block:: yaml
- name: set connection information for aws modules and run tasks
As explained in the :ref:`Integration Test guide <testing_integration>`
there are defined IAM policies in `mattclay/aws-terminator <https://github.com/mattclay/aws-terminator>`_ that contain the necessary permissions
to run the AWS integration test.
If your module interacts with a new service or otherwise requires new permissions, tests will fail when you submit a pull request and the
`Ansibullbot <https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md>`_ will tag your PR as needing revision.
We do not automatically grant additional permissions to the roles used by the continuous integration builds.
You will need to raise a Pull Request against `mattclay/aws-terminator <https://github.com/mattclay/aws-terminator>`_ to add them.
If your PR has test failures, check carefully to be certain the failure is only due to the missing permissions. If you've ruled out other sources of failure, add a comment with the `ready_for_review`
tag and explain that it's due to missing permissions.
Your pull request cannot be merged until the tests are passing. If your pull request is failing due to missing permissions,
you must collect the minimum IAM permissions required to
run the tests.
There are two ways to figure out which IAM permissions you need for your PR to pass:
* Start with the most permissive IAM policy, run the tests to collect information about which resources your tests actually use, then construct a policy based on that output. This approach only works on modules that use `AnsibleAWSModule`.
* Start with the least permissive IAM policy, run the tests to discover a failure, add permissions for the resource that addresses that failure, then repeat. If your module uses `AnsibleModule` instead of `AnsibleAWSModule`, you must use this approach.
To start with the most permissive IAM policy:
1) `Create an IAM policy <https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create.html#access_policies_create-start>`_ that allows all actions (set ``Action`` and ``Resource`` to ``*```).
2) Run your tests locally with this policy. On AnsibleAWSModule-based modules, the ``debug_botocore_endpoint_logs`` option is automatically set to ``yes``, so you should see a list of AWS ACTIONS after the PLAY RECAP showing all the permissions used. If your tests use a boto/AnsibleModule module, you must start with the least permissive policy (see below).
3) Modify your policy to allow only the actions your tests use. Restrict account, region, and prefix where possible. Wait a few minutes for your policy to update.
4) Run the tests again with a user or role that allows only the new policy.
5) If the tests fail, troubleshoot (see tips below), modify the policy, run the tests again, and repeat the process until the tests pass with a restrictive policy.
6) Open a pull request proposing the minimum required policy to the `CI policies <https://github.com/mattclay/aws-terminator/tree/master/aws/policy>`_.
To start from the least permissive IAM policy:
1) Run the integration tests locally with no IAM permissions.
2) Examine the error when the tests reach a failure.
a) If the error message indicates the action used in the request, add the action to your policy.
b) If the error message does not indicate the action used in the request:
- Usually the action is a CamelCase version of the method name - for example, for an ec2 client the method `describe_security_groups` correlates to the action `ec2:DescribeSecurityGroups`.
- Refer to the documentation to identify the action.
c) If the error message indicates the resource ARN used in the request, limit the action to that resource.
d) If the error message does not indicate the resource ARN used:
- Determine if the action can be restricted to a resource by examining the documentation.
- If the action can be restricted, use the documentation to construct the ARN and add it to the policy.
3) Add the action or resource that caused the failure to `an IAM policy <https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create.html#access_policies_create-start>`_. Wait a few minutes for your policy to update.
4) Run the tests again with this policy attached to your user or role.
5) If the tests still fail at the same place with the same error you will need to troubleshoot (see tips below). If the first test passes, repeat steps 2 and 3 for the next error. Repeat the process until the tests pass with a restrictive policy.
6) Open a pull request proposing the minimum required policy to the `CI policies <https://github.com/mattclay/aws-terminator/tree/master/aws/policy>`_.
Troubleshooting IAM policies
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- When you make changes to a policy, wait a few minutes for the policy to update before re-running the tests.
- Use the `policy simulator <https://policysim.aws.amazon.com/>`_ to verify that each action (limited by resource when applicable) in your policy is allowed.
- If you're restricting actions to certain resources, replace resources temporarily with `*`. If the tests pass with wildcard resources, there is a problem with the resource definition in your policy.
- If the initial troubleshooting above doesn't provide any more insight, AWS may be using additional undisclosed resources and actions.
- Examine the AWS FullAccess policy for the service for clues.
- Re-read the AWS documentation, especially the list of `Actions, Resources and Condition Keys <https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_actions-resources-contextkeys.html>`_ for the various AWS services.
- Look at the `cloudonaut <https://iam.cloudonaut.io>`_ documentation as a troubleshooting cross-reference.
- Use a search engine.
- Ask in the #ansible-aws chat channel (using Matrix at ansible.im or using IRC at `irc.libera.chat <https://libera.chat/>`_).
Unsupported Integration tests
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
There are a limited number of reasons why it may not be practical to run integration
tests for a module within CI. Where these apply you should add the keyword
`unsupported` to the aliases file in `test/integration/targets/MODULE_NAME/aliases`.
Some cases where tests should be marked as unsupported:
1) The tests take longer than 10 or 15 minutes to complete
2) The tests create expensive resources
3) The tests create inline policies
4) The tests require the existence of external resources
5) The tests manage Account level security policies such as the password policy or AWS Organizations.
Where one of these reasons apply you should open a pull request proposing the minimum required policy to the
`unsupported test policies <https://github.com/mattclay/aws-terminator/tree/master/hacking/aws_config/test_policies>`_.
Unsupported integration tests will not be automatically run by CI. However, the
necessary policies should be available so that the tests can be manually run by
someone performing a PR review or writing a patch.
Mario Lenz
2 years ago
committed by