From 836d7a42a42cd4040c60e382f6012927753c0ba3 Mon Sep 17 00:00:00 2001 From: Will Thames Date: Sat, 3 Mar 2018 05:03:12 +1000 Subject: [PATCH] Remove old boto documentation and promote AnsibleAWSModule (#36763) There should no longer be a need for boto documentation - it just adds significant extra clutter to the guidelines. AnsibleAWSModule greatly improves the codebase to such an extent that we should mandate it for new code unless there is a documented good reason. --- .../modules/cloud/amazon/GUIDELINES.md | 162 +++++++----------- 1 file changed, 60 insertions(+), 102 deletions(-) diff --git a/lib/ansible/modules/cloud/amazon/GUIDELINES.md b/lib/ansible/modules/cloud/amazon/GUIDELINES.md index f4117d69375..af14edcc59f 100644 --- a/lib/ansible/modules/cloud/amazon/GUIDELINES.md +++ b/lib/ansible/modules/cloud/amazon/GUIDELINES.md @@ -14,9 +14,35 @@ Please do not add new dependencies on the old boto library. Prior to 2.0, modules may have been written in boto or boto3. The effort to port all modules to boto3 has begun. From Ansible 2.4 it is permissible for modules which previously required boto to -start to require boto3 in order to deliver the functionality previously supported with boto and the +start to migrate to boto3 in order to deliver the functionality previously supported with boto and the boto dependency can be deleted. +From 2.6, all new modules should use AnsibleAWSModule as a base, or have a documented reason not +to. Using AnsibleAWSModule greatly simplifies exception handling and library management, reducing +the amount of boilerplate code. + +## Porting code to AnsibleAWSModule + +Change + +``` +from ansible.module_utils.basic import AnsibleModule +... +module = AnsibleModule(...) +``` + +to + +``` +from ansible.module_utils.aws.core import AnsibleAWSModule +... +module = AnsibleAWSModule(...) +``` + +Few other changes are required. One possible issue that you might encounter is that AnsibleAWSModule +does not inherit methods from AnsibleModule by default, but most useful methods +are included. If you do find an issue, please raise a bug report. + ## Bug fixing Bug fixes to code that relies on boto will still be accepted. When possible, the code should be @@ -57,35 +83,17 @@ else: The `ansible.module_utils.ec2` module and `ansible.module_utils.core.aws` modules will both automatically import boto3 and botocore. If boto3 is missing from the system then the variable -`HAS_BOTO3` will be set to false. Normally, this means that modules don't need to import either -botocore or boto3 directly. There is no need to check `HAS_BOTO3` when using AnsibleAWSModule +`HAS_BOTO3` will be set to false. Normally, this means that modules don't need to import +boto3 directly. There is no need to check `HAS_BOTO3` when using AnsibleAWSModule as the module does that check. -If you want to import the modules anyway (for example `from botocore.exception import -ClientError`) Wrap import statements in a try block and fail the module later using `HAS_BOTO3` if -the import fails. See below as well (`Exception Handling for boto3 and botocore`) for more detail -on how to safely import botocore exceptions. - -#### boto - ```python +from ansible.module_utils.aws.core import AnsibleAWSModule + try: - import boto.ec2 - from boto.exception import BotoServerError - HAS_BOTO = True + import botocore except ImportError: - HAS_BOTO = False - -def main(): - - if not HAS_BOTO: - module.fail_json(msg='boto required for this module') -``` - -#### boto3 - -```python -from ansible.module_utils.aws.core import AnsibleAWSModule + pass # handled by AnsibleAWSModule ``` or @@ -94,6 +102,10 @@ or from ansible.module_utils.basic import AnsibleModule from ansible.module_utils.ec2 import HAS_BOTO3 +try: + import botocore +except ImportError: + pass # handled by imported HAS_BOTO3 def main(): @@ -103,25 +115,8 @@ def main(): #### boto and boto3 combined -Ensure that you clearly document if a new parameter requires requires a specific version. Import -boto3 at the top of the module as normal and then use the `HAS_BOTO3` bool when necessary, before the -new feature. - -```python -from ansible.module_utils.ec2 import HAS_BOTO3 - -try: - import boto - HAS_BOTO = True -except ImportError: - HAS_BOTO = False +Modules should be ported to use boto3 rather than use both boto and boto3. -if my_new_feature_Parameter_is_set: - if HAS_BOTO3: - # do feature - else: - module.fail_json(msg="boto3 is required for this feature") -``` ### Connecting to AWS @@ -133,8 +128,6 @@ to connect to AWS as these handle the same range of connection options. These helpers also for missing profiles or a region not set when it needs to be, so you don't have to. -#### boto3 - An example of connecting to ec2 is shown below. Note that unlike boto there is no `NoAuthHandlerFound` exception handling like in boto. Instead, an `AuthFailure` exception will be thrown when you use the connection. To ensure that authorization, parameter validation and permissions errors are all caught, @@ -158,22 +151,9 @@ region, ec2_url, aws_connect_params = get_aws_connection_info(module, boto3=True connection = boto3_conn(module, conn_type='client', resource='ec2', region=region, endpoint=ec2_url, **aws_connect_params) ``` -#### boto - -An example of connecting to ec2: - -Some boto services require that the region is specified. You should check for the region parameter -if required. - ```python -region, ec2_url, aws_connect_params = get_aws_connection_info(module) -if region: - try: - connection = connect_to_aws(boto.ec2, region, **aws_connect_params) - except (boto.exception.NoAuthHandlerFound, AnsibleAWSError) as e: - module.fail_json(msg=str(e)) -else: - module.fail_json(msg="region must be specified") +region, ec2_url, aws_connect_params = get_aws_connection_info(module, boto3=True) +connection = boto3_conn(module, conn_type='client', resource='ec2', region=region, endpoint=ec2_url, **aws_connect_params) ``` ### Common Documentation Fragments for Connection Parameters @@ -198,35 +178,7 @@ extends_documentation_fragment: ''' ``` -### Exception Handling for boto - -You should wrap any boto call in a try block. If an exception is thrown, it is up to you decide how -to handle it but usually calling fail_json with the error or helpful message and traceback will -suffice. - -#### boto - -```python -# Import BotoServerError -try: - import boto.ec2 - from boto.exception import BotoServerError - HAS_BOTO = True -except ImportError: - HAS_BOTO = False - -# Connect to AWS -... - -# Make a call to AWS -try: - result = connection.aws_call() -except BotoServerError as e: - module.fail_json(msg="helpful message here", exception=traceback.format_exc(), - **camel_dict_to_snake_dict(e.message)) -``` - -### Exception Handling for boto3 and botocore +### Exception Handling You should wrap any boto3 or botocore call in a try block. If an exception is thrown, then there are a number of possibilities for handling it. @@ -240,14 +192,14 @@ For more information on botocore exception handling see [the botocore error docu #### using fail_json_aws() -_fail_json_aws() is a new method and may be subject to change. You can use it in modules which are -being contributed back to Ansible, however if you are publishing your module separately please -don't use it before the start of 2018 / Ansible 2.4_ - In the AnsibleAWSModule there is a special method, `module.fail_json_aws()` for nice reporting of exceptions. Call this on your exception and it will report the error together with a traceback for use in Ansible verbose mode. +You should use the AnsibleAWSModule for all new modules, unless not possible. If adding significant +amounts of exception handling to existing modules, we recommend migrating the module to use AnsibleAWSModule +(there are very few changes required to do this) + ```python from ansible.module_utils.aws.core import AnsibleAWSModule @@ -473,6 +425,20 @@ functions detailed below. boto3 returns results in a dict. The keys of the dict are in CamelCase format. In keeping with Ansible format, this function will convert the keys to snake_case. +`camel_dict_to_snake_dict` takes an optional parameter called `ignore_list` which is a list of +keys not to convert (this is usually useful for the `tags` dict, whose child keys should remain with +case preserved) + +Another optional parameter is `reversible`. By default, `HTTPEndpoint` is converted to `http_endpoint`, +which would then be converted by `snake_dict_to_camel_dict` to `HttpEndpoint`. +Passing `reversible=True` converts HTTPEndpoint to `h_t_t_p_endpoint` which converts back to `HTTPEndpoint`. + +#### snake_dict_to_camel_dict + +`snake_dict_to_camel_dict` converts snake cased keys to camel case. By default, because it was +first introduced for ECS purposes, this converts to dromedaryCase. An optional +parameter called `capitalize_first`, which defaults to `False`, can be used to convert to CamelCase. + #### ansible_dict_to_boto3_filter_list Converts a an Ansible list of filters to a boto3 friendly list of dicts. This is useful for any @@ -482,15 +448,7 @@ boto3 `_facts` modules. Pass an exception returned from boto or boto3, and this function will consistently get the message from the exception. -``` -import traceback -from ansible.module_utils.ec2 import boto_exception -try: - ... -except boto.exception.BotoServerError as err: - error_msg = boto_exception(err) - module.fail_json(msg=error_msg, exception=traceback.format_exc()) -``` +Deprecated: use `AnsibleAWSModule`'s `fail_json_aws` instead. #### boto3_tag_list_to_ansible_dict