[cloud][docs] Improve exception handling guidelines for AWS modules(#30745)

Better document what exceptions to handle, when and why.
Describe how to handle client auth exceptions, and that
AWSRetry retries on `XYZNotFound` exceptions.

lib/ansible/modules/cloud/amazon/GUIDELINES.md

@@ -57,11 +57,12 @@ else:
 
 The `ansible.module_utils.ec2` module and `ansible.module_utils.core.aws` modules will both
 automatically import boto3 and botocore.  If boto3 is missing from the system then the variable
-HAS_BOTO3 will be set to false.  Normally, this means that modules don't need to import either
+`HAS_BOTO3` will be set to false.  Normally, this means that modules don't need to import either
-botocore or boto3 directly.
+botocore or boto3 directly. There is no need to check `HAS_BOTO3` when using AnsibleAWSModule
+as the module does that check.
 
 If you want to import the modules anyway (for example `from botocore.exception import
-ClientError`) Wrap import statements in a try block and fail the module later using HAS_BOTO3 if
+ClientError`) Wrap import statements in a try block and fail the module later using `HAS_BOTO3` if
 the import fails.
 
 #### boto
@@ -83,7 +84,15 @@ def main():
 #### boto3
 
 ```python
-from ansible.module_utils.aws.core import HAS_BOTO3
+from ansible.module_utils.aws.core import AnsibleAWSModule
+```
+
+or
+
+```python
+from ansible.module_utils.basic import AnsibleModule
+from ansible.module_utils.ec2 import HAS_BOTO3
+
 
 def main():
 
@@ -94,7 +103,7 @@ def main():
 #### boto and boto3 combined
 
 Ensure that you clearly document if a new parameter requires requires a specific version. Import
-boto3 at the top of the module as normal and then use the HAS_BOTO3 bool when necessary, before the
+boto3 at the top of the module as normal and then use the `HAS_BOTO3` bool when necessary, before the
 new feature.
 
 ```python
@@ -142,7 +151,8 @@ else:
 
 An example of connecting to ec2 is shown below.  Note that there is no `NoAuthHandlerFound`
 exception handling like in boto.  Instead, an `AuthFailure` exception will be thrown when you use
-'connection'. See exception handling.
+'connection'. To ensure that authorization, parameter validation and permissions errors are all
+caught, you should catch `ClientError` and `BotoCoreError` exceptions with every boto3 connection call.
 
 ```python
 region, ec2_url, aws_connect_params = get_aws_connection_info(module, boto3=True)
@@ -203,7 +213,7 @@ exceptions.  Call this on your exception and it will report the error together w
 use in Ansible verbose mode.
 
 ```python
-from ansible.module_utils.aws.core import HAS_BOTO3, AnsibleAWSModule
+from ansible.module_utils.aws.core AnsibleAWSModule
 
 # Set up module parameters
 ...
@@ -212,10 +222,11 @@ from ansible.module_utils.aws.core import HAS_BOTO3, AnsibleAWSModule
 ...
 
 # Make a call to AWS
+name = module.params.get['name']
 try:
-    result = connection.aws_call()
+    result = connection.describe_frooble(FroobleName=name)
-except Exception as e:
+except (botocore.exceptions.BotoCoreError, botocore.exceptions.ClientError) as e:
-    module.fail_json_aws(e, msg="trying to do aws_call")
+    module.fail_json_aws(e, msg="Couldn't obtain frooble %s" % name)
 ```
 
 Note that it should normally be acceptable to catch all normal exceptions here, however if you
@@ -225,13 +236,16 @@ If you need to perform an action based on the error boto3 returned, use the erro
 
 ```python
 # Make a call to AWS
+name = module.params.get['name']
 try:
-    result = connection.aws_call()
+    result = connection.describe_frooble(FroobleName=name)
-except ClientError, e:
+except botocore.exceptions.ClientError as e:
-    if e.response['Error']['Code'] == 'NoSuchEntity':
+    if e.response['Error']['Code'] == 'FroobleNotFound':
         return None
     else:
-        module.fail_json_aws(e, msg="trying to do aws_call")
+        module.fail_json_aws(e, msg="Couldn't obtain frooble %s" % name)
+except botocore.exceptions.BotoCoreError as e:
+    module.fail_json_aws(e, msg="Couldn't obtain frooble %s" % name)
 ```
 
 #### using fail_json() and avoiding ansible.module_utils.aws.core
@@ -240,36 +254,45 @@ Boto3 provides lots of useful information when an exception is thrown so pass th
 along with the message.
 
 ```python
-# Import ClientError from botocore
+from ansible.module_utils.ec2 import HAS_BOTO3
+
 try:
-    from botocore.exceptions import ClientError
+    import botocore
-    HAS_BOTO3 = True
 except ImportError:
-    HAS_BOTO3 = False
+    pass  # caught by imported HAS_BOTO3
 
 # Connect to AWS
 ...
 
 # Make a call to AWS
+name = module.params.get['name']
 try:
-    result = connection.aws_call()
+    result = connection.describe_frooble(FroobleName=name)
-except ClientError as e:
+except botocore.exceptions.ClientError as e:
-    module.fail_json(msg=e.message, exception=traceback.format_exc(),
+    module.fail_json(msg="Couldn't obtain frooble %s: %s" % (name, str(e)),
+                     exception=traceback.format_exc(),
                      **camel_dict_to_snake_dict(e.response))
 ```
 
+Note: we use `str(e)` rather than `e.message` as the latter doesn't
+work with python3
+
 If you need to perform an action based on the error boto3 returned, use the error code.
 
 ```python
 # Make a call to AWS
+name = module.params.get['name']
 try:
-    result = connection.aws_call()
+    result = connection.describe_frooble(FroobleName=name)
-except ClientError as e:
+except botocore.exceptions.ClientError as e:
-    if e.response['Error']['Code'] == 'NoSuchEntity':
+    if e.response['Error']['Code'] == 'FroobleNotFound':
         return None
     else:
-        module.fail_json(msg=e.message, exception=traceback.format_exc(),
+        module.fail_json(msg="Couldn't obtain frooble %s: %s" % (name, str(e)),
+                         exception=traceback.format_exc(),
                          **camel_dict_to_snake_dict(e.response))
+except botocore.exceptions.BotoCoreError as e:
+    module.fail_json_aws(e, msg="Couldn't obtain frooble %s" % name)
 ```
 
 ### API throttling and pagination
@@ -295,7 +318,7 @@ for more details.
 The combination of these two approaches is then
 
 ```
-@AWSRetry.exponential_backoff(tries=5, delay=5)
+@AWSRetry.exponential_backoff(retries=5, delay=5)
 def describe_some_resource_with_backoff(client, **kwargs):
      paginator = client.get_paginator('describe_some_resource')
      return paginator.paginate(**kwargs).build_full_result()['SomeResource']
@@ -309,6 +332,36 @@ def describe_some_resource(client, module):
         module.fail_json_aws(e, msg="Could not describe some resource")
 ```
 
+If the underlying `describe_some_resources` API call throws a `ResourceNotFound`
+exception, `AWSRetry` takes this as a cue to retry until it's not thrown (this
+is so that when creating a resource, we can just retry until it exists).
+
+To handle authorization failures or parameter validation errors in 
+`describe_some_resource_with_backoff`, where we just want to return `None` if
+the resource doesn't exist and not retry, we need:
+
+```
+@AWSRetry.exponential_backoff(retries=5, delay=5)
+def describe_some_resource_with_backoff(client, **kwargs):
+     try:
+         return client.describe_some_resource(ResourceName=kwargs['name'])['Resources']
+     except botocore.exceptions.ClientError as e:
+         if e.response['Error']['Code'] == 'ResourceNotFound':
+             return None
+         else:
+             raise
+     except BotoCoreError as e:
+         raise
+
+
+def describe_some_resource(client, module):
+    name = module.params.get['name']
+    try:
+        return describe_some_resource_with_backoff(client, name=name)
+    except (botocore.exceptions.BotoCoreError, botocore.exceptions.ClientError) as e:
+        module.fail_json_aws(e, msg="Could not describe resource %s" % name)
+```
+
 ### Returning Values
 
 When you make a call using boto3, you will probably get back some useful information that you