* added logic to handle multiple actions in an ALB listener rule (#41861)
* fix linting and pep8 issues
* added test for multiple actions using OIDC authentication
* added error messages related to old versions of botocore and multiple actions
* fix action validation error checks (need to check the exception string)
* added logic to make oidc configs idempotent (remove clientsecret for check)
* modified TargetGroupName to TargetGroupArn substitution to account for multiple rule actions
* refactored tests so that it can be run against different versions of botocore
* fix runme.sh to refelct changes to cloud testsuite
* add UseExistingClientSecret to oidc config (AWS api change)
* remove tests for OIDC auth action; add tests for redirect and fixed-response
* add in fixes from markuman and mjmayer
* remove documentation for cognito integration (not sure how to test); added example config for fixed-response and redirect actions
* renamed oidc/multiple action tests; leaving commented due to some AWS API changes
* pep8 fix
* more pep8 fixes
* Restructure elb_application_lb test suite
Move from runme.sh to virtualenv based roles
Update policies to fix tests
Don't log temp dir deletion, so many files in the diff!
* When the release scripts were changed, this file to implement release
announcements was left out
* Add announce.py to the compile skip lists as only the release engineers
run this during package build.
* Generate galaxy.yml based on single source of truth
* Fix up tests and align file names
* Minor Makefile tweak
* Remove link in galaxy.yml file and make it a template file
* Moved collections docs to dev_guide
* change Makefile clean path
* Added readme to example meta file
* review fixes
* Use newer style for doc generation script
* Fix mistake in dev_guide index
* removed uneeded file, fixed links and added preview banner
* Moved banner for sanity test
* Fix dump_keywords to require attribute docs
* Change the documentation commands to make definition files positional
Since the definition files are mandatory, make them positional
parameters instead of options.
We have some common code used by several docs scripts. Migrate that
into the build-only shared code repository.
* Move lib/ansible/utils/_build_helpers.py to the directory for common
build code
* Migrate docs/bin/dump_config.py to a build-ansible subcommand
* Migrate dump_keywords to the build-ansible framework
* Make the script more maintainable by using functions and good
variable names
* Port to Python3 idioms
* Fix bug so that private attributes will be undocumented
* Move generate_man to a build-ansible subcommand
* Port plugin_formatter to a build-ansible subcommand
* Rework command_plugins so that docs scripts can target Python-3.4+ and
releng-only subcommands can use more recent versions of Python.
The architecture is now that command_plugins/* need to be importable
on Python-3.4. The init_parsers() method needs to run on Python-3.4.
But the main() method can utilize features of more recent Python as
long as it fits within those parameters.
* Update docs build requirements
Port the plugin_formatter to build-ansible framework
ansible-test only passes files which have the .py suffix for sanity
tests on python files. This change will allow sanity tests to run on
the Python files in hacking/
* Rename test-module to test-module.py
* Symlink test-module for backwards compat since end users may be using
test-module
* Fix test-module sanity errors that are now triggered
* Rename ansible_profile to ansible-profile.py
* Rename build-ansible
* Update testing policies to ensure all required permissions are present
* Tidy up security policies to reduce duplicate permissions
* Make roles static so that they can be present before CI is run,
meaning that role creation permission is not required by the CI
itself, only by someone setting up the roles prior to testing
* Move contents to cloudfront policy to network policy to ensure policy
count (maximum of 10) stays low
* Maintain compute policy below 6144 bytes
Not waiting outputs results in a format that will never
be matched by the tests
Ensure instances get tidied up
Allow ec2:ReportInstanceStatus
ec2_instance: Improve test cleanup on failing tests
Improve describe/modify attribute error handling
Address feedback on PR
* Get the most detailed version number from distro.version() for CentOS and Debian
* Update tests and fixtures
* Update fixture generation script to gather distro info and work with Python 3
* Update LinuxMint fixtures
* Cleanup fixture formatting
* Improvements based on feedback from abadger:
- use unicode since that is what distro returns
- use frozenset with a tuple
- include link Debian to bug
* new module uses modern ansible AWS standards
* adds additional tests for snapshots
* Update return_skeleton_generator for python3 - should
set type to `str`, not `string`.
* Update testing policy to be correct for RDS test suite
* Create read replica in same region to avoid more permissions being
required
* Ensure modifying DB doesn't try to downgrade engine version
* Add tags to main test suite to limit number of tests run for problem
solving
* Combine testing policies
Because of the maximum of 10 policies per group, need to
consolidate testing policies as best we can.
* Tidy put-account-setting tasks and add permission
Using `environment` and `command` rather than `shell` avoids the
need for `no_log` and means that people can fix the problem
* refactor ecs_cluster test suite
move from runme.sh technique to virtualenv
use ec2_instance rather than ec2 module to
avoid need for boto
* Add waiter for AWSRegional
* Add support for WAF Regional
* Add support for regional waf web acl
* Remove set_trace, pep formatting
* Add paginator for regional_waf
* Change name of param for waf_regional
This is more in line with how AWS refers to the service. Additional
changes made to how client is called. Used ternary to reduce if
statements
* Change parameter name to waf_regional
* Add support for removal waf regional condition
* Change parameter from cloudfront to waf_regional
* Added state: absent waf rule
* Remove set_trace
* Add integration tests for waf regional
* WIP: adding region parameter to tests
* Add support for waf facts module
* Add region to waf regional integration tests
* Update security policy for waf regional testing
* Add type to documentation for waf_regional param
* Implement a framework for having common code for release scripts
* Release scripts will go through hacking/build-ansible. build-ansible is
a pluggable script which will set a directory that has common code for
non-enduser scripts. It will then invoke the plugin which implements
that subcommand. Uses straight.plugin for loading each sub-command.
* We're going to add tools which are needed to test ansible (the changelog
generation, for instance) so we need to include the pieces relevant to
that in the tarball.
* Add straight.plugin to the sanity test requirements for the same
reason
* Skip compile test just for build-ansible plugins which won't be run as
part of sanity tests.
* Make test-module use default value for interpreter
* Changing from static interpreter path to sys.executable as per #54053
* A little ntegration test for #54053
Figure out how to format the release announcement so a link isn't
directly followed by a period which would hinder cutand paste but uses
proper grammar and punctuation.
Announcements taken from https://github.com/ansible/community/wiki/RelEng:-ReleaseProcess
and then cleaned up:
* Update issue reporting blurb from feedback from acozine and gundalow
* Add a subject and to line for email output
* Ignore long line tests on the jinja templates (as jinja doesn't give
enough control to get rid of newlines when text wrapping)
* Skip shebang and compile tests for older pythons since this is a
release engineer-only script. (ok'd by mattclay)
* Add an option to parse botocore.endpoint logs for the AWS actions performed during a task
Add a callback to consolidate all AWS actions used by modules
Added some documentation to the AWS guidelines
* Enable aws_resource_actions callback only for AWS tests
* Add script to help generate policies
* Set debug_botocore_endpoint_logs via environment variable for all AWS integration tests
Ensure AWS tests inherit environment
(also remove AWS CLI in aws_rds inventory tests and use the module)
* AWS: new module ec2_transit_gateway fixes#49376
* Add permissions neeeded for integration tests
* uncomment nolog on creds
* add unsupported to integration test aliases
* remove the shippable/aws/group alias so doesn't conflict with unsupported
* Do not return 'instances' when wait is false
* Added integration tests for wait: false
* Added changelog fragment
* Fix test suite to work with ec2_instance
* Additional permissions
* Enforce boto3 version
* Fix broken tests
* Improve error messages
* fix linter issues
* adds fargate launch_type to ecs_task module
* White space changes
* fix documentation for running ecs task on fargate
* remove extraneous example from ecs_task
* White space changes
* Adds changelog fragment
* Pluralize minor_changes in changelog fragment
* Add Stop and Start task permissions
* fix parameters check and port module to boto3
* begin with integration tests
* allow redshift iam policy
* Wait for cluster to be created before moving on to delete it
* Allow sts credentials so this can be run in CI
Don't log credentials
ensure cluster can be removed
* - Replace DIY waiters with boto3 waiters
- test multi node cluster
* catch specific boto3 error codes
* remove wait from test
* add missing alias for shippable
* - Rework modify function.
- Default unavailable parameters to none.
- Add cluster modify test
* Ensure resources are cleaned up if tests fail
* Ensure all botocore ClientError and BotoCoreError exceptions are handled
* Allow creation and deletion of keys (deletion just schedules for
deletion, recreating an old key is just cancelling its deletion)
* Allow grants to be set, thus enabling encryption contexts to be
used with keys
* Allow tags to be added and modified
* Add testing for KMS module
* Tidy up aws_kms module to latest standards
* csv of memory usage
* Fix var
* Configurable output file
* Add cpu profiling
* Valdiate the existence of cgroup files
* Add guard to prevent exception when trying to reset max memory value
* to_bytes/to_text and docs updates
* Add support for CPU results
* Just track the max, don't log all results, and then calculate max
* Restore cgroup_memory_recap, and move new functionality into cgroup_perf_recap
* Add pid count tracking, restructure to support more profilers
* Add cli tool for graphing cgroup_perf_recap data
* csv_output_dir is a path
* Correct CALLBACK_NAME
* Include uuid in csv data
* fix linting errors
* Bump version_added
* Create helper funciton to create dict from list of keys, with callable default
* Updated notes to include pids
* Print a newline after each section
* Plugin improvements
* Add option to supporess recap display
* Add default for output directory
* Add option to dictate whether or not to write files
* Add JSON-seq output option
* s/uuid/task_uuid
* Use bytes for paths
* Increase polling interval length for pids/memory
* Reduce instance attrs, change how we invoke profilers
* Shorten some line lengths
* Remove more instance attrs
* Fix some typos
* document directory creation, and catch exceptions
* Enable per task file outputs, and filename customization
* s/per_task_file/file_per_task/g
* Add module ses_rule_set for Amazon SES
* Update behaviours and naming to be consistent with other aws_ses_ modules.
* Add global lock around tests using active rule sets to prevent intermittent test failures.
* Fix deletion of rule sets so that we don't inactivate the active rule set
when force deleting an inactive rule set.
Executed command:
./hacking/test-module -m lib/ansible/modules/cloud/scaleway/scaleway_security_group.py -a ...
Fix this exception found while testing scaleway_security_group module:
Traceback (most recent call last):
File "~/debug_dir/__main__.py", line 240, in <module>
main()
File "~/debug_dir/__main__.py", line 236, in main
core(module)
File "~/debug_dir/__main__.py", line 209, in core
api = Scaleway(module=module)
File "~/debug_dir/ansible/module_utils/scaleway.py", line 58, in __init__
'User-Agent': self.get_user_agent_string(module),
File "~/debug_dir/ansible/module_utils/scaleway.py", line 99, in get_user_agent_string
return "ansible %s Python %s" % (module.ansible_version, sys.version.split(' ')[0])
AttributeError: 'AnsibleModule' object has no attribute 'ansible_version'
* kick off
* done for the day
* beta code and test
* fix a typo
* boto3_conn and boto_exception aren't used in this code, ec2_argument_spec is used but unneeded.
* Returning when find a match avoids doing extra work, especially when pagination is involved
* add new permissions for test
* (output is changed) is preferred over accessing the attribute directly.
* pass the result through camel_dict_to_snake_dict() before returning it.
* AnsibleAWSModule automatically merges the argument_spec.
* deletes the created resources even if a test fails.
* AnsibleAWSModule automatically merges the argument_spec.
* fix typo
* fix pep8
* paginate list_repositories
* specify permissions for test
* cut the unnecessary code.
* add return doc string
* add missed ':'
* fix syntax error: mapping values are not allowed here
* add description for return
* fix syntax error
* rename module name and turn off automated integration test.
* Start of work on pylint plugin to catch due/past-due deprecated calls
* Improve deprecated pylint plugin
* Catch call to AnsibleModule.deprecate also
* Skip splatted kwargs, we can't infer that info
* Add error for invalid version in deprecation
* Skip version if it's a reference to a var
* Disable ansible-deprecated-no-version for displaying deprecated module info
* fix comments
* is None
* Force specifying a version, this can be disabled on a per case basis
* Disable ansible-deprecated-version by default
* Remove to look for 2.8 deprecated
* Revert "Remove to look for 2.8 deprecated"
This reverts commit 4e84034fd1.
* Add script and template used for creating issues for deprecated issues
* Fix underscore var
* Improve iam_group exception handling
Use AnsibleAWSModule for iam_group and handle BotoCoreErrors
as well as ClientErrors. Use fail_json_aws to improve error messages
* Add minimal iam_group test suite
Update some of the read-only IAM permissions (this is not sufficient
to run the test suite but it gets further than it did until it tries
to add a (non-existent) user)
* Clean up after tests
It seems that on some Linux distribution (Fedora 28, Debian), man will
not fallback on a default path if MANPATH is set. So using the env-setup
script will prevent man from working.
* added account_alias in the response of module aws_caller_facts
* added comment to explain list_account_aliases
* renamed caller_identity to caller_facts as the content is extended
* created changelog
* security-policy needs the iam:ListAccountAliases for this module to work
* test now checks for the added field account_alias
* gracefully handle missing iam:ListAccountAliases permission
* Remove use of simplejson throughout code base. Fixes#42761
* Address failing tests
* Remove simplejson from contrib and other outlying files
* Add changelog fragment for simplejson removal
Now that we don't need to worry about python-2.4 and 2.5, we can make
some improvements to the way AnsiballZ handles modules.
* Change AnsiballZ wrapper to use import to invoke the module
We need the module to think of itself as a script because it could be
coded as:
main()
or as:
if __name__ == '__main__':
main()
Or even as:
if __name__ == '__main__':
random_function_name()
A script will invoke all of those. Prior to this change, we invoked
a second Python interpreter on the module so that it really was
a script. However, this means that we have to run python twice (once
for the AnsiballZ wrapper and once for the module). This change makes
the module think that it is a script (because __name__ in the module ==
'__main__') but it's actually being invoked by us importing the module
code.
There's three ways we've come up to do this.
* The most elegant is to use zipimporter and tell the import mechanism
that the module being loaded is __main__:
* 5959f11c9d/lib/ansible/executor/module_common.py (L175)
* zipimporter is nice because we do not have to extract the module from
the zip file and save it to the disk when we do that. The import
machinery does it all for us.
* The drawback is that modules do not have a __file__ which points
to a real file when they do this. Modules could be using __file__
to for a variety of reasons, most of those probably have
replacements (the most common one is to find a writable directory
for temporary files. AnsibleModule.tmpdir should be used instead)
We can monkeypatch __file__ in fom AnsibleModule initialization
but that's kind of gross. There's no way I can see to do this
from the wrapper.
* Next, there's imp.load_module():
* https://github.com/abadger/ansible/blob/340edf7489/lib/ansible/executor/module_common.py#L151
* imp has the nice property of allowing us to set __name__ to
__main__ without changing the name of the file itself
* We also don't have to do anything special to set __file__ for
backwards compatibility (although the reason for that is the
drawback):
* Its drawback is that it requires the file to exist on disk so we
have to explicitly extract it from the zipfile and save it to
a temporary file
* The last choice is to use exec to execute the module:
* https://github.com/abadger/ansible/blob/f47a4ccc76/lib/ansible/executor/module_common.py#L175
* The code we would have to maintain for this looks pretty clean.
In the wrapper we create a ModuleType, set __file__ on it, read
the module's contents in from the zip file and then exec it.
* Drawbacks: We still have to explicitly extract the file's contents
from the zip archive instead of letting python's import mechanism
handle it.
* Exec also has hidden performance issues and breaks certain
assumptions that modules could be making about their own code:
http://lucumr.pocoo.org/2011/2/1/exec-in-python/
Our plan is to use imp.load_module() for now, deprecate the use of
__file__ in modules, and switch to zipimport once the deprecation
period for __file__ is over (without monkeypatching a fake __file__ in
via AnsibleModule).
* Rename the name of the AnsiBallZ wrapped module
This makes it obvious that the wrapped module isn't the module file that
we distribute. It's part of trying to mitigate the fact that the module
is now named __main)).py in tracebacks.
* Shield all wrapper symbols inside of a function
With the new import code, all symbols in the wrapper become visible in
the module. To mitigate the chance of collisions, move most symbols
into a toplevel function. The only symbols left in the global namespace
are now _ANSIBALLZ_WRAPPER and _ansiballz_main.
revised porting guide entry
Integrate code coverage collection into AnsiballZ.
ci_coverage
ci_complete
* aws_eks: New module for managing AWS EKS
aws_eks module is used for creating and removing EKS clusters.
Includes full test suite and updates to IAM policies to enable it.
* Clean up all security groups
* appease shippable
* Rename aws_eks module to aws_eks_cluster
The compute policy was exceeding maximum size and contained
policies that already exist in ecs-policy.
Look up suitable AMIs rather than hardcode
We don't want to maintain multiple image IDs for multiple regions
so use ec2_ami_facts to set a suitable image ID
Improve exception handling
* Fix test-module failing to validate args
The test-module pass a wrong argument _ansible_tmp cause the validation failed.
Change the argument _ansible_tmp to _ansible_tmpdir to fix this.
* Add a integration test for test-module.
Prior to this change, we don't have a test for test-module.
This change ensure the correctness of test-module script.
Remove VPC permissions from network-policy.json as they mostly duplicate
compute-policy.json permissions - separating the VPC and compute permissions
would likely lead to further confusion.
We only needed it for migrating cherrypicks between the unified repo and
the ansible-modules-* repos. Now that we aren't supporting 2.3, we no
longer need this script.
* New module: ec2_vpc_vpn_facts
* Add integration tests for ec2_vpc_vpn_facts and the IAM permissions
* Add retry to VPC removal
* Use unique name for VGW
* Always clean up after tests and add retries
* create module tmpdir based on remote_tmp
* Source remote_tmp from controller if possible
* Fixed sanity test and not use lambda
* Added expansion of env vars to the remote tmp
* Fixed sanity issues
* Added note around shell remote_tmp option
* Changed fallback tmp dir to ~/.ansible/tmp to make shell defaults
* Add helpful failure message if target_type=ip is not supported
Create test case for target_type=ip not supported
* Update elb_target_group module to latest standards
Use AnsibleAWSModule
Improve exception handling
Improve connection handling
Enable awsvpc network mode for ECS services and tasks and
their underlying task definitions
Improve test suite to thoroughly test the changes
Use runme.sh technique to run old and new versions of botocore to
ensure that the modules work with older botocore and older network modes
and fail gracefully if awsvpc network mode is used with older botocore
* Add aws_ses_identity_policy module for managing SES sending policies
* Add option to AnsibleAWSModule for applying a retry decorator to all calls.
* Add per-callsite opt in to retry behaviours in AnsibleAWSModule
* Update aws_ses_identity_policy module to opt in to retries at all callsites.
* Add test for aws_ses_identity_policy module with inline policy.
* Remove implicit retrys on boto resources since they're not working yet.
Cloudfront needs CreateOriginAccessIdentity
Add profile parameter to setup-iam.yml. Could arguably just use
AWS_PROFILE but given that other tasks are using profile, should
be consistent.
* module_common: set required parameter templar
Fix the following error (related to b455901):
$ ./hacking/test-module -m ./lib/ansible/modules/system/ping.py -I ansible_python_interpreter=/usr/bin/python
Traceback (most recent call last):
File "./hacking/test-module", line 268, in <module>
main()
File "./hacking/test-module", line 249, in main
(modfile, modname, module_style) = boilerplate_module(options.module_path, options.module_args, interpreters, options.check, options.filename)
File "./hacking/test-module", line 152, in boilerplate_module
task_vars=task_vars
File "ansible/lib/ansible/executor/module_common.py", line 910, in modify_module
environment=environment)
File "ansible/lib/ansible/executor/module_common.py", line 736, in _find_module_utils
shebang, interpreter = _get_shebang(u'/usr/bin/python', task_vars, templar)
File "ansible/lib/ansible/executor/module_common.py", line 452, in _get_shebang
interpreter = templar.template(task_vars[interpreter_config].strip())
AttributeError: 'NoneType' object has no attribute 'template'
* module_common.modify_module: templar is required
* Add aws_ses_identity module
* Update CI alias, add BotoCoreError exception handling.
* Add SES and SNS permissions to hacking/aws_config to run aws_ses_identity integration tests
The search string used to look for Clear Linux
was changed in 45a9f96774 to
be more specific, but was too specific. Now finding
a substring match for 'Clear Linux' in /usr/lib/os-release
is enough to consider a match.
Since the details of the full name in os-release varies
('Clear Linux Software for Intel Architecture',
'Clear Linux OS for Intel Architecture', etc) the
search string match was failing and would fall back to the
'first word in the release file' method resulting in
ansible_distribution='NAME="Clear'
Also add a meta fact indicating which search string
was matched.
Test case info from:
https://github.com/ansible/ansible/issues/31501#issuecomment-340861535Fixes#31501
People expect to be able to upload files to s3 using standard
locations for files.
Providing an action plugin that effectively rewrites the `src`
key to the result of finding such a file is a great help.
Tests added, and IAM permissions corrected
* added cloudfont.py, modified cloudfront_facts.py class name and fixed a minor bug
* Improvements to cloudfront_distribution
* Reduce the scope of the cloudfront_distribution module
* Remove presigning
* Remove streaming distribution functionality
* Add full test suite for cloudfront distribution
* Meet Ansible AWS guidelines
* Make requested changes
Fix tests
Use built-in waiter
Update copyright
Tests for:
* ecs_cluster
* ecs_service
* ecs_service_facts
* ecs_taskdefinition
* ecs_taskdefinition_facts
* Add idempotency testing
Test ecs_cluster, ecs_service and ecs_taskdefinition for trivial
idempotency. Add FIXMEs to the tests because the latter two fail.
Remove unused dependencies
This fix adds replacement for exit() to sys.exit(), as
exit() is not recommended way to exit from the program.
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Add some integration tests for ec2_vpc_net module
* Add a couple tests for check mode
fix typo
ensure the DHCP option set is cleaned up
* Add permissions to test policy
* Warn on tests used as filters
* Update docs, add aliases for tests that fit more gramatically with test syntax
* Fix rst formatting
* Add successful filter, alias of success
* Remove renamed_deprecation, it was overkill
* Make directory alias for is_dir
* Update tests to use proper jinja test syntax
* Update additional documentation, living outside of YAML files, to reflect proper jinja test syntax
* Add conversion script, porting guide updates, and changelog updates
* Update newly added uses of tests as filters
* No underscore variable
* Convert recent tests as filter changes to win_stat
* Fix some changes related to rebasing a few integration tests
* Make tests_as_filters_warning explicitly accept the name of the test, instead of inferring the name
* Add test for tests_as_filters_warning
* Update tests as filters in newly added/modified tests
* Address recent changes to several integration tests
* Address recent changes in cs_vpc
Exception was:
Traceback (most recent call last):
File "./hacking/test-module", line 268, in <module>
main()
File "./hacking/test-module", line 249, in main
(modfile, modname, module_style) = boilerplate_module(options.module_path, options.module_args, interpreters, options.check, options.filename)
File "./hacking/test-module", line 155, in boilerplate_module
if module_style == 'new' and 'ANSIBALLZ_WRAPPER = True' in module_data:
TypeError: a bytes-like object is required, not 'str'
* Add integration test suite for ec2_vpc_subnet
* wrap boto3 connection in try/except
update module documentation and add RETURN docs
add IPv6 support to VPC subnet module
rename ipv6cidr to ipv6_cidr, use required_if for parameter testing, update some failure messages to be more descriptive
DryRun mode was removed from this function a while ago but exception handling was still checking for it, removed
add wait and timeout for subnet creation process
fixup the ipv6 cidr disassociation logic a bit per review
update RETURN values per review
added module parameter check
removed DryRun parameter from boto3 call since it would always be false here
fix subnet wait loop
add a purge_tags parameter, fix the ensure_tags function, update to use compare_aws_tags func
fix tags type error per review
remove **kwargs use in create_subnet function per review
* rebased on #31870, fixed merge conflicts, and updated error messages
* fixes to pass tests
* add test for failure on invalid ipv6 block and update tags test for purge_tags=true function
* fix pylint issue
* fix exception handling error when run with python3
* add ipv6 tests and fix module code
* Add permissions to hacking/aws_config/testing_policies/ec2-policy.json for adding IPv6 cidr blocks to VPC and subnets
* fix type in tests and update assert conditional to check entire returned value
* add AWS_SESSION_TOKEN into environment for aws cli commands to work in CI
* remove key and value options from call to boto3_tag_list_to_ansible_dict
* remove wait loop and use boto3 EC2 waiter
* remove unused register: result vars
* revert az argument default value to original setting default=None
* Allow backoff for describe_subnets
Improve exception handling to latest standards
* Add integration test suite for ec2_vpc_subnet
* Add test for creating subnet without AZ
Fix bug identified by test
Fixes#31905
* ec2_group: add support for rule descriptions.
* Document rule description feature and add an example using it.
* Fix removing rule descriptions.
* Add integration tests to verify adding/modifying/removing rule descriptions works as expected.
* Add permissions to hacking/aws_config/testing_policies/ec2-policy.json for updating ingress and egress rule descriptions.
* ec2_group: add backwards compatibility with older versions of botocore for rule descriptions.
* Add compatibility with older version of botocore for ec2_group integration tests.
* ec2_group: move HAS_RULE_DESCRIPTION to be checked first.
* Make requested change
* Pass around a variable instead of client
* Make sure has_rule_description defaults to None
* Fail if rule_desc is in any ingress/egress rules and the the botocore version < 1.7.2
* Remove unnecessary variable
* Fix indentation for changed=True when updating rule descriptions.
* minor refactor to remove duplicate code
* add missing parameter
* Fix pep8
* Update test policy.
Provide all necessary permissions for AMI tests
Allow tests to run in us-east-2
Ensure `always` section gets used
Update tests to ensure that cleanup works better, and add
deletion idempotency test
After running hacking/test-module to generate some output,
the JSON output can be fed into the return skeletion generator
to create an excellent starting point for RETURN docs
* Update RDS parameter group for boto3
* Update to boto3
* Update to latest ansible standards
* Remove choices list for valid engines (See #19221 for context)
* Allow tagging
* Return some useful information, and document that information
* Add tests for rds_param_group
* Improve testing of rds_param_group
* Add purge_tags option for rds_param_group
* Fix remaining broken rds_param_group tests
* Ensure the group name is lowercased. Fixes integration tests when run on OSX
* Add network value to support_by field.
* New support_by value, certified
* Deprecate curated in favor of certified
* Add conversion from 1.0 to 1.1 to metadata-tool
* Add supported by Red Hat field to ansible-doc output
* Add test runner to PATH
* Add Python3 support
* Updating env-setup.fish to use more portable '-exec' rather than '-delete'
* Create gen_egg_info function
Move code into a function similar to env-setup.
Silence all output when run with -q
* Add tests for group in a VPC
* Improve ec2_group output and documentation
Update ec2_group to provide full security group information
Add RETURN documentation to match
* Fix ec2_group creation within a VPC
Ensure VPC ID gets passed when creating security group
* Add test for auto creating SG
* Fix ec2_group auto group creation
* Add backoff to describe_security_groups
Getting LimitExceeded from describe_security_groups is definitely
possible (source: me) so add backoff to increase likelihood of
success.
To ensure that all `describe_security_group` calls are backed off,
remove implicit ones that use `ec2.SecurityGroup`. From there,
the decision to remove the `ec2` boto3 resource and rely on the client
alone makes good sense.
* Tidy up auto created security group
Add resource_prefix to auto created security group and delete
it in the `always` section.
Use YAML argument form for all module parameters
* Use Boto3 for ec2_group
Currently boto doesn't support ipv6. To support ipv6 in ec2_group, we need boto3.
boto3 has significant API changes, which caused more re-factoring for ec2_group module.
Added additional integration test to test_ec2_group role.
* Follow the standard for boto3 ansible
Fixed imports. Use boto3 ansible exception with camel_dict_to_snake_dict.
Refactored the call to authorize/revoke ingress and egress.
* Removed dependancy with module ipaddress
Added new parameter called cidr_ipv6 for specifying
ipv6 addresses inline with how boto3 handles ipv6 addresses.
* Updated integration test
* Added ipv6 integration test for ec2_group
* Set purge_rules to false for integration test
* Fixed import statements
Added example for ipv6.
Removed defining HAS_BOTO3 variable and import HAS_BOTO3 from ec2.
Cleaned up import statements.
* Fixed exception handling
* Add IAM permissions for ec2_group tests
Missing AuthorizeSecurityGroupEgress necessary for latest tests
* Wrapped botocore import in try/except block
Import just botocore to be more similar to other modules
* Fix ansible-doc traceback when a plugin doesn't parse correctly
* Change extract_metadata ivocation to take either an ast or source
code. When given source code, it can find file offsets for the start
and end of dict. When given the ast, it is quicker as it doesn't have
to reparse the source. Requires changing the call to the function to
use a keyword arg.
* Fix reading of metadata to find the last occurrence of
ANSIBLE_METADATA instead of the first.
* Add some more unittests to get closer to complete coverage
Make pyca/cryptography the preferred backend for cryptographic needs (mainly vault) falling back to pycrypto
pyca/cryptography is already implicitly a dependency in many cases
through paramiko (2.0+) as well as the new openssl_publickey module,
which requires pyOpenSSL 16.0+. Additionally, pyca/cryptography is
an optional dep for better performance with vault already.
This commit leverages cryptography's padding, constant time comparisons,
and CBC/CTR modes to reduce the amount of code ansible needs to
maintain.
* Handle wrong password given for VaultAES format
* Do not display deprecation warning for cryptography on python-2.6
* Namespace all of the pycrypto imports and always import them
Makes unittests better and the code less likely to get stupid mistakes
(like using HMAC from cryptogrpahy when the one from pycrypto is needed)
* Add back in atfork since we need pycrypto to reinitialize its RNG just in case we're being used with old paramiko
* contrib/inventory/gce: Remove spurious require on pycrypto
(cherry picked from commit 9e16b9db275263b3ea8d1b124966fdebfc9ab271)
* Add cryptography to ec2_win_password module requirements
* Fix python3 bug which would pass text strings to a function which
requires byte strings.
* Attempt to add pycrypto version to setup deps
* Change hacking README for dual pycrypto/cryptography
* update dependencies for various CI scripts
* additional CI dockerfile/script updates
* add paramiko to the windows and sanity requirement set
This is needed because ansible lists it as a requirement. Previously
the missing dep wasn't enforced, but cryptography imports pkg_resources
so you can't ignore a requirement any more
* Add integration test cases for old vault and for wrong passwords
* helper script for manual testing of pycrypto/cryptography
* Skip the pycrypto tests so that users without it installed can still run the unittests
* Run unittests for vault with both cryptography and pycrypto backend
* Start of ansible config project
moved configuration definitions to external yaml file vs hardcoded
* updated constants to be a data strcutures that are looped over and also return origin of setting
changed to manager/data scheme for base classes
new cli ansible-config to view/manage ansible configuration settings
* prints green for default/unchanged and yellow for those that have been overriden
* added list action to show all configurable settings and their associated ini and env var names
* allows specifying config file to see what result would look like
* TBD update, edit and view options
removed test for functions that have been removed
env_Vars are now list of dicts
allows for version_added and deprecation in future
added a couple of descriptions for future doc autogeneration
ensure test does not fail if delete_me exists
normalized 'path expansion'
added yaml config to setup packaging
removed unused imports
better encoding handling
updated as per feedback
* pep8
* aws integration tests - provide an IAM policy that can be used for running them
* move documentation of aws policies into main integration testing documentation + some updates there
* Update testing_integration.rst
Edits
Made ansible-doc more plugin agnostic
We can have docs in lookup, callback, connectionm strategy, etc
Use first docstring and make pepizis happy
generalized module_docs to plugin_docs
documented cartesian, ssh, default, jsonfile, etc as examples
changed lack of docs to warning when listing
made smarter about bad docstrings
better blacklisting
added handling of options/config/envs/etc
move blacklist to find_plugins, only need once
Changes to the metadata format were approved here:
https://github.com/ansible/proposals/issues/54
* Update documentation to the new metadata format
* Changes to metadata-tool to account for new metadata
* Add GPL license header
* Add upgrade subcommand to upgrade metadata version
* Change default metadata to the new format
* Fix exclusion of non-modules from the metadata report
* Fix ansible-doc for new module metadata
* Exclude metadata version from ansible-doc output
* Fix website docs generation for the new metadata
* Update metadata schema in valiate-modules test
* Update the metadata in all modules to the new version
Raise the bar for module `DOCUMENTAION`
This validator update was used to find the issues in https://github.com/ansible/ansible/pull/22297/files
**Validation**
* Updated Validation and docs to enforce more (items fixed in https://github.com/ansible/ansible/pull/22297/files)
* Use `suboptions` to document complex options
* Validate module name
* Validate deprecated modules have correct ANSIBLE_METADATA
**Module Documentation Generation**
* Document `suboptions:` Example https://gist.github.com/gundalow/4bdc3669d696268328ccc18528cc6718
* Tidy up HTML generation (valid HTML, no empty lists, etc)
**Documentation**
* Clarify the steps for deprecating a module
* Use correct RST headings
* Document `suboptions:` (options)
* Document `contains:` (returns)
**Details**
The aim is to get this (and corresponding module updates) complete by the time `devel` becomes `2.4`, as this allows us to raise the bar for new modules
Example `suboptions` https://gist.github.com/gundalow/4bdc3669d696268328ccc18528cc6718
The aim is to get this PR integrated into `devel` *before* we branch `stable-2.3`, this will allows us to:
* Raise the bar for new modules in 2.4
* Ensure the generated module documentation for 2.3 and higher is improved, important as we will be doing versioned docs moving forward.
* Update test-module
Ensuring invoke is assigned
Traceback (most recent call last):
File "ansible/hacking/test-module", line 267, in <module>
main()
File "ansible/hacking/test-module", line 263, in main
runtest(modfile, argspath, modname, module_style, interpreters)
File "ansible/hacking/test-module", line 207, in runtest
invoke = "%s%s" % (invoke, modfile)
UnboundLocalError: local variable 'invoke' referenced before assignment
* Update test-module
Made the change to only require a single if, making the function more 'DRY'.
* Use ansible_python_interpreter to run modules
Use ansible_python_interpreter to run modules if
`-I ansible_python_interpreter` is set.
Remove unused default from `-I` help text.
* Update test-module to pep8 standards
- Replace nose usage with pytest.
- Remove legacy Shippable integration.sh.
- Update Makefile to use pytest and ansible-test.
- Convert most yield unit tests to pytest parametrize.
- Remove shebangs from:
- ini files
- unit tests
- module_utils
- plugins
- module_docs_fragments
- non-executable Makefiles
- Change non-modules from '/usr/bin/python' to '/usr/bin/env python'.
- Change '/bin/env' to '/usr/bin/env'.
Also removed main functions from unit tests (since they no longer
have a shebang) and fixed a python 3 compatibility issue with
update_bundled.py so it does not need to specify a python 2 shebang.
A script was added to check for unexpected shebangs in files.
This script is run during CI on Shippable.
* Remove old egg-info files before creating new ones
Currently, setup.py generates egg files then they are deleted. This change
fixes this behavior and matches that in env-setup.
* Do not try to move ansible*egg-info to lib/
setup.py creates the ansible.egg-info in lib/ so this step is unnecessary. Matches env-setup behavior.
* Better test for number of arguments in argv
This prevents an erronous error message from being thrown since set -q returns an error code with the number of variables not defined, resulting in a non-zero exit if no arguments are passed.
Indent case statement within switch statement.
Currently, "ansible localhost -m setup" can fail silently during the
run of gen_distribution_version_testcase.py, resulting in incorrect
output. Use check_output() rather than communicate() and handle
the exception if we get a nonzero return value.
As suggested in feedback on
https://github.com/ansible/ansible/pull/17575, add
os_family to test_distribution_version. Add the
correct os_family to the existing testcase data
entries.
Also add os_family to the output of
gen_distribution_version_testcase.py so any new
generated entries will contain this data.
We couldn't copy to_unicode, to_bytes, to_str into module_utils because
of licensing. So once created it we had two sets of functions that did
the same things but had different implementations. To remedy that, this
change removes the ansible.utils.unicode versions of those functions.
Previously it also built the MANPAGES target
requiring asciidoc and libxml, before starting
a 'make docs' in docsite.
Also change the #! line in
hacking/dump_playbook_attributes.py to not specify
python2... yet.
* add tests for centos6, rhel6 and rhel7
* gen_distribution_version_testcase with python2.6
* remove unused imports
* fix redhat/vmware/... parsing
* add centos7 test case
This makes our recursive, ast.parse performance measures as fast as
pre-ziploader baseline.
Since this unittest isn't testing that the returned module data is
correct we don't need to worry about os.rename not having any module
data. Should devise a separate test for the module and caching code
* Could only have one alias before. Subsequent aliases overrode the
previous ones. Now multiple aliases work.
* Fix BLACKLISTED_MODULES. Previously, modules were listed in the
generated documentation despite being blacklisted
* Deprecated modules form extras were showing the (E) tag and not the
(D) tag. Reversed that now (Probably not necessary to also show the
E tag).
* Sort the deprecated modules alphabetically in the Category docs as
well as the list of all modules
* Optimization: Previously rendered the modules to rst twice once in all
group and once in individual categories. Fixed to only render them
once.
* Add fireball to blacklist and remove async_status (as people need to
use that).
* Ziploader proof of concept (jimi-c)
* Cleanups to proof of concept ziploader branch:
* python3 compatible base64 encoding
* zipfile compression (still need to enable toggling this off for
systems without zlib support in python)
* Allow non-wildcard imports (still need to make this recusrsive so that
we can have module_utils code that imports other module_utils code.)
* Better tracebacks: module filename is kept and module_utils directory
is kept so that tracebacks show the real filenames that the errors
appear in.
* Make sure we import modules that are used into the module_utils files that they are used in.
* Set ansible version in a more pythonic way for ziploader than we were doing in module replacer
* Make it possible to set the module compression as an inventory var
This may be necessary on systems where python has been compiled without
zlib compression.
* Refactoring of module_common code:
* module replacer only replaces values that make sense for that type of
file (example: don't attempt to replace python imports if we're in
a powershell module).
* Implement configurable shebang support for ziploader wrapper
* Implement client-side constants (for SELINUX_SPECIAL_FS and SYSLOG)
via environment variable.
* Remove strip_comments param as we're never going to use it (ruins line
numbering)
* Don't repeat ourselves about detecting REPLACER
* Add an easy way to debug
* Port test-module to the ziploader-aware modify_module()
* strip comments and blank lines from the wrapper so we send less over the wire.
* Comments cleanup
* Remember to output write the module line itself in powershell modules
* for line in lines strips the newlines so we have to add them back in
Matt Clay