We have some common code used by several docs scripts. Migrate that
into the build-only shared code repository.
* Move lib/ansible/utils/_build_helpers.py to the directory for common
build code
* Migrate docs/bin/dump_config.py to a build-ansible subcommand
* Migrate dump_keywords to the build-ansible framework
* Make the script more maintainable by using functions and good
variable names
* Port to Python3 idioms
* Fix bug so that private attributes will be undocumented
* Move generate_man to a build-ansible subcommand
* Port plugin_formatter to a build-ansible subcommand
* Rework command_plugins so that docs scripts can target Python-3.4+ and
releng-only subcommands can use more recent versions of Python.
The architecture is now that command_plugins/* need to be importable
on Python-3.4. The init_parsers() method needs to run on Python-3.4.
But the main() method can utilize features of more recent Python as
long as it fits within those parameters.
* Update docs build requirements
Port the plugin_formatter to build-ansible framework
ansible-test only passes files which have the .py suffix for sanity
tests on python files. This change will allow sanity tests to run on
the Python files in hacking/
* Rename test-module to test-module.py
* Symlink test-module for backwards compat since end users may be using
test-module
* Fix test-module sanity errors that are now triggered
* Rename ansible_profile to ansible-profile.py
* Rename build-ansible
* Update testing policies to ensure all required permissions are present
* Tidy up security policies to reduce duplicate permissions
* Make roles static so that they can be present before CI is run,
meaning that role creation permission is not required by the CI
itself, only by someone setting up the roles prior to testing
* Move contents to cloudfront policy to network policy to ensure policy
count (maximum of 10) stays low
* Maintain compute policy below 6144 bytes
Not waiting outputs results in a format that will never
be matched by the tests
Ensure instances get tidied up
Allow ec2:ReportInstanceStatus
ec2_instance: Improve test cleanup on failing tests
Improve describe/modify attribute error handling
Address feedback on PR
* Get the most detailed version number from distro.version() for CentOS and Debian
* Update tests and fixtures
* Update fixture generation script to gather distro info and work with Python 3
* Update LinuxMint fixtures
* Cleanup fixture formatting
* Improvements based on feedback from abadger:
- use unicode since that is what distro returns
- use frozenset with a tuple
- include link Debian to bug
* new module uses modern ansible AWS standards
* adds additional tests for snapshots
* Update return_skeleton_generator for python3 - should
set type to `str`, not `string`.
* Update testing policy to be correct for RDS test suite
* Create read replica in same region to avoid more permissions being
required
* Ensure modifying DB doesn't try to downgrade engine version
* Add tags to main test suite to limit number of tests run for problem
solving
* Combine testing policies
Because of the maximum of 10 policies per group, need to
consolidate testing policies as best we can.
* Tidy put-account-setting tasks and add permission
Using `environment` and `command` rather than `shell` avoids the
need for `no_log` and means that people can fix the problem
* refactor ecs_cluster test suite
move from runme.sh technique to virtualenv
use ec2_instance rather than ec2 module to
avoid need for boto
* Add waiter for AWSRegional
* Add support for WAF Regional
* Add support for regional waf web acl
* Remove set_trace, pep formatting
* Add paginator for regional_waf
* Change name of param for waf_regional
This is more in line with how AWS refers to the service. Additional
changes made to how client is called. Used ternary to reduce if
statements
* Change parameter name to waf_regional
* Add support for removal waf regional condition
* Change parameter from cloudfront to waf_regional
* Added state: absent waf rule
* Remove set_trace
* Add integration tests for waf regional
* WIP: adding region parameter to tests
* Add support for waf facts module
* Add region to waf regional integration tests
* Update security policy for waf regional testing
* Add type to documentation for waf_regional param
* Implement a framework for having common code for release scripts
* Release scripts will go through hacking/build-ansible. build-ansible is
a pluggable script which will set a directory that has common code for
non-enduser scripts. It will then invoke the plugin which implements
that subcommand. Uses straight.plugin for loading each sub-command.
* We're going to add tools which are needed to test ansible (the changelog
generation, for instance) so we need to include the pieces relevant to
that in the tarball.
* Add straight.plugin to the sanity test requirements for the same
reason
* Skip compile test just for build-ansible plugins which won't be run as
part of sanity tests.
* Make test-module use default value for interpreter
* Changing from static interpreter path to sys.executable as per #54053
* A little ntegration test for #54053
Figure out how to format the release announcement so a link isn't
directly followed by a period which would hinder cutand paste but uses
proper grammar and punctuation.
Announcements taken from https://github.com/ansible/community/wiki/RelEng:-ReleaseProcess
and then cleaned up:
* Update issue reporting blurb from feedback from acozine and gundalow
* Add a subject and to line for email output
* Ignore long line tests on the jinja templates (as jinja doesn't give
enough control to get rid of newlines when text wrapping)
* Skip shebang and compile tests for older pythons since this is a
release engineer-only script. (ok'd by mattclay)
* Add an option to parse botocore.endpoint logs for the AWS actions performed during a task
Add a callback to consolidate all AWS actions used by modules
Added some documentation to the AWS guidelines
* Enable aws_resource_actions callback only for AWS tests
* Add script to help generate policies
* Set debug_botocore_endpoint_logs via environment variable for all AWS integration tests
Ensure AWS tests inherit environment
(also remove AWS CLI in aws_rds inventory tests and use the module)
* AWS: new module ec2_transit_gateway fixes#49376
* Add permissions neeeded for integration tests
* uncomment nolog on creds
* add unsupported to integration test aliases
* remove the shippable/aws/group alias so doesn't conflict with unsupported
* Do not return 'instances' when wait is false
* Added integration tests for wait: false
* Added changelog fragment
* Fix test suite to work with ec2_instance
* Additional permissions
* Enforce boto3 version
* Fix broken tests
* Improve error messages
* fix linter issues
* adds fargate launch_type to ecs_task module
* White space changes
* fix documentation for running ecs task on fargate
* remove extraneous example from ecs_task
* White space changes
* Adds changelog fragment
* Pluralize minor_changes in changelog fragment
* Add Stop and Start task permissions
* fix parameters check and port module to boto3
* begin with integration tests
* allow redshift iam policy
* Wait for cluster to be created before moving on to delete it
* Allow sts credentials so this can be run in CI
Don't log credentials
ensure cluster can be removed
* - Replace DIY waiters with boto3 waiters
- test multi node cluster
* catch specific boto3 error codes
* remove wait from test
* add missing alias for shippable
* - Rework modify function.
- Default unavailable parameters to none.
- Add cluster modify test
* Ensure resources are cleaned up if tests fail
* Ensure all botocore ClientError and BotoCoreError exceptions are handled
* Allow creation and deletion of keys (deletion just schedules for
deletion, recreating an old key is just cancelling its deletion)
* Allow grants to be set, thus enabling encryption contexts to be
used with keys
* Allow tags to be added and modified
* Add testing for KMS module
* Tidy up aws_kms module to latest standards
* csv of memory usage
* Fix var
* Configurable output file
* Add cpu profiling
* Valdiate the existence of cgroup files
* Add guard to prevent exception when trying to reset max memory value
* to_bytes/to_text and docs updates
* Add support for CPU results
* Just track the max, don't log all results, and then calculate max
* Restore cgroup_memory_recap, and move new functionality into cgroup_perf_recap
* Add pid count tracking, restructure to support more profilers
* Add cli tool for graphing cgroup_perf_recap data
* csv_output_dir is a path
* Correct CALLBACK_NAME
* Include uuid in csv data
* fix linting errors
* Bump version_added
* Create helper funciton to create dict from list of keys, with callable default
* Updated notes to include pids
* Print a newline after each section
* Plugin improvements
* Add option to supporess recap display
* Add default for output directory
* Add option to dictate whether or not to write files
* Add JSON-seq output option
* s/uuid/task_uuid
* Use bytes for paths
* Increase polling interval length for pids/memory
* Reduce instance attrs, change how we invoke profilers
* Shorten some line lengths
* Remove more instance attrs
* Fix some typos
* document directory creation, and catch exceptions
* Enable per task file outputs, and filename customization
* s/per_task_file/file_per_task/g
* Add module ses_rule_set for Amazon SES
* Update behaviours and naming to be consistent with other aws_ses_ modules.
* Add global lock around tests using active rule sets to prevent intermittent test failures.
* Fix deletion of rule sets so that we don't inactivate the active rule set
when force deleting an inactive rule set.
Executed command:
./hacking/test-module -m lib/ansible/modules/cloud/scaleway/scaleway_security_group.py -a ...
Fix this exception found while testing scaleway_security_group module:
Traceback (most recent call last):
File "~/debug_dir/__main__.py", line 240, in <module>
main()
File "~/debug_dir/__main__.py", line 236, in main
core(module)
File "~/debug_dir/__main__.py", line 209, in core
api = Scaleway(module=module)
File "~/debug_dir/ansible/module_utils/scaleway.py", line 58, in __init__
'User-Agent': self.get_user_agent_string(module),
File "~/debug_dir/ansible/module_utils/scaleway.py", line 99, in get_user_agent_string
return "ansible %s Python %s" % (module.ansible_version, sys.version.split(' ')[0])
AttributeError: 'AnsibleModule' object has no attribute 'ansible_version'
* kick off
* done for the day
* beta code and test
* fix a typo
* boto3_conn and boto_exception aren't used in this code, ec2_argument_spec is used but unneeded.
* Returning when find a match avoids doing extra work, especially when pagination is involved
* add new permissions for test
* (output is changed) is preferred over accessing the attribute directly.
* pass the result through camel_dict_to_snake_dict() before returning it.
* AnsibleAWSModule automatically merges the argument_spec.
* deletes the created resources even if a test fails.
* AnsibleAWSModule automatically merges the argument_spec.
* fix typo
* fix pep8
* paginate list_repositories
* specify permissions for test
* cut the unnecessary code.
* add return doc string
* add missed ':'
* fix syntax error: mapping values are not allowed here
* add description for return
* fix syntax error
* rename module name and turn off automated integration test.
* Start of work on pylint plugin to catch due/past-due deprecated calls
* Improve deprecated pylint plugin
* Catch call to AnsibleModule.deprecate also
* Skip splatted kwargs, we can't infer that info
* Add error for invalid version in deprecation
* Skip version if it's a reference to a var
* Disable ansible-deprecated-no-version for displaying deprecated module info
* fix comments
* is None
* Force specifying a version, this can be disabled on a per case basis
* Disable ansible-deprecated-version by default
* Remove to look for 2.8 deprecated
* Revert "Remove to look for 2.8 deprecated"
This reverts commit 4e84034fd1.
* Add script and template used for creating issues for deprecated issues
* Fix underscore var
* Improve iam_group exception handling
Use AnsibleAWSModule for iam_group and handle BotoCoreErrors
as well as ClientErrors. Use fail_json_aws to improve error messages
* Add minimal iam_group test suite
Update some of the read-only IAM permissions (this is not sufficient
to run the test suite but it gets further than it did until it tries
to add a (non-existent) user)
* Clean up after tests
It seems that on some Linux distribution (Fedora 28, Debian), man will
not fallback on a default path if MANPATH is set. So using the env-setup
script will prevent man from working.
* added account_alias in the response of module aws_caller_facts
* added comment to explain list_account_aliases
* renamed caller_identity to caller_facts as the content is extended
* created changelog
* security-policy needs the iam:ListAccountAliases for this module to work
* test now checks for the added field account_alias
* gracefully handle missing iam:ListAccountAliases permission
* Remove use of simplejson throughout code base. Fixes#42761
* Address failing tests
* Remove simplejson from contrib and other outlying files
* Add changelog fragment for simplejson removal
Now that we don't need to worry about python-2.4 and 2.5, we can make
some improvements to the way AnsiballZ handles modules.
* Change AnsiballZ wrapper to use import to invoke the module
We need the module to think of itself as a script because it could be
coded as:
main()
or as:
if __name__ == '__main__':
main()
Or even as:
if __name__ == '__main__':
random_function_name()
A script will invoke all of those. Prior to this change, we invoked
a second Python interpreter on the module so that it really was
a script. However, this means that we have to run python twice (once
for the AnsiballZ wrapper and once for the module). This change makes
the module think that it is a script (because __name__ in the module ==
'__main__') but it's actually being invoked by us importing the module
code.
There's three ways we've come up to do this.
* The most elegant is to use zipimporter and tell the import mechanism
that the module being loaded is __main__:
* 5959f11c9d/lib/ansible/executor/module_common.py (L175)
* zipimporter is nice because we do not have to extract the module from
the zip file and save it to the disk when we do that. The import
machinery does it all for us.
* The drawback is that modules do not have a __file__ which points
to a real file when they do this. Modules could be using __file__
to for a variety of reasons, most of those probably have
replacements (the most common one is to find a writable directory
for temporary files. AnsibleModule.tmpdir should be used instead)
We can monkeypatch __file__ in fom AnsibleModule initialization
but that's kind of gross. There's no way I can see to do this
from the wrapper.
* Next, there's imp.load_module():
* https://github.com/abadger/ansible/blob/340edf7489/lib/ansible/executor/module_common.py#L151
* imp has the nice property of allowing us to set __name__ to
__main__ without changing the name of the file itself
* We also don't have to do anything special to set __file__ for
backwards compatibility (although the reason for that is the
drawback):
* Its drawback is that it requires the file to exist on disk so we
have to explicitly extract it from the zipfile and save it to
a temporary file
* The last choice is to use exec to execute the module:
* https://github.com/abadger/ansible/blob/f47a4ccc76/lib/ansible/executor/module_common.py#L175
* The code we would have to maintain for this looks pretty clean.
In the wrapper we create a ModuleType, set __file__ on it, read
the module's contents in from the zip file and then exec it.
* Drawbacks: We still have to explicitly extract the file's contents
from the zip archive instead of letting python's import mechanism
handle it.
* Exec also has hidden performance issues and breaks certain
assumptions that modules could be making about their own code:
http://lucumr.pocoo.org/2011/2/1/exec-in-python/
Our plan is to use imp.load_module() for now, deprecate the use of
__file__ in modules, and switch to zipimport once the deprecation
period for __file__ is over (without monkeypatching a fake __file__ in
via AnsibleModule).
* Rename the name of the AnsiBallZ wrapped module
This makes it obvious that the wrapped module isn't the module file that
we distribute. It's part of trying to mitigate the fact that the module
is now named __main)).py in tracebacks.
* Shield all wrapper symbols inside of a function
With the new import code, all symbols in the wrapper become visible in
the module. To mitigate the chance of collisions, move most symbols
into a toplevel function. The only symbols left in the global namespace
are now _ANSIBALLZ_WRAPPER and _ansiballz_main.
revised porting guide entry
Integrate code coverage collection into AnsiballZ.
ci_coverage
ci_complete
Toshio Kuratomi