* parallelize getting mount info
* fixed timeout and made 8 max thread count
- minor cleanup
- avoid empty mount entries
- set timeout on get
- enforce timeout per mount/thread
- make note on failure per mount
- make note on timeout per mount
- ensure proper pool control
- minor fixes
- less vars, simpler code
- move filter 'pre threading'
- remove timeout for all mounts, now per mount
- also use cpu count from multiprocessing lib
- moved 'bind' options out of thread as per comments
- warn on error, more info on failure to get info
* removed info declaration from documentation fragment as this is not implemented
* added optional headers for POST and PUT requests
* updated documentation
* added missing headers field decalaration
* removed info choice from state field
* added tests for the new utm_utils function
* fixed class invocation
* added missing required params
* fixed the pytests
* Move get_all_subclasses out of sys_info as it is unrelated to system
information.
* get_all_subclasses now returns a set() instead of a list.
* Don't port get_platform to sys_info as it is deprecated. Code using
the common API should just use platform.system() directly.
* Rename load_platform_subclass() to get_platform_subclass and do not
instantiate the rturned class.
* Test the compat shims in module_utils/basic.py separately from the new
API in module_utils/common/sys_info.py and module_utils/common/_utils.py
* Revert "allow caller to deal with timeout (#49449)"
This reverts commit 63279823a7.
Flawed on many levels
* Adds poor API to a public function
* Papers over the fact that the public function is doing something bad
by catching exceptions it cannot handle in the first place
* Papers over the real cause of the issue which is a bug in the timeout
decorator
* Doesn't reraise properly
* Catches the wrong exception
Fixes#49824Fixes#49817
* Make the timeout decorator properly raise an exception outside of the function's scope
signal handlers which raise exceptions will never work well because the
exception can be raised anywhere in the called code. This leads to
exception race conditions where the exceptions could end up being
hanlded by unintended pieces of the called code.
The timeout decorator was using just that idiom. It was especially bad
because the decorator syntactically occurs outside of the called code
but because of the signal handler, the exception was being raised inside
of the called code.
This change uses a thread instead of a signal to manage the timeout in
parallel to the execution of the decorated function. Since raising of
the exception happens inside of the decorator, now, instead of inside of
a signal handler, the timeout exception is raised from outside of the
called code as expected which makes reasoning about where exceptions are
to be expected intuitive again.
Fixes#43884
* Add a common case test.
Adding an integration test driven from our unittests. Most of the time
we'll timeout in run_command which is running things in a subprocess.
Create a test for that specific case in case anything funky comes up
between threading and execve.
* Don't use OSError-based TimeoutError as a base class
Unlike most standard exceptions, OSError has a specific parameter list
with specific meanings. Instead follow the example of other stdlib
functions, concurrent.futures and multiprocessing and define a separate
TimeoutException.
* Add comment and docstring to point out that this is not hte Python3 TimeoutError
Since the 'platform.dist()' and 'platform.linux_distribution()'
methods will be removed from future versions of python, this
provides an alternative to replace ansibles use of those
methods.
lib/ansible/module_utils/distro.py is a copy of
https://github.com/nir0s/distro/blob/master/distro.py
This module is originally from https://github.com/nir0s/distro
and is license under the Apache License, Version 2.0.
* set ansible_os_family from name variable in os-release for clearlinux system
Signed-off-by: Josue David Hernandez Gutierrez <josue.d.hernandez.gutierrez@intel.com>
* Add os_family for clear linux and clear linux mixes
Signed-off-by: Josue David Hernandez Gutierrez <josue.d.hernandez.gutierrez@intel.com>
* Fix for changes in clearlinux
clearlinux is now providing /etc/os-release file and ansible is identifying as NA
then this change allow ansible to find it
Signed-off-by: Josue David Hernandez Gutierrez <josue.d.hernandez.gutierrez@intel.com>
* Add changelog fragment for clearlinux changes
Signed-off-by: Josue David Hernandez Gutierrez <josue.d.hernandez.gutierrez@intel.com>
* VMware: Fix module usages in module_utils
* Skip test for Python 2.6 as SSL context is not available in Python 2.6
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* FTD modules: bug fixes and upsert functionality
* Fix sanity checks
* Fix unit tests for Python 2.6
* Log status code for login/logout
* Use string formatting in logging
* More generic comparison code from docker_container to docker_common.
* More flexibility if a is None and method is allow_to_present.
Note that this odes not affect docker_container, as there a is never None.
* Update docker_secret and docker_config: simplify labels comparison.
* Added unit tests.
* Use proper subsequence test for allow_more_present for lists.
Note that this does not affect existing code in docker_container, since lists
don't use allow_more_present. Using allow_more_present will only be possible
in Ansible 2.8.
* pep8
* Move ansible.compat.tests to test/units/compat/.
* Fix unit test references to ansible.compat.tests.
* Move builtins compat to separate file.
* Fix classification of test/units/compat/ dir.
* Separate networking tools that may be used by modules outside of networking so changes to networking-only utilities don't trigger AWS integration tests
* Add unit tests for moved network utils
* Add comment to prevent imports from being mistakenly removed
* Move to_bits as well
* Add common and Swagger client utils for FTD modules
* Update FTD HTTP API plugin and add unit tests for it
* Add configuration layer handling object idempotency
* Add ftd_configuration module with unit tests
* Add ftd_file_download and ftd_file_upload modules with unit tests
* Validate operation data and parameters
* Fix ansible-doc, boilerplate and import errors
* Fix pip8 sanity errors
* Update object comparison to work recursively
* Add copyright
* Remove use of simplejson throughout code base. Fixes#42761
* Address failing tests
* Remove simplejson from contrib and other outlying files
* Add changelog fragment for simplejson removal
Now that we don't need to worry about python-2.4 and 2.5, we can make
some improvements to the way AnsiballZ handles modules.
* Change AnsiballZ wrapper to use import to invoke the module
We need the module to think of itself as a script because it could be
coded as:
main()
or as:
if __name__ == '__main__':
main()
Or even as:
if __name__ == '__main__':
random_function_name()
A script will invoke all of those. Prior to this change, we invoked
a second Python interpreter on the module so that it really was
a script. However, this means that we have to run python twice (once
for the AnsiballZ wrapper and once for the module). This change makes
the module think that it is a script (because __name__ in the module ==
'__main__') but it's actually being invoked by us importing the module
code.
There's three ways we've come up to do this.
* The most elegant is to use zipimporter and tell the import mechanism
that the module being loaded is __main__:
* 5959f11c9d/lib/ansible/executor/module_common.py (L175)
* zipimporter is nice because we do not have to extract the module from
the zip file and save it to the disk when we do that. The import
machinery does it all for us.
* The drawback is that modules do not have a __file__ which points
to a real file when they do this. Modules could be using __file__
to for a variety of reasons, most of those probably have
replacements (the most common one is to find a writable directory
for temporary files. AnsibleModule.tmpdir should be used instead)
We can monkeypatch __file__ in fom AnsibleModule initialization
but that's kind of gross. There's no way I can see to do this
from the wrapper.
* Next, there's imp.load_module():
* https://github.com/abadger/ansible/blob/340edf7489/lib/ansible/executor/module_common.py#L151
* imp has the nice property of allowing us to set __name__ to
__main__ without changing the name of the file itself
* We also don't have to do anything special to set __file__ for
backwards compatibility (although the reason for that is the
drawback):
* Its drawback is that it requires the file to exist on disk so we
have to explicitly extract it from the zipfile and save it to
a temporary file
* The last choice is to use exec to execute the module:
* https://github.com/abadger/ansible/blob/f47a4ccc76/lib/ansible/executor/module_common.py#L175
* The code we would have to maintain for this looks pretty clean.
In the wrapper we create a ModuleType, set __file__ on it, read
the module's contents in from the zip file and then exec it.
* Drawbacks: We still have to explicitly extract the file's contents
from the zip archive instead of letting python's import mechanism
handle it.
* Exec also has hidden performance issues and breaks certain
assumptions that modules could be making about their own code:
http://lucumr.pocoo.org/2011/2/1/exec-in-python/
Our plan is to use imp.load_module() for now, deprecate the use of
__file__ in modules, and switch to zipimport once the deprecation
period for __file__ is over (without monkeypatching a fake __file__ in
via AnsibleModule).
* Rename the name of the AnsiBallZ wrapped module
This makes it obvious that the wrapped module isn't the module file that
we distribute. It's part of trying to mitigate the fact that the module
is now named __main)).py in tracebacks.
* Shield all wrapper symbols inside of a function
With the new import code, all symbols in the wrapper become visible in
the module. To mitigate the chance of collisions, move most symbols
into a toplevel function. The only symbols left in the global namespace
are now _ANSIBALLZ_WRAPPER and _ansiballz_main.
revised porting guide entry
Integrate code coverage collection into AnsiballZ.
ci_coverage
ci_complete
d7df072b96 changed how we call
journal.send() from positional arguments to keyword arguments. So we
need to update the test to check for the arguments it was called with in
the keyword args, not in the positional args.
* Properly handle default package manager vs apt
For distros where apt might be installed but is not the default
package manager for the distro, properly identify the default distro
package manager during fact finding and re-use fact finding from
DistributionFactCollector and instead of reimplementing small
portions of it in PkgMgrFactCollector
Add unit test to always check the apt + Fedora combination to test
the new code.
Fixes#34014
Signed-off-by: Adam Miller <admiller@redhat.com>
* remove q debugging output I accidentally left behind
Signed-off-by: Adam Miller <admiller@redhat.com>
* add os_family to the conditional so we're only hitting that code path when needed
Signed-off-by: Adam Miller <admiller@redhat.com>
* setup for a _check* pattern for general os_family group pkg_mgr checking
Signed-off-by: Adam Miller <admiller@redhat.com>
* use Mock.patch decorator for os.path.exists in TestPkgMgrFactsAptFedora
Signed-off-by: Adam Miller <admiller@redhat.com>
Move dict_merge from azure_rm_resource module to
module_utils.common.dict_transformations and add tests.
Use dict_merge to provide a fairly realistic, reliable
diff output when k8s-based modules are run in check_mode.
Rename unit tests so that they actually run and reflect
the module_utils they're based on.
* Fix tmpdir on non root become
- also avoid exception if tmpdir and remote_tmp are None
- give 'None' on deescalation so tempfile will fallback to it's default behaviour
and use system dirs
- fix issue with bad tempdir (not existing/not createable/not writeable)
i.e nobody and ~/.ansible/tmp
- added tests for blockfile case
* Revert "Temporarily revert c119d54"
This reverts commit 5c614a59a6.
* changes based on PR feedback and changelog fragment
* changes based on the review
* Fix tmpdir when makedirs failed so we just use the system tmp
* Let missing remote_tmp fail
If remote_tmp is missing then there's something more basic wrong in the
communication from the controller to the module-side. It's better to
be alerted in this case than to silently ignore it.
jborean and I have independently checked what happens if the user sets
ansible_remote_tmp to empty string and !!null and both cases work fine.
(null is turned into a default value controller-side. empty string
triggers the warning because it is probably not a directory that the
become user is able to use).
* Issue #39860: Add 'not_contains' method to parsing.py
* Issue #30860 Adds self.negate to Conditional class in lib/ansible/module_utils/network/common/parsing.py
* Issue #39860 Fix singleton-comparison issue per sanity tests
* Issue #39860 'test/integration/targets/nxos_command/tests/cli/not_comparison_operator.yaml' integration test
* Issue #39860 Add unit tests to '../../test/units/module_utils/network/common/test_parsing.py'
* Issue #39860 Fix singleton comparison issue
* Fix E302 expected 2 blank lines, found 1
* Issue #39860 Add license header to unit tests
* Issue #39860 Move integration test to 'test/integration/targets/nxos_command/tests/common/'; remove unnecessary comment from unit test
* Issue #39860 remove unnecessary comment from unit test
When parsing the distribution files such as /etc/os-release, we extract
the full distribution version but not the major version. As such, the
ansible_distribution_major_version ends up being 'NA' whereas the
ansible_distribution_version contains the full version.
Before this patch we get this on openSUSE Leap 15
ansible -o localhost -m setup -a filter=ansible_distribution_major_version
localhost | SUCCESS => {"ansible_facts": {"ansible_distribution_major_version": "NA"}, "changed": false}
After this patch we get this
ansible -o localhost -m setup -a filter=ansible_distribution_major_version
localhost | SUCCESS => {"ansible_facts": {"ansible_distribution_major_version": "15"}, "changed": false}
This also fixes the Tumbleweed distribution test to report a proper
major version and also adds a test for openSUSE Leap 15.0 to avoid
potential future regressions.
Fixes: #41410
* Adds requests.Session like class
* py2 syntax fix
* Add a few examples to the Request docstrings
* Add helper methods and docs
* Fix test failures
* Switch tests to test Request instead of open_url, add simple open_url test to validate funcitonality
* Fix filename in replace-urlopen code smell test
* To fix following github issues 35774, 36574 and 39494
* To fix following github issues 35774, 36574 and 39494
* To fix following github issues 35774, 36574 and 39494
* To fix following github issues 35774, 36574 and 39494
* To fix following github issues 35774, 36574 and 39494
* To fix following github issues 35774, 36574 and 39494
* removed old_name new entry to make ui cleaner
* removed old_name new entry to make ui cleaner
* removed old_name new entry to make ui cleaner
* removed old_name new entry to make ui cleaner
* removed old_name new entry to make ui cleaner
* removed old_name new entry to make ui cleaner
* to resolve the bug 40709
* reslove shippable error
* reslove shippable error
* reslove shippable error
* reslove shippable error
* reslove shippable error
* reslove shippable error
* reslove shippable error
* reslove shippable error
* reslove shippable error
* to fix shippable nios automation error
* modified the name input parsing method
* modified the name input parsing method
* modified the name input parsing method
* modified the name input parsing method
* modified the name input parsing method
* modified the name input parsing method
* modified the name input parsing method
* modified the name input parsing method
* modified the name input parsing method
* shippable error fix
* shippable error fix
* shippable error fix
* shippable error fix
* shippable error fix
* review comment fix
* shippable error fix
* shippable error fix
Brian Coca