If the temp directory creation failed in mkdtemp then temp_path is never
given a value. This would lead to a NameError exception which would
obfuscate the original error (out of disk space being a common one). By
catching NameError, python will raise the original exception as we want.
Fixes#17215
* fixes#22441
* fixes#22655
* moves all env handling into the exec wrapper; this should work for everything but raw, which is consistent with non-Windows.
* Update module_utils.six to latest
We've been held back on the version of six we could use on the module
side to 1.4.x because of python-2.4 compatibility. Now that our minimum
is Python-2.6, we can update to the latest version of six in
module_utils and get rid of the second copy in lib/ansible/compat.
* Make the module_utils path configurable
* Add a config value to define the path site module_utils files
* Handle module_utils that do not have source as an error
* Make an integration test for module_utils envvar working
* Add documentation for the ANSIBLE_MODULE_UTILS config option/envvar
* Add it to the sample ansible.cfg
* Add it to intro_configuration.
* Also modify intro_configuration to place envvars on equal footing with
the config options (will need to document the envvar names in the
future)
* Also add the ANSIBLE_LIBRARY use case from
https://github.com/ansible/ansible/issues/15432 so we can close out
that bug.
This version just gets the relevant paths from PluginLoader and then
uses the existing imp.find_plugin() calls in the AnsiballZ code to load
the proper module_utils.
Modify PluginLoader to optionally omit subdirectories (module_utils
needs to operate on top level dirs, not on subdirs because it has
a hierarchical namespace whereas all other plugins use a flat
namespace).
Rename snippet* variables to module_utils*
Add a small number of unittests for recursive_finder
Add a larger number of integration tests to demonstrate that
module_utils is working.
Whitelist module-style shebang in test target library dirs
Prefix module_data variable with b_ to be clear that it holds bytes data
On Ubuntu the scriptdir gets placed into sys.path. This makes some
modules (copy) fail because the ansible module gets loaded instead of
the stdlib copy module. So we remove scriptdir there. Unfortunately,
the scriptdir code uses abspath(). When pipelining, abspath() has to
find the cwd. On OSX, finding the cwd when that directory is not
executable by the user raises an OSError. Since OSX does not suffer
from the scriptdir problem we're able to just skip scriptdir handling if
we get that exception.
Fixes#19729
Some machines have system clocks which can fall behind (for instance,
a host without a CMOS battery like Raspberry Pi). When managing those
machines we have to workaround the fact that the zip format does not
handle file timestamps before 1980. The workaround is to substitute in
the timestamp from the controller instead of from the managed machine.
Fixes#18640
While doing evil things with action plugins, I hit a code path in which
the mkdir here was failing due to lack of parent dir. Changing this to
makedirs made everything happy. Now, I'd obviously like to understand
why the parent dir exists in some places and not others - but I could
not find anywhere that C.DEFAULT_LOCAL_TMP is ensured to be created.
We couldn't copy to_unicode, to_bytes, to_str into module_utils because
of licensing. So once created it we had two sets of functions that did
the same things but had different implementations. To remedy that, this
change removes the ansible.utils.unicode versions of those functions.
I'm not sure why that would be desirable -- we really want __version__
to come from the controller whereas importing will come from the client
node. If it turns out there was a reason to do that, please be sure to
use an exception handler that catches all exceptions instead of only
catching ImportError:
```
try:
from ansible.release import __version__, __author__
except:
__version__ = [...]
```
Fixes#16523
Since Ansiballz, we no longer need to import basic directly into
a new-style module. Some modules, like the Networking modules, may
import basic in their own module_utils files and the module will import
that specialized module_util file rather than basic.
When setuptools installs a python module (as is done via python setup.py
install) It puts the module into a subdirectory of site-packages and
then creates an entry in easy-install.pth to load that directory. This
makes it difficult for Ansiballz to function correctly as the .pth file
overrides the sys.path that the wrapper constructs. Using
sitecustomize.py fixes this because sitecustomize overrides the
directories handled in .pth files.
Fixes#16187
* Port urls.py to python3
Fixes (largely normalizing byte vs text strings) for python3
* Rework what we do with attributes that aren't set already.
* Comments
This was reinitialized every time we forked before so we weren't sharing
the same Locks. It also was not accounting for modules which were
directly invoked by an action plugin instead of going through the
strategy plguins.
* Make ziploader's ansible and ansible.module_utils libraries into
namespace packages.
* Move __version__ and __author__ from ansible/__init__ to
ansible/release.py. This is because namespace packages only load one
__init__.py. If that is not the __init__.py with the author and
version info then those won't be available.
* In ziplaoder, move the version ito ANSIBLE_CONSTANTS.
* Change PluginLoader to properly construct the path to the plugins even
when namespace packages are present.
* Move zipcache temp dir creation into the locked section otherwise it
races with other workers.
* Catch IOError and turn it into an AnsibleError. IOErrors can hang
multiprocessng.
Updated python module wrapper explode method to drop 'args' file next to module.
Both execute() and excommunicate() debug methods now pass the module args via file to enable debuggers that are picky about stdin.
Updated unit tests to use a context manager for masking/restoring default streams and argv.
This makes our recursive, ast.parse performance measures as fast as
pre-ziploader baseline.
Since this unittest isn't testing that the returned module data is
correct we don't need to worry about os.rename not having any module
data. Should devise a separate test for the module and caching code
Some debuggers are easier to work with when we do everything in a single
process. This debug option caters to that at the expense of being
different from what Ansible will actually do to invoke a module.
When we document this we should be clear that this shouldn't be used for
general purpose debugging and that some modules may show strange
"errors" when used with this. Those won't be considered real bugs as
it's not how ansible really invokes the modules.
* Run the module as a script from the wrapper instead of executing in the same process.
Fixes cornercases where the module could potentially be executed twice
if we import and then run the main() function without calling sys.exit()
somewhere.
Also fixes problem with concurrent.futures() hanging. Not sure
precisely how this is being triggered but it is related to invoking the
main() function outside of an if __name__ == '__main__' conditional.
* Fix for python-2.6
* Ziploader proof of concept (jimi-c)
* Cleanups to proof of concept ziploader branch:
* python3 compatible base64 encoding
* zipfile compression (still need to enable toggling this off for
systems without zlib support in python)
* Allow non-wildcard imports (still need to make this recusrsive so that
we can have module_utils code that imports other module_utils code.)
* Better tracebacks: module filename is kept and module_utils directory
is kept so that tracebacks show the real filenames that the errors
appear in.
* Make sure we import modules that are used into the module_utils files that they are used in.
* Set ansible version in a more pythonic way for ziploader than we were doing in module replacer
* Make it possible to set the module compression as an inventory var
This may be necessary on systems where python has been compiled without
zlib compression.
* Refactoring of module_common code:
* module replacer only replaces values that make sense for that type of
file (example: don't attempt to replace python imports if we're in
a powershell module).
* Implement configurable shebang support for ziploader wrapper
* Implement client-side constants (for SELINUX_SPECIAL_FS and SYSLOG)
via environment variable.
* Remove strip_comments param as we're never going to use it (ruins line
numbering)
* Don't repeat ourselves about detecting REPLACER
* Add an easy way to debug
* Port test-module to the ziploader-aware modify_module()
* strip comments and blank lines from the wrapper so we send less over the wire.
* Comments cleanup
* Remember to output write the module line itself in powershell modules
* for line in lines strips the newlines so we have to add them back in
- Add support for inserting module args into PowerShell modules. Fixes#11661.
- Support Windows paths containing spaces. Applies changes from #10727 to v2. Fixes#9999. Should also fixansible/ansible-modules-core#944 and ansible/ansible-modules-core#1007.
- Change how execution policy is set for running remote scripts. Applies changes from #11092 to v2. Also fixesansible/ansible-modules-core#1776.
- Use codepage 65001 (UTF-8) for WinRM connection instead of default (CP437), convert command to UTF-8 and results from UTF-8. Replaces changes from #10024. Fixes#11198.
- Close WinRM connection when task completes.
- Use win_stat, win_file and win_copy modules instead of stat, file and copy when called from within other action plugins (only when using WinRM+PowerShell).
- Unquote Windows path arguments before passing to win_stat, win_file, win_copy and slurp modules (only when using WinRM/PowerShell).
- Check for win_ping module to determine if core modules are missing (only when using WinRM/PowerShell).
- Add stdout_lines to result from running low level commands (so stdout_lines is available when using raw/script).
- Update copy action plugin to use shell functions for joining paths and checking for trailing slash.
- Update fetch action plugin to unquote source path when using Windows paths.
- Add win_copy and win_template action plugins that inherit from copy and template.
- Support running .bat and .cmd scripts using default system encoding instead of UTF-8.
- Always send PowerShell commands as base64-encoded blobs to allow for running simple PowerShell commands via raw.
- Support running modules on Windows with interpreters other than PowerShell.
- Update integration tests to support above changes and test unicode fixes.
- Add test for win_user error from ansible/ansible-modules-core#1241 (fixed by ansible/ansible-modules-core#1774).
- Add test for additional win_stat output values (implemented by ansible/ansible-modules-core#1473).
- Add test for OS architecture and name from setup.ps1 (implemented by ansible/ansible-modules-core#1100).
All WinRM integration tests pass for me with these changes.
Matt Davis