Commit Graph

519 Commits (faf446a895590344e1563455edba05809401f9c8)

Author SHA1 Message Date
Jordan Borean 0cdc410dce
no_log mask suboption fallback values and defaults CVE-2021-20228 (#73487)
* no_log mask suboption fallback values and defaults

* Added changelog

* Remove lambda expression
5 years ago
Brian Coca c7d4acc12f
allow env to override unspecified unsafe_writes (#73282)
* allow env var for fallback value for unspecified unsafe_writes
5 years ago
Alexander Sowitzki e6da544310 Let `command` always return stdout & stderr (#73004)
The `command` module does not return stdout & sterr when calling
a non existing executable or an unknown exception arises. This fix
lets the module return empty byte strings in those cases.
5 years ago
Brian Coca 932ba36160
ensure unsafe writes fallback (#70722)
* Ensure we actually fallback to unsafe_writes when set to true

 add integration test
 add fix for get_url not passing the parameter from args
5 years ago
Sam Doooran 5ecfb19cad
arg_spec - move validator lookup method to a function (#72667)
* arg_spec - move type checking lookup method to a function

* Change get_wanted_type name and behavior

Change the name to get_validator to bette describe what it is doing.

Change the interface to always return a value. This lines up with the behavior of get_*
functions always returning something or None and check_* functions raising an
Exception if something went wrong during the check.

* Add param to check_type_str()

Not meant to be a long term fix, but gets tests passing. More work is needed to figure
out how to solve this cleanly.

* Remove private attribute mapping types to validator

Since the function that needs it has moved to parameters.py, there is no need to have it as
a attribute of AnsibleModule.

Update tests that were referencing the private attribute.

* Use private method for 'str' type

To avoid having to put the string conversion warning behavior in the check_type_str() method,
use the private _check_type_str() method for 'str' type.

Import CHECK_ARGUMENT_TYPES_DISPATCHER for backwards compalitibility and store it as
a private attribute.

Revert changes to support plugins that are referencing serf._CHECK_ARGUMENT_TYPES_DISPATCHER.

* Add changelog

* Change function name to better reflect its... function

* Change dict name to better reflect its contents

CHECK_ARGUMENT_TYPES_DISPATCHER --> DEFAULT_TYPE_VALIDATORS

* Fix changelog
5 years ago
Pilou b464d18fd1
AnsibleModule.set_mode_if_different: handle symlink is in a sticky directory (#45198)
* file: add symlink is in a sticky directory tests
* file: handle symlink in a sticky directory

Co-Authored-By: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>

* Add changelog and fix unit test
The builtins import was removed since it was unused, but it is now needed.
5 years ago
Sam Doran e889b1063f
arg_spec - rework _check_arguments() (#72447)
* Move _syslog_facitily to __init__
  No good reason it should not be set for each object

* Move internal property setting to private method
* Create check_arguments() function
* Remove unused import
* Rename function to better match its behavior
  Change the behavior to return a set, either empty or populated, with unsupported keys.
  Accept legal_inputs as optional which will not required calling handle_aliases before calling
  get_unsupported_parameters().

* Add changelog
* Rework function behavior and documentation
  I realized I missed the original intent of this method when moving it to a function. It
  is meant to compared the parameter keys to legal inputs always, not compare
  parameter keys to argument spec keys, even though the argument spec keys should
  be a subset of legal inputs.

* Add tests
* Fix typo.
* Set internal properties when handling suboptions
5 years ago
Brian Coca 07248e5ec1
avoid key errors on environment access (#72620) 5 years ago
Alexei Znamensky 5654de6fce
Return error if cwd directory does not exist (#72390)
* Return warning or error if cwd directory does not exist, in AnsibleModule.run_command()
5 years ago
Abhijeet Kasurde 9ffa84cc1c
basic: Handle exception in default selector (#72101)
In Python 2.7, default selector raises IOError
instead of OSError. Fix handles this exception.

Fixes: #71704

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
5 years ago
Rick Elrod 5cd489af06
Let get_file_attributes() work without `lsattr -v` (#71845)
* Let get_file_attributes() work without `lsattr -v`

Change:
- module_utils's get_file_attributes() expects `lsattr -v` to work, but
  in some cases, it may not.
- The function now takes an optional include_version bool parameter,
  which removes this expectation.
- Places where we call get_file_attributes() without using the 'version'
  it returns, we now call it with include_version=False.

Test Plan:
- New unit tests

Signed-off-by: Rick Elrod <rick@elrod.me>
5 years ago
Sam Doran 888be697cb
Revert "Change default file permissions so they are not world readable (#70221)" (#71231)
* Revert "Change default file permissions so they are not world readable (#70221)"

This reverts commit 5260527c4a.

* Revert "Fix warning for new default permissions when mode is not specified (#70976)"

This reverts commit dc79528cc6.
5 years ago
Sam Doran dc79528cc6
Fix warning for new default permissions when mode is not specified (#70976)
Follow up to #70221
Related to #67794
CVE-2020-1736

When set_mode_if_different() is called with mode of 'None', ensure we issue
a warning about the change in default permissions.

Add integration tests to ensure the warning works properly.

* Fix tests
- actually use custom module 🤦‍♂️
- verify file permission on created files
- use remote_tmp_dir so we're ready for split controller
- improve test module so we can skip the call to set_fs_attributes_if_different()
- fix tests for CentOS 6
5 years ago
Abhijeet Kasurde e439194c8c
basic: Add name of aliases in error message (#69427)
With this patch, user will be notified with available aliases
of arg parameter.

Fixes: #58752

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
5 years ago
Sam Doran 5260527c4a
Change default file permissions so they are not world readable (#70221)
* Change default file permissions so they are not world readable

CVE-2020-1736

Set the default permissions for files we create with atomic_move() to 0o0660. Track
which files we create that did not exist and warn if the module supports 'mode'
and it was not specified and the module did not call set_mode_if_different(). This allows the user to take action and specify a mode rather than using the defaults.

A code audit is needed to find all instances of modules that call atomic_move()
but do not call set_mode_if_different(). The findings need to be documented in
a changelog since we are not warning. Warning in those instances would be frustrating
to the user since they have no way to change the module code.

- use a set for storing list of created files
- just check the argument spac and params rather than using another property
- improve the warning message to include the default permissions
5 years ago
David Shrewsbury bf98f031f3
Sanitize URI module keys with no_log values (#70762)
* Add sanitize_keys() to module_utils.

* More robust tests

* Revert 69653 change

* Allow list or dict

* fix pep8

* Sanitize lists within dict values

* words

* First pass at uri module

* Fix insane sanity tests

* fix integration tests

* Add changelog

* Remove unit test introduced in 69653

* Add ignore_keys param

* Sanitize all-the-things

* Ignore '_ansible*' keys

* cleanup

* Use module.no_log_values

* Avoid deep recursion issues by using deferred removal structure.

* Nit cleanups

* Add doc blurb

* spelling

* ci_complete
5 years ago
Abhijeet Kasurde 8cccede0d4
basic: use PollSelector implementation (#70352)
Some platform such as ESXi does not implement EpollSelector,
which is selected by DefaultSelector. Use SelectSelector which is
based upon 'select' implementation. This works perfectly with
a platform like VMware ESXi.

Fixes: #70238

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
5 years ago
jabdr 0690b68bd3
Support datetime.date object in module result (#70595)
* Support datetime.date object in module result

Fixes #70583

* change blank lines for pep8 sanity test
5 years ago
Rick Elrod 688cd8657b
Fix string/bytestring comparsion in m_u.basic (#70439)
Change:
- module_utils.basic.is_special_selinux_path() used a string ==
  bytestring comparison which returned False and made Ansible think that
  certain filesystems aren't, in fact, special-cased, when they should
  be. Ensure both sides of the == are bytestrings.

Test Plan:
- Added `copy` integration tests for this case.

Tickets:
- Fixes #70244

Signed-off-by: Rick Elrod <rick@elrod.me>
6 years ago
Rick Elrod f7078c1f8f
Throw a prettier error in m_u.basic syslog (#70312)
Change:
- In certain situations, such as when the input string contains null
  bytes (\0), syslog.syslog will throw a TypeError. Handle that and
  fail_json instead.

Test Plan:
- New test
- ansible-test --docker centos[68] (for py2 and py3 respectively)

Tickets:
- Refs #70269

Signed-off-by: Rick Elrod <rick@elrod.me>
6 years ago
Matt Clay 234994fc07 Fix boilerplate in setup.py and lib/ansible/ dir. 6 years ago
Sam Doran bc05415109
Only pass kwargs to our string checker not callable checkers (#70151)
Since only check_type_str() accepts extra param, only pass to our checker and
do not pass kwargs to custom checkers.

* Add unit tests
6 years ago
Jordan Borean 15f07e7a6c
Fix up schema for collection deprecation (#69977)
* Fix up schema for collection deprecation

* Fix up modules using wrong name

* Another fix
6 years ago
Felix Fontein a862ff2d43
Deprecation revisited (#69926)
* Allow to specify collection_name separately for deprecation.

* Use new functionality in Ansible.

* Use new functionality in tests.

* Update tagging/untagging functions.

* Update pylint deprecated sanity test.

* Update validate-modules. Missing are basic checks for version_added (validate semantic version format for collections).

* Improve version validation. Re-add version_added validation.

* Make sure collection names are added to return docs before schema validation.

* Extra checks to avoid crashes on bad data.

* Make C# module utils code work, and update/extend tests.

* Add changelog fragment.

* Stop extracting collection name from potentially tagged versions/dates.

* Simplify C# code.

* Update Windows modules docs.

* Forgot semicolons.
6 years ago
Sam Doran 6065638e00
Display parameter name in string conversion warning (#57145)
Add prefix to error message for nested options. This is helpful if a subelement key has the same name as a top level key.
6 years ago
Felix Fontein ea04e0048d
Allow to deprecate options and aliases by date (#68177)
* Allow to deprecate options and aliases by date instead of only by version.

* Update display.deprecate().

* Adjust behavior to conform to tested behavior, extend tests, and improve C# style.

* Parse date and fail on invalid date.

This is mainly to make sure that people start using invalid dates, and we eventually have a mess to clean up.

* C# code: improve validation and update/extend tests.

* Make sure that deprecate() is not called with both date and version.

* Forgot to remove no longer necessary formatting.

* Adjust order of warnings in C# code.

* Adjust unrelated test.

* Fix grammar (and make that test pass).

* Don't parse date, and adjust message to be same as in #67684.

* Sanity tests: disable date in past test.

* Validate-modules: validate ISO 8601 date format.

* Validate-modules: switch schema declaration for deprecated_aliases to improve error messages for invalid dates.

* Use DateTime instead of string for date deprecation.

* Validate that date in deprecated_aliases is actually a DateTime.

* Fix tests.

* Fix rebasing error.

* Adjust error codes for pylint, and add removed_at_date and deprecated_aliases.date checks to validate-modules.

* Make deprecation date in the past error codes optional.

* Make sure not both version and date are specified for AnsibleModule.deprecate() calls.

* Stop using Python 3.7+ API.

* Make sure errors are actually reported. Re-add 'ansible-' prefix.

* Avoid crashing when 'name' isn't there.

* Linting.

* Update lib/ansible/module_utils/csharp/Ansible.Basic.cs

Co-authored-by: Jordan Borean <jborean93@gmail.com>

* Adjust test to latest change.

* Prefer date over version if both end up in Display.deprecated().

Co-authored-by: Jordan Borean <jborean93@gmail.com>
6 years ago
Matt Martz e0f25a2b1f
Strip no log values from module response keys. Fixes #68400 (#69653) 6 years ago
Bob Weinand f200487414
Fix filedescriptor out of range in select() when running commands (#65058)
* Fix filedescriptor out of range in select() when running commands

* Simplify the run_command() code

Now that we're using selectors in run_command(), we can simplify some of
the code.

* Use fileobj.read() instead of os.read()
* No longer use get_buffer_size() as we can just slurp all of the data
  instead.

Also use a simpler conditional check of whether the selector map is
empty

Co-authored-by: Toshio Kuratomi <a.badger@gmail.com>
6 years ago
Rylan Polster eb40ecc843
Check for correct version of systemd Python library (#60692)
Fix issue where some versions of systemd don't have journal.sendv
6 years ago
Abhijeet Kasurde 0f8937f2ed
basic: Update required library message (#68657)
Updated missing_required_lib API message.

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
6 years ago
Toshio Kuratomi 6531ba38f8 Allow the msg argument to fail_json() to be a positional argument
fial_json() requires a message be given to it to inform the end user of
why the module failed.  Prior to this commit, the message had to be a
keyword argument:

    module.fail_json(msg='Failed due to error')

Since this is a required parameter, this commit allows the message to be
given as a positional argument instead:

   module.fail_json('Failed due to an error')
6 years ago
Felix Fontein 8ff62799c9
AnsibleModule.load_file_common_arguments: allow to override path (#61659)
* Allow to overwrite path in AnsibleModule.load_file_common_arguments

* Use path override.

* Add changelog.
6 years ago
Felix Fontein f6815040fd
add_file_common_arguments: fix defaults and tpyes, improve sanity checking (#67243)
* Make validate-modules stop ignore FILE_COMMON_ARGUMENTS.

* Add types to FILE_COMMON_ARGUMENTS and update document fragment to match it.

* Update ignore.txt.

* Add changelog.
6 years ago
Felix Fontein f725dce936
Clean up FILE_COMMON_ARGUMENTS (#66389)
* Clean up FILE_COMMON_ARGUMENTS.

* postgresql_pg_hba doesn't declare the backup option.

* uri doesn't declare the remote_src option.

* Add documentation.

* maven_artifact seems to use directory_mode, which it doesn't declare.

* Update changelogs/fragments/66389-file-common-arguments.yml
Update docs/docsite/rst/porting_guides/porting_guide_2.10.rst

ci_complete

Co-Authored-By: Jill R <4121322+jillr@users.noreply.github.com>
6 years ago
Felix Fontein fe454d27a1
Fix removed_in_version to support honor suboptions (#66918)
* Add unit tests.
* Fix reporting for removed_in_version.
* Add changelog.
6 years ago
Sam Doran 5112feeace
Make get_bin_path() always raise an exception (#56813)
This makes it behave in a more idiomatic way

* Fix bug in Darwin facts for free memory
    If the vm_stat command is not found, fact gathering would fail with an unhelpful 
    error message. Handle this gracefully and return a default value for free memory.

* Add unit tests
6 years ago
Sam Doran 3461c682c3
Add mechanism for storing warnings and deprecations outside of AnsibleModule (#58993)
* Move warn() and deprecate() methods out of basic.py
* Use _global_warnings and _global_deprications and create accessor functions
    - This lays the foundation for future functions being moved outside of AnsibleModule
       that need an interface to warnings and deprecations without modifying them.
* Add unit tests for new warn and deprecate functions
6 years ago
Felix Fontein c58d8ed1f5 Improve deprecate sanity check; remove deprecated features which should have been removed for 2.9 (#65745)
* Update deprecation check.

* Remove support for check_invalid_arguments.
6 years ago
Sam Doran 617fbad743
synchronize - fix password authentication (#66542)
On Python 2, leave all fds open since there is no mechanism to close specific fds with subprocess.Popen() on Python 2

Add unit tests.

Co-authored-by: Matt Martz <matt@sivel.net>
6 years ago
kaorihinata 3ca4580cb4 Allow no_log=False to silence the no_log warnings for module parameters (#64733)
As AnsibleModule._log_invocation is currently implemented, any parameter
with a name that matches PASSWORD_MATCH triggers the no_log warning as a
precaution against parameters that may contain sensitive data, but have not
been marked as sensitive by the module author.

This patch would allow module authors to explicitly mark the aforementioned
parameters as not sensitive thereby bypassing an erroneous warning message,
while still catching parameters which have not been marked at all by the
author.

Adds tests for various no_log states including True, False, and None (as
extracted by AnsibleModule._log_invocation) when applied to an argument with
a name that matches PASSWORD_MATCH.

Fixes: #49465 #64656
6 years ago
Sam Doran e9d29b1fe4 Properly mask no_log values is sub parameters during failure (#63405)
* Get no_log parameters from subspec

* Add changelog and unit tests

* Handle list of dicts in suboptions

Add fancy error message (this will probably haunt me)

* Update unit tests to test for list of dicts in suboptions

* Add integration tests

* Validate parameters in dict and list

In case it comes in as a string

* Make changes based on feedback, fix tests

* Simplify validators since we only need to validate dicts

Add test for suboptions passed in as strings to ensure they get validated properly and turned into a dictionary.

ci_complete

* Add a few more integration tests
6 years ago
Jill R 8d167bdaef Deprecate alias 'thirsty' from all usages (#61245)
* Deprecate alias 'thirsty' from all usages

Fixes: #61236

* Now with added version quoting

* Handle deprecated aliases in mod_utils

* Make alias deprecation a subkey of the canonical arg, and not a separate argument
6 years ago
Matt Clay f510d59943
Support relative imports in AnsiballZ. (#61196) 6 years ago
Felix Fontein cd4736d893 missing_required_lib: add information how to change Python interpreter (#60736)
* Add information how to change Python interpreter used by Ansible.

* Update lib/ansible/module_utils/basic.py

Co-Authored-By: Abhijeet Kasurde <akasurde@redhat.com>

* Make test less dependent on exact message.
6 years ago
Felix Fontein 4a574c4d0c Option parsing: warn if both an option and its alias are specified for a module (#53698)
* Print warning when both an option and its alias is specified.

* Improve output.

* Put warnings into self._warnings directly, resp. use self.warn() when handling subspecs.

* Add changelog.

* Add unit test.
6 years ago
Abhijeet Kasurde 4b152235ff
Typo fixes (#59227)
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
6 years ago
Toshio Kuratomi 33d2728879 Rename python files in hacking/ directory to have .py suffix
ansible-test only passes files which have the .py suffix for sanity
tests on python files.  This change will allow sanity tests to run on
the Python files in hacking/

* Rename test-module to test-module.py
* Symlink test-module for backwards compat since end users may be using
  test-module
* Fix test-module sanity errors that are now triggered
* Rename ansible_profile to ansible-profile.py
* Rename build-ansible
7 years ago
Brian Coca ee4cba1b85 ensure bytes in run_command (#58411)
* ensure we pass bytes to run_command

* changed tests for new behaviour

* dont b the pytest

* fixes by sivel
7 years ago
dpoon a9fcd1d197 Handle mode-setting of symlink to RO filesystem (#58564)
Fix for https://github.com/ansible/ansible/issues/58562
7 years ago
Trishna Guha 2e8a3efccb
Revert nxos, ios, iosxr return_timestamps (#56206)
* Revert "nxos_command:run_commands results failure when commands array size >1 (#52670)"
This reverts commit 0df5b92af3.
* Revert "added timestamps to nxos_command module (#50261)"
This reverts commit e150943314.
* Revert "added timestamps to ios_command module (#50323)"
This reverts commit 2a432a093b.
* Revert "added response_timestamps to iosxr_command module (#50095)"
This reverts commit 2a0c356da9.
7 years ago
Brian Coca bda541fa0d
fix missing attribs with dirct module execution (#53875)
* fix missing attribs with dirct module execution
* also make remote tmp handling smarter
 update tests
* set default if attrib does not exist
* add simple test
7 years ago
Matt Clay 56418cc274 Fix file path encoding bugs on Python 3. 7 years ago
Sam Doran ff88bd82b5
Move type checking methods out of basic.py and add unit tests (#53687)
* Move check_type_str() out of basic.py

* Move check_type_list() out of basic.py

* Move safe_eval() out of basic.py

* Move check_type_dict() out of basic.py

* Move json importing code to common location

* Move check_type_bool() out of basic.py

* Move _check_type_int() out of basic.py

* Move _check_type_float() out of basic.py

* Move _check_type_path() out of basic.py

* Move _check_type_raw() out of basic.py

* Move _check_type_bytes() out of basic.py

* Move _check_type_bits() out of basic.py

* Create text.formatters.py

Move human_to_bytes, bytes_to_human, and _lenient_lowercase out of basic.py into text.formatters.py
Change references in modules to point to function at new location

* Move _check_type_jsonarg() out of basic.py

* Rename json related functions and put them in common.text.converters

Move formatters.py to common.text.formatters.py and update references in modules.

* Rework check_type_str()

Add allow_conversion option to make the function more self-contained.
Move the messaging back to basic.py since those error messages are more relevant to using this function in the context of AnsibleModule and not when using the function in isolation.

* Add unit tests for type checking functions

* Change _lenient_lowercase to lenient_lowercase per feedback
7 years ago
Sam Doran 43a44e6f35
Move utility functions out of basic.py (#51715)
Move the following methods to lib/anisble/module_utils/common/validation.py:

- _count_terms()
- _check_mutually_exclusive()
- _check_required_one_of()
- _check_required_together()
- _check_required_by()
- _check_required_arguments()
- _check_required_if
- fail_on_missing_params() --> create check_missing_parameters()
7 years ago
Brian Coca e280f2f7b0
Try to get correct buffer size to avoid races (#53547)
* Try to get correct buffer size to avoid races

  fixes #51393

* fix test, mock buffer function since all is mocked
7 years ago
Felix Fontein 07fcb60d55 Python 2: accept both long and int for type=int (module options) (#53289)
* Added unit test
7 years ago
Sam Doran aba4bed803
Move _handle_no_log_values() out of basic.py (#48628)
* Rename method and make private
* Use is_iterable, combine transformations
* Remove unused return_values from network modules
* Improve docstrings in new functions
* Add new PASS_VAR
* Add unit tests for list_no_log_values
* Fix unit tests for Python 2.6
7 years ago
Sam Doran 2a98faee2b
Move _handle_aliases() out of basic.py (#48578)
Refinements:
- return legal_inputs and update class properties
- remove redundant arguments from method and handle in caller
- add better exception types to method

* Add unit tests for handle_aliases
7 years ago
Sam Doran 23a6b88dd2
Modify the correct variable when setting available hashing algorithms (#52994)
* Revert "use list instead of tuple and remove md5 on ValueError (#51357)" c459f040da.
* Modify the correct variable when determining available hashing algorithms
7 years ago
Sam Doran cc9c72d6f8 Do not add state: absent when a non-existent path is returned (#51350)
Leave it up to the module to return the state in the results.

I went through all the modules in files/ and only found one case where the module needed to return this. No other modules return paths that do not exists.

Signed-off-by: Sam Doran <sdoran@redhat.com>
7 years ago
Sam Doran f52a088862
Add option to ignore, warn, or error when a module parameter is converted to a string (#51404)
* Add new module property to Windows modules
* Add brief pause to file tests to ensure the stat times are not equal, which was happening sometimes.
* Raise TypeError on error rather than fail_json()
* Rework error message to be less verbose
* Add porting guide entry
7 years ago
Ganesh Nalawade 41e2bd1df5
Add support for elements validation in argspec (#50335)
* Add support for elements validation in argspec

Fixes #48473

*  Add support to validate the elements value in argspec
   when type is `list`

* Fix unit test failures

* Add unit test for elements validation

* Fix CI failures

* Fix review comments

* Fix unit test and CI failures after rebase
7 years ago
Dag Wieers c2fb581414 Fix various sonarcloud issues
This fixes various reported bugs through sonarcloud at:
https://sonarcloud.io/project/issues?id=Rodney-Reis_ansible&resolved=false&types=BUG
7 years ago
Dag Wieers cd9471ef17 Introduce new 'required_by' argument_spec option (#28662)
* Introduce new "required_by' argument_spec option

This PR introduces a new **required_by** argument_spec option which allows you to say *"if parameter A is set, parameter B and C are required as well"*.

- The difference with **required_if** is that it can only add dependencies if a parameter is set to a specific value, not when it is just defined.
- The difference with **required_together** is that it has a commutative property, so: *"Parameter A and B are required together, if one of them has been defined"*.

As an example, we need this for the complex options that the xml module provides. One of the issues we often see is that users are not using the correct combination of options, and then are surprised that the module does not perform the requested action(s).

This would be solved by adding the correct dependencies, and mutual exclusives. For us this is important to get this shipped together with the new xml module in Ansible v2.4. (This is related to bugfix https://github.com/ansible/ansible/pull/28657)

```python
    module = AnsibleModule(
        argument_spec=dict(
            path=dict(type='path', aliases=['dest', 'file']),
            xmlstring=dict(type='str'),
            xpath=dict(type='str'),
            namespaces=dict(type='dict', default={}),
            state=dict(type='str', default='present', choices=['absent',
'present'], aliases=['ensure']),
            value=dict(type='raw'),
            attribute=dict(type='raw'),
            add_children=dict(type='list'),
            set_children=dict(type='list'),
            count=dict(type='bool', default=False),
            print_match=dict(type='bool', default=False),
            pretty_print=dict(type='bool', default=False),
            content=dict(type='str', choices=['attribute', 'text']),
            input_type=dict(type='str', default='yaml', choices=['xml',
'yaml']),
            backup=dict(type='bool', default=False),
        ),
        supports_check_mode=True,
        required_by=dict(
            add_children=['xpath'],
            attribute=['value', 'xpath'],
            content=['xpath'],
            set_children=['xpath'],
            value=['xpath'],
        ),
        required_if=[
            ['count', True, ['xpath']],
            ['print_match', True, ['xpath']],
        ],
        required_one_of=[
            ['path', 'xmlstring'],
            ['add_children', 'content', 'count', 'pretty_print', 'print_match', 'set_children', 'value'],
        ],
        mutually_exclusive=[
            ['add_children', 'content', 'count', 'print_match','set_children', 'value'],
            ['path', 'xmlstring'],
        ],
    )
```

* Rebase and fix conflict

* Add modules that use required_by functionality

* Update required_by schema

* Fix rebase issue
7 years ago
Jordan Borean a39c4ad464
Final round of moving modules to new import error msg (#51852)
* Final round of moving modules to new import error msg

* readd URL to jenkins install guide

* fix unit tests
7 years ago
Mike Sgarbossa c459f040da use list instead of tuple and remove md5 on ValueError (#51357)
* use list instead of tuple and remove md5 on ValueError

Signed-off-by: michael.sgarbossa <msgarbossa@cvs.com>

* convert algorithms to list and add comment

Signed-off-by: michael.sgarbossa <msgarbossa@cvs.com>

* only convert to list if algorithms is not None

Signed-off-by: michael.sgarbossa <msgarbossa@cvs.com>

* new fragment for PR 51357

Signed-off-by: michael.sgarbossa <msgarbossa@cvs.com>

* fix lint: remove blank line
7 years ago
Jordan Borean 0f0d33a954 start getting modules to use shared import error code (#51787) 7 years ago
vaneuk e150943314 added timestamps to nxos_command module (#50261)
* added timestamps to nxos_command module

nxos_command module now returns timestamps field, which shows command execution time

* –fixed unit test failure for /lib/ansible/module_utils/basic

* cosmetic changes to align with PEP 8
7 years ago
Toshio Kuratomi 5844c8c7f0 Cleanups to the common.sys_info API
* Move get_all_subclasses out of sys_info as it is unrelated to system
  information.
* get_all_subclasses now returns a set() instead of a list.
* Don't port get_platform to sys_info as it is deprecated.  Code using
  the common API should just use platform.system() directly.
* Rename load_platform_subclass() to get_platform_subclass and do not
  instantiate the rturned class.
* Test the compat shims in module_utils/basic.py separately from the new
  API in module_utils/common/sys_info.py and module_utils/common/_utils.py
7 years ago
Toshio Kuratomi bd072fe83a
Make the timeout decorator raise an exception out of the function's scope (#49921)
* Revert "allow caller to deal with timeout (#49449)"

This reverts commit 63279823a7.

Flawed on many levels

* Adds poor API to a public function
* Papers over the fact that the public function is doing something bad
  by catching exceptions it cannot handle in the first place
* Papers over the real cause of the issue which is a bug in the timeout
  decorator
* Doesn't reraise properly
* Catches the wrong exception

Fixes #49824
Fixes #49817

* Make the timeout decorator properly raise an exception outside of the function's scope

signal handlers which raise exceptions will never work well because the
exception can be raised anywhere in the called code.  This leads to
exception race conditions where the exceptions could end up being
hanlded by unintended pieces of the called code.

The timeout decorator was using just that idiom.  It was especially bad
because the decorator syntactically occurs outside of the called code
but because of the signal handler, the exception was being raised inside
of the called code.

This change uses a thread instead of a signal to manage the timeout in
parallel to the execution of the decorated function.  Since raising of
the exception happens inside of the decorator, now, instead of inside of
a signal handler, the timeout exception is raised from outside of the
called code as expected which makes reasoning about where exceptions are
to be expected intuitive again.

Fixes #43884

* Add a common case test.

Adding an integration test driven from our unittests.  Most of the time
we'll timeout in run_command which is running things in a subprocess.
Create a test for that specific case in case anything funky comes up
between threading and execve.

* Don't use OSError-based TimeoutError as a base class

Unlike most standard exceptions, OSError has a specific parameter list
with specific meanings.  Instead follow the example of other stdlib
functions, concurrent.futures and multiprocessing and define a separate
TimeoutException.

* Add comment and docstring to point out that this is not hte Python3 TimeoutError
7 years ago
Toshio Kuratomi 3fba006207 Update bare exceptions to specify Exception.
This will keep us from accidentally catching program-exiting exceptions
like KeyboardInterupt and SystemExit.
7 years ago
Brian Coca 63279823a7
allow caller to deal with timeout (#49449)
* allow caller to deal with timeout

  a start for #43884 and timeout issues in long operations
7 years ago
Andreas Krüger 18bf48cec2 Pull documentation of ansible.module_utils.basic from (improved) doc strings. (#48416) 7 years ago
Andreas Calminder 876b637208 move some of basic into common (#48078)
* move file functions into common.file

* move *_PERM_BITS and mark as private (_*_PERM_BITS)

* move get_{platform, distribution, distribution_version} get_all_subclasses and load_platform_subclass into common.sys_info

* forgot get_distribution_version, properly rename get_all_subclasses

* add common/sys_info.py to recursive finder test

* update module paths in test_platform_distribution.py

* update docstrings, _get_all_subclasses -> get_all_subclasses

* forgot to update names

* remove trailing whitespace
7 years ago
Abhijeet Kasurde 789b0ef0c9 Pass string command in run_command (#48805)
When there are spaces in command args passed as a list,
then run_command and underlying subprocess fails.
This can be overcome by passing command as string rather than list.

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
7 years ago
Yauhen Kirylau 14037443de fix(tasks: synchronize): wrap in sshpass if ssh password was provided (#30743)
* fix(tasks: synchronize): wrap in sshpass if ssh password was provided

Closes #16616

* fix(tasks: synchronize): pass rsync password to sshpass via fd

* fix(tasks: synchronize): use fail_json instead of AnsibleError

* fixup! fix(tasks: synchronize): use fail_json instead of AnsibleError

fix python2 handling

* feat(module_utils: basic: run_command): add optional arguments `pass_fds` and `before_communicate_callback`

* fix(tasks: synchronize): use module.run_command instead of subprocess.Popen

* fixup! fix(tasks: synchronize): use module.run_command instead of subprocess.Popen

remove unused import

* fixup! fixup! fix(tasks: synchronize): use module.run_command instead of subprocess.Popen

pass_fds only if they passed to run_command()
7 years ago
Jordan Borean ddfd1dbfc6
Add helper function to return helpful import error msg (#47409) 7 years ago
Jonathan Oddy f2dccb90e8 Restore SIGPIPE handler to DFL on POpen
Python sets the SIGPIPE handler to SIG_IGN. On execv() signal handlers are
reset to their defaults, EXCEPT those that are SIG_IGN which are left ignored.
In Python 3 subprocess.popen explicitly resets the SIGPIPE handler to SIG_DFL,
but unfortunately in Python 2.7 it does not. This leads to subprocesses being
executed with SIGPIPE ignored. This is often a problem with bash scripts which
rely on SIGPIPE to terminate commands in a pipe, but can easily be a problem
with other applications.

This implements the Python 3 behaviour for Python 2.7 by using a preexec_fn.
7 years ago
Dag Wieers 66eec42f53 Fix calling deprecate with correct arguments (#44726)
This fixes #44702
7 years ago
Sviatoslav Sydorenko a978d7f283 Fix collections.deque import in compat module 7 years ago
Matt Martz 07b2698c03 Add new expand_shell argument for run_command, to disable expanding shellisms (#45620)
* Add new expand_shell argument for run_command, to disable expanding shellisms. Fixes #45418

* s/expand_shell/expand_user_and_vars/g
7 years ago
Jordan Borean e07352b9de basic.py: catch ValueError when trying to import hash algorithms (#44551)
* basic.py: catch more Exceptions when trying to import md5 hash algorithms
7 years ago
Matt Martz c1c229c6d4
Remove use of simplejson throughout code base (#43548)
* Remove use of simplejson throughout code base. Fixes #42761

* Address failing tests

* Remove simplejson from contrib and other outlying files

* Add changelog fragment for simplejson removal
7 years ago
Brian Coca 222c907ffb
actually check we can run scm command for roles (#43315)
* actually check we can run scm command for roles
* a better error message than file not found
* more narrow exception hanlding
* refactor common functions for more extended use and further 'basic.py' separation
7 years ago
Toshio Kuratomi 52449cc01a AnsiballZ improvements
Now that we don't need to worry about python-2.4 and 2.5, we can make
some improvements to the way AnsiballZ handles modules.

* Change AnsiballZ wrapper to use import to invoke the module
  We need the module to think of itself as a script because it could be
  coded as:

      main()

  or as:

      if __name__ == '__main__':
          main()

  Or even as:

      if __name__ == '__main__':
          random_function_name()

  A script will invoke all of those.  Prior to this change, we invoked
  a second Python interpreter on the module so that it really was
  a script.  However, this means that we have to run python twice (once
  for the AnsiballZ wrapper and once for the module).  This change makes
  the module think that it is a script (because __name__ in the module ==
  '__main__') but it's actually being invoked by us importing the module
  code.

  There's three ways we've come up to do this.
  * The most elegant is to use zipimporter and tell the import mechanism
    that the module being loaded is __main__:
    * 5959f11c9d/lib/ansible/executor/module_common.py (L175)
    * zipimporter is nice because we do not have to extract the module from
      the zip file and save it to the disk when we do that.  The import
      machinery does it all for us.
    * The drawback is that modules do not have a __file__ which points
      to a real file when they do this.  Modules could be using __file__
      to for a variety of reasons, most of those probably have
      replacements (the most common one is to find a writable directory
      for temporary files.  AnsibleModule.tmpdir should be used instead)
      We can monkeypatch __file__ in fom AnsibleModule initialization
      but that's kind of gross.  There's no way I can see to do this
      from the wrapper.

  * Next, there's imp.load_module():
    * https://github.com/abadger/ansible/blob/340edf7489/lib/ansible/executor/module_common.py#L151
    * imp has the nice property of allowing us to set __name__ to
      __main__ without changing the name of the file itself
    * We also don't have to do anything special to set __file__ for
      backwards compatibility (although the reason for that is the
      drawback):
    * Its drawback is that it requires the file to exist on disk so we
      have to explicitly extract it from the zipfile and save it to
      a temporary file

  * The last choice is to use exec to execute the module:
    * https://github.com/abadger/ansible/blob/f47a4ccc76/lib/ansible/executor/module_common.py#L175
    * The code we would have to maintain for this looks pretty clean.
      In the wrapper we create a ModuleType, set __file__ on it, read
      the module's contents in from the zip file and then exec it.
    * Drawbacks: We still have to explicitly extract the file's contents
      from the zip archive instead of letting python's import mechanism
      handle it.
    * Exec also has hidden performance issues and breaks certain
      assumptions that modules could be making about their own code:
      http://lucumr.pocoo.org/2011/2/1/exec-in-python/

  Our plan is to use imp.load_module() for now, deprecate the use of
  __file__ in modules, and switch to zipimport once the deprecation
  period for __file__ is over (without monkeypatching a fake __file__ in
  via AnsibleModule).

* Rename the name of the AnsiBallZ wrapped module
  This makes it obvious that the wrapped module isn't the module file that
  we distribute.  It's part of trying to mitigate the fact that the module
  is now named __main)).py in tracebacks.

* Shield all wrapper symbols inside of a function
  With the new import code, all symbols in the wrapper become visible in
  the module.  To mitigate the chance of collisions, move most symbols
  into a toplevel function.  The only symbols left in the global namespace
  are now _ANSIBALLZ_WRAPPER and _ansiballz_main.

revised porting guide entry

Integrate code coverage collection into AnsiballZ.

ci_coverage
ci_complete
7 years ago
Dag Wieers 9bf22309b3 Clarify uncertain nature of tracebacks on Python 2 (#42513)
* Only add exception/traceback on Python 3

On Python 2 the traceback could be any exception from the stack frame
and likely unrelated to the fail_json call.

On Python 3 the traceback is cleared outside any exception frame, so the
call always returns the most inner traceback (if any), and therefor is
most likely related to the fail_json call.

* Add uncertainty to traceback on Python 2

On Python 2 the last exception in the stack frame is being returned,
this could be unrelated to the actual error, especially if fail_json()
is called outside an except: block.
7 years ago
Brian Coca 9217fbb7dd
better error messasge (#42770)
* better error messasge
8 years ago
Jordan Borean 8bdd04c147 Fix remote_tmp when become with non admin user (#42396)
* Fix tmpdir on non root become

 - also avoid exception if tmpdir and remote_tmp are None
 - give 'None' on deescalation so tempfile will fallback to it's default behaviour
   and use system dirs
 - fix issue with bad tempdir (not existing/not createable/not writeable)
   i.e nobody and ~/.ansible/tmp
 - added tests for blockfile case

* Revert "Temporarily revert c119d54"

This reverts commit 5c614a59a6.

* changes based on PR feedback and changelog fragment

* changes based on the review

* Fix tmpdir when makedirs failed so we just use the system tmp

* Let missing remote_tmp fail

If remote_tmp is missing then there's something more basic wrong in the
communication from the controller to the module-side.  It's better to
be alerted in this case than to silently ignore it.

jborean and I have independently checked what happens if the user sets
ansible_remote_tmp to empty string and !!null and both cases work fine.
(null is turned into a default value controller-side.  empty string
triggers the warning because it is probably not a directory that the
become user is able to use).
8 years ago
Abhijeet Kasurde 087fc0c20c Restore BOOLEANS import in basic.py (#42008)
This import was removed by mistake. This is required for backward
compatibility.

Fixes: #41988

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
8 years ago
Andreas Calminder 75e9b76c15 accept + and - modifiers for file attributes (#40061)
* accept + and - modifiers for file attributes
* tests for adding/removing file attributes
8 years ago
Hideki Saito d7df072b96 Add syslog_facility parameter handling with systemd.journal (#41078)
* Add syslog_facility parameter handling with systemd.journal

- Fixed issue #41072

Signed-off-by: Hideki Saito <saito@fgrep.org>
8 years ago
Sviatoslav Sydorenko eb209e92c9 Move collections abc shim to _collections_compat 8 years ago
Sviatoslav Sydorenko 912bd25a4e Drop deprecated Python 2.4 compat shim 8 years ago
Sviatoslav Sydorenko e1ae082a1e Partially python3fy ansible.module_utils.basic
Adding
    __metaclass__ = type
breaks things, so I didn't include it.
8 years ago
Toshio Kuratomi 3760cc3a7d Fix python-2.4ism that was still present in basic.py's use of tempfile 8 years ago
Jordan Borean 5c39c3b2d1
Module basic.py to create parent dirs of tmpdir if needed (#40201)
* Module basic.py to create parent dirs of tmpdir if needed

* Added warning to dir creation

* Assert if make_dirs was called or not in unit tests
8 years ago
Jordan Borean 44ab948e5d
create module tmpdir based on remote_tmp (#39833)
* create module tmpdir based on remote_tmp

* Source remote_tmp from controller if possible

* Fixed sanity test and not use lambda

* Added expansion of env vars to the remote tmp

* Fixed sanity issues

* Added note around shell remote_tmp option

* Changed fallback tmp dir to ~/.ansible/tmp to make shell defaults
8 years ago
Toshio Kuratomi e17ee368e3 Improve comments about file_commmon_arguments 8 years ago
Toshio Kuratomi 6b6c4914d2 Initial cleanup of file module
* Remove use of six.b as Python-2.6+ have byte literals.
* Make AnsibleModule a global object so we'll have access to it in all
  the functions we're going to break this up into.
* Rework the parameters so things that are in file_common_args are used
  from file_common_args or the reason for deviation is documented.
* Remove validate as a parameter: this should be taken care of by
  removing it from params before the copy and template action plugin
  invoke file.
* Rename diff_peek to _diff_peek as it is an internal parameter.
* add module_name execute_module call to assemble so that it is more greppable
8 years ago
Toshio Kuratomi b9a00372c2 Categorize FILE_COMMON_ARGS
Some file_common_args are not common to all file modules.  Divide the
args into categories for a future refator of this into a separate
file_argspec.
8 years ago