ansible

Commit Graph

Author	SHA1	Message	Date
Matt Clay	4e489d1be8	Update Shippable integration test groups. (#43118 ) * Update Shippable integration test groups. * Update integration test group aliases. * Rebalance AWS and Azure tests with extra group. * Rebalance Windows tests with another group.	6 years ago
Adrian Likins	1613a739ad	fix decrypted vault utf8 values (#37539 ) * Fix errors decrypted non-ascii vault vars AnsibleVaultEncryptedUnicode was just using b"".decode() instead of to_text() on the bytestrings returned from vault.decrypt() and could cause errors on python2 if non-ascii since decode() defaults to ascii. Use to_text() to default to decoding utf-8. add intg and unit tests for value of vaulted vars being non-ascii utf8 based on https://github.com/ansible/ansible/issues/37258 Fixes #37258 * yamllint fixups	7 years ago
Matt Clay	c262dbfd30	Use https for links to ansible.com domains.	7 years ago
Adrian Likins	6e737c8cb6	Fix 'New Vault password' on vault 'edit' (#35923 ) * Fix 'New Vault password' on vault 'edit' `ffe0ddea96` introduce a change on 'ansible-vault edit' that tried to check for --encrypt-vault-id in that mode. But '--encrypt-vault-id' is not intended for 'edit' since the 'edit' should always reuse the vault secret that was used to decrypt the text. Change cli to not check for --encrypt-vault-id on 'edit'. VaultLib.decrypt_and_get_vault_id() was change to return the vault secret used to decrypt (in addition to vault_id and the plaintext). VaultEditor.edit_file() will now use 'vault_secret_used' as returned from decrypt_and_get_vault_id() so that an edited file always gets reencrypted with the same secret, regardless of any vault id configuration or cli options. Fixes #35834	7 years ago
Adrian Likins	ffe0ddea96	add a vault --encrypt-vault-to specify vault id to use for encrypt (#31067 ) Enforce that there can be only one --new-vault-id or --new-vault-password-file and use this instead of --encrypt-vault-id * Add a config option for default vault encrypt id	7 years ago
Adrian Likins	fc180a378a	Support using vault password files that are themselves vault encrypted (#27668 ) Extract vault related bits of DataLoader._get_file_contents to DataLoader._decrypt_if_vault_data When loading vault password files, detect if they are vault encrypted, and if so, try to decrypt with any already known vault secrets. This implements the 'Allow vault password files to be vault encrypted' (#31002) feature card from the 2.5.0 project at https://github.com/ansible/ansible/projects/9 Fixes #31002	7 years ago
Adrian Likins	86dc3c09ac	Fix vault --ask-vault-pass with no tty (#31493 ) * Fix vault --ask-vault-pass with no tty 2.4.0 added a check for isatty() that would skip setting up interactive vault password prompts if not running on a tty. But... getpass.getpass() will fallback to reading from stdin if it gets that far without a tty. Since 2.4.0 skipped the interactive prompts / getpass.getpass() in that case, it would never get a chance to fall back to stdin. So if 'echo $VAULT_PASSWORD\| ansible-playbook --ask-vault-pass site.yml' was ran without a tty (ie, from a jenkins job or via the vagrant ansible provisioner) the 2.4 behavior was different than 2.3. 2.4 would never read the password from stdin, resulting in a vault password error like: ERROR! Attempting to decrypt but no vault secrets found Fix is just to always call the interactive password prompts based on getpass.getpass() on --ask-vault-pass or --vault-id @prompt and let getpass sort it out. * up test_prompt_no_tty to expect prompt with no tty We do call the PromptSecret class if there is no tty, but we are back to expecting it to read from stdin in that case. * Fix logic for when to auto-prompt vault pass If --ask-vault-pass is used, then pretty much always prompt. If it is not used, then prompt if there are no other vault ids provided and 'auto_prompt==True'. Fixes vagrant bug https://github.com/hashicorp/vagrant/issues/9033 Fixes #30993	7 years ago
Adrian Likins	9c58827410	Better handling of malformed vault data envelope (#32515 ) * Better handling of malformed vault data envelope If an embedded vaulted variable ('!vault' in yaml) had an invalid format, it would eventually cause an error for seemingly unrelated reasons. "Invalid" meaning not valid hexlify (extra chars, non-hex chars, etc). For ex, if a host_vars file had invalid vault format variables, on py2, it would cause an error like: 'ansible.vars.hostvars.HostVars object' has no attribute u'broken.example.com' Depending on where the invalid vault is, it could also cause "VARIABLE IS NOT DEFINED!". The behavior can also change if ansible-playbook is py2 or py3. Root cause is errors from binascii.unhexlify() not being handled consistently. Fix is to add a AnsibleVaultFormatError exception and raise it on any unhexlify() errors and to handle it properly elsewhere. Add a _unhexlify() that try/excepts around a binascii.unhexlify() and raises an AnsibleVaultFormatError on invalid vault data. This is so the same exception type is always raised for this case. Previous it was different between py2 and py3. binascii.unhexlify() raises a binascii.Error if the hexlified blobs in a vault data blob are invalid. On py2, binascii.Error is a subclass of Exception. On py3, binascii.Error is a subclass of TypeError When decrypting content of vault encrypted variables, if a binascii.Error is raised it propagates up to playbook.base.Base.post_validate(). post_validate() handles exceptions for TypeErrors but not for base Exception subclasses (like py2 binascii.Error). * Add a display.warning on vault format errors * Unit tests for _unhexlify, parse_vaulttext* * Add intg test cases for invalid vault formats Fixes #28038	7 years ago
Adrian Likins	297dfb1d50	Vault secrets script client inc new 'keyring' client (#27669 ) This adds a new type of vault-password script (a 'client') that takes advantage of and enhances the multiple vault password support. If a vault password script basename ends with the name '-client', consider it a vault password script client. A vault password script 'client' just means that the script will take a '--vault-id' command line arg. The previous vault password script (as invoked by --vault-password-file pointing to an executable) takes no args and returns the password on stdout. But it doesnt know anything about --vault-id or multiple vault passwords. The new 'protocol' of the vault password script takes a cli arg ('--vault-id') so that it can lookup that specific vault-id and return it's password. Since existing vault password scripts don't know the new 'protocol', a way to distinguish password scripts that do understand the protocol was needed. The convention now is to consider password scripts that are named like 'something-client.py' (and executable) to be vault password client scripts. The new client scripts get invoked with the '--vault-id' they were requested for. An example: ansible-playbook --vault-id my_vault_id@contrib/vault/vault-keyring-client.py some_playbook.yml That will cause the 'contrib/vault/vault-keyring-client.py' script to be invoked as: contrib/vault/vault-keyring-client.py --vault-id my_vault_id The previous vault-keyring.py password script was extended to become vault-keyring-client.py. It uses the python 'keyring' module to request secrets from various backends. The plain 'vault-keyring.py' script would determine which key id and keyring name to use based on values that had to be set in ansible.cfg. So it was also limited to one keyring name. The new vault-keyring-client.py will request the secret for the vault id provided via the '--vault-id' option. The script can be used without config and can be used for multiple keyring ids (and keyrings). On success, a vault password client script will print the password to stdout and exit with a return code of 0. If the 'client' script can't find a secret for the --vault-id, the script will exit with return code of 2 and print an error to stderr.	7 years ago
Adrian Likins	278ff19bea	Handle vault decrypt --output=- (#31066 ) In cli.CLI.unfrack_path callback, special case if the value of '--output' is '-', and avoid expanding it to a full path. vault cli already has special cases for '-', so it just needs to get the original value to work. Fixes #30550	7 years ago
Adrian Likins	a14d0f3586	Use vault_id when encrypted via vault-edit (#30772 ) * Use vault_id when encrypted via vault-edit On the encryption stage of 'ansible-vault edit --vault-id=someid@passfile somefile', the vault id was not being passed to encrypt() so the files were always saved with the default vault id in the 1.1 version format. When trying to edit that file a second time, also with a --vault-id, the file would be decrypted with the secret associated with the provided vault-id, but since the encrypted file had no vault id in the envelope there would be no match for 'default' secrets. (Only the --vault-id was included in the potential matches, so the vault id actually used to decrypt was not). If that list was empty, there would be an IndexError when trying to encrypted the changed file. This would result in the displayed error: ERROR! Unexpected Exception, this is probably a bug: list index out of range Fix is two parts: 1) use the vault id when encrypting from edit 2) when matching the secret to use for encrypting after edit, include the vault id that was used for decryption and not just the vault id (or lack of vault id) from the envelope. add unit tests for #30575 and intg tests for 'ansible-vault edit' Fixes #30575	7 years ago
Adrian Likins	1f962bd937	Fix config value type for VAULT_IDENTITY_LIST (#28678 ) Was using the 'value_type' key, but didn't get updated to the new 'type' key in merge. Fix playbooks cli so it uses VAULT_IDENTITY_LIST as well.	7 years ago
Adrian Likins	5739bb075f	Vault secrets default vault ids list (#28190 ) * Add config option for a default list of vault-ids This is the vault-id equilivent of ANSIBLE_DEFAULT_PASSWORD_FILE except ANSIBLE_DEFAULT_VAULT_IDENTITY_LIST is a list.	7 years ago
Adrian Likins	e287af1ac8	Vault secrets empty password (#28186 ) * Better handling of empty/invalid passwords empty password files are global error and cause an exit. A warning is also emitted with more detail. ie, if any of the password/secret sources provide a bogus password (ie, empty) or fail (exception, ctrl-d, EOFError), we stop at the first error and exit. This makes behavior when entering empty password at prompt match 2.3 (ie, an error)	7 years ago
Adrian Likins	75a8be9a5d	Add back support for vault_password_file config var (#27597 ) Got removed in arg parsing updates. Now added back in setup_vault_secrets(). The default value for DEFAULT_VAULT_PASSWORD_FILE was also set to '~' for some reason, change to to no default. Add integration tests.	7 years ago
Adrian Likins	934b645191	Support multiple vault passwords (#22756 ) Fixes #13243 Add --vault-id to name/identify multiple vault passwords Use --vault-id to indicate id and path/type --vault-id=prompt # prompt for default vault id password --vault-id=myorg@prompt # prompt for a vault_id named 'myorg' --vault-id=a_password_file # load ./a_password_file for default id --vault-id=myorg@a_password_file # load file for 'myorg' vault id vault_id's are created implicitly for existing --vault-password-file and --ask-vault-pass options. Vault ids are just for UX purposes and bookkeeping. Only the vault payload and the password bytestring is needed to decrypt a vault blob. Replace passing password around everywhere with a VaultSecrets object. If we specify a vault_id, mention that in password prompts Specifying multiple -vault-password-files will now try each until one works Rev vault format in a backwards compatible way The 1.2 vault format adds the vault_id to the header line of the vault text. This is backwards compatible with older versions of ansible. Old versions will just ignore it and treat it as the default (and only) vault id. Note: only 2.4+ supports multiple vault passwords, so while earlier ansible versions can read the vault-1.2 format, it does not make them magically support multiple vault passwords. use 1.1 format for 'default' vault_id Vaulted items that need to include a vault_id will be written in 1.2 format. If we set a new DEFAULT_VAULT_IDENTITY, then the default will use version 1.2 vault will only use a vault_id if one is specified. So if none is specified and C.DEFAULT_VAULT_IDENTITY is 'default' we use the old format. Changes/refactors needed to implement multiple vault passwords raise exceptions on decrypt fail, check vault id early split out parsing the vault plaintext envelope (with the sha/original plaintext) to _split_plaintext_envelope() some cli fixups for specifying multiple paths in the unfrack_paths optparse callback fix py3 dict.keys() 'dict_keys object is not indexable' error pluralize cli.options.vault_password_file -> vault_password_files pluralize cli.options.new_vault_password_file -> new_vault_password_files pluralize cli.options.vault_id -> cli.options.vault_ids Add a config option (vault_id_match) to force vault id matching. With 'vault_id_match=True' and an ansible vault that provides a vault_id, then decryption will require that a matching vault_id is required. (via --vault-id=my_vault_id@password_file, for ex). In other words, if the config option is true, then only the vault secrets with matching vault ids are candidates for decrypting a vault. If option is false (the default), then all of the provided vault secrets will be selected. If a user doesn't want all vault secrets to be tried to decrypt any vault content, they can enable this option. Note: The vault id used for the match is not encrypted or cryptographically signed. It is just a label/id/nickname used for referencing a specific vault secret.	7 years ago
Toshio Kuratomi	e238ae999b	Cyptography pr 20566 rebase (#25560 ) Make pyca/cryptography the preferred backend for cryptographic needs (mainly vault) falling back to pycrypto pyca/cryptography is already implicitly a dependency in many cases through paramiko (2.0+) as well as the new openssl_publickey module, which requires pyOpenSSL 16.0+. Additionally, pyca/cryptography is an optional dep for better performance with vault already. This commit leverages cryptography's padding, constant time comparisons, and CBC/CTR modes to reduce the amount of code ansible needs to maintain. * Handle wrong password given for VaultAES format * Do not display deprecation warning for cryptography on python-2.6 * Namespace all of the pycrypto imports and always import them Makes unittests better and the code less likely to get stupid mistakes (like using HMAC from cryptogrpahy when the one from pycrypto is needed) * Add back in atfork since we need pycrypto to reinitialize its RNG just in case we're being used with old paramiko * contrib/inventory/gce: Remove spurious require on pycrypto (cherry picked from commit 9e16b9db275263b3ea8d1b124966fdebfc9ab271) * Add cryptography to ec2_win_password module requirements * Fix python3 bug which would pass text strings to a function which requires byte strings. * Attempt to add pycrypto version to setup deps * Change hacking README for dual pycrypto/cryptography * update dependencies for various CI scripts * additional CI dockerfile/script updates * add paramiko to the windows and sanity requirement set This is needed because ansible lists it as a requirement. Previously the missing dep wasn't enforced, but cryptography imports pkg_resources so you can't ignore a requirement any more * Add integration test cases for old vault and for wrong passwords * helper script for manual testing of pycrypto/cryptography * Skip the pycrypto tests so that users without it installed can still run the unittests * Run unittests for vault with both cryptography and pycrypto backend	8 years ago
Matt Martz	004e99316c	Allow template files to be vaulted (#22951 ) * Allow template files to be vaulted * Make sure to import exceptions we need * get_real_file can't take bytes, since it looks specifically for string_types * Now that we aren't using open() we don't need b_source * Expand playbooks_vault docs to include modules that support vaulted src files * Add vaulted template test	8 years ago
Adrian Likins	ae3d7fb29e	Fix vault reading from stdin (avoid realpath() on non-links) (#23583 ) * Fix vault reading from stdin (avoid realpath() on non-links) os.path.realpath() is used to find the target of file paths that are symlinks so vault operations happen directly on the target. However, in addition to resolving symlinks, realpath() also returns a full path. when reading from stdin, vault cli uses '-' as a special file path so VaultEditor() will replace with stdin. realpath() was expanding '-' with the CWD to something like '/home/user/playbooks/-' causing errors like: ERROR! [Errno 2] No such file or directory: u'/home/user/ansible/-' Fix is to specialcase '-' to not use realpath() Fixes #23567 * to_text decrypt output when writing to stdout	8 years ago
Adrian Likins	a2907c4d5c	Add smoke/intg tests for vault cli (#21679 ) Add smoke/intg tests for vault cli	8 years ago
Brian Coca	a2c38c47aa	added docs for vault and made trigger shorter: !vault (#20985 ) * added docs for vault and made trigger shorter: !vault * added single var valuting * Update playbooks_vault.rst Edit pass for spelling and grammar. Ship it! * Update playbooks_vault.rst Typo fixes.	8 years ago
Matt Clay	6bbd92e422	Initial ansible-test implementation. (#18556 )	8 years ago
Adrian Likins	25aa757e80	add a intg test for vault encrypted inventory (#18550 )	8 years ago
Adrien Vergé	0e834fc9e4	Fix cosmetic problems in YAML source This change corrects problems reported by the `yamllint` linter. Since key duplication problems were removed in `4d48711`, this commit mainly fixes trailing spaces and extra empty lines at beginning/end of files.	8 years ago
Matt Clay	80a5c70ad7	Split integration tests out from Makefile. (#17976 )	8 years ago

25 Commits (9858f4162857b61ec5079276f97cd80f75e427f8)