28 Commits (8b3ca4c5376972f0c52aa96dbe3084046fd3327c)

Author	SHA1	Message	Date
Adrian Likins	297dfb1d50	Vault secrets script client inc new 'keyring' client (#27669) ... This adds a new type of vault-password script (a 'client') that takes advantage of and enhances the multiple vault password support. If a vault password script basename ends with the name '-client', consider it a vault password script client. A vault password script 'client' just means that the script will take a '--vault-id' command line arg. The previous vault password script (as invoked by --vault-password-file pointing to an executable) takes no args and returns the password on stdout. But it doesnt know anything about --vault-id or multiple vault passwords. The new 'protocol' of the vault password script takes a cli arg ('--vault-id') so that it can lookup that specific vault-id and return it's password. Since existing vault password scripts don't know the new 'protocol', a way to distinguish password scripts that do understand the protocol was needed. The convention now is to consider password scripts that are named like 'something-client.py' (and executable) to be vault password client scripts. The new client scripts get invoked with the '--vault-id' they were requested for. An example: ansible-playbook --vault-id my_vault_id@contrib/vault/vault-keyring-client.py some_playbook.yml That will cause the 'contrib/vault/vault-keyring-client.py' script to be invoked as: contrib/vault/vault-keyring-client.py --vault-id my_vault_id The previous vault-keyring.py password script was extended to become vault-keyring-client.py. It uses the python 'keyring' module to request secrets from various backends. The plain 'vault-keyring.py' script would determine which key id and keyring name to use based on values that had to be set in ansible.cfg. So it was also limited to one keyring name. The new vault-keyring-client.py will request the secret for the vault id provided via the '--vault-id' option. The script can be used without config and can be used for multiple keyring ids (and keyrings). On success, a vault password client script will print the password to stdout and exit with a return code of 0. If the 'client' script can't find a secret for the --vault-id, the script will exit with return code of 2 and print an error to stderr.	7 years ago
Adrian Likins	a14d0f3586	Use vault_id when encrypted via vault-edit (#30772) ... * Use vault_id when encrypted via vault-edit On the encryption stage of 'ansible-vault edit --vault-id=someid@passfile somefile', the vault id was not being passed to encrypt() so the files were always saved with the default vault id in the 1.1 version format. When trying to edit that file a second time, also with a --vault-id, the file would be decrypted with the secret associated with the provided vault-id, but since the encrypted file had no vault id in the envelope there would be no match for 'default' secrets. (Only the --vault-id was included in the potential matches, so the vault id actually used to decrypt was not). If that list was empty, there would be an IndexError when trying to encrypted the changed file. This would result in the displayed error: ERROR! Unexpected Exception, this is probably a bug: list index out of range Fix is two parts: 1) use the vault id when encrypting from edit 2) when matching the secret to use for encrypting after edit, include the vault id that was used for decryption and not just the vault id (or lack of vault id) from the envelope. add unit tests for #30575 and intg tests for 'ansible-vault edit' Fixes #30575	7 years ago
Matt Clay	7714dcd04e	Enable more pylint rules and fix reported issues. (#30539) ... * Enable pylint unreachable test. * Enable pylint suppressed-message test. * Enable pylint redundant-unittest-assert test. * Enable pylint bad-open-mode test. * Enable pylint signature-differs test. * Enable pylint unnecessary-pass test. * Enable pylint unnecessary-lambda test. * Enable pylint raising-bad-type test. * Enable pylint logging-not-lazy test. * Enable pylint logging-format-interpolation test. * Enable pylint useless-else-on-loop test.	7 years ago
Adrian Likins	e287af1ac8	Vault secrets empty password (#28186) ... * Better handling of empty/invalid passwords empty password files are global error and cause an exit. A warning is also emitted with more detail. ie, if any of the password/secret sources provide a bogus password (ie, empty) or fail (exception, ctrl-d, EOFError), we stop at the first error and exit. This makes behavior when entering empty password at prompt match 2.3 (ie, an error)	7 years ago
Adrian Likins	220db9cdd2	Better vault pass prompt behav on EOF, more unit tests (#27981)	7 years ago
Adrian Likins	c38ff3b8f8	pylint fixes for vault related code (#27721) ... * rm unneeded parens following assert * rm unused parse_vaulttext_envelope from yaml.constructor * No longer need index/enumerate over vault_ids * rm unnecessary else * rm unused VaultCli.secrets * rm unused vault_id arg on VaultAES.decrypt() pylint: Unused argument 'vault_id' pylint: Unused parse_vaulttext_envelope imported from ansible.parsing.vault pylint: Unused variable 'index' pylint: Unnecessary parens after 'assert' keyword pylint: Unnecessary "else" after "return" (no-else-return) pylint: Attribute 'editor' defined outside __init__ * use 'dummy' for unused variables instead of _ Based on pylint unused variable warnings. Existing code use '_' for this, but that is old and busted. The hot new thing is 'dummy'. It is so fetch. Except for where we get warnings for reusing the 'dummy' var name inside of a list comprehension. * Add super().__init__ call to PromptVaultSecret.__init__ pylint: __init__ method from base class 'VaultSecret' is not called (super-init-not-called) * Make FileVaultSecret.read_file reg method again The base class read_file() doesnt need self but the sub classes do. Rm now unneeded loader arg to read_file() * Fix err msg string literal that had no effect pylint: String statement has no effect The indent on the continuation of the msg_format was wrong so the second half was dropped. There was also no need to join() filename (copy/paste from original with a command list I assume...) * Use local cipher_name in VaultEditor.edit_file not instance pylint: Unused variable 'cipher_name' pylint: Unused variable 'b_ciphertext' Use the local cipher_name returned from parse_vaulttext_envelope() instead of the instance self.cipher_name var. Since there is only one valid cipher_name either way, it was equilivent, but it will not be with more valid cipher_names * Rm unused b_salt arg on VaultAES256._encrypt* pylint: Unused argument 'b_salt' Previously the methods computed the keys and iv themselves so needed to be passed in the salt, but now the key/iv are built before and passed in so b_salt arg is not used anymore. * rm redundant import of call from subprocess pylint: Imports from package subprocess are not grouped use via subprocess module now instead of direct import. * self._bytes is set in super init now, rm dup * Make FileVaultSecret.read_file() -> _read_file() _read_file() is details of the implementation of load(), so now 'private'.	7 years ago
Adrian Likins	2b0a7338d4	Handle win style CRLF newlines in vault text (#27590) ... When parsing a vaulttext blob, use .splitlines() instead of split(b'\n') to handle \n newlines and windows style \r\n (CRLF) new lines. The vaulttext enevelope at this point is just the header line and a hexlify()'ed blob, so CRLF is a valid newline here. Fixes #22914	7 years ago
Adrian Likins	934b645191	Support multiple vault passwords (#22756) ... Fixes #13243 ** Add --vault-id to name/identify multiple vault passwords Use --vault-id to indicate id and path/type --vault-id=prompt # prompt for default vault id password --vault-id=myorg@prompt # prompt for a vault_id named 'myorg' --vault-id=a_password_file # load ./a_password_file for default id --vault-id=myorg@a_password_file # load file for 'myorg' vault id vault_id's are created implicitly for existing --vault-password-file and --ask-vault-pass options. Vault ids are just for UX purposes and bookkeeping. Only the vault payload and the password bytestring is needed to decrypt a vault blob. Replace passing password around everywhere with a VaultSecrets object. If we specify a vault_id, mention that in password prompts Specifying multiple -vault-password-files will now try each until one works ** Rev vault format in a backwards compatible way The 1.2 vault format adds the vault_id to the header line of the vault text. This is backwards compatible with older versions of ansible. Old versions will just ignore it and treat it as the default (and only) vault id. Note: only 2.4+ supports multiple vault passwords, so while earlier ansible versions can read the vault-1.2 format, it does not make them magically support multiple vault passwords. use 1.1 format for 'default' vault_id Vaulted items that need to include a vault_id will be written in 1.2 format. If we set a new DEFAULT_VAULT_IDENTITY, then the default will use version 1.2 vault will only use a vault_id if one is specified. So if none is specified and C.DEFAULT_VAULT_IDENTITY is 'default' we use the old format. ** Changes/refactors needed to implement multiple vault passwords raise exceptions on decrypt fail, check vault id early split out parsing the vault plaintext envelope (with the sha/original plaintext) to _split_plaintext_envelope() some cli fixups for specifying multiple paths in the unfrack_paths optparse callback fix py3 dict.keys() 'dict_keys object is not indexable' error pluralize cli.options.vault_password_file -> vault_password_files pluralize cli.options.new_vault_password_file -> new_vault_password_files pluralize cli.options.vault_id -> cli.options.vault_ids ** Add a config option (vault_id_match) to force vault id matching. With 'vault_id_match=True' and an ansible vault that provides a vault_id, then decryption will require that a matching vault_id is required. (via --vault-id=my_vault_id@password_file, for ex). In other words, if the config option is true, then only the vault secrets with matching vault ids are candidates for decrypting a vault. If option is false (the default), then all of the provided vault secrets will be selected. If a user doesn't want all vault secrets to be tried to decrypt any vault content, they can enable this option. Note: The vault id used for the match is not encrypted or cryptographically signed. It is just a label/id/nickname used for referencing a specific vault secret.	7 years ago
Toshio Kuratomi	e238ae999b	Cyptography pr 20566 rebase (#25560) ... Make pyca/cryptography the preferred backend for cryptographic needs (mainly vault) falling back to pycrypto pyca/cryptography is already implicitly a dependency in many cases through paramiko (2.0+) as well as the new openssl_publickey module, which requires pyOpenSSL 16.0+. Additionally, pyca/cryptography is an optional dep for better performance with vault already. This commit leverages cryptography's padding, constant time comparisons, and CBC/CTR modes to reduce the amount of code ansible needs to maintain. * Handle wrong password given for VaultAES format * Do not display deprecation warning for cryptography on python-2.6 * Namespace all of the pycrypto imports and always import them Makes unittests better and the code less likely to get stupid mistakes (like using HMAC from cryptogrpahy when the one from pycrypto is needed) * Add back in atfork since we need pycrypto to reinitialize its RNG just in case we're being used with old paramiko * contrib/inventory/gce: Remove spurious require on pycrypto (cherry picked from commit 9e16b9db275263b3ea8d1b124966fdebfc9ab271) * Add cryptography to ec2_win_password module requirements * Fix python3 bug which would pass text strings to a function which requires byte strings. * Attempt to add pycrypto version to setup deps * Change hacking README for dual pycrypto/cryptography * update dependencies for various CI scripts * additional CI dockerfile/script updates * add paramiko to the windows and sanity requirement set This is needed because ansible lists it as a requirement. Previously the missing dep wasn't enforced, but cryptography imports pkg_resources so you can't ignore a requirement any more * Add integration test cases for old vault and for wrong passwords * helper script for manual testing of pycrypto/cryptography * Skip the pycrypto tests so that users without it installed can still run the unittests * Run unittests for vault with both cryptography and pycrypto backend	8 years ago
Abhijeet Kasurde	b89cb95609	Fix spelling mistakes (comments only) (#25564) ... Original Author : klemens <ka7@github.com> Taking over previous PR as per https://github.com/ansible/ansible/pull/23644#issuecomment-307334525 Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>	8 years ago
Dag Wieers	4efec414e7	test/: PEP8 compliancy (#24803) ... * test/: PEP8 compliancy - Make PEP8 compliant * Python3 chokes on casting int to bytes (#24952) But if we tell the formatter that the var is a number, it works	8 years ago
Adrian Likins	ae3d7fb29e	Fix vault reading from stdin (avoid realpath() on non-links) (#23583) ... * Fix vault reading from stdin (avoid realpath() on non-links) os.path.realpath() is used to find the target of file paths that are symlinks so vault operations happen directly on the target. However, in addition to resolving symlinks, realpath() also returns a full path. when reading from stdin, vault cli uses '-' as a special file path so VaultEditor() will replace with stdin. realpath() was expanding '-' with the CWD to something like '/home/user/playbooks/-' causing errors like: ERROR! [Errno 2] No such file or directory: u'/home/user/ansible/-' Fix is to specialcase '-' to not use realpath() Fixes #23567 * to_text decrypt output when writing to stdout	8 years ago
Toshio Kuratomi	2fff690caa	Update module_utils.six to latest (#22855) ... * Update module_utils.six to latest We've been held back on the version of six we could use on the module side to 1.4.x because of python-2.4 compatibility. Now that our minimum is Python-2.6, we can update to the latest version of six in module_utils and get rid of the second copy in lib/ansible/compat.	8 years ago
Adrian Likins	6c6b647182	Fix 'ansible-vault edit /some/symlink' (#20417) ... Since vault edit attempts to unlink edited files before creating a new file with the same name and writing to it, if the file was a symlink, the symlink would be replaced with a regular file. VaultEditor file ops now check if files it is changing are symlinks and instead works directly on the target, so that os.rename() and shutils do the right thing. Add unit tests cases for this case and assorted VaultEditor test cases. Fixes #20264	8 years ago
Matt Clay	8c270ac75f	Add empty-init code-smell script. (#18406) ... Also removed boilerplate from otherwise empty __init__.py files which should not contain any code (checked by empty-init script).	8 years ago
Toshio Kuratomi	e70066a6f7	Many Cleanups to vault ... * Make is_encrypted_file handle both files opened in text and binary mode On python3, by default files are opened in text mode. Since we know the encoding of vault files (and especially the header which is the first set of bytes) we can decide whether the file is an encrypted vault file in either case. * Fix is_encrypted_file not resetting the file position * Update is_encrypted_file to check that all the data in the file is ascii * For is_encrypted_file(), add start_pos and count parameters This allows callers to specify reading vaulttext from the middle of a file if necessary. * Combine VaultLib.encrypt() and VaultLib.encrypt_bytestring() * Change vault's is_encrypted() to take either text or byte strings and to return False if any part of the data is non-ascii. * Remove unnecessary use of six.b * Vault Cipher: mark a few methods as private. * VaultAES256._is_equal throws a TypeError if given non byte strings * Make VaultAES256 methods that don't need self staticmethods and classmethods * Mark VaultAES and is_encrypted as deprecated * Get rid of VaultFile (unused and feature implemented in a different way) * Normalize variable and parameter names on plaintext, ciphertext, vaulttext * Normalize variable and parameter names on "b_" prefix when dealing with bytes * Test changes: * Remove redundant tests( both checking the same byte string) * Fix use of format string without format operator * Enable vault editor tests on python3 * Initialize the vault_cipher for VaultAES256 testing in setUp() * Make assertTrue and assertFalse take the actual method calls for better error messages. * Test that non-ascii byte strings compare correctly. * Test that unicode strings and ints raise TypeError * Test-specific: * Removed test_methods_exist(). We only have one VaultLib so the implementation is the assurance that the methods exist. (Can use an abc for this if it changes). * Add tests for both byte string and text string input where the API takes either. * Convert "assert" to unittest assert functions or add a custom message where that will make failures easier to debug. * Move instantiating the VaultLib into setUp().	8 years ago
Toshio Kuratomi	4ed88512e4	Move uses of to_bytes, to_text, to_native to use the module_utils version (#17423) ... We couldn't copy to_unicode, to_bytes, to_str into module_utils because of licensing. So once created it we had two sets of functions that did the same things but had different implementations. To remedy that, this change removes the ansible.utils.unicode versions of those functions.	8 years ago
Toshio Kuratomi	44d979c8f5	Enable most unittests on python3 (just some vault unittests and a logging one left) (#17240) ... Make some python3 fixes to make the unittests pass: * galaxy imports * dictionary iteration in role requirements * swap_stdout helper for unittests * Normalize to text string in a facts.py function	8 years ago
Adrian Likins	e396d5d508	Implement vault encrypted yaml variables. (#16274) ... Make !vault-encrypted create a AnsibleVaultUnicode yaml object that can be used as a regular string object. This allows a playbook to include a encrypted vault blob for the value of a yaml variable. A 'secret_password' variable can have it's value encrypted instead of having to vault encrypt an entire vars file. Add __ENCRYPTED__ to the vault yaml types so template.Template can treat it similar to __UNSAFE__ flags. vault.VaultLib api changes: - Split VaultLib.encrypt to encrypt and encrypt_bytestring - VaultLib.encrypt() previously accepted the plaintext data as either a byte string or a unicode string. Doing the right thing based on the input type would fail on py3 if given a arg of type 'bytes'. To simplify the API, vaultlib.encrypt() now assumes input plaintext is a py2 unicode or py3 str. It will encode to utf-8 then call the new encrypt_bytestring(). The new methods are less ambiguous. - moved VaultLib.is_encrypted logic to vault module scope and split to is_encrypted() and is_encrypted_file(). Add a test/unit/mock/yaml_helper.py It has some helpers for testing parsing/yaml Integration tests added as roles test_vault and test_vault_embedded	8 years ago
Marius Gedminas	ec3ada1cda	Fix test on Python 3: vault code expects bytes ... (All tests now succeed on Python 3.5)	9 years ago
Marius Gedminas	5c70f932bd	Fix test on Python 3: vault code expects bytes ... (Third failing test out of four.)	9 years ago
Marius Gedminas	a1d95536f9	Fix test on Python 3: vault code expects bytes ... (Different test than the last commit.)	9 years ago
Marius Gedminas	f58f0c62e1	Fix test on Python 3: vault code expects bytes	9 years ago
Abhijit Menon-Sen	4f3a98eff6	Update Vault tests to make sure AES decryption works ... Note that this test was broken in devel because it was really just duplicating the AES256 test because setting v.cipher_name to 'AES' no longer selected AES after it was de-write-whitelisted. Now that we've removed the VaultAES encryption code, we embed static output from an earlier version and test that we can decrypt it.	9 years ago
Abhijit Menon-Sen	b84053019a	Make the filename the first argument to rekey_file	9 years ago
Abhijit Menon-Sen	c4b2540ecc	Update tests for VaultEditor API changes	9 years ago
Toshio Kuratomi	a3fd4817ef	Unicode and other fixes for vault	9 years ago
James Cammarata	ce3ef7f4c1	Making the switch to v2	10 years ago