Commit Graph

26 Commits (24d7ccd182004e73b655226fb34e2079e21bb68f)

Author SHA1 Message Date
Matt Martz 617372f8c0
Mass nuke deprecated items that are easily removed. ci_complete (#44320) 6 years ago
Matt Clay 4e489d1be8
Update Shippable integration test groups. (#43118)
* Update Shippable integration test groups.
* Update integration test group aliases.
* Rebalance AWS and Azure tests with extra group.
* Rebalance Windows tests with another group.
6 years ago
Adrian Likins 1613a739ad fix decrypted vault utf8 values (#37539)
* Fix errors decrypted non-ascii vault vars

AnsibleVaultEncryptedUnicode was just using b"".decode()
instead of to_text() on the bytestrings returned from
vault.decrypt() and could cause errors on python2
if non-ascii since decode() defaults to ascii.
Use to_text() to default to decoding utf-8.

add intg and unit tests for value of vaulted vars
being non-ascii utf8

based on https://github.com/ansible/ansible/issues/37258

Fixes #37258

* yamllint fixups
7 years ago
Matt Clay c262dbfd30 Use https for links to ansible.com domains. 7 years ago
Adrian Likins 6e737c8cb6
Fix 'New Vault password' on vault 'edit' (#35923)
* Fix 'New Vault password' on vault 'edit'

ffe0ddea96 introduce a
change on 'ansible-vault edit' that tried to check
for --encrypt-vault-id in that mode. But '--encrypt-vault-id'
is not intended for 'edit' since the 'edit' should always
reuse the vault secret that was used to decrypt the text.

Change cli to not check for --encrypt-vault-id on 'edit'.

VaultLib.decrypt_and_get_vault_id() was change to return
the vault secret used to decrypt (in addition to vault_id
and the plaintext).

VaultEditor.edit_file() will now use 'vault_secret_used'
as returned from decrypt_and_get_vault_id() so that
an edited file always gets reencrypted with the same
secret, regardless of any vault id configuration or
cli options.

Fixes #35834
7 years ago
Adrian Likins ffe0ddea96
add a vault --encrypt-vault-to specify vault id to use for encrypt (#31067)
Enforce that there can be only one --new-vault-id or
--new-vault-password-file and use this instead of
--encrypt-vault-id

* Add a config option for default vault encrypt id
7 years ago
Adrian Likins fc180a378a
Support using vault password files that are themselves vault encrypted (#27668)
Extract vault related bits of DataLoader._get_file_contents to DataLoader._decrypt_if_vault_data

When loading vault password files, detect if they are vault encrypted, and if so, try to decrypt with any already known vault secrets.

This implements the 'Allow vault password files to be vault encrypted'  (#31002) feature card from
the 2.5.0 project at https://github.com/ansible/ansible/projects/9

Fixes #31002
7 years ago
Adrian Likins 86dc3c09ac
Fix vault --ask-vault-pass with no tty (#31493)
* Fix vault --ask-vault-pass with no tty

2.4.0 added a check for isatty() that would skip setting up interactive
vault password prompts if not running on a tty.

But... getpass.getpass() will fallback to reading from stdin if
it gets that far without a tty. Since 2.4.0 skipped the interactive
prompts / getpass.getpass() in that case, it would never get a chance
to fall back to stdin.

So if 'echo $VAULT_PASSWORD| ansible-playbook --ask-vault-pass site.yml'
was ran without a tty (ie, from a jenkins job or via the vagrant
ansible provisioner) the 2.4 behavior was different than 2.3. 2.4
would never read the password from stdin, resulting in a vault password
error like:

        ERROR! Attempting to decrypt but no vault secrets found

Fix is just to always call the interactive password prompts based
on getpass.getpass() on --ask-vault-pass or --vault-id @prompt and
let getpass sort it out.

* up test_prompt_no_tty to expect prompt with no tty

We do call the PromptSecret class if there is no tty, but
we are back to expecting it to read from stdin in that case.

* Fix logic for when to auto-prompt vault pass

If --ask-vault-pass is used, then pretty much always
prompt.

If it is not used, then prompt if there are no other
vault ids provided and 'auto_prompt==True'.

Fixes vagrant bug https://github.com/hashicorp/vagrant/issues/9033

Fixes #30993
7 years ago
Adrian Likins 9c58827410
Better handling of malformed vault data envelope (#32515)
* Better handling of malformed vault data envelope

If an embedded vaulted variable ('!vault' in yaml)
had an invalid format, it would eventually cause
an error for seemingly unrelated reasons.
"Invalid" meaning not valid hexlify (extra chars,
non-hex chars, etc).

For ex, if a host_vars file had invalid vault format
variables, on py2, it would cause an error like:

  'ansible.vars.hostvars.HostVars object' has no
  attribute u'broken.example.com'

Depending on where the invalid vault is, it could
also cause "VARIABLE IS NOT DEFINED!". The behavior
can also change if ansible-playbook is py2 or py3.

Root cause is errors from binascii.unhexlify() not
being handled consistently.

Fix is to add a AnsibleVaultFormatError exception and
raise it on any unhexlify() errors and to handle it
properly elsewhere.

Add a _unhexlify() that try/excepts around a binascii.unhexlify()
and raises an AnsibleVaultFormatError on invalid vault data.
This is so the same exception type is always raised for this
case. Previous it was different between py2 and py3.

binascii.unhexlify() raises a binascii.Error if the hexlified
blobs in a vault data blob are invalid.

On py2, binascii.Error is a subclass of Exception.
On py3, binascii.Error is a subclass of TypeError

When decrypting content of vault encrypted variables,
if a binascii.Error is raised it propagates up to
playbook.base.Base.post_validate(). post_validate()
handles exceptions for TypeErrors but not for
base Exception subclasses (like py2 binascii.Error).

* Add a display.warning on vault format errors
* Unit tests for _unhexlify, parse_vaulttext*
* Add intg test cases for invalid vault formats

Fixes #28038
7 years ago
Adrian Likins 297dfb1d50 Vault secrets script client inc new 'keyring' client (#27669)
This adds a new type of vault-password script  (a 'client') that takes advantage of and enhances the 
multiple vault password support.

If a vault password script basename ends with the name '-client', consider it a vault password script client. 

A vault password script 'client' just means that the script will take a '--vault-id' command line arg.

The previous vault password script (as invoked by --vault-password-file pointing to an executable) takes
no args and returns the password on stdout. But it doesnt know anything about --vault-id or multiple vault
passwords.

The new 'protocol' of the vault password script takes a cli arg ('--vault-id') so that it can lookup that specific
vault-id and return it's password.

Since existing vault password scripts don't know the new 'protocol', a way to distinguish password scripts
that do understand the protocol was needed.  The convention now is to consider password scripts that are
named like 'something-client.py' (and executable) to be vault password client scripts.

The new client scripts get invoked with the '--vault-id' they were requested for. An example:

     ansible-playbook --vault-id my_vault_id@contrib/vault/vault-keyring-client.py some_playbook.yml

That will cause the 'contrib/vault/vault-keyring-client.py' script to be invoked as:

     contrib/vault/vault-keyring-client.py --vault-id my_vault_id

The previous vault-keyring.py password script was extended to become vault-keyring-client.py. It uses
the python 'keyring' module to request secrets from various backends. The plain 'vault-keyring.py' script
would determine which key id and keyring name to use based on values that had to be set in ansible.cfg.
So it was also limited to one keyring name.

The new vault-keyring-client.py will request the secret for the vault id provided via the '--vault-id' option.
The script can be used without config and can be used for multiple keyring ids (and keyrings).

On success, a vault password client script will print the password to stdout and exit with a return code of 0.
If the 'client' script can't find a secret for the --vault-id, the script will exit with return code of 2 and print an error to stderr.
7 years ago
Adrian Likins 278ff19bea Handle vault decrypt --output=- (#31066)
In cli.CLI.unfrack_path callback, special case if the
value of '--output' is '-', and avoid expanding
it to a full path.

vault cli already has special cases for '-', so it
just needs to get the original value to work.

Fixes #30550
7 years ago
Adrian Likins a14d0f3586 Use vault_id when encrypted via vault-edit (#30772)
* Use vault_id when encrypted via vault-edit

On the encryption stage of
'ansible-vault edit --vault-id=someid@passfile somefile',
the vault id was not being passed to encrypt() so the files were
always saved with the default vault id in the 1.1 version format.

When trying to edit that file a second time, also with a --vault-id,
the file would be decrypted with the secret associated with the
provided vault-id, but since the encrypted file had no vault id
in the envelope there would be no match for 'default' secrets.
(Only the --vault-id was included in the potential matches, so
the vault id actually used to decrypt was not).

If that list was empty, there would be an IndexError when trying
to encrypted the changed file. This would result in the displayed
error:

ERROR! Unexpected Exception, this is probably a bug: list index out of range

Fix is two parts:

1) use the vault id when encrypting from edit

2) when matching the secret to use for encrypting after edit,
include the vault id that was used for decryption and not just
the vault id (or lack of vault id) from the envelope.

add unit tests for #30575 and intg tests for 'ansible-vault edit'

Fixes #30575
7 years ago
Adrian Likins 1f962bd937 Fix config value type for VAULT_IDENTITY_LIST (#28678)
Was using the 'value_type' key, but didn't get updated
to the new 'type' key in merge.

Fix playbooks cli so it uses VAULT_IDENTITY_LIST as well.
7 years ago
Adrian Likins 5739bb075f Vault secrets default vault ids list (#28190)
* Add config option for a default list of vault-ids

This is the vault-id equilivent of ANSIBLE_DEFAULT_PASSWORD_FILE
except ANSIBLE_DEFAULT_VAULT_IDENTITY_LIST is a list.
7 years ago
Adrian Likins e287af1ac8 Vault secrets empty password (#28186)
* Better handling of empty/invalid passwords

empty password files are global error and cause an
exit. A warning is also emitted with more detail.

ie, if any of the password/secret sources provide
a bogus password (ie, empty) or fail (exception,
 ctrl-d, EOFError), we stop at the first error and exit. 

This makes behavior when entering empty password at
prompt match 2.3 (ie, an error)
7 years ago
Adrian Likins 75a8be9a5d Add back support for vault_password_file config var (#27597)
Got removed in arg parsing updates. Now added back in
setup_vault_secrets().

The default value for DEFAULT_VAULT_PASSWORD_FILE was also
set to '~' for some reason, change to to no default.

Add integration tests.
7 years ago
Adrian Likins 934b645191 Support multiple vault passwords (#22756)
Fixes #13243

** Add --vault-id to name/identify multiple vault passwords

Use --vault-id to indicate id and path/type

 --vault-id=prompt  # prompt for default vault id password
 --vault-id=myorg@prompt  # prompt for a vault_id named 'myorg'
 --vault-id=a_password_file  # load ./a_password_file for default id
 --vault-id=myorg@a_password_file # load file for 'myorg' vault id

vault_id's are created implicitly for existing --vault-password-file
and --ask-vault-pass options.

Vault ids are just for UX purposes and bookkeeping. Only the vault
payload and the password bytestring is needed to decrypt a
vault blob.

Replace passing password around everywhere with
a VaultSecrets object.

If we specify a vault_id, mention that in password prompts

Specifying multiple -vault-password-files will
now try each until one works

** Rev vault format in a backwards compatible way

The 1.2 vault format adds the vault_id to the header line
of the vault text. This is backwards compatible with older
versions of ansible. Old versions will just ignore it and
treat it as the default (and only) vault id.

Note: only 2.4+ supports multiple vault passwords, so while
earlier ansible versions can read the vault-1.2 format, it
does not make them magically support multiple vault passwords.

use 1.1 format for 'default' vault_id

Vaulted items that need to include a vault_id will be
written in 1.2 format.

If we set a new DEFAULT_VAULT_IDENTITY, then the default will
use version 1.2

vault will only use a vault_id if one is specified. So if none
is specified and C.DEFAULT_VAULT_IDENTITY is 'default'
we use the old format.

** Changes/refactors needed to implement multiple vault passwords

raise exceptions on decrypt fail, check vault id early

split out parsing the vault plaintext envelope (with the
sha/original plaintext) to _split_plaintext_envelope()

some cli fixups for specifying multiple paths in
the unfrack_paths optparse callback

fix py3 dict.keys() 'dict_keys object is not indexable' error

pluralize cli.options.vault_password_file -> vault_password_files
pluralize cli.options.new_vault_password_file -> new_vault_password_files
pluralize cli.options.vault_id -> cli.options.vault_ids

** Add a config option (vault_id_match) to force vault id matching.

With 'vault_id_match=True' and an ansible
vault that provides a vault_id, then decryption will require
that a matching vault_id is required. (via
--vault-id=my_vault_id@password_file, for ex).

In other words, if the config option is true, then only
the vault secrets with matching vault ids are candidates for
decrypting a vault. If option is false (the default), then
all of the provided vault secrets will be selected.

If a user doesn't want all vault secrets to be tried to
decrypt any vault content, they can enable this option.

Note: The vault id used for the match is not encrypted or
cryptographically signed. It is just a label/id/nickname used
for referencing a specific vault secret.
7 years ago
Toshio Kuratomi e238ae999b Cyptography pr 20566 rebase (#25560)
Make pyca/cryptography the preferred backend for cryptographic needs (mainly vault) falling back to pycrypto

pyca/cryptography is already implicitly a dependency in many cases
through paramiko (2.0+) as well as the new openssl_publickey module,
which requires pyOpenSSL 16.0+. Additionally, pyca/cryptography is
an optional dep for better performance with vault already.

This commit leverages cryptography's padding, constant time comparisons,
and CBC/CTR modes to reduce the amount of code ansible needs to
maintain.

* Handle wrong password given for VaultAES format

* Do not display deprecation warning for cryptography on python-2.6

* Namespace all of the pycrypto imports and always import them

  Makes unittests better and the code less likely to get stupid mistakes
  (like using HMAC from cryptogrpahy when the one from pycrypto is needed)

* Add back in atfork since we need pycrypto to reinitialize its RNG just in case we're being used with old paramiko

* contrib/inventory/gce: Remove spurious require on pycrypto

(cherry picked from commit 9e16b9db275263b3ea8d1b124966fdebfc9ab271)

* Add cryptography to ec2_win_password module requirements
  * Fix python3 bug which would pass text strings to a function which
    requires byte strings.

* Attempt to add pycrypto version to setup deps

* Change hacking README for dual pycrypto/cryptography

* update dependencies for various CI scripts

* additional CI dockerfile/script updates

* add paramiko to the windows and sanity requirement set

  This is needed because ansible lists it as a requirement. Previously
  the missing dep wasn't enforced, but cryptography imports pkg_resources
  so you can't ignore a requirement any more

* Add integration test cases for old vault and for wrong passwords

* helper script for manual testing of pycrypto/cryptography

* Skip the pycrypto tests so that users without it installed can still run the unittests

* Run unittests for vault with both cryptography and pycrypto backend
8 years ago
Matt Martz 004e99316c Allow template files to be vaulted (#22951)
* Allow template files to be vaulted

* Make sure to import exceptions we need

* get_real_file can't take bytes, since it looks specifically for string_types

* Now that we aren't using open() we don't need b_source

* Expand playbooks_vault docs to include modules that support vaulted src files

* Add vaulted template test
8 years ago
Adrian Likins ae3d7fb29e Fix vault reading from stdin (avoid realpath() on non-links) (#23583)
* Fix vault reading from stdin (avoid realpath() on non-links)

os.path.realpath() is used to find the target of file paths that
are symlinks so vault operations happen directly on the target.

However, in addition to resolving symlinks, realpath() also returns
a full path. when reading from stdin, vault cli uses '-' as a special
file path so VaultEditor() will replace with stdin.
realpath() was expanding '-' with the CWD to something like
'/home/user/playbooks/-' causing errors like:

        ERROR! [Errno 2] No such file or directory: u'/home/user/ansible/-'

Fix is to specialcase '-' to not use realpath()

Fixes #23567

* to_text decrypt output when writing to stdout
8 years ago
Adrian Likins a2907c4d5c Add smoke/intg tests for vault cli (#21679)
Add smoke/intg tests for vault cli
8 years ago
Brian Coca a2c38c47aa added docs for vault and made trigger shorter: !vault (#20985)
* added docs for vault and made trigger shorter: !vault

* added single var valuting

* Update playbooks_vault.rst

Edit pass for spelling and grammar. Ship it!

* Update playbooks_vault.rst

Typo fixes.
8 years ago
Matt Clay 6bbd92e422 Initial ansible-test implementation. (#18556) 8 years ago
Adrian Likins 25aa757e80 add a intg test for vault encrypted inventory (#18550) 8 years ago
Adrien Vergé 0e834fc9e4 Fix cosmetic problems in YAML source
This change corrects problems reported by the `yamllint` linter.

Since key duplication problems were removed in 4d48711, this commit
mainly fixes trailing spaces and extra empty lines at beginning/end of
files.
8 years ago
Matt Clay 80a5c70ad7 Split integration tests out from Makefile. (#17976) 8 years ago