diff --git a/CHANGELOG.md b/CHANGELOG.md index 4d190293651..05bfff03965 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,10 +8,18 @@ Ansible Changes By Release * Added fact namespacing, from now on facts will be available under 'ansible_facts' namespace (i.e. `ansible_facts.ansible_os_distribution`), they will still also be added into the main namespace directly but now also having a configuration toggle to disable this. Eventually this will be on by default. This is done to avoid collisions and possible security issues as facts come from the remote targets and they might be compromised. * new 'order' play level keyword that allows the user to change the order in which Ansible processes hosts when dispatching tasks. * Users can now set group merge priority for groups of the same depth (parent child relationship), using the new `ansible_group_priority` variable, when values are the same or don't exist it will fallback to the previous 'sorting by name'. -* Support for Python-2.4 and Python-2.5 on the managed system's side was - dropped. If you need to manage a system that ships with Python-2.4 or - Python-2.5 you'll need to install Python-2.6 or better there or run - Ansible-2.3 until you can upgrade the system. +* Support for Python-2.4 and Python-2.5 on the managed system's side was dropped. If you need to manage a system that ships with Python-2.4 or Python-2.5, you'll need to install Python-2.6 or better on the managed system or run Ansible-2.3 until you can upgrade the system. +* Inventory has been revamped: + - Inventory classes have been split to allow for better management and deduplication + - Logic that each inventory source duplicated is now common and pushed up to reconciliation + - VariableManager has been updated for better interaction with inventory + - Updated CLI with helper method to initialize base objects for plays + - Inventory plugins are a new type of plugin (can generate and/or update inventory) + - Old inventory formats are still supported via plugins + - The vars_plugins have been eliminated in favor of inventory_plugins #TODO: repurpose/implement inv plugin that loads vars ones? + - Loading group_vars/host_vars is now a plugin and can be overridden (for inventory) + - It is now possible to specify mulitple inventory sources in the command line (-i /etc/hosts1 -i /opt/hosts2) + - Inventory plugins can use the cache plugin (i.e. virtualbox) and is affected by `meta: refresh_inventory` ### Deprecations * The behaviour when specifying --tags (or --skip-tags) multiple times on the command line diff --git a/Makefile b/Makefile index 4f52dc8af14..7c34cb9a6e8 100644 --- a/Makefile +++ b/Makefile @@ -121,6 +121,9 @@ tests: tests-py3: $(ANSIBLE_TEST) units -v --python $(PYTHON3_VERSION) $(TEST_FLAGS) +tests-nonet: + $(ANSIBLE_TEST) units -v --python $(PYTHON_VERSION) $(TEST_FLAGS) --exclude test/units/modules/network/ + integration: $(ANSIBLE_TEST) integration -v --docker $(IMAGE) $(TARGET) $(TEST_FLAGS) @@ -179,6 +182,7 @@ clean: @echo "Cleaning up docsite" $(MAKE) -C docs/docsite clean $(MAKE) -C docs/api clean + find test/ -type f -name '*.retry' -delete python: $(PYTHON) setup.py build diff --git a/bin/ansible b/bin/ansible index 24550b92e92..7c1a3ca4f92 100755 --- a/bin/ansible +++ b/bin/ansible @@ -37,9 +37,6 @@ import shutil import sys import traceback -# for debug -from multiprocessing import Lock - import ansible.constants as C from ansible.errors import AnsibleError, AnsibleOptionsError, AnsibleParserError from ansible.utils.display import Display diff --git a/docs/docsite/rst/intro_inventory.rst b/docs/docsite/rst/intro_inventory.rst index 2334c041073..10e42cd5989 100644 --- a/docs/docsite/rst/intro_inventory.rst +++ b/docs/docsite/rst/intro_inventory.rst @@ -5,22 +5,22 @@ Inventory .. contents:: Topics -Ansible works against multiple systems in your infrastructure at the -same time. It does this by selecting portions of systems listed in -Ansible's inventory file, which defaults to being saved in -the location ``/etc/ansible/hosts``. You can specify a different inventory file using the -``-i `` option on the command line. - -Not only is this inventory configurable, but you can also use -multiple inventory files at the same time (explained below) and also +Ansible works against multiple systems in your infrastructure at the same time. +It does this by selecting portions of systems listed in Ansible's inventory, +which defaults to being saved in the location ``/etc/ansible/hosts``. +You can specify a different inventory file using the ``-i `` option on the command line. + +Not only is this inventory configurable, but you can also use multiple inventory files at the same time and pull inventory from dynamic or cloud sources, as described in :doc:`intro_dynamic_inventory`. +Introduced in version 2.4, Ansible has inventory plugins to make this flexible and customizable. .. _inventoryformat: Hosts and Groups ++++++++++++++++ -The format for ``/etc/ansible/hosts`` is an INI-like format and looks like this: +The inventory file can be in one of many formats, depending on the inventory plugins you have. +For this example, the format for ``/etc/ansible/hosts`` is an INI-like (one of Ansible's defaults) and looks like this:: .. code-block:: ini @@ -118,6 +118,8 @@ Variables can also be applied to an entire group at once:: ntp_server=ntp.atlanta.example.com proxy=proxy.atlanta.example.com +Be aware that this is only a convenient way to apply variables to multiple hosts at once; even though you can target hosts by group, variables are always flattened to the host level before a play is executed. + .. _subgroups: Groups of Groups, and Group Variables @@ -149,8 +151,11 @@ It is also possible to make groups of groups using the ``:children`` suffix. Jus southwest northwest -If you need to store lists or hash data, or prefer to keep host and group specific variables -separate from the inventory file, see the next section. +If you need to store lists or hash data, or prefer to keep host and group specific variables separate from the inventory file, see the next section. +Child groups have a couple of properties to note: + + - First, any host that is member of a child group is automatically a member of the parent group. + - Second, a child group's variables will have higher precedence (override) a parent group's variables. .. _default_groups: @@ -228,7 +233,7 @@ ansible_connection .. include:: ../rst_common/ansible_ssh_changes_note.rst -SSH connection: +General for all connections: ansible_host The name of the host to connect to, if different from the alias you wish to give to it. @@ -236,6 +241,10 @@ ansible_port The ssh port number, if not 22 ansible_user The default ssh user name to use. + + +Specific to the SSH connection: + ansible_ssh_pass The ssh password to use (never store this variable in plain text; always use a vault. See :ref:`best_practices_for_variables_and_vaults`) ansible_ssh_private_key_file @@ -252,10 +261,7 @@ ansible_ssh_extra_args This setting is always appended to the default :command:`ssh` command line. ansible_ssh_pipelining Determines whether or not to use SSH pipelining. This can override the ``pipelining`` setting in :file:`ansible.cfg`. - -.. versionadded:: 2.2 - -ansible_ssh_executable +ansible_ssh_executable (added in version 2.2) This setting overrides the default behavior to use the system :command:`ssh`. This can override the ``ssh_executable`` setting in :file:`ansible.cfg`. @@ -295,7 +301,7 @@ ansible_shell_executable to use :command:`/bin/sh` (i.e. :command:`/bin/sh` is not installed on the target machine or cannot be run from sudo.). -Examples from a host file:: +Examples from an Ansible-INI host file:: some_host ansible_port=2222 ansible_user=manager aws_host ansible_ssh_private_key_file=/home/example/.ssh/aws.pem @@ -360,3 +366,4 @@ Here is an example of how to instantly deploy to created containers:: Questions? Help? Ideas? Stop by the list on Google Groups `irc.freenode.net `_ #ansible IRC chat channel + diff --git a/docs/docsite/rst/playbooks_variables.rst b/docs/docsite/rst/playbooks_variables.rst index 68282f6d0c5..39496530710 100644 --- a/docs/docsite/rst/playbooks_variables.rst +++ b/docs/docsite/rst/playbooks_variables.rst @@ -835,12 +835,12 @@ In 1.x, the precedence is as follows (with the last listed variables winning pri In 2.x, we have made the order of precedence more specific (with the last listed variables winning prioritization): * role defaults [1]_ - * inventory INI or script group vars [2]_ + * inventory file or script group vars [2]_ * inventory group_vars/all * playbook group_vars/all * inventory group_vars/* * playbook group_vars/* - * inventory INI or script host vars [2]_ + * inventory file or script host vars [2]_ * inventory host_vars/* * playbook host_vars/* * host facts diff --git a/examples/ansible.cfg b/examples/ansible.cfg index 77ba5d20d41..069055539d4 100644 --- a/examples/ansible.cfg +++ b/examples/ansible.cfg @@ -60,11 +60,21 @@ # uncomment this to disable SSH key host checking #host_key_checking = False -# change the default callback +# change the default callback, you can only have one 'stdout' type enabled at a time. #stdout_callback = skippy -# enable additional callbacks + + +## Ansible ships with some plugins that require whitelisting, +## this is done to avoid running all of a type by default. +## These setting lists those that you want enabled for your system. +## Custom plugins should not need this unless plugin author specifies it. + +# enable callback plugins, they can output to stdout but cannot be 'stdout' type. #callback_whitelist = timer, mail +# enable inventory plugins, default: 'host_list', 'script', 'ini', 'yaml' +#inventory_enabled = host_list, aws, openstack, docker + # Determine whether includes in tasks and handlers are "static" by # default. As of 2.0, includes are dynamic by default. Setting these # values to True will make includes behave more like they did in the diff --git a/lib/ansible/cli/__init__.py b/lib/ansible/cli/__init__.py index ce51245b3b3..01b89674372 100644 --- a/lib/ansible/cli/__init__.py +++ b/lib/ansible/cli/__init__.py @@ -35,9 +35,13 @@ import ansible from ansible.release import __version__ from ansible import constants as C from ansible.errors import AnsibleError, AnsibleOptionsError -from ansible.module_utils.six import with_metaclass +from ansible.inventory.manager import InventoryManager +from ansible.module_utils.six import with_metaclass, string_types from ansible.module_utils._text import to_bytes, to_text +from ansible.parsing.dataloader import DataLoader from ansible.utils.path import unfrackpath +from ansible.utils.vars import load_extra_vars, load_options_vars +from ansible.vars.manager import VariableManager try: from __main__ import display @@ -49,8 +53,6 @@ except ImportError: class SortedOptParser(optparse.OptionParser): '''Optparser which sorts the options by opt before outputting --help''' - #FIXME: epilog parsing: OptionParser.format_epilog = lambda self, formatter: self.epilog - def format_help(self, formatter=None, epilog=None): self.option_list.sort(key=operator.methodcaller('get_opt_string')) return optparse.OptionParser.format_help(self, formatter=None) @@ -294,9 +296,8 @@ class CLI(with_metaclass(ABCMeta, object)): help="verbose mode (-vvv for more, -vvvv to enable connection debugging)") if inventory_opts: - parser.add_option('-i', '--inventory-file', dest='inventory', - help="specify inventory host path (default=%s) or comma separated host list." % C.DEFAULT_HOST_LIST, - default=C.DEFAULT_HOST_LIST, action="callback", callback=CLI.expand_tilde, type=str) + parser.add_option('-i', '--inventory', '--inventory-file', dest='inventory', action="append", + help="specify inventory host path (default=[%s]) or comma separated host list. --inventory-file is deprecated" % C.DEFAULT_HOST_LIST) parser.add_option('--list-hosts', dest='listhosts', action='store_true', help='outputs a list of matching hosts; does not execute anything else') parser.add_option('-l', '--limit', default=C.DEFAULT_SUBSET, dest='subset', @@ -318,12 +319,12 @@ class CLI(with_metaclass(ABCMeta, object)): parser.add_option('--ask-vault-pass', default=C.DEFAULT_ASK_VAULT_PASS, dest='ask_vault_pass', action='store_true', help='ask for vault password') parser.add_option('--vault-password-file', default=C.DEFAULT_VAULT_PASSWORD_FILE, dest='vault_password_file', - help="vault password file", action="callback", callback=CLI.expand_tilde, type=str) + help="vault password file", action="callback", callback=CLI.expand_tilde, type='string') parser.add_option('--new-vault-password-file', dest='new_vault_password_file', - help="new vault password file for rekey", action="callback", callback=CLI.expand_tilde, type=str) + help="new vault password file for rekey", action="callback", callback=CLI.expand_tilde, type='string') parser.add_option('--output', default=None, dest='output_file', help='output file name for encrypt or decrypt; use - for stdout', - action="callback", callback=CLI.expand_tilde, type=str) + action="callback", callback=CLI.expand_tilde, type='string') if subset_opts: parser.add_option('-t', '--tags', dest='tags', default=[], action='append', @@ -342,8 +343,7 @@ class CLI(with_metaclass(ABCMeta, object)): connect_group.add_option('-k', '--ask-pass', default=C.DEFAULT_ASK_PASS, dest='ask_pass', action='store_true', help='ask for connection password') connect_group.add_option('--private-key','--key-file', default=C.DEFAULT_PRIVATE_KEY_FILE, dest='private_key_file', - help='use this file to authenticate the connection', - action="callback", callback=CLI.unfrack_path, type=str) + help='use this file to authenticate the connection', action="callback", callback=CLI.unfrack_path, type='string') connect_group.add_option('-u', '--user', default=C.DEFAULT_REMOTE_USER, dest='remote_user', help='connect as this user (default=%s)' % C.DEFAULT_REMOTE_USER) connect_group.add_option('-c', '--connection', dest='connection', default=C.DEFAULT_TRANSPORT, @@ -439,7 +439,10 @@ class CLI(with_metaclass(ABCMeta, object)): # If some additional transformations are needed for the # arguments and options, do it here. """ + self.options, self.args = self.parser.parse_args(self.args[1:]) + + # process tags if hasattr(self.options, 'tags') and not self.options.tags: # optparse defaults does not do what's expected self.options.tags = ['all'] @@ -457,6 +460,7 @@ class CLI(with_metaclass(ABCMeta, object)): tags.add(tag.strip()) self.options.tags = list(tags) + # process skip_tags if hasattr(self.options, 'skip_tags') and self.options.skip_tags: if not C.MERGE_MULTIPLE_CLI_TAGS: if len(self.options.skip_tags) > 1: @@ -471,6 +475,23 @@ class CLI(with_metaclass(ABCMeta, object)): skip_tags.add(tag.strip()) self.options.skip_tags = list(skip_tags) + # process inventory options + if hasattr(self.options, 'inventory'): + + if self.options.inventory: + + # should always be list + if isinstance(self.options.inventory, string_types): + self.options.inventory = [self.options.inventory] + + # Ensure full paths when needed + self.options.inventory = [unfrackpath(opt) if ',' not in opt else opt for opt in self.options.inventory] + + else: + # set default if it exists + if os.path.exists(C.DEFAULT_HOST_LIST): + self.options.inventory = [ C.DEFAULT_HOST_LIST ] + @staticmethod def version(prog): ''' return ansible version ''' @@ -654,18 +675,33 @@ class CLI(with_metaclass(ABCMeta, object)): return vault_pass - def get_opt(self, k, defval=""): - """ - Returns an option from an Optparse values instance. - """ - try: - data = getattr(self.options, k) - except: - return defval - # FIXME: Can this be removed if cli and/or constants ensures it's a - # list? - if k == "roles_path": - if os.pathsep in data: - data = data.split(os.pathsep)[0] - return data + @staticmethod + def _play_prereqs(options): + + # all needs loader + loader = DataLoader() + + # vault + b_vault_pass = None + if options.vault_password_file: + # read vault_pass from a file + b_vault_pass = CLI.read_vault_password_file(options.vault_password_file, loader=loader) + elif options.ask_vault_pass: + b_vault_pass = CLI.ask_vault_passwords() + + if b_vault_pass is not None: + loader.set_vault_password(b_vault_pass) + + # create the inventory, and filter it based on the subset specified (if any) + inventory = InventoryManager(loader=loader, sources=options.inventory) + + # create the variable manager, which will be shared throughout + # the code, ensuring a consistent view of global variables + variable_manager = VariableManager(loader=loader, inventory=inventory) + + # load vars from cli options + variable_manager.extra_vars = load_extra_vars(loader=loader, options=options) + variable_manager.options_vars = load_options_vars(options, CLI.version_info(gitinfo=False)) + + return loader, inventory, variable_manager diff --git a/lib/ansible/cli/adhoc.py b/lib/ansible/cli/adhoc.py index a7e1633b7b4..f1e94b6c3e4 100644 --- a/lib/ansible/cli/adhoc.py +++ b/lib/ansible/cli/adhoc.py @@ -26,15 +26,10 @@ from ansible import constants as C from ansible.cli import CLI from ansible.errors import AnsibleError, AnsibleOptionsError from ansible.executor.task_queue_manager import TaskQueueManager -from ansible.inventory import Inventory from ansible.module_utils._text import to_text -from ansible.parsing.dataloader import DataLoader from ansible.parsing.splitter import parse_kv from ansible.playbook.play import Play from ansible.plugins import get_all_plugin_loaders -from ansible.utils.vars import load_extra_vars -from ansible.utils.vars import load_options_vars -from ansible.vars import VariableManager try: from __main__ import display @@ -105,29 +100,12 @@ class AdHocCLI(CLI): sshpass = None becomepass = None - b_vault_pass = None self.normalize_become_options() (sshpass, becomepass) = self.ask_passwords() passwords = { 'conn_pass': sshpass, 'become_pass': becomepass } - loader = DataLoader() - - if self.options.vault_password_file: - # read vault_pass from a file - b_vault_pass = CLI.read_vault_password_file(self.options.vault_password_file, loader=loader) - loader.set_vault_password(b_vault_pass) - elif self.options.ask_vault_pass: - b_vault_pass = self.ask_vault_passwords() - loader.set_vault_password(b_vault_pass) - - variable_manager = VariableManager() - variable_manager.extra_vars = load_extra_vars(loader=loader, options=self.options) - - variable_manager.options_vars = load_options_vars(self.options) - - inventory = Inventory(loader=loader, variable_manager=variable_manager, host_list=self.options.inventory) - variable_manager.set_inventory(inventory) + loader, inventory, variable_manager = self._play_prereqs(self.options) no_hosts = False if len(inventory.list_hosts()) == 0: diff --git a/lib/ansible/cli/console.py b/lib/ansible/cli/console.py index 69c11014c32..2dbcb4be0ca 100644 --- a/lib/ansible/cli/console.py +++ b/lib/ansible/cli/console.py @@ -40,15 +40,12 @@ from ansible import constants as C from ansible.cli import CLI from ansible.errors import AnsibleError from ansible.executor.task_queue_manager import TaskQueueManager -from ansible.inventory import Inventory from ansible.module_utils._text import to_native, to_text -from ansible.parsing.dataloader import DataLoader from ansible.parsing.splitter import parse_kv from ansible.playbook.play import Play from ansible.plugins import module_loader from ansible.utils import plugin_docs from ansible.utils.color import stringc -from ansible.vars import VariableManager try: from __main__ import display @@ -277,11 +274,6 @@ class ConsoleCLI(CLI, cmd.Cmd): """ if not arg: self.options.cwd = '*' - elif arg == '..': - try: - self.options.cwd = self.inventory.groups_for_host(self.options.cwd)[1].name - except Exception: - self.options.cwd = '' elif arg in '/*': self.options.cwd = 'all' elif self.inventory.get_hosts(arg): @@ -402,7 +394,6 @@ class ConsoleCLI(CLI, cmd.Cmd): sshpass = None becomepass = None - vault_pass = None # hosts if len(self.args) != 1: @@ -421,19 +412,7 @@ class ConsoleCLI(CLI, cmd.Cmd): (sshpass, becomepass) = self.ask_passwords() self.passwords = { 'conn_pass': sshpass, 'become_pass': becomepass } - self.loader = DataLoader() - - if self.options.vault_password_file: - # read vault_pass from a file - vault_pass = CLI.read_vault_password_file(self.options.vault_password_file, loader=self.loader) - self.loader.set_vault_password(vault_pass) - elif self.options.ask_vault_pass: - vault_pass = self.ask_vault_passwords() - self.loader.set_vault_password(vault_pass) - - self.variable_manager = VariableManager() - self.inventory = Inventory(loader=self.loader, variable_manager=self.variable_manager, host_list=self.options.inventory) - self.variable_manager.set_inventory(self.inventory) + self.loader, self.inventory, self.variable_manager = self._play_prereqs(self.options) no_hosts = False if len(self.inventory.list_hosts()) == 0: diff --git a/lib/ansible/cli/doc.py b/lib/ansible/cli/doc.py index ceba403032d..52afd548d66 100644 --- a/lib/ansible/cli/doc.py +++ b/lib/ansible/cli/doc.py @@ -28,8 +28,8 @@ import yaml from ansible import constants as C from ansible.cli import CLI from ansible.errors import AnsibleError, AnsibleOptionsError -from ansible.module_utils.six import iteritems, string_types -from ansible.plugins import module_loader, action_loader, lookup_loader, callback_loader, cache_loader, connection_loader, strategy_loader +from ansible.module_utils.six import string_types +from ansible.plugins import module_loader, action_loader, lookup_loader, callback_loader, cache_loader, connection_loader, strategy_loader, PluginLoader from ansible.utils import plugin_docs try: @@ -66,7 +66,7 @@ class DocCLI(CLI): self.parser.add_option("-a", "--all", action="store_true", default=False, dest='all_plugins', help='Show documentation for all plugins') self.parser.add_option("-t", "--type", action="store", default='module', dest='type', type='choice', - help='Choose which plugin type', choices=['module','cache', 'connection', 'callback', 'lookup', 'strategy']) + help='Choose which plugin type', choices=['module','cache', 'connection', 'callback', 'lookup', 'strategy', 'inventory']) super(DocCLI, self).parse() @@ -89,6 +89,8 @@ class DocCLI(CLI): loader = lookup_loader elif plugin_type == 'strategy': loader = strategy_loader + elif plugin_type == 'inventory': + loader = PluginLoader( 'InventoryModule', 'ansible.plugins.inventory', 'inventory_plugins', 'inventory_plugins') else: loader = module_loader diff --git a/lib/ansible/cli/galaxy.py b/lib/ansible/cli/galaxy.py index 3b0192d2a4d..a89435a0110 100644 --- a/lib/ansible/cli/galaxy.py +++ b/lib/ansible/cli/galaxy.py @@ -677,3 +677,19 @@ class GalaxyCLI(CLI): display.display(resp['status']) return True + + def get_opt(self, k, defval=""): + """ + Returns an option from an Optparse values instance. + """ + try: + data = getattr(self.options, k) + except: + return defval + # FIXME: Can this be removed if cli and/or constants ensures it's a + # list? + if k == "roles_path": + if os.pathsep in data: + data = data.split(os.pathsep)[0] + return data + diff --git a/lib/ansible/cli/playbook.py b/lib/ansible/cli/playbook.py index 679396c85c1..d7ea0295866 100644 --- a/lib/ansible/cli/playbook.py +++ b/lib/ansible/cli/playbook.py @@ -26,13 +26,8 @@ import stat from ansible.cli import CLI from ansible.errors import AnsibleError, AnsibleOptionsError from ansible.executor.playbook_executor import PlaybookExecutor -from ansible.inventory import Inventory -from ansible.parsing.dataloader import DataLoader from ansible.playbook.block import Block from ansible.playbook.play_context import PlayContext -from ansible.utils.vars import load_extra_vars -from ansible.utils.vars import load_options_vars -from ansible.vars import VariableManager try: from __main__ import display @@ -93,7 +88,6 @@ class PlaybookCLI(CLI): # Manage passwords sshpass = None becomepass = None - b_vault_pass = None passwords = {} # initial error check, to make sure all specified playbooks are accessible @@ -110,26 +104,7 @@ class PlaybookCLI(CLI): (sshpass, becomepass) = self.ask_passwords() passwords = { 'conn_pass': sshpass, 'become_pass': becomepass } - loader = DataLoader() - - if self.options.vault_password_file: - # read vault_pass from a file - b_vault_pass = CLI.read_vault_password_file(self.options.vault_password_file, loader=loader) - loader.set_vault_password(b_vault_pass) - elif self.options.ask_vault_pass: - b_vault_pass = self.ask_vault_passwords() - loader.set_vault_password(b_vault_pass) - - # create the variable manager, which will be shared throughout - # the code, ensuring a consistent view of global variables - variable_manager = VariableManager() - variable_manager.extra_vars = load_extra_vars(loader=loader, options=self.options) - - variable_manager.options_vars = load_options_vars(self.options) - - # create the inventory, and filter it based on the subset specified (if any) - inventory = Inventory(loader=loader, variable_manager=variable_manager, host_list=self.options.inventory) - variable_manager.set_inventory(inventory) + loader, inventory, variable_manager = self._play_prereqs(self.options) # (which is not returned in list_hosts()) is taken into account for # warning if inventory is empty. But it can't be taken into account for @@ -147,6 +122,7 @@ class PlaybookCLI(CLI): # Invalid limit raise AnsibleError("Specified --limit does not match any hosts") + # flush fact cache if requested if self.options.flush_cache: self._flush_cache(inventory, variable_manager) @@ -207,7 +183,7 @@ class PlaybookCLI(CLI): return taskmsg - all_vars = variable_manager.get_vars(loader=loader, play=play) + all_vars = variable_manager.get_vars(play=play) play_context = PlayContext(play=play, options=self.options) for block in play.compile(): block = block.filter_tagged_tasks(play_context, all_vars) diff --git a/lib/ansible/cli/pull.py b/lib/ansible/cli/pull.py index d7a81655b7d..492c619c911 100644 --- a/lib/ansible/cli/pull.py +++ b/lib/ansible/cli/pull.py @@ -155,10 +155,15 @@ class PullCLI(CLI): # Attempt to use the inventory passed in as an argument # It might not yet have been downloaded so use localhost as default - if not self.options.inventory or ( ',' not in self.options.inventory and not os.path.exists(self.options.inventory)): - inv_opts = 'localhost,' + inv_opts = '' + if getattr(self.options, 'inventory'): + for inv in self.options.inventory: + if isinstance(inv, list): + inv_opts += " -i '%s' " % ','.join(inv) + elif ',' in inv or os.path.exists(inv): + inv_opts += ' -i %s ' % inv else: - inv_opts = self.options.inventory + inv_opts = "-i 'localhost,'" #FIXME: enable more repo modules hg/svn? if self.options.module_name == 'git': @@ -190,7 +195,7 @@ class PullCLI(CLI): bin_path = os.path.dirname(os.path.abspath(sys.argv[0])) # hardcode local and inventory/host as this is just meant to fetch the repo - cmd = '%s/ansible -i "%s" %s -m %s -a "%s" all -l "%s"' % (bin_path, inv_opts, base_opts, self.options.module_name, repo_opts, limit_opts) + cmd = '%s/ansible %s %s -m %s -a "%s" all -l "%s"' % (bin_path, inv_opts, base_opts, self.options.module_name, repo_opts, limit_opts) for ev in self.options.extra_vars: cmd += ' -e "%s"' % ev @@ -222,8 +227,8 @@ class PullCLI(CLI): cmd = '%s/ansible-playbook %s %s' % (bin_path, base_opts, playbook) if self.options.vault_password_file: cmd += " --vault-password-file=%s" % self.options.vault_password_file - if self.options.inventory: - cmd += ' -i "%s"' % self.options.inventory + if inv_opts: + cmd += ' %s' % inv_opts for ev in self.options.extra_vars: cmd += ' -e "%s"' % ev if self.options.ask_sudo_pass or self.options.ask_su_pass or self.options.become_ask_pass: diff --git a/lib/ansible/constants.py b/lib/ansible/constants.py index ce924e5120f..eb7f12284b1 100644 --- a/lib/ansible/constants.py +++ b/lib/ansible/constants.py @@ -30,7 +30,7 @@ from ansible.module_utils._text import to_text from ansible.parsing.quoting import unquote from ansible.utils.path import makedirs_safe -BOOL_TRUE = frozenset(["true", "t", "y", "1", "yes", "on"]) +BOOL_TRUE = frozenset( [ "true", "t", "y", "1", "yes", "on" ] ) def mk_boolean(value): @@ -170,52 +170,51 @@ def load_config_file(): p, CONFIG_FILE = load_config_file() -# check all of these extensions when looking for yaml files for things like -# group variables -- really anything we can load -YAML_FILENAME_EXTENSIONS = ["", ".yml", ".yaml", ".json"] - +# non configurable but used as defaults +BLACKLIST_EXTS = ('.pyc', '.pyo', '.swp', '.bak', '~', '.rpm', '.md', '.txt') # the default whitelist for cow stencils -DEFAULT_COW_WHITELIST = ['bud-frogs', 'bunny', 'cheese', 'daemon', 'default', 'dragon', 'elephant-in-snake', 'elephant', - 'eyes', 'hellokitty', 'kitty', 'luke-koala', 'meow', 'milk', 'moofasa', 'moose', 'ren', 'sheep', - 'small', 'stegosaurus', 'stimpy', 'supermilker', 'three-eyes', 'turkey', 'turtle', 'tux', 'udder', - 'vader-koala', 'vader', 'www'] +DEFAULT_COW_WHITELIST = [ 'bud-frogs', 'bunny', 'cheese', 'daemon', 'default', 'dragon', 'elephant-in-snake', 'elephant', + 'eyes', 'hellokitty', 'kitty', 'luke-koala', 'meow', 'milk', 'moofasa', 'moose', 'ren', 'sheep', + 'small', 'stegosaurus', 'stimpy', 'supermilker', 'three-eyes', 'turkey', 'turtle', 'tux', 'udder', + 'vader-koala', 'vader', 'www', ] # sections in config file -DEFAULTS = 'defaults' - - -# FIXME: add deprecation warning when these get set -# DEPRECATED VARS - -# If --tags or --skip-tags is given multiple times on the CLI and this is -# True, merge the lists of tags together. If False, let the last argument -# overwrite any previous ones. Behaviour is overwrite through 2.2. 2.3 -# overwrites but prints deprecation. 2.4 the default is to merge. -MERGE_MULTIPLE_CLI_TAGS = get_config(p, DEFAULTS, 'merge_multiple_cli_tags', 'ANSIBLE_MERGE_MULTIPLE_CLI_TAGS', True, value_type='boolean') - -# GENERALLY CONFIGURABLE THINGS -DEFAULT_DEBUG = get_config(p, DEFAULTS, 'debug', 'ANSIBLE_DEBUG', False, value_type='boolean') -DEFAULT_VERBOSITY = get_config(p, DEFAULTS, 'verbosity', 'ANSIBLE_VERBOSITY', 0, value_type='integer') -DEFAULT_HOST_LIST = get_config(p, DEFAULTS, 'inventory', 'ANSIBLE_INVENTORY', '/etc/ansible/hosts', value_type='path') -DEFAULT_ROLES_PATH = get_config(p, DEFAULTS, 'roles_path', 'ANSIBLE_ROLES_PATH', - '~/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles', - value_type='pathlist', expand_relative_paths=True) -DEFAULT_REMOTE_TMP = get_config(p, DEFAULTS, 'remote_tmp', 'ANSIBLE_REMOTE_TEMP', '~/.ansible/tmp') -DEFAULT_LOCAL_TMP = get_config(p, DEFAULTS, 'local_tmp', 'ANSIBLE_LOCAL_TEMP', '~/.ansible/tmp', value_type='tmppath') -DEFAULT_MODULE_NAME = get_config(p, DEFAULTS, 'module_name', None, 'command') -DEFAULT_FACT_PATH = get_config(p, DEFAULTS, 'fact_path', 'ANSIBLE_FACT_PATH', None, value_type='path') -DEFAULT_FORKS = get_config(p, DEFAULTS, 'forks', 'ANSIBLE_FORKS', 5, value_type='integer') -DEFAULT_MODULE_ARGS = get_config(p, DEFAULTS, 'module_args', 'ANSIBLE_MODULE_ARGS', '') -DEFAULT_MODULE_LANG = get_config(p, DEFAULTS, 'module_lang', 'ANSIBLE_MODULE_LANG', os.getenv('LANG', 'en_US.UTF-8')) -DEFAULT_MODULE_SET_LOCALE = get_config(p, DEFAULTS, 'module_set_locale', 'ANSIBLE_MODULE_SET_LOCALE', False, value_type='boolean') -DEFAULT_MODULE_COMPRESSION = get_config(p, DEFAULTS, 'module_compression', None, 'ZIP_DEFLATED') -DEFAULT_TIMEOUT = get_config(p, DEFAULTS, 'timeout', 'ANSIBLE_TIMEOUT', 10, value_type='integer') -DEFAULT_POLL_INTERVAL = get_config(p, DEFAULTS, 'poll_interval', 'ANSIBLE_POLL_INTERVAL', 15, value_type='integer') -DEFAULT_REMOTE_USER = get_config(p, DEFAULTS, 'remote_user', 'ANSIBLE_REMOTE_USER', None) -DEFAULT_ASK_PASS = get_config(p, DEFAULTS, 'ask_pass', 'ANSIBLE_ASK_PASS', False, value_type='boolean') -DEFAULT_PRIVATE_KEY_FILE = get_config(p, DEFAULTS, 'private_key_file', 'ANSIBLE_PRIVATE_KEY_FILE', None, value_type='path') -DEFAULT_REMOTE_PORT = get_config(p, DEFAULTS, 'remote_port', 'ANSIBLE_REMOTE_PORT', None, value_type='integer') -DEFAULT_ASK_VAULT_PASS = get_config(p, DEFAULTS, 'ask_vault_pass', 'ANSIBLE_ASK_VAULT_PASS', False, value_type='boolean') +DEFAULTS='defaults' + +#### DEPRECATED VARS ### # FIXME: add deprecation warning when these get set +#none left now + +#### DEPRECATED FEATURE TOGGLES: these will eventually be removed as it becomes the standard #### + +# If --tags or --skip-tags is given multiple times on the CLI and this is True, merge the lists of tags together. +# If False, let the last argument overwrite any previous ones. +# Behaviour is overwrite through 2.2. 2.3 overwrites but prints deprecation. 2.4 the default is to merge. +MERGE_MULTIPLE_CLI_TAGS = get_config(p, DEFAULTS, 'merge_multiple_cli_tags', 'ANSIBLE_MERGE_MULTIPLE_CLI_TAGS', True, value_type='boolean') + +# Controls which 'precedence path' to take, remove when decide on which! +SOURCE_OVER_GROUPS = get_config(p, 'vars', 'source_over_groups', 'ANSIBLE_SOURCE_OVER_GROUPS', True, value_type='boolean') + +#### GENERALLY CONFIGURABLE THINGS #### +DEFAULT_DEBUG = get_config(p, DEFAULTS, 'debug', 'ANSIBLE_DEBUG', False, value_type='boolean') +DEFAULT_VERBOSITY = get_config(p, DEFAULTS, 'verbosity', 'ANSIBLE_VERBOSITY', 0, value_type='integer') +DEFAULT_ROLES_PATH = get_config(p, DEFAULTS, 'roles_path', 'ANSIBLE_ROLES_PATH', + '~/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles', value_type='pathlist', expand_relative_paths=True) +DEFAULT_REMOTE_TMP = get_config(p, DEFAULTS, 'remote_tmp', 'ANSIBLE_REMOTE_TEMP', '~/.ansible/tmp') +DEFAULT_LOCAL_TMP = get_config(p, DEFAULTS, 'local_tmp', 'ANSIBLE_LOCAL_TEMP', '~/.ansible/tmp', value_type='tmppath') +DEFAULT_MODULE_NAME = get_config(p, DEFAULTS, 'module_name', None, 'command') +DEFAULT_FACT_PATH = get_config(p, DEFAULTS, 'fact_path', 'ANSIBLE_FACT_PATH', None, value_type='path') +DEFAULT_FORKS = get_config(p, DEFAULTS, 'forks', 'ANSIBLE_FORKS', 5, value_type='integer') +DEFAULT_MODULE_ARGS = get_config(p, DEFAULTS, 'module_args', 'ANSIBLE_MODULE_ARGS', '') +DEFAULT_MODULE_LANG = get_config(p, DEFAULTS, 'module_lang', 'ANSIBLE_MODULE_LANG', os.getenv('LANG', 'en_US.UTF-8')) +DEFAULT_MODULE_SET_LOCALE = get_config(p, DEFAULTS, 'module_set_locale','ANSIBLE_MODULE_SET_LOCALE',False, value_type='boolean') +DEFAULT_MODULE_COMPRESSION= get_config(p, DEFAULTS, 'module_compression', None, 'ZIP_DEFLATED') +DEFAULT_TIMEOUT = get_config(p, DEFAULTS, 'timeout', 'ANSIBLE_TIMEOUT', 10, value_type='integer') +DEFAULT_POLL_INTERVAL = get_config(p, DEFAULTS, 'poll_interval', 'ANSIBLE_POLL_INTERVAL', 15, value_type='integer') +DEFAULT_REMOTE_USER = get_config(p, DEFAULTS, 'remote_user', 'ANSIBLE_REMOTE_USER', None) +DEFAULT_ASK_PASS = get_config(p, DEFAULTS, 'ask_pass', 'ANSIBLE_ASK_PASS', False, value_type='boolean') +DEFAULT_PRIVATE_KEY_FILE = get_config(p, DEFAULTS, 'private_key_file', 'ANSIBLE_PRIVATE_KEY_FILE', None, value_type='path') +DEFAULT_REMOTE_PORT = get_config(p, DEFAULTS, 'remote_port', 'ANSIBLE_REMOTE_PORT', None, value_type='integer') +DEFAULT_ASK_VAULT_PASS = get_config(p, DEFAULTS, 'ask_vault_pass', 'ANSIBLE_ASK_VAULT_PASS', False, value_type='boolean') DEFAULT_VAULT_PASSWORD_FILE = get_config(p, DEFAULTS, 'vault_password_file', 'ANSIBLE_VAULT_PASSWORD_FILE', None, value_type='path') DEFAULT_TRANSPORT = get_config(p, DEFAULTS, 'transport', 'ANSIBLE_TRANSPORT', 'smart') DEFAULT_SCP_IF_SSH = get_config(p, 'ssh_connection', 'scp_if_ssh', 'ANSIBLE_SCP_IF_SSH', 'smart') @@ -227,14 +226,12 @@ DEFAULT_KEEP_REMOTE_FILES = get_config(p, DEFAULTS, 'keep_remote_files', 'ANSIBL DEFAULT_HASH_BEHAVIOUR = get_config(p, DEFAULTS, 'hash_behaviour', 'ANSIBLE_HASH_BEHAVIOUR', 'replace') DEFAULT_PRIVATE_ROLE_VARS = get_config(p, DEFAULTS, 'private_role_vars', 'ANSIBLE_PRIVATE_ROLE_VARS', False, value_type='boolean') DEFAULT_JINJA2_EXTENSIONS = get_config(p, DEFAULTS, 'jinja2_extensions', 'ANSIBLE_JINJA2_EXTENSIONS', None) -DEFAULT_EXECUTABLE = get_config(p, DEFAULTS, 'executable', 'ANSIBLE_EXECUTABLE', '/bin/sh') -DEFAULT_GATHERING = get_config(p, DEFAULTS, 'gathering', 'ANSIBLE_GATHERING', 'implicit').lower() -DEFAULT_GATHER_SUBSET = get_config(p, DEFAULTS, 'gather_subset', 'ANSIBLE_GATHER_SUBSET', 'all').lower() -DEFAULT_GATHER_TIMEOUT = get_config(p, DEFAULTS, 'gather_timeout', 'ANSIBLE_GATHER_TIMEOUT', 10, value_type='integer') -DEFAULT_LOG_PATH = get_config(p, DEFAULTS, 'log_path', 'ANSIBLE_LOG_PATH', '', value_type='path') -DEFAULT_FORCE_HANDLERS = get_config(p, DEFAULTS, 'force_handlers', 'ANSIBLE_FORCE_HANDLERS', False, value_type='boolean') -DEFAULT_INVENTORY_IGNORE = get_config(p, DEFAULTS, 'inventory_ignore_extensions', 'ANSIBLE_INVENTORY_IGNORE', - ["~", ".orig", ".bak", ".ini", ".cfg", ".retry", ".pyc", ".pyo"], value_type='list') +DEFAULT_EXECUTABLE = get_config(p, DEFAULTS, 'executable', 'ANSIBLE_EXECUTABLE', '/bin/sh') +DEFAULT_GATHERING = get_config(p, DEFAULTS, 'gathering', 'ANSIBLE_GATHERING', 'implicit').lower() +DEFAULT_GATHER_SUBSET = get_config(p, DEFAULTS, 'gather_subset', 'ANSIBLE_GATHER_SUBSET', 'all').lower() +DEFAULT_GATHER_TIMEOUT = get_config(p, DEFAULTS, 'gather_timeout', 'ANSIBLE_GATHER_TIMEOUT', 10, value_type='integer') +DEFAULT_LOG_PATH = get_config(p, DEFAULTS, 'log_path', 'ANSIBLE_LOG_PATH', '', value_type='path') +DEFAULT_FORCE_HANDLERS = get_config(p, DEFAULTS, 'force_handlers', 'ANSIBLE_FORCE_HANDLERS', False, value_type='boolean') DEFAULT_VAR_COMPRESSION_LEVEL = get_config(p, DEFAULTS, 'var_compression_level', 'ANSIBLE_VAR_COMPRESSION_LEVEL', 0, value_type='integer') DEFAULT_INTERNAL_POLL_INTERVAL = get_config(p, DEFAULTS, 'internal_poll_interval', None, 0.001, value_type='float') DEFAULT_ALLOW_UNSAFE_LOOKUPS = get_config(p, DEFAULTS, 'allow_unsafe_lookups', None, False, value_type='boolean') @@ -242,16 +239,27 @@ ERROR_ON_MISSING_HANDLER = get_config(p, DEFAULTS, 'error_on_missing_handler', ' SHOW_CUSTOM_STATS = get_config(p, DEFAULTS, 'show_custom_stats', 'ANSIBLE_SHOW_CUSTOM_STATS', False, value_type='boolean') NAMESPACE_FACTS = get_config(p, DEFAULTS, 'restrict_facts_namespace', 'ANSIBLE_RESTRICT_FACTS', False, value_type='boolean') -# static includes +# Inventory +DEFAULT_HOST_LIST = get_config(p, DEFAULTS,'inventory', 'ANSIBLE_INVENTORY', '/etc/ansible/hosts', value_type='path', expand_relative_paths=True) +INVENTORY_ENABLED = get_config(p, DEFAULTS,'inventory_enabled', 'ANSIBLE_INVENTORY_ENABLED', + [ 'host_list', 'script', 'ini', 'yaml' ], value_type='list') +INVENTORY_IGNORE_EXTS = get_config(p, DEFAULTS, 'inventory_ignore_extensions', 'ANSIBLE_INVENTORY_IGNORE', + BLACKLIST_EXTS + (".orig", ".ini", ".cfg", ".retry"), value_type='list') +INVENTORY_IGNORE_PATTERNS = get_config(p, DEFAULTS, 'inventory_ignore_patterns', 'ANSIBLE_INVENTORY_IGNORE_REGEX', [], value_type='list') +VARIABLE_PRECEDENCE = get_config(p, DEFAULTS, 'precedence', 'ANSIBLE_PRECEDENCE', + ['all_inventory', 'groups_inventory', 'all_plugins_inventory', 'all_plugins_play', + 'groups_plugins_inventory', 'groups_plugins_play'], + value_type='list') +# Static includes DEFAULT_TASK_INCLUDES_STATIC = get_config(p, DEFAULTS, 'task_includes_static', 'ANSIBLE_TASK_INCLUDES_STATIC', False, value_type='boolean') DEFAULT_HANDLER_INCLUDES_STATIC = get_config(p, DEFAULTS, 'handler_includes_static', 'ANSIBLE_HANDLER_INCLUDES_STATIC', False, value_type='boolean') -# disclosure +# Disclosure DEFAULT_NO_LOG = get_config(p, DEFAULTS, 'no_log', 'ANSIBLE_NO_LOG', False, value_type='boolean') DEFAULT_NO_TARGET_SYSLOG = get_config(p, DEFAULTS, 'no_target_syslog', 'ANSIBLE_NO_TARGET_SYSLOG', False, value_type='boolean') ALLOW_WORLD_READABLE_TMPFILES = get_config(p, DEFAULTS, 'allow_world_readable_tmpfiles', None, False, value_type='boolean') -# selinux +# Selinux DEFAULT_SELINUX_SPECIAL_FS = get_config(p, 'selinux', 'special_context_filesystems', None, 'fuse, nfs, vboxsf, ramfs, 9p', value_type='list') DEFAULT_LIBVIRT_LXC_NOSECLABEL = get_config(p, 'selinux', 'libvirt_lxc_noseclabel', 'LIBVIRT_LXC_NOSECLABEL', False, value_type='boolean') @@ -443,7 +451,8 @@ VAULT_VERSION_MAX = 1.0 TREE_DIR = None LOCALHOST = frozenset(['127.0.0.1', 'localhost', '::1']) # module search -BLACKLIST_EXTS = ('.pyc', '.swp', '.bak', '~', '.rpm', '.md', '.txt') IGNORE_FILES = ["COPYING", "CONTRIBUTING", "LICENSE", "README", "VERSION", "GUIDELINES"] INTERNAL_RESULT_KEYS = ['add_host', 'add_group'] RESTRICTED_RESULT_KEYS = ['ansible_rsync_path', 'ansible_playbook_python'] +# check all of these extensions when looking for 'variable' files which should be YAML or JSON. +YAML_FILENAME_EXTENSIONS = [ ".yml", ".yaml", ".json" ] diff --git a/lib/ansible/executor/play_iterator.py b/lib/ansible/executor/play_iterator.py index b3808021f0e..4cce30e5009 100644 --- a/lib/ansible/executor/play_iterator.py +++ b/lib/ansible/executor/play_iterator.py @@ -22,11 +22,9 @@ __metaclass__ = type import fnmatch from ansible import constants as C -from ansible.errors import AnsibleError from ansible.module_utils.six import iteritems from ansible.playbook.block import Block from ansible.playbook.task import Task -from ansible.playbook.role_include import IncludeRole boolean = C.mk_boolean @@ -205,10 +203,6 @@ class PlayIterator: start_at_matched = False for host in inventory.get_hosts(self._play.hosts): self._host_states[host.name] = HostState(blocks=self._blocks) - # if the host's name is in the variable manager's fact cache, then set - # its _gathered_facts flag to true for smart gathering tests later - if host.name in variable_manager._fact_cache and variable_manager._fact_cache.get(host.name).get('module_setup', False): - host._gathered_facts = True # if we're looking to start at a specific task, iterate through # the tasks for this host until we find the specified task if play_context.start_at_task is not None and not start_at_done: @@ -265,7 +259,6 @@ class PlayIterator: display.debug("host %s is done iterating, returning" % host.name) return (s, None) - old_s = s (s, task) = self._get_next_task_from_state(s, host=host, peek=peek) if not peek: @@ -310,16 +303,12 @@ class PlayIterator: if (gathering == 'implicit' and implied) or \ (gathering == 'explicit' and boolean(self._play.gather_facts)) or \ - (gathering == 'smart' and implied and not host._gathered_facts): + (gathering == 'smart' and implied and not (variable_manager._fact_cache.get(host.name,{}).get('module_setup', False))): # The setup block is always self._blocks[0], as we inject it # during the play compilation in __init__ above. setup_block = self._blocks[0] if setup_block.has_tasks() and len(setup_block.block) > 0: task = setup_block.block[0] - if not peek: - # mark the host as having gathered facts, because we're - # returning the setup task to be executed - host.set_gathered_facts(True) else: # This is the second trip through ITERATING_SETUP, so we clear # the flag and move onto the next block in the list while setting diff --git a/lib/ansible/executor/playbook_executor.py b/lib/ansible/executor/playbook_executor.py index 5923deafc01..c97e27527ec 100644 --- a/lib/ansible/executor/playbook_executor.py +++ b/lib/ansible/executor/playbook_executor.py @@ -79,7 +79,7 @@ class PlaybookExecutor: try: for playbook_path in self._playbooks: pb = Playbook.load(playbook_path, variable_manager=self._variable_manager, loader=self._loader) - self._inventory.set_playbook_basedir(os.path.realpath(os.path.dirname(playbook_path))) + #FIXME: move out of inventory self._inventory.set_playbook_basedir(os.path.realpath(os.path.dirname(playbook_path))) if self._tqm is None: # we are doing a listing entry = {'playbook': playbook_path} @@ -122,7 +122,7 @@ class PlaybookExecutor: # Create a temporary copy of the play here, so we can run post_validate # on it without the templating changes affecting the original object. - all_vars = self._variable_manager.get_vars(loader=self._loader, play=play) + all_vars = self._variable_manager.get_vars(play=play) templar = Templar(loader=self._loader, variables=all_vars) new_play = play.copy() new_play.post_validate(templar) diff --git a/lib/ansible/executor/task_queue_manager.py b/lib/ansible/executor/task_queue_manager.py index 252b4946af0..b7ab020e4c0 100644 --- a/lib/ansible/executor/task_queue_manager.py +++ b/lib/ansible/executor/task_queue_manager.py @@ -210,7 +210,7 @@ class TaskQueueManager: if not self._callbacks_loaded: self.load_callbacks() - all_vars = self._variable_manager.get_vars(loader=self._loader, play=play) + all_vars = self._variable_manager.get_vars(play=play) warn_if_reserved(all_vars) templar = Templar(loader=self._loader, variables=all_vars) diff --git a/lib/ansible/galaxy/role.py b/lib/ansible/galaxy/role.py index 1a8b9ace54c..c7183498107 100644 --- a/lib/ansible/galaxy/role.py +++ b/lib/ansible/galaxy/role.py @@ -30,7 +30,6 @@ import yaml from distutils.version import LooseVersion from shutil import rmtree -import ansible.constants as C from ansible.errors import AnsibleError from ansible.module_utils.urls import open_url from ansible.playbook.role.requirement import RoleRequirement diff --git a/lib/ansible/inventory/__init__.py b/lib/ansible/inventory/__init__.py index de5fe4c72fc..e69de29bb2d 100644 --- a/lib/ansible/inventory/__init__.py +++ b/lib/ansible/inventory/__init__.py @@ -1,903 +0,0 @@ -# (c) 2012-2014, Michael DeHaan -# -# This file is part of Ansible -# -# Ansible is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# Ansible is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Ansible. If not, see . - -############################################# -from __future__ import (absolute_import, division, print_function) -__metaclass__ = type - -import fnmatch -import os -import subprocess -import sys -import re -import itertools - - -from ansible import constants as C -from ansible.errors import AnsibleError -from ansible.inventory.dir import InventoryDirectory, get_file_parser -from ansible.inventory.group import Group -from ansible.inventory.host import Host -from ansible.module_utils.six import string_types, iteritems -from ansible.module_utils._text import to_bytes, to_text -from ansible.parsing.utils.addresses import parse_address -from ansible.plugins import vars_loader -from ansible.utils.vars import combine_vars -from ansible.utils.path import unfrackpath - -try: - from __main__ import display -except ImportError: - from ansible.utils.display import Display - display = Display() - - -HOSTS_PATTERNS_CACHE = {} - - -class Inventory(object): - """ - Host inventory for ansible. - """ - - def __init__(self, loader, variable_manager, host_list=C.DEFAULT_HOST_LIST): - - # the host file file, or script path, or list of hosts - # if a list, inventory data will NOT be loaded - self.host_list = unfrackpath(host_list, follow=False) - self._loader = loader - self._variable_manager = variable_manager - self.localhost = None - - # caching to avoid repeated calculations, particularly with - # external inventory scripts. - - self._vars_per_host = {} - self._vars_per_group = {} - self._hosts_cache = {} - self._pattern_cache = {} - self._group_dict_cache = {} - self._vars_plugins = [] - - self._basedir = self.basedir() - - # Contains set of filenames under group_vars directories - self._group_vars_files = self._find_group_vars_files(self._basedir) - self._host_vars_files = self._find_host_vars_files(self._basedir) - - # to be set by calling set_playbook_basedir by playbook code - self._playbook_basedir = None - - # the inventory object holds a list of groups - self.groups = {} - - # a list of host(names) to contain current inquiries to - self._restriction = None - self._subset = None - - # clear the cache here, which is only useful if more than - # one Inventory objects are created when using the API directly - self.clear_pattern_cache() - self.clear_group_dict_cache() - - self.parse_inventory(host_list) - - def serialize(self): - data = dict() - return data - - def deserialize(self, data): - pass - - def parse_inventory(self, host_list): - - if isinstance(host_list, string_types): - if "," in host_list: - host_list = host_list.split(",") - host_list = [ h for h in host_list if h and h.strip() ] - - self.parser = None - - # Always create the 'all' and 'ungrouped' groups, even if host_list is - # empty: in this case we will subsequently add the implicit 'localhost' to it. - - ungrouped = Group('ungrouped') - all = Group('all') - all.add_child_group(ungrouped) - base_groups = frozenset([all, ungrouped]) - - self.groups = dict(all=all, ungrouped=ungrouped) - - if host_list is None: - pass - elif isinstance(host_list, list): - for h in host_list: - try: - (host, port) = parse_address(h, allow_ranges=False) - except AnsibleError as e: - display.vvv("Unable to parse address from hostname, leaving unchanged: %s" % to_text(e)) - host = h - port = None - - new_host = Host(host, port) - if h in C.LOCALHOST: - # set default localhost from inventory to avoid creating an implicit one. Last localhost defined 'wins'. - if self.localhost is not None: - display.warning("A duplicate localhost-like entry was found (%s). First found localhost was %s" % (h, self.localhost.name)) - display.vvvv("Set default localhost to %s" % h) - self.localhost = new_host - all.add_host(new_host) - elif self._loader.path_exists(host_list): - # TODO: switch this to a plugin loader and a 'condition' per plugin on which it should be tried, restoring 'inventory pllugins' - if self.is_directory(host_list): - # Ensure basedir is inside the directory - host_list = os.path.join(self.host_list, "") - self.parser = InventoryDirectory(loader=self._loader, groups=self.groups, filename=host_list) - else: - self.parser = get_file_parser(host_list, self.groups, self._loader) - vars_loader.add_directory(self._basedir, with_subdir=True) - - if not self.parser: - # should never happen, but JIC - raise AnsibleError("Unable to parse %s as an inventory source" % host_list) - else: - display.warning("Host file not found: %s" % to_text(host_list)) - - self._vars_plugins = [ x for x in vars_loader.all(self) ] - - ### POST PROCESS groups and hosts after specific parser was invoked - - hosts = [] - group_names = set() - # set group vars from group_vars/ files and vars plugins - for g in self.groups: - group = self.groups[g] - group.vars = combine_vars(group.vars, self.get_group_variables(group.name)) - self.get_group_vars(group) - group_names.add(group.name) - hosts.extend(group.get_hosts()) - - host_names = set() - # get host vars from host_vars/ files and vars plugins - for host in hosts: - host.vars = combine_vars(host.vars, self.get_host_variables(host.name)) - self.get_host_vars(host) - host_names.add(host.name) - - mygroups = host.get_groups() - - # ensure hosts are always in 'all' - if all not in mygroups: - all.add_host(host) - - if ungrouped in mygroups: - # clear ungrouped of any incorrectly stored by parser - if set(mygroups).difference(base_groups): - host.remove_group(ungrouped) - else: - # add ungrouped hosts to ungrouped - length = len(mygroups) - if length == 0 or (length == 1 and all in mygroups): - ungrouped.add_host(host) - - # warn if overloading identifier as both group and host - for conflict in group_names.intersection(host_names): - display.warning("Found both group and host with same name: %s" % conflict) - - def _match(self, str, pattern_str): - try: - if pattern_str.startswith('~'): - return re.search(pattern_str[1:], str) - else: - return fnmatch.fnmatch(str, pattern_str) - except Exception: - raise AnsibleError('invalid host pattern: %s' % pattern_str) - - def _match_list(self, items, item_attr, pattern_str): - results = [] - try: - if not pattern_str.startswith('~'): - pattern = re.compile(fnmatch.translate(pattern_str)) - else: - pattern = re.compile(pattern_str[1:]) - except Exception: - raise AnsibleError('invalid host pattern: %s' % pattern_str) - - for item in items: - if pattern.match(getattr(item, item_attr)): - results.append(item) - return results - - def get_hosts(self, pattern="all", ignore_limits=False, ignore_restrictions=False, order=None): - """ - Takes a pattern or list of patterns and returns a list of matching - inventory host names, taking into account any active restrictions - or applied subsets - """ - - # Check if pattern already computed - if isinstance(pattern, list): - pattern_hash = u":".join(pattern) - else: - pattern_hash = pattern - - if not ignore_limits and self._subset: - pattern_hash += u":%s" % to_text(self._subset) - - if not ignore_restrictions and self._restriction: - pattern_hash += u":%s" % to_text(self._restriction) - - if pattern_hash not in HOSTS_PATTERNS_CACHE: - - patterns = Inventory.split_host_pattern(pattern) - hosts = self._evaluate_patterns(patterns) - - # mainly useful for hostvars[host] access - if not ignore_limits and self._subset: - # exclude hosts not in a subset, if defined - subset = self._evaluate_patterns(self._subset) - hosts = [ h for h in hosts if h in subset ] - - if not ignore_restrictions and self._restriction: - # exclude hosts mentioned in any restriction (ex: failed hosts) - hosts = [ h for h in hosts if h.name in self._restriction ] - - seen = set() - HOSTS_PATTERNS_CACHE[pattern_hash] = [x for x in hosts if x not in seen and not seen.add(x)] - - # sort hosts list if needed (should only happen when called from strategy) - if order in ['sorted', 'reverse_sorted']: - from operator import attrgetter - hosts = sorted(HOSTS_PATTERNS_CACHE[pattern_hash][:], key=attrgetter('name'), reverse=(order == 'reverse_sorted')) - elif order == 'reverse_inventory': - hosts = sorted(HOSTS_PATTERNS_CACHE[pattern_hash][:], reverse=True) - else: - hosts = HOSTS_PATTERNS_CACHE[pattern_hash][:] - if order == 'shuffle': - from random import shuffle - shuffle(hosts) - elif order not in [None, 'inventory']: - AnsibleError("Invalid 'order' specified for inventory hosts: %s" % order) - - return hosts - - @classmethod - def split_host_pattern(cls, pattern): - """ - Takes a string containing host patterns separated by commas (or a list - thereof) and returns a list of single patterns (which may not contain - commas). Whitespace is ignored. - - Also accepts ':' as a separator for backwards compatibility, but it is - not recommended due to the conflict with IPv6 addresses and host ranges. - - Example: 'a,b[1], c[2:3] , d' -> ['a', 'b[1]', 'c[2:3]', 'd'] - """ - - if isinstance(pattern, list): - return list(itertools.chain(*map(cls.split_host_pattern, pattern))) - - # If it's got commas in it, we'll treat it as a straightforward - # comma-separated list of patterns. - - elif ',' in pattern: - patterns = re.split('\s*,\s*', pattern) - - # If it doesn't, it could still be a single pattern. This accounts for - # non-separator uses of colons: IPv6 addresses and [x:y] host ranges. - else: - try: - (base, port) = parse_address(pattern, allow_ranges=True) - patterns = [pattern] - except: - # The only other case we accept is a ':'-separated list of patterns. - # This mishandles IPv6 addresses, and is retained only for backwards - # compatibility. - patterns = re.findall( - r'''(?: # We want to match something comprising: - [^\s:\[\]] # (anything other than whitespace or ':[]' - | # ...or... - \[[^\]]*\] # a single complete bracketed expression) - )+ # occurring once or more - ''', pattern, re.X - ) - - return [p.strip() for p in patterns] - - @classmethod - def order_patterns(cls, patterns): - - # Host specifiers should be sorted to ensure consistent behavior - pattern_regular = [] - pattern_intersection = [] - pattern_exclude = [] - for p in patterns: - if p.startswith("!"): - pattern_exclude.append(p) - elif p.startswith("&"): - pattern_intersection.append(p) - elif p: - pattern_regular.append(p) - - # if no regular pattern was given, hence only exclude and/or intersection - # make that magically work - if pattern_regular == []: - pattern_regular = ['all'] - - # when applying the host selectors, run those without the "&" or "!" - # first, then the &s, then the !s. - return pattern_regular + pattern_intersection + pattern_exclude - - def _evaluate_patterns(self, patterns): - """ - Takes a list of patterns and returns a list of matching host names, - taking into account any negative and intersection patterns. - """ - - patterns = Inventory.order_patterns(patterns) - hosts = [] - - for p in patterns: - # avoid resolving a pattern that is a plain host - if p in self._hosts_cache: - hosts.append(self.get_host(p)) - else: - that = self._match_one_pattern(p) - if p.startswith("!"): - hosts = [ h for h in hosts if h not in that ] - elif p.startswith("&"): - hosts = [ h for h in hosts if h in that ] - else: - to_append = [ h for h in that if h.name not in [ y.name for y in hosts ] ] - hosts.extend(to_append) - return hosts - - def _match_one_pattern(self, pattern): - """ - Takes a single pattern and returns a list of matching host names. - Ignores intersection (&) and exclusion (!) specifiers. - - The pattern may be: - - 1. A regex starting with ~, e.g. '~[abc]*' - 2. A shell glob pattern with ?/*/[chars]/[!chars], e.g. 'foo*' - 3. An ordinary word that matches itself only, e.g. 'foo' - - The pattern is matched using the following rules: - - 1. If it's 'all', it matches all hosts in all groups. - 2. Otherwise, for each known group name: - (a) if it matches the group name, the results include all hosts - in the group or any of its children. - (b) otherwise, if it matches any hosts in the group, the results - include the matching hosts. - - This means that 'foo*' may match one or more groups (thus including all - hosts therein) but also hosts in other groups. - - The built-in groups 'all' and 'ungrouped' are special. No pattern can - match these group names (though 'all' behaves as though it matches, as - described above). The word 'ungrouped' can match a host of that name, - and patterns like 'ungr*' and 'al*' can match either hosts or groups - other than all and ungrouped. - - If the pattern matches one or more group names according to these rules, - it may have an optional range suffix to select a subset of the results. - This is allowed only if the pattern is not a regex, i.e. '~foo[1]' does - not work (the [1] is interpreted as part of the regex), but 'foo*[1]' - would work if 'foo*' matched the name of one or more groups. - - Duplicate matches are always eliminated from the results. - """ - - if pattern.startswith("&") or pattern.startswith("!"): - pattern = pattern[1:] - - if pattern not in self._pattern_cache: - (expr, slice) = self._split_subscript(pattern) - hosts = self._enumerate_matches(expr) - try: - hosts = self._apply_subscript(hosts, slice) - except IndexError: - raise AnsibleError("No hosts matched the subscripted pattern '%s'" % pattern) - self._pattern_cache[pattern] = hosts - - return self._pattern_cache[pattern] - - def _split_subscript(self, pattern): - """ - Takes a pattern, checks if it has a subscript, and returns the pattern - without the subscript and a (start,end) tuple representing the given - subscript (or None if there is no subscript). - - Validates that the subscript is in the right syntax, but doesn't make - sure the actual indices make sense in context. - """ - - # Do not parse regexes for enumeration info - if pattern.startswith('~'): - return (pattern, None) - - # We want a pattern followed by an integer or range subscript. - # (We can't be more restrictive about the expression because the - # fnmatch semantics permit [\[:\]] to occur.) - - pattern_with_subscript = re.compile( - r'''^ - (.+) # A pattern expression ending with... - \[(?: # A [subscript] expression comprising: - (-?[0-9]+)| # A single positive or negative number - ([0-9]+)([:-]) # Or an x:y or x: range. - ([0-9]*) - )\] - $ - ''', re.X - ) - - subscript = None - m = pattern_with_subscript.match(pattern) - if m: - (pattern, idx, start, sep, end) = m.groups() - if idx: - subscript = (int(idx), None) - else: - if not end: - end = -1 - subscript = (int(start), int(end)) - if sep == '-': - display.warning("Use [x:y] inclusive subscripts instead of [x-y] which has been removed") - - return (pattern, subscript) - - def _apply_subscript(self, hosts, subscript): - """ - Takes a list of hosts and a (start,end) tuple and returns the subset of - hosts based on the subscript (which may be None to return all hosts). - """ - - if not hosts or not subscript: - return hosts - - (start, end) = subscript - - if end: - if end == -1: - end = len(hosts)-1 - return hosts[start:end+1] - else: - return [ hosts[start] ] - - def _enumerate_matches(self, pattern): - """ - Returns a list of host names matching the given pattern according to the - rules explained above in _match_one_pattern. - """ - - results = [] - - def __append_host_to_results(host): - if host.name not in results: - if not host.implicit: - results.append(host) - - groups = self.get_groups() - matched = False - for group in groups.values(): - if self._match(group.name, pattern): - matched = True - for host in group.get_hosts(): - __append_host_to_results(host) - else: - matching_hosts = self._match_list(group.get_hosts(), 'name', pattern) - if matching_hosts: - matched = True - for host in matching_hosts: - __append_host_to_results(host) - - if pattern in C.LOCALHOST and len(results) == 0: - new_host = self._create_implicit_localhost(pattern) - results.append(new_host) - matched = True - - if not matched: - display.warning("Could not match supplied host pattern, ignoring: %s" % pattern) - return results - - def _create_implicit_localhost(self, pattern): - - if self.localhost: - new_host = self.localhost - else: - new_host = Host(pattern) - new_host.address = "127.0.0.1" - new_host.implicit = True - new_host.vars = self.get_host_vars(new_host) - new_host.set_variable("ansible_connection", "local") - if "ansible_python_interpreter" not in new_host.vars: - py_interp = sys.executable - if not py_interp: - # sys.executable is not set in some cornercases. #13585 - display.warning('Unable to determine python interpreter from sys.executable. Using /usr/bin/python default.' - ' You can correct this by setting ansible_python_interpreter for localhost') - py_interp = '/usr/bin/python' - new_host.set_variable("ansible_python_interpreter", py_interp) - self.get_group("ungrouped").add_host(new_host) - self.localhost = new_host - return new_host - - def clear_pattern_cache(self): - ''' called exclusively by the add_host plugin to allow patterns to be recalculated ''' - global HOSTS_PATTERNS_CACHE - HOSTS_PATTERNS_CACHE = {} - self._pattern_cache = {} - - def clear_group_dict_cache(self): - ''' called exclusively by the add_host and group_by plugins ''' - self._group_dict_cache = {} - - def groups_for_host(self, host): - if host in self._hosts_cache: - return self._hosts_cache[host].get_groups() - else: - return [] - - def get_groups(self): - return self.groups - - def get_host(self, hostname): - if hostname not in self._hosts_cache: - self._hosts_cache[hostname] = self._get_host(hostname) - return self._hosts_cache[hostname] - - def _get_host(self, hostname): - matching_host = None - if hostname in C.LOCALHOST: - if self.localhost: - matching_host= self.localhost - else: - for host in self.get_group('all').get_hosts(): - if host.name in C.LOCALHOST: - matching_host = host - break - if not matching_host: - matching_host = self._create_implicit_localhost(hostname) - # update caches - self._hosts_cache[hostname] = matching_host - for host in C.LOCALHOST.difference((hostname,)): - self._hosts_cache[host] = self._hosts_cache[hostname] - else: - for group in self.groups.values(): - for host in group.get_hosts(): - if host not in self._hosts_cache: - self._hosts_cache[host.name] = host - if hostname == host.name: - matching_host = host - return matching_host - - def get_group(self, groupname): - return self.groups.get(groupname) - - def get_group_variables(self, groupname, update_cached=False, vault_password=None): - if groupname not in self._vars_per_group or update_cached: - self._vars_per_group[groupname] = self._get_group_variables(groupname, vault_password=vault_password) - return self._vars_per_group[groupname] - - def _get_group_variables(self, groupname, vault_password=None): - - group = self.get_group(groupname) - if group is None: - raise Exception("group not found: %s" % groupname) - - vars = {} - - # plugin.get_group_vars retrieves just vars for specific group - vars_results = [ plugin.get_group_vars(group, vault_password=vault_password) for plugin in self._vars_plugins if hasattr(plugin, 'get_group_vars')] - for updated in vars_results: - if updated is not None: - vars = combine_vars(vars, updated) - - # Read group_vars/ files - vars = combine_vars(vars, self.get_group_vars(group)) - - return vars - - def get_group_dict(self): - """ - In get_vars() we merge a 'magic' dictionary 'groups' with group name - keys and hostname list values into every host variable set. - - Cache the creation of this structure here - """ - - if not self._group_dict_cache: - for (group_name, group) in iteritems(self.groups): - self._group_dict_cache[group_name] = [h.name for h in group.get_hosts()] - - return self._group_dict_cache - - def get_vars(self, hostname, update_cached=False, vault_password=None): - - host = self.get_host(hostname) - if not host: - raise AnsibleError("no vars as host is not in inventory: %s" % hostname) - return host.get_vars() - - def get_host_variables(self, hostname, update_cached=False, vault_password=None): - - if hostname not in self._vars_per_host or update_cached: - self._vars_per_host[hostname] = self._get_host_variables(hostname, vault_password=vault_password) - return self._vars_per_host[hostname] - - def _get_host_variables(self, hostname, vault_password=None): - - host = self.get_host(hostname) - if host is None: - raise AnsibleError("no host vars as host is not in inventory: %s" % hostname) - - vars = {} - - # plugin.run retrieves all vars (also from groups) for host - vars_results = [ plugin.run(host, vault_password=vault_password) for plugin in self._vars_plugins if hasattr(plugin, 'run')] - for updated in vars_results: - if updated is not None: - vars = combine_vars(vars, updated) - - # plugin.get_host_vars retrieves just vars for specific host - vars_results = [ plugin.get_host_vars(host, vault_password=vault_password) for plugin in self._vars_plugins if hasattr(plugin, 'get_host_vars')] - for updated in vars_results: - if updated is not None: - vars = combine_vars(vars, updated) - - # still need to check InventoryParser per host vars - # which actually means InventoryScript per host, - # which is not performant - if self.parser is not None: - vars = combine_vars(vars, self.parser.get_host_variables(host)) - - return vars - - def add_group(self, group): - if group.name not in self.groups: - self.groups[group.name] = group - else: - raise AnsibleError("group already in inventory: %s" % group.name) - - def list_hosts(self, pattern="all"): - - """ return a list of hostnames for a pattern """ - - result = [ h for h in self.get_hosts(pattern) ] - if len(result) == 0 and pattern in C.LOCALHOST: - result = [pattern] - return result - - def list_groups(self): - return sorted(self.groups.keys(), key=lambda x: x) - - def restrict_to_hosts(self, restriction): - """ - Restrict list operations to the hosts given in restriction. This is used - to batch serial operations in main playbook code, don't use this for other - reasons. - """ - if restriction is None: - return - elif not isinstance(restriction, list): - restriction = [ restriction ] - self._restriction = [ h.name for h in restriction ] - - def subset(self, subset_pattern): - """ - Limits inventory results to a subset of inventory that matches a given - pattern, such as to select a given geographic of numeric slice amongst - a previous 'hosts' selection that only select roles, or vice versa. - Corresponds to --limit parameter to ansible-playbook - """ - if subset_pattern is None: - self._subset = None - else: - subset_patterns = Inventory.split_host_pattern(subset_pattern) - results = [] - # allow Unix style @filename data - for x in subset_patterns: - if x.startswith("@"): - fd = open(x[1:]) - results.extend(fd.read().split("\n")) - fd.close() - else: - results.append(x) - self._subset = results - - def remove_restriction(self): - """ Do not restrict list operations """ - self._restriction = None - - def is_file(self): - """ - Did inventory come from a file? We don't use the equivalent loader - methods in inventory, due to the fact that the loader does an implict - DWIM on the path, which may be incorrect for inventory paths relative - to the playbook basedir. - """ - if not isinstance(self.host_list, string_types): - return False - return os.path.isfile(self.host_list) or self.host_list == os.devnull - - def is_directory(self, path): - """ - Is the inventory host list a directory? Same caveat for here as with - the is_file() method above. - """ - if not isinstance(self.host_list, string_types): - return False - return os.path.isdir(path) - - def basedir(self): - """ if inventory came from a file, what's the directory? """ - dname = self.host_list - if self.is_directory(self.host_list): - dname = self.host_list - elif not self.is_file(): - dname = None - else: - dname = os.path.dirname(self.host_list) - if dname is None or dname == '' or dname == '.': - dname = os.getcwd() - if dname: - dname = os.path.abspath(dname) - return dname - - def src(self): - """ if inventory came from a file, what's the directory and file name? """ - if not self.is_file(): - return None - return self.host_list - - def playbook_basedir(self): - """ returns the directory of the current playbook """ - return self._playbook_basedir - - def set_playbook_basedir(self, dir_name): - """ - sets the base directory of the playbook so inventory can use it as a - basedir for host_ and group_vars, and other things. - """ - # Only update things if dir is a different playbook basedir - if dir_name != self._playbook_basedir: - # we're changing the playbook basedir, so if we had set one previously - # clear the host/group vars entries from the VariableManager so they're - # not incorrectly used by playbooks from different directories - if self._playbook_basedir: - self._variable_manager.clear_playbook_hostgroup_vars_files(self._playbook_basedir) - - self._playbook_basedir = dir_name - # get group vars from group_vars/ files - # TODO: excluding the new_pb_basedir directory may result in group_vars - # files loading more than they should, however with the file caching - # we do this shouldn't be too much of an issue. Still, this should - # be fixed at some point to allow a "first load" to touch all of the - # directories, then later runs only touch the new basedir specified - found_group_vars = self._find_group_vars_files(self._playbook_basedir) - if found_group_vars: - self._group_vars_files = self._group_vars_files.union(found_group_vars) - for group in self.groups.values(): - self.get_group_vars(group) - - found_host_vars = self._find_host_vars_files(self._playbook_basedir) - if found_host_vars: - self._host_vars_files = self._host_vars_files.union(found_host_vars) - # get host vars from host_vars/ files - for host in self.get_hosts(): - self.get_host_vars(host) - # invalidate cache - self._vars_per_host = {} - self._vars_per_group = {} - - def get_host_vars(self, host, new_pb_basedir=False, return_results=False): - """ Read host_vars/ files """ - return self._get_hostgroup_vars(host=host, group=None, new_pb_basedir=new_pb_basedir, return_results=return_results) - - def get_group_vars(self, group, new_pb_basedir=False, return_results=False): - """ Read group_vars/ files """ - return self._get_hostgroup_vars(host=None, group=group, new_pb_basedir=new_pb_basedir, return_results=return_results) - - def _find_group_vars_files(self, basedir): - """ Find group_vars/ files """ - if basedir in ('', None): - basedir = './' - path = os.path.realpath(os.path.join(basedir, 'group_vars')) - found_vars = set() - if os.path.exists(path): - if os.path.isdir(path): - found_vars = set(os.listdir(to_text(path))) - else: - display.warning("Found group_vars that is not a directory, skipping: %s" % path) - return found_vars - - def _find_host_vars_files(self, basedir): - """ Find host_vars/ files """ - if basedir in ('', None): - basedir = './' - path = os.path.realpath(os.path.join(basedir, 'host_vars')) - found_vars = set() - if os.path.exists(path): - found_vars = set(os.listdir(to_text(path))) - return found_vars - - def _get_hostgroup_vars(self, host=None, group=None, new_pb_basedir=False, return_results=False): - """ - Loads variables from group_vars/ and host_vars/ in directories parallel - to the inventory base directory or in the same directory as the playbook. Variables in the playbook - dir will win over the inventory dir if files are in both. - """ - - results = {} - scan_pass = 0 - _basedir = self._basedir - _playbook_basedir = self._playbook_basedir - - # look in both the inventory base directory and the playbook base directory - # unless we do an update for a new playbook base dir - if not new_pb_basedir and _playbook_basedir: - basedirs = [_basedir, _playbook_basedir] - else: - basedirs = [_basedir] - - for basedir in basedirs: - # this can happen from particular API usages, particularly if not run - # from /usr/bin/ansible-playbook - if basedir in ('', None): - basedir = './' - - scan_pass = scan_pass + 1 - - # it's not an eror if the directory does not exist, keep moving - if not os.path.exists(basedir): - continue - - # save work of second scan if the directories are the same - if _basedir == _playbook_basedir and scan_pass != 1: - continue - - # Before trying to load vars from file, check that the directory contains relvant file names - if host is None and any(map(lambda ext: group.name + ext in self._group_vars_files, C.YAML_FILENAME_EXTENSIONS)): - # load vars in dir/group_vars/name_of_group - base_path = to_text(os.path.abspath(os.path.join(to_bytes(basedir), b"group_vars/" + to_bytes(group.name))), errors='surrogate_or_strict') - host_results = self._variable_manager.add_group_vars_file(base_path, self._loader) - if return_results: - results = combine_vars(results, host_results) - elif group is None and any(map(lambda ext: host.name + ext in self._host_vars_files, C.YAML_FILENAME_EXTENSIONS)): - # same for hostvars in dir/host_vars/name_of_host - base_path = to_text(os.path.abspath(os.path.join(to_bytes(basedir), b"host_vars/" + to_bytes(host.name))), errors='surrogate_or_strict') - group_results = self._variable_manager.add_host_vars_file(base_path, self._loader) - if return_results: - results = combine_vars(results, group_results) - - # all done, results is a dictionary of variables for this particular host. - return results - - def refresh_inventory(self): - - self.clear_pattern_cache() - self.clear_group_dict_cache() - - self._hosts_cache = {} - self._vars_per_host = {} - self._vars_per_group = {} - self.groups = {} - - self.parse_inventory(self.host_list) diff --git a/lib/ansible/inventory/data.py b/lib/ansible/inventory/data.py new file mode 100644 index 00000000000..a0d31086174 --- /dev/null +++ b/lib/ansible/inventory/data.py @@ -0,0 +1,281 @@ +# (c) 2012-2014, Michael DeHaan +# +# This file is part of Ansible +# +# Ansible is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# Ansible is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with Ansible. If not, see . + +############################################# +from __future__ import (absolute_import, division, print_function) +__metaclass__ = type + +import os +import sys +import re + +from ansible import constants as C +from ansible.errors import AnsibleError +from ansible.inventory.group import Group +from ansible.inventory.host import Host +from ansible.module_utils.six import iteritems +from ansible.module_utils._text import to_bytes +from ansible.plugins.cache import FactCache +from ansible.utils.vars import combine_vars +from ansible.utils.path import basedir + +try: + from __main__ import display +except ImportError: + from ansible.utils.display import Display + display = Display() + +class InventoryData(object): + """ + Holds inventory data (host and group objects). + Using it's methods should guarantee expected relationships and data. + """ + + def __init__(self): + + # the inventory object holds a list of groups + self.groups = {} + self.hosts = {} + + # provides 'groups' magic var, host object has group_names + self._groups_dict_cache = {} + + # current localhost, implicit or explicit + self.localhost = None + + self.current_source = None + + # Always create the 'all' and 'ungrouped' groups, + for group in ('all', 'ungrouped'): + self.add_group(group) + self.add_child('all', 'ungrouped') + + # prime cache + self.cache = FactCache() + + def serialize(self): + data = dict() + return data + + def deserialize(self, data): + pass + + def _create_implicit_localhost(self, pattern): + + if self.localhost: + new_host = self.localhost + else: + new_host = Host(pattern) + + # use 'all' vars but not part of all group + new_host.vars = self.groups['all'].get_vars() + + new_host.address = "127.0.0.1" + new_host.implicit = True + + if "ansible_python_interpreter" not in new_host.vars: + py_interp = sys.executable + if not py_interp: + # sys.executable is not set in some cornercases. #13585 + py_interp = '/usr/bin/python' + display.warning('Unable to determine python interpreter from sys.executable. Using /usr/bin/python default.' + ' You can correct this by setting ansible_python_interpreter for localhost') + new_host.set_variable("ansible_python_interpreter", py_interp) + + if "ansible_connection" not in new_host.vars: + new_host.set_variable("ansible_connection", 'local') + + self.localhost = new_host + + return new_host + + + def _scan_groups_for_host(self, hostname, localhost=False): + ''' in case something did not update inventory correctly, fallback to group scan ''' + + found = None + for group in self.groups.values(): + for host in group.get_hosts(): + if hostname == host.name: + found = host + break + if found: + break + + if found: + display.debug('Found host (%s) in groups but it was missing from main inventory' % hostname) + + return found + + + def reconcile_inventory(self): + ''' Ensure inventory basic rules, run after updates ''' + + display.debug('Reconcile groups and hosts in inventory.') + self.current_source = None + + group_names = set() + # set group vars from group_vars/ files and vars plugins + for g in self.groups: + group = self.groups[g] + group_names.add(group.name) + + host_names = set() + # get host vars from host_vars/ files and vars plugins + for host in self.hosts.values(): + host_names.add(host.name) + + mygroups = host.get_groups() + + # ensure hosts are always in 'all' + if 'all' not in mygroups and not host.implicit: + self.add_child('all', host.name) + + if self.groups['ungrouped'] in mygroups: + # clear ungrouped of any incorrectly stored by parser + if set(mygroups).difference(set([self.groups['all'], self.groups['ungrouped']])): + host.remove_group(self.groups['ungrouped']) + + elif not host.implicit: + # add ungrouped hosts to ungrouped, except implicit + length = len(mygroups) + if length == 0 or (length == 1 and all in mygroups): + self.add_child('ungrouped', host.name) + + # special case for implicit hosts + if host.implicit: + host.vars = combine_vars(self.groups['all'].get_vars(), host.vars) + + # warn if overloading identifier as both group and host + for conflict in group_names.intersection(host_names): + display.warning("Found both group and host with same name: %s" % conflict) + + self._groups_dict_cache = {} + + def get_host(self, hostname): + ''' fetch host object using name + deal with implicit localhost + and possible inconsistent inventory ''' + + matching_host = self.hosts.get(hostname, None) + + # if host is not in hosts dict + if matching_host is None: + + # might need to create implicit localhost + if hostname in C.LOCALHOST: + matching_host = self._create_implicit_localhost(hostname) + + # might be inconsistent inventory, search groups + if matching_host is None: + matching_host = self._scan_groups_for_host(hostname) + + # if found/created update hosts dict + if matching_host: + self.hosts[hostname] = matching_host + + return matching_host + + + def add_group(self, group): + ''' adds a group to inventory if not there already ''' + + if group not in self.groups: + g = Group(group) + self.groups[group] = g + self._groups_dict_cache = {} + display.debug("Added group %s to inventory" % group) + else: + display.debug("group %s already in inventory" % group) + + def add_host(self, host, group=None, port=None): + ''' adds a host to inventory and possibly a group if not there already ''' + + g = None + if group: + if group in self.groups: + g = self.groups[group] + else: + raise AnsibleError("Could not find group %s in inventory" % group) + + if host not in self.hosts: + h = Host(host, port) + self.hosts[host] = h + if self.current_source: # set to 'first source' in which host was encountered + self.set_variable(host, 'inventory_file', os.path.basename(self.current_source)) + self.set_variable(host, 'inventory_dir', basedir(self.current_source)) + else: + self.set_variable(host, 'inventory_file', None) + self.set_variable(host, 'inventory_dir', None) + display.debug("Added host %s to inventory" % (host)) + + # set default localhost from inventory to avoid creating an implicit one. Last localhost defined 'wins'. + if host in C.LOCALHOST: + if self.localhost is None: + self.localhost = self.hosts[host] + display.vvvv("Set default localhost to %s" % h) + else: + display.warning("A duplicate localhost-like entry was found (%s). First found localhost was %s" % (h, self.localhost.name)) + else: + h = self.hosts[host] + + if g and host not in g.get_hosts(): + g.add_host(h) + self._groups_dict_cache = {} + display.debug("Added host %s to group %s" % (host,group)) + + + def set_variable(self, entity, varname, value): + ''' sets a varible for an inventory object ''' + + if entity in self.groups: + inv_object = self.groups[entity] + elif entity in self.hosts: + inv_object = self.hosts[entity] + else: + raise AnsibleError("Could not identify group or host named %s" % entity) + + inv_object.set_variable(varname, value) + display.debug('set %s for %s' % (varname, entity)) + + + def add_child(self, group, child): + ''' Add host or group to group ''' + + if group in self.groups: + g = self.groups[group] + if child in self.groups: + g.add_child_group(self.groups[child]) + elif child in self.hosts: + g.add_host(self.hosts[child]) + else: + raise AnsibleError("%s is not a known host nor group" % child) + self._groups_dict_cache = {} + display.debug('Group %s now contains %s' % (group, child)) + else: + raise AnsibleError("%s is not a known group" % group) + + def get_groups_dict(self): + """ + We merge a 'magic' var 'groups' with group name keys and hostname list values into every host variable set. Cache for speed. + """ + if not self._groups_dict_cache: + for (group_name, group) in iteritems(self.groups): + self._groups_dict_cache[group_name] = [h.name for h in group.get_hosts()] + + return self._groups_dict_cache + diff --git a/lib/ansible/inventory/dir.py b/lib/ansible/inventory/dir.py deleted file mode 100644 index faccb79b853..00000000000 --- a/lib/ansible/inventory/dir.py +++ /dev/null @@ -1,299 +0,0 @@ -# (c) 2013, Daniel Hokka Zakrisson -# (c) 2014, Serge van Ginderachter -# -# This file is part of Ansible -# -# Ansible is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# Ansible is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Ansible. If not, see . - -############################################# -from __future__ import (absolute_import, division, print_function) -__metaclass__ = type - -import os - -from ansible import constants as C -from ansible.errors import AnsibleError -from ansible.utils.vars import combine_vars -from ansible.module_utils._text import to_native - -#FIXME: make into plugins -from ansible.inventory.ini import InventoryParser as InventoryINIParser -from ansible.inventory.yaml import InventoryParser as InventoryYAMLParser -from ansible.inventory.script import InventoryScript - -__all__ = ['get_file_parser'] - -def get_file_parser(hostsfile, groups, loader): - # check to see if the specified file starts with a - # shebang (#!/), so if an error is raised by the parser - # class we can show a more apropos error - - shebang_present = False - processed = False - myerr = [] - parser = None - - try: - with open(hostsfile, 'rb') as inv_file: - initial_chars = inv_file.read(2) - if initial_chars.startswith(b'#!'): - shebang_present = True - except: - pass - - #FIXME: make this 'plugin loop' - # script - if loader.is_executable(hostsfile): - try: - parser = InventoryScript(loader=loader, groups=groups, filename=hostsfile) - processed = True - except Exception as e: - myerr.append('Attempted to execute "%s" as inventory script: %s' % (hostsfile, to_native(e))) - elif shebang_present: - - myerr.append("The inventory file \'%s\' looks like it should be an executable inventory script, but is not marked executable. " - "Perhaps you want to correct this with `chmod +x %s`?" % (hostsfile, hostsfile)) - - # YAML/JSON - if not processed and not shebang_present and os.path.splitext(hostsfile)[-1] in C.YAML_FILENAME_EXTENSIONS: - try: - parser = InventoryYAMLParser(loader=loader, groups=groups, filename=hostsfile) - processed = True - except Exception as e: - myerr.append('Attempted to read "%s" as YAML: %s' % (to_native(hostsfile), to_native(e))) - - # ini - if not processed and not shebang_present: - try: - parser = InventoryINIParser(loader=loader, groups=groups, filename=hostsfile) - processed = True - except Exception as e: - myerr.append('Attempted to read "%s" as ini file: %s ' % (to_native(hostsfile), to_native(e))) - - if not processed and myerr: - raise AnsibleError('\n'.join(myerr)) - - return parser - -class InventoryDirectory(object): - ''' Host inventory parser for ansible using a directory of inventories. ''' - - def __init__(self, loader, groups=None, filename=C.DEFAULT_HOST_LIST): - if groups is None: - groups = dict() - - self.names = os.listdir(filename) - self.names.sort() - self.directory = filename - self.parsers = [] - self.hosts = {} - self.groups = groups - - self._loader = loader - - for i in self.names: - - # Skip files that end with certain extensions or characters - if any(i.endswith(ext) for ext in C.DEFAULT_INVENTORY_IGNORE): - continue - # Skip hidden files - if i.startswith('.') and not i.startswith('./'): - continue - # These are things inside of an inventory basedir - if i in ("host_vars", "group_vars", "vars_plugins"): - continue - fullpath = os.path.join(self.directory, i) - if os.path.isdir(fullpath): - parser = InventoryDirectory(loader=loader, groups=groups, filename=fullpath) - else: - parser = get_file_parser(fullpath, self.groups, loader) - if parser is None: - #FIXME: needs to use display - import warnings - warnings.warning("Could not find parser for %s, skipping" % fullpath) - continue - - self.parsers.append(parser) - - # retrieve all groups and hosts form the parser and add them to - # self, don't look at group lists yet, to avoid - # recursion trouble, but just make sure all objects exist in self - newgroups = parser.groups.values() - for group in newgroups: - for host in group.hosts: - self._add_host(host) - for group in newgroups: - self._add_group(group) - - # now check the objects lists so they contain only objects from - # self; membership data in groups is already fine (except all & - # ungrouped, see later), but might still reference objects not in self - for group in self.groups.values(): - # iterate on a copy of the lists, as those lists get changed in - # the loop - # list with group's child group objects: - for child in group.child_groups[:]: - if child != self.groups[child.name]: - group.child_groups.remove(child) - group.child_groups.append(self.groups[child.name]) - # list with group's parent group objects: - for parent in group.parent_groups[:]: - if parent != self.groups[parent.name]: - group.parent_groups.remove(parent) - group.parent_groups.append(self.groups[parent.name]) - # list with group's host objects: - for host in group.hosts[:]: - if host != self.hosts[host.name]: - group.hosts.remove(host) - group.hosts.append(self.hosts[host.name]) - # also check here that the group that contains host, is - # also contained in the host's group list - if group not in self.hosts[host.name].groups: - self.hosts[host.name].groups.append(group) - - # extra checks on special groups all and ungrouped - # remove hosts from 'ungrouped' if they became member of other groups - if 'ungrouped' in self.groups: - ungrouped = self.groups['ungrouped'] - # loop on a copy of ungrouped hosts, as we want to change that list - for host in frozenset(ungrouped.hosts): - if len(host.groups) > 1: - host.groups.remove(ungrouped) - ungrouped.hosts.remove(host) - - # remove hosts from 'all' if they became member of other groups - # all should only contain direct children, not grandchildren - # direct children should have dept == 1 - if 'all' in self.groups: - allgroup = self.groups['all' ] - # loop on a copy of all's child groups, as we want to change that list - for group in allgroup.child_groups[:]: - # groups might once have beeen added to all, and later be added - # to another group: we need to remove the link wit all then - if len(group.parent_groups) > 1 and allgroup in group.parent_groups: - # real children of all have just 1 parent, all - # this one has more, so not a direct child of all anymore - group.parent_groups.remove(allgroup) - allgroup.child_groups.remove(group) - elif allgroup not in group.parent_groups: - # this group was once added to all, but doesn't list it as - # a parent any more; the info in the group is the correct - # info - allgroup.child_groups.remove(group) - - def _add_group(self, group): - """ Merge an existing group or add a new one; - Track parent and child groups, and hosts of the new one """ - - if group.name not in self.groups: - # it's brand new, add him! - self.groups[group.name] = group - # the Group class does not (yet) implement __eq__/__ne__, - # so unlike Host we do a regular comparison here - if self.groups[group.name] != group: - # different object, merge - self._merge_groups(self.groups[group.name], group) - - def _add_host(self, host): - if host.name not in self.hosts: - # Papa's got a brand new host - self.hosts[host.name] = host - # because the __eq__/__ne__ methods in Host() compare the - # name fields rather than references, we use id() here to - # do the object comparison for merges - if self.hosts[host.name] != host: - # different object, merge - self._merge_hosts(self.hosts[host.name], host) - - def _merge_groups(self, group, newgroup): - """ Merge all of instance newgroup into group, - update parent/child relationships - group lists may still contain group objects that exist in self with - same name, but was instanciated as a different object in some other - inventory parser; these are handled later """ - - # name - if group.name != newgroup.name: - raise AnsibleError("Cannot merge inventory group %s with %s" % (group.name, newgroup.name)) - - # depth - group.depth = max([group.depth, newgroup.depth]) - - # hosts list (host objects are by now already added to self.hosts) - for host in newgroup.hosts: - grouphosts = dict([(h.name, h) for h in group.hosts]) - if host.name in grouphosts: - # same host name but different object, merge - self._merge_hosts(grouphosts[host.name], host) - else: - # new membership, add host to group from self - # group from self will also be added again to host.groups, but - # as different object - group.add_host(self.hosts[host.name]) - # now remove this the old object for group in host.groups - for hostgroup in [g for g in host.groups]: - if hostgroup.name == group.name and hostgroup != self.groups[group.name]: - self.hosts[host.name].groups.remove(hostgroup) - - # group child membership relation - for newchild in newgroup.child_groups: - # dict with existing child groups: - childgroups = dict([(g.name, g) for g in group.child_groups]) - # check if child of new group is already known as a child - if newchild.name not in childgroups: - self.groups[group.name].add_child_group(newchild) - - # group parent membership relation - for newparent in newgroup.parent_groups: - # dict with existing parent groups: - parentgroups = dict([(g.name, g) for g in group.parent_groups]) - # check if parent of new group is already known as a parent - if newparent.name not in parentgroups: - if newparent.name not in self.groups: - # group does not exist yet in self, import him - self.groups[newparent.name] = newparent - # group now exists but not yet as a parent here - self.groups[newparent.name].add_child_group(group) - - # variables - group.vars = combine_vars(group.vars, newgroup.vars) - - def _merge_hosts(self,host, newhost): - """ Merge all of instance newhost into host """ - - # name - if host.name != newhost.name: - raise AnsibleError("Cannot merge host %s with %s" % (host.name, newhost.name)) - - # group membership relation - for newgroup in newhost.groups: - # dict with existing groups: - hostgroups = dict([(g.name, g) for g in host.groups]) - # check if new group is already known as a group - if newgroup.name not in hostgroups: - if newgroup.name not in self.groups: - # group does not exist yet in self, import him - self.groups[newgroup.name] = newgroup - # group now exists but doesn't have host yet - self.groups[newgroup.name].add_host(host) - - # variables - host.vars = combine_vars(host.vars, newhost.vars) - - def get_host_variables(self, host): - """ Gets additional host variables from all inventories """ - vars = {} - for i in self.parsers: - vars.update(i.get_host_variables(host)) - return vars diff --git a/lib/ansible/inventory/group.py b/lib/ansible/inventory/group.py index 44799930506..b1059122e60 100644 --- a/lib/ansible/inventory/group.py +++ b/lib/ansible/inventory/group.py @@ -18,6 +18,7 @@ from __future__ import (absolute_import, division, print_function) __metaclass__ = type from ansible.errors import AnsibleError +from ansible.utils.vars import combine_vars class Group: ''' a group of ansible hosts ''' @@ -35,13 +36,12 @@ class Group: self._hosts_cache = None self.priority = 1 - #self.clear_hosts_cache() - #if self.name is None: - # raise Exception("group name is required") - def __repr__(self): return self.get_name() + def __str__(self): + return self.get_name() + def __getstate__(self): return self.serialize() @@ -139,7 +139,6 @@ class Group: if self._hosts_cache is None: self._hosts_cache = self._get_hosts() - return self._hosts_cache def _get_hosts(self): diff --git a/lib/ansible/inventory/helpers.py b/lib/ansible/inventory/helpers.py new file mode 100644 index 00000000000..7ea9dea7c30 --- /dev/null +++ b/lib/ansible/inventory/helpers.py @@ -0,0 +1,35 @@ +# (c) 2017, Ansible by RedHat Inc, +# +# This file is part of Ansible +# +# Ansible is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# Ansible is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with Ansible. If not, see . + +############################################# +from __future__ import (absolute_import, division, print_function) +__metaclass__ = type + +from ansible.utils.vars import combine_vars + + +def sort_groups(groups): + return sorted(groups, key=lambda g: (g.depth, g.priority, g.name)) + + +def get_group_vars(groups): + + results = {} + for group in sort_groups(groups): + results = combine_vars(results, group.get_vars()) + + return results diff --git a/lib/ansible/inventory/host.py b/lib/ansible/inventory/host.py index b520454e7cc..03927cc2ebc 100644 --- a/lib/ansible/inventory/host.py +++ b/lib/ansible/inventory/host.py @@ -46,6 +46,12 @@ class Host: def __hash__(self): return hash(self.name) + def __str__(self): + return self.get_name() + + def __repr__(self): + return self.get_name() + def serialize(self): groups = [] for group in self.groups: @@ -56,7 +62,6 @@ class Host: vars=self.vars.copy(), address=self.address, uuid=self._uuid, - gathered_facts=self._gathered_facts, groups=groups, implicit=self.implicit, ) @@ -78,47 +83,37 @@ class Host: def __init__(self, name=None, port=None, gen_uuid=True): - self.name = name self.vars = {} self.groups = [] + self._uuid = None + self.name = name self.address = name if port: self.set_variable('ansible_port', int(port)) - self._gathered_facts = False - self._uuid = None if gen_uuid: self._uuid = get_unique_id() self.implicit = False - def __repr__(self): - return self.get_name() - def get_name(self): return self.name - @property - def gathered_facts(self): - return self._gathered_facts - - def set_gathered_facts(self, gathered): - self._gathered_facts = gathered def populate_ancestors(self): - # populate ancestors for group in self.groups: self.add_group(group) def add_group(self, group): - # populate ancestors + # populate ancestors first for oldg in group.get_ancestors(): if oldg not in self.groups: self.add_group(oldg) + # actually add group if group not in self.groups: self.groups.append(group) @@ -136,25 +131,21 @@ class Host: else: self.remove_group(oldg) - def set_variable(self, key, value): + def set_variable(self, key, value): self.vars[key]=value def get_groups(self): return self.groups - def get_vars(self): - + def get_magic_vars(self): results = {} - results = combine_vars(results, self.vars) results['inventory_hostname'] = self.name results['inventory_hostname_short'] = self.name.split('.')[0] results['group_names'] = sorted([ g.name for g in self.get_groups() if g.name != 'all']) - return results - def get_group_vars(self): - results = {} - groups = self.get_groups() - for group in sorted(groups, key=lambda g: (g.depth, g.priority, g.name)): - results = combine_vars(results, group.get_vars()) - return results + return combine_vars(self.vars, results) + + def get_vars(self): + return combine_vars(self.vars, self.get_magic_vars()) + diff --git a/lib/ansible/inventory/manager.py b/lib/ansible/inventory/manager.py new file mode 100644 index 00000000000..64cb36584cf --- /dev/null +++ b/lib/ansible/inventory/manager.py @@ -0,0 +1,599 @@ +# (c) 2012-2014, Michael DeHaan +# +# This file is part of Ansible +# +# Ansible is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# Ansible is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with Ansible. If not, see . + +############################################# +from __future__ import (absolute_import, division, print_function) +__metaclass__ = type + +import fnmatch +import os +import re +import itertools + +from ansible import constants as C +from ansible.errors import AnsibleError, AnsibleOptionsError, AnsibleParserError +from ansible.inventory.data import InventoryData +from ansible.module_utils.six import string_types +from ansible.module_utils._text import to_bytes, to_text +from ansible.parsing.utils.addresses import parse_address +from ansible.plugins import PluginLoader +from ansible.utils.path import unfrackpath + +try: + from __main__ import display +except ImportError: + from ansible.utils.display import Display + display = Display() + +HOSTS_PATTERNS_CACHE = {} + +IGNORED_ALWAYS = [b"^\.", b"^host_vars$", b"^group_vars$", b"^vars_plugins$"] +IGNORED_PATTERNS = [ to_bytes(x) for x in C.INVENTORY_IGNORE_PATTERNS ] +IGNORED_EXTS = [b'%s$' % to_bytes(re.escape(x)) for x in C.INVENTORY_IGNORE_EXTS] + +IGNORED = re.compile(b'|'.join(IGNORED_ALWAYS + IGNORED_PATTERNS + IGNORED_EXTS)) + +def order_patterns(patterns): + ''' takes a list of patterns and reorders them by modifier to apply them consistently ''' + + # FIXME: this goes away if we apply patterns incrementally or by groups + pattern_regular = [] + pattern_intersection = [] + pattern_exclude = [] + for p in patterns: + if p.startswith("!"): + pattern_exclude.append(p) + elif p.startswith("&"): + pattern_intersection.append(p) + elif p: + pattern_regular.append(p) + + # if no regular pattern was given, hence only exclude and/or intersection + # make that magically work + if pattern_regular == []: + pattern_regular = ['all'] + + # when applying the host selectors, run those without the "&" or "!" + # first, then the &s, then the !s. + return pattern_regular + pattern_intersection + pattern_exclude + + +def split_host_pattern(pattern): + """ + Takes a string containing host patterns separated by commas (or a list + thereof) and returns a list of single patterns (which may not contain + commas). Whitespace is ignored. + + Also accepts ':' as a separator for backwards compatibility, but it is + not recommended due to the conflict with IPv6 addresses and host ranges. + + Example: 'a,b[1], c[2:3] , d' -> ['a', 'b[1]', 'c[2:3]', 'd'] + """ + + if isinstance(pattern, list): + return list(itertools.chain(*map(split_host_pattern, pattern))) + + # If it's got commas in it, we'll treat it as a straightforward + # comma-separated list of patterns. + + elif ',' in pattern: + patterns = re.split('\s*,\s*', pattern) + + # If it doesn't, it could still be a single pattern. This accounts for + # non-separator uses of colons: IPv6 addresses and [x:y] host ranges. + else: + try: + (base, port) = parse_address(pattern, allow_ranges=True) + patterns = [pattern] + except: + # The only other case we accept is a ':'-separated list of patterns. + # This mishandles IPv6 addresses, and is retained only for backwards + # compatibility. + patterns = re.findall( + r'''(?: # We want to match something comprising: + [^\s:\[\]] # (anything other than whitespace or ':[]' + | # ...or... + \[[^\]]*\] # a single complete bracketed expression) + )+ # occurring once or more + ''', pattern, re.X + ) + + return [p.strip() for p in patterns] + +class InventoryManager(object): + ''' Creates and manages inventory ''' + + def __init__(self, loader, sources=None): + + # base objects + self._loader = loader + self._inventory = InventoryData() + + # a list of host(names) to contain current inquiries to + self._restriction = None + self._subset = None + + # caches + self._pattern_cache = {} # resolved host patterns + self._inventory_plugins = {} # for generating inventory + + # the inventory dirs, files, script paths or lists of hosts + if sources is None: + self._sources = [] + elif isinstance(sources, string_types): + self._sources = [ sources ] + else: + self._sources = sources + + # get to work! + self.parse_sources() + + @property + def localhost(self): + return self._inventory.localhost + + @property + def groups(self): + return self._inventory.groups + + @property + def hosts(self): + return self._inventory.hosts + + def get_vars(self, *args, **kwargs): + return self._inventory.get_vars(args, kwargs) + + def add_host(self, host, group=None, port=None): + return self._inventory.add_host(host, group, port) + + def add_group(self, group): + return self._inventory.add_group(group) + + def get_groups_dict(self): + return self._inventory.get_groups_dict() + + def reconcile_inventory(self): + return self._inventory.reconcile_inventory() + + def get_host(self, hostname): + return self._inventory.get_host(hostname) + + def _setup_inventory_plugins(self): + ''' sets up loaded inventory plugins for usage ''' + + inventory_loader = PluginLoader( 'InventoryModule', 'ansible.plugins.inventory', 'inventory_plugins', 'inventory_plugins') + display.vvvv('setting up inventory plugins') + + for name in C.INVENTORY_ENABLED: + plugin = inventory_loader.get(name) + name = os.path.splitext(os.path.basename(plugin._original_path))[0] + self._inventory_plugins[name] = plugin + + if not self._inventory_plugins: + raise AnsibleError("No inventory plugins available to generate inventory, make sure you have at least one whitelisted.") + + def parse_sources(self, cache=True): + ''' iterate over inventory sources and parse each one to populate it''' + + self._setup_inventory_plugins() + + parsed = False + # allow for multiple inventory parsing + for source in self._sources: + + if source: + if ',' not in source: + source = unfrackpath(source, follow=False) + parse = self.parse_source(source, cache=cache) + if parse and not parsed: + parsed = True + + if parsed: + # do post processing + self._inventory.reconcile_inventory() + else: + display.warning("No inventory was parsed, only implicit localhost is available") + + self._inventory_plugins = {} + + def parse_source(self, source, cache=True): + ''' Generate or update inventory for the source provided ''' + + parsed = False + display.debug(u'Examining possible inventory source: %s' % source) + + b_source = to_bytes(source) + # process directories as a collection of inventories + if os.path.isdir(b_source): + display.debug(u'Searching for inventory files in directory: %s' % source) + for i in sorted(os.listdir(b_source)): + + display.debug(u'Considering %s' % i) + # Skip hidden files and stuff we explicitly ignore + if IGNORED.search(i): + continue + + # recursively deal with directory entries + fullpath = os.path.join(b_source, i) + parsed_this_one = self.parse_source(to_text(fullpath)) + display.debug(u'parsed %s as %s' % (fullpath, parsed_this_one)) + if not parsed: + parsed = parsed_this_one + else: + # left with strings or files, let plugins figure it out + + # set so new hosts can use for inventory_file/dir vasr + self._inventory.current_source = source + + # get inventory plugins if needed, there should always be at least one generator + if not self._inventory_plugins: + self._setup_inventory_plugins() + + # try source with each plugin + failures = [] + for plugin in self._inventory_plugins: + display.debug(u'Attempting to use plugin %s' % plugin) + + # initialize + inv = self._inventory_plugins[plugin] + if inv.verify_file(source): + try: + inv.parse(self._inventory, self._loader, source, cache=cache) + parsed = True + display.vvv(u'Parsed %s inventory source with %s plugin' % (to_text(source), plugin)) + break + except AnsibleParserError as e: + failures.append(u'\n* Failed to parse %s with %s inventory plugin: %s\n' %(to_text(source), plugin, to_text(e))) + else: + display.debug(u'%s did not meet %s requirements' % (to_text(source), plugin)) + else: + if failures: + # only if no plugin processed files should we show errors. + for fail in failures: + display.warning(fail) + + if not parsed: + display.warning(u"Unable to parse %s as an inventory source" % to_text(source)) + + # clear up, jic + self._inventory.current_source = None + + return parsed + + def clear_caches(self): + ''' clear all caches ''' + global HOSTS_PATTERNS_CACHE + HOSTS_PATTERNS_CACHE = {} + self._pattern_cache = {} + #FIXME: flush inventory cache + + def refresh_inventory(self): + ''' recalculate inventory ''' + + self.clear_caches() + self._inventory = InventoryData() + self.parse_sources(cache=False) + + def _match(self, string, pattern_str): + try: + if pattern_str.startswith('~'): + return re.search(pattern_str[1:], string) + else: + return fnmatch.fnmatch(string, pattern_str) + except Exception as e: + raise AnsibleError('invalid host pattern (%s): %s' % (pattern_str, str(e))) + + def _match_list(self, items, item_attr, pattern_str): + results = [] + try: + if not pattern_str.startswith('~'): + pattern = re.compile(fnmatch.translate(pattern_str)) + else: + pattern = re.compile(pattern_str[1:]) + except Exception: + raise AnsibleError('invalid host list pattern: %s' % pattern_str) + + for item in items: + if pattern.match(getattr(item, item_attr)): + results.append(item) + return results + + def get_hosts(self, pattern="all", ignore_limits=False, ignore_restrictions=False, order=None): + """ + Takes a pattern or list of patterns and returns a list of matching + inventory host names, taking into account any active restrictions + or applied subsets + """ + + # Check if pattern already computed + if isinstance(pattern, list): + pattern_hash = u":".join(pattern) + else: + pattern_hash = pattern + + if not ignore_limits and self._subset: + pattern_hash += u":%s" % to_text(self._subset) + + if not ignore_restrictions and self._restriction: + pattern_hash += u":%s" % to_text(self._restriction) + + if pattern_hash not in HOSTS_PATTERNS_CACHE: + + patterns = split_host_pattern(pattern) + hosts = self._evaluate_patterns(patterns) + + # mainly useful for hostvars[host] access + if not ignore_limits and self._subset: + # exclude hosts not in a subset, if defined + subset = self._evaluate_patterns(self._subset) + hosts = [ h for h in hosts if h in subset ] + + if not ignore_restrictions and self._restriction: + # exclude hosts mentioned in any restriction (ex: failed hosts) + hosts = [ h for h in hosts if h.name in self._restriction ] + + seen = set() + HOSTS_PATTERNS_CACHE[pattern_hash] = [x for x in hosts if x not in seen and not seen.add(x)] + + # sort hosts list if needed (should only happen when called from strategy) + if order in ['sorted', 'reverse_sorted']: + from operator import attrgetter + hosts = sorted(HOSTS_PATTERNS_CACHE[pattern_hash][:], key=attrgetter('name'), reverse=(order == 'reverse_sorted')) + elif order == 'reverse_inventory': + hosts = sorted(HOSTS_PATTERNS_CACHE[pattern_hash][:], reverse=True) + else: + hosts = HOSTS_PATTERNS_CACHE[pattern_hash][:] + if order == 'shuffle': + from random import shuffle + shuffle(hosts) + elif order not in [None, 'inventory']: + AnsibleOptionsError("Invalid 'order' specified for inventory hosts: %s" % order) + + return hosts + + + def _evaluate_patterns(self, patterns): + """ + Takes a list of patterns and returns a list of matching host names, + taking into account any negative and intersection patterns. + """ + + patterns = order_patterns(patterns) + hosts = [] + + for p in patterns: + # avoid resolving a pattern that is a plain host + if p in self._inventory.hosts: + hosts.append(self._inventory.get_host(p)) + else: + that = self._match_one_pattern(p) + if p.startswith("!"): + hosts = [ h for h in hosts if h not in that ] + elif p.startswith("&"): + hosts = [ h for h in hosts if h in that ] + else: + to_append = [ h for h in that if h.name not in [ y.name for y in hosts ] ] + hosts.extend(to_append) + return hosts + + def _match_one_pattern(self, pattern): + """ + Takes a single pattern and returns a list of matching host names. + Ignores intersection (&) and exclusion (!) specifiers. + + The pattern may be: + + 1. A regex starting with ~, e.g. '~[abc]*' + 2. A shell glob pattern with ?/*/[chars]/[!chars], e.g. 'foo*' + 3. An ordinary word that matches itself only, e.g. 'foo' + + The pattern is matched using the following rules: + + 1. If it's 'all', it matches all hosts in all groups. + 2. Otherwise, for each known group name: + (a) if it matches the group name, the results include all hosts + in the group or any of its children. + (b) otherwise, if it matches any hosts in the group, the results + include the matching hosts. + + This means that 'foo*' may match one or more groups (thus including all + hosts therein) but also hosts in other groups. + + The built-in groups 'all' and 'ungrouped' are special. No pattern can + match these group names (though 'all' behaves as though it matches, as + described above). The word 'ungrouped' can match a host of that name, + and patterns like 'ungr*' and 'al*' can match either hosts or groups + other than all and ungrouped. + + If the pattern matches one or more group names according to these rules, + it may have an optional range suffix to select a subset of the results. + This is allowed only if the pattern is not a regex, i.e. '~foo[1]' does + not work (the [1] is interpreted as part of the regex), but 'foo*[1]' + would work if 'foo*' matched the name of one or more groups. + + Duplicate matches are always eliminated from the results. + """ + + if pattern.startswith("&") or pattern.startswith("!"): + pattern = pattern[1:] + + if pattern not in self._pattern_cache: + (expr, slice) = self._split_subscript(pattern) + hosts = self._enumerate_matches(expr) + try: + hosts = self._apply_subscript(hosts, slice) + except IndexError: + raise AnsibleError("No hosts matched the subscripted pattern '%s'" % pattern) + self._pattern_cache[pattern] = hosts + + return self._pattern_cache[pattern] + + def _split_subscript(self, pattern): + """ + Takes a pattern, checks if it has a subscript, and returns the pattern + without the subscript and a (start,end) tuple representing the given + subscript (or None if there is no subscript). + + Validates that the subscript is in the right syntax, but doesn't make + sure the actual indices make sense in context. + """ + + # Do not parse regexes for enumeration info + if pattern.startswith('~'): + return (pattern, None) + + # We want a pattern followed by an integer or range subscript. + # (We can't be more restrictive about the expression because the + # fnmatch semantics permit [\[:\]] to occur.) + + pattern_with_subscript = re.compile( + r'''^ + (.+) # A pattern expression ending with... + \[(?: # A [subscript] expression comprising: + (-?[0-9]+)| # A single positive or negative number + ([0-9]+)([:-]) # Or an x:y or x: range. + ([0-9]*) + )\] + $ + ''', re.X + ) + + subscript = None + m = pattern_with_subscript.match(pattern) + if m: + (pattern, idx, start, sep, end) = m.groups() + if idx: + subscript = (int(idx), None) + else: + if not end: + end = -1 + subscript = (int(start), int(end)) + if sep == '-': + display.warning("Use [x:y] inclusive subscripts instead of [x-y] which has been removed") + + return (pattern, subscript) + + def _apply_subscript(self, hosts, subscript): + """ + Takes a list of hosts and a (start,end) tuple and returns the subset of + hosts based on the subscript (which may be None to return all hosts). + """ + + if not hosts or not subscript: + return hosts + + (start, end) = subscript + + if end: + if end == -1: + end = len(hosts)-1 + return hosts[start:end+1] + else: + return [ hosts[start] ] + + def _enumerate_matches(self, pattern): + """ + Returns a list of host names matching the given pattern according to the + rules explained above in _match_one_pattern. + """ + + results = [] + + def __append_host_to_results(host): + if host.name not in results: + if not host.implicit: + results.append(host) + + matched = False + for group in self._inventory.groups.values(): + if self._match(to_text(group.name), pattern): + matched = True + for host in group.get_hosts(): + __append_host_to_results(host) + else: + matching_hosts = self._match_list(group.get_hosts(), 'name', pattern) + if matching_hosts: + matched = True + for host in matching_hosts: + __append_host_to_results(host) + + if not results and pattern in C.LOCALHOST: + # get_host autocreates implicit when needed + implicit = self._inventory.get_host(pattern) + if implicit: + results.append(implicit) + matched = True + + if not matched: + display.warning("Could not match supplied host pattern, ignoring: %s" % pattern) + return results + + def list_hosts(self, pattern="all"): + """ return a list of hostnames for a pattern """ + #FIXME: cache? + result = [ h for h in self.get_hosts(pattern) ] + + # allow implicit localhost if pattern matches and no other results + if len(result) == 0 and pattern in C.LOCALHOST: + result = [pattern] + + return result + + def list_groups(self): + #FIXME: cache? + return sorted(self._inventory.groups.keys(), key=lambda x: x) + + def restrict_to_hosts(self, restriction): + """ + Restrict list operations to the hosts given in restriction. This is used + to batch serial operations in main playbook code, don't use this for other + reasons. + """ + if restriction is None: + return + elif not isinstance(restriction, list): + restriction = [ restriction ] + self._restriction = [ h.name for h in restriction ] + + def subset(self, subset_pattern): + """ + Limits inventory results to a subset of inventory that matches a given + pattern, such as to select a given geographic of numeric slice amongst + a previous 'hosts' selection that only select roles, or vice versa. + Corresponds to --limit parameter to ansible-playbook + """ + if subset_pattern is None: + self._subset = None + else: + subset_patterns = split_host_pattern(subset_pattern) + results = [] + # allow Unix style @filename data + for x in subset_patterns: + if x.startswith("@"): + fd = open(x[1:]) + results.extend(fd.read().split("\n")) + fd.close() + else: + results.append(x) + self._subset = results + + def remove_restriction(self): + """ Do not restrict list operations """ + self._restriction = None + + def clear_pattern_cache(self): + self._pattern_cache = {} diff --git a/lib/ansible/inventory/script.py b/lib/ansible/inventory/script.py deleted file mode 100644 index d9cc9d0a9f0..00000000000 --- a/lib/ansible/inventory/script.py +++ /dev/null @@ -1,170 +0,0 @@ -# (c) 2012-2014, Michael DeHaan -# -# This file is part of Ansible -# -# Ansible is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# Ansible is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Ansible. If not, see . - -############################################# -from __future__ import (absolute_import, division, print_function) -__metaclass__ = type - -import os -import subprocess -import sys -from collections import Mapping - -from ansible import constants as C -from ansible.errors import AnsibleError -from ansible.inventory.host import Host -from ansible.inventory.group import Group -from ansible.module_utils.basic import json_dict_bytes_to_unicode -from ansible.module_utils.six import iteritems -from ansible.module_utils._text import to_native, to_text - - -class InventoryScript: - ''' Host inventory parser for ansible using external inventory scripts. ''' - - def __init__(self, loader, groups=None, filename=C.DEFAULT_HOST_LIST): - if groups is None: - groups = dict() - - self._loader = loader - self.groups = groups - - # Support inventory scripts that are not prefixed with some - # path information but happen to be in the current working - # directory when '.' is not in PATH. - self.filename = os.path.abspath(filename) - cmd = [ self.filename, "--list" ] - try: - sp = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) - except OSError as e: - raise AnsibleError("problem running %s (%s)" % (' '.join(cmd), e)) - (stdout, stderr) = sp.communicate() - - if sp.returncode != 0: - raise AnsibleError("Inventory script (%s) had an execution error: %s " % (filename,stderr)) - - # make sure script output is unicode so that json loader will output - # unicode strings itself - try: - self.data = to_text(stdout, errors="strict") - except Exception as e: - raise AnsibleError("inventory data from {0} contained characters that cannot be interpreted as UTF-8: {1}".format(to_native(self.filename), - to_native(e))) - - # see comment about _meta below - self.host_vars_from_top = None - self._parse(stderr) - - def _parse(self, err): - - all_hosts = {} - - # not passing from_remote because data from CMDB is trusted - try: - self.raw = self._loader.load(self.data) - except Exception as e: - sys.stderr.write(to_native(err) + "\n") - raise AnsibleError("failed to parse executable inventory script results from {0}: {1}".format(to_native(self.filename), to_native(e))) - - if not isinstance(self.raw, Mapping): - sys.stderr.write(to_native(err) + "\n") - raise AnsibleError("failed to parse executable inventory script results from {0}: data needs to be formatted " - "as a json dict".format(to_native(self.filename))) - - group = None - for (group_name, data) in self.raw.items(): - - # in Ansible 1.3 and later, a "_meta" subelement may contain - # a variable "hostvars" which contains a hash for each host - # if this "hostvars" exists at all then do not call --host for each - # host. This is for efficiency and scripts should still return data - # if called with --host for backwards compat with 1.2 and earlier. - - if group_name == '_meta': - if 'hostvars' in data: - self.host_vars_from_top = data['hostvars'] - continue - - if group_name not in self.groups: - group = self.groups[group_name] = Group(group_name) - - group = self.groups[group_name] - host = None - - if not isinstance(data, dict): - data = {'hosts': data} - # is not those subkeys, then simplified syntax, host with vars - elif not any(k in data for k in ('hosts','vars','children')): - data = {'hosts': [group_name], 'vars': data} - - if 'hosts' in data: - if not isinstance(data['hosts'], list): - raise AnsibleError("You defined a group \"%s\" with bad " - "data for the host list:\n %s" % (group_name, data)) - - for hostname in data['hosts']: - if hostname not in all_hosts: - all_hosts[hostname] = Host(hostname) - host = all_hosts[hostname] - group.add_host(host) - - if 'vars' in data: - if not isinstance(data['vars'], dict): - raise AnsibleError("You defined a group \"%s\" with bad " - "data for variables:\n %s" % (group_name, data)) - - for k, v in iteritems(data['vars']): - group.set_variable(k, v) - - # Separate loop to ensure all groups are defined - for (group_name, data) in self.raw.items(): - if group_name == '_meta': - continue - if isinstance(data, dict) and 'children' in data: - for child_name in data['children']: - if child_name in self.groups: - self.groups[group_name].add_child_group(self.groups[child_name]) - - # Finally, add all top-level groups as children of 'all'. - # We exclude ungrouped here because it was already added as a child of - # 'all' at the time it was created. - - for group in self.groups.values(): - if group.depth == 0 and group.name not in ('all', 'ungrouped'): - self.groups['all'].add_child_group(group) - - def get_host_variables(self, host): - """ Runs