You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
ansible/lib/ansible/vars/plugins.py

125 lines
4.4 KiB
Python

# Copyright (c) 2018 Ansible Project
# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
optimize host_group_vars and vars plugin loading (#79945) * Improve host_group_vars efficiency: * normalize the basedir with `os.path.realpath()` once and cache it * cache missing paths/files * reduce the calls to `isinstance` Add a couple more general improvements in vars/plugins.py get_vars_from_path(): * call `PluginLoader.all()` once for vars plugins and reload specific plugins subsequently * don't reload legacy/builtin vars plugins that are not enabled Add a test for host_group_vars and legacy plugin loading Co-authored-by: Matt Davis <mrd@redhat.com> * changelog * Add a new is_stateless attribute to the vars plugin baseclass update integration tests to be quieter and use the same test pattern Fix deprecation and adjust test that didn't catch the issue (deprecation only occured when the value was False) move realpath cache to host_group_vars (do not smuggle call state as instance data) refactor under a single 'if cache:' statement Call os.path.isdir instead of always calling os.path.exists first. Just call os.path.exists to differentiate between missing and non-directory. remove call to super(VarsModule, self).get_vars() use the entity name as the cache key instead of variable location Remove isinstance checks and use a class attribute just in case any plugins are subclassing Host/Group Replace startswith by checking index 0 of the name instead, since host/group names are required * rename is_stateless to cache_instance to make it more clear what it does * add plugin instance cache using the path to plugin loader reduce loading stage option if a new instance isn't created don't require a known subdir on PluginLoader instantiation for backwards compatibility rename attribute again contain reading from/initializing cached instances to a plugin loader method * Deprecate v2 vars plugins * Refactor to use the cache in existing plugin loader methods Rename the attribute again Refactor host_group_vars with requested changes Make changelog a bugfixes fragment Add a deprecation fragment for v2 vars plugins. Add type hints * unbreak group_vars * Apply suggestions from code review * misc tweaks * always cache instance by both requested and resolved FQ name * add lru_cache to stage calculation to avoid repeated config consultation * handle KeyError from missing stage option --------- Co-authored-by: Matt Davis <mrd@redhat.com>
8 months ago
from __future__ import annotations
import os
optimize host_group_vars and vars plugin loading (#79945) * Improve host_group_vars efficiency: * normalize the basedir with `os.path.realpath()` once and cache it * cache missing paths/files * reduce the calls to `isinstance` Add a couple more general improvements in vars/plugins.py get_vars_from_path(): * call `PluginLoader.all()` once for vars plugins and reload specific plugins subsequently * don't reload legacy/builtin vars plugins that are not enabled Add a test for host_group_vars and legacy plugin loading Co-authored-by: Matt Davis <mrd@redhat.com> * changelog * Add a new is_stateless attribute to the vars plugin baseclass update integration tests to be quieter and use the same test pattern Fix deprecation and adjust test that didn't catch the issue (deprecation only occured when the value was False) move realpath cache to host_group_vars (do not smuggle call state as instance data) refactor under a single 'if cache:' statement Call os.path.isdir instead of always calling os.path.exists first. Just call os.path.exists to differentiate between missing and non-directory. remove call to super(VarsModule, self).get_vars() use the entity name as the cache key instead of variable location Remove isinstance checks and use a class attribute just in case any plugins are subclassing Host/Group Replace startswith by checking index 0 of the name instead, since host/group names are required * rename is_stateless to cache_instance to make it more clear what it does * add plugin instance cache using the path to plugin loader reduce loading stage option if a new instance isn't created don't require a known subdir on PluginLoader instantiation for backwards compatibility rename attribute again contain reading from/initializing cached instances to a plugin loader method * Deprecate v2 vars plugins * Refactor to use the cache in existing plugin loader methods Rename the attribute again Refactor host_group_vars with requested changes Make changelog a bugfixes fragment Add a deprecation fragment for v2 vars plugins. Add type hints * unbreak group_vars * Apply suggestions from code review * misc tweaks * always cache instance by both requested and resolved FQ name * add lru_cache to stage calculation to avoid repeated config consultation * handle KeyError from missing stage option --------- Co-authored-by: Matt Davis <mrd@redhat.com>
8 months ago
from functools import lru_cache
from ansible import constants as C
from ansible.errors import AnsibleError
optimize host_group_vars and vars plugin loading (#79945) * Improve host_group_vars efficiency: * normalize the basedir with `os.path.realpath()` once and cache it * cache missing paths/files * reduce the calls to `isinstance` Add a couple more general improvements in vars/plugins.py get_vars_from_path(): * call `PluginLoader.all()` once for vars plugins and reload specific plugins subsequently * don't reload legacy/builtin vars plugins that are not enabled Add a test for host_group_vars and legacy plugin loading Co-authored-by: Matt Davis <mrd@redhat.com> * changelog * Add a new is_stateless attribute to the vars plugin baseclass update integration tests to be quieter and use the same test pattern Fix deprecation and adjust test that didn't catch the issue (deprecation only occured when the value was False) move realpath cache to host_group_vars (do not smuggle call state as instance data) refactor under a single 'if cache:' statement Call os.path.isdir instead of always calling os.path.exists first. Just call os.path.exists to differentiate between missing and non-directory. remove call to super(VarsModule, self).get_vars() use the entity name as the cache key instead of variable location Remove isinstance checks and use a class attribute just in case any plugins are subclassing Host/Group Replace startswith by checking index 0 of the name instead, since host/group names are required * rename is_stateless to cache_instance to make it more clear what it does * add plugin instance cache using the path to plugin loader reduce loading stage option if a new instance isn't created don't require a known subdir on PluginLoader instantiation for backwards compatibility rename attribute again contain reading from/initializing cached instances to a plugin loader method * Deprecate v2 vars plugins * Refactor to use the cache in existing plugin loader methods Rename the attribute again Refactor host_group_vars with requested changes Make changelog a bugfixes fragment Add a deprecation fragment for v2 vars plugins. Add type hints * unbreak group_vars * Apply suggestions from code review * misc tweaks * always cache instance by both requested and resolved FQ name * add lru_cache to stage calculation to avoid repeated config consultation * handle KeyError from missing stage option --------- Co-authored-by: Matt Davis <mrd@redhat.com>
8 months ago
from ansible.inventory.group import InventoryObjectType
from ansible.plugins.loader import vars_loader
from ansible.utils.display import Display
from ansible.utils.vars import combine_vars
display = Display()
optimize host_group_vars and vars plugin loading (#79945) * Improve host_group_vars efficiency: * normalize the basedir with `os.path.realpath()` once and cache it * cache missing paths/files * reduce the calls to `isinstance` Add a couple more general improvements in vars/plugins.py get_vars_from_path(): * call `PluginLoader.all()` once for vars plugins and reload specific plugins subsequently * don't reload legacy/builtin vars plugins that are not enabled Add a test for host_group_vars and legacy plugin loading Co-authored-by: Matt Davis <mrd@redhat.com> * changelog * Add a new is_stateless attribute to the vars plugin baseclass update integration tests to be quieter and use the same test pattern Fix deprecation and adjust test that didn't catch the issue (deprecation only occured when the value was False) move realpath cache to host_group_vars (do not smuggle call state as instance data) refactor under a single 'if cache:' statement Call os.path.isdir instead of always calling os.path.exists first. Just call os.path.exists to differentiate between missing and non-directory. remove call to super(VarsModule, self).get_vars() use the entity name as the cache key instead of variable location Remove isinstance checks and use a class attribute just in case any plugins are subclassing Host/Group Replace startswith by checking index 0 of the name instead, since host/group names are required * rename is_stateless to cache_instance to make it more clear what it does * add plugin instance cache using the path to plugin loader reduce loading stage option if a new instance isn't created don't require a known subdir on PluginLoader instantiation for backwards compatibility rename attribute again contain reading from/initializing cached instances to a plugin loader method * Deprecate v2 vars plugins * Refactor to use the cache in existing plugin loader methods Rename the attribute again Refactor host_group_vars with requested changes Make changelog a bugfixes fragment Add a deprecation fragment for v2 vars plugins. Add type hints * unbreak group_vars * Apply suggestions from code review * misc tweaks * always cache instance by both requested and resolved FQ name * add lru_cache to stage calculation to avoid repeated config consultation * handle KeyError from missing stage option --------- Co-authored-by: Matt Davis <mrd@redhat.com>
8 months ago
def _prime_vars_loader():
optimize host_group_vars and vars plugin loading (#79945) * Improve host_group_vars efficiency: * normalize the basedir with `os.path.realpath()` once and cache it * cache missing paths/files * reduce the calls to `isinstance` Add a couple more general improvements in vars/plugins.py get_vars_from_path(): * call `PluginLoader.all()` once for vars plugins and reload specific plugins subsequently * don't reload legacy/builtin vars plugins that are not enabled Add a test for host_group_vars and legacy plugin loading Co-authored-by: Matt Davis <mrd@redhat.com> * changelog * Add a new is_stateless attribute to the vars plugin baseclass update integration tests to be quieter and use the same test pattern Fix deprecation and adjust test that didn't catch the issue (deprecation only occured when the value was False) move realpath cache to host_group_vars (do not smuggle call state as instance data) refactor under a single 'if cache:' statement Call os.path.isdir instead of always calling os.path.exists first. Just call os.path.exists to differentiate between missing and non-directory. remove call to super(VarsModule, self).get_vars() use the entity name as the cache key instead of variable location Remove isinstance checks and use a class attribute just in case any plugins are subclassing Host/Group Replace startswith by checking index 0 of the name instead, since host/group names are required * rename is_stateless to cache_instance to make it more clear what it does * add plugin instance cache using the path to plugin loader reduce loading stage option if a new instance isn't created don't require a known subdir on PluginLoader instantiation for backwards compatibility rename attribute again contain reading from/initializing cached instances to a plugin loader method * Deprecate v2 vars plugins * Refactor to use the cache in existing plugin loader methods Rename the attribute again Refactor host_group_vars with requested changes Make changelog a bugfixes fragment Add a deprecation fragment for v2 vars plugins. Add type hints * unbreak group_vars * Apply suggestions from code review * misc tweaks * always cache instance by both requested and resolved FQ name * add lru_cache to stage calculation to avoid repeated config consultation * handle KeyError from missing stage option --------- Co-authored-by: Matt Davis <mrd@redhat.com>
8 months ago
# find 3rd party legacy vars plugins once, and look them up by name subsequently
list(vars_loader.all(class_only=True))
optimize host_group_vars and vars plugin loading (#79945) * Improve host_group_vars efficiency: * normalize the basedir with `os.path.realpath()` once and cache it * cache missing paths/files * reduce the calls to `isinstance` Add a couple more general improvements in vars/plugins.py get_vars_from_path(): * call `PluginLoader.all()` once for vars plugins and reload specific plugins subsequently * don't reload legacy/builtin vars plugins that are not enabled Add a test for host_group_vars and legacy plugin loading Co-authored-by: Matt Davis <mrd@redhat.com> * changelog * Add a new is_stateless attribute to the vars plugin baseclass update integration tests to be quieter and use the same test pattern Fix deprecation and adjust test that didn't catch the issue (deprecation only occured when the value was False) move realpath cache to host_group_vars (do not smuggle call state as instance data) refactor under a single 'if cache:' statement Call os.path.isdir instead of always calling os.path.exists first. Just call os.path.exists to differentiate between missing and non-directory. remove call to super(VarsModule, self).get_vars() use the entity name as the cache key instead of variable location Remove isinstance checks and use a class attribute just in case any plugins are subclassing Host/Group Replace startswith by checking index 0 of the name instead, since host/group names are required * rename is_stateless to cache_instance to make it more clear what it does * add plugin instance cache using the path to plugin loader reduce loading stage option if a new instance isn't created don't require a known subdir on PluginLoader instantiation for backwards compatibility rename attribute again contain reading from/initializing cached instances to a plugin loader method * Deprecate v2 vars plugins * Refactor to use the cache in existing plugin loader methods Rename the attribute again Refactor host_group_vars with requested changes Make changelog a bugfixes fragment Add a deprecation fragment for v2 vars plugins. Add type hints * unbreak group_vars * Apply suggestions from code review * misc tweaks * always cache instance by both requested and resolved FQ name * add lru_cache to stage calculation to avoid repeated config consultation * handle KeyError from missing stage option --------- Co-authored-by: Matt Davis <mrd@redhat.com>
8 months ago
for plugin_name in C.VARIABLE_PLUGINS_ENABLED:
if not plugin_name:
continue
vars_loader.get(plugin_name)
optimize host_group_vars and vars plugin loading (#79945) * Improve host_group_vars efficiency: * normalize the basedir with `os.path.realpath()` once and cache it * cache missing paths/files * reduce the calls to `isinstance` Add a couple more general improvements in vars/plugins.py get_vars_from_path(): * call `PluginLoader.all()` once for vars plugins and reload specific plugins subsequently * don't reload legacy/builtin vars plugins that are not enabled Add a test for host_group_vars and legacy plugin loading Co-authored-by: Matt Davis <mrd@redhat.com> * changelog * Add a new is_stateless attribute to the vars plugin baseclass update integration tests to be quieter and use the same test pattern Fix deprecation and adjust test that didn't catch the issue (deprecation only occured when the value was False) move realpath cache to host_group_vars (do not smuggle call state as instance data) refactor under a single 'if cache:' statement Call os.path.isdir instead of always calling os.path.exists first. Just call os.path.exists to differentiate between missing and non-directory. remove call to super(VarsModule, self).get_vars() use the entity name as the cache key instead of variable location Remove isinstance checks and use a class attribute just in case any plugins are subclassing Host/Group Replace startswith by checking index 0 of the name instead, since host/group names are required * rename is_stateless to cache_instance to make it more clear what it does * add plugin instance cache using the path to plugin loader reduce loading stage option if a new instance isn't created don't require a known subdir on PluginLoader instantiation for backwards compatibility rename attribute again contain reading from/initializing cached instances to a plugin loader method * Deprecate v2 vars plugins * Refactor to use the cache in existing plugin loader methods Rename the attribute again Refactor host_group_vars with requested changes Make changelog a bugfixes fragment Add a deprecation fragment for v2 vars plugins. Add type hints * unbreak group_vars * Apply suggestions from code review * misc tweaks * always cache instance by both requested and resolved FQ name * add lru_cache to stage calculation to avoid repeated config consultation * handle KeyError from missing stage option --------- Co-authored-by: Matt Davis <mrd@redhat.com>
8 months ago
def get_plugin_vars(loader, plugin, path, entities):
data = {}
try:
data = plugin.get_vars(loader, path, entities)
except AttributeError:
optimize host_group_vars and vars plugin loading (#79945) * Improve host_group_vars efficiency: * normalize the basedir with `os.path.realpath()` once and cache it * cache missing paths/files * reduce the calls to `isinstance` Add a couple more general improvements in vars/plugins.py get_vars_from_path(): * call `PluginLoader.all()` once for vars plugins and reload specific plugins subsequently * don't reload legacy/builtin vars plugins that are not enabled Add a test for host_group_vars and legacy plugin loading Co-authored-by: Matt Davis <mrd@redhat.com> * changelog * Add a new is_stateless attribute to the vars plugin baseclass update integration tests to be quieter and use the same test pattern Fix deprecation and adjust test that didn't catch the issue (deprecation only occured when the value was False) move realpath cache to host_group_vars (do not smuggle call state as instance data) refactor under a single 'if cache:' statement Call os.path.isdir instead of always calling os.path.exists first. Just call os.path.exists to differentiate between missing and non-directory. remove call to super(VarsModule, self).get_vars() use the entity name as the cache key instead of variable location Remove isinstance checks and use a class attribute just in case any plugins are subclassing Host/Group Replace startswith by checking index 0 of the name instead, since host/group names are required * rename is_stateless to cache_instance to make it more clear what it does * add plugin instance cache using the path to plugin loader reduce loading stage option if a new instance isn't created don't require a known subdir on PluginLoader instantiation for backwards compatibility rename attribute again contain reading from/initializing cached instances to a plugin loader method * Deprecate v2 vars plugins * Refactor to use the cache in existing plugin loader methods Rename the attribute again Refactor host_group_vars with requested changes Make changelog a bugfixes fragment Add a deprecation fragment for v2 vars plugins. Add type hints * unbreak group_vars * Apply suggestions from code review * misc tweaks * always cache instance by both requested and resolved FQ name * add lru_cache to stage calculation to avoid repeated config consultation * handle KeyError from missing stage option --------- Co-authored-by: Matt Davis <mrd@redhat.com>
8 months ago
if hasattr(plugin, 'get_host_vars') or hasattr(plugin, 'get_group_vars'):
display.deprecated(
f"The vars plugin {plugin.ansible_name} from {plugin._original_path} is relying "
"on the deprecated entrypoints 'get_host_vars' and 'get_group_vars'. "
"This plugin should be updated to inherit from BaseVarsPlugin and define "
"a 'get_vars' method as the main entrypoint instead.",
version="2.20",
)
try:
for entity in entities:
optimize host_group_vars and vars plugin loading (#79945) * Improve host_group_vars efficiency: * normalize the basedir with `os.path.realpath()` once and cache it * cache missing paths/files * reduce the calls to `isinstance` Add a couple more general improvements in vars/plugins.py get_vars_from_path(): * call `PluginLoader.all()` once for vars plugins and reload specific plugins subsequently * don't reload legacy/builtin vars plugins that are not enabled Add a test for host_group_vars and legacy plugin loading Co-authored-by: Matt Davis <mrd@redhat.com> * changelog * Add a new is_stateless attribute to the vars plugin baseclass update integration tests to be quieter and use the same test pattern Fix deprecation and adjust test that didn't catch the issue (deprecation only occured when the value was False) move realpath cache to host_group_vars (do not smuggle call state as instance data) refactor under a single 'if cache:' statement Call os.path.isdir instead of always calling os.path.exists first. Just call os.path.exists to differentiate between missing and non-directory. remove call to super(VarsModule, self).get_vars() use the entity name as the cache key instead of variable location Remove isinstance checks and use a class attribute just in case any plugins are subclassing Host/Group Replace startswith by checking index 0 of the name instead, since host/group names are required * rename is_stateless to cache_instance to make it more clear what it does * add plugin instance cache using the path to plugin loader reduce loading stage option if a new instance isn't created don't require a known subdir on PluginLoader instantiation for backwards compatibility rename attribute again contain reading from/initializing cached instances to a plugin loader method * Deprecate v2 vars plugins * Refactor to use the cache in existing plugin loader methods Rename the attribute again Refactor host_group_vars with requested changes Make changelog a bugfixes fragment Add a deprecation fragment for v2 vars plugins. Add type hints * unbreak group_vars * Apply suggestions from code review * misc tweaks * always cache instance by both requested and resolved FQ name * add lru_cache to stage calculation to avoid repeated config consultation * handle KeyError from missing stage option --------- Co-authored-by: Matt Davis <mrd@redhat.com>
8 months ago
if entity.base_type is InventoryObjectType.HOST:
data |= plugin.get_host_vars(entity.name)
else:
data |= plugin.get_group_vars(entity.name)
except AttributeError:
if hasattr(plugin, 'run'):
raise AnsibleError("Cannot use v1 type vars plugin %s from %s" % (plugin._load_name, plugin._original_path))
else:
raise AnsibleError("Invalid vars plugin %s from %s" % (plugin._load_name, plugin._original_path))
return data
optimize host_group_vars and vars plugin loading (#79945) * Improve host_group_vars efficiency: * normalize the basedir with `os.path.realpath()` once and cache it * cache missing paths/files * reduce the calls to `isinstance` Add a couple more general improvements in vars/plugins.py get_vars_from_path(): * call `PluginLoader.all()` once for vars plugins and reload specific plugins subsequently * don't reload legacy/builtin vars plugins that are not enabled Add a test for host_group_vars and legacy plugin loading Co-authored-by: Matt Davis <mrd@redhat.com> * changelog * Add a new is_stateless attribute to the vars plugin baseclass update integration tests to be quieter and use the same test pattern Fix deprecation and adjust test that didn't catch the issue (deprecation only occured when the value was False) move realpath cache to host_group_vars (do not smuggle call state as instance data) refactor under a single 'if cache:' statement Call os.path.isdir instead of always calling os.path.exists first. Just call os.path.exists to differentiate between missing and non-directory. remove call to super(VarsModule, self).get_vars() use the entity name as the cache key instead of variable location Remove isinstance checks and use a class attribute just in case any plugins are subclassing Host/Group Replace startswith by checking index 0 of the name instead, since host/group names are required * rename is_stateless to cache_instance to make it more clear what it does * add plugin instance cache using the path to plugin loader reduce loading stage option if a new instance isn't created don't require a known subdir on PluginLoader instantiation for backwards compatibility rename attribute again contain reading from/initializing cached instances to a plugin loader method * Deprecate v2 vars plugins * Refactor to use the cache in existing plugin loader methods Rename the attribute again Refactor host_group_vars with requested changes Make changelog a bugfixes fragment Add a deprecation fragment for v2 vars plugins. Add type hints * unbreak group_vars * Apply suggestions from code review * misc tweaks * always cache instance by both requested and resolved FQ name * add lru_cache to stage calculation to avoid repeated config consultation * handle KeyError from missing stage option --------- Co-authored-by: Matt Davis <mrd@redhat.com>
8 months ago
# optimized for stateless plugins; non-stateless plugin instances will fall out quickly
@lru_cache(maxsize=10)
def _plugin_should_run(plugin, stage):
# if a plugin-specific setting has not been provided, use the global setting
# older/non shipped plugins that don't support the plugin-specific setting should also use the global setting
allowed_stages = None
try:
allowed_stages = plugin.get_option('stage')
except (AttributeError, KeyError):
pass
if allowed_stages:
return allowed_stages in ('all', stage)
# plugin didn't declare a preference; consult global config
config_stage_override = C.RUN_VARS_PLUGINS
if config_stage_override == 'demand' and stage == 'inventory':
return False
elif config_stage_override == 'start' and stage == 'task':
return False
return True
def get_vars_from_path(loader, path, entities, stage):
data = {}
if vars_loader._paths is None:
# cache has been reset, reload all()
_prime_vars_loader()
for plugin_name in vars_loader._plugin_instance_cache:
optimize host_group_vars and vars plugin loading (#79945) * Improve host_group_vars efficiency: * normalize the basedir with `os.path.realpath()` once and cache it * cache missing paths/files * reduce the calls to `isinstance` Add a couple more general improvements in vars/plugins.py get_vars_from_path(): * call `PluginLoader.all()` once for vars plugins and reload specific plugins subsequently * don't reload legacy/builtin vars plugins that are not enabled Add a test for host_group_vars and legacy plugin loading Co-authored-by: Matt Davis <mrd@redhat.com> * changelog * Add a new is_stateless attribute to the vars plugin baseclass update integration tests to be quieter and use the same test pattern Fix deprecation and adjust test that didn't catch the issue (deprecation only occured when the value was False) move realpath cache to host_group_vars (do not smuggle call state as instance data) refactor under a single 'if cache:' statement Call os.path.isdir instead of always calling os.path.exists first. Just call os.path.exists to differentiate between missing and non-directory. remove call to super(VarsModule, self).get_vars() use the entity name as the cache key instead of variable location Remove isinstance checks and use a class attribute just in case any plugins are subclassing Host/Group Replace startswith by checking index 0 of the name instead, since host/group names are required * rename is_stateless to cache_instance to make it more clear what it does * add plugin instance cache using the path to plugin loader reduce loading stage option if a new instance isn't created don't require a known subdir on PluginLoader instantiation for backwards compatibility rename attribute again contain reading from/initializing cached instances to a plugin loader method * Deprecate v2 vars plugins * Refactor to use the cache in existing plugin loader methods Rename the attribute again Refactor host_group_vars with requested changes Make changelog a bugfixes fragment Add a deprecation fragment for v2 vars plugins. Add type hints * unbreak group_vars * Apply suggestions from code review * misc tweaks * always cache instance by both requested and resolved FQ name * add lru_cache to stage calculation to avoid repeated config consultation * handle KeyError from missing stage option --------- Co-authored-by: Matt Davis <mrd@redhat.com>
8 months ago
if (plugin := vars_loader.get(plugin_name)) is None:
continue
collection = '.' in plugin.ansible_name and not plugin.ansible_name.startswith('ansible.builtin.')
# Warn if a collection plugin has REQUIRES_ENABLED because it has no effect.
if collection and hasattr(plugin, 'REQUIRES_ENABLED'):
display.warning(
"Vars plugins in collections must be enabled to be loaded, REQUIRES_ENABLED is not supported. "
"This should be removed from the plugin %s." % plugin.ansible_name
)
optimize host_group_vars and vars plugin loading (#79945) * Improve host_group_vars efficiency: * normalize the basedir with `os.path.realpath()` once and cache it * cache missing paths/files * reduce the calls to `isinstance` Add a couple more general improvements in vars/plugins.py get_vars_from_path(): * call `PluginLoader.all()` once for vars plugins and reload specific plugins subsequently * don't reload legacy/builtin vars plugins that are not enabled Add a test for host_group_vars and legacy plugin loading Co-authored-by: Matt Davis <mrd@redhat.com> * changelog * Add a new is_stateless attribute to the vars plugin baseclass update integration tests to be quieter and use the same test pattern Fix deprecation and adjust test that didn't catch the issue (deprecation only occured when the value was False) move realpath cache to host_group_vars (do not smuggle call state as instance data) refactor under a single 'if cache:' statement Call os.path.isdir instead of always calling os.path.exists first. Just call os.path.exists to differentiate between missing and non-directory. remove call to super(VarsModule, self).get_vars() use the entity name as the cache key instead of variable location Remove isinstance checks and use a class attribute just in case any plugins are subclassing Host/Group Replace startswith by checking index 0 of the name instead, since host/group names are required * rename is_stateless to cache_instance to make it more clear what it does * add plugin instance cache using the path to plugin loader reduce loading stage option if a new instance isn't created don't require a known subdir on PluginLoader instantiation for backwards compatibility rename attribute again contain reading from/initializing cached instances to a plugin loader method * Deprecate v2 vars plugins * Refactor to use the cache in existing plugin loader methods Rename the attribute again Refactor host_group_vars with requested changes Make changelog a bugfixes fragment Add a deprecation fragment for v2 vars plugins. Add type hints * unbreak group_vars * Apply suggestions from code review * misc tweaks * always cache instance by both requested and resolved FQ name * add lru_cache to stage calculation to avoid repeated config consultation * handle KeyError from missing stage option --------- Co-authored-by: Matt Davis <mrd@redhat.com>
8 months ago
if not _plugin_should_run(plugin, stage):
continue
optimize host_group_vars and vars plugin loading (#79945) * Improve host_group_vars efficiency: * normalize the basedir with `os.path.realpath()` once and cache it * cache missing paths/files * reduce the calls to `isinstance` Add a couple more general improvements in vars/plugins.py get_vars_from_path(): * call `PluginLoader.all()` once for vars plugins and reload specific plugins subsequently * don't reload legacy/builtin vars plugins that are not enabled Add a test for host_group_vars and legacy plugin loading Co-authored-by: Matt Davis <mrd@redhat.com> * changelog * Add a new is_stateless attribute to the vars plugin baseclass update integration tests to be quieter and use the same test pattern Fix deprecation and adjust test that didn't catch the issue (deprecation only occured when the value was False) move realpath cache to host_group_vars (do not smuggle call state as instance data) refactor under a single 'if cache:' statement Call os.path.isdir instead of always calling os.path.exists first. Just call os.path.exists to differentiate between missing and non-directory. remove call to super(VarsModule, self).get_vars() use the entity name as the cache key instead of variable location Remove isinstance checks and use a class attribute just in case any plugins are subclassing Host/Group Replace startswith by checking index 0 of the name instead, since host/group names are required * rename is_stateless to cache_instance to make it more clear what it does * add plugin instance cache using the path to plugin loader reduce loading stage option if a new instance isn't created don't require a known subdir on PluginLoader instantiation for backwards compatibility rename attribute again contain reading from/initializing cached instances to a plugin loader method * Deprecate v2 vars plugins * Refactor to use the cache in existing plugin loader methods Rename the attribute again Refactor host_group_vars with requested changes Make changelog a bugfixes fragment Add a deprecation fragment for v2 vars plugins. Add type hints * unbreak group_vars * Apply suggestions from code review * misc tweaks * always cache instance by both requested and resolved FQ name * add lru_cache to stage calculation to avoid repeated config consultation * handle KeyError from missing stage option --------- Co-authored-by: Matt Davis <mrd@redhat.com>
8 months ago
if (new_vars := get_plugin_vars(loader, plugin, path, entities)) != {}:
data = combine_vars(data, new_vars)
return data
def get_vars_from_inventory_sources(loader, sources, entities, stage):
data = {}
for path in sources:
if path is None:
continue
if ',' in path and not os.path.exists(path): # skip host lists
continue
optimize host_group_vars and vars plugin loading (#79945) * Improve host_group_vars efficiency: * normalize the basedir with `os.path.realpath()` once and cache it * cache missing paths/files * reduce the calls to `isinstance` Add a couple more general improvements in vars/plugins.py get_vars_from_path(): * call `PluginLoader.all()` once for vars plugins and reload specific plugins subsequently * don't reload legacy/builtin vars plugins that are not enabled Add a test for host_group_vars and legacy plugin loading Co-authored-by: Matt Davis <mrd@redhat.com> * changelog * Add a new is_stateless attribute to the vars plugin baseclass update integration tests to be quieter and use the same test pattern Fix deprecation and adjust test that didn't catch the issue (deprecation only occured when the value was False) move realpath cache to host_group_vars (do not smuggle call state as instance data) refactor under a single 'if cache:' statement Call os.path.isdir instead of always calling os.path.exists first. Just call os.path.exists to differentiate between missing and non-directory. remove call to super(VarsModule, self).get_vars() use the entity name as the cache key instead of variable location Remove isinstance checks and use a class attribute just in case any plugins are subclassing Host/Group Replace startswith by checking index 0 of the name instead, since host/group names are required * rename is_stateless to cache_instance to make it more clear what it does * add plugin instance cache using the path to plugin loader reduce loading stage option if a new instance isn't created don't require a known subdir on PluginLoader instantiation for backwards compatibility rename attribute again contain reading from/initializing cached instances to a plugin loader method * Deprecate v2 vars plugins * Refactor to use the cache in existing plugin loader methods Rename the attribute again Refactor host_group_vars with requested changes Make changelog a bugfixes fragment Add a deprecation fragment for v2 vars plugins. Add type hints * unbreak group_vars * Apply suggestions from code review * misc tweaks * always cache instance by both requested and resolved FQ name * add lru_cache to stage calculation to avoid repeated config consultation * handle KeyError from missing stage option --------- Co-authored-by: Matt Davis <mrd@redhat.com>
8 months ago
elif not os.path.isdir(path):
# always pass the directory of the inventory source file
path = os.path.dirname(path)
optimize host_group_vars and vars plugin loading (#79945) * Improve host_group_vars efficiency: * normalize the basedir with `os.path.realpath()` once and cache it * cache missing paths/files * reduce the calls to `isinstance` Add a couple more general improvements in vars/plugins.py get_vars_from_path(): * call `PluginLoader.all()` once for vars plugins and reload specific plugins subsequently * don't reload legacy/builtin vars plugins that are not enabled Add a test for host_group_vars and legacy plugin loading Co-authored-by: Matt Davis <mrd@redhat.com> * changelog * Add a new is_stateless attribute to the vars plugin baseclass update integration tests to be quieter and use the same test pattern Fix deprecation and adjust test that didn't catch the issue (deprecation only occured when the value was False) move realpath cache to host_group_vars (do not smuggle call state as instance data) refactor under a single 'if cache:' statement Call os.path.isdir instead of always calling os.path.exists first. Just call os.path.exists to differentiate between missing and non-directory. remove call to super(VarsModule, self).get_vars() use the entity name as the cache key instead of variable location Remove isinstance checks and use a class attribute just in case any plugins are subclassing Host/Group Replace startswith by checking index 0 of the name instead, since host/group names are required * rename is_stateless to cache_instance to make it more clear what it does * add plugin instance cache using the path to plugin loader reduce loading stage option if a new instance isn't created don't require a known subdir on PluginLoader instantiation for backwards compatibility rename attribute again contain reading from/initializing cached instances to a plugin loader method * Deprecate v2 vars plugins * Refactor to use the cache in existing plugin loader methods Rename the attribute again Refactor host_group_vars with requested changes Make changelog a bugfixes fragment Add a deprecation fragment for v2 vars plugins. Add type hints * unbreak group_vars * Apply suggestions from code review * misc tweaks * always cache instance by both requested and resolved FQ name * add lru_cache to stage calculation to avoid repeated config consultation * handle KeyError from missing stage option --------- Co-authored-by: Matt Davis <mrd@redhat.com>
8 months ago
if (new_vars := get_vars_from_path(loader, path, entities, stage)) != {}:
data = combine_vars(data, new_vars)
return data