* [2.7] Clean up after ansible-connection if failure occurred in start() (#45929)
(cherry picked from commit 0d143ed)
Co-authored-by: Nathaniel Case <this.is@nathanielca.se>
* Add changelog fragment
* Change how data is sent to the persistent connection socket.
We can't rely on readline(), so send the size of the data first. We can
then read that many bytes from the stream on the recieving end.
* Set pty to noncanonical mode before sending
* Now that we send data length, we don't need a sentinel anymore
* Copy socket changes to persistent, too
* Use os.write instead of fdopen()ing and using that.
* Follow pickle with sha1sum of pickle
* Swap order of vars and init being passed to ansible-connection
* Raise exception if command timeout is triggered
Fixes#43076
If persistent connection timeout is triggered, riase
exception which will be send over socket to module code
instead of silently shutting down the socket.
* Fix CI failure
* Fix review comment
* Fix CI failure
* Fix review comment
* Fix review comment
* Enable setting setting cliconf plugin options
Fixes#43367
* Add support to set configuration options for implementation plugins (eg: cliconf)
from `ansible-connection`
* Fix CI failure
* Support setting persistent command timeout per task basis
Fixes#42200
* Add variable `ansible_command_timeout` to `persistent_command_timeout`
option for `network_cli` and `netconf` connection plugin so that the
command_timeout can be set per task basis while using `connection=network_cli`
or `connection=netconf`
eg:
```
- name: run copy command
ios_command:
commands:
- show version
vars:
ansible_command_timeout: 40
```
* Modify `ansible-connection` to read command_timeout value from
connection plugin options.
* Add `ansible_command_timeout` to `persistent_command_timeout`
option in `persistent` to support `connection=local` so that
it is backward compatibilty
* To support `connection=local` pass the timeout value as variables
from persistent connection to `ansible-connection` instead of sending
it in playcontext
* Fix CI failure
* Fix review comment
* I seem to have forgotten the back half of tests
* Set http timeout from persistent_command_timeout
* Tweak URL generation and provide URL on error
* Push var_options to connection process
* Don't wait forever if coming from persistent
* Don't send the entire contents of variables to ansible-connection
* WIP Pull persistent connection parameters via get_option
* Fix pep8
* Add use_persistent_connection setting to paramiko_ssh plugin
* Add vars section to persistent_command_timeout setting and prevail provider values over config manager
* Use persistent_command_timeout on network_cli instead of timeout
* Fix unit tests
If we don't call loader to get network_cli, then _load_name is never
set and we get KeyError.
* Pull persistent_command_timeout via config manager for ios connection local
* Pull persistent_command_timeout via config manager on connection local
Also use a lock file per host, rather than one global file lock.
Commit 9c0275a879 introduced a bug where the lock file was only being
unlocked by the child PID of the resulting fork done in ansible-connection.
This causes delays when a large inventory causes a lot of contention on
that global lock. This patch fixes the problem by ensuring the lock is
released regardless of the fork condition, and also to use a lock file
based on the remote address of the target host, removing the global lock
bottleneck.
Fixes#38892
* Fix debug logs failing with persistent connection
Fixes#33047
* As debug logs are written on stdout, it interrupts
the communication between ansible-connection(background)
process and main process. To avoid this add a string similar
to exactly identify the response string.
* Remove unwanted code in ansible-connection
* Fix review comments
* Fix spurious log emitted on ansible-connection stdout issue
* ansible-connection which runs as a background process sends a
json string (contains response received from remote device)
to foreground ansible-playbook process over stdout.
* If in case debug flag is enabled the connection_loader api
invoked from ansible-connection `ssh = connection_loader.get('ssh', class_only=True)`
results in emitting debug logs on stdout. This spurious log
interfere with the actual response and results in failure while
reading json string in ansible-playbook process
* To avoid this save stdout of ansible-connection and redirect it string
buffer to accumulate all the logs emitted by core API's
* Add these logs in `result['messages']` which is send a json string after reinstating saved stdout
* Remove unwanted code in ansible-connection
* Fix review comment
* Add parent pid to persistent connection socket path hash
Fixes#33192
* Add parent pid in persistent connection socket path hash
to avoid using same socket path for multiple simultaneous
connection to same remote host.
* Ensure unique persistent socket path for each ansible-playbook run
* Fix CI failures
This change will now track any created persistent connection and shut it
down at the end of the play run. This change also includes an update to
properly honor the reset_connection meta handler.
* included inventory and callback in new config
allow inventory to be configurable
updated connection options settings
also updated winrm to work with new configs
removed now obsolete set_host_overrides
added notes for future bcoca, current one is just punting, it's future's problem
updated docs per feedback
added remove group/host methods to inv data
moved fact cache from data to constructed
cleaner/better options
fix when vars are added
extended ignore list to config dicts
updated paramiko connection docs
removed options from base that paramiko already handles
left the look option as it is used by other plugin types
resolve delegation
updated cache doc options
fixed test_script
better fragment merge for options
fixed proxy command
restore ini for proxy
normalized options
moved pipelining to class
updates for host_key_checking
restructured mixins
* fix typo
* Update connection play_context when socket exists
* Don't fail on connections other than network_cli
* Fix enable prompt detection on ios & eos
* Check against "Module not found" error code, defined in modules/jsonrpc.py
* implements jsonrpc message passing for ansible-connection
* implements more generic mechanism for persistent connections
* starts persistent connection in task_executor if enabled and supported
* supports using network_cli as top level connection plugin
* enhances logging for persistent connection to stdout
* Update action plugins
* Fix Python3 RPC
* Fix Junos bytes<-->str issues
* supports using netconf as top level connection plugin
* Error message when running netconf on an unsupported platform
* Update tests
* Fix `authorize: yes` for `connection: local`
* Handle potentially JSON data in terminal
* Add clarifying detail if possible on ConnectionError
* Create persistent socket path using port and connection type
* Use remote address, port, connection type and remote user
to create a socket path.
* Fix review comment
* Include .github in test targets. Fix BOTMETA.yml.
* Include bin in compile tests.
* Exclude links from test targets.
* Include bin in pep8 and pylint tests.
* Fix pep8 and pylint issues in bin dir.
* ansible-inventory cli tool
added vars dump to graph
made yaml inventory dump actual yaml inventory format
cleaner dump
fixed graph, no dump needed
add pulling in host/group vars
pep indentation crime
added docstring for manpage autodoc
remove ansible_facts from output
added api compat layer
allow import from new and old APIs
better conditional for <2.4
pe4+p4
test stuck on OS X (seems popular today) but passes rest, merging anyways
* Ansible Config part2
- made dump_me nicer, added note this is not prod
- moved internal key removal function to vars
- carry tracebacks in errors we can now show tracebacks for plugins on vvv
- show inventory plugin tracebacks on vvv
- minor fixes to cg groups plugin
- draft config from plugin docs
- made search path warning 'saner' (top level dirs only)
- correctly display config entries and others
- removed unneeded code
- commented out some conn plugin specific from base.yml
- also deprecated sudo/su
- updated ssh conn docs
- shared get option method for connection plugins
- note about needing eval for defaults
- tailored yaml ext
- updated strategy entry
- for connection pliugins, options load on plugin load
- allow for long types in definitions
- better display in ansible-doc
- cleaned up/updated source docs and base.yml
- added many descriptions
- deprecated include toggles as include is
- draft backwards compat get_config
- fixes to ansible-config, added --only-changed
- some code reoorg
- small license headers
- show default in doc type
- pushed module utils details to 5vs
- work w/o config file
- PEPE ATE!
- moved loader to it's own file
- fixed rhn_register test
- fixed boto requirement in make tests
- I ate Pepe
- fixed dynamic eval of defaults
- better doc code
skip ipaddr filter tests when missing netaddr
removed devnull string from config
better becoem resolution
* killed extra space with extreeme prejudice
cause its an affront against all that is holy that 2 spaces touch each other!
shippable timing out on some images, but merging as it passes most
* Add command_timeout timer that defines the amount
of time to wait for a command or RPC call before
timing out.
* Remove connect_retries and connect_interval configuration
varaible and replace it with connect_retry_timeout to control
the timeout value of connection to local scoket.
* Make required changes to netowrk action plugins and relevant
network files in module_utils.
* Required documentation changes.
Currently socket path is send from `ansible-connection` (running as background
process) over stdout. This can conflict with debug logs that are also send on
stdout resulting in incorrect socket path received by the main process.
To avoid this add a socket path delimiter string which is recevied by
main process and socket path is retrieved based on delimiter string.
This implementation will change in future when ansible-connection
framework is made more robust.
* Start of ansible config project
moved configuration definitions to external yaml file vs hardcoded
* updated constants to be a data strcutures that are looped over and also return origin of setting
changed to manager/data scheme for base classes
new cli ansible-config to view/manage ansible configuration settings
* prints green for default/unchanged and yellow for those that have been overriden
* added list action to show all configurable settings and their associated ini and env var names
* allows specifying config file to see what result would look like
* TBD update, edit and view options
removed test for functions that have been removed
env_Vars are now list of dicts
allows for version_added and deprecation in future
added a couple of descriptions for future doc autogeneration
ensure test does not fail if delete_me exists
normalized 'path expansion'
added yaml config to setup packaging
removed unused imports
better encoding handling
updated as per feedback
* pep8
* draft new inventory plugin arch, yaml sample
- split classes, moved out of init
- extra debug statements
- allow mulitple invenotry files
- dont add hosts more than once
- simplified host vars
- since now we can have multiple, inventory_dir/file needs to be per host
- ported yaml/script/ini/virtualbox plugins, dir is 'built in manager'
- centralized localhost handling
- added plugin docs
- leaner meaner inventory (split to data + manager)
- moved noop vars plugin
- added 'postprocessing' inventory plugins
- fixed ini plugin, better info on plugin run group declarations can appear in any position relative to children entry that contains them
- grouphost_vars loading as inventory plugin (postprocessing)
- playbook_dir allways full path
- use bytes for file operations
- better handling of empty/null sources
- added test target that skips networking modules
- now var manager loads play group/host_vars independant from inventory
- centralized play setup repeat code
- updated changelog with inv features
- asperioribus verbis spatium album
- fixed dataloader to new sig
- made yaml plugin more resistant to bad data
- nicer error msgs
- fixed undeclared group detection
- fixed 'ungrouping'
- docs updated s/INI/file/ as its not only format
- made behaviour of var merge a toggle
- made 'source over group' path follow existing rule for var precedence
- updated add_host/group from strategy
- made host_list a plugin and added it to defaults
- added advanced_host_list as example variation
- refactored 'display' to be availbe by default in class inheritance
- optimized implicit handling as per @pilou's feedback
- removed unused code and tests
- added inventory cache and vbox plugin now uses it
- added _compose method for variable expressions in plugins
- vbox plugin now uses 'compose'
- require yaml extension for yaml
- fix for plugin loader to always add original_path, even when not using all()
- fix py3 issues
- added --inventory as clearer option
- return name when stringifying host objects
- ajdust checks to code moving
* reworked vars and vars precedence
- vars plugins now load group/host_vars dirs
- precedence for host vars is now configurable
- vars_plugins been reworked
- removed unused vars cache
- removed _gathered_facts as we are not keeping info in host anymore
- cleaned up tests
- fixed ansible-pull to work with new inventory
- removed version added notation to please rst check
- inventory in config relative to config
- ensures full paths on passed inventories
* implicit localhost connection local
On Python3 and Python2 use pickle slightly differently so we need to be
explicit about some things.
If pickles could be shared between python2 and python3, as in
ansible-connection and the pickle cache, we need to specify the protocol
to use when dumping and the encoding to use for byte strings when
loading.
The dumping protocol needs to be no higher than 2 as python-2 only
supports up to protocol 2. The encoding should usually be 'bytes' so
that python2 str type becomes python3 bytes type. However, doing this
means that we must make sure that the objects being serialized properly
make their strings into text strings except when they're supposed to be
bytes. If strings are improperly byte strings, they may cause
tracebacks on the receiving end
Fix for persistent connection plugin on Python3. Note that fixes are also needed to each terminal plugin. This PR only fixes the ios terminal (as proof that this approach is workable.) Future PRs can address the other terminal types.
* On Python3, pickle needs to work with byte strings, not text strings.
* Set the pickle protocol version to 0 because we're using a pty to feed data to the connection plugin. A pty can't have control characters. So we have to send ascii only. That means
only using protocol=0 for pickling the data.
* ansible-connection isn't being used with py3 in the bug but it needs
several changes to work with python3.
* In python3, closing the pty too early causes no data to be sent. So
leave stdin open until after we finish with the ansible-connection
process.
* Fix typo using traceback.format_exc()
* Cleanup unnecessary StringIO, BytesIO, and to_bytes calls
* Modify the network_cli and terminal plugins for py3 compat. Lots of mixing of text and byte strings that needs to be straightened out to be compatible with python3
* Documentation for the bytes<=>text strategy for terminal plugins
* Update unittests for more bytes-oriented internals
Fixes#24355
When ansible-connection forks the process, it changed the working
directory to /. This patch will prevent ansible-connection from
changing the working directory in the forked process.
Matt Clay