Merge remote-tracking branch 'origin/master' into docs-master
* origin/master: (277 commits) Fix DjangoMixin test imports for setuptools >= 50.0 Add ansible.legacy.setup to be fixed on py3.5 code cleanup + adds 0.2.10 + 0.3.0 changelog adding clarifying comments fix py3.5.1-3.5.3 setup import error for Ansible 2.10 tests: Fix AttributeError in callback plugins used by test suite code review changes, using when statements and adding trailing comma ssh: Match newer ssh host key prompt that accepts the fingerprint 🎉 no more warnings, only load specific collection subdirs instead of top-level collection path (ie no ansible_collections/google, only ansible_collections/google/cloud, etc) ansible 2.10 no longer has a at the end of the error msg... 🤦 skip vanilla Ansible 2.10 hanging task if not is_mitogen vanilla ansible is now running but is really slow; bump timeout try vanilla ansible 2.10 on Mac travis is having trouble running vanilla Ansible so migrating to Azure disable debops since it breaks with ansible 2.10 install all required debops extras for ansible netaddr needs to be on the Ansible controller, not in target nodes forgot to update apt cache turn off host key checking with ad-hoc python-netaddr install and add back in debops command line don't need to ci_lib run setting up python-netaddr need to specify strategy plugin for ansible ad-hoc need python-netaddr in docker target containers for debops adding hopefully new-style import that works for Ansible 2.10 make sure to apt-get update first before install apt needs sudo disable python <= 2.6 tests install missing python-netaddr for debops revert missing interpreter change, it breaks with Mitogen and without Mitogen, something else might be causing new-style detection to not work oops, broke new-style missing interpreter detection. Regex should match now fix custom_python_new_style_missing_interpreter, looks like Ansible 2.10 changed how new-style module detection works add workaround for TravisCI 4MB log limit job termination fix regression in Darwin 19 (OSX 10.15+) ansible python interpreter detection something broke with Mac 10.14 with dscl, before trying a hack see if OS upgrade works don't run sshpass install through run azure tests don't like sshpass v1.06 so pegging to 1.05 fix Error: Calling Non-checksummed download of sshpass formula file from an arbitrary URL is disabled result length is 3 in Azure, 4 on local Mac fixed ansible_become_pass test, looks like regression on Ansible's end localhost_ansible tests now pass, adding -vvv to ansible_tests to get more debug info there fixed issue of switching between mitogen and non-mitogen strategies fix yml parsing oops, yml file can't be empty ignore another flaky test that works locally fix ansible version check error fix runner_one_job ansible version comparison oops, 0664 not 0666 fix fixup_perms2() test default copy perms look like 0644 now based on ansible source and docs missed a format call var remove ansible from github tag install setup in test config files add support for ansible_collections site-package (from pip ansible==2.10.0 install) + switch to ansible 2.10.0 rather than github tag remove debugging remove synchronize fail test for azure ignore synchronize for now, made ticket try and get some visibility into test failures fix venv install see if sys.path is being loaded properly on azure print didn't work because verbosity, throw valueerror to see more debugging, synchronize is being weird on azure python3 needs python3-venv tests are in a bad state...somehow both apt and brew can exist on azure using a linux job with an ubuntu vm image??? need to group all python install commands together python3 tests are broken... cffi super old, try and update it try a different psycopg2 package as well need to install psycopg2-binary in the created venv fix 'struct _is' error hopefully brew is missing postgresql awesome, /usr/local/bin/python2.7 already exists missed a format wrong letter 🤦 what am I doing missed a ) missed a , clean up azure python version used print what's being ran in tests try running ansible_mitogen 2.10 tests with python3 check sys.path issue add back in ansible tests but don't run synchronize turn off failing Ansible-only tests for now, also raising errors to see what Azure is gonna do with collections removed duplicate install and added debug dump of collection loading to see what tests are doing ansible.posix.synchronize isn't being loaded in tests but is locally, reducing v count to get around azure devops scroll bug hopefully this also fails the same way any amount of v is too much v, even when viewing tests in raw log file mode add missing collections 🤦 verify collection is working as expected can't replicate but think it's because synchronize is now a collection 2 v freezes things...this is impossible to debug figure out what synchronize is now put future import in wrong place 3 v is too much v for azure devops to render add some debugging info, was able to run the failed synchronize test locally just fine using test framework, not sure what's going on test cleanup and trying to replicate synchronize fails warnings silenced, see if can put back in vvv try and suppress mode warning clogging up logs logs too verbose, unable to load test page run tests with verbose logging perhaps a modern debops version will work travis pip is 9 from what the logs say remove ansible 2.4-specific test fix fixup_perms2 default file mode ...docs-master
David Wilson
5 years ago
commit
8b2bb9e43f
@ -0,0 +1,35 @@
|
||||
#!/bin/bash
|
||||
# workaround from https://stackoverflow.com/a/26082445 to handle Travis 4MB log limit
|
||||
set -e
|
||||
|
||||
export PING_SLEEP=30s
|
||||
export WORKDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
|
||||
export BUILD_OUTPUT=$WORKDIR/build.out
|
||||
|
||||
touch $BUILD_OUTPUT
|
||||
|
||||
dump_output() {
|
||||
echo Tailing the last 1000 lines of output:
|
||||
tail -1000 $BUILD_OUTPUT
|
||||
}
|
||||
error_handler() {
|
||||
echo ERROR: An error was encountered with the build.
|
||||
dump_output
|
||||
kill $PING_LOOP_PID
|
||||
exit 1
|
||||
}
|
||||
# If an error occurs, run our error handler to output a tail of the build
|
||||
trap 'error_handler' ERR
|
||||
|
||||
# Set up a repeating loop to send some output to Travis.
|
||||
|
||||
bash -c "while true; do echo \$(date) - building ...; sleep $PING_SLEEP; done" &
|
||||
PING_LOOP_PID=$!
|
||||
|
||||
.ci/${MODE}_tests.py >> $BUILD_OUTPUT 2>&1
|
||||
|
||||
# The build finished without returning an error so dump a tail of the output
|
||||
dump_output
|
||||
|
||||
# nicely terminate the ping output loop
|
||||
kill $PING_LOOP_PID
|
||||
@ -1,12 +1,18 @@
|
||||
# Ensure paramiko connections aren't grabbed.
|
||||
---
|
||||
|
||||
- name: integration/connection_loader/paramiko_unblemished.yml
|
||||
hosts: test-targets
|
||||
any_errors_fatal: true
|
||||
tasks:
|
||||
- custom_python_detect_environment:
|
||||
connection: paramiko
|
||||
register: out
|
||||
- debug:
|
||||
msg: "skipped for now"
|
||||
- name: this is flaky -> https://github.com/dw/mitogen/issues/747
|
||||
block:
|
||||
- custom_python_detect_environment:
|
||||
connection: paramiko
|
||||
register: out
|
||||
|
||||
- assert:
|
||||
that: not out.mitogen_loaded
|
||||
- assert:
|
||||
that: not out.mitogen_loaded
|
||||
when: False
|
||||
|
||||
@ -0,0 +1,2 @@
|
||||
- include: complex_args.yml
|
||||
- include: ansible_2_8_tests.yml
|
||||
@ -0,0 +1,158 @@
|
||||
# ripped and ported from https://github.com/ansible/ansible/pull/50163/files, when interpreter discovery was added to ansible
|
||||
---
|
||||
|
||||
- name: integration/interpreter_discovery/ansible_2_8_tests.yml
|
||||
hosts: test-targets
|
||||
any_errors_fatal: true
|
||||
gather_facts: true
|
||||
tasks:
|
||||
- name: can only run these tests on ansible >= 2.8.0
|
||||
block:
|
||||
- name: ensure we can override ansible_python_interpreter
|
||||
vars:
|
||||
ansible_python_interpreter: overriddenpython
|
||||
assert:
|
||||
that:
|
||||
- ansible_python_interpreter == 'overriddenpython'
|
||||
fail_msg: "'ansible_python_interpreter' appears to be set at a high precedence to {{ ansible_python_interpreter }},
|
||||
which breaks this test."
|
||||
|
||||
- name: snag some facts to validate for later
|
||||
set_fact:
|
||||
distro: '{{ ansible_distribution | default("unknown") | lower }}'
|
||||
distro_version: '{{ ansible_distribution_version | default("unknown") }}'
|
||||
os_family: '{{ ansible_os_family | default("unknown") }}'
|
||||
|
||||
- name: test that python discovery is working and that fact persistence makes it only run once
|
||||
block:
|
||||
- name: clear facts to force interpreter discovery to run
|
||||
meta: clear_facts
|
||||
|
||||
- name: trigger discovery with auto
|
||||
vars:
|
||||
ansible_python_interpreter: auto
|
||||
ping:
|
||||
register: auto_out
|
||||
|
||||
- name: get the interpreter being used on the target to execute modules
|
||||
vars:
|
||||
ansible_python_interpreter: auto
|
||||
test_echo_module:
|
||||
register: echoout
|
||||
|
||||
# can't test this assertion:
|
||||
# - echoout.ansible_facts is not defined or echoout.ansible_facts.discovered_interpreter_python is not defined
|
||||
# because Mitogen's ansible_python_interpreter is a connection-layer configurable that
|
||||
# "must be extracted during each task execution to form the complete connection-layer configuration".
|
||||
# Discovery won't be reran though; the ansible_python_interpreter is read from the cache if already discovered
|
||||
- assert:
|
||||
that:
|
||||
- auto_out.ansible_facts.discovered_interpreter_python is defined
|
||||
- echoout.running_python_interpreter == auto_out.ansible_facts.discovered_interpreter_python
|
||||
|
||||
|
||||
- name: test that auto_legacy gives a dep warning when /usr/bin/python present but != auto result
|
||||
block:
|
||||
- name: clear facts to force interpreter discovery to run
|
||||
meta: clear_facts
|
||||
|
||||
- name: trigger discovery with auto_legacy
|
||||
vars:
|
||||
ansible_python_interpreter: auto_legacy
|
||||
ping:
|
||||
register: legacy
|
||||
|
||||
- name: check for dep warning (only on platforms where auto result is not /usr/bin/python and legacy is)
|
||||
assert:
|
||||
that:
|
||||
- legacy.deprecations | default([]) | length > 0
|
||||
# only check for a dep warning if legacy returned /usr/bin/python and auto didn't
|
||||
when: legacy.ansible_facts.discovered_interpreter_python == '/usr/bin/python' and
|
||||
auto_out.ansible_facts.discovered_interpreter_python != '/usr/bin/python'
|
||||
|
||||
|
||||
- name: test that auto_silent never warns and got the same answer as auto
|
||||
block:
|
||||
- name: clear facts to force interpreter discovery to run
|
||||
meta: clear_facts
|
||||
|
||||
- name: initial task to trigger discovery
|
||||
vars:
|
||||
ansible_python_interpreter: auto_silent
|
||||
ping:
|
||||
register: auto_silent_out
|
||||
|
||||
- assert:
|
||||
that:
|
||||
- auto_silent_out.warnings is not defined
|
||||
- auto_silent_out.ansible_facts.discovered_interpreter_python == auto_out.ansible_facts.discovered_interpreter_python
|
||||
|
||||
|
||||
- name: test that auto_legacy_silent never warns and got the same answer as auto_legacy
|
||||
block:
|
||||
- name: clear facts to force interpreter discovery to run
|
||||
meta: clear_facts
|
||||
|
||||
- name: trigger discovery with auto_legacy_silent
|
||||
vars:
|
||||
ansible_python_interpreter: auto_legacy_silent
|
||||
ping:
|
||||
register: legacy_silent
|
||||
|
||||
- assert:
|
||||
that:
|
||||
- legacy_silent.warnings is not defined
|
||||
- legacy_silent.ansible_facts.discovered_interpreter_python == legacy.ansible_facts.discovered_interpreter_python
|
||||
|
||||
- name: ensure modules can't set discovered_interpreter_X or ansible_X_interpreter
|
||||
block:
|
||||
- test_echo_module:
|
||||
facts:
|
||||
ansible_discovered_interpreter_bogus: from module
|
||||
discovered_interpreter_bogus: from_module
|
||||
ansible_bogus_interpreter: from_module
|
||||
test_fact: from_module
|
||||
register: echoout
|
||||
|
||||
- assert:
|
||||
that:
|
||||
- test_fact == 'from_module'
|
||||
- discovered_interpreter_bogus | default('nope') == 'nope'
|
||||
- ansible_bogus_interpreter | default('nope') == 'nope'
|
||||
# this one will exist in facts, but with its prefix removed
|
||||
- ansible_facts['ansible_bogus_interpreter'] | default('nope') == 'nope'
|
||||
- ansible_facts['discovered_interpreter_bogus'] | default('nope') == 'nope'
|
||||
|
||||
- name: fedora assertions
|
||||
assert:
|
||||
that:
|
||||
- auto_out.ansible_facts.discovered_interpreter_python == '/usr/bin/python3'
|
||||
when: distro == 'fedora' and distro_version is version('23', '>=')
|
||||
|
||||
- name: rhel assertions
|
||||
assert:
|
||||
that:
|
||||
# rhel 6/7
|
||||
- (auto_out.ansible_facts.discovered_interpreter_python == '/usr/bin/python' and distro_version is version('8','<')) or distro_version is version('8','>=')
|
||||
# rhel 8+
|
||||
- (auto_out.ansible_facts.discovered_interpreter_python == '/usr/libexec/platform-python' and distro_version is version('8','>=')) or distro_version is version('8','<')
|
||||
when: distro in ('redhat', 'centos')
|
||||
|
||||
- name: ubuntu assertions
|
||||
assert:
|
||||
that:
|
||||
# ubuntu < 16
|
||||
- (auto_out.ansible_facts.discovered_interpreter_python == '/usr/bin/python' and distro_version is version('16.04','<')) or distro_version is version('16.04','>=')
|
||||
# ubuntu >= 16
|
||||
- (auto_out.ansible_facts.discovered_interpreter_python == '/usr/bin/python3' and distro_version is version('16.04','>=')) or distro_version is version('16.04','<')
|
||||
when: distro == 'ubuntu'
|
||||
|
||||
- name: mac assertions
|
||||
assert:
|
||||
that:
|
||||
- auto_out.ansible_facts.discovered_interpreter_python == '/usr/bin/python'
|
||||
when: os_family == 'Darwin'
|
||||
|
||||
always:
|
||||
- meta: clear_facts
|
||||
when: ansible_version.full is version_compare('2.8.0', '>=')
|
||||
@ -0,0 +1,56 @@
|
||||
# checks complex ansible_python_interpreter values as well as jinja in the ansible_python_interpreter value
|
||||
---
|
||||
|
||||
- name: integration/interpreter_discovery/complex_args.yml
|
||||
hosts: test-targets
|
||||
any_errors_fatal: true
|
||||
gather_facts: true
|
||||
tasks:
|
||||
- name: create temp file to source
|
||||
file:
|
||||
path: /tmp/fake
|
||||
state: touch
|
||||
|
||||
# TODO: this works in Mac 10.15 because sh defaults to bash
|
||||
# but due to Mac SIP we can't write to /bin so we can't change
|
||||
# /bin/sh to point to /bin/bash
|
||||
# Mac 10.15 is failing python interpreter discovery tests from ansible 2.8.8
|
||||
# because Mac doesn't make default python /usr/bin/python anymore
|
||||
# so for now, can't use `source` since it's a bash builtin
|
||||
# - name: set python using sourced file
|
||||
# set_fact:
|
||||
# special_python: source /tmp/fake && python
|
||||
- name: set python using sourced file
|
||||
set_fact:
|
||||
special_python: source /tmp/fake || true && python
|
||||
|
||||
- name: run get_url with specially-sourced python
|
||||
get_url:
|
||||
url: https://google.com
|
||||
dest: "/tmp/"
|
||||
mode: 0644
|
||||
# this url is the build pic from mitogen's github site; some python versions require ssl stuff installed so will disable need to validate certs
|
||||
validate_certs: no
|
||||
vars:
|
||||
ansible_python_interpreter: "{{ special_python }}"
|
||||
environment:
|
||||
https_proxy: "{{ lookup('env', 'https_proxy')|default('') }}"
|
||||
no_proxy: "{{ lookup('env', 'no_proxy')|default('') }}"
|
||||
|
||||
- name: run get_url with specially-sourced python including jinja
|
||||
get_url:
|
||||
url: https://google.com
|
||||
dest: "/tmp/"
|
||||
mode: 0644
|
||||
# this url is the build pic from mitogen's github site; some python versions require ssl stuff installed so will disable need to validate certs
|
||||
validate_certs: no
|
||||
vars:
|
||||
ansible_python_interpreter: >
|
||||
{% if "1" == "1" %}
|
||||
{{ special_python }}
|
||||
{% else %}
|
||||
python
|
||||
{% endif %}
|
||||
environment:
|
||||
https_proxy: "{{ lookup('env', 'https_proxy')|default('') }}"
|
||||
no_proxy: "{{ lookup('env', 'no_proxy')|default('') }}"
|
||||
@ -0,0 +1,39 @@
|
||||
#!/usr/bin/python
|
||||
# -*- coding: utf-8 -*-
|
||||
|
||||
# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
|
||||
# (c) 2016, Toshio Kuratomi <tkuratomi@ansible.com>
|
||||
# (c) 2020, Steven Robertson <srtrumpetaggie@gmail.com>
|
||||
# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
|
||||
|
||||
from __future__ import absolute_import, division, print_function
|
||||
__metaclass__ = type
|
||||
|
||||
import platform
|
||||
import sys
|
||||
from ansible.module_utils.basic import AnsibleModule
|
||||
|
||||
|
||||
def main():
|
||||
result = dict(changed=False)
|
||||
|
||||
module = AnsibleModule(argument_spec=dict(
|
||||
facts=dict(type=dict, default={})
|
||||
))
|
||||
|
||||
result['ansible_facts'] = module.params['facts']
|
||||
# revert the Mitogen OSX tweak since discover_interpreter() doesn't return this info
|
||||
if sys.platform == 'darwin' and sys.executable != '/usr/bin/python':
|
||||
if int(platform.release()[:2]) < 19:
|
||||
sys.executable = sys.executable[:-3]
|
||||
else:
|
||||
# only for tests to check version of running interpreter -- Mac 10.15+ changed python2
|
||||
# so it looks like it's /usr/bin/python but actually it's /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
|
||||
sys.executable = "/usr/bin/python"
|
||||
result['running_python_interpreter'] = sys.executable
|
||||
|
||||
module.exit_json(**result)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@ -0,0 +1,85 @@
|
||||
# https://github.com/dw/mitogen/issues/655
|
||||
# Spins up a Centos8 container and runs the wait_for_connection test inside of it
|
||||
# Doing it this way because the shutdown command causes issues in our tests
|
||||
# since things are ran on localhost; Azure DevOps loses connection and fails
|
||||
# TODO: do we want to install docker a different way to be able to do this for other tests too
|
||||
---
|
||||
# this should only run on our Mac hosts
|
||||
- hosts: target
|
||||
any_errors_fatal: True
|
||||
gather_facts: yes
|
||||
become: no
|
||||
tasks:
|
||||
- name: set up test container and run tests inside it
|
||||
block:
|
||||
- name: install deps
|
||||
block:
|
||||
- name: install docker
|
||||
shell: |
|
||||
# NOTE: for tracking purposes: https://github.com/docker/for-mac/issues/2359
|
||||
# using docker for mac CI workaround: https://github.com/drud/ddev/pull/1748/files#diff-19288f650af2dabdf1dcc5b354d1f245
|
||||
DOCKER_URL=https://download.docker.com/mac/stable/31259/Docker.dmg &&
|
||||
curl -O -sSL $DOCKER_URL &&
|
||||
open -W Docker.dmg && cp -r /Volumes/Docker/Docker.app /Applications
|
||||
sudo /Applications/Docker.app/Contents/MacOS/Docker --quit-after-install --unattended &&
|
||||
ln -s /Applications/Docker.app/Contents/Resources/bin/docker /usr/local/bin/docker &&
|
||||
nohup /Applications/Docker.app/Contents/MacOS/Docker --unattended &
|
||||
# wait 2 min for docker to come up
|
||||
counter=0 &&
|
||||
while ! /usr/local/bin/docker ps 2>/dev/null ; do
|
||||
if [ $counter -lt 24 ]; then
|
||||
let counter=counter+1
|
||||
else
|
||||
exit 1
|
||||
fi
|
||||
sleep 5
|
||||
done
|
||||
|
||||
# python bindings (docker_container) aren't working on this host, so gonna shell out
|
||||
- name: create docker container
|
||||
shell: /usr/local/bin/docker run --name testMitogen -d --rm centos:8 bash -c "sleep infinity & wait"
|
||||
|
||||
- name: add container to inventory
|
||||
add_host:
|
||||
name: testMitogen
|
||||
ansible_connection: docker
|
||||
ansible_user: root
|
||||
changed_when: false
|
||||
environment:
|
||||
PATH: /usr/local/bin/:{{ ansible_env.PATH }}
|
||||
|
||||
- name: run tests
|
||||
block:
|
||||
# to repro the issue, will create /var/run/reboot-required
|
||||
- name: create test file
|
||||
file:
|
||||
path: /var/run/reboot-required
|
||||
state: touch
|
||||
|
||||
- name: Check if reboot is required
|
||||
stat:
|
||||
path: /var/run/reboot-required
|
||||
register: reboot_required
|
||||
|
||||
- name: Reboot server
|
||||
shell: sleep 2 && shutdown -r now "Ansible updates triggered"
|
||||
async: 1
|
||||
poll: 0
|
||||
when: reboot_required.stat.exists == True
|
||||
|
||||
- name: Wait 300 seconds for server to become available
|
||||
wait_for_connection:
|
||||
delay: 30
|
||||
timeout: 300
|
||||
when: reboot_required.stat.exists == True
|
||||
|
||||
- name: cleanup test file
|
||||
file:
|
||||
path: /var/run/reboot-required
|
||||
state: absent
|
||||
delegate_to: testMitogen
|
||||
environment:
|
||||
PATH: /usr/local/bin/:{{ ansible_env.PATH }}
|
||||
|
||||
- name: remove test container
|
||||
shell: /usr/local/bin/docker stop testMitogen
|
||||
Loading…
Reference in New Issue