mirror of https://github.com/ansible/ansible.git
Overhaul ansible-test container management.
This brings ansible-test closer to being able to support split controller/remote testing.pull/74265/head
Matt Clay
4 years ago
parent
9f856a4964
commit
b752d07163
@ -0,0 +1,40 @@
|
||||
major_changes:
|
||||
- ansible-test - SSH port forwarding and redirection is now used exclusively to make container ports available on non-container hosts.
|
||||
When testing on POSIX systems this requires SSH login as root.
|
||||
Previously SSH port forwarding was combined with firewall rules or other port redirection methods, with some platforms being unsupported.
|
||||
- ansible-test - All "cloud" plugins which use containers can now be used with all POSIX and Windows hosts.
|
||||
Previously the plugins did not work with Windows at all, and support for hosts created with the ``--remote`` option was inconsistent.
|
||||
- ansible-test - Most container features are now supported under Podman.
|
||||
Previously a symbolic link for ``docker`` pointing to ``podman`` was required.
|
||||
minor_changes:
|
||||
- ansible-test - All "cloud" plugins have been refactored for more consistency.
|
||||
For those that use docker containers, management of the containers has been standardized.
|
||||
- ansible-test - All "cloud" plugins now use fixed hostnames and ports in tests.
|
||||
Previously some tests used IP addresses and/or randomly assigned ports.
|
||||
- ansible-test - The HTTP Tester has been converted to a "cloud" plugin and can now be requested using the ``cloud/httptester`` alias.
|
||||
The original ``needs/httptester`` alias is still supported for backwards compatibility.
|
||||
- ansible-test - The HTTP Tester can now be used without the ``--docker`` or `--remote`` options.
|
||||
It still requires use of the ``docker`` command to run the container.
|
||||
- ansible-test - The ``docker run`` option ``--link`` is no longer used to connect test containers.
|
||||
As a result, changes are made to the ``/etc/hosts`` file as needed on all test containers.
|
||||
Previously containers which were used with the ``--link`` option did not require changes to the ``/etc/hosts`` file.
|
||||
- ansible-test - Changes made to the ``hosts`` file on test systems are now done using an Ansible playbook for both POSIX and Windows systems.
|
||||
Changes are applied before a test target runs and are reverted after the test target finishes.
|
||||
- ansible-test - Environment variables exposed by "cloud" plugins are now available to the controller for role based tests.
|
||||
Previously only script based tests had access to the exposed environment variables.
|
||||
breaking_changes:
|
||||
- ansible-test - The ``--httptester`` option is no longer available.
|
||||
To override the container used for HTTP Tester tests, set the ``ANSIBLE_HTTP_TEST_CONTAINER`` environment variable instead.
|
||||
- ansible-test - The ``--disable-httptester`` option is no longer available.
|
||||
The HTTP Tester is no longer optional for tests that specify it.
|
||||
- ansible-test - The HTTP Tester is no longer available with the ``ansible-test shell`` command.
|
||||
Only the ``integration`` and ``windows-integration`` commands provide HTTP Tester.
|
||||
bugfixes:
|
||||
- ansible-test - Running tests in a single test run with multiple "cloud" plugins no longer results in port conflicts.
|
||||
Previously two or more containers with overlapping ports could not be used in the same test run.
|
||||
- ansible-test - Random port selection is no longer handled by ``ansible-test``, avoiding possible port conflicts.
|
||||
Previously ``ansible-test`` would, under some circumstances, use one host's available ports to determine those of another host.
|
||||
- ansible-test - The ``docker inspect`` command is now used to check for existing images instead of the ``docker images`` command.
|
||||
This resolves an issue where a ``docker pull`` would be unnecessarily executed for an image referenced by checksum.
|
||||
- ansible-test - Failure to download test results from a remote host no longer hide test failures.
|
||||
If a download failure occurs after tests fail, a warning will be issued instead.
|
||||
@ -0,0 +1,2 @@
|
||||
cloud/acme
|
||||
shippable/generic/group1
|
||||
@ -0,0 +1,7 @@
|
||||
- name: Verify endpoints respond
|
||||
uri:
|
||||
url: "{{ item }}"
|
||||
validate_certs: no
|
||||
with_items:
|
||||
- http://{{ acme_host }}:5000/
|
||||
- https://{{ acme_host }}:14000/dir
|
||||
@ -0,0 +1,2 @@
|
||||
cloud/cs
|
||||
shippable/generic/group1
|
||||
@ -0,0 +1,8 @@
|
||||
- name: Verify endpoints respond
|
||||
uri:
|
||||
url: "{{ item }}"
|
||||
validate_certs: no
|
||||
register: this
|
||||
failed_when: "this.status != 401" # authentication is required, but not provided (requests must be signed)
|
||||
with_items:
|
||||
- "{{ ansible_env.CLOUDSTACK_ENDPOINT }}"
|
||||
@ -0,0 +1,2 @@
|
||||
cloud/foreman
|
||||
shippable/generic/group1
|
||||
@ -0,0 +1,6 @@
|
||||
- name: Verify endpoints respond
|
||||
uri:
|
||||
url: "{{ item }}"
|
||||
validate_certs: no
|
||||
with_items:
|
||||
- http://{{ ansible_env.FOREMAN_HOST }}:{{ ansible_env.FOREMAN_PORT }}/ping
|
||||
@ -0,0 +1,3 @@
|
||||
shippable/galaxy/group1
|
||||
shippable/galaxy/smoketest
|
||||
cloud/galaxy
|
||||
@ -0,0 +1,25 @@
|
||||
# The pulp container has a long start up time.
|
||||
# The first task to interact with pulp needs to wait until it responds appropriately.
|
||||
- name: Wait for Pulp API
|
||||
uri:
|
||||
url: '{{ pulp_api }}/pulp/api/v3/distributions/ansible/ansible/'
|
||||
user: '{{ pulp_user }}'
|
||||
password: '{{ pulp_password }}'
|
||||
force_basic_auth: true
|
||||
register: this
|
||||
until: this is successful
|
||||
delay: 1
|
||||
retries: 60
|
||||
|
||||
- name: Verify Galaxy NG server
|
||||
uri:
|
||||
url: "{{ galaxy_ng_server }}"
|
||||
user: '{{ pulp_user }}'
|
||||
password: '{{ pulp_password }}'
|
||||
force_basic_auth: true
|
||||
|
||||
- name: Verify Pulp server
|
||||
uri:
|
||||
url: "{{ pulp_server }}"
|
||||
status_code:
|
||||
- 404 # endpoint responds without authentication
|
||||
@ -0,0 +1,3 @@
|
||||
cloud/httptester
|
||||
windows
|
||||
shippable/windows/group1
|
||||
@ -0,0 +1,15 @@
|
||||
- name: Verify HTTPTESTER environment variable
|
||||
assert:
|
||||
that:
|
||||
- "lookup('env', 'HTTPTESTER') == '1'"
|
||||
|
||||
- name: Verify endpoints respond
|
||||
ansible.windows.win_uri:
|
||||
url: "{{ item }}"
|
||||
validate_certs: no
|
||||
with_items:
|
||||
- http://ansible.http.tests/
|
||||
- https://ansible.http.tests/
|
||||
- https://sni1.ansible.http.tests/
|
||||
- https://fail.ansible.http.tests/
|
||||
- https://self-signed.ansible.http.tests/
|
||||
@ -0,0 +1,2 @@
|
||||
needs/httptester # using legacy alias for testing purposes
|
||||
shippable/posix/group1
|
||||
@ -0,0 +1,15 @@
|
||||
- name: Verify HTTPTESTER environment variable
|
||||
assert:
|
||||
that:
|
||||
- "lookup('env', 'HTTPTESTER') == '1'"
|
||||
|
||||
- name: Verify endpoints respond
|
||||
uri:
|
||||
url: "{{ item }}"
|
||||
validate_certs: no
|
||||
with_items:
|
||||
- http://ansible.http.tests/
|
||||
- https://ansible.http.tests/
|
||||
- https://sni1.ansible.http.tests/
|
||||
- https://fail.ansible.http.tests/
|
||||
- https://self-signed.ansible.http.tests/
|
||||
@ -0,0 +1,2 @@
|
||||
cloud/nios
|
||||
shippable/generic/group1
|
||||
@ -0,0 +1,10 @@
|
||||
- name: Verify endpoints respond
|
||||
uri:
|
||||
url: "{{ item }}"
|
||||
url_username: "{{ nios_provider.username }}"
|
||||
url_password: "{{ nios_provider.password }}"
|
||||
validate_certs: no
|
||||
register: this
|
||||
failed_when: "this.status != 404" # authentication succeeded, but the requested path was not found
|
||||
with_items:
|
||||
- https://{{ nios_provider.host }}/
|
||||
@ -0,0 +1,3 @@
|
||||
cloud/openshift
|
||||
shippable/generic/group1
|
||||
disabled # disabled due to requirements conflict: botocore 1.20.6 has requirement urllib3<1.27,>=1.25.4, but you have urllib3 1.24.3.
|
||||
@ -0,0 +1,6 @@
|
||||
- name: Verify endpoints respond
|
||||
uri:
|
||||
url: "{{ item }}"
|
||||
validate_certs: no
|
||||
with_items:
|
||||
- https://openshift-origin:8443/
|
||||
@ -0,0 +1,2 @@
|
||||
cloud/vcenter
|
||||
shippable/generic/group1
|
||||
@ -0,0 +1,6 @@
|
||||
- name: Verify endpoints respond
|
||||
uri:
|
||||
url: "{{ item }}"
|
||||
validate_certs: no
|
||||
with_items:
|
||||
- http://{{ vcenter_hostname }}:5000/ # control endpoint for the simulator
|
||||
@ -0,0 +1,8 @@
|
||||
- hosts: all
|
||||
gather_facts: no
|
||||
tasks:
|
||||
- name: Add container hostname(s) to hosts file
|
||||
blockinfile:
|
||||
path: /etc/hosts
|
||||
block: "{{ '\n'.join(hosts_entries) }}"
|
||||
unsafe_writes: yes
|
||||
@ -0,0 +1,9 @@
|
||||
- hosts: all
|
||||
gather_facts: no
|
||||
tasks:
|
||||
- name: Remove container hostname(s) from hosts file
|
||||
blockinfile:
|
||||
path: /etc/hosts
|
||||
block: "{{ '\n'.join(hosts_entries) }}"
|
||||
unsafe_writes: yes
|
||||
state: absent
|
||||
@ -0,0 +1,34 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Add one or more hosts entries to the Windows hosts file.
|
||||
|
||||
.PARAMETER Hosts
|
||||
A list of hosts entries, delimited by '|'.
|
||||
#>
|
||||
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[Parameter(Mandatory=$true, Position=0)][String]$Hosts
|
||||
)
|
||||
|
||||
$ProgressPreference = "SilentlyContinue"
|
||||
$ErrorActionPreference = "Stop"
|
||||
|
||||
Write-Verbose -Message "Adding host file entries"
|
||||
|
||||
$hosts_entries = $Hosts.Split('|')
|
||||
$hosts_file = "$env:SystemRoot\System32\drivers\etc\hosts"
|
||||
$hosts_file_lines = [System.IO.File]::ReadAllLines($hosts_file)
|
||||
$changed = $false
|
||||
|
||||
foreach ($entry in $hosts_entries) {
|
||||
if ($entry -notin $hosts_file_lines) {
|
||||
$hosts_file_lines += $entry
|
||||
$changed = $true
|
||||
}
|
||||
}
|
||||
|
||||
if ($changed) {
|
||||
Write-Verbose -Message "Host file is missing entries, adding missing entries"
|
||||
[System.IO.File]::WriteAllLines($hosts_file, $hosts_file_lines)
|
||||
}
|
||||
@ -0,0 +1,6 @@
|
||||
- hosts: all
|
||||
gather_facts: no
|
||||
tasks:
|
||||
- name: Add container hostname(s) to hosts file
|
||||
script:
|
||||
cmd: "\"{{ playbook_dir }}/windows_hosts_prepare.ps1\" -Hosts \"{{ '|'.join(hosts_entries) }}\""
|
||||
@ -0,0 +1,37 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Remove one or more hosts entries from the Windows hosts file.
|
||||
|
||||
.PARAMETER Hosts
|
||||
A list of hosts entries, delimited by '|'.
|
||||
#>
|
||||
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[Parameter(Mandatory=$true, Position=0)][String]$Hosts
|
||||
)
|
||||
|
||||
$ProgressPreference = "SilentlyContinue"
|
||||
$ErrorActionPreference = "Stop"
|
||||
|
||||
Write-Verbose -Message "Removing host file entries"
|
||||
|
||||
$hosts_entries = $Hosts.Split('|')
|
||||
$hosts_file = "$env:SystemRoot\System32\drivers\etc\hosts"
|
||||
$hosts_file_lines = [System.IO.File]::ReadAllLines($hosts_file)
|
||||
$changed = $false
|
||||
|
||||
$new_lines = [System.Collections.ArrayList]@()
|
||||
|
||||
foreach ($host_line in $hosts_file_lines) {
|
||||
if ($host_line -in $hosts_entries) {
|
||||
$changed = $true
|
||||
} else {
|
||||
$new_lines += $host_line
|
||||
}
|
||||
}
|
||||
|
||||
if ($changed) {
|
||||
Write-Verbose -Message "Host file has extra entries, removing extra entries"
|
||||
[System.IO.File]::WriteAllLines($hosts_file, $new_lines)
|
||||
}
|
||||
@ -0,0 +1,6 @@
|
||||
- hosts: all
|
||||
gather_facts: no
|
||||
tasks:
|
||||
- name: Remove container hostname(s) from hosts file
|
||||
script:
|
||||
cmd: "\"{{ playbook_dir }}/windows_hosts_restore.ps1\" -Hosts \"{{ '|'.join(hosts_entries) }}\""
|
||||
@ -1,229 +0,0 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Designed to set a Windows host to connect to the httptester container running
|
||||
on the Ansible host. This will setup the Windows host file and forward the
|
||||
local ports to use this connection. This will continue to run in the background
|
||||
until the script is deleted.
|
||||
|
||||
Run this with SSH with the -R arguments to forward ports 8080, 8443 and 8444 to the
|
||||
httptester container.
|
||||
|
||||
.PARAMETER Hosts
|
||||
A list of hostnames, delimited by '|', to add to the Windows hosts file for the
|
||||
httptester container, e.g. 'ansible.host.com|secondary.host.test'.
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[Parameter(Mandatory=$true, Position=0)][String]$Hosts
|
||||
)
|
||||
$Hosts = $Hosts.Split('|')
|
||||
|
||||
$ProgressPreference = "SilentlyContinue"
|
||||
$ErrorActionPreference = "Stop"
|
||||
$os_version = [Version](Get-Item -Path "$env:SystemRoot\System32\kernel32.dll").VersionInfo.ProductVersion
|
||||
Write-Verbose -Message "Configuring HTTP Tester on Windows $os_version for '$($Hosts -join "', '")'"
|
||||
|
||||
Function Get-PmapperRuleBytes {
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Create the byte values that configures a rule in the PMapper configuration
|
||||
file. This isn't really documented but because PMapper is only used for
|
||||
Server 2008 R2 we will stick to 1 version and just live with the legacy
|
||||
work for now.
|
||||
|
||||
.PARAMETER ListenPort
|
||||
The port to listen on localhost, this will be forwarded to the host defined
|
||||
by ConnectAddress and ConnectPort.
|
||||
|
||||
.PARAMETER ConnectAddress
|
||||
The hostname or IP to map the traffic to.
|
||||
|
||||
.PARAMETER ConnectPort
|
||||
This port of ConnectAddress to map the traffic to.
|
||||
#>
|
||||
param(
|
||||
[Parameter(Mandatory=$true)][UInt16]$ListenPort,
|
||||
[Parameter(Mandatory=$true)][String]$ConnectAddress,
|
||||
[Parameter(Mandatory=$true)][Int]$ConnectPort
|
||||
)
|
||||
|
||||
$connect_field = "$($ConnectAddress):$ConnectPort"
|
||||
$connect_bytes = [System.Text.Encoding]::ASCII.GetBytes($connect_field)
|
||||
$data_length = [byte]($connect_bytes.Length + 6) # size of payload minus header, length, and footer
|
||||
$port_bytes = [System.BitConverter]::GetBytes($ListenPort)
|
||||
|
||||
$payload = [System.Collections.Generic.List`1[Byte]]@()
|
||||
$payload.Add([byte]16) > $null # header is \x10, means Configure Mapping rule
|
||||
$payload.Add($data_length) > $null
|
||||
$payload.AddRange($connect_bytes)
|
||||
$payload.AddRange($port_bytes)
|
||||
$payload.AddRange([byte[]]@(0, 0)) # 2 extra bytes of padding
|
||||
$payload.Add([byte]0) > $null # 0 is TCP, 1 is UDP
|
||||
$payload.Add([byte]0) > $null # 0 is Any, 1 is Internet
|
||||
$payload.Add([byte]31) > $null # footer is \x1f, means end of Configure Mapping rule
|
||||
|
||||
return ,$payload.ToArray()
|
||||
}
|
||||
|
||||
Write-Verbose -Message "Adding host file entries"
|
||||
$hosts_file = "$env:SystemRoot\System32\drivers\etc\hosts"
|
||||
$hosts_file_lines = [System.IO.File]::ReadAllLines($hosts_file)
|
||||
$changed = $false
|
||||
foreach ($httptester_host in $Hosts) {
|
||||
$host_line = "127.0.0.1 $httptester_host # ansible-test httptester"
|
||||
if ($host_line -notin $hosts_file_lines) {
|
||||
$hosts_file_lines += $host_line
|
||||
$changed = $true
|
||||
}
|
||||
}
|
||||
if ($changed) {
|
||||
Write-Verbose -Message "Host file is missing entries, adding missing entries"
|
||||
[System.IO.File]::WriteAllLines($hosts_file, $hosts_file_lines)
|
||||
}
|
||||
|
||||
# forward ports
|
||||
$forwarded_ports = @{
|
||||
80 = 8080
|
||||
443 = 8443
|
||||
444 = 8444
|
||||
}
|
||||
if ($os_version -ge [Version]"6.2") {
|
||||
Write-Verbose -Message "Using netsh to configure forwarded ports"
|
||||
foreach ($forwarded_port in $forwarded_ports.GetEnumerator()) {
|
||||
$port_set = netsh interface portproxy show v4tov4 | `
|
||||
Where-Object { $_ -match "127.0.0.1\s*$($forwarded_port.Key)\s*127.0.0.1\s*$($forwarded_port.Value)" }
|
||||
|
||||
if (-not $port_set) {
|
||||
Write-Verbose -Message "Adding netsh portproxy rule for $($forwarded_port.Key) -> $($forwarded_port.Value)"
|
||||
$add_args = @(
|
||||
"interface",
|
||||
"portproxy",
|
||||
"add",
|
||||
"v4tov4",
|
||||
"listenaddress=127.0.0.1",
|
||||
"listenport=$($forwarded_port.Key)",
|
||||
"connectaddress=127.0.0.1",
|
||||
"connectport=$($forwarded_port.Value)"
|
||||
)
|
||||
$null = netsh $add_args 2>&1
|
||||
}
|
||||
}
|
||||
} else {
|
||||
Write-Verbose -Message "Using Port Mapper to configure forwarded ports"
|
||||
# netsh interface portproxy doesn't work on local addresses in older
|
||||
# versions of Windows. Use custom application Port Mapper to acheive the
|
||||
# same outcome
|
||||
# http://www.analogx.com/contents/download/Network/pmapper/Freeware.htm
|
||||
$s3_url = "https://ansible-ci-files.s3.amazonaws.com/ansible-test/pmapper-1.04.exe"
|
||||
|
||||
# download the Port Mapper executable to a temporary directory
|
||||
$pmapper_folder = Join-Path -Path ([System.IO.Path]::GetTempPath()) -ChildPath ([System.IO.Path]::GetRandomFileName())
|
||||
$pmapper_exe = Join-Path -Path $pmapper_folder -ChildPath pmapper.exe
|
||||
$pmapper_config = Join-Path -Path $pmapper_folder -ChildPath pmapper.dat
|
||||
New-Item -Path $pmapper_folder -ItemType Directory > $null
|
||||
|
||||
$stop = $false
|
||||
do {
|
||||
try {
|
||||
Write-Verbose -Message "Attempting download of '$s3_url'"
|
||||
(New-Object -TypeName System.Net.WebClient).DownloadFile($s3_url, $pmapper_exe)
|
||||
$stop = $true
|
||||
} catch { Start-Sleep -Second 5 }
|
||||
} until ($stop)
|
||||
|
||||
# create the Port Mapper rule file that contains our forwarded ports
|
||||
$fs = [System.IO.File]::Create($pmapper_config)
|
||||
try {
|
||||
foreach ($forwarded_port in $forwarded_ports.GetEnumerator()) {
|
||||
Write-Verbose -Message "Creating forwarded port rule for $($forwarded_port.Key) -> $($forwarded_port.Value)"
|
||||
$pmapper_rule = Get-PmapperRuleBytes -ListenPort $forwarded_port.Key -ConnectAddress 127.0.0.1 -ConnectPort $forwarded_port.Value
|
||||
$fs.Write($pmapper_rule, 0, $pmapper_rule.Length)
|
||||
}
|
||||
} finally {
|
||||
$fs.Close()
|
||||
}
|
||||
|
||||
Write-Verbose -Message "Starting Port Mapper '$pmapper_exe' in the background"
|
||||
$start_args = @{
|
||||
CommandLine = $pmapper_exe
|
||||
CurrentDirectory = $pmapper_folder
|
||||
}
|
||||
$res = Invoke-CimMethod -ClassName Win32_Process -MethodName Create -Arguments $start_args
|
||||
if ($res.ReturnValue -ne 0) {
|
||||
$error_msg = switch($res.ReturnValue) {
|
||||
2 { "Access denied" }
|
||||
3 { "Insufficient privilege" }
|
||||
8 { "Unknown failure" }
|
||||
9 { "Path not found" }
|
||||
21 { "Invalid parameter" }
|
||||
default { "Undefined Error: $($res.ReturnValue)" }
|
||||
}
|
||||
Write-Error -Message "Failed to start pmapper: $error_msg"
|
||||
}
|
||||
$pmapper_pid = $res.ProcessId
|
||||
Write-Verbose -Message "Port Mapper PID: $pmapper_pid"
|
||||
}
|
||||
|
||||
Write-Verbose -Message "Wait for current script at '$PSCommandPath' to be deleted before running cleanup"
|
||||
$fsw = New-Object -TypeName System.IO.FileSystemWatcher
|
||||
$fsw.Path = Split-Path -Path $PSCommandPath -Parent
|
||||
$fsw.Filter = Split-Path -Path $PSCommandPath -Leaf
|
||||
$fsw.WaitForChanged([System.IO.WatcherChangeTypes]::Deleted, 3600000) > $null
|
||||
Write-Verbose -Message "Script delete or timeout reached, cleaning up Windows httptester artifacts"
|
||||
|
||||
Write-Verbose -Message "Cleanup host file entries"
|
||||
$hosts_file_lines = [System.IO.File]::ReadAllLines($hosts_file)
|
||||
$new_lines = [System.Collections.ArrayList]@()
|
||||
$changed = $false
|
||||
foreach ($host_line in $hosts_file_lines) {
|
||||
if ($host_line.EndsWith("# ansible-test httptester")) {
|
||||
$changed = $true
|
||||
continue
|
||||
}
|
||||
$new_lines.Add($host_line) > $null
|
||||
}
|
||||
if ($changed) {
|
||||
Write-Verbose -Message "Host file has extra entries, removing extra entries"
|
||||
[System.IO.File]::WriteAllLines($hosts_file, $new_lines)
|
||||
}
|
||||
|
||||
if ($os_version -ge [Version]"6.2") {
|
||||
Write-Verbose -Message "Cleanup of forwarded port configured in netsh"
|
||||
foreach ($forwarded_port in $forwarded_ports.GetEnumerator()) {
|
||||
$port_set = netsh interface portproxy show v4tov4 | `
|
||||
Where-Object { $_ -match "127.0.0.1\s*$($forwarded_port.Key)\s*127.0.0.1\s*$($forwarded_port.Value)" }
|
||||
|
||||
if ($port_set) {
|
||||
Write-Verbose -Message "Removing netsh portproxy rule for $($forwarded_port.Key) -> $($forwarded_port.Value)"
|
||||
$delete_args = @(
|
||||
"interface",
|
||||
"portproxy",
|
||||
"delete",
|
||||
"v4tov4",
|
||||
"listenaddress=127.0.0.1",
|
||||
"listenport=$($forwarded_port.Key)"
|
||||
)
|
||||
$null = netsh $delete_args 2>&1
|
||||
}
|
||||
}
|
||||
} else {
|
||||
Write-Verbose -Message "Stopping Port Mapper executable based on pid $pmapper_pid"
|
||||
Stop-Process -Id $pmapper_pid -Force
|
||||
|
||||
# the process may not stop straight away, try multiple times to delete the Port Mapper folder
|
||||
$attempts = 1
|
||||
do {
|
||||
try {
|
||||
Write-Verbose -Message "Cleanup temporary files for Port Mapper at '$pmapper_folder' - Attempt: $attempts"
|
||||
Remove-Item -Path $pmapper_folder -Force -Recurse
|
||||
break
|
||||
} catch {
|
||||
Write-Verbose -Message "Cleanup temporary files for Port Mapper failed, waiting 5 seconds before trying again:$($_ | Out-String)"
|
||||
if ($attempts -ge 5) {
|
||||
break
|
||||
}
|
||||
$attempts += 1
|
||||
Start-Sleep -Second 5
|
||||
}
|
||||
} until ($true)
|
||||
}
|
||||
@ -0,0 +1,92 @@
|
||||
"""HTTP Tester plugin for integration tests."""
|
||||
from __future__ import (absolute_import, division, print_function)
|
||||
__metaclass__ = type
|
||||
|
||||
import os
|
||||
|
||||
from . import (
|
||||
CloudProvider,
|
||||
CloudEnvironment,
|
||||
CloudEnvironmentConfig,
|
||||
)
|
||||
|
||||
from ..util import (
|
||||
display,
|
||||
generate_password,
|
||||
)
|
||||
|
||||
from ..config import (
|
||||
IntegrationConfig,
|
||||
)
|
||||
|
||||
from ..containers import (
|
||||
run_support_container,
|
||||
)
|
||||
|
||||
KRB5_PASSWORD_ENV = 'KRB5_PASSWORD'
|
||||
|
||||
|
||||
class HttptesterProvider(CloudProvider):
|
||||
"""HTTP Tester provider plugin. Sets up resources before delegation."""
|
||||
def __init__(self, args): # type: (IntegrationConfig) -> None
|
||||
super(HttptesterProvider, self).__init__(args)
|
||||
|
||||
self.image = os.environ.get('ANSIBLE_HTTP_TEST_CONTAINER', 'quay.io/ansible/http-test-container:1.3.0')
|
||||
|
||||
self.uses_docker = True
|
||||
|
||||
def setup(self): # type: () -> None
|
||||
"""Setup resources before delegation."""
|
||||
super(HttptesterProvider, self).setup()
|
||||
|
||||
ports = [
|
||||
80,
|
||||
88,
|
||||
443,
|
||||
444,
|
||||
749,
|
||||
]
|
||||
|
||||
aliases = [
|
||||
'ansible.http.tests',
|
||||
'sni1.ansible.http.tests',
|
||||
'fail.ansible.http.tests',
|
||||
'self-signed.ansible.http.tests',
|
||||
]
|
||||
|
||||
descriptor = run_support_container(
|
||||
self.args,
|
||||
self.platform,
|
||||
self.image,
|
||||
'http-test-container',
|
||||
ports,
|
||||
aliases=aliases,
|
||||
start=True,
|
||||
allow_existing=True,
|
||||
cleanup=True,
|
||||
env={
|
||||
KRB5_PASSWORD_ENV: generate_password(),
|
||||
},
|
||||
)
|
||||
|
||||
descriptor.register(self.args)
|
||||
|
||||
# Read the password from the container environment.
|
||||
# This allows the tests to work when re-using an existing container.
|
||||
# The password is marked as sensitive, since it may differ from the one we generated.
|
||||
krb5_password = descriptor.details.container.env_dict()[KRB5_PASSWORD_ENV]
|
||||
display.sensitive.add(krb5_password)
|
||||
|
||||
self._set_cloud_config(KRB5_PASSWORD_ENV, krb5_password)
|
||||
|
||||
|
||||
class HttptesterEnvironment(CloudEnvironment):
|
||||
"""HTTP Tester environment plugin. Updates integration test environment after delegation."""
|
||||
def get_environment_config(self): # type: () -> CloudEnvironmentConfig
|
||||
"""Returns the cloud environment config."""
|
||||
return CloudEnvironmentConfig(
|
||||
env_vars=dict(
|
||||
HTTPTESTER='1', # backwards compatibility for tests intended to work with or without HTTP Tester
|
||||
KRB5_PASSWORD=self._get_cloud_config(KRB5_PASSWORD_ENV),
|
||||
)
|
||||
)
|
||||
@ -0,0 +1,755 @@
|
||||
"""High level functions for working with containers."""
|
||||
from __future__ import (absolute_import, division, print_function)
|
||||
__metaclass__ = type
|
||||
|
||||
import atexit
|
||||
import contextlib
|
||||
import json
|
||||
import random
|
||||
import time
|
||||
import uuid
|
||||
|
||||
from . import types as t
|
||||
|
||||
from .encoding import (
|
||||
Text,
|
||||
)
|
||||
|
||||
from .util import (
|
||||
ApplicationError,
|
||||
SubprocessError,
|
||||
display,
|
||||
get_host_ip,
|
||||
sanitize_host_name,
|
||||
)
|
||||
|
||||
from .util_common import (
|
||||
named_temporary_file,
|
||||
)
|
||||
|
||||
from .config import (
|
||||
EnvironmentConfig,
|
||||
IntegrationConfig,
|
||||
WindowsIntegrationConfig,
|
||||
)
|
||||
|
||||
from .docker_util import (
|
||||
ContainerNotFoundError,
|
||||
DockerInspect,
|
||||
docker_exec,
|
||||
docker_inspect,
|
||||
docker_pull,
|
||||
docker_rm,
|
||||
docker_run,
|
||||
docker_start,
|
||||
get_docker_command,
|
||||
get_docker_container_id,
|
||||
get_docker_host_ip,
|
||||
)
|
||||
|
||||
from .ansible_util import (
|
||||
run_playbook,
|
||||
)
|
||||
|
||||
from .core_ci import (
|
||||
SshKey,
|
||||
)
|
||||
|
||||
from .target import (
|
||||
IntegrationTarget,
|
||||
)
|
||||
|
||||
from .ssh import (
|
||||
SshConnectionDetail,
|
||||
SshProcess,
|
||||
create_ssh_port_forwards,
|
||||
create_ssh_port_redirects,
|
||||
generate_ssh_inventory,
|
||||
)
|
||||
|
||||
# information about support containers provisioned by the current ansible-test instance
|
||||
support_containers = {} # type: t.Dict[str, ContainerDescriptor]
|
||||
|
||||
|
||||
class HostType:
|
||||
"""Enum representing the types of hosts involved in running tests."""
|
||||
origin = 'origin'
|
||||
control = 'control'
|
||||
managed = 'managed'
|
||||
|
||||
|
||||
def run_support_container(
|
||||
args, # type: EnvironmentConfig
|
||||
context, # type: str
|
||||
image, # type: str
|
||||
name, # type: name
|
||||
ports, # type: t.List[int]
|
||||
aliases=None, # type: t.Optional[t.List[str]]
|
||||
start=True, # type: bool
|
||||
allow_existing=False, # type: bool
|
||||
cleanup=None, # type: t.Optional[bool]
|
||||
cmd=None, # type: t.Optional[t.List[str]]
|
||||
env=None, # type: t.Optional[t.Dict[str, str]]
|
||||
): # type: (...) -> ContainerDescriptor
|
||||
"""
|
||||
Start a container used to support tests, but not run them.
|
||||
Containers created this way will be accessible from tests.
|
||||
"""
|
||||
if name in support_containers:
|
||||
raise Exception('Container already defined: %s' % name)
|
||||
|
||||
# SSH is required for publishing ports, as well as modifying the hosts file.
|
||||
# Initializing the SSH key here makes sure it is available for use after delegation.
|
||||
SshKey(args)
|
||||
|
||||
aliases = aliases or [sanitize_host_name(name)]
|
||||
|
||||
current_container_id = get_docker_container_id()
|
||||
|
||||
publish_ports = True
|
||||
docker_command = get_docker_command().command
|
||||
|
||||
if docker_command == 'docker':
|
||||
if args.docker:
|
||||
publish_ports = False # publishing ports is not needed when test hosts are on the docker network
|
||||
|
||||
if current_container_id:
|
||||
publish_ports = False # publishing ports is pointless if already running in a docker container
|
||||
|
||||
options = ['--name', name]
|
||||
|
||||
if start:
|
||||
options.append('-d')
|
||||
|
||||
if publish_ports:
|
||||
for port in ports:
|
||||
options.extend(['-p', str(port)])
|
||||
|
||||
if env:
|
||||
for key, value in env.items():
|
||||
options.extend(['--env', '%s=%s' % (key, value)])
|
||||
|
||||
support_container_id = None
|
||||
|
||||
if allow_existing:
|
||||
try:
|
||||
container = docker_inspect(args, name)
|
||||
except ContainerNotFoundError:
|
||||
container = None
|
||||
|
||||
if container:
|
||||
support_container_id = container.id
|
||||
|
||||
if not container.running:
|
||||
display.info('Ignoring existing "%s" container which is not running.' % name, verbosity=1)
|
||||
support_container_id = None
|
||||
elif not container.image:
|
||||
display.info('Ignoring existing "%s" container which has the wrong image.' % name, verbosity=1)
|
||||
support_container_id = None
|
||||
elif publish_ports and not all(port and len(port) == 1 for port in [container.get_tcp_port(port) for port in ports]):
|
||||
display.info('Ignoring existing "%s" container which does not have the required published ports.' % name, verbosity=1)
|
||||
support_container_id = None
|
||||
|
||||
if not support_container_id:
|
||||
docker_rm(args, name)
|
||||
|
||||
if support_container_id:
|
||||
display.info('Using existing "%s" container.' % name)
|
||||
running = True
|
||||
existing = True
|
||||
else:
|
||||
display.info('Starting new "%s" container.' % name)
|
||||
docker_pull(args, image)
|
||||
support_container_id = docker_run(args, image, options, create_only=not start, cmd=cmd)
|
||||
running = start
|
||||
existing = False
|
||||
|
||||
if cleanup is None:
|
||||
cleanup = not existing
|
||||
|
||||
descriptor = ContainerDescriptor(
|
||||
image,
|
||||
context,
|
||||
name,
|
||||
support_container_id,
|
||||
ports,
|
||||
aliases,
|
||||
publish_ports,
|
||||
running,
|
||||
existing,
|
||||
cleanup,
|
||||
env,
|
||||
)
|
||||
|
||||
if not support_containers:
|
||||
atexit.register(cleanup_containers, args)
|
||||
|
||||
support_containers[name] = descriptor
|
||||
|
||||
return descriptor
|
||||
|
||||
|
||||
def get_container_database(args): # type: (EnvironmentConfig) -> ContainerDatabase
|
||||
"""Return the current container database, creating it as needed, or returning the one provided on the command line through delegation."""
|
||||
if not args.containers:
|
||||
args.containers = create_container_database(args)
|
||||
elif isinstance(args.containers, (str, bytes, Text)):
|
||||
args.containers = ContainerDatabase.from_dict(json.loads(args.containers))
|
||||
|
||||
display.info('>>> Container Database\n%s' % json.dumps(args.containers.to_dict(), indent=4, sort_keys=True), verbosity=3)
|
||||
|
||||
return args.containers
|
||||
|
||||
|
||||
class ContainerAccess:
|
||||
"""Information needed for one test host to access a single container supporting tests."""
|
||||
def __init__(self, host_ip, names, ports, forwards): # type: (str, t.List[str], t.Optional[t.List[int]], t.Optional[t.Dict[int, int]]) -> None
|
||||
# if forwards is set
|
||||
# this is where forwards are sent (it is the host that provides an indirect connection to the containers on alternate ports)
|
||||
# /etc/hosts uses 127.0.0.1 (since port redirection will be used)
|
||||
# else
|
||||
# this is what goes into /etc/hosts (it is the container's direct IP)
|
||||
self.host_ip = host_ip
|
||||
|
||||
# primary name + any aliases -- these go into the hosts file and reference the appropriate ip for the origin/control/managed host
|
||||
self.names = names
|
||||
|
||||
# ports available (set if forwards is not set)
|
||||
self.ports = ports
|
||||
|
||||
# port redirections to create through host_ip -- if not set, no port redirections will be used
|
||||
self.forwards = forwards
|
||||
|
||||
def port_map(self): # type: () -> t.List[t.Tuple[int, int]]
|
||||
"""Return a port map for accessing this container."""
|
||||
if self.forwards:
|
||||
ports = list(self.forwards.items())
|
||||
else:
|
||||
ports = [(port, port) for port in self.ports]
|
||||
|
||||
return ports
|
||||
|
||||
@staticmethod
|
||||
def from_dict(data): # type: (t.Dict[str, t.Any]) -> ContainerAccess
|
||||
"""Return a ContainerAccess instance from the given dict."""
|
||||
forwards = data.get('forwards')
|
||||
|
||||
if forwards:
|
||||
forwards = dict((int(key), value) for key, value in forwards.items())
|
||||
|
||||
return ContainerAccess(
|
||||
host_ip=data['host_ip'],
|
||||
names=data['names'],
|
||||
ports=data.get('ports'),
|
||||
forwards=forwards,
|
||||
)
|
||||
|
||||
def to_dict(self): # type: () -> t.Dict[str, t.Any]
|
||||
"""Return a dict of the current instance."""
|
||||
value = dict(
|
||||
host_ip=self.host_ip,
|
||||
names=self.names,
|
||||
)
|
||||
|
||||
if self.ports:
|
||||
value.update(ports=self.ports)
|
||||
|
||||
if self.forwards:
|
||||
value.update(forwards=self.forwards)
|
||||
|
||||
return value
|
||||
|
||||
|
||||
class ContainerDatabase:
|
||||
"""Database of running containers used to support tests."""
|
||||
def __init__(self, data): # type: (t.Dict[str, t.Dict[str, t.Dict[str, ContainerAccess]]]) -> None
|
||||
self.data = data
|
||||
|
||||
@staticmethod
|
||||
def from_dict(data): # type: (t.Dict[str, t.Any]) -> ContainerDatabase
|
||||
"""Return a ContainerDatabase instance from the given dict."""
|
||||
return ContainerDatabase(dict((access_name,
|
||||
dict((context_name,
|
||||
dict((container_name, ContainerAccess.from_dict(container))
|
||||
for container_name, container in containers.items()))
|
||||
for context_name, containers in contexts.items()))
|
||||
for access_name, contexts in data.items()))
|
||||
|
||||
def to_dict(self): # type: () -> t.Dict[str, t.Any]
|
||||
"""Return a dict of the current instance."""
|
||||
return dict((access_name,
|
||||
dict((context_name,
|
||||
dict((container_name, container.to_dict())
|
||||
for container_name, container in containers.items()))
|
||||
for context_name, containers in contexts.items()))
|
||||
for access_name, contexts in self.data.items())
|
||||
|
||||
|
||||
def local_ssh(args): # type: (EnvironmentConfig) -> SshConnectionDetail
|
||||
"""Return SSH connection details for localhost, connecting as root to the default SSH port."""
|
||||
return SshConnectionDetail('localhost', 'localhost', None, 'root', SshKey(args).key, args.python_executable)
|
||||
|
||||
|
||||
def create_container_database(args): # type: (EnvironmentConfig) -> ContainerDatabase
|
||||
"""Create and return a container database with information necessary for all test hosts to make use of relevant support containers."""
|
||||
origin = {} # type: t.Dict[str, t.Dict[str, ContainerAccess]]
|
||||
control = {} # type: t.Dict[str, t.Dict[str, ContainerAccess]]
|
||||
managed = {} # type: t.Dict[str, t.Dict[str, ContainerAccess]]
|
||||
|
||||
for name, container in support_containers.items():
|
||||
if container.details.published_ports:
|
||||
published_access = ContainerAccess(
|
||||
host_ip=get_docker_host_ip(),
|
||||
names=container.aliases,
|
||||
ports=None,
|
||||
forwards=dict((port, published_port) for port, published_port in container.details.published_ports.items()),
|
||||
)
|
||||
else:
|
||||
published_access = None # no published access without published ports (ports are only published if needed)
|
||||
|
||||
if container.details.container_ip:
|
||||
# docker containers, and rootfull podman containers should have a container IP address
|
||||
container_access = ContainerAccess(
|
||||
host_ip=container.details.container_ip,
|
||||
names=container.aliases,
|
||||
ports=container.ports,
|
||||
forwards=None,
|
||||
)
|
||||
elif get_docker_command().command == 'podman':
|
||||
# published ports for rootless podman containers should be accessible from the host's IP
|
||||
container_access = ContainerAccess(
|
||||
host_ip=get_host_ip(),
|
||||
names=container.aliases,
|
||||
ports=None,
|
||||
forwards=dict((port, published_port) for port, published_port in container.details.published_ports.items()),
|
||||
)
|
||||
else:
|
||||
container_access = None # no container access without an IP address
|
||||
|
||||
if get_docker_container_id():
|
||||
if not container_access:
|
||||
raise Exception('Missing IP address for container: %s' % name)
|
||||
|
||||
origin_context = origin.setdefault(container.context, {})
|
||||
origin_context[name] = container_access
|
||||
elif not published_access:
|
||||
pass # origin does not have network access to the containers
|
||||
else:
|
||||
origin_context = origin.setdefault(container.context, {})
|
||||
origin_context[name] = published_access
|
||||
|
||||
if args.remote:
|
||||
pass # SSH forwarding required
|
||||
elif args.docker or get_docker_container_id():
|
||||
if container_access:
|
||||
control_context = control.setdefault(container.context, {})
|
||||
control_context[name] = container_access
|
||||
else:
|
||||
raise Exception('Missing IP address for container: %s' % name)
|
||||
else:
|
||||
if not published_access:
|
||||
raise Exception('Missing published ports for container: %s' % name)
|
||||
|
||||
control_context = control.setdefault(container.context, {})
|
||||
control_context[name] = published_access
|
||||
|
||||
data = {
|
||||
HostType.origin: origin,
|
||||
HostType.control: control,
|
||||
HostType.managed: managed,
|
||||
}
|
||||
|
||||
data = dict((key, value) for key, value in data.items() if value)
|
||||
|
||||
return ContainerDatabase(data)
|
||||
|
||||
|
||||
class SupportContainerContext:
|
||||
"""Context object for tracking information relating to access of support containers."""
|
||||
def __init__(self, containers, process): # type: (ContainerDatabase, t.Optional[SshProcess]) -> None
|
||||
self.containers = containers
|
||||
self.process = process
|
||||
|
||||
def close(self): # type: () -> None
|
||||
"""Close the process maintaining the port forwards."""
|
||||
if not self.process:
|
||||
return # forwarding not in use
|
||||
|
||||
self.process.terminate()
|
||||
|
||||
display.info('Waiting for the session SSH port forwarding process to terminate.', verbosity=1)
|
||||
|
||||
self.process.wait()
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def support_container_context(
|
||||
args, # type: EnvironmentConfig
|
||||
ssh, # type: t.Optional[SshConnectionDetail]
|
||||
): # type: (...) -> t.Optional[ContainerDatabase]
|
||||
"""Create a context manager for integration tests that use support containers."""
|
||||
if not isinstance(args, IntegrationConfig):
|
||||
yield None # containers are only used for integration tests
|
||||
return
|
||||
|
||||
containers = get_container_database(args)
|
||||
|
||||
if not containers.data:
|
||||
yield ContainerDatabase({}) # no containers are being used, return an empty database
|
||||
return
|
||||
|
||||
context = create_support_container_context(args, ssh, containers)
|
||||
|
||||
try:
|
||||
yield context.containers
|
||||
finally:
|
||||
context.close()
|
||||
|
||||
|
||||
def create_support_container_context(
|
||||
args, # type: EnvironmentConfig
|
||||
ssh, # type: t.Optional[SshConnectionDetail]
|
||||
containers, # type: ContainerDatabase
|
||||
): # type: (...) -> SupportContainerContext
|
||||
"""Context manager that provides SSH port forwards. Returns updated container metadata."""
|
||||
host_type = HostType.control
|
||||
|
||||
revised = ContainerDatabase(containers.data.copy())
|
||||
source = revised.data.pop(HostType.origin, None)
|
||||
|
||||
container_map = {} # type: t.Dict[t.Tuple[str, int], t.Tuple[str, str, int]]
|
||||
|
||||
if host_type not in revised.data:
|
||||
if not source:
|
||||
raise Exception('Missing origin container details.')
|
||||
|
||||
for context_name, context in source.items():
|
||||
for container_name, container in context.items():
|
||||
for port, access_port in container.port_map():
|
||||
container_map[(container.host_ip, access_port)] = (context_name, container_name, port)
|
||||
|
||||
if not container_map:
|
||||
return SupportContainerContext(revised, None)
|
||||
|
||||
if not ssh:
|
||||
raise Exception('The %s host was not pre-configured for container access and SSH forwarding is not available.' % host_type)
|
||||
|
||||
forwards = list(container_map.keys())
|
||||
process = create_ssh_port_forwards(args, ssh, forwards)
|
||||
result = SupportContainerContext(revised, process)
|
||||
|
||||
try:
|
||||
port_forwards = process.collect_port_forwards()
|
||||
contexts = {}
|
||||
|
||||
for forward, forwarded_port in port_forwards.items():
|
||||
access_host, access_port = forward
|
||||
context_name, container_name, container_port = container_map[(access_host, access_port)]
|
||||
container = source[context_name][container_name]
|
||||
context = contexts.setdefault(context_name, {})
|
||||
|
||||
forwarded_container = context.setdefault(container_name, ContainerAccess('127.0.0.1', container.names, None, {}))
|
||||
forwarded_container.forwards[container_port] = forwarded_port
|
||||
|
||||
display.info('Container "%s" port %d available at %s:%d is forwarded over SSH as port %d.' % (
|
||||
container_name, container_port, access_host, access_port, forwarded_port,
|
||||
), verbosity=1)
|
||||
|
||||
revised.data[host_type] = contexts
|
||||
|
||||
return result
|
||||
except Exception:
|
||||
result.close()
|
||||
raise
|
||||
|
||||
|
||||
class ContainerDescriptor:
|
||||
"""Information about a support container."""
|
||||
def __init__(self,
|
||||
image, # type: str
|
||||
context, # type: str
|
||||
name, # type: str
|
||||
container_id, # type: str
|
||||
ports, # type: t.List[int]
|
||||
aliases, # type: t.List[str]
|
||||
publish_ports, # type: bool
|
||||
running, # type: bool
|
||||
existing, # type: bool
|
||||
cleanup, # type: bool
|
||||
env, # type: t.Optional[t.Dict[str, str]]
|
||||
): # type: (...) -> None
|
||||
self.image = image
|
||||
self.context = context
|
||||
self.name = name
|
||||
self.container_id = container_id
|
||||
self.ports = ports
|
||||
self.aliases = aliases
|
||||
self.publish_ports = publish_ports
|
||||
self.running = running
|
||||
self.existing = existing
|
||||
self.cleanup = cleanup
|
||||
self.env = env
|
||||
self.details = None # type: t.Optional[SupportContainer]
|
||||
|
||||
def start(self, args): # type: (EnvironmentConfig) -> None
|
||||
"""Start the container. Used for containers which are created, but not started."""
|
||||
docker_start(args, self.name)
|
||||
|
||||
def register(self, args): # type: (EnvironmentConfig) -> SupportContainer
|
||||
"""Record the container's runtime details. Must be used after the container has been started."""
|
||||
if self.details:
|
||||
raise Exception('Container already registered: %s' % self.name)
|
||||
|
||||
try:
|
||||
container = docker_inspect(args, self.container_id)
|
||||
except ContainerNotFoundError:
|
||||
if not args.explain:
|
||||
raise
|
||||
|
||||
# provide enough mock data to keep --explain working
|
||||
container = DockerInspect(args, dict(
|
||||
Id=self.container_id,
|
||||
NetworkSettings=dict(
|
||||
IPAddress='127.0.0.1',
|
||||
Ports=dict(('%d/tcp' % port, [dict(HostPort=random.randint(30000, 40000) if self.publish_ports else port)]) for port in self.ports),
|
||||
),
|
||||
Config=dict(
|
||||
Env=['%s=%s' % (key, value) for key, value in self.env.items()] if self.env else [],
|
||||
),
|
||||
))
|
||||
|
||||
support_container_ip = container.get_ip_address()
|
||||
|
||||
if self.publish_ports:
|
||||
# inspect the support container to locate the published ports
|
||||
tcp_ports = dict((port, container.get_tcp_port(port)) for port in self.ports)
|
||||
|
||||
if any(not config or len(config) != 1 for config in tcp_ports.values()):
|
||||
raise ApplicationError('Unexpected `docker inspect` results for published TCP ports:\n%s' % json.dumps(tcp_ports, indent=4, sort_keys=True))
|
||||
|
||||
published_ports = dict((port, int(config[0]['HostPort'])) for port, config in tcp_ports.items())
|
||||
else:
|
||||
published_ports = {}
|
||||
|
||||
self.details = SupportContainer(
|
||||
container,
|
||||
support_container_ip,
|
||||
published_ports,
|
||||
)
|
||||
|
||||
return self.details
|
||||
|
||||
|
||||
class SupportContainer:
|
||||
"""Information about a running support container available for use by tests."""
|
||||
def __init__(self,
|
||||
container, # type: DockerInspect
|
||||
container_ip, # type: str
|
||||
published_ports, # type: t.Dict[int, int]
|
||||
): # type: (...) -> None
|
||||
self.container = container
|
||||
self.container_ip = container_ip
|
||||
self.published_ports = published_ports
|
||||
|
||||
|
||||
def wait_for_file(args, # type: EnvironmentConfig
|
||||
container_name, # type: str
|
||||
path, # type: str
|
||||
sleep, # type: int
|
||||
tries, # type: int
|
||||
check=None, # type: t.Optional[t.Callable[[str], bool]]
|
||||
): # type: (...) -> str
|
||||
"""Wait for the specified file to become available in the requested container and return its contents."""
|
||||
display.info('Waiting for container "%s" to provide file: %s' % (container_name, path))
|
||||
|
||||
for _iteration in range(1, tries):
|
||||
if _iteration > 1:
|
||||
time.sleep(sleep)
|
||||
|
||||
try:
|
||||
stdout = docker_exec(args, container_name, ['dd', 'if=%s' % path], capture=True)[0]
|
||||
except SubprocessError:
|
||||
continue
|
||||
|
||||
if not check or check(stdout):
|
||||
return stdout
|
||||
|
||||
raise ApplicationError('Timeout waiting for container "%s" to provide file: %s' % (container_name, path))
|
||||
|
||||
|
||||
def cleanup_containers(args): # type: (EnvironmentConfig) -> None
|
||||
"""Clean up containers."""
|
||||
for container in support_containers.values():
|
||||
if container.cleanup:
|
||||
docker_rm(args, container.container_id)
|
||||
else:
|
||||
display.notice('Remember to run `docker rm -f %s` when finished testing.' % container.name)
|
||||
|
||||
|
||||
def create_hosts_entries(context): # type: (t.Dict[str, ContainerAccess]) -> t.List[str]
|
||||
"""Return hosts entries for the specified context."""
|
||||
entries = []
|
||||
unique_id = uuid.uuid4()
|
||||
|
||||
for container in context.values():
|
||||
# forwards require port redirection through localhost
|
||||
if container.forwards:
|
||||
host_ip = '127.0.0.1'
|
||||
else:
|
||||
host_ip = container.host_ip
|
||||
|
||||
entries.append('%s %s # ansible-test %s' % (host_ip, ' '.join(container.names), unique_id))
|
||||
|
||||
return entries
|
||||
|
||||
|
||||
def create_container_hooks(
|
||||
args, # type: IntegrationConfig
|
||||
managed_connections, # type: t.Optional[t.List[SshConnectionDetail]]
|
||||
): # type: (...) -> t.Tuple[t.Optional[t.Callable[[IntegrationTarget], None]], t.Optional[t.Callable[[IntegrationTarget], None]]]
|
||||
"""Return pre and post target callbacks for enabling and disabling container access for each test target."""
|
||||
containers = get_container_database(args)
|
||||
|
||||
control_contexts = containers.data.get(HostType.control)
|
||||
|
||||
if control_contexts:
|
||||
managed_contexts = containers.data.get(HostType.managed)
|
||||
|
||||
if not managed_contexts:
|
||||
managed_contexts = create_managed_contexts(control_contexts)
|
||||
|
||||
control_type = 'posix'
|
||||
|
||||
if isinstance(args, WindowsIntegrationConfig):
|
||||
managed_type = 'windows'
|
||||
else:
|
||||
managed_type = 'posix'
|
||||
|
||||
control_state = {}
|
||||
managed_state = {}
|
||||
|
||||
control_connections = [local_ssh(args)]
|
||||
|
||||
def pre_target(target):
|
||||
forward_ssh_ports(args, control_connections, '%s_hosts_prepare.yml' % control_type, control_state, target, HostType.control, control_contexts)
|
||||
forward_ssh_ports(args, managed_connections, '%s_hosts_prepare.yml' % managed_type, managed_state, target, HostType.managed, managed_contexts)
|
||||
|
||||
def post_target(target):
|
||||
cleanup_ssh_ports(args, control_connections, '%s_hosts_restore.yml' % control_type, control_state, target, HostType.control)
|
||||
cleanup_ssh_ports(args, managed_connections, '%s_hosts_restore.yml' % managed_type, managed_state, target, HostType.managed)
|
||||
else:
|
||||
pre_target, post_target = None, None
|
||||
|
||||
return pre_target, post_target
|
||||
|
||||
|
||||
def create_managed_contexts(control_contexts): # type: (t.Dict[str, t.Dict[str, ContainerAccess]]) -> t.Dict[str, t.Dict[str, ContainerAccess]]
|
||||
"""Create managed contexts from the given control contexts."""
|
||||
managed_contexts = {}
|
||||
|
||||
for context_name, control_context in control_contexts.items():
|
||||
managed_context = managed_contexts[context_name] = {}
|
||||
|
||||
for container_name, control_container in control_context.items():
|
||||
managed_context[container_name] = ContainerAccess(control_container.host_ip, control_container.names, None, dict(control_container.port_map()))
|
||||
|
||||
return managed_contexts
|
||||
|
||||
|
||||
def forward_ssh_ports(
|
||||
args, # type: IntegrationConfig
|
||||
ssh_connections, # type: t.Optional[t.List[SshConnectionDetail]]
|
||||
playbook, # type: str
|
||||
target_state, # type: t.Dict[str, t.Tuple[t.List[str], t.List[SshProcess]]]
|
||||
target, # type: IntegrationTarget
|
||||
host_type, # type: str
|
||||
contexts, # type: t.Dict[str, t.Dict[str, ContainerAccess]]
|
||||
): # type: (...) -> None
|
||||
"""Configure port forwarding using SSH and write hosts file entries."""
|
||||
if ssh_connections is None:
|
||||
return
|
||||
|
||||
test_context = None
|
||||
|
||||
for context_name, context in contexts.items():
|
||||
context_alias = 'cloud/%s/' % context_name
|
||||
|
||||
if context_alias in target.aliases:
|
||||
test_context = context
|
||||
break
|
||||
|
||||
if not test_context:
|
||||
return
|
||||
|
||||
if not ssh_connections:
|
||||
raise Exception('The %s host was not pre-configured for container access and SSH forwarding is not available.' % host_type)
|
||||
|
||||
redirects = [] # type: t.List[t.Tuple[int, str, int]]
|
||||
messages = []
|
||||
|
||||
for container_name, container in test_context.items():
|
||||
explain = []
|
||||
|
||||
for container_port, access_port in container.port_map():
|
||||
if container.forwards:
|
||||
redirects.append((container_port, container.host_ip, access_port))
|
||||
|
||||
explain.append('%d -> %s:%d' % (container_port, container.host_ip, access_port))
|
||||
else:
|
||||
explain.append('%s:%d' % (container.host_ip, container_port))
|
||||
|
||||
if explain:
|
||||
if container.forwards:
|
||||
message = 'Port forwards for the "%s" container have been established on the %s host' % (container_name, host_type)
|
||||
else:
|
||||
message = 'Ports for the "%s" container are available on the %s host as' % (container_name, host_type)
|
||||
|
||||
messages.append('%s:\n%s' % (message, '\n'.join(explain)))
|
||||
|
||||
hosts_entries = create_hosts_entries(test_context)
|
||||
inventory = generate_ssh_inventory(ssh_connections)
|
||||
|
||||
with named_temporary_file(args, 'ssh-inventory-', '.json', None, inventory) as inventory_path:
|
||||
run_playbook(args, inventory_path, playbook, dict(hosts_entries=hosts_entries))
|
||||
|
||||
ssh_processes = [] # type: t.List[SshProcess]
|
||||
|
||||
if redirects:
|
||||
for ssh in ssh_connections:
|
||||
ssh_processes.append(create_ssh_port_redirects(args, ssh, redirects))
|
||||
|
||||
target_state[target.name] = (hosts_entries, ssh_processes)
|
||||
|
||||
for message in messages:
|
||||
display.info(message, verbosity=1)
|
||||
|
||||
|
||||
def cleanup_ssh_ports(
|
||||
args, # type: IntegrationConfig
|
||||
ssh_connections, # type: t.List[SshConnectionDetail]
|
||||
playbook, # type: str
|
||||
target_state, # type: t.Dict[str, t.Tuple[t.List[str], t.List[SshProcess]]]
|
||||
target, # type: IntegrationTarget
|
||||
host_type, # type: str
|
||||
): # type: (...) -> None
|
||||
"""Stop previously configured SSH port forwarding and remove previously written hosts file entries."""
|
||||
state = target_state.pop(target.name, None)
|
||||
|
||||
if not state:
|
||||
return
|
||||
|
||||
(hosts_entries, ssh_processes) = state
|
||||
|
||||
inventory = generate_ssh_inventory(ssh_connections)
|
||||
|
||||
with named_temporary_file(args, 'ssh-inventory-', '.json', None, inventory) as inventory_path:
|
||||
run_playbook(args, inventory_path, playbook, dict(hosts_entries=hosts_entries))
|
||||
|
||||
if ssh_processes:
|
||||
for process in ssh_processes:
|
||||
process.terminate()
|
||||
|
||||
display.info('Waiting for the %s host SSH port forwarding processs(es) to terminate.' % host_type, verbosity=1)
|
||||
|
||||
for process in ssh_processes:
|
||||
process.wait()
|
||||
@ -0,0 +1,264 @@
|
||||
"""High level functions for working with SSH."""
|
||||
from __future__ import (absolute_import, division, print_function)
|
||||
__metaclass__ = type
|
||||
|
||||
import json
|
||||
import os
|
||||
import random
|
||||
import re
|
||||
import subprocess
|
||||
|
||||
from . import types as t
|
||||
|
||||
from .encoding import (
|
||||
to_bytes,
|
||||
to_text,
|
||||
)
|
||||
|
||||
from .util import (
|
||||
ApplicationError,
|
||||
cmd_quote,
|
||||
common_environment,
|
||||
devnull,
|
||||
display,
|
||||
exclude_none_values,
|
||||
sanitize_host_name,
|
||||
)
|
||||
|
||||
from .config import (
|
||||
EnvironmentConfig,
|
||||
)
|
||||
|
||||
|
||||
class SshConnectionDetail:
|
||||
"""Information needed to establish an SSH connection to a host."""
|
||||
def __init__(self,
|
||||
name, # type: str
|
||||
host, # type: str
|
||||
port, # type: t.Optional[int]
|
||||
user, # type: str
|
||||
identity_file, # type: str
|
||||
python_interpreter=None, # type: t.Optional[str]
|
||||
shell_type=None, # type: t.Optional[str]
|
||||
): # type: (...) -> None
|
||||
self.name = sanitize_host_name(name)
|
||||
self.host = host
|
||||
self.port = port
|
||||
self.user = user
|
||||
self.identity_file = identity_file
|
||||
self.python_interpreter = python_interpreter
|
||||
self.shell_type = shell_type
|
||||
|
||||
|
||||
class SshProcess:
|
||||
"""Wrapper around an SSH process."""
|
||||
def __init__(self, process): # type: (t.Optional[subprocess.Popen]) -> None
|
||||
self._process = process
|
||||
self.pending_forwards = None # type: t.Optional[t.Set[t.Tuple[str, int]]]
|
||||
|
||||
self.forwards = {} # type: t.Dict[t.Tuple[str, int], int]
|
||||
|
||||
def terminate(self): # type: () -> None
|
||||
"""Terminate the SSH process."""
|
||||
if not self._process:
|
||||
return # explain mode
|
||||
|
||||
# noinspection PyBroadException
|
||||
try:
|
||||
self._process.terminate()
|
||||
except Exception: # pylint: disable=broad-except
|
||||
pass
|
||||
|
||||
def wait(self): # type: () -> None
|
||||
"""Wait for the SSH process to terminate."""
|
||||
if not self._process:
|
||||
return # explain mode
|
||||
|
||||
self._process.wait()
|
||||
|
||||
def collect_port_forwards(self): # type: (SshProcess) -> t.Dict[t.Tuple[str, int], int]
|
||||
"""Collect port assignments for dynamic SSH port forwards."""
|
||||
errors = []
|
||||
|
||||
display.info('Collecting %d SSH port forward(s).' % len(self.pending_forwards), verbosity=2)
|
||||
|
||||
while self.pending_forwards:
|
||||
if self._process:
|
||||
line_bytes = self._process.stderr.readline()
|
||||
|
||||
if not line_bytes:
|
||||
if errors:
|
||||
details = ':\n%s' % '\n'.join(errors)
|
||||
else:
|
||||
details = '.'
|
||||
|
||||
raise ApplicationError('SSH port forwarding failed%s' % details)
|
||||
|
||||
line = to_text(line_bytes).strip()
|
||||
|
||||
match = re.search(r'^Allocated port (?P<src_port>[0-9]+) for remote forward to (?P<dst_host>[^:]+):(?P<dst_port>[0-9]+)$', line)
|
||||
|
||||
if not match:
|
||||
if re.search(r'^Warning: Permanently added .* to the list of known hosts\.$', line):
|
||||
continue
|
||||
|
||||
display.warning('Unexpected SSH port forwarding output: %s' % line, verbosity=2)
|
||||
|
||||
errors.append(line)
|
||||
continue
|
||||
|
||||
src_port = int(match.group('src_port'))
|
||||
dst_host = str(match.group('dst_host'))
|
||||
dst_port = int(match.group('dst_port'))
|
||||
|
||||
dst = (dst_host, dst_port)
|
||||
else:
|
||||
# explain mode
|
||||
dst = list(self.pending_forwards)[0]
|
||||
src_port = random.randint(40000, 50000)
|
||||
|
||||
self.pending_forwards.remove(dst)
|
||||
self.forwards[dst] = src_port
|
||||
|
||||
display.info('Collected %d SSH port forward(s):\n%s' % (
|
||||
len(self.forwards), '\n'.join('%s -> %s:%s' % (src_port, dst[0], dst[1]) for dst, src_port in sorted(self.forwards.items()))), verbosity=2)
|
||||
|
||||
return self.forwards
|
||||
|
||||
|
||||
def create_ssh_command(
|
||||
ssh, # type: SshConnectionDetail
|
||||
options=None, # type: t.Optional[t.Dict[str, t.Union[str, int]]]
|
||||
cli_args=None, # type: t.List[str]
|
||||
command=None, # type: t.Optional[str]
|
||||
): # type: (...) -> t.List[str]
|
||||
"""Create an SSH command using the specified options."""
|
||||
cmd = [
|
||||
'ssh',
|
||||
'-n', # prevent reading from stdin
|
||||
'-i', ssh.identity_file, # file from which the identity for public key authentication is read
|
||||
]
|
||||
|
||||
if not command:
|
||||
cmd.append('-N') # do not execute a remote command
|
||||
|
||||
if ssh.port:
|
||||
cmd.extend(['-p', str(ssh.port)]) # port to connect to on the remote host
|
||||
|
||||
if ssh.user:
|
||||
cmd.extend(['-l', ssh.user]) # user to log in as on the remote machine
|
||||
|
||||
ssh_options = dict(
|
||||
BatchMode='yes',
|
||||
ExitOnForwardFailure='yes',
|
||||
LogLevel='ERROR',
|
||||
ServerAliveCountMax=4,
|
||||
ServerAliveInterval=15,
|
||||
StrictHostKeyChecking='no',
|
||||
UserKnownHostsFile='/dev/null',
|
||||
)
|
||||
|
||||
ssh_options.update(options or {})
|
||||
|
||||
for key, value in sorted(ssh_options.items()):
|
||||
cmd.extend(['-o', '='.join([key, str(value)])])
|
||||
|
||||
cmd.extend(cli_args or [])
|
||||
cmd.append(ssh.host)
|
||||
|
||||
if command:
|
||||
cmd.append(command)
|
||||
|
||||
return cmd
|
||||
|
||||
|
||||
def run_ssh_command(
|
||||
args, # type: EnvironmentConfig
|
||||
ssh, # type: SshConnectionDetail
|
||||
options=None, # type: t.Optional[t.Dict[str, t.Union[str, int]]]
|
||||
cli_args=None, # type: t.List[str]
|
||||
command=None, # type: t.Optional[str]
|
||||
): # type: (...) -> SshProcess
|
||||
"""Run the specified SSH command, returning the created SshProcess instance created."""
|
||||
cmd = create_ssh_command(ssh, options, cli_args, command)
|
||||
env = common_environment()
|
||||
|
||||
cmd_show = ' '.join([cmd_quote(c) for c in cmd])
|
||||
display.info('Run background command: %s' % cmd_show, verbosity=1, truncate=True)
|
||||
|
||||
cmd_bytes = [to_bytes(c) for c in cmd]
|
||||
env_bytes = dict((to_bytes(k), to_bytes(v)) for k, v in env.items())
|
||||
|
||||
if args.explain:
|
||||
process = SshProcess(None)
|
||||
else:
|
||||
process = SshProcess(subprocess.Popen(cmd_bytes, env=env_bytes, bufsize=-1, stdin=devnull(), stdout=subprocess.PIPE, stderr=subprocess.PIPE))
|
||||
|
||||
return process
|
||||
|
||||
|
||||
def create_ssh_port_forwards(
|
||||
args, # type: EnvironmentConfig
|
||||
ssh, # type: SshConnectionDetail
|
||||
forwards, # type: t.List[t.Tuple[str, int]]
|
||||
): # type: (...) -> SshProcess
|
||||
"""
|
||||
Create SSH port forwards using the provided list of tuples (target_host, target_port).
|
||||
Port bindings will be automatically assigned by SSH and must be collected with a subseqent call to collect_port_forwards.
|
||||
"""
|
||||
options = dict(
|
||||
LogLevel='INFO', # info level required to get messages on stderr indicating the ports assigned to each forward
|
||||
)
|
||||
|
||||
cli_args = []
|
||||
|
||||
for forward_host, forward_port in forwards:
|
||||
cli_args.extend(['-R', ':'.join([str(0), forward_host, str(forward_port)])])
|
||||
|
||||
process = run_ssh_command(args, ssh, options, cli_args)
|
||||
process.pending_forwards = forwards
|
||||
|
||||
return process
|
||||
|
||||
|
||||
def create_ssh_port_redirects(
|
||||
args, # type: EnvironmentConfig
|
||||
ssh, # type: SshConnectionDetail
|
||||
redirects, # type: t.List[t.Tuple[int, str, int]]
|
||||
): # type: (...) -> SshProcess
|
||||
"""Create SSH port redirections using the provided list of tuples (bind_port, target_host, target_port)."""
|
||||
options = {}
|
||||
cli_args = []
|
||||
|
||||
for bind_port, target_host, target_port in redirects:
|
||||
cli_args.extend(['-R', ':'.join([str(bind_port), target_host, str(target_port)])])
|
||||
|
||||
process = run_ssh_command(args, ssh, options, cli_args)
|
||||
|
||||
return process
|
||||
|
||||
|
||||
def generate_ssh_inventory(ssh_connections): # type: (t.List[SshConnectionDetail]) -> str
|
||||
"""Return an inventory file in JSON format, created from the provided SSH connection details."""
|
||||
inventory = dict(
|
||||
all=dict(
|
||||
hosts=dict((ssh.name, exclude_none_values(dict(
|
||||
ansible_host=ssh.host,
|
||||
ansible_port=ssh.port,
|
||||
ansible_user=ssh.user,
|
||||
ansible_ssh_private_key_file=os.path.abspath(ssh.identity_file),
|
||||
ansible_connection='ssh',
|
||||
ansible_ssh_pipelining='yes',
|
||||
ansible_python_interpreter=ssh.python_interpreter,
|
||||
ansible_shell_type=ssh.shell_type,
|
||||
ansible_ssh_extra_args='-o UserKnownHostsFile=/dev/null', # avoid changing the test environment
|
||||
ansible_ssh_host_key_checking='no',
|
||||
))) for ssh in ssh_connections),
|
||||
),
|
||||
)
|
||||
|
||||
inventory_text = json.dumps(inventory, indent=4, sort_keys=True)
|
||||
|
||||
display.info('>>> SSH Inventory\n%s' % inventory_text, verbosity=3)
|
||||
|
||||
return inventory_text
|
||||
@ -0,0 +1,518 @@
|
||||
# Copyright (c) 2020 Ansible Project
|
||||
# Simplified BSD License (see licenses/simplified_bsd.txt or https://opensource.org/licenses/BSD-2-Clause)
|
||||
|
||||
Function Get-AnsibleWindowsWebRequest {
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Creates a System.Net.WebRequest object based on common URL module options in Ansible.
|
||||
|
||||
.DESCRIPTION
|
||||
Will create a WebRequest based on common input options within Ansible. This can be used manually or with
|
||||
Invoke-AnsibleWindowsWebRequest.
|
||||
|
||||
.PARAMETER Uri
|
||||
The URI to create the web request for.
|
||||
|
||||
.PARAMETER UrlMethod
|
||||
The protocol method to use, if omitted, will use the default value for the URI protocol specified.
|
||||
|
||||
.PARAMETER FollowRedirects
|
||||
Whether to follow redirect reponses. This is only valid when using a HTTP URI.
|
||||
all - Will follow all redirects
|
||||
none - Will follow no redirects
|
||||
safe - Will only follow redirects when GET or HEAD is used as the UrlMethod
|
||||
|
||||
.PARAMETER Headers
|
||||
A hashtable or dictionary of header values to set on the request. This is only valid for a HTTP URI.
|
||||
|
||||
.PARAMETER HttpAgent
|
||||
A string to set for the 'User-Agent' header. This is only valid for a HTTP URI.
|
||||
|
||||
.PARAMETER MaximumRedirection
|
||||
The maximum number of redirections that will be followed. This is only valid for a HTTP URI.
|
||||
|
||||
.PARAMETER UrlTimeout
|
||||
The timeout in seconds that defines how long to wait until the request times out.
|
||||
|
||||
.PARAMETER ValidateCerts
|
||||
Whether to validate SSL certificates, default to True.
|
||||
|
||||
.PARAMETER ClientCert
|
||||
The path to PFX file to use for X509 authentication. This is only valid for a HTTP URI. This path can either
|
||||
be a filesystem path (C:\folder\cert.pfx) or a PSPath to a credential (Cert:\CurrentUser\My\<thumbprint>).
|
||||
|
||||
.PARAMETER ClientCertPassword
|
||||
The password for the PFX certificate if required. This is only valid for a HTTP URI.
|
||||
|
||||
.PARAMETER ForceBasicAuth
|
||||
Whether to set the Basic auth header on the first request instead of when required. This is only valid for a
|
||||
HTTP URI.
|
||||
|
||||
.PARAMETER UrlUsername
|
||||
The username to use for authenticating with the target.
|
||||
|
||||
.PARAMETER UrlPassword
|
||||
The password to use for authenticating with the target.
|
||||
|
||||
.PARAMETER UseDefaultCredential
|
||||
Whether to use the current user's credentials if available. This will only work when using Become, using SSH with
|
||||
password auth, or WinRM with CredSSP or Kerberos with credential delegation.
|
||||
|
||||
.PARAMETER UseProxy
|
||||
Whether to use the default proxy defined in IE (WinINet) for the user or set no proxy at all. This should not
|
||||
be set to True when ProxyUrl is also defined.
|
||||
|
||||
.PARAMETER ProxyUrl
|
||||
An explicit proxy server to use for the request instead of relying on the default proxy in IE. This is only
|
||||
valid for a HTTP URI.
|
||||
|
||||
.PARAMETER ProxyUsername
|
||||
An optional username to use for proxy authentication.
|
||||
|
||||
.PARAMETER ProxyPassword
|
||||
The password for ProxyUsername.
|
||||
|
||||
.PARAMETER ProxyUseDefaultCredential
|
||||
Whether to use the current user's credentials for proxy authentication if available. This will only work when
|
||||
using Become, using SSH with password auth, or WinRM with CredSSP or Kerberos with credential delegation.
|
||||
|
||||
.PARAMETER Module
|
||||
The AnsibleBasic module that can be used as a backup parameter source or a way to return warnings back to the
|
||||
Ansible controller.
|
||||
|
||||
.EXAMPLE
|
||||
$spec = @{
|
||||
options = @{}
|
||||
}
|
||||
$module = [Ansible.Basic.AnsibleModule]::Create($args, $spec, @(Get-AnsibleWindowsWebRequestSpec))
|
||||
|
||||
$web_request = Get-AnsibleWindowsWebRequest -Module $module
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
[OutputType([System.Net.WebRequest])]
|
||||
Param (
|
||||
[Alias("url")]
|
||||
[System.Uri]
|
||||
$Uri,
|
||||
|
||||
[Alias("url_method")]
|
||||
[System.String]
|
||||
$UrlMethod,
|
||||
|
||||
[Alias("follow_redirects")]
|
||||
[ValidateSet("all", "none", "safe")]
|
||||
[System.String]
|
||||
$FollowRedirects = "safe",
|
||||
|
||||
[System.Collections.IDictionary]
|
||||
$Headers,
|
||||
|
||||
[Alias("http_agent")]
|
||||
[System.String]
|
||||
$HttpAgent = "ansible-httpget",
|
||||
|
||||
[Alias("maximum_redirection")]
|
||||
[System.Int32]
|
||||
$MaximumRedirection = 50,
|
||||
|
||||
[Alias("url_timeout")]
|
||||
[System.Int32]
|
||||
$UrlTimeout = 30,
|
||||
|
||||
[Alias("validate_certs")]
|
||||
[System.Boolean]
|
||||
$ValidateCerts = $true,
|
||||
|
||||
# Credential params
|
||||
[Alias("client_cert")]
|
||||
[System.String]
|
||||
$ClientCert,
|
||||
|
||||
[Alias("client_cert_password")]
|
||||
[System.String]
|
||||
$ClientCertPassword,
|
||||
|
||||
[Alias("force_basic_auth")]
|
||||
[Switch]
|
||||
$ForceBasicAuth,
|
||||
|
||||
[Alias("url_username")]
|
||||
[System.String]
|
||||
$UrlUsername,
|
||||
|
||||
[Alias("url_password")]
|
||||
[System.String]
|
||||
$UrlPassword,
|
||||
|
||||
[Alias("use_default_credential")]
|
||||
[Switch]
|
||||
$UseDefaultCredential,
|
||||
|
||||
# Proxy params
|
||||
[Alias("use_proxy")]
|
||||
[System.Boolean]
|
||||
$UseProxy = $true,
|
||||
|
||||
[Alias("proxy_url")]
|
||||
[System.String]
|
||||
$ProxyUrl,
|
||||
|
||||
[Alias("proxy_username")]
|
||||
[System.String]
|
||||
$ProxyUsername,
|
||||
|
||||
[Alias("proxy_password")]
|
||||
[System.String]
|
||||
$ProxyPassword,
|
||||
|
||||
[Alias("proxy_use_default_credential")]
|
||||
[Switch]
|
||||
$ProxyUseDefaultCredential,
|
||||
|
||||
[ValidateScript({ $_.GetType().FullName -eq 'Ansible.Basic.AnsibleModule' })]
|
||||
[System.Object]
|
||||
$Module
|
||||
)
|
||||
|
||||
# Set module options for parameters unless they were explicitly passed in.
|
||||
if ($Module) {
|
||||
foreach ($param in $PSCmdlet.MyInvocation.MyCommand.Parameters.GetEnumerator()) {
|
||||
if ($PSBoundParameters.ContainsKey($param.Key)) {
|
||||
# Was set explicitly we want to use that value
|
||||
continue
|
||||
}
|
||||
|
||||
foreach ($alias in @($Param.Key) + $param.Value.Aliases) {
|
||||
if ($Module.Params.ContainsKey($alias)) {
|
||||
$var_value = $Module.Params.$alias -as $param.Value.ParameterType
|
||||
Set-Variable -Name $param.Key -Value $var_value
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Disable certificate validation if requested
|
||||
# FUTURE: set this on ServerCertificateValidationCallback of the HttpWebRequest once .NET 4.5 is the minimum
|
||||
if (-not $ValidateCerts) {
|
||||
[System.Net.ServicePointManager]::ServerCertificateValidationCallback = { $true }
|
||||
}
|
||||
|
||||
# Enable TLS1.1/TLS1.2 if they're available but disabled (eg. .NET 4.5)
|
||||
$security_protocols = [System.Net.ServicePointManager]::SecurityProtocol -bor [System.Net.SecurityProtocolType]::SystemDefault
|
||||
if ([System.Net.SecurityProtocolType].GetMember("Tls11").Count -gt 0) {
|
||||
$security_protocols = $security_protocols -bor [System.Net.SecurityProtocolType]::Tls11
|
||||
}
|
||||
if ([System.Net.SecurityProtocolType].GetMember("Tls12").Count -gt 0) {
|
||||
$security_protocols = $security_protocols -bor [System.Net.SecurityProtocolType]::Tls12
|
||||
}
|
||||
[System.Net.ServicePointManager]::SecurityProtocol = $security_protocols
|
||||
|
||||
$web_request = [System.Net.WebRequest]::Create($Uri)
|
||||
if ($UrlMethod) {
|
||||
$web_request.Method = $UrlMethod
|
||||
}
|
||||
$web_request.Timeout = $UrlTimeout * 1000
|
||||
|
||||
if ($UseDefaultCredential -and $web_request -is [System.Net.HttpWebRequest]) {
|
||||
$web_request.UseDefaultCredentials = $true
|
||||
} elseif ($UrlUsername) {
|
||||
if ($ForceBasicAuth) {
|
||||
$auth_value = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $UrlUsername, $UrlPassword)))
|
||||
$web_request.Headers.Add("Authorization", "Basic $auth_value")
|
||||
} else {
|
||||
$credential = New-Object -TypeName System.Net.NetworkCredential -ArgumentList $UrlUsername, $UrlPassword
|
||||
$web_request.Credentials = $credential
|
||||
}
|
||||
}
|
||||
|
||||
if ($ClientCert) {
|
||||
# Expecting either a filepath or PSPath (Cert:\CurrentUser\My\<thumbprint>)
|
||||
$cert = Get-Item -LiteralPath $ClientCert -ErrorAction SilentlyContinue
|
||||
if ($null -eq $cert) {
|
||||
Write-Error -Message "Client certificate '$ClientCert' does not exist" -Category ObjectNotFound
|
||||
return
|
||||
}
|
||||
|
||||
$crypto_ns = 'System.Security.Cryptography.X509Certificates'
|
||||
if ($cert.PSProvider.Name -ne 'Certificate') {
|
||||
try {
|
||||
$cert = New-Object -TypeName "$crypto_ns.X509Certificate2" -ArgumentList @(
|
||||
$ClientCert, $ClientCertPassword
|
||||
)
|
||||
} catch [System.Security.Cryptography.CryptographicException] {
|
||||
Write-Error -Message "Failed to read client certificate at '$ClientCert'" -Exception $_.Exception -Category SecurityError
|
||||
return
|
||||
}
|
||||
}
|
||||
$web_request.ClientCertificates = New-Object -TypeName "$crypto_ns.X509Certificate2Collection" -ArgumentList @(
|
||||
$cert
|
||||
)
|
||||
}
|
||||
|
||||
if (-not $UseProxy) {
|
||||
$proxy = $null
|
||||
} elseif ($ProxyUrl) {
|
||||
$proxy = New-Object -TypeName System.Net.WebProxy -ArgumentList $ProxyUrl, $true
|
||||
} else {
|
||||
$proxy = $web_request.Proxy
|
||||
}
|
||||
|
||||
# $web_request.Proxy may return $null for a FTP web request. We only set the credentials if we have an actual
|
||||
# proxy to work with, otherwise just ignore the credentials property.
|
||||
if ($null -ne $proxy) {
|
||||
if ($ProxyUseDefaultCredential) {
|
||||
# Weird hack, $web_request.Proxy returns an IWebProxy object which only gurantees the Credentials
|
||||
# property. We cannot set UseDefaultCredentials so we just set the Credentials to the
|
||||
# DefaultCredentials in the CredentialCache which does the same thing.
|
||||
$proxy.Credentials = [System.Net.CredentialCache]::DefaultCredentials
|
||||
} elseif ($ProxyUsername) {
|
||||
$proxy.Credentials = New-Object -TypeName System.Net.NetworkCredential -ArgumentList @(
|
||||
$ProxyUsername, $ProxyPassword
|
||||
)
|
||||
} else {
|
||||
$proxy.Credentials = $null
|
||||
}
|
||||
}
|
||||
|
||||
$web_request.Proxy = $proxy
|
||||
|
||||
# Some parameters only apply when dealing with a HttpWebRequest
|
||||
if ($web_request -is [System.Net.HttpWebRequest]) {
|
||||
if ($Headers) {
|
||||
foreach ($header in $Headers.GetEnumerator()) {
|
||||
switch ($header.Key) {
|
||||
Accept { $web_request.Accept = $header.Value }
|
||||
Connection { $web_request.Connection = $header.Value }
|
||||
Content-Length { $web_request.ContentLength = $header.Value }
|
||||
Content-Type { $web_request.ContentType = $header.Value }
|
||||
Expect { $web_request.Expect = $header.Value }
|
||||
Date { $web_request.Date = $header.Value }
|
||||
Host { $web_request.Host = $header.Value }
|
||||
If-Modified-Since { $web_request.IfModifiedSince = $header.Value }
|
||||
Range { $web_request.AddRange($header.Value) }
|
||||
Referer { $web_request.Referer = $header.Value }
|
||||
Transfer-Encoding {
|
||||
$web_request.SendChunked = $true
|
||||
$web_request.TransferEncoding = $header.Value
|
||||
}
|
||||
User-Agent { continue }
|
||||
default { $web_request.Headers.Add($header.Key, $header.Value) }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# For backwards compatibility we need to support setting the User-Agent if the header was set in the task.
|
||||
# We just need to make sure that if an explicit http_agent module was set then that takes priority.
|
||||
if ($Headers -and $Headers.ContainsKey("User-Agent")) {
|
||||
$options = (Get-AnsibleWindowsWebRequestSpec).options
|
||||
if ($HttpAgent -eq $options.http_agent.default) {
|
||||
$HttpAgent = $Headers['User-Agent']
|
||||
} elseif ($null -ne $Module) {
|
||||
$Module.Warn("The 'User-Agent' header and the 'http_agent' was set, using the 'http_agent' for web request")
|
||||
}
|
||||
}
|
||||
$web_request.UserAgent = $HttpAgent
|
||||
|
||||
switch ($FollowRedirects) {
|
||||
none { $web_request.AllowAutoRedirect = $false }
|
||||
safe {
|
||||
if ($web_request.Method -in @("GET", "HEAD")) {
|
||||
$web_request.AllowAutoRedirect = $true
|
||||
} else {
|
||||
$web_request.AllowAutoRedirect = $false
|
||||
}
|
||||
}
|
||||
all { $web_request.AllowAutoRedirect = $true }
|
||||
}
|
||||
|
||||
if ($MaximumRedirection -eq 0) {
|
||||
$web_request.AllowAutoRedirect = $false
|
||||
} else {
|
||||
$web_request.MaximumAutomaticRedirections = $MaximumRedirection
|
||||
}
|
||||
}
|
||||
|
||||
return $web_request
|
||||
}
|
||||
|
||||
Function Invoke-AnsibleWindowsWebRequest {
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Invokes a ScriptBlock with the WebRequest.
|
||||
|
||||
.DESCRIPTION
|
||||
Invokes the ScriptBlock and handle extra information like accessing the response stream, closing those streams
|
||||
safely as well as setting common module return values.
|
||||
|
||||
.PARAMETER Module
|
||||
The Ansible.Basic module to set the return values for. This will set the following return values;
|
||||
elapsed - The total time, in seconds, that it took to send the web request and process the response
|
||||
msg - The human readable description of the response status code
|
||||
status_code - An int that is the response status code
|
||||
|
||||
.PARAMETER Request
|
||||
The System.Net.WebRequest to call. This can either be manually crafted or created with
|
||||
Get-AnsibleWindowsWebRequest.
|
||||
|
||||
.PARAMETER Script
|
||||
The ScriptBlock to invoke during the web request. This ScriptBlock should take in the params
|
||||
Param ([System.Net.WebResponse]$Response, [System.IO.Stream]$Stream)
|
||||
|
||||
This scriptblock should manage the response based on what it need to do.
|
||||
|
||||
.PARAMETER Body
|
||||
An optional Stream to send to the target during the request.
|
||||
|
||||
.PARAMETER IgnoreBadResponse
|
||||
By default a WebException will be raised for a non 2xx status code and the Script will not be invoked. This
|
||||
parameter can be set to process all responses regardless of the status code.
|
||||
|
||||
.EXAMPLE Basic module that downloads a file
|
||||
$spec = @{
|
||||
options = @{
|
||||
path = @{ type = "path"; required = $true }
|
||||
}
|
||||
}
|
||||
$module = Ansible.Basic.AnsibleModule]::Create($args, $spec, @(Get-AnsibleWindowsWebRequestSpec))
|
||||
|
||||
$web_request = Get-AnsibleWindowsWebRequest -Module $module
|
||||
|
||||
Invoke-AnsibleWindowsWebRequest -Module $module -Request $web_request -Script {
|
||||
Param ([System.Net.WebResponse]$Response, [System.IO.Stream]$Stream)
|
||||
|
||||
$fs = [System.IO.File]::Create($module.Params.path)
|
||||
try {
|
||||
$Stream.CopyTo($fs)
|
||||
$fs.Flush()
|
||||
} finally {
|
||||
$fs.Dispose()
|
||||
}
|
||||
}
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param (
|
||||
[Parameter(Mandatory=$true)]
|
||||
[System.Object]
|
||||
[ValidateScript({ $_.GetType().FullName -eq 'Ansible.Basic.AnsibleModule' })]
|
||||
$Module,
|
||||
|
||||
[Parameter(Mandatory=$true)]
|
||||
[System.Net.WebRequest]
|
||||
$Request,
|
||||
|
||||
[Parameter(Mandatory=$true)]
|
||||
[ScriptBlock]
|
||||
$Script,
|
||||
|
||||
[AllowNull()]
|
||||
[System.IO.Stream]
|
||||
$Body,
|
||||
|
||||
[Switch]
|
||||
$IgnoreBadResponse
|
||||
)
|
||||
|
||||
$start = Get-Date
|
||||
if ($null -ne $Body) {
|
||||
$request_st = $Request.GetRequestStream()
|
||||
try {
|
||||
$Body.CopyTo($request_st)
|
||||
$request_st.Flush()
|
||||
} finally {
|
||||
$request_st.Close()
|
||||
}
|
||||
}
|
||||
|
||||
try {
|
||||
try {
|
||||
$web_response = $Request.GetResponse()
|
||||
} catch [System.Net.WebException] {
|
||||
# A WebResponse with a status code not in the 200 range will raise a WebException. We check if the
|
||||
# exception raised contains the actual response and continue on if IgnoreBadResponse is set. We also
|
||||
# make sure we set the status_code return value on the Module object if possible
|
||||
|
||||
if ($_.Exception.PSObject.Properties.Name -match "Response") {
|
||||
$web_response = $_.Exception.Response
|
||||
|
||||
if (-not $IgnoreBadResponse -or $null -eq $web_response) {
|
||||
$Module.Result.msg = $_.Exception.StatusDescription
|
||||
$Module.Result.status_code = $_.Exception.Response.StatusCode
|
||||
throw $_
|
||||
}
|
||||
} else {
|
||||
throw $_
|
||||
}
|
||||
}
|
||||
|
||||
if ($Request.RequestUri.IsFile) {
|
||||
# A FileWebResponse won't have these properties set
|
||||
$Module.Result.msg = "OK"
|
||||
$Module.Result.status_code = 200
|
||||
} else {
|
||||
$Module.Result.msg = $web_response.StatusDescription
|
||||
$Module.Result.status_code = $web_response.StatusCode
|
||||
}
|
||||
|
||||
$response_stream = $web_response.GetResponseStream()
|
||||
try {
|
||||
# Invoke the ScriptBlock and pass in WebResponse and ResponseStream
|
||||
&$Script -Response $web_response -Stream $response_stream
|
||||
} finally {
|
||||
$response_stream.Dispose()
|
||||
}
|
||||
} finally {
|
||||
if ($web_response) {
|
||||
$web_response.Close()
|
||||
}
|
||||
$Module.Result.elapsed = ((Get-date) - $start).TotalSeconds
|
||||
}
|
||||
}
|
||||
|
||||
Function Get-AnsibleWindowsWebRequestSpec {
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Used by modules to get the argument spec fragment for AnsibleModule.
|
||||
|
||||
.EXAMPLES
|
||||
$spec = @{
|
||||
options = @{}
|
||||
}
|
||||
$module = [Ansible.Basic.AnsibleModule]::Create($args, $spec, @(Get-AnsibleWindowsWebRequestSpec))
|
||||
|
||||
.NOTES
|
||||
The options here are reflected in the doc fragment 'ansible.windows.web_request' at
|
||||
'plugins/doc_fragments/web_request.py'.
|
||||
#>
|
||||
@{
|
||||
options = @{
|
||||
url_method = @{ type = 'str' }
|
||||
follow_redirects = @{ type = 'str'; choices = @('all', 'none', 'safe'); default = 'safe' }
|
||||
headers = @{ type = 'dict' }
|
||||
http_agent = @{ type = 'str'; default = 'ansible-httpget' }
|
||||
maximum_redirection = @{ type = 'int'; default = 50 }
|
||||
url_timeout = @{ type = 'int'; default = 30 }
|
||||
validate_certs = @{ type = 'bool'; default = $true }
|
||||
|
||||
# Credential options
|
||||
client_cert = @{ type = 'str' }
|
||||
client_cert_password = @{ type = 'str'; no_log = $true }
|
||||
force_basic_auth = @{ type = 'bool'; default = $false }
|
||||
url_username = @{ type = 'str' }
|
||||
url_password = @{ type = 'str'; no_log = $true }
|
||||
use_default_credential = @{ type = 'bool'; default = $false }
|
||||
|
||||
# Proxy options
|
||||
use_proxy = @{ type = 'bool'; default = $true }
|
||||
proxy_url = @{ type = 'str' }
|
||||
proxy_username = @{ type = 'str' }
|
||||
proxy_password = @{ type = 'str'; no_log = $true }
|
||||
proxy_use_default_credential = @{ type = 'bool'; default = $false }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
$export_members = @{
|
||||
Function = "Get-AnsibleWindowsWebRequest", "Get-AnsibleWindowsWebRequestSpec", "Invoke-AnsibleWindowsWebRequest"
|
||||
}
|
||||
Export-ModuleMember @export_members
|
||||
@ -0,0 +1,219 @@
|
||||
#!powershell
|
||||
|
||||
# Copyright: (c) 2015, Corwin Brown <corwin@corwinbrown.com>
|
||||
# Copyright: (c) 2017, Dag Wieers (@dagwieers) <dag@wieers.com>
|
||||
# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
|
||||
|
||||
#AnsibleRequires -CSharpUtil Ansible.Basic
|
||||
#Requires -Module Ansible.ModuleUtils.CamelConversion
|
||||
#Requires -Module Ansible.ModuleUtils.FileUtil
|
||||
#Requires -Module Ansible.ModuleUtils.Legacy
|
||||
#AnsibleRequires -PowerShell ..module_utils.WebRequest
|
||||
|
||||
$spec = @{
|
||||
options = @{
|
||||
url = @{ type = "str"; required = $true }
|
||||
content_type = @{ type = "str" }
|
||||
body = @{ type = "raw" }
|
||||
dest = @{ type = "path" }
|
||||
creates = @{ type = "path" }
|
||||
removes = @{ type = "path" }
|
||||
return_content = @{ type = "bool"; default = $false }
|
||||
status_code = @{ type = "list"; elements = "int"; default = @(200) }
|
||||
|
||||
# Defined for ease of use and backwards compatibility
|
||||
url_timeout = @{
|
||||
aliases = "timeout"
|
||||
}
|
||||
url_method = @{
|
||||
aliases = "method"
|
||||
default = "GET"
|
||||
}
|
||||
|
||||
# Defined for the alias backwards compatibility, remove once aliases are removed
|
||||
url_username = @{
|
||||
aliases = @("user", "username")
|
||||
deprecated_aliases = @(
|
||||
@{ name = "user"; date = [DateTime]::ParseExact("2022-07-01", "yyyy-MM-dd", $null); collection_name = 'ansible.windows' },
|
||||
@{ name = "username"; date = [DateTime]::ParseExact("2022-07-01", "yyyy-MM-dd", $null); collection_name = 'ansible.windows' }
|
||||
)
|
||||
}
|
||||
url_password = @{
|
||||
aliases = @("password")
|
||||
deprecated_aliases = @(
|
||||
@{ name = "password"; date = [DateTime]::ParseExact("2022-07-01", "yyyy-MM-dd", $null); collection_name = 'ansible.windows' }
|
||||
)
|
||||
}
|
||||
}
|
||||
supports_check_mode = $true
|
||||
}
|
||||
$module = [Ansible.Basic.AnsibleModule]::Create($args, $spec, @(Get-AnsibleWindowsWebRequestSpec))
|
||||
|
||||
$url = $module.Params.url
|
||||
$method = $module.Params.url_method.ToUpper()
|
||||
$content_type = $module.Params.content_type
|
||||
$body = $module.Params.body
|
||||
$dest = $module.Params.dest
|
||||
$creates = $module.Params.creates
|
||||
$removes = $module.Params.removes
|
||||
$return_content = $module.Params.return_content
|
||||
$status_code = $module.Params.status_code
|
||||
|
||||
$JSON_CANDIDATES = @('text', 'json', 'javascript')
|
||||
|
||||
$module.Result.elapsed = 0
|
||||
$module.Result.url = $url
|
||||
|
||||
Function ConvertFrom-SafeJson {
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Safely convert a JSON string to an object, this is like ConvertFrom-Json except it respect -ErrorAction.
|
||||
|
||||
.PAREMTER InputObject
|
||||
The input object string to convert from.
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param (
|
||||
[Parameter(Mandatory=$true)]
|
||||
[AllowEmptyString()]
|
||||
[AllowNull()]
|
||||
[String]
|
||||
$InputObject
|
||||
)
|
||||
|
||||
if (-not $InputObject) {
|
||||
return
|
||||
}
|
||||
|
||||
try {
|
||||
# Make sure we output the actual object without unpacking with the unary comma
|
||||
,[Ansible.Basic.AnsibleModule]::FromJson($InputObject)
|
||||
} catch [System.ArgumentException] {
|
||||
Write-Error -Message "Invalid json string as input object: $($_.Exception.Message)" -Exception $_.Exception
|
||||
}
|
||||
}
|
||||
|
||||
if (-not ($method -cmatch '^[A-Z]+$')) {
|
||||
$module.FailJson("Parameter 'method' needs to be a single word in uppercase, like GET or POST.")
|
||||
}
|
||||
|
||||
if ($creates -and (Test-AnsiblePath -Path $creates)) {
|
||||
$module.Result.skipped = $true
|
||||
$module.Result.msg = "The 'creates' file or directory ($creates) already exists."
|
||||
$module.ExitJson()
|
||||
}
|
||||
|
||||
if ($removes -and -not (Test-AnsiblePath -Path $removes)) {
|
||||
$module.Result.skipped = $true
|
||||
$module.Result.msg = "The 'removes' file or directory ($removes) does not exist."
|
||||
$module.ExitJson()
|
||||
}
|
||||
|
||||
$client = Get-AnsibleWindowsWebRequest -Uri $url -Module $module
|
||||
|
||||
if ($null -ne $content_type) {
|
||||
$client.ContentType = $content_type
|
||||
}
|
||||
|
||||
$response_script = {
|
||||
param($Response, $Stream)
|
||||
|
||||
ForEach ($prop in $Response.PSObject.Properties) {
|
||||
$result_key = Convert-StringToSnakeCase -string $prop.Name
|
||||
$prop_value = $prop.Value
|
||||
# convert and DateTime values to ISO 8601 standard
|
||||
if ($prop_value -is [System.DateTime]) {
|
||||
$prop_value = $prop_value.ToString("o", [System.Globalization.CultureInfo]::InvariantCulture)
|
||||
}
|
||||
$module.Result.$result_key = $prop_value
|
||||
}
|
||||
|
||||
# manually get the headers as not all of them are in the response properties
|
||||
foreach ($header_key in $Response.Headers.GetEnumerator()) {
|
||||
$header_value = $Response.Headers[$header_key]
|
||||
$header_key = $header_key.Replace("-", "") # replace - with _ for snake conversion
|
||||
$header_key = Convert-StringToSnakeCase -string $header_key
|
||||
$module.Result.$header_key = $header_value
|
||||
}
|
||||
|
||||
# we only care about the return body if we need to return the content or create a file
|
||||
if ($return_content -or $dest) {
|
||||
# copy to a MemoryStream so we can read it multiple times
|
||||
$memory_st = New-Object -TypeName System.IO.MemoryStream
|
||||
try {
|
||||
$Stream.CopyTo($memory_st)
|
||||
|
||||
if ($return_content) {
|
||||
$memory_st.Seek(0, [System.IO.SeekOrigin]::Begin) > $null
|
||||
$content_bytes = $memory_st.ToArray()
|
||||
$module.Result.content = [System.Text.Encoding]::UTF8.GetString($content_bytes)
|
||||
if ($module.Result.ContainsKey("content_type") -and $module.Result.content_type -Match ($JSON_CANDIDATES -join '|')) {
|
||||
$json = ConvertFrom-SafeJson -InputObject $module.Result.content -ErrorAction SilentlyContinue
|
||||
if ($json) {
|
||||
$module.Result.json = $json
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if ($dest) {
|
||||
$memory_st.Seek(0, [System.IO.SeekOrigin]::Begin) > $null
|
||||
$changed = $true
|
||||
|
||||
if (Test-AnsiblePath -Path $dest) {
|
||||
$actual_checksum = Get-FileChecksum -path $dest -algorithm "sha1"
|
||||
|
||||
$sp = New-Object -TypeName System.Security.Cryptography.SHA1CryptoServiceProvider
|
||||
$content_checksum = [System.BitConverter]::ToString($sp.ComputeHash($memory_st)).Replace("-", "").ToLower()
|
||||
|
||||
if ($actual_checksum -eq $content_checksum) {
|
||||
$changed = $false
|
||||
}
|
||||
}
|
||||
|
||||
$module.Result.changed = $changed
|
||||
if ($changed -and (-not $module.CheckMode)) {
|
||||
$memory_st.Seek(0, [System.IO.SeekOrigin]::Begin) > $null
|
||||
$file_stream = [System.IO.File]::Create($dest)
|
||||
try {
|
||||
$memory_st.CopyTo($file_stream)
|
||||
} finally {
|
||||
$file_stream.Flush()
|
||||
$file_stream.Close()
|
||||
}
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
$memory_st.Close()
|
||||
}
|
||||
}
|
||||
|
||||
if ($status_code -notcontains $Response.StatusCode) {
|
||||
$module.FailJson("Status code of request '$([int]$Response.StatusCode)' is not in list of valid status codes $status_code : $($Response.StatusCode)'.")
|
||||
}
|
||||
}
|
||||
|
||||
$body_st = $null
|
||||
if ($null -ne $body) {
|
||||
if ($body -is [System.Collections.IDictionary] -or $body -is [System.Collections.IList]) {
|
||||
$body_string = ConvertTo-Json -InputObject $body -Compress
|
||||
} elseif ($body -isnot [String]) {
|
||||
$body_string = $body.ToString()
|
||||
} else {
|
||||
$body_string = $body
|
||||
}
|
||||
$buffer = [System.Text.Encoding]::UTF8.GetBytes($body_string)
|
||||
|
||||
$body_st = New-Object -TypeName System.IO.MemoryStream -ArgumentList @(,$buffer)
|
||||
}
|
||||
|
||||
try {
|
||||
Invoke-AnsibleWindowsWebRequest -Module $module -Request $client -Script $response_script -Body $body_st -IgnoreBadResponse
|
||||
} catch {
|
||||
$module.FailJson("Unhandled exception occurred when sending web request. Exception: $($_.Exception.Message)", $_)
|
||||
} finally {
|
||||
if ($null -ne $body_st) {
|
||||
$body_st.Dispose()
|
||||
}
|
||||
}
|
||||
|
||||
$module.ExitJson()
|
||||
@ -0,0 +1,155 @@
|
||||
#!/usr/bin/python
|
||||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright: (c) 2015, Corwin Brown <corwin@corwinbrown.com>
|
||||
# Copyright: (c) 2017, Dag Wieers (@dagwieers) <dag@wieers.com>
|
||||
# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
|
||||
|
||||
DOCUMENTATION = r'''
|
||||
---
|
||||
module: win_uri
|
||||
short_description: Interacts with webservices
|
||||
description:
|
||||
- Interacts with FTP, HTTP and HTTPS web services.
|
||||
- Supports Digest, Basic and WSSE HTTP authentication mechanisms.
|
||||
- For non-Windows targets, use the M(ansible.builtin.uri) module instead.
|
||||
options:
|
||||
url:
|
||||
description:
|
||||
- Supports FTP, HTTP or HTTPS URLs in the form of (ftp|http|https)://host.domain:port/path.
|
||||
type: str
|
||||
required: yes
|
||||
content_type:
|
||||
description:
|
||||
- Sets the "Content-Type" header.
|
||||
type: str
|
||||
body:
|
||||
description:
|
||||
- The body of the HTTP request/response to the web service.
|
||||
type: raw
|
||||
dest:
|
||||
description:
|
||||
- Output the response body to a file.
|
||||
type: path
|
||||
creates:
|
||||
description:
|
||||
- A filename, when it already exists, this step will be skipped.
|
||||
type: path
|
||||
removes:
|
||||
description:
|
||||
- A filename, when it does not exist, this step will be skipped.
|
||||
type: path
|
||||
return_content:
|
||||
description:
|
||||
- Whether or not to return the body of the response as a "content" key in
|
||||
the dictionary result. If the reported Content-type is
|
||||
"application/json", then the JSON is additionally loaded into a key
|
||||
called C(json) in the dictionary results.
|
||||
type: bool
|
||||
default: no
|
||||
status_code:
|
||||
description:
|
||||
- A valid, numeric, HTTP status code that signifies success of the request.
|
||||
- Can also be comma separated list of status codes.
|
||||
type: list
|
||||
elements: int
|
||||
default: [ 200 ]
|
||||
|
||||
url_method:
|
||||
default: GET
|
||||
aliases:
|
||||
- method
|
||||
url_timeout:
|
||||
aliases:
|
||||
- timeout
|
||||
|
||||
# Following defined in the web_request fragment but the module contains deprecated aliases for backwards compatibility.
|
||||
url_username:
|
||||
description:
|
||||
- The username to use for authentication.
|
||||
- The alias I(user) and I(username) is deprecated and will be removed on
|
||||
the major release after C(2022-07-01).
|
||||
aliases:
|
||||
- user
|
||||
- username
|
||||
url_password:
|
||||
description:
|
||||
- The password for I(url_username).
|
||||
- The alias I(password) is deprecated and will be removed on the major
|
||||
release after C(2022-07-01).
|
||||
aliases:
|
||||
- password
|
||||
extends_documentation_fragment:
|
||||
- ansible.windows.web_request
|
||||
|
||||
seealso:
|
||||
- module: ansible.builtin.uri
|
||||
- module: ansible.windows.win_get_url
|
||||
author:
|
||||
- Corwin Brown (@blakfeld)
|
||||
- Dag Wieers (@dagwieers)
|
||||
'''
|
||||
|
||||
EXAMPLES = r'''
|
||||
- name: Perform a GET and Store Output
|
||||
ansible.windows.win_uri:
|
||||
url: http://example.com/endpoint
|
||||
register: http_output
|
||||
|
||||
# Set a HOST header to hit an internal webserver:
|
||||
- name: Hit a Specific Host on the Server
|
||||
ansible.windows.win_uri:
|
||||
url: http://example.com/
|
||||
method: GET
|
||||
headers:
|
||||
host: www.somesite.com
|
||||
|
||||
- name: Perform a HEAD on an Endpoint
|
||||
ansible.windows.win_uri:
|
||||
url: http://www.example.com/
|
||||
method: HEAD
|
||||
|
||||
- name: POST a Body to an Endpoint
|
||||
ansible.windows.win_uri:
|
||||
url: http://www.somesite.com/
|
||||
method: POST
|
||||
body: "{ 'some': 'json' }"
|
||||
'''
|
||||
|
||||
RETURN = r'''
|
||||
elapsed:
|
||||
description: The number of seconds that elapsed while performing the download.
|
||||
returned: always
|
||||
type: float
|
||||
sample: 23.2
|
||||
url:
|
||||
description: The Target URL.
|
||||
returned: always
|
||||
type: str
|
||||
sample: https://www.ansible.com
|
||||
status_code:
|
||||
description: The HTTP Status Code of the response.
|
||||
returned: success
|
||||
type: int
|
||||
sample: 200
|
||||
status_description:
|
||||
description: A summary of the status.
|
||||
returned: success
|
||||
type: str
|
||||
sample: OK
|
||||
content:
|
||||
description: The raw content of the HTTP response.
|
||||
returned: success and return_content is True
|
||||
type: str
|
||||
sample: '{"foo": "bar"}'
|
||||
content_length:
|
||||
description: The byte size of the response.
|
||||
returned: success
|
||||
type: int
|
||||
sample: 54447
|
||||
json:
|
||||
description: The json structure returned under content as a dictionary.
|
||||
returned: success and Content-Type is "application/json" or "application/javascript" and return_content is True
|
||||
type: dict
|
||||
sample: {"this-is-dependent": "on the actual return content"}
|
||||
'''
|
||||
@ -1,131 +0,0 @@
|
||||
# This file is part of Ansible
|
||||
# -*- coding: utf-8 -*-
|
||||
#
|
||||
#
|
||||
# Ansible is free software: you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation, either version 3 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# Ansible is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License
|
||||
# along with Ansible. If not, see <http://www.gnu.org/licenses/>.
|
||||
#
|
||||
|
||||
# Make coding more python3-ish
|
||||
from __future__ import (absolute_import, division, print_function)
|
||||
__metaclass__ = type
|
||||
|
||||
import os
|
||||
|
||||
import pytest
|
||||
from units.compat.mock import call, patch, MagicMock
|
||||
|
||||
# docker images quay.io/ansible/centos7-test-container --format '{{json .}}'
|
||||
DOCKER_OUTPUT_MULTIPLE = """
|
||||
{"Containers":"N/A","CreatedAt":"2020-06-11 17:05:58 -0500 CDT","CreatedSince":"3 months ago","Digest":"\u003cnone\u003e","ID":"b0f914b26cc1","Repository":"quay.io/ansible/centos7-test-container","SharedSize":"N/A","Size":"556MB","Tag":"1.17.0","UniqueSize":"N/A","VirtualSize":"555.6MB"}
|
||||
{"Containers":"N/A","CreatedAt":"2020-06-11 17:05:58 -0500 CDT","CreatedSince":"3 months ago","Digest":"\u003cnone\u003e","ID":"b0f914b26cc1","Repository":"quay.io/ansible/centos7-test-container","SharedSize":"N/A","Size":"556MB","Tag":"latest","UniqueSize":"N/A","VirtualSize":"555.6MB"}
|
||||
{"Containers":"N/A","CreatedAt":"2019-04-01 19:59:39 -0500 CDT","CreatedSince":"18 months ago","Digest":"\u003cnone\u003e","ID":"dd3d10e03dd3","Repository":"quay.io/ansible/centos7-test-container","SharedSize":"N/A","Size":"678MB","Tag":"1.8.0","UniqueSize":"N/A","VirtualSize":"678MB"}
|
||||
""".lstrip() # noqa: E501
|
||||
|
||||
PODMAN_OUTPUT = """
|
||||
[
|
||||
{
|
||||
"id": "dd3d10e03dd3580de865560c3440c812a33fd7a1fca8ed8e4a1219ff3d809e3a",
|
||||
"names": [
|
||||
"quay.io/ansible/centos7-test-container:1.8.0"
|
||||
],
|
||||
"digest": "sha256:6e5d9c99aa558779715a80715e5cf0c227a4b59d95e6803c148290c5d0d9d352",
|
||||
"created": "2019-04-02T00:59:39.234584184Z",
|
||||
"size": 702761933
|
||||
},
|
||||
{
|
||||
"id": "b0f914b26cc1088ab8705413c2f2cf247306ceeea51260d64c26894190d188bd",
|
||||
"names": [
|
||||
"quay.io/ansible/centos7-test-container:latest"
|
||||
],
|
||||
"digest": "sha256:d8431aa74f60f4ff0f1bd36bc9a227bbb2066330acd8bf25e29d8614ee99e39c",
|
||||
"created": "2020-06-11T22:05:58.382459136Z",
|
||||
"size": 578513505
|
||||
}
|
||||
]
|
||||
""".lstrip()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def docker_images():
|
||||
from ansible_test._internal.docker_util import docker_images
|
||||
return docker_images
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def ansible_test(ansible_test):
|
||||
import ansible_test
|
||||
return ansible_test
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def subprocess_error():
|
||||
from ansible_test._internal.util import SubprocessError
|
||||
return SubprocessError
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
('returned_items_count', 'patched_dc_stdout'),
|
||||
(
|
||||
(3, (DOCKER_OUTPUT_MULTIPLE, '')),
|
||||
(2, (PODMAN_OUTPUT, '')),
|
||||
(0, ('', '')),
|
||||
),
|
||||
ids=('docker JSONL', 'podman JSON sequence', 'empty output'))
|
||||
def test_docker_images(docker_images, mocker, returned_items_count, patched_dc_stdout):
|
||||
mocker.patch(
|
||||
'ansible_test._internal.docker_util.docker_command',
|
||||
return_value=patched_dc_stdout)
|
||||
ret = docker_images('', 'quay.io/ansible/centos7-test-container')
|
||||
assert len(ret) == returned_items_count
|
||||
|
||||
|
||||
def test_podman_fallback(ansible_test, docker_images, subprocess_error, mocker):
|
||||
'''Test podman >2 && <2.2 fallback'''
|
||||
|
||||
cmd = ['docker', 'images', 'quay.io/ansible/centos7-test-container', '--format', '{{json .}}']
|
||||
docker_command_results = [
|
||||
subprocess_error(cmd, status=1, stderr='function "json" not defined'),
|
||||
(PODMAN_OUTPUT, ''),
|
||||
]
|
||||
mocker.patch(
|
||||
'ansible_test._internal.docker_util.docker_command',
|
||||
side_effect=docker_command_results)
|
||||
|
||||
ret = docker_images('', 'quay.io/ansible/centos7-test-container')
|
||||
calls = [
|
||||
call(
|
||||
'',
|
||||
['images', 'quay.io/ansible/centos7-test-container', '--format', '{{json .}}'],
|
||||
capture=True,
|
||||
always=True),
|
||||
call(
|
||||
'',
|
||||
['images', 'quay.io/ansible/centos7-test-container', '--format', 'json'],
|
||||
capture=True,
|
||||
always=True),
|
||||
]
|
||||
ansible_test._internal.docker_util.docker_command.assert_has_calls(calls)
|
||||
assert len(ret) == 2
|
||||
|
||||
|
||||
def test_podman_no_such_image(ansible_test, docker_images, subprocess_error, mocker):
|
||||
'''Test podman "no such image" error'''
|
||||
|
||||
cmd = ['docker', 'images', 'quay.io/ansible/centos7-test-container', '--format', '{{json .}}']
|
||||
exc = subprocess_error(cmd, status=1, stderr='no such image'),
|
||||
mocker.patch(
|
||||
'ansible_test._internal.docker_util.docker_command',
|
||||
side_effect=exc)
|
||||
ret = docker_images('', 'quay.io/ansible/centos7-test-container')
|
||||
assert ret == []
|
||||
Loading…
Reference in New Issue