ansible: fix put_file() for large temporary files.

Reverts 49736b3a, large file copies can't avoid the RTT.

The parent stack must be blocked while FileService progresses, as unlike
the small file path, it does not make a snapshot of the (possibly
temporary) file passed by the action plug-in. So we need to keep that
file alive while the service runs.

Add a new integration test and a new soak test to cover both.
pull/372/head
David Wilson 6 years ago
parent 530fd18e4c
commit 638b196a45

@ -880,13 +880,11 @@ class Connection(ansible.plugins.connection.ConnectionBase):
path=mitogen.utils.cast(in_path)
)
# A roundtrip is always necessary for the target to request the file
# from FileService, however, by pipelining the transfer function, the
# subsequent step (probably a module invocation) can get its
# dependencies and function call in-flight before the transfer is
# complete. This saves at least 1 RTT between the transfer completing
# and the start of the follow-up task.
self.get_chain().call_no_reply(
# For now this must remain synchronous, as the action plug-in may have
# passed us a temporary file to transfer. A future FileService could
# maintain an LRU list of open file descriptors to keep the temporary
# file alive, but that requires more work.
self.get_chain().call(
ansible_mitogen.target.transfer_file,
context=self.parent,
in_path=in_path,

@ -53,11 +53,6 @@ Enhancements
a 250 ms link from 30 seconds to 10 seconds compared to v0.2.2, down from 120
seconds compared to vanilla.
* `49736b3a <https://github.com/dw/mitogen/commit/49736b3a>`_: avoid a
roundtrip when transferring files larger than 124KiB, removing a delay
between waiting for the transfer to complete and start of the follow-up
action.
* `#337 <https://github.com/dw/mitogen/issues/337>`_: To avoid a scaling
limitation, a PTY is no longer allocated for an SSH connection unless the
configuration specifies a password.

@ -1,3 +1,4 @@
- import_playbook: copy.yml
- import_playbook: fixup_perms2__copy.yml
- import_playbook: low_level_execute_command.yml
- import_playbook: make_tmp_path.yml

@ -0,0 +1,66 @@
# Verify copy module for small and large files, and inline content.
- name: integration/action/synchronize.yml
hosts: test-targets
any_errors_fatal: true
tasks:
- copy:
dest: /tmp/copy-tiny-file
content:
this is a tiny file.
connection: local
- copy:
dest: /tmp/copy-large-file
# Must be larger than Connection.SMALL_SIZE_LIMIT.
content: "{% for x in range(200000) %}x{% endfor %}"
connection: local
# end of making files
- file:
state: absent
path: "{{item}}"
with_items:
- /tmp/copy-tiny-file.out
- /tmp/copy-large-file.out
- /tmp/copy-tiny-inline-file.out
- /tmp/copy-large-inline-file.out
# end of cleaning out files
- copy:
dest: /tmp/copy-large-file.out
src: /tmp/copy-large-file
- copy:
dest: /tmp/copy-tiny-file.out
src: /tmp/copy-tiny-file
- copy:
dest: /tmp/copy-tiny-inline-file.out
content: "tiny inline content"
- copy:
dest: /tmp/copy-large-inline-file.out
content: |
{% for x in range(200000) %}y{% endfor %}
# stat results
- stat:
path: "{{item}}"
with_items:
- /tmp/copy-tiny-file.out
- /tmp/copy-large-file.out
- /tmp/copy-tiny-inline-file.out
- /tmp/copy-large-inline-file.out
register: stat
- assert:
that:
- stat.results[0].stat.checksum == "f29faa9a6f19a700a941bf2aa5b281643c4ec8a0"
- stat.results[1].stat.checksum == "62951f943c41cdd326e5ce2b53a779e7916a820d"
- stat.results[2].stat.checksum == "b26dd6444595e2bdb342aa0a91721b57478b5029"
- stat.results[3].stat.checksum == "d675f47e467eae19e49032a2cc39118e12a6ee72"

@ -0,0 +1,6 @@
- file:
path: /tmp/foo-{{inventory_hostname}}
state: absent
- copy:
dest: /tmp/foo-{{inventory_hostname}}
content: "{{content}}"

@ -0,0 +1,6 @@
- hosts: all
tasks:
- set_fact:
content: "{% for x in range(126977) %}x{% endfor %}"
- include_tasks: _file_service_loop.yml
with_sequence: start=1 end=100
Loading…
Cancel
Save