Stream image data in load_image() to avoid out-of-memory

Reading the entire tar file into memory can result in out-of-memory
conditions such as this traceback:

Traceback (most recent call last):
  File "/tmp/ansible_YELTSu/ansible_module_docker_image.py", line 486, in load_image
    self.client.load_image(image_data)
  File "/usr/local/lib/python2.7/dist-packages/docker/api/image.py", line 147, in load_image
    res = self._post(self._url("/images/load"), data=data)
  ...
  File "/usr/lib/python2.7/httplib.py", line 997, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 848, in _send_output
    msg += message_body
MemoryError

Luckily docker-py's load_image(), which calls requests post(), accepts a
file-like object instead of a string.  Pass in the file object to avoid
reading the full file into memory.  This allows larger tar files to load
succesfully.

Signed-off-by: Stefan Hajnoczi <stefanha@gmail.com>
reviewable/pr18780/r1
Stefan Hajnoczi 9 years ago
parent a8072f9ef0
commit 47cc422b8a

@ -474,19 +474,22 @@ class ImageManager(DockerBaseClass):
:return: image dict :return: image dict
''' '''
try: try:
self.log("Reading image data from %s" % self.load_path) self.log("Opening image %s" % self.load_path)
image_tar = open(self.load_path, 'r') image_tar = open(self.load_path, 'r')
image_data = image_tar.read()
image_tar.close()
except Exception as exc: except Exception as exc:
self.fail("Error reading image data %s - %s" % (self.load_path, str(exc))) self.fail("Error opening image %s - %s" % (self.load_path, str(exc)))
try: try:
self.log("Loading image from %s" % self.load_path) self.log("Loading image from %s" % self.load_path)
self.client.load_image(image_data) self.client.load_image(image_tar)
except Exception as exc: except Exception as exc:
self.fail("Error loading image %s - %s" % (self.name, str(exc))) self.fail("Error loading image %s - %s" % (self.name, str(exc)))
try:
image_tar.close()
except Exception as exc:
self.fail("Error closing image %s - %s" % (self.name, str(exc)))
return self.client.find_image(self.name, self.tag) return self.client.find_image(self.name, self.tag)

Loading…
Cancel
Save