Lazy loading images when creating a pdf

What did you do?

I'm trying to create a pdf with a lot of images, I decided to lazy load them to avoid high memory usage so I built a generator that loads the images and yields them, here is the minimal reproducible example

import os
import sys
from PIL import Image
from random import choice
import os
import psutil

process = psutil.Process(os.getpid())
rotations = [i * i for i in range(12)]
names = ["pdf%i.pdf" % i for i in range(20)]

def image_generator(image_paths, do_break=False):
    print("running generator")
    curr_image = 0
    for img_p in image_paths:
        print("mem usage in bytes: %s " % str(process.memory_info().rss))
        with open(img_p, "rb") as fp:
            with Image.open(fp) as img:
                try:
                    img.verify()
                except Exception as e:
                    print("invalid image continuing")
                    print("Exception: %s" % str(e))
                    continue
            with open(img_p, "rb") as fp:
                with Image.open(fp) as img:
                    img.load()
                    yield img

def gather_images(img_dir):
    image_paths = []
    print("gathering images")
    for root, __, files in os.walk(img_dir, topdown=False):
        for file in files:
            source_path = os.path.join(root, file)
            image_paths.append(source_path)
    return image_paths

def run_main(imgs_dir):
    image_paths = gather_images(imgs_dir)
    if len(image_paths) < 3:
        print("not enough images")
    else:
        print("starting")
        first_img = None
        for img in image_generator(image_paths):
            first_img = img
            break
        image_paths.pop(0)
        print("Images to process: %i" % len(image_paths))
        name = choice(names)
        for name in names:
            if os.path.isfile(name):
                name = choice(name)
        if os.path.isfile(name):
            print("Unable to create name")
            raise FileExistsError(name)
        else:
            first_img.save(name, "PDF", resolution=90, save_all=True,
                           append_images=image_generator(image_paths))

if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("need path argument (location of images)")
    elif not os.path.isdir(os.path.abspath(sys.argv[1])):
        print("Provided path does not exists")
    else:
        run_main(sys.argv[1])

What did you expect to happen?

I expected the memory usage to remain somewhat constant

What actually happened?

The memory usage kept rising until I had to forcefully close the program to avoid a system freeze
I have also tried to close the images after yielding them which did maintain the memory usage constant but that only raised another issue ValueError: Operation on closed image

full traceback:

Traceback (most recent call last):
  File "minimal_memory_example.py", line 70, in <module>
    run_main(sys.argv[1])
  File "minimal_memory_example.py", line 62, in run_main
    append_images=image_generator(image_paths))
  File "/home/hallowf/Documents/Github/STPDF/venv/lib/python3.7/site-packages/PIL/Image.py", line 2088, in save
    save_handler(self, fp, filename)
  File "/home/hallowf/Documents/Github/STPDF/venv/lib/python3.7/site-packages/PIL/PdfImagePlugin.py", line 45, in _save_all
    _save(im, fp, filename, save_all=True)
  File "/home/hallowf/Documents/Github/STPDF/venv/lib/python3.7/site-packages/PIL/PdfImagePlugin.py", line 174, in _save
    Image.SAVE["JPEG"](im, op, filename)
  File "/home/hallowf/Documents/Github/STPDF/venv/lib/python3.7/site-packages/PIL/JpegImagePlugin.py", line 779, in _save
    ImageFile._save(im, fp, [("jpeg", (0, 0) + im.size, 0, rawmode)], bufsize)
  File "/home/hallowf/Documents/Github/STPDF/venv/lib/python3.7/site-packages/PIL/ImageFile.py", line 485, in _save
    im.load()
  File "/home/hallowf/Documents/Github/STPDF/venv/lib/python3.7/site-packages/PIL/ImageFile.py", line 144, in load
    pixel = Image.Image.load(self)
  File "/home/hallowf/Documents/Github/STPDF/venv/lib/python3.7/site-packages/PIL/Image.py", line 879, in load
    return self.im.pixel_access(self.readonly)
  File "/home/hallowf/Documents/Github/STPDF/venv/lib/python3.7/site-packages/PIL/_util.py", line 43, in __getattr__
    raise self.ex
ValueError: Operation on closed image

What are your OS, Python and Pillow versions?

• OS: Ubuntu 18.04 LTS
• Python: python 3.7
• Pillow: 6.1.0

[radarhere]

Here is an idea that still sees memory usage increase, but does so at half the rate - rather than using 'append_images', you can use 'append' - https://pillow.readthedocs.io/en/stable/handbook/image-file-formats.html#pdf

first_img.save(name, "PDF", resolution=90)
for img in image_generator(image_paths):
    img.save(name, "PDF", resolution=90, append=True)

but what is cause of the increasing memory usage?

Is the PDF loaded onto memory when images are appended to it?

Also Is there another way that would avoid me looping through the functionality when there are too much images to handle, the images have a really large size or there are just not enough system resources to process all of the images on a "single run"

[radarhere]

I wonder if @vashek, as the author of #2965, might be able to provide an informative answer here?

[vashek]

@radarhere, are you sure it actually uses memory (at some significant rate), and not just address space? The PdfParser does memory-map the PDF being appended to (see https://github.com/python-pillow/Pillow/blob/master/src/PIL/PdfParser.py#L545, called from https://github.com/python-pillow/Pillow/blob/master/src/PIL/PdfParser.py#L380). I don't think it actually reads the data of the existing image, though.

But when you use append_images, @hallowf, then yes, the generator will be exhausted and all the images loaded and kept in memory, sorry. See https://github.com/python-pillow/Pillow/blob/master/src/PIL/PdfImagePlugin.py#L87

@radarhere Well the solution you provided will have to do for now (and yes memory increases at a much lower rate), I will probably just implement a failsafe in case there is too much memory being used to avoid system freezes

[radarhere]

Testing now, I find that the memory increase from my earlier solution is not consistent.

If I run the script with a 100 images, memory starts to decrease towards the end.

Closing, unless someone is able to more consistently demonstrate a problem with a given series of images - at the moment, there are no images attached here, so the problem is not as clearly defined as it could be.