C Unpickler memory leak via memo

[kale-smoothie]

BPO	41422
Nosy	@serhiy-storchaka, @kale-smoothie
PRs	bpo-41422: Visit the Unpickler's memo in Unpickler_traverse #21664
Files	leak_pickler.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-07-28.18:26:29.426>
labels = ['3.8', '3.9', '3.10', 'performance', '3.7', 'library']
title = 'C Unpickler memory leak via memo'
updated_at = <Date 2020-07-28.19:23:14.683>
user = 'https://github.com/kale-smoothie'

bugs.python.org fields:

activity = <Date 2020-07-28.19:23:14.683>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2020-07-28.18:26:29.426>
creator = 'kale-smoothie'
dependencies = []
files = ['49345']
hgrepos = []
issue_num = 41422
keywords = ['patch']
message_count = 2.0
messages = ['374518', '374521']
nosy_count = 3.0
nosy_names = ['python-dev', 'serhiy.storchaka', 'kale-smoothie']
pr_nums = ['21664']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue41422'
versions = ['Python 3.5', 'Python 3.6', 'Python 3.7', 'Python 3.8', 'Python 3.9', 'Python 3.10']

[kale-smoothie]

I'm not familiar with the workings of GC/pickle, but it looks like the traverse code in the C Unpickler omits a visit to the memo, potentially causing a memory leak?

[kale-smoothie]

The leak demonstrated in the attachment is, to my understanding, caused by memoizing the closure returned from the find_class method that's used to intercept global references. The cycle is then: Unpickler, memo table, closure, Unpickler (via cell reference to self).

My proposed patch visits every entry in the memo table.

Pre-patch run of valgrind on leak_pickler.py:

==20339== HEAP SUMMARY:
==20339== in use at exit: 190,189,238 bytes in 2,406,919 blocks
==20339== total heap usage: 3,150,288 allocs, 743,369 frees, 233,766,596 bytes allocated
==20339==
==20339== LEAK SUMMARY:
==20339== definitely lost: 0 bytes in 0 blocks
==20339== indirectly lost: 0 bytes in 0 blocks
==20339== possibly lost: 190,176,150 bytes in 2,406,835 blocks
==20339== still reachable: 13,088 bytes in 84 blocks
==20339== suppressed: 0 bytes in 0 blocks
==20339== Rerun with --leak-check=full to see details of leaked memory

Post-patch run of valgrind on leak_pickler.py:

==20880== HEAP SUMMARY:
==20880== in use at exit: 667,277 bytes in 6,725 blocks
==20880== total heap usage: 2,853,739 allocs, 2,847,014 frees, 216,473,216 bytes allocated
==20880==
==20880== LEAK SUMMARY:
==20880== definitely lost: 0 bytes in 0 blocks
==20880== indirectly lost: 0 bytes in 0 blocks
==20880== possibly lost: 654,624 bytes in 6,646 blocks
==20880== still reachable: 12,653 bytes in 79 blocks
==20880== suppressed: 0 bytes in 0 blocks
==20880== Rerun with --leak-check=full to see details of leaked memory