pickle + weakref.proxy(self)

[phdru]

BPO	18068
Nosy	@phdru, @pitrou, @avassalotti, @serhiy-storchaka

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = None
created_at = <Date 2013-05-26.18:59:09.881>
labels = ['type-feature', 'library']
title = 'pickle + weakref.proxy(self)'
updated_at = <Date 2015-11-28.14:11:50.631>
user = 'https://github.com/phdru'

bugs.python.org fields:

activity = <Date 2015-11-28.14:11:50.631>
actor = 'serhiy.storchaka'
assignee = 'serhiy.storchaka'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2013-05-26.18:59:09.881>
creator = 'phd'
dependencies = []
files = []
hgrepos = []
issue_num = 18068
keywords = []
message_count = 3.0
messages = ['190105', '199902', '255541']
nosy_count = 4.0
nosy_names = ['phd', 'pitrou', 'alexandre.vassalotti', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue18068'
versions = ['Python 3.4']

[phdru]

Hello! I've found a problematic behaviour of pickle when it pickles
(direct or indirect) weak proxies to self. I suspect pickle cannot
detect reference loops with weak proxies. I managed to narrow the case
to the following example:

import pickle, weakref

class TestPickle(object):
    def __init__(self):
        self.recursive = weakref.proxy(self)

    def __getstate__(self):
        print "__getstate__", id(self)
        return self.__dict__.copy()

    def __setstate__(self, d):
        print "__setstate__", id(self)
        self.__dict__.update(d)

print "- 1 -"
test = TestPickle()
print "- 2 -"
data = pickle.dumps(test, pickle.HIGHEST_PROTOCOL)
print "- 3 -"
test2 = pickle.loads(data)
print "- 4 -"
print "Result:", id(test2)
print "- 5 -"

It prints:

• 1 -

• 2 -
  __getstate__ 3075348620
  __getstate__ 3075348620

• 3 -
  __setstate__ 3075348844
  __setstate__ 3075349004

• 4 -
  Result: 3075349004

• 5 -

  That is, __getstate__ is called twice for the same object. And what
  is worse, __setstate__ is called twice for different objects. The
  resulting unpickled object is the last one, but in the library that I
  have been debugging creation of two different objects during unpickling
  is a bug.

  I can fix it by avoiding pickling the proxy and recreating the proxy
  on unpickling:

import pickle, weakref

class TestPickle(object):
    def __init__(self):
        self.recursive = weakref.proxy(self)

    def __getstate__(self):
        print "__getstate__", id(self)
        d = self.__dict__.copy()
        del d['recursive']
        return d

    def __setstate__(self, d):
        print "__setstate__", id(self)
        self.__dict__.update(d)
        self.recursive = weakref.proxy(self)

print "- 1 -"
test = TestPickle()
print "- 2 -"
data = pickle.dumps(test, pickle.HIGHEST_PROTOCOL)
print "- 3 -"
test2 = pickle.loads(data)
print "- 4 -"
print "Result:", id(test2)
print "- 5 -"

• 1 -

• 2 -
  __getstate__ 3075070092

• 3 -
  __setstate__ 3075070188

• 4 -
  Result: 3075070188

• 5 -

  But I wonder if it's a bug that should be fixed? If it's an expected
  behaviour it perhaps should be documented as a warning in docs for
  pickle or weakref or both.

[pitrou]

I don't think weakref objects are meant to be picklable at all. At least there's no support code for that, so pickling weak proxies only works by chance.

[serhiy-storchaka]

The weakref.proxy object delegates all attribute access to referred object, including special attributes used in pickling, like __class__ or __getstate__.

test.recursive is not test, but it looks as TestPickle instance for the pickler. It is pickled and unpickled as TestPickle instance.

>>> test.recursive.__class__
<class '__main__.TestPickle'>
>>> test.recursive.__getstate__
<bound method TestPickle.__getstate__ of <__main__.TestPickle object at 0xb7130bac>>
>>> test.recursive.__reduce_ex__(2)
__getstate__ 3071478700
(<function __newobj__ at 0xb74b76f4>, (<class '__main__.TestPickle'>,), {'recursive': <weakproxy at 0xb7127a04 to TestPickle at 0xb7130bac>}, None, None)
>>> pickle.loads(pickle.dumps(test.recursive, 2))
__getstate__ 3071478700
__setstate__ 3071525356
<__main__.TestPickle object at 0xb713c1ec>

Since test.recursive.recursive is test.recursive, this recursion is detected by pickler.

This issue is similar to bpo-6395, but since weakref.proxy is not subclassable, we can just implement correct __reduce__.