Can't cache a ReturnDict.

[StevenVeshkini]

Has anyone been able to cache serializer.data from a custom Serializer on one object? Seems like there's no way to store the ReturnDict that results from serializer.data.

[tomchristie]

Could you also include a concrete use case to work against. In get the question generally, but it'd be helpful to see the specific case you're trying to achieve.

[tomchristie]

Also what error is raised?

[xordoquy]

It seems that the ReturnDict having a link to the serializer breaks the pickling:

>>> from rest_framework import serializers
>>> class NestedSerializer(serializers.Serializer):
...             one = serializers.IntegerField(max_value=10)
...             two = serializers.IntegerField(max_value=10)
... 
>>> class TestSerializer(serializers.Serializer):
...             nested = NestedSerializer()
... 
>>> input_data = {
...             'nested': {
...                 'one': 1,
...                 'two': 2,
...             }
...         }
>>> s = TestSerializer(input_data)
>>> s.data
ReturnDict([('nested', OrderedDict([('one', 1), ('two', 2)]))])
>>> d = s.data
>>> import pickle
>>> pickle.dumps(d)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
_pickle.PicklingError: Can't pickle <class 'TestSerializer'>: attribute lookup TestSerializer on builtins failed
>>> pickle.dumps({'nested': [{'one': 1, 'two': 2}]})
b'\x80\x03}q\x00X\x06\x00\x00\x00nestedq\x01]q\x02}q\x03(X\x03\x00\x00\x00twoq\x04K\x02X\x03\x00\x00\x00oneq\x05K\x01uas.'

[xordoquy]

Note that Django uses pickle to cache things.

[StevenVeshkini]

@xordoquy Yes, that is the exact problem I am having (nested serializers).

[xordoquy]

It was a bit late yesterday. I'll add a proper test case this evening.

[StevenVeshkini]

@xordoquy Is this a bug or was this the way DRF was designed?

[xordoquy]

It's both ;)

[StevenVeshkini]

hahahaha nice :P

[StevenVeshkini]

Any luck?

[tomchristie]

So the right way to resolve this would be either...

• Make serializer instances picklable, which oughta resolve this.

Or...

• Make ReturnDict's pickle representation not include the serializer backlink. Given that we render the .content that oughta be okay in most cases anyways, not clear if accessing response.data from cached responses is something that should happen or not.

My personal take is that using pickle to deal with HTTP caching is a wonky approach. Rather store the raw headers and content, and return a newly created response instance when returning cached content. (Then you don't bump into all this horrible sorts of cases.)

[picturedots]

I encountered this problem (I think) when I was trying to set up caching
and gave up on this approach. I ended up caching the entier rendered
response instead, using the URL as the key, but only if the accepted
renderer wasn't api, i.e.
should_cache = self.request.accepted_renderer.format != 'api'

On Mon, Jan 5, 2015 at 12:31 PM, Tom Christie [email protected]
wrote:

So the right way to resolve this would be either...

• Make serializer instances picklable, which oughta resolve this.

Or...

• Make ReturnDict's pickle representation not include the serializer
  backlink. Given that we render the .content that oughta be okay in
  most cases anyways, not clear if accessing response.data from cached
  responses is something that should happen or not.

My personal take is that using pickle to deal with HTTP caching is a wonky
approach. Rather store the raw headers and content, and return a newly
created response instance when returning cached content. (Then you don't
bump into all this horrible sorts of cases.)

—
Reply to this email directly or view it on GitHub
#2360 (comment)
.

Mark Chackerian

VP Engineering, Architizer http://www.architizer.com/
646.812.0994

[gaffney]

I just encountered this issue as well while working on the upgrade. It also happens with a ReturnList:

(Error lines: cPickle.c #L2100-2106)

  subl /Users/_/.virtualenvs/_/lib/python2.7/site-packages/django/core/cache/backends/locmem.py:67
    pickled = pickle.dumps(value, pickle.HIGHEST_PROTOCOL)
PicklingError: Can't pickle <type 'generator'>: attribute lookup __builtin__.generator failed