MultiPartParser is skipped when uploaded file size > DATA_UPLOAD_MAX_MEMORY_SIZE

[eshaan7]

I posted this question on StackOverflow but after much debugging I'm inclined to believe this is a bug.

Original Post:
https://stackoverflow.com/questions/67621576/django-multipartparser-is-skipped-when-uploaded-file-size-data-upload-max-mem

I duplicate the same question here:

I have a view in my DRF app which accepts Content-type: multipart/form-data which I use to POST multiple files together,

• sample represents a binary file
• data is a string-yfied JSON.

Code:

from rest_framework.parsers import MultiPartParser
from rest_framework.decorators import api_view, parser_classes
from rest_framework import serializers as rfs

class CustomMultiPartParser(MultiPartParser):
    def parse(self, stream, *args, **kwargs):
        print("----- inside MultiPartJSONParser ------") ## for debug
        req = super().parse(stream, *args, **kwargs)
        data_str = req.data.get("data", None)
        if data_str:
            parsed = json.loads(data_str)
            mutable_data = {"data": parsed}
            mutable_files = {}
            for key, value in req.files.items():
                if key != "data":
                    mutable_files[key] = value
            return parsers.DataAndFiles(mutable_data, mutable_files)

        return req

class CustomFileSerializer(rfs.Serializer):
    data = AnalyseDataFieldSerializer()
    sample = rfs.FileField()

@api_view(["POST"])
@parser_classes((CustomMultiPartParser,))
def my_view(request):
   print(type(request.data)) ## for debug
   serializer = CustomFileSerializer(data=request.data)
   serializer.is_valid(raise_exception=True)
    # ...snip...

When the sample POSTed to this endpoint is < 2.5 mb in size, the type(request.data) is dict whereas if file size > 2.5 mb, the type(request.data) is django's QueryDict.

Logs:

# file size < 2.5mb
----- inside MultiPartJSONParser ------
{'data': {'private': False}, 'sample': <InMemoryUploadedFile: pafish.exe (application/x-ms-dos-executable)>}
# file size > 2.5mb
<QueryDict: {'data': '[{"private": false}]', 'sample': [<TemporaryUploadedFile: wannacry.bin (application/octet-stream)>]}>>

On debugging more, I find out that:

• The 2.5mb threshold is the django's DATA_UPLOAD_MAX_MEMORY_SIZE. When I set this variable to a higher value, the threshold becomes exactly that.

• the CustomMultiPartParser is skipped all-together in the second case. Since the parsing is skipped the request.data["data"] is still a string (as can be seen in the logs) which breaks the serialization and the response is:

  {"errors":{"data":["This field is required."]}}

• I thought that perhaps the issue is with the TemporaryUploadedFile, so in my settings.py, I forced it to only use TemporaryUploadFile with:

  FILE_UPLOAD_HANDLERS =   ["django.core.files.uploadhandler.TemporaryFileUploadHandler"]

  But I get the same problem with this as well, the CustomMultiPartParser is skipped all-together when the file size > 2.5mb (DATA_UPLOAD_MAX_MEMORY_SIZE).

I am not able to understand why the CustomMultiPartParser is skipped which breaks the serialization.

[eshaan7]

I debug further to pin point the exact lines of code. My findings about the control flow for the file size > DATA_UPLOAD_MAX_MEMORY_SIZE case:

• We need to look at the DRF's Request class. More precisely, the Request._parse() function since our issue is with parsing only.
• On line 329, it tries to access self.stream (self being the Request instance) which in turn calls self._load_stream() which in turn calls self.body, here I am not sure but I believe the body property comes from django's HttpRequest class.
• The call to self.body results in an exception RawPostDataException. (code ref.)
• On looking back at lines 328-330 of Request._parse() function we can see the same exception is caught. Due to which the function returns at line 337 i.e. it never reaches line 354 where there is the actual call to the CustomMultiPartParser class' parse function.

This answers why the parsing is skipped all together but I was unable to pinpoint the role of DATA_UPLOAD_MAX_MEMORY_SIZE variable.