Non-required fields with 'allow_null=True' should not imply a default value

[RomuloOliveira]

Prior to the #5518, all fields with required=False and without a explicit default value were not included in the output.

After the patch, even non-required fields now had a default value, which is counter-intuitive if you look at the required documentation, which says Set to false if this field is not required to be present during deserialization.

This PR restore the original required=False behaviour. Now, any required=False without a explicit default value should not be returned in the output.
However, it keeps the implicit default=None for required fields introduced in #5518.

Question I: Did I wrote the test in the correct class (TestNotRequiredOutput) or should it be in the TestDefaultOutput class?

Question II: Is there a reason that the implicit default None value for allow_null is not in the Field.get_default function? Looking at the except handling code, at first it did not sound right.

Question III: Should I add to the documentation that allow_null implicitly set a default=None for required fields?

That issue is already closed and it is not my intention to reopen it, but it's a special case that I did not fully understand why not forcing a explicit default value. E.g, what about allow_blank? And many=True? Shouldn't they have a implicit default value too, using the same logic?

[rpkilby]

Hi @RomuloOliveira, thanks for raising this. The docs do say that required fields may be omitted:

Setting this to False also allows the object attribute or dictionary key to be omitted from output when serializing the instance. If the key is not present it will simply not be included in the output representation.

To address a few things...

However, it keeps the implicit default=None for required fields introduced in #5518.

To clarify, the PR description is not completely correct, and this was not the final behavior. default values affect both serialization and deserialization, and allow_null implying default=None was not the correct behavior (you can see this in the commit/build history for #5518). Instead, allow_null only implies a default value for serialization output.

Did I wrote the test in the correct class (TestNotRequiredOutput) or should it be in the TestDefaultOutput class?

TestNotRequiredOutput is correct, as this isn't testing the default argument.

Is there a reason that the implicit default None value for allow_null is not in the Field.get_default function?

Yes, this is related to allow_null implying default=None not being the correct behavior.

Should I add to the documentation that allow_null implicitly set a default=None for required fields?

I don't think it's necessary for this PR. I'll address it separately.

[rpkilby]

👍

[rpkilby]

In hindsight, while consistent with the required docs, I don't think this is the correct behavior. I'm thinking that allow_null should always imply a default serialization value, regardless of whether a field is required.

@RomuloOliveira, was there a specific reason why these fields needed to be omitted from the serialized output? It's not clear to me what behavior this is enabling.

[RomuloOliveira]

@rpkilby I've made this to made required=False output consistent and simple (not required -> omitted from output, period) and restore this behaviour, which existed before 3.7.2.

In my opinion, it keeps things explicit and do not depend on any other flags combinations to work as expected. In this way, an user do not need to know all caveats and special behaviour that occurs when flags are combined.

I noticed this because after upgrading to 3.7.2 some fields were not consistent with others in an API that I was developing and it took a lot of time to figure out what was causing it. I've thought at first it was a bug before seeing the discussion #5518.

I thought it's a bit weird that allow_null=True implies a default serialization output. It is not consistent with python/DRF "philosophy" of keeping things simple and explicit.

This discussion was closed and so was #5518, but as an DRF user, I'm asked myself why use implicit values (which can interfere with another flags) and, why use it only in allow_null. I think it made inconsistent, since allow_blank=True do not imply a blank default output and I really don't think it should.