Inclusion of Null data values in serialized API output

[codingfiddler]

Current API standards indicate that if a data value is null, it shouldn't be included in the deserialized output that a client would receive. This definitely has advantages and disadvantages, but there are few things that could be tricky:

• GET/PUT actions become more complicated since a GET request would only return some of the fields on the object, and the resulting PUT would have to be designed to accept a limited subset of fields and isn't really a PUT anymore but more a PATCH.
• Sometimes you need to null out a given value, and if that value isn't returned after your update call there may be confusion on whether or not that value was actually updated.
• APIs that create static type definitions for these returned API objects would have to account for various fields not always being present vs. a traditional type definition

One way to handle some of these concerns, per Nick Clarity (@nickclarity) is to allow for deserializing objects into different typed objects (using a custom parser and factory) based on what properties are in available in the payload.

I'd love to get everyone's thoughts on this one, thanks a ton!

[nicholas-s-perkins]

I've talked about a few examples before in slack about why you might sometimes want to include null properties, but I can add an expanded use-case here.

Example Problem

The one case that I find most interesting is in testing API expansion.
We can use a real example from network-configuration-service as to why (for one reason at least) it returns nulls.

So let's consider an endpoint that fetches a dc4 company.
So you make your GET /company endpoint, which has a few properties:

{
   "id": 10,
   "name": "MyCoolCompany Inc",
   "active": true
}

But after a while of NCS being released, there was a desire to tie Identity orgs and dc4 companies together. They had been tied somewhat through Identity, but there was an architectural plan that showed that it worked better if the dc4 company itself had an org_id. So org_id is added to the company table and the endpoint is updated to return it.

{
   "id": 10,
   "name": "MyCoolCompany Inc",
   "active": true,
   "org_id": "121312341412442"
}

But there's a problem, org_id strings didn't exist for a lot of old dc4 companies. They had to be nullable. In fact, when the first release was made all existing dc4 companies had null org_ids.

So, in trying to write tests that confirm the output of GET /company is correct, imagine you have an API that filters out null properties in responses.
What does it then look like if:

1. The org_id is null
   or
2. The org_id property was not serialized correctly in the new response?

The answer is that they are identical, there is actually no possible way to detect the serialization bug when org_id=null.

{
   "id": 10,
   "name": "MyCoolCompany Inc",
   "active": true
}

Follow up thoughts

The use-case above alone makes me think that if we had any general recommendation for APIs it would be that they always include nulls when it's reasonable.
I tend to think of it as a static vs dynamic modeling problem. Some models need to be dynamic. E.g. some search endpoint that might look through 1000s of properties that you only want to see the properties returned that you search for.
But a lot of models are just representing some static sql table with just a handful of properties. In those cases if the value is correctly represented as null I think it makes sense for it to just be null.
If we're worried about payload bloat gzip is pretty efficient.

[eggilbert]

This was basically my argument when this topic was discussed in one of the working group meetings. I was on the dissenting side of the ledger from what was ultimately proposed.

Similar to NCS, in some stuff we're working through right now with Rules Management API, there is occasionally a property that is nullable. And returning it explicitly as such has been intentional to make it clear and unambiguous. Usually there's a value present, but in the cases where it doesn't have one, that's important to call out. It seems like it would be more weird if the field was just suddenly gone, because if it's a rare case, you might have incorrectly modeled things to assume that field was always present and then things break in deserialization.

In our specific case, we're returning what is really a "map" but doing so in an object form so the response structure is static (per other standards). So there's one field that is the key/name, and another that is the value. If the value is null and gets omitted per this standard, now this would just sometimes return an object with only a key specified? 🤔

I understand the other extreme that was brought up for Assortment where there's 1000+ fields and most of them are not used at a given time. But maybe that should be more the exception and not the rule? And Nick's statement of include nulls when it's reasonable is a better default stance?

[travisgosselin]

I think I'm pretty much in agreement with this conversation. In fact, spent a bit of time trying to review why we initially documented it this way in the API Standards (to ignore NULLs) and it looks like that meeting/review was not recorded (before and after was). We are really talking about the difference between 'undefined' and 'null'.

The Standards right now take this stance:
https://atlassian.spscommerce.com/wiki/display/SIP/API+-+Serialization#APISerialization-Null

Moving ahead with ignoring NULLs really agitates me in a particular way. Dynamic shaping of the content based on NULL fields is really in contradiction to keeping our shapes the same for a resource regardless of query string parameters. Additionally, it would technically be awkward for required content or other fields that are present, as it would mean that theoretically you could just remove any field from a REST API in a contract-breaking way and say it's NULL as a loophole to get rid of it. This does not seem to be in support of our direction, developer experience, and building for our consumers /clients. Keeping a static schema "is just simpler" (including nulls).

I'm in favor of updating this as you describe @eggilbert , with the default to not ignore NULLs and some exceptions. I'm also not clear that the exception scenario with 1000+ concrete fields is appropriate in an API, they should be in a list.

Any other thoughts? Anyone not in favor of this change?

[travisgosselin]

I have now inverted this as per the suggestions and reasoning available here. Any more updates/terminology changes? https://atlassian.spscommerce.com/wiki/display/SIP/API+-+Serialization#APISerialization-Null