Handling complex object payloads in API Responses

[nickclarity]

Taken from API-URL Structure: "A resource SHOULD NOT dynamically change its response body schema or shape based on query parameters or other dynamic behavior. If you require a differently shaped body to be returned, consider breaking it into different resource endpoints."

What if our response object includes a payload of some complex object type that happens to be JSON?

The best example I can think of this is from the Rules Management API. They define a standard shape for their response models and have a dynamic shape for the property "payload" which is simply defined as an object in the API spec. In this example, the payload happens to be JSON, so it's returned in-line as a JSON object. Does this mean the whole response model could then be considered dynamic?

This might fall into the category of APIs that serve up complex objects, or that wrap or manage blob/clob data.

Some topics for conversation are:

• Should payloads as part of a response be in the form of a string (this might include JSON, XML, plain text, etc...)? Do we make an exception for JSON? The difference (for JSON) being:
  • { "payload": { ... } }
  • { "payload": ""{\"...\"}""
• If we are returning payloads, should those always be defined at different routes?
  • Per the example below should we get meta data at /rules/v2/types/rules/instances/{instanceID} and the payload at /rules/v2/types/rules/instances/{instanceID}/payload?
  • If so, does the response content-type for /payload become application/octet-stream?
  • Or, is it sufficient to include the {typeID} as a route param too? /rules/v2/types/{typeID}/rules/instances/{instanceID}/payload
• Do we care to add some definition for encoding? Whether as part of a string in a response object or a separate route.
• Is there a difference between how we deal with requests vs responses?

Examples from /rules/v2/types/rules/instances/{instanceID}:
Response model including a Document Identifier rule payload:

{
    "ID": 28097,
    "active": false,
    "createdBy": "USER: 789436214106946475209379604081598495864968",
    "createdDate": "2021-03-09T20:15:23.586606Z",
    "fingerprint": "7f2fa7c5-bb34-3f58-868c-b2ee85bb0d7g",
    "payload": {
        "identifiersBySource": [
            {
                "source": "ParcelPayload",
                "identifiers": [
                    {
                        "identifier": "PurchaseOrderNumber",
                        "pathsByLanguage": [
                            {
                                "paths": [
                                    {
                                        "path": "Order/Header/OrderHeader/PurchaseOrderNumber"
                                    }
                                ],
                                "queryLanguage": "xpath"
                            }
                        ]
                    }
                ],
                "identifiersVersion": {
                    "source": "Keystone",
                    "version": "1.0"
                }
            },
            {
                "source": "ParcelMeta",
                "identifiers": [
                    {
                        "identifier": "ParcelCreatedDate",
                        "pathsByLanguage": [
                            {
                                "paths": [
                                    {
                                        "path": "$.createdDate"
                                    }
                                ],
                                "queryLanguage": "jsonPath"
                            }
                        ]
                    }
                ],
                "identifiersVersion": {
                    "source": "ParcelService",
                    "version": "v3"
                }
            }
        ]
    },
    "rule": {
        "ID": 7189,
        "createdBy": "USER: +564654654654",
        "createdDate": "2023-01-09T20:15:23.422438Z",
        "description": "Order values that identify the document",
        "modifiedBy": "USER: 548598798468464654654",
        "modifiedDate": "2022-09-09T20:15:23.422438Z",
        "name": "Order Identifiers",
        "selectors": [
            {
                "name": "documentType",
                "value": "Order"
            },
            {
                "name": "documentVersion",
                "value": "7.7"
            },
            {
                "name": "organization",
                "value": "654987654654654654"
            },
            {
                "name": "organizationSecondary",
                "value": null
            }
        ],
        "type": {
            "ID": 31
        }
    }
}

Response model including a Document Matching rule payload:

{
    "ID": 27720,
    "active": true,
    "createdBy": "USER: 6546874984654654",
    "createdDate": "2022-01-05T17:57:25.313157Z",
    "fingerprint": "ae23a388-39e3-3ce9-9213-8254e5d10454",
    "payload": {
        "matchingSources": [
            {
                "sourceFilters": [
                    {
                        "value": null,
                        "operator": "max",
                        "priority": 1,
                        "identifier": "ParcelCreatedDate",
                        "compareType": "date"
                    }
                ],
                "sourceDocument": "Order",
                "matchingIdentifiers": [
                    {
                        "operator": "equals",
                        "required": false,
                        "compareType": "string",
                        "sourceIdentifier": "PurchaseOrderNumber",
                        "targetIdentifier": "PurchaseOrderNumber"
                    },
                    {
                        "operator": "equals",
                        "required": false,
                        "compareType": "string",
                        "sourceIdentifier": "CustomerOrderNumber",
                        "targetIdentifier": "CustomerOrderNumber"
                    },
                    {
                        "operator": "equals",
                        "required": false,
                        "compareType": "string",
                        "sourceIdentifier": "ReleaseNumber",
                        "targetIdentifier": "ReleaseNumber"
                    }
                ]
            }
        ]
    },
    "rule": {
        "ID": 5827,
        "createdBy": "USER: 674654654654654654654654654",
        "createdDate": "2022-01-05T17:57:24.918303Z",
        "description": "Document matching instructions for Shipment",
        "modifiedBy": "USER: 65464684654654654654",
        "modifiedDate": "2019-01-05T17:57:24.918303Z",
        "name": "Shipment Document Matching Rule",
        "selectors": [
            {
                "name": "documentType",
                "value": "Shipment"
            },
            {
                "name": "documentVersion",
                "value": "7.7"
            },
            {
                "name": "organization",
                "value": null
            },
            {
                "name": "organizationSecondary",
                "value": null
            }
        ],
        "type": {
            "ID": 32
        }
    }
}

https://spscommerce.stoplight.io/docs/93864e4e-da8c-43a9-ba1d-7ccc8dd93ca5/b3A6MzQwMzI3NzE-get-a-specific-rule-instance

[nickclarity]

Further Elaboration

For the example of the rule instance payload, I'm not sure how it is being stored, but the schema a rule type is well defined at /rules/v2/types/{typeID} but itself could be considered a dynamic response type. The purpose of rules/v2 as I understand it was specifically to structure the routes in this way for maintainability. As new rule types are added, they didn't want to continue to add new routes to service those rule types. Also, specifically for Rules Management, a rule type and rule instances are entities, so returning them as JSON object would be preferred. However, how we define routes, queries and response types might also have implications in other services that I am not aware of.

For the DEX API, we have a number of endpoints that are only used for testing (at the moment) that take in both encoded RSX payloads as strings and also in-line JSON objects (which are the dynamically shaped rule instance objects). Hence, the bullet for requests vs responses.

Other considerations are deserialize in-line dynamically shaped object into generic types. From my experience this can be tricky in Java/Kotlin because of variance. I am not an expert on this topic (as I am new to the Java world) but in my experience it causes a limited amount of flexibility when requesting/parsing data into a List<Request<T>() where T : RuleInstanceResponsePayload (pardon my mixed syntax). From my understanding of Java, type information is lost at runtime including the type of T. Also T and List<T> aren't interchangeable.

[eggilbert]

Here's the docs for creating a Rule Type which show both the request and response:
https://docs.api.spscommerce.com/docs/93864e4e-da8c-43a9-ba1d-7ccc8dd93ca5/b3A6MzQwMzI3Njc-create-a-new-rule-type

In this case, the validation schema object is also JSON so the guidance at the time was to combine them together. But what if it was a parcel? Or maybe something binary?

Is this still what we want to do with JSON in JSON? Is that a special case? Do we want some encoding as a big base64 string like DEX is doing with parcel payloads in requests? Do we want to leave it transparent as a non-encoded string but just escape the nested JSON so the whole thing is a single valid JSON string?

CC: @acchristensen since I think you were part of discussions here a year ago. 😄

[travisgosselin]

Got it, I understand now, thanks for clearing that up a bit in my mind. I don't think I have a problem at all with the way you have currently done this. I also don't think its in conflict with the API Standards or the direct intended target of the statement.

• URL is changing with instance ID - the core intent was to ensure that the same URL (without query string) does not dynamically change its schema. The wording is a bit ambiguous by saying "resource". I think your balance of the same wrapper payload for every response combined with a defined dynamic portion is a good marriage towards the intent here. The fact that the creation of this resource takes dynamic content (as the content itself) is also worth noting as something that seems in line with the spirit of the standards for this type of request.
• Should the payload be merged JSON or some other type of string to differentiate it / parse it differently? - For this, I think my head keeps coming back to developer experience, thinking "how would I want to receive this dynamic data"? As a consumer I think merged JSON is the easiest because the payload /json was provided as part of the request payload to create it. In most languages I could easily create a consumer that uses "payload" as a dynamic "expando-like object, JObject... etc whatever you want to call it". The alternative would have me doing more work to pull the string and parse it (double parse)... but I"m struggling to see the reason behind doing that differentiation? Did you have considerations on that? Are there security thoughts here?

It sounds like this discussion has been had already during build time, so if there are other considerations you weighed back and forth it would be helpful to document them perhaps for future clarity. I assume this is question was not posed as you are wanting to change how it is done today, but just furthering the standards?

Do you have suggestions or thoughts on how / what you would change in the reference for dynamic resources/endpoints in the API Standards? (I think based on this case study we should add something). Thoughts?

[eggilbert]

I can't speak for @nickclarity, but for me, yeah I mostly just wanted this to get brought up so potential gaps in the standards around serialization could be discussed. Lots of things deal with parcels, and that's an interesting example of generally non-JSON payloads that get passed around and I couldn't find anything written down about how to handle that, just more simple types like dates, booleans, etc.

I don't think any of the 3 ways that have been proposed here about merging the JSON, encoding it as a string, or escaping it as a string are particularly bad or wrong. And I've seen most if not all of them in play in various services. Should there just be one preferred method, or do they all feel acceptable?

[nickclarity]

"URL is changing with instance ID"

Thank you @travisgosselin for clarifying. That makes sense to me now. The instanceID defines the shape of a specific resource. Seems obvious when I type it.

I also agree with @eggilbert that there might be more to explore regarding serialization and non-JSON payloads. I don't want to get too hypothetical, but if Parcel Service returned a JSON object with metadata:

{
  "id": "123",
  "createdBy": "987",
  "payload": "<Order><Header>..."
}

{
  "id": "456",
  "createdBy": "987",
  "payload": { "order":  { "header": {  } } }
}

(Sorry for the "RSX" example)

In this case the response object could contain either JSON or XML for the payload property. Would you imagine that because the payload for instance 456 happens to be JSON that it should be merged into the response object? Also, if the payload isn't JSON, should it be encoded to save space in transit? This might get more to the heart of what @eggilbert was talking about.

[travisgosselin]

Thanks for the examples - this is great. I chatted a bit with Evan this morning as well. The final question/concern here is if there should be more direct guidance in the REST API Standards on handling potentially complex objects and payloads and how those should be mixed with JSON result objects.

I do not think there is necessarily a clear statement to make at the level of REST API Standards for this. Seems more of a case-by-case basis. But I would be interested to hear if you have an opinion forming on statements that could be made in the standards?

Thoughts on different examples:

• With rules-management, the input is ALWAYS JSON dynamic request, and output is always JSON. Therefore merging it together with the JSON result object, is easy, and makes sense. If rules-management also supported XML results (return content-type), I'd expect that the "payload" would still be returned in JSON while the other content is xml
• With the example you provide above for RSX, this is a good use case in a few ways and starting to get to the heart of some long-term questions about RSX integration with our APIs. Leaving out the "RSX" specific nature, I think a lot depends on that payload. If that payload is more opaque in general (an unknown format or ambiguous format), keeping it as a string that you either escape (as necessary) or encode seems most appropriate (different from rules-management where you know the format). Should that payload be converted back and forth between different types dynamically in the single json result output? Probably not, for what reason?

This starts to bring up additional patterns related to media types (Content-Type). As per one of your original thoughts, you'd also have to determine if it makes sense to return the "payload" in the current response (for rules mgmt it would seem it did, as payload size was generally small). However, when we start talking more about documents as a whole, it would be unreasonable potentially to always provide the entirety of a document payload as it relates to a resource in your API. in this case I think it would be a standard pattern to separate it using hypermedia style link:

GET /resource/123
Content-Type: application/json
RESPONSE:
{
  "id": "123",
  "createdBy": "987",
  "payloadLink": "/resource/123/payload"
}

GET /resource/123/payload
Content-Type: application/xml
RESPONSE:
<Order><Header>...

**Can use status code 415 to indicate errors if given the wrong content type.

This enables consumers to get larger payload content only when they explicitly need it. It also enables you to handle content-type specification if this object can be requested in multiple formats:

GET /resource/123/payload
Content-Type: application/json
RESPONSE:
{ "order":  { "header": {  } } }

I think there are things to add to the API Standards with regards to a new section on media-types / content-type in general indicating the above. But that I don't think generally answers your original question, since "it depends" :)

Perhaps there is an opportunity to add a statement like:
Escaped JSON SHOULD NOT appear within a request and response object using Content-Type of application/json where the format of that value is known to always be JSON. Instead it should appear as merged JSON.

For encoding of the payload, I'm not sure I have an opinion. If the intention of the API is to ensure opaqueness of the value, that seems helpful.

[mmerth]

I think another interesting example that can be brought up, is the Transformation Design Artifact. I know we've had various issues with this in the past. At the time there were a few different tools that all used their own defined models. IIRC new properties were added over time to help handle new features for Transformation Designer, and any tools that used those JSON designs would need to be updated.

At the time I was working on a couple tools that needed to understand the complex design JSON, so I made a python package that both of my tools used. https://github.com/SPSCommerce/transformation-design-tools

This required the package to be properly maintained and updated as needed, but overall I think it improved the developer experience when working with the JSON designs (in python at least)

[travisgosselin]

Nice, distributed and shared SDKs for the APIs is a great place to be in (even if it's only models that are shared). While we will be designing reuseable "domains" to represent shared modeling in API design first, this is a good reminder of the benefits in those models moving to shared packages as well. Even a system where we update the spec/contract theoretically could materialize as updated modeling distributed to source code (maybe a pipe dream at an organizational scale, but fun to think about).

[acchristensen]

This is a great concern to bring up since it's one of my favorite patterns that I see wide applicability for in many areas for us. Some existing examples have been called out and there are some others:

• Rules Mgmt: provide a JSON schema (metamodel) that defines the schema validation for a given rule type.
• Transformation Designer with the JSON re-encoding and double parse that is... I think a bit unfortunate
• Rules v1 - the specs are a kind of metamodel that defines data requirements for its translations
• ICA - I believe kind of has a thing like this. The configuration could be different between between account types... But I think it's the UI that has to have the logic to provide the right params (might be wrong on that).
• Events for Event Store/REP that has a specific attribute set aside for metadata. No defined metamodel for that, but if we had one based on the event type or source it could be immensely useful.

And I think there are some other possible places this could crop up, as well.

• Metadata for parcels or doc processing stage snapshots.
• Possibly for capturing a snapshot of which config was used at a given time for doc processing.
• Requirements for documents sent to a Doc API based on a specific retailer requirements or supplier business rules.

If we can add some thoughts around standardization for this pattern I'd be all for it. I think the use of JSON schema metamodel for JSON payloads is an "easy" example. It makes sense to me that the metamodel would be defined using other standards or DSLs, and applied against different formats of data. For example, for a Doc API we'd likely have some artifact (TD design?) that describes the validation rules for a given document (between a specified set of trading partners, doc type, and fulfillment model). Which validation artifact to use will depend on those dynamic parameters. And the structure that it's applied against would likely be RSX, but maybe eventually a new API standards compliant structure.
Anyway, just to throw my 2 cents in there. Metamodels to be applied to a payload based on some parameters makes a lot of sense for our domain.

[travisgosselin]

Any defined thoughts on a proposal? Or concrete examples of standards you'd make?

[acchristensen]

Regarding payload serialization, I do think this has got to be something we lean heavily on Content-type and Accept headers for. Encoding a payload inside a different format is cumbersome. For request data, I think the multi-part stuff can handle multiple mime-types with multiple payloads in a request. I'd imagine the same can be done for a response. Something to look at. The Ship Doc transformation API does this for passing in an xml payload and corresponding stylesheet.

[travisgosselin]

Do you have an API spec reference or docs to link directly to this example?

[acchristensen]

For what ShipDoc's Transformation API did on receiving multi-part, multi-mime requests (not returning them): https://developercenter.spscommerce.com/#/docs/transformation-api/transformation . Or for the code: https://github.com/SPSCommerce/devcenter-transformation-api/blob/main/xsl-transformation-service/src/main/java/com/spscommerce/workflow/xsltransformationapi/resource/TransformResource.java#L51-L55