Submitting both user and instance data via both the uri and message body

[handrews]

[Edited for changes made between #159 and #179]
[Edited to replace hrefVars with hrefSchema based on later discussions]

The purpose of this issue is to explore the limitations of the current draft's usage of method, schema, and href (in terms of resolving template variables). There are ideas here that could become a proposal, but I am not directly proposing anything in the example schema. I'm really just trying to find a consensus on the limitations.

The main point here is the need to both specify URL parameters and a message body. The API has events and invitations. The invitation collection can be filtered from either the event perspective or (via email address) the invitee perspective. I did not provide a schema for invitees as it made the example too long without adding anything.

Since the query string is as much a part of the identifier as the path components, this API treats /invitations?event=1234 as the collection of invitations for the given event. The API supports creating invitations for event 1234 by POSTing to that URI, just the same as it would for a URI of /events/elements/1234/invitations. However, not all of the fields needed to create an invite can be a part of the URI, so a creation also takes a message body providing the remaining parameters.

The current href specification does not allow it to define user input. But if schema is used for the body, there is no other mechanism for getting user input into the URI. This example uses hrefSchema as a way to map the URI template parameters to a schema and/or a value from the instance. This is to help illustrate the two routes of user input, not propose hrefSchema as necessarily being a solution.

hrefSchema is a schema which treats the template variables as if they were properties of a JSON object. Each variable property's schema is used to validate "user" input (which may or may not be from an actual human user). It can use $data with default to take a default value from the instance when no user input is supplied. To implement the current behavior of only resolving from the instance, you would use a schema like this:

{
    "properties": {
        "email": {
            "enum": {"$data": "0/email"},
            "default": {"$data": "0/email"}
        }
    }
}

For convenience, the above can be specified by just giving the "0/email" relative JSON pointer instead of a schema (this is what you will see in the example). [EDIT: originally hrefSchema was hrefVars and did not quite use regular schema syntax. Now that hrefSchema is a regular schema, the shortcut might be a bad idea]. The default behavior for template variable {foo} is "hrefSchema": {"properties": {"foo": "0/foo"}} (which replicates the current behavior).

I am also not advocating for or against this specific design of collections and individual resources. It was constructed to illustrate my points. You could get around some issues by redesigning the resources, but aside from expecting APIs to not blatantly abuse protocols, I do not think that hyper-schema should be opinionated about resource design.

{
    "definitions": {
        "identifier": {
            "description": "Assigned and managed by the server.",
            "type": "integer",
            "minimum": 1,
            "readOnly": true
        },
        "event": {
            "type": "object",
            "properties": {
                "id": {"$ref": "#/definitions/identifier"},
                "name": {"type": "string"}
            }
        },
        "invitation": {
            "type": "object",
            "properties": {
                "id": {"$ref": "#/definitions/identifier"},
                "email": {"$ref": "#/definitions/identifier"},
                "event": {"$ref": "#/definitions/identifier"},
                "headline": {"type": "string"},
                "message": {"type": "string"}
            }
        },
        "eventResource": {
            "allOf": [{"$ref": "#/definitions/event"}],
            "links": [
                {
                    "rel": "self",
                    "href": "/events/elements/{id}"
                },
                {
                    "rel": "collection",
                    "href": "/events"
                },
                {
                    "rel": "tag:example.com,2016:invitations",
                    "href": "/invitations{?email,event}",
                    "hrefSchema": {
                        "properties": {
                            "email": {"type": "string", "format": "email"},
                            "event": "0/id"
                        }
                    },
                    "schema": {
                        "properties": {
                            "headline": {
                                "type": "string",
                               "default": "0/name"
                            },
                            "message": {"type": "string"}
                        }
                    }
                }
            ]
        },
        "invitationResource": {
            "allOf": [{"$ref": "#/definitions/invitation"}],
            "links": [
                {"rel": "self", "href": "/invitations/elements/{id}"},
                {"rel": "collection", "href": "/invitations"}
            ]
        },
        "eventCollection": {
            "type": "object",
            "properties": {
                "elements": {
                    "type": "array",
                    "items": {
                        "allOf": [{"$ref": "#/definitinos/event"}],
                        "links": [
                            {"rel": "item", "href": "/events/elements/{id}"}
                        ]
                    }
                },
                "filters": {"name": {"type": "string"}}
            },
            "links": [
                {
                    "rel": "self",
                    "href": "/events{?name}",
                    "hrefSchema": {
                        "properties": {"name": "0/filters/name"}
                    }
                }
            ]
        },
        "invitationCollection": {
            "type": "object",
            "properties": {
                "elements": {
                    "type": "array",
                    "items": {
                        "allOf": [{"$ref": "#/definitions/invitation"}],
                        "links": [
                            { "rel": "item", "href": "/invitations/elements/{id}"}
                        ]
                    }
                },
                "filters": {
                    "type": "object",
                    "properties": {
                        "email": {"type": "string", "format": "email"},
                        "event": {"$ref": "#/definitions/identifier"}
                    }
                }
            },
            "links": [
                {
                    "rel": "self",
                    "href": "/invitations{?email,event}",
                    "hrefSchema": {
                        "properties": {
                            "email": "0/filters/email",
                            "event": "0/filters/event"
                        }
                    }
                }
            ]
        }
    },
    "links": [
        {
            "rel": "tag:example.com,2016:events",
            "href": {"$ref": "/events{?name}"},
            "hrefSchema": {"properties": {"name": {"type": "string"}}},
            "schema": {"$ref": "#/definitions/event"}
        },
        {
            "rel": "tag:example.com,2016:invitations",
            "href": {"$ref": "/invitations{?email,event}"},
            "hrefSchema": {
                "properties": {
                    "email": {"type": "string", "format": "email"},
                    "event": {"$ref": "#/definitions/identifier"}
                }
            }
        }
    ]
}

Given this schema, I should be able to do something like this (separate params and data arguments lifted from the design of Python's "requests" library- again, not insisting this is the solution, just trying to illustrate the challenge):

from magic import restclient

# Assume restclient uses describedBy HTTP
# header links to fetch schemas as needed
entry_point = restclient("https://api.example.com")

# For simplicity, we'll assume the first result is right.
party = restclient.follow("tag:example.com,2016:events",
                          params={"name": "Awesome Party"})[0]

# This lets the headline be filled in with the event's name automatically
party.submit("tag:example.com,2016:invitations",
             params={"email": "[email protected]"},
             data={"message": "Come to my party!"})

So in this example, there are several combinations of data sources and destinations:

• "event" is put in the URI, and is only allowed to come from the instance
• "email" is put in the URI, and must come from the user
• "headline" is put in the body, and if not specified, it comes from the instance
• "message" is put in the body, and must come from the user

Currently, this cannot be expressed in hyper-schema. Should it be possible? If we get agreement on that, we can talk about how to do it.

[handrews]

@awwright @Relequestual @jdesrosiers @slurmulon @Anthropic

Any reaction at all to this? Am I the only person who has ever wanted to try to make this work? What alternatives am I missing?

[Anthropic]

@handrews would an accurate TL;DR be that you want to define body data instead of just a querystring in links, but also type define the params?

Regarding alternative requirements, from a UI Schema perspective one issue I find is that we need to be able to define data that is not added to the model represented by the schema, for example, you have a data set that is made into a wizard stepper sequence when the form is defined, the details of the wizard's current workflow step is not part of your model, but you want it in the uri, not in the content sent to the server. My server processes the entire data set sent to it, I need an additional scope that is not in the main definition to run conditions against. Almost like a scope for model and one for state. I guess being able to define what goes in the body would be useful for that.

PS. Still unsure why links is an array, anyone know? Doesn't make sense to me, you don't want duplicate links and you need a rel, so why not make that the key in an object so it can easily be referenced.

[handrews]

PS. Still unsure why links is an array, anyone know? Doesn't make sense to me, you don't want duplicate links and you need a rel, so why not make that the key in an object so it can easily be referenced.

There are several link relations from various RFCs that specifically state that they can appear multiple times. Of course I can't remember any of them off the top of my head, but I assume that's the reason. It's a bit annoying for the common case, I agree.

(I'll reply to the main points once I've had a chance to think through your example a bit more).

[handrews]

@Anthropic if I understand your UI example correctly, your wizard sends a resource representation to update at each step, but each step also needs to send information that is outside of that representation? So the representation is sent in the body, and the other data goes in the URI?

This is similar to what I'm talking about, and may even be the same in many cases. For me, anything that goes in the URI is part of resource identification. So URI query parameters that implement filtering and pagination of collections are identifying a subset of the larger collection. I suppose such parameters could identify states in a workflow as well.

HTML provides only one direct avenue for user input, and HTML forms put that input into either the URI or the body. But I don't see a reason to preserve that either/or behavior in an API. I have frequently needed to interact with both, as the URI and the request body serve different purposes.

would an accurate TL;DR be that you want to define body data instead of just a querystring in links, but also type define the params?

I'm not sure- what does "type define the params" mean?

---

In an API, start from an entry point resource, and follow links, interacting with the various resources we encounter. We usually send representations of those resources back and forth, or (in the case of patching) documents related to the representation in some way defined by the media type being used for patching. Occasionally we send data that is less directly related to the resource, and in HTTP this is done using POST.

What I've found missing in all hypermedia API systems that I've tried to work with so far is a comprehensive system for mapping data from resource A's representation into both the URI for related resource B and into the request body that is sent to resource B.

The last system that I worked with adapted and extended the templating from JSON Hyper-Schema (based on #52), which solved the problem of mapping into the URI, but did not have a solution for mapping into the request body. This turned out to be a major flaw in certain workflows, breaking what was otherwise a very nice HATEOAS-based programming experience and making developers do unnecessary work to pick data out of resource A and stuff it into the request for resource B. In truth, the relation from A to B already dictated that behavior, we just did not have a mechanism to capture that and use it.

Does that help at all?

[awwright]

JSON Schema gets its link listing form from the HTTP Link header. But other JSON-based formats follow an object based approach, see #124 that I just filed.

[Anthropic]

@handrews
Yes so pagination and my wizard example are essentially the same thing, and I follow your reasoning better now, so yes I see a need for it :)

"type define the params" is that in the current link you could have params "wahtever/{id}" but you seem to use the hrefSchema to define what that id would actually need to be or would be in the case of the relative pointer. That's what I meant by that.

@awwright You're awesome.

[handrews]

in the current link you could have params "wahtever/{id}" but you seem to use the hrefVars to define what that id would actually need to be or would be in the case of the relative pointer.

Correct. We were already implicitly defining it by the schema of whatever the "id" property was. This allows for two things: Changing where to look up "id" using a relative pointer, and allowing explicit specification of the schema. Even when there is an implicit schema based on the instance property used to resolve the variable, in the case of complex schemas with "allOf"/"anyOf"/"oneOf"/"not"/"dependencies", extracting the correct schema automatically to use with user input can be very challenging. So it makes it much easier to implement and more clear to read if supporting validation of user input requires an explicit schema.

Explicit specification of URI template variables accomplishes two other things: supporting user input for URI variables through the same mechanism as instance-based URI variables, and completely separating the URI variable mechanism from the request body mechanism.

[handrews]

@awwright I'm trying to stay out of #96 at your request, so could you comment here about how you would fit this use case into your views on method, schema, and API interactions in general? In your last #96 (comment) you continued to fit everything into a web form analogy using HTML, but that analogy has produced a specification for method, schema and href that cannot address my use cases.

[handrews]

PR #129 is one approach to implementing part of this (hrefSchema [EDIT: which was called hrefVars at the time] with only instance resolution supported, and getting rid of pre-processing). The correlating issue is #142 (which I split out of the rather messy #52 which is why it has a higher number than the PR).

@awwright suggested an alternate first step during an IRC combination, of hrefSchema used only to allow user input, deferring the question of whether to use relative JSON pointers or some other mechanism. In this approach, method, which at this point is just used to select between URI encoding or the request body, would be dropped. hrefSchema would handle URI input (not just query parameters) and schema would always apply to the request body. Anything else that method used to do or has been proposed to do would be reconsidered for another keyword (such as the allow keyword proposed in #73).

Just writing this here to keep any other readers up-to-date on various ideas of how to move this forward.

[Relequestual]

This sounds reasonable to me!

[Relequestual]

Was there some attempt to fix this problem? I'm not totally clear. Seems like it's still a problem.

[handrews]

#179 is a first step. There's more to it, but that's the part that is targeted for draft 6.

[slurmulon]

@handrews @Relequestual I would find this feature highly valuable for the reasons described above, generally:

1. It removes ambiguities over what sort of data is needed / expected in order to resolve the URI template into a URL
2. It decouples request body data from URL query parameter values to be sent
3. And, my favorite, resources concerning multiple entities (aka "nested resources") are gracefully supported:

"links": [
    {
        "rel": "self",
        "href": "/api/event/{event}/invitations/{email}",
        "hrefSchema": {
            "properties": {
                "email": "0/filters/email",
                "event": "0/filters/event"
            }
        }
    }
]

The old solution to this (and this is from memory and may be misguided due to lack of information) is where you have URI template variables that are associated with schemas by encoding a JSON Pointer into the template variable itself. I'm not sure if this is truly supported, so please ignore this example if so:

"links": [
    {
        "rel": "self",
        "href": "/api/event/{ 0%2Ffilters%2Fuser }/invitations/{ 0%2Ffilters%2Fuser }"
    }
]

So yeah, the new/proposed approach a much needed improvement IMO.

---

Regarding the state of things:

There's more to it, but that's the part that is targeted for draft 6.

What concepts need working on the most? What are the biggest concerns?

[handrews]

@slurmulon PR #228 (which was earlier submitted as #179 but bogged down as it took on too much) ONLY adds "hrefSchema", and "hrefSchema" may only use normal schema constructs- no relative JSON Pointers (with or without "$data").

It leaves everything else alone. If there is an "hrefSchema" and external input is supplied, that is used first, but for everything else the existing URI Template preprocessing and resolution from instance applies, and then "method" and "schema" work as they have been. Using "hrefSchema", "method": "get", and "schema" all together is a little weird, but the PR establishes a clear order of actions so the result is unambiguous.

@awwright is skeptical of both "$data" and JSON Pointers, an @epoberezkin has is own issues with JSON Pointers as well. I don't want to derail this issue into a discussion of those concerns: There are (or should be) issues devoted to each. If we can agree on all of the necessary tools, then we can fix the rest of this. It might take a few steps, though.

But reaching something that is the functional equivalent of the example in this PR is my overall top priority for hyper-schema.

[handrews]

I'm going to go ahead and close this, as the key point was addressed by splitting "schema" into "hrefSchema" and "submissionSchema". The only thing here that was not addressed is the idea of a mechanism for Hyper-Schema to define an automatic mapping of instance data as client input (instead of relying on the application to decide whether and how to fill out some or all of the input from the instance instead of requesting additional input).

$data, plus the separation of the two schemas, would automatically solve this unless I am really forgetting something. If $data is ultimately rejected, I think it will be more clear what, if anything, should be filed for an alternate approach.

In the meantime, we could use fewer gigantic confusing issues lying around :-)