Decision Proposal 010 - Standard HTTP Headers

[JamesMBligh]

This decision proposal outlines a recommendation on the standard HTTP headers to be used for the API standards. Note that security headers have been specifically left out awaiting the working of the Security Working Group.

Feedback is now open for this proposal. Feedback is planned to be closed on the 14th September.
Decision Proposal 010 - Standard HTTP Headers.pdf

[perlboy]

Starting point here: https://openbanking.atlassian.net/wiki/spaces/DZ/pages/127009221/Read+Write+Data+API+Specification+-+v2.0.0#Read/WriteDataAPISpecification-v2.0.0-Return&ErrorCodes

[JamesMBligh]

Thanks for the tip

-JB-

[da-banking]

Versioning Headers
As noted on #11 , we think that the versioning negotiation could be considered a subset of content negotiation.

For example: Accept: application/vnd.data61.v2+json;q=0.1, application/vnd..v5+json;q=0.9. Here the data consumer is requesting v2 of the Data61 APIs, prefering v5 of the vendor specific API from vendor .

By negotiating versions with this mechanism, the response in the case where the server cannot find a good match would be 406 Not Acceptable.

This approach would eliminate the need for the custom headers - x-v, x-min-v, x-v, and x--v - instead relying on built in features of the HTTP specs to support this negotiation and returning standard errors if negotiation failed.

Throttling Headers
Really a question on throttling since the Retry-After response header is specified. The Farrell report suggested that:

Recommendation 5.10 – access frequency
The Data Standards Body should determine how to limit the number of data requests that can be made.

@JamesMBligh - it would be great if you could clarify your interpretation of how throttling will work.

We will leave our views off this item as its tangential to the question of what standard headers are included. However maybe another Decision Proposal for throttling would be appropriate?

[JamesMBligh]

I understand the point on the version header but having separate custom headers for versioning will be a lot easier for provider and client implementations as well as readability of the requests during troubleshooting and testing. Custom headers are also valid parts of the HTTP spec so we’re not really stepping outside of standards.

With regards to throttling, I picked up the headers and errors from the U.K. spec verbatim as they seemed reasonable and not constraining in any way. I will add a decision to the list for NFRs in general as the topic does recur in questions.

Thanks for the feedback.

-JB-

[da-banking]

@JamesMBligh - your points on custom headers are good.

As a guiding principal, we would prefer the design to use established features of the HTTP standard, to re-inventing new ones. Inventing new ones is good if the existing standards fall materially short.

Versioning Headers
There is considerable flexibility in the existing Accepts header. An alternative approach to vendor specific mime types is to use the accept-params.

For example, Accepts: application/json;v=3;foo-v=4;bar-v=5 would allow a client to specify a standard header containing all of the versions it supported. The server implementation would only be concerned with the standard v and the specific data provider v associated with it. If the server couldn’t respond to this, a 406 would be appropriate.

On success, the server for data provider bar might respond with Content-Type: application/json;v=3;bar-v=5

Accepts: application/json would be synonymous with Accepts: application/json;v=1

If the client wanted to specify a min version too, to avoid the 406, this is supported: Accepts: application/json;q=1.0v=3;foo-v=4;bar-v=5, application/json;q=0.5v=1;foo-v=2;bar-v=3. Here the q=1.0 stuff is preferred, but q=0.5 is an acceptable minimum. Any versions lower than those specified in q=0.5 would be a 406 as the client is stating it cannot support anything lower.

Another advantage of the Accepts approach is that all supported headers are known now at design time. This allows us to set the CORS Access-Control-Allow-Headers value to Accepts, and then clients can send a standard Accepts header to all data providers. If custom data provider headers are being added all the time, then data providers will either have to accept all headers which would be a security concern, or the client would have to customise the request for each data provider.

Compression Headers
It would also be good to allow data consumers to send other common, optional request headers like Accept-Encoding. Gzip compression would be good for data consumers, data providers and PSUs. This would also require the server to respond with Content-Encoding if it supported the requested compression.

Caching Headers
Similarly optional inclusion of the ETag response header and the corresponding If-None-Match request header would be a good way for data consumers to cache data and keep it current while not transmitting large payloads from data providers unnecessarily.

[ajmcmiddlin]

The Queensland FP Lab (Data61) agree with @da-banking. We believe that standard HTTP headers should be used when possible and that the Accept header is the appropriate place for version negotiation. We don't have an opinion on the format of the header, other than that it should comply with HTTP standards and include all relevant versioning information.

[deboraelkin2]

I agree with @JamesMBligh. I understand that using the Accept header to negotiate versions is closer to the standard, but I'm also a strong advocate of easy to understand APIs. So even though it is not a standard header, I think using x-v and x-min-v contributes to the overall "readability" of the API specification and, as it's been pointed out before, makes it easier to test and troubleshoot.

[tonymorris]

I am of the opinion that using an Accept header for versions results in an API specification that is easier to read and debug. The compliance and appropriateness for version negotiation in the header makes tools and libraries easy to write and use. The argument that using a non-standard header will be easy seems to rest only on the idea that parsing, and subsequently reader a header implies some penalty (please correct if wrong, as I do not understand any penalty otherwise).

Parsing is relatively trivial. See, for example, the Data61 parser exercise, which is done on day 2 of an introductory programming course and the benefits of compliance and subsequent tools are too great to forgo in light of this.

• https://github.com/data61/fp-course/blob/master/src/Course/Parser.hs

This model of parsing, which goes back many decades, has even been implemented (and is still supported) in Java:

• https://github.com/functionaljava/functionaljava/tree/master/core/src/main/java/fj/parser
• https://github.com/jon-hanson/parsecj

[JamesMBligh]

My thoughts on the feedback so far are as follows:

Additional Headers
It is a well made point that this standard should not be exclusive and that other standard HTTP behaviors should be accommodated and expected by a consumer. It was not the intent of this proposal to preclude the usual compression and caching headers. It was also not the intent to mandate there use in a specific way. This should probably be left to the data providers to optimize for their own implementations. Language to that effect will be brought into the final proposal unless there is additional feedback that is counter.

Accept Header
I disagree with this approach. The Accept header is a well defined part of the HTTP protocol designed for content negotiation as stated. While the content of this header is flexible it was not designed to carry two independent sets of information. The content type of the payload is entirely separate to the schema version. I also do not see the negative connotations of the use of custom headers. They are an entirely valid part of the HTTP spec and we will be using them in this standard for security anyway. I would see the overloaded use of Accept in this way, when compared to the use of simple custom headers, as a less compliant approach to HTTP rather than a more compliant approach.

CORS
The feedback around CORS and allow-headers is interesting. From a provider perspective this is of minimal concern as they will only support one or two extension headers at most and this support will be system wide rather than end point specific. From a consumer perspective the argument that it will be difficult to customize requests per provider is noted but somewhat offset by the fact that the additional fields obtained (that will be entirely provider specific) will require special handling regardless. I would also note that the argument that parsers are easy to build could be just as easily applied to the creation of a series of conditional statements to include or exclude a provider specific header. I would recommend conditional statements be included in day 1 of the intro to programming course ;)

A final observation regarding CORS is that it is mainly relevant for direct API calls from the client. Some of the arguments presented are relevant for calls to the APIs directly from the browser or mobile client but less relevant for the more common use case of server to server calls supporting a long running authorisation. It is not yet clear if an implicit model will even be supported during the first phase. That said, the decision could go either way and could change in the future so it is not a point that would be definitive.

-JB-

[JamesMBligh]

Feedback will close on this item tonight and the decision will likely align with my previous comment pending additional feedback this afternoon.

-JB-

[NationalAustraliaBank]

With respect to Accept Headers we agree with the statement below, i.e. we should in general not overload the meaning of data. This can only lead to confusion; and opens the door for more overloading in future.

The content type of the payload is entirely separate to the schema version

For more advanced contract negotiation use cases the ability to extend and correlate request and response headers will be invaluable.

Can we assume that:

1. Standard browser headers are supported such as User-Agent, Origin and Referer

2. Standard CORS headers will be supported for browsers:

Access-Control-Request-Method

Access-Control-Request-Headers

Access-Control-Allow-Origin

Access-Control-Allow-Methods

Access-Control-Allow-Headers

Access-Control-Max-Age

3. There will be a consistent defined set of headers for throttling, rate limiting, and cache control.

We recommend including:
4. Every request and response will have a unique correlation id:

x-Correlation-Id

Some useful reading on the topic:
https://blog.rapid7.com/2016/12/23/the-value-of-correlation-ids/

[bazzat]

The consensus of the ABA Online Banking Technical Working Group is that we collectively support Data61's recommended base set of headers. Additional comments from WG members regarding expanding/clarifying the recommended set have been posted previously in this thread.

[TKCOBA]

COBA supports the adoption of a standard HTTP header format, as this would be more consistent with the API Principles (particularly Principle 2: APIs use open standards). COBA notes that there are a number of common web application attacks that focus on header information (such as cross-site scripting) and, in this context, suggest that this decision proposal also be considered by Data61’s Information Security Working Group.

[JamesMBligh]

I have now closed consultation on this decision. A recommendation incorporating feedback will be made to the Chair in due course.

-JB-

[JamesMBligh]

The finalised decision for this topic has been endorsed. Please refer to the attached document.
Decision 010 - Standard HTTP Headers.pdf

-JB-