Decision Proposal 004 - Versioning strategy

[JamesMBligh]

This decision proposal outlines a recommendation for how versioning would be managed for the standard. The detail for the proposal is in the attached PDF.

Feedback is now open for this proposal. Feedback is planned to be closed on the 24th August.
Decision Proposal 004 - Versioning Strategy.pdf

[c-j-hughes]

Full disclosure - I work for DA, and we are implementing the standards for a number of ADIs.

The proposed option 4 gives the most flexibility.

I think Alternative 1 is the better one. Offering more than one supported option, complicates the implementation, without giving any more power. It would require the application, and analytics to coalesce two different representations of the same thing, and we would need to determine precedence if two different values were provided.

Additionally HTTP caching is invalidated when the query string changes, but not an HTTP header. A browser, or a proxy may return the wrongly versioned content from a cache if an HTTP header is used, but would not for a query string version.

Not suggesting that an API versioning strategy is not essential, but a great many changes can be made to an API without breaking backwards compatibility with consumers of the API.

Is the proposal that all changes to the standards will increment the endpoint version, or only breaking changes?

To go a little deeper, in the context of #3, where mechanisms are defined for providers to extend the standard endpoints, is it assumed that providers will also version their extensions in the same way? So If I had an additional property on a response "da-foo", and I added "da-bar", should I increment a provider-specific version ?v-da=2 on the endpoint?

I would prefer to add the new property da-bar without versioning, since it wouldn't break any consumer.

Is the expectation that all versions would need to be supported indefinitely, or would newer versions declare sufficiently old versions deprecated? I'm thinking of the scenario where a new ADI enters the market in 5y time, would they have to implement all versions and variants of the standard released between now and then, as a client conceivably could still be requesting v1 stuff.

[gsblue]

@JamesMBligh does the recommended option(3) also cater for consumers having the flexibility to use different standard version in different endpoints ? e.g.

GET	http://www.bank.com.au/api/cds-au/v1/banking/accounts?v=1
GET	http://www.bank.com.au/api/cds-au/v2/banking/products?v=1

Or the flexibility is more for choosing different endpoint version under the same standard version?

Are the contracts for any given endpoint going to be defined by the standard version or endpoint version?

[JamesMBligh]

It would be the latter. The URI based version is for the overall standard with the implication being that this would only change when a significant standards change has occurred (maybe every few years). This could even imply moving beyond JSON for payload (assuming something else comes along.

When this occurred all of the end points under the new standard may be altered. The versions for the end points may even revert back to 1.

-JB-

[percolatorly]

As you’ve mentioned - we’ve taken a simpler implementation approach in the UK with block versioning the APIs. However, as you note, there are advantages with clarity on what versions are live, and coordinating API providers to update versions together. We also have the concept of a “resource group” which we version together. E.g., in the last release, the account information APIs were updated to v2, but the payment APIs remained at v1.

I like the flexibility of Option 4, but there will already be some complexity for third party clients to manage the implementation nuances for each API provider (on the same version). Third party clients may only want to code to a newer version of the API standard, when the majority of API providers support a newer version.
As a related aside - the bigger challenge for third party clients to overcome in the UK has been to understand what optional fields are populated in the response by an API provider (and under what circumstances).

Are you also looking to make a decision on backwards / forwards compatibility here? As an example, in the UK - we have the concept of a third party client registering a “consent object” with an API provider (in agreement with an end-user).
One of the main learnings here was that third party clients need clarity on how these “consent objects” (which control access permissions) are migrated across versions of the API. The worst possible outcome is that a third party client will need to re-register the “consent object” for every newer version of an API. This also ties into how access tokens will be managed across API versions.
Our approach for backwards / forwards compatibility has been specified here for the account information APIs:
https://openbanking.atlassian.net/wiki/spaces/WOR/pages/540541118/Account+and+Transaction+API+Specification+-+v3.0-RC3#AccountandTransactionAPISpecification-v3.0-RC3-ReleaseManagement

Would appreciate feedback on our approach as well.

[NationalAustraliaBank]

The proposal for End Point versioning makes sense over the block versioning approach followed in the UK, we believe this will allow for more targeted revisions over time.

• The “Standard Version” is unlikely to change as fast as the “End Point Version” yet the standard version has more prominence in the URI. We would like to promote the “End Point” version in the base path of the resource. This will help in making the implementation easier to implement, consume and iterate. Then every version change of the standard points to a new logical resource; without needing to use a query sting for the End Point versioning. i.e.:

http://www.bank.com.au/api/cds-au/<End Point Version>/banking/accounts 
http://www.bank.com.au/api/cds-au/v1/banking/accounts 

• With respect to the alternative proposals, there are pros and cons of using a query parameter Vs. a header parameter approach for the End Point versioning. Regardless of this decision we should in our opinion consistently follow one method. The consumer experience of having to dynamically choose per implementation or End Point version would create unnecessary friction on the scheme. We would prefer that query parameters are NOT used to identify the resource, rather query parameters will be used for things like filtering within a resource collection etc.

• Additionally, To cater for provider specific extensibility, it would be cleaner to make use of an optional header parameter for End Point versioning for the following reasons:

o v=1 of the End Point standard might be stable for two years, but specific data providers may want to innovate faster and provide more compelling offerings at a faster rate, additionally provider versions may change over time too. Meaning that one v=1 End Point might have multiple provider version implementations. As such providers will need a mechanism to differentiate their variations too. Allowing the scheme to foster an environment where innovation is possible within the constraints of any particular End Point major version. (i.e. data extensibility is allowed so long as no breaking changes are implemented as a result).
o Additionally, to implement Option 4’s end point Negotiation approach the consumer would need to supply a “minimum-version” parameter along with the End-Point version parameter; adding multiple query version parameters becomes clunky.
o It would be cleaner to use header parameters and response parameters to consistently communicate this negotiation. i.e. what is asked for Vs. what is received both appear as request and response header parameters respectively.
o Following this logic, the choice then becomes do we implement multiple separate headers? Or do we implement a versioned header object as a header parameter. E.g. as per the proposed version object below). This approach would also allow for the adoption of an explicit minor version to be added down the track). A fully elaborated version header block example:

x-version-block = 

{
	"standard-version": 1,
	"version": 4,
	"minor-version": 1,
	"minimum-version": 2,
	"provider-version": 9,
	"security-patch": 3
}

• We would like to propose an approach where only breaking changes increment the End Point version. In the proposed model every change will increment the version. The legislation talks about the potential for banks/(providers) to add enriched attributes to any particular End Point as a premium pay-per-use service. In this scenario the “proposed” version number would need to change, however this would not be possible if every change needs to increment the version.
o For example

{
	"transactionDetails": "shoes"
}

might be in a v=1 End Point, however a Data Provider might want to offer an additional field to the transaction as a premium service extending the v=1 standard with a “non-breaking” change e.g.

{
	"transactionDetails": "shoes",
	"x-NAB-recieptImageURL": "www.recieptimage.capture/3456787652s23"
}

In this example it would still be a compliant v=1 End Point however it would now allow each data provider to extend their own implementation.

o For this to be true we would need to agree on:

1. What “non-breaking” changes mean;
2. An appropriate mark-up for attribute level extension i.e. x--attribute;
3. How over time the things that are done consistently amongst providers transition into later versions of the official standard.

• There are times when it makes sense for the payloads of a resource which return a single object to be consistent with the payload of a separate End Point which returns the related collection; in this instance we recommend to follow the principle of upgrading their versions at the same time. Now, “as a developer I have confidence that there is consistency between logical resources and their versioned payloads”.

• From an implementation perspective we recommend to also version the Swagger/OpenAPI specification itsef? i.e. with a “timestamp”, “commit date” or “git tag”? The standards for a particular version can change with every GitHub commit, this means that changes might be missed based on the date each delivery team downloads the specification.

Having something like an x-standard-commit-date in the spec allows for a diff compare from the current version of the standard to the working specification each specific organisation/team is delivering against. i.e. a baseline reference point.

This would also allow for compliance to be baselined to a particular date of specification. Any changes to the standard post this date could then be considered a “change request”. I.e. Without this level of explicit control, it is possible that changes could be made to a version=1 End Point specification the day before the planned go-live date. How would we manage a scenario like this?

• To contain the complexity of delivery we propose a lock down on the specification version on/after a set date; from this date on we can iterate the standards further but allow a certain consolidation period for others to catch-up to the same consistent version before new changes are introduced. (Note: this means we can still iterate the standards, but the scope is locked)

• From a security point of view, can we also provision for a separate security/security control-set versioning strategy. The data security controls may need patching independent of the data version standards. There may come a time when certain implementations need to be certified to a certain patch level to retain compliance in the scheme. This would make it explicitly clear to consumers and providers of data when not to consume risky resources.

This provides a reasonable mechanism for Data Consumers not to accept the risk of collecting data from a Data Provider with an inappropriate declared security standard in place. This logic also follows for a Provider who is reluctant to share their data with a consumer with an inappropriately declared security control-set in place. In the case of data loss and liability dissputes these claims may serve as a mechanism to determine/resolve liability claims.

• Other things to consider: How will backwards compatibility be managed? how far back will we be required to support older versions? (n-1 & n-2 & n-x for example is only required to be supported)

[JamesMBligh]

Wow. Some excellent and extremely comprehensive feedback in these posts. Much appreciated. It will take some time to digest these comments and respond.

-JB-

[JamesMBligh]

I have now closed consultation on this decision. A recommendation incorporating feedback will be made to the Chair in due course. I'll also give a response shortly on my reading of the feedback.

-JB-

[JamesMBligh]

Thinking on the feedback received:

Min/Max Version vs Single Version
In general the feedback seems to support the inclusion of the possibility of version negotiation via the specification of a requested version and minimum alternative version.

Headers vs Query Parameters
Headers sound like the preferred option and aligns with HTTP usage for APIs where headers are used to request specific media types, etc. It also supports version negotiation in that the actual version provided can be nominated in the response headers. As there is no equivalent to query parameters on the response the use of headers would be more consistent. The cache impact of Query Parameters is a valid concern but caching can probably be manipulated appropriately using cache headers.

Optionality
For simplicity implementing both options doesn't seem supportable.

Consent Object
This is outside the scope of this specific decision but it is a good question. The Authorisation Granularity decision would be pertinent to this feedback. I would also suggest that consent should apply to the resource and not the version of the payload so any of the proposed solutions would work in the same way with consent.

Backwards Compatibility
A policy on backwards compatibility (and also compliance with new versions) needs to be determined but is not in the scope of this specific decision.

End Point Version in URI
Changing the resource location based on URI is less appropriate than headers or Query Parameters. This feedback is not aligned with other feedback.

Versioning of Extensions
This is an important omission from this decision proposal and some form of accommodation for versioning of extensions is definitely needed. Accommodation for extension versions will be added.

Breaking Changes
Since the standard is really a contract between consumers and providers the concept of incorporating breaking changes without a version change is probably unhelpful. A client that accommodates a non-breaking change will still have to code for providers that haven't supported that non-breaking change yet anyway. For simplicity the preference is likely to be that any changes to an end point that is already approved should invoke a new version.

Other feedback
Use of a commit date or security patch level is additional complexity that does not seem to give commensurate value. These could be considered in the future but seem like an over complication for the first standard draft.

Hope all that makes sense.

-JB-

[JamesMBligh]

The finalised decision for this topic has been endorsed. Please refer to the attached document.
Decision 004 - Versioning Strategy.pdf

-JB-