Obligation date-based standards

[commbankoss]

While we recognize the the efforts made by the regime to improve the engagement process, including the introduction of regular working groups, workshops, detailed minutes, and most recently the support portal, it has become quite difficult to interpret which parts of the standards have what obligation dates.

To mitigate these difficulties, we propose the standards be split into different files, each describing the obligations for a specific date/milestone.

[perlboy]

How is this possible when there are inline changes tied to existing obligation dates? This is related to the request to improve how documentation is published that was "being looked at" at the start of the year in #56 as well as related to #61, #41 and more recently #265, #266 (which, incidentally wasn't supported by maintenance iteration attendees), #310 and #311. Since then changes have continued to involve diffs in excess of 100,000+ lines and as of this afternoons implementation call the engineering team is "looking at options" although there doesn't appear to have been any formal kick off of requirements analysis so I guess "DSB knows best" is still ringing true.

In an effort to make it easier for participants to focus on delivering real things, Biza.io has started maintaining a redocs version of the Standards which while focusing (at this stage) on endpoint changes provides much smaller diffs, structurally tagged, that are 2% the size of the upstream along with OpenAPI 3 support (and Swagger backport), versioned schema, documented enumerations and, where possible, bounded value ranges.

Realistically, in the absence of an approach that understands the software engineering process, perhaps a more detailed obligation date table would suffice? For instance, something (very and definitely inaccurate!) roughly like this:

Endpoint	Version	Introduction Date	Mandatory Delivery Date	Deprecation Date	Retirement Date
/products	V1	2020-01-01	2020-07-01	2020-09-01	2020-10-01
/products	V2	2020-06-01	2020-10-01	N/A	N/A
/products	V3	2020-09-01	2020-02-01	N/A	N/A
/product/{productId}	V1	2020-01-01	2020-07-01	2020-09-01	2020-10-01
/product/{productId}	V2	2020-06-01	2020-10-01	N/A	N/A
/product/{productId}	V3	2020-09-01	2020-02-01	N/A	N/A
InfoSec Profile	1.3.1	2020-01-01	2020-11-01	N/A	N/A
InfoSec Profile	1.4.0	2020-10-01	2020-02-01	N/A	N/A

Of course, this doesn't protect against undocumented and inline version changes (of which there have been quite a few in successive versions) but at least a dev lead can give an update the project manager can get onboard with.

Finally, I'll add a note here @commbankoss that what you are proposing would be a lot easier (and indeed quite logical) had the versioning strategy been aligned to the standards version as I outlined, at the time on the Register GitHub (because attempts within the DSB team and externally after I left fell on deaf ears). I suspect it is only now that internal dev teams are experiencing the pain that those who have been operating as recipients have forecast and experienced since the beginning.

[perlboy]

Further on this, the PR for 1.5.0 changes is 112,822 additions, 2,240 removals. Not withstanding the infosec changes the actual payload endpoint changes is 167 additions, 128 removals representing a 99.74% reduction.

[perlboy]

There has been some related commentary on this at #328

[WestpacOpenBanking]

Westpac is supportive of the approach mentioned by Commonwealth Bank, but the use of endpoint versions significantly complicates an approach of this type. We are supportive of a more detailed obligation date table. We would also be supportive of allowing a review period for new standards versions in entirety prior to releases – this would help to ensure that documentation errors are avoided in future.

[CDR-API-Stream]

Thanks all, as discussed on the maintenance iteration call held on 7/10/2020 this issue should not be considered in isolation and the recommendations proposed in #328 warrant consideration as a holistic solution.

The core issue at the heart of the change request is to have greater certainty on obligations for each compliance milestone. In the context of a more comprehensive obligations table, the DSB would first like to elicit requirements and what dimensions (along with an agreed definition) should be expressed e.g Rules section/version, earliest implementation date, etc. @perlboy has provided a number of dimensions:

1. Endpoint or Section of the data standards
2. Section Version
3. Introduction Date
4. Mandatory Obligation Date
5. Deprecation Date (if applicable)
6. Retirement Date

@perlboy,

• is "Introduction Date" the date this was first introduced in the data standards or an optional "go early implementation date"?
• how do you consider "Deprecation Date" and "Retirement Date" are different?

[perlboy]

Hi @CDR-API-Stream,

• is "Introduction Date" the date this was first introduced in the data standards or an optional "go early implementation date"?

Introduced into Data Standards is probably most appropriate. I'm not aware of any requirement to limit the time by which a holder can go live (ie. "Next Day" would be great - if it was possible!).

• how do you consider "Deprecation Date" and "Retirement Date" are different?

Deprecation Date refers to when it was declared in the Standards as scheduled for Retirement. Retirement Date is the "last date of support" for Recipients. For Babelfish we immediately mark methods with a @deprecated tag, which starts doing strikethrough in IDEs giving developers an indicator of work to do but not breaking the code itself, and then drop the method entirely post retirement whereby compile errors would ensue.

[CDR-API-Stream]

Several changes have been made, or are planned to be made for v1.7.0, based on feedback to this issue and other similar issues related to the DSB's publishing processes:

1. The DSB has recently published a noting paper regarding DSB's standards promotion process. The DSB is now trialling a new staging area for standards so that changes can be drafted before they are formally adopted into the Consumer Data Standards.
2. The DSB is also reviewing its publishing tools and has recently trialled using redoc for publication of standards.
3. The DSB has launched a more interactive site for Consumer Experience guidelines: https://www.notion.so/Consumer-Experience-Standards-and-Guidelines-dffe42d39d4942c5b4f2c7612ba4f6e0.
4. The DSB has ported the Consumer Experience Data Standards from the previous PDF format
   to the Technical Data Standards page to be released as part of v1.7.0
5. The Future Dated Obligations table in the standards is being enhanced based on feedback
   arising from Change Request Obligation date-based standards #315.

This change request will be kept open so that further feedback can be tracked. The original poster may wish to close this issue or create a separate issue that specifically targets other areas for consideration as they see fit.

[nils-work]

As noted in the comment above, a number of changes in respect to this issue have been made since it was raised, including these items introduced in v1.17.0 -

• Obligation highlighting
• Obligation milestone windows
• Navigation for diffs and code examples

If there are no further comments or feedback on this issue, I propose it be closed on 30 June 2023.