Decision Proposal 164 - Endpoint Metrics

[JamesMBligh]

This decision proposal provides options for modifying the metrics endpoints in the standards to include metrics for specific industry resource endpoints.

The proposal for consultation is attached below:
Decision Proposal 164 - Endpoint metrics.pdf

Feedback is now open for this proposal. Feedback is planned to be closed on Friday 9th April 2021.

[ryderwj]

The proposal suggests possible usage scenarios of API operation level metrics, however these are not specifically linked to a firm requirement.  As the ecosystem is still in its infancy and there has been limited application of information from the existing Metrics endpoint, recommend Option 4 - Dont implement endpoints Metrics and keep current metrics API until a firm use case exists. 

Should a use case be determined, the preferred option is Option 2, providing specific resource level endpoint and maintaining the summary level metrics as is. 

It is noted that Availability information is specified in the endpoint level metrics. This could imply a much more granular approach to availability monitoring for data holders than is currently required under the existing non functional requirements, driving additional cost into implementations.

[commbankoss]

Commonwealth Bank recommends Option 4 (Dont implement endpoints Metrics and keep current metrics API). To enable us to provide more specific feedback, Commonwealth Bank would be grateful if DSB could provide further explanation as to the specific objectives and requirements of these proposed changes.

[WestpacOpenBanking]

Westpac welcomes the opportunity to comment in relation to the inclusion of endpoint metrics.

• Aggregated metrics should be retained - Individual endpoint metrics have limited value for assessing overall conformance to non-functional requirements. As an example the peak TPS for a transactions endpoint and an accounts endpoint may or may not occur at the same time - the statistics for each endpoint aren’t sufficient for the ACCC assess whether non-functional requirements have been met.
• Option 1 and Option 3 are very similar in terms of the technical impacts required to implement them and the data available to consumers.
• Option 2 may require a discovery mechanism as the available endpoints will vary by participant. This may be disadvantageous if Get Metrics is eventually made public - browser based clients might have additional latency before displaying data if they need to make a synchronous call to determine which endpoints are available before accessing the relevant metrics.
• The proposal doesn’t address endpoint versioning – the DSB may wish to consider if metrics are to be included for different endpoint versions. Additionally, it would be preferable to define the standard in a way which avoids new Get Metrics versions when new endpoint versions are released.
• It should be made clear whether the intention is to include information security endpoints as part of any endpoint statistics.
• If it is later to be proposed that endpoint metrics are made public, then the security implications of providing additional knowledge to threat actors should be considered.
• Due to existing delivery priorities, we propose that any change to get Metrics API occurs post the Feb 2022 release for phase 3 products for non-majors.

[anzbankau]

Our recommendation is "Option 4 – Don’t implement endpoint metrics and keep current metrics API as is".

Similar to other participants this is our preference as there are other delivery priorities, and our view is the current level of information provided is sufficient at this point in time.

[EnergyAustraliaCDR]

Hi All,
Feedback from EnergyAustralia below:

EnergyAustralia supports Option 4 – Don’t implement endpoint metrics and keep current metrics API “as is” at this stage given the level of maturity we are at and the information shared to-date.
There are still details that need to be standardized from an API governance perspective by DSB, which can influence the details of our proposed changes as below:

Of the 3 proposed options to provide metrics at the endpoint level, Option 2 - Create a new metrics endpoint API to retrieve individual endpoint metrics is supported with changes suggested below.

Proposed metrics API URI for option 2 is GET /admin/metrics/endpoint/{operationId}

We recommend to make use of the URI standard GET /admin/{operationid}/metrics/ as this provides a generalisation for all future APIs to add to (e.g. /admin/getcustomer/metrics) and also provides features to add additional details in future to add-on for e.g. GET /admin/{operationid}/metrics/{parameter}?daterange=? (and substitute {paramater} with the fields mentioned in Decision Proposal 164).

Another advantage is having the platform level metrics API at the root-level i.e. GET /admin/metrics which provides options to enhance to add additional features to list the APIs that support individual API metrics operations along with Hyperlinks if required as well.

[NationalAustraliaBank]

NAB recommends Option 4 (Don’t implement endpoint metrics and keep current metrics API as is) for the similar reasons mentioned above by others.

NAB understands that the DSB is interested in the likely lead time for implementation for each of the options in the paper. Options 1, 2 and 3 require considerable development effort by NAB, with Option 2 being the highest.

The implementation lead time depends on the amount of other changes going on at the same time. For example, DH’s have significant work already underway to meet the November 2021 obligations so we would appreciate consideration be given to the current workload on DH’s when determining priority of this change.

[CDR-API-Stream]

Thank you to everyone who's put forward feedback. Consultation has been closed and feedback will now be reviewed and considered.

[CDR-API-Stream]

A response is being formulated for this consultation. It has been delayed as we obtain internal feedback as to the requirements for ongoing regulation and enforcement. We will endeavour to respond as soon as possible.

[CDR-API-Stream]

The feedback on this proposal has been reviewed. This response has been delayed while internal conversations were undertaking with the CDR operational, planning and enforcement teams to discuss the feedback and determine the priority of obtaining this data in near real time.

The feedback here has been overwhelmingly supportive of Option 4 (Don’t implement endpoint metrics and keep current metrics API as is). It is noted, however, that the feedback has only been provided by data holders and that a number of data recipients have reported inconsistent performance for individual end points and that this is a concern for the overall integrity of the CDR regime and data on the extent of this issue is currently difficult to obtain.

To address these potentially conflicting positions the DSB is intending to proceed as follows:

• Support the clear feedback proposing the adoption of Option 4
• This means that this consultation will be closed with the status of No Decision Taken
• Incorporate the possibility of obtaining data holder endpoint performance from accredited data recipients under a proposal for an ADR Metrics API as an alternative solution for the operational needs articulated

[ACCC-CDR]

Metrics data provides the ACCC with valuable information to monitor the ecosystem and inform our compliance activities. However, the currently mandated metrics are high level, and therefore there are clear opportunities for improvements.

We are of the view the additional metrics outlined in this decision proposal will meaningfully improve the data available to the ACCC, and will allow us to proactively understand issues and undertake activities to improve the health of the ecosystem. It will also improve the value of data the ACCC may publish on the health and usage of the ecosystem.

Our preferred option is Option 3 (i.e. modify existing Get Metrics API to include endpoint metrics), which involves an update to the GetMetrics schema. This would involve a similar amount of technical changes to Option 1 (i.e. create a new metrics endpoint API (bulk)) and we consider it is better to update the existing endpoint which involves reusing existing configurations, which we expect will reduce the complexity of implementation for data holders. We acknowledge the benefits of the preferred option outlined by the DSB, and while we agree that the recommendation in the Decision Proposal will provide flexibility to invoke Get Metrics for a single API or return metrics for all APIs is ideal, we don’t consider the increase in flexibility necessarily justifies the additional build and implementation costs likely to be involved.

We note data holders’ views regarding the implications of this decision, in terms of the cost in both time and resources. Therefore, we would support providing additional time for data holders to comply with these obligations. We note that this presents an opportunity to provide long-term clarity to current data holders, and data holders who will join the ecosystem as part of the expansion into the energy and other future sectors.