Decision Proposal 166 - CX metrics for Data Holders

[CDR-API-Stream]

This decision proposal provides options for including consumer experience metrics in the metrics endpoint.

The proposal for consultation is attached below:
Decision Proposal 166 - CX Metrics.pdf

Feedback is now open for this proposal. Feedback is planned to be closed on Friday 18th June 2021.

[nils-work]

Hi @CDR-API-Stream

Just a few questions/observations -

Option 1 - General CX Metrics
Authorisation - Number of Failed Authorisations - Total number of failed Authorisations for the period.

• Is this referring to any technical errors in the authorisation flow? (The list of expected 'error codes' did not appear to be linked in the proposal PDF and may clarify a few questions.)
• Would it cover cases where the user could tap a button to 'Cancel' the flow and return to the ADR, or would that be classified as an abandon?

Authentication - Number of Authentications - Total number of authentications attempted for the period.

• Would this be the number of times an ADR passes a 'consumer' into the authentication journey, or the total number of OTP login attempts?

Authentication - Number of Failed Authentications Total number of consents requiring authentication that failed.

• I believe all consents require authentication?
• Similar to above questions about errors, is this expected to be total login attempt failures, or count of oauth error codes returned to the ADR?

Authentication - Number of Abandoned Authentications Total number of consents requiring authentication that were abandoned.

• Is this to be inferred from the total (minus success and fail), or will there also be a standardised way, or error code to capture an Abandon? (e.g. when cancelling and returning to the ADR?)

Option 2 - Detailed CX Metrics
All areas - Abandoned Status

• The definition of the abandoned metric may need to be clarified in order to determine the 'Time to complete' metric for an abandoned flow.

Thanks

[WestpacOpenBanking]

Westpac welcomes the opportunity to comment on Decision Proposal 166: CX Metrics for Data Holders.

We agree with the assessment that CX metrics would be valuable to understand the user experience, adoption and preferences in the CDR ecosystem and are supportive of Option 1 as a way to do this.

We suggest that this proposal could be improved by:

• Ensuring definitions for each metric are unambiguous. For example:

  • Does ‘Number of New Authorisations’ include re-authorised/amended consents?
  • What does ‘Active’ mean in metric ‘Number of Active Authorisations’? Does it mean not revoked?
  • Does ‘Number of Withdrawals’ include revocations via means other than dashboard?
  • Does ‘Number of Failed Authorisations’ include customers not completing an authorisation? Is it related to a technical issue?
  • Are ‘Failures per error code’ regarding technical issues? What is the final list of error codes?
  • Does ‘Average time for Authorisation’ cover the time for authorisation end to end (ADR and Data Holder steps) or just the steps on the Data Holder end? Are failures included? Is a median better?
  • What are the scenarios for ‘Number of Failed Authentications’?

• Advising what time period the metrics need to cover. We recommend that data frequencies (such as weekly, monthly, 6-monthly, etc) should be chosen based on the intended usage of data.

• Total Number of Eligible Users should be non-mandatory. Many data holder implementations determine eligibility “on-demand” at the time of user or third party actions, for example when granting consent or sharing data, and not store it in a way which readily allows the computation of an accurate metric. This is because eligibility could change at any point in time. Computing this on a regular basis for every user will cause a large system overhead.

Considering the current and intended usage of the Get Metrics API, only historical data should be provided. That is, the “CURRENT” period should not be supported for this endpoint. Near real time metrics likely to be more costly and time-consuming for participants to implement.

In addition to understanding the CX metrics from the Data Holder perspective, we recommend an equivalent set of CX metrics to be implemented by ADRs as these would cover the end to end flow (e.g. authorisation flow is initiated and completed on the ADR end).

We also remark that prior to Option 2 being implemented, there should be a trial period (minimum of 3 months) of Option 1 by Data Holders and ADR’s to enable a better understanding of what other metrics may be beneficial. We also note that having to provide detailed metrics as proposed by Option 2 may negatively impact performance of implementations due to the increased requirements for audit logging.

We agree that Option 3, manual reporting is not a timely or scalable solution, however we do support it as secondary option for sharing metrics.

[commbankoss]

Commonwealth Bank recommends against implementing CX Metrics at this point in time, as to the extent that the proposed benefits have been articulated, it is our view that they would not outweigh the cost and effort required to implement this change. We recommend this proposal be revisited following a post-implementation review after the full implementation of the current CDR regime which will enable the future design of the CDR to benefit from data-driven findings.

[CDR-API-Stream]

In response to community request, and the fact that some of the links in the PDF seem to be inactive, we have extended the consultation by a week to the 18th June.

[anzbankau]

This proposal suggests that “CX Metrics would be invaluable for reporting and dashboards on the user”. Whilst we are very supportive of understanding the details of the consumer experience in our capacity as a data holder, the proposed CX metrics go well beyond the CDR quality of service requirements and firmly into the customer research domain.

Our position is that mandating the provision of this data is beyond the scope of the CDR and the accompanying standards and is a substantial impost upon and with very little benefit to data holders. As such we are not supportive of mandating CX metrics to support research and we recommend that such activity be undertaken by the Data61 CX workstream, in line with similar customer research activity undertaken to date.

[RobHale-RAB]

In terms of the overall suggested approach, Option 1 seems to be the right approach to take for the time being. Currently, there are insufficient participants in the ecosystem to fully understand and confidently commit to an appropriate and final scope of detailed CX metrics.

Some good points have been raised above ☝ (from @nils-work and @WestpacOpenBanking) requesting clarification on definition of failed authentication. As always, when we get into the detail, it gets more complicated than we initially thought. There seem to be three potential phases of authentication that might result in a failure:

• There could be issues where the customer wants to authenticate, but is not eligible, perhaps they're not registered for OTP or they don't have digital channel access. These scenarios would cause authentication to fail before a technical attempt has been made, but in CX terms, it is arguably still a failure.
• Then there is a middle piece where the DH is unable to send the OTP. Perhaps it can't be sent, there is some technical issue, or it is sent but isn't received in time, or at all. Some of these can't be detected by the DH, some can.
• The final piece is where a OTP was sent by the DH, and it was received but the outcome wasn't successful - perhaps the consumer entered it incorrectly, or it was deemed invalid, perhaps it wasn't entered in time, or not at all.

There will inevitably be similar issues with authorisation so just echoing the sentiment from others that we'd need really clear definitions of what success and failure are.

Given the majority of the direct CDR CX is delivered by ADRs, if the intent is to understand and enhance the CX, it will be important for ADRs to also supply an equivalent set of CX metrics. By providing their own metrics, ADRs may also shed light on issues not visible to DHs, such as when an attempted consent initiation with a DH has not been successful (maybe the DH is offline or the endpoint is not operating as expected).

The proposed metrics will help understand the outcome of individual steps in a customer CDR data sharing journey such as authentication or authorisation, but it won't be possible to associate that step with prior or subsequent steps. For this reason, and if the objective is to understand and improve the overall CX, then the following 2 consent success metrics could be useful:

• Percentage of consents initiated with a DH, resulting in a successful authentication
• Percentage of consents initiated with a DH, resulting in a successful authorisation

This would reveal if consumers drop-out (for some reason) at the authentication or authorisation step. The use of percentages will hide data volumes, so ideally the percentage would be derived from separate reported numerator and denominator figures for each metric.

Overall, having metrics available by ADR, DH and ADR software application will be very useful.

A final thought. Given that there is some debate about the value or need for consumer dashboards in the current form. Could it be worthwhile including some basic consumer dashboard usage statistics in this scope? Ideally they should be provided by both ADRs and DHs.

• Number of consumer dashboard logins
• Number of times data sharing was viewed via the consumer dashboard
• Number of times data sharing was stopped via the consumer dashboard

Where data sharing was stopped, it would be interesting to know the software application. This might highlight those applications where consumers were not comfortable maintaining ongoing data sharing.

[NationalAustraliaBank]

NAB supports the comments and recommendations by CBA and ANZ above.

[SelenaLiuEA]

Hi All,

EnergyAustralia's feedback is below.

We support Option 1 – General CX Metrics - to provide a summary of Consent and Authorisation metrics for each Data Holder. However, there are still details that need to be standardised from an API governance perspective by the DSB.

Option 2 will significantly increase the breadth of data being reported on every transaction and has the potential to materially increase costs to Data Holders. EnergyAustralia questions the value of reporting on this data for each Accredited Data Recipient and Data Holder on a step-by-step basis for every transaction. We ask the DSB to consider the value of this data vs the cost of implementing, testing and reporting on these metrics.

The manual option 3 would require almost as much work in order to produce the metrics as Option 1. Given the level of maturity of the energy CDR and the likely changes on the CX workflows, we also question whether designing and implementing these CX metrics now is appropriate timing.

[CDR-API-Stream]

A response is being formulated for this consultation. It has been delayed as we obtain internal feedback as to the requirements for ongoing regulation and enforcement. We will endeavour to respond as soon as possible.

[CDR-API-Stream]

Thanks everyone for the feedback. We have taken some time to respond in response to the comments querying the value of such metrics being obtained via API. After internal consultation with operational teams it would appear that there is significant value to understanding the consumer experience of the CDR (whether this is visually presented or used mainly for monitoring the performance of the regime).

The bulk of the feedback indicates that option 1 would be preferred. The DSB understands the argument that manual reporting would be costly and that adding too much detail too early may result in unwarranted costs.

The feedback that the existing proposal is unclear, needs to address time boundaries and potentially should include dashboard metrics as well has been noted.

As a result of this feedback a new version of the proposal is being developed focusing on option 1 with an intent to be more specific in the proposal for a new CX metric API. This will be posted to this thread for continuity and for further consultation.

[ACCC-CDR]

Consumers are at the heart of the CDR and the ACCC supports the DSB’s proposal to iteratively increase understanding of the consumer experience.

We support Option 1, with a slight alteration to include the aggregation of the three metric areas per software product ID. This will enable us to identify the brands which are experiencing the most issues.

We note data holders’ views regarding the implications of this decision, in terms of the cost in both time and resources. Therefore, we would support providing additional time for data holders to comply with these obligations. We note that this presents an opportunity to provide long-term clarity to current data holders, and data holders who will join the ecosystem as part of the expansion into the energy and other future sectors.