API Uplift for Data Holder Multi-Sector support

[CDR-API-Stream]

Description

Data holders will need to decide how to structure their brands in the CDR Register to cater for more than one sector. The introduction of energy raises the question on how data holder brands should support different sectors and how the dynamic client registrations generated should map to different sectors

Area Affected

Register APIs will need to be structured to expose single or sector agnostic discovery and status

Dynamic client registration specification will need to define how the registration process will support single and multi-sector scenarios

Change Proposed

Provide requirements to data holders on how a data holder brand will cater to both single and multi-sector scenarios

[CDR-API-Stream]

The following describes the changes proposed and constraints for this issue.

Please refer to #425 to address multi-sector support for Data Recipients

Introduction:

Data holders utilise CDR Register APIs to obtain information about data recipients in the ecosystem.

With the introduction of the energy sector to the CDR, there is a need to redefine the APIs to simplify alignment to multi-sector support.

Along with these changes, there is an opportunity to address outstanding work to align CDR Register APIs to the conventions defined in the Consumer Data Standards. These changes are to allow for the APIs to be more usable and align their usage across multiple sectors.

Background:

Currently, data holders retrieve data recipient information from the following endpoints:

• GetDataRecipientsStatus
• GetSoftwareProductsStatus
• GetDataRecipients

These APIs have <industry> within the URI path, inferring this call filters by industry / sector. However, data recipients do not currently align with a sector at the entity level, rather they have sector-specific scopes assigned in the CDR Register and this is reflected in the generated SSA.
Therefore, it is not useful to retrieve data recipient and software product information by industry.

This also is reflected in participant statuses. It is expected that a data holder will be able to observe a data recipient and associated software products move through their various states. However, if a data recipient / software product can jump between sectors, data holders will lose this visibility

Proposed Changes:

Issue a new set of CDR Register APIs

A new set of CDR Register APIs will be exposed to allow new builds to align to the new CDR Register standards while keeping the original set maintained until a reasonable deprecation schedule has been decided.

The new set of CDR Register APIs will be exposed targeting the energy sector and allow new builds to align to the new CDR Register standards. The previous set of APIs will be maintained until a reasonable deprecation schedule has been decided.

The new set of APIs will have the following differences from the old:

1. Remove {industry} from the URI path for the following APIs:
   • GetDataHolderStatus
   • GetDataRecipientsStatus
   • GetSoftwareProductsStatus
   • GetDataRecipients
   • GetDataHolderBrands
   • GetSoftwareStatementAssertion

2. Structure of the response aligns to data, links, and meta requirements as defined in the CDS. Note that this does NOT include the introduction of pagination

3. Move x-v header from optional to mandatory

4. Support x-min-v header as per CDS

5. Specify If-None-Match and ETag request and response headers in the public APIs

6. Add energy value to industry enumeration

7. Where sector is returned in the response field will move to an array to reflect multiple sectors

8. Align response error codes to the CDS
   • 400 for missing required header / Invalid Version
   • 406 for unsupported version
   • 304: Not modified to align to the ETag pattern
   • 404: Add support as per CDS to handle missing resources

To reiterate, these changes would be implemented against the new set of APIs, with the previous set retained so current implementations remain unaffected. However, an expectation would be set that data holders would eventually migrate to these APIs and a reasonable deprecation schedule would be decided.

Considerations:

Versioning

Options:

1. Update the major version of the endpoint to V2: https:// <register-base-path>/ cdr-register / v2 / data-holders / brands
   This approach is deemed inappropriate as this value represents the version of the standards, not individual or sets of endpoints.
2. Reusing the current APIs and simply setting the industry parameter to all and increment the endpoint version number: https:// <register-base-path>/ cdr-register / v1 / all / data-holders / brands
   This couples the endpoints together and does not allow for both sets of endpoints to be separately maintained
3. Reissue APIs without the industry parameter: https:// <register-base-path>/ cdr-register / v1 / data-holders / brands
   This has the result of the new APIs having unique paths and not clashing with the old. Documentation will be updated to reflect that the new APIs should be targeted for new builds and the old APIs will be maintained until deprecated.

Our recommendation is option 3.

Deprecation Schedule

The intent of these changes is to make them available for energy builds so no changes are imposed on the banking sector. However, consideration would need to be made to determine when the old APIs are to be deprecated. Setting expectations early on what a reasonable timeframe looks like would aid participant development roadmap planning.

Add an unauthenticated GetDataHolderBrands endpoint exposed as a public API

Rationale:

The GetDataHolderBrands API definition evolved to meet the need to store additional metadata for the discovery and authentication of Data Holder Brands.

The GetDataHolderBrands API allows for the following:

1. Discover Data Holder Brands within the CDR
2. Retrieve metadata for the brands to render in client applications (such as name, description, logos etc)
3. Map which legal entity the brand belongs to
4. Retrieve the status of Data Holder Brand as set by the Registrar
5. Retrieve URI information to allow reference to:
   a. OIDC Discovery Configuration Document
   b. Public APIs including PRD
   c. Admin APIs
   d. Extension APIs
   e. Public Website
6. Retrieve Authentication JWKS for outbound authentication for DH to ADR calls (e.g. CDR Arrangement revocation)
7. Detect change over time for the above information

This endpoint is only available to authenticated data recipients. This design approach has resulted in the following:

1. Increased build effort for data recipients to access this API
2. Less scalability when this API is used for change detection through polling
3. Public clients do not have access to this information. This means they are unable to derive the location of data holder PRD, status, and outage endpoints from the CDR Register

Allowing access to these CDR Register endpoints to authenticated users only adds a layer of obfuscation. Other security mechanisms are present to protect these data holder endpoints, therefore the benefit of securing this information behind an authentication mechanism is being re-examined.

Allowing unauthenticated access at an equivalent endpoint simplifies the usage of this endpoint for change detection. There are additional benefits to exposing this endpoint as a public API and using the same pattern as GetDataRecipients. This simplifies usage and allows clients to leverage ETag and CDN functionality.

Proposed Changes:

The following changes are proposed:

1. Create a new GetDataHolderBrands endpoint returning the same payload in time for energy
2. Move this endpoint out of the secure domain, removing MTLS and authentication requirements
3. Move this endpoint into the public API domain, allowing the API to be accessible via CDN and leverage ETags
4. Add industry filtering behaviour:
   • Default: All data holder brands will be returned
   • Query parameter filter: allow filtering by industry sector
5. Remove paging, aligning the response definition to other public APIs

Options:

1. Create an unauthenticated Data Holder Brands API returning the same content as the current authenticated API. The authenticated version of the API will be deprecated once a schedule has been determined

2. Create an unauthenticated Data Holder Brands API returning a subset of the content currently returned in the authenticated API. Security endpoint information would NOT be included in this API. Both the authenticated and unauthenticated APIs would be maintained.

Considerations:

What are the consequences of making this information available publicly?

Add Constraint to Data Holder Brand to support one industry only

Rationale:

With the introduction of the energy sector to the Consumer Data Right, the relationship between data holder brands and sectors needs to be defined.

With the initial roll-out of energy, the current model will NOT change and data holders which span multiple industries will be required to create one brand per industry on the CDR Register.

Further design work is scheduled to examine how multiple industries may be facilitated by one brand on the CDR Register. This work will happen in parallel but is not guaranteed to be complete and available in the initial energy rollout timeframe.

Proposed Change:

Add documentation describing the constraint of one sector per brand when configuring the Data Holder Brand in the RAAP

Considerations:

1. What is the impact of this decision on future energy data holders?

The current assumption is the initial data holders in the energy sector will be unaffected by this constraint as their initial rollout will not span multiple sectors.

2. When will multi-sector support per brand need to be available?

[CDR-API-Stream]

Add an unauthenticated GetDataHolderBrands endpoint exposed as a public API
Further work is required to consider these changes. Issue #444 has been raised to continue this work post maintenance iteration 9.

The rest of the change request will be addressed through maintenance iteration 9.

[CDR-API-Stream]

Versioning

Upon review, it has been determined that the benefits for option 3, allowing two sets of APIs to be maintained in parallel until one is deprecated, can be equivalently achieved using option 2, incrementing the endpoint version numbers, and allowing multiple endpoint versions to be made available in parallel.

Option 3's original requirement to maintain two separate sets of endpoints independently adds complexity to the versioning model with limited benefit. The constraint with adopting option 2 is, if changes are anticipated in the future to one of the endpoints, participants will be required to migrate to the latest version of that endpoint. This is a minimal constraint as participants will eventually migrate to the latter versions of the endpoints anyway.

Therefore, option 2 will be adopted and the the Endpoint Version Schedule details the different versions of the endpoints which will be made available:

This approach has the advantage of maintaining consistency with the current endpoint versioning mechanism described in the Consumer Data Standards and allows participants to plan endpoint migration accordingly.

Previous endpoint deprecation and retirement dates will be explored in a future maintenance iteration.
Issue #452 has been raised to track this work.

[CDR-API-Stream]

This change was incorporated into release v1.15.0. Refer to Decision 212 for further details.