A status of POSTED should indicate the final update for a transaction

[markskript]

Description

The CDR standards are quite flexible regarding how a Banking Transaction is represented as it progresses through its lifecycle. The majority of Data Holders appear to be following a predictable flow where a transaction's details quite often change while the transaction status is PENDING but stabilises when the transaction finally lands in a status of POSTED.

We have identified at least one data holder where as the transaction moves through it's intermediate states, they are presenting a status of POSTED (including a non-null postedDateTime value) on those intermediate states.

For example - a transaction with a description of "AUTHORISATION", which is obviously in a transitional state, is coming through with a status of POSTED and a non-null posting date time. This brings into question the value of the status field at all.

Additionally, the standards are currently flexible enough to allow data holders to NOT provide a traceable transaction ID as the transaction moves through these states.

When combined, these aspects make it impossible for the consumer to identify what transactions are pending and what are posted. This results in what appear to be "duplicate" transactions to the consumer. The solution suggested by one DH was to delete all data from the consumers end and refresh it every time - but this is not scalable and would only exacerbate NFR issues.

We could propose that the transaction ID need to be consistent as a transaction moves through its various states, but we know that this isn't possible for all data holders, and was specifically called out as not feasible during the standard's origination.

Instead, we propose a change to the wording of the standards to align the implementations of the data holders and make it mandatory that if a transaction is in a pending state (i.e. not in it's final posted form) then the status of the transaction MUST be PENDING.

Intention and Value of Change

This change will allow consumers to weed out duplicates by ignoring, if they wish, all transactions that have a status of PENDING.

Area Affected

The description on the "status" field of https://consumerdatastandardsaustralia.github.io/standards/#cdr-banking-api_schemas_tocSbankingtransaction.

Change Proposed

We propose updating the description of this field as follows

Status of the transaction whether pending or posted. Note that there is currently no provision in the standards to guarantee the ability to correlate a pending transaction with an associated posted transaction. If a transaction is in a state where it has not been posted to the account and is still being authorised then the status MUST be presented as PENDING.

Alternatives

We are more than happy to hear any alternatives that people can suggest. Our end goal is for consumers to be able to achieve the same view of transactions as they see in their Internet Banking, without needing to totally delete all transactions and reload them fresh every day.

Another alternate approach could be the following wording

Status of the transaction whether pending or posted. Note that there is currently no provision in the standards to guarantee the ability to correlate a pending transaction with an associated posted transaction. Once a transaction has a status of POSTED the transaction_id value MUST not change.

[kambasiq]

Basiq is in support of this request as well. When we have raised tickets against DHs for issues regarding transaction status, we get a reference to this Zendesk ticket. The specific line in the answer "As to the marking of specific account types as pending, this is at the discretion and interpretation of the data holder." makes it impossible to have consistent approach to managing transactions if there is ongoing consent.

[markskript]

The use case we are trying to support with this change is a consumer importing transactions from the CDR into their ERP system. Their ERP system expects a SINGLE entry to represent a transaction as it moves through its lifecycle. Consumers don't commonly see a PENDING transaction and its POSTED equivalent as two different transactions - they are the one transaction moving through two various states.

This change is trying to achieve two things, which together should solve the issues with this use case. Can we obtain feedback from DHs as to how compatible their systems are with the following changes:

1. That once a transaction has a status of POSTED, its transaction ID never changes
2. That a transaction which is pending or pre-authorised (i.e. has NOT been posted to the ledged) is never presented with a POSTED status.
3. That a transaction with a status of POSTED should only ever disappear in edge cases, such as relating to fraud.

If DHs are happy with that then I think the wording below (which is a combined version of the two suggestions in the original description) might suffice?

Status of the transaction whether pending or posted. Note that there is currently no provision in the standards to guarantee the ability to correlate a pending transaction with an associated posted transaction. Once a transaction has a status of POSTED the transaction_id value of that transaction MUST not change. Once a transaction has a status of POSTED that transaction should be persistent in all situations except edge cases such as fraud.

[joshuanicholson]

As an ADR, we agree with Mark. Our thoughts are as follows. Like Mark, we would like to have some insights from DH in order to deliver a consistent and accurate solution for consumers.

• Posted for us means final; the transaction has moved into a state considered complete, and all details are available and correct, be that date, amount, description or transaction ID. Ideally, once posted, it should not change#1

• Pending. Unless we open this up to several other changes, pending is pretty much anything other than posted, be that a
  hold, temporary charge, transaction waiting for clearance, initial charge pending details, etc.

• It would be ideal to have transaction IDs remain in the transition from pending to posted, but as Mark has articulated, the business case for most accounting systems is that only posted transactions are required due to the chance of pending transactions changing. This means our primary concern is the permanence of IDs once posted.

• As an aside, some banks on their Internet bank banking site do not indicate which transactions are pending and which are posted, whereas mobile apps generally indicate the state of a transaction. This lack of disclosure causes significant expectation gaps about CDR data. While not within the scope of CDR rules, it would be appreciated by all stakeholders that there be some consistency across the digital experiences with the disclosure of "posted" and "pending" The terms used by the DH are not so much the issue, just that there is an indication a transaction has reached a state of completion, finalisation or posted which means it's unlikely to change.

#1. We appreciate that there are circumstances where something may change, like the topic of transactions being deleted, which has come up around related discussions.

[nils-work]

I would suggest that transactions disclosed with an incorrect status would be a compliance concern according to the Rules:
7.10 Rule relating to privacy safeguard 11—quality of CDR data and Act: 56EN Privacy safeguard 11—quality of CDR data

Disclosures by data holders

(1) If a data holder of CDR data is required or authorised under the consumer data rules to disclose the CDR data, the data holder must take reasonable steps to ensure that the CDR data is, having regard to the purpose for which it is held, accurate, up to date and complete.

If so, it would seem unnecessary to add a statement declaring that if a transaction has not been posted on the account (impacting the currentBalance), it must not be disclosed as POSTED.

In terms of changes to the status description, the current statement is:

Status of the transaction whether pending or posted. Note that there is currently no provision in the standards to guarantee the ability to correlate a pending transaction with an associated posted transaction.

Possible additions to the description in relation to this change request may be:

Once a transaction has a status of POSTED, the transactionId (when available) MUST NOT change.

or

Once a transaction has a status of POSTED, all associated details are expected to be final and MUST NOT change.

and/or

Only in special circumstances such as fraud, a POSTED transaction MAY be removed.

Could participants provide any details of special circumstances that would necessitate a POSTED transaction being changed?

[da-banking]

Hi all

To add commentary to this discussion.

Our Core Banking System (adopted by 15 Data Holders) allows:

1. POSTED transactions to have the transaction description amended - NB: the transaction id does not change

2. Transactions to be POSTED with a simple description, which is then updated to the full description when the overnight process runs

In both cases, customers see the same in digital channels and these transactions affect the account balance immediately (hence are not PENDING)

Thanks
DA Banking

[nils-work]

Would additional status values or another status or 'key' field help to convey more granular stages of a transaction lifecycle, which may assist ADRs with reconciliation?

There may be two key aspects to consider:

1. Which balance the transaction has been applied to
   1. PENDING meaning it has affected the availableBalance
   2. POSTED meaning it has (also) affected the currentBalance
2. Whether details of the transaction (transactionId, type, description, amount, reference, or any available ...DateTime values) are complete in the provided record, or can be expected to be further amended by a nightly or other process, including whether, or when such a process is expected to be completed.
   1. This information may be most important when either a transactionId is completely unavailable, or where one may be added or amended during such a process, as significant changes to the record after it is initially disclosed are likely to make reconciliation difficult.
   2. Could a consistent interpretation and application of unique executionDateTime assist with identification and deduplication where changes in other fields may occur, or, could Data Holders generate/derive and provide a separate transaction key based on any data they know to be immutable according to their process? The availability of such implementation-specific detail for a similar purpose has been discussed in relation to issue 553 (point 4 in comment).

[joshuanicholson]

Which balance the transaction has been applied to
PENDING meaning it has affected the availableBalance
POSTED meaning it has (also) affected the currentBalance

We agree with point 1: clear definitions and expectations are set around the current (and, if applicable, future) transaction status. The proposal is simple but effective. The transaction status and balance MUST also work in tandem. It seems illogical to have balances updated "immediately", be it a new pending transaction or a transaction moving from pending to posted. Having delays of hours or days between these two data points being synchronised is one of our core issues requiring unique code DH by DH (and even more brittle code product by product within a DH). We suggest that the wording be updated to reflect a timeliness requirement. This is important as consumers have two experiences: the DH digital experience, where there is NO delay, and CDR, where there are delays.

[joshuanicholson]

Regarding point 2
We are somewhat conflicted about this approach. Ideally, we would prefer that changes to data MUST NOT occur once they are posted. As an ADR, we definitely see DHs make changes to transactions after they've moved to posted; we thank the DHs who have reduced this, but it does lead to the practicality of this point.

To maintain some flexibility (i.e., allow changes to data), we alternately suggest having a new "last updated date/time" field and allowing this to be a parameter for ADR to collect transactions based on the date last updated. And yes, we appreciate some DH reaction saying we can do this by "over collecting" and comparing, but how far back should we collect? And do DHs really want ADRs to over-collect 3 days, 3 months, 3 years, considering some businesses may have, say, 10, 100 or 500 transactions per day?

To be clear, we prefer data to remain unchanged once posted, but we always want to ensure the data we collect is current and matches what consumers see via other channels (paper statements, PDF statements, mobile apps, browser IB, export files CSV etc, and other data feed sources). I am sure most DHs know that Bank Statements are used in audits, by lenders to make decisions, by financial planners and lawyers in court cases/legal matters. Therefore, as a DH, you ensure bank statements present the DH and data with integrity; CDR must be held to the same standard as most user cases of a bank statement apply to CDR data.

Yes, I am a broken record on matching digital experiences, but at present, so many DHs have different experiences, causing significant friction, data accuracy issues, and support costs for ADRs. To be brutal about this, we are losing paying consumers because of poor data quality from DHs.

[nils-work]

Hi @joshuanicholson

Some thoughts re: your comment:

we would prefer that changes to data MUST NOT occur once they are posted

The option to provide distinction between a transaction being POSTED, and being in a 'final' (detail) state as described in my point 2 above was in response to point 2 in this comment, in particular for transactions that don't have a consistent transaction id through any subsequent processing:

[Our System allows...] Transactions to be POSTED with a simple description, which is then updated to the full description when the overnight process runs

Having a 'last modified date' (which I'm not opposed to) doesn't signal that modifications of the transaction detail may be pending (e.g. nightly processes or other enrichment) for a POSTED transaction that has already hit the balances. Knowing that further processes are pending may not help with reconciliation, but may assist with deciding when to collect data (i.e. only after that process is complete). If any type of unique transaction key could be provided, knowledge of pending processes may be redundant because you can update records in place.

Other DH channels can probably deal with transaction detail changes because they may not need to reconcile and deduplicate the transaction feed in the same way an ADR may need to, where they have already collected and stored that data and want to limit repeat collection. That is, unless the ADR has chosen to 're-collect' or 'over collect' as you stated, treating each daily (or longer duration) collection as only being valid as at that point in time, and then completely replacing all those transactions the next day, with whatever is then available. This may mean dropping, re-collecting and re-calculating a lot of data, rather than simply appending updates each day. NFRs may make the 'over collecting' approach impractical in some cases though, as you stated.

[nils-work]

Following the above discussion, I would suggest the following changes to resolve issues related to the collection and reconciliation of Transaction data:

1. Clarify the expectations of the BankingTransaction status field by adding the following statement to the description:

   • PENDING meaning the transaction has affected the availableBalance of the account,
   • POSTED meaning it has (also) affected the currentBalance.

2. Add a statement to the introductory text of the Get Transactions For Account endpoint to state:

   • Once a transaction reaches the POSTED status, the amount value MUST NOT change. This means that a reversal or other adjustment to a transaction amount must be disclosed as a separate transaction to accommodate accounting requirements.

3. Add a status query parameter to the Get Transactions For Account endpoint with the description:

   Used to filter results according to the PENDING/POSTED status. If absent then all transactions are returned.

4. Add a modifiedDateTime field to the BankingTransaction schema with the description:

   The date and time any detail in the transaction was added or updated. Note: the amount value MUST NOT change for POSTED transactions.

5. Add a detailPending Boolean field to the BankingTransaction schema with the description:

   true if any subsequent Data Holder processing is expected to change any detail provided in the disclosed transaction. false if all expected processing or enrichment is known to be complete at the time of disclosure.

6. Add a transactionDeleted Boolean field to the BankingTransaction schema with the description:

   true if the transaction has been removed from other Data Holder channels and no longer impacts any disclosed account balances (as at the modifiedDateTime field value). false if the transaction remains valid as disclosed.

[markskript]

Hi @nils-work

I believe that change 1 above would address the specific problem that trigger us to raise this DH and I would be happy to promote that in isolation of the other changes.