[Work_Item]: Augment support for correction handling

[ijurica]

1. Problem Statement *

Describe the problem, issue, use case, or opportunity that this work item addresses.
Include practitioner quotes illustrating real examples a) of questions being asked by practitioners and b) value unlocked by answering these questions, if available.

• What is the problem?: Explain the context and why it needs resolution.
• Impact: Describe how the problem affects users, systems, or the project.

While some correction handling rules have been incorporated into individual columns' normative requirements - primarily within metrics and related unit dimensions - the FOCUS specification lacks a comprehensive, unified framework (i.e., attribute) for managing corrections across all columns. The absence of an overarching correction handling standard may lead to inconsistencies, oversights, and ambiguities in the specification, particularly in multi-dimensional and metric-based data contexts, ultimately leading to misinterpretation and the generation of non-conforming correction records, compromising data integrity, reconciliation, and overall data quality.

To address this, an overarching correction handling attribute should be introduced. Additionally, a review of existing correction handling rules within individual columns' normative requirements is essential to ensure consistency and alignment with this unified approach.

2. Objective *

State the objective of this work item. What outcome is expected?

• Success Criteria: Define how success will be measured (e.g. metrics and KPIs).

The goal of this work item is to formalize a consistent approach to correction handling across all columns within the FOCUS dataset.
By providing clear guidelines for producing conforming correction records and managing corrections across complex cost records in multi-dimensional and metric-based data contexts, this work item aims to minimize misinterpretation by both producers and consumers, ultimately enhancing data quality, integrity, and reliability.

Expected Outcomes:

• Clear, unambiguous rules and expectations for correction records as a whole, along with a unified approach to specifying correction handling normative requirements for individual columns, are established.
• An overarching correction handling attribute is introduced.
• Correction-related normative requirements for individual columns (e.g., metrics and related unit dimensions) are consistent and aligned with the agreed unified approach and overarching correction-handling attribute.

Success Criteria:

Success can be measured by improvements in clarity, consistency, and accuracy regarding correction handling, as well as positive feedback from providers and practitioners on the ease of producing and validating conforming correction records.

3. Supporting Documentation *

Include links to supporting documents such as:

• Data Examples: [Link to data or relevant files; DO NOT share proprietary information]
• Related Use Cases or Discussion Documents: [Link to discussion]
• PRs or Other References: [Link to relevant references]

Data Examples:

TBD

Prerequisites:

• Issue [DISCUSSION]: Formally document the backwards compatibility process for FOCUS #578
• Issue [EDITORIAL]: Editorial guidelines should be consistently applied throughout the specification. #594

Related Issues:

• Issue [Work_Item] Refine normative requirements for all existing spec content #557
• Issue [DISCUSSION]: Handling of corrections in the event of USAGE errors #511

4. Proposed Solution / Approach

Outline any proposed solutions, approaches, or potential paths forward. Do not submit detailed solutions; please keep suggestions high-level.

• Initial Ideas: Describe potential solution paths, tools, or technologies.
• Considerations: Include any constraints, dependencies, or risks.
• Feasibility: Include any information that helps quantify feasibility, such as perceived level of effort to augment the spec, or existing fields in current data generator exports.
• Benchmarks: Are there established best practices for solving this problem available to practitioners today (e.g. mappings from existing CSP exports that are widely used)?

Initial Ideas:

• Define clear, unambiguous rules for the production, validation, and application of correction records.
• Establish a unified approach to correction handling normative requirements for individual columns.
• Introduce an overarching correction handling framework (attribute) that addresses all FOCUS columns under a holistic correction-handling guideline, aligned with previously defined rules.
• Revisit existing correction handling-related normative requirements for individual columns to ensure consistency and alignment with the overarching correction handling attribute and the agreed unified approach.

Considerations:

• There is a possibility that the results of this work item could introduce backward incompatibilities. Therefore, potential impacts will be carefully assessed at each stage of the process (e.g., defining rules, specifying the correction-handling attribute, and revisiting existing correction-related normative requirements for individual columns).
• Criteria for backward compatibility assessment are out of scope for this issue but must be prepared as a prerequisite.
• Normative requirements for existing FOCUS columns and attributes will be reviewed comprehensively in a separate issue.

Feasibility:

• Since this issue involves establishing rules and guidelines, introducing an overarching attribute, and addressing multiple columns, dividing the effort into multiple PRs will help manage the overall workload more effectively.

5. Epic or Theme Association

This section will be completed by the Maintainers.

• Epic: [Epic Name]
• Theme: [Theme Name, if applicable]

TBD

6. Stakeholders *

List the main stakeholders for this issue.

• Primary Stakeholder: [Name/Role]
• Other Involved Parties: [Names/Roles]

TBD

[jpradocueva]

Summary TF-2 call on Oct 16:

#556 [SPEC CHANGE]: Add Correction Handling Attribute and/or Appendix
Key Discussion Items: The discussion centered around adding a correction handling attribute and ensuring corrections are correctly specified across all charge categories.
Problem Identification: Corrections for metrics, costs, and unit prices are inconsistently handled.
Divergent Views: Some participants suggested this is a minor issue, while others felt it needed more thorough specification.
Final Agreement: Irena will create a work item for this topic using the work item template.
Action Items:
[TF-2-#556] Irena, @ijurica was assigned to backfill the issue or create a new work item following the standard template.

[jpradocueva]

Notes from the Maintainers' call on November 4:

Context: Current spec guidelines for cost corrections are insufficient, leading to discrepancies in how corrections are applied. This task aims to provide more comprehensive correction handling guidance to improve data consistency and accuracy.
Level of Effort Required: Medium — Implementing improved correction handling is feasible but requires collaboration to define and document correction practices clearly.

[jpradocueva]

Summary Maintainers's call on Nov 25

Context:
This item aims to enhance the specification's ability to address and process corrections in billing data, ensuring accuracy in scenarios such as error adjustments or retroactive changes.
Maintainers Assigned:
Irena, probably Alex (Shawl will check with him)
Task Force Assigned:
Task Force 1 (TF1).

[jpradocueva]

Action Items from the TF-1 call on December 3:

• [#556] Alex @ahullah : Review the current straw man proposal for the correction handling appendix (or attribute), which is currently more focused on refunds and credits, and revise it to better align with general correction handling. Note that a parallel discussion (see WorkItem #636) is addressing the use case involving charge records for refunds not necessarily tied to previously invoiced billing periods. This discussion may lead to material changes in the definition of the Charge Class column, a key component for correction handling.

Action Items from the Members' call on December 5:

• [#556] Volunteers: Develop a taxonomy for correction types, including normative requirements for each type.
• [#556] Irena @ijurica : Draft a proposal for phased implementation of augmented correction handling and circulate it for feedback.
• [#636] Chris @cnharris10 : Document detailed refund scenarios, particularly for corrections within open billing periods, to inform resolution.
• [#636] Alex @ahullah : Review and propose updates to refund guidance, ensuring alignment with the expanded correction handling framework.

[jpradocueva]

Action Items from the Maintainers' call on December 9:

• [#556 and #636] Alex @ahullah to draft strawman definitions for corrections and refunds and present them at the next TF1 meeting.
• [#556 and #636] Irena @ijurica to analyze existing use cases to identify potential overlaps or unique characteristics between corrections and refunds.
• [#556 and #636] Riley @rileyjenk to prepare examples of corrections and refunds to aid in the definition process.

[jpradocueva]

Action Items from the TF-1 call on December 10:

• [#556] Irena @ijurica : Draft a glossary update to include a definition of refunds and circulate it for review.
• [#556] All Members: Provide feedback on use cases discussed, particularly customer-initiated refunds versus corrections, in GitHub.
• [#556] Chris @cnharris10 : Analyze current charge class definitions and propose adjustments, if necessary, for alignment with these use cases.

[jpradocueva]

Action Items from the Maintainers' call on December 16:

• [#556] Alex @ahullah : Draft an appendix that separates credits and refunds to clarify correction handling and prepare the PR
• [#556] Irena @ijurica : Provide suggestions for introducing glossary updates to clarify concepts and present them during the next TF1 meeting.

[jpradocueva]

Action Items from TF-1 call on December 17:

• [#556] Irena @ijurica : Provide a PR with candidate definitions for "refund" and "correction" based on the discussion.