[DISCUSSION]: Formally document the backwards compatibility process for FOCUS

[AWS-ZachErdman]

Description

FOCUS has been receiving some backwards incompatible changes in 1.1. We need to:

1. Document our definition of backwards incompatibility
2. Document our process for validating and merging backwards incompatible changes to a future version

Proposed Approach

This documentation should be in the FOCUS repo for future reference, but need not be in the specification document.

GitHub Issue or Reference

No response

Context

No response

Data Submission for Discussion

No response

[jpradocueva]

Action Items from TF-2 call on Oct 3rd:

• [TF2-#578] Zach, @AWS-ZachErdman : Draft initial documentation for review next week.

[jpradocueva]

Action Items from the Mebmers' call on October 3rd:

• [Maintainers-#578] Zach, @AWS-ZachErdman to draft a process for breaking change documentation.

[ijurica]

A few notes on (potential) breaking changes introduced in FOCUS version 1.1 compared to version 1.0...

---

Assuming a breaking change is defined as any modification to an existing FOCUS column, attribute, or Metadata element that could cause previously functioning SQL queries (supporting a specific use case) to fail or behave differently, the following changes introduced in FOCUS version 1.1, compared to version 1.0, are considered breaking changes:

• ConsumedQuantity column update:

  • According to the normative requirements specified in version 1.1, ConsumedQuantity MUST be null if ChargeCategory is 'Usage', CommitmentDiscountStatus is 'Unused', and ChargeClass is not 'Correction'.
  • In contrast, version 1.0 specified that ConsumedQuantity MUST NOT be null if ChargeCategory is 'Usage' and ChargeClass is not 'Correction', regardless of the CommitmentDiscountStatus.
  • Note: An alternative to this breaking change was considered (see PR Have consumedquantity be 0 for unused commitment #598 for details). We discussed the pros and cons and decided to proceed with the current version.

• ConsumedUnit column update:

  • According to the normative requirements specified in version 1.1, ConsumedUnit MUST be null if ChargeCategory is 'Usage', CommitmentDiscountStatus is 'Unused', and ChargeClass is not 'Correction'.
  • In contrast, version 1.0 specified that ConsumedUnit MUST NOT be null if ChargeCategory is 'Usage' and ChargeClass is not 'Correction', regardless of the CommitmentDiscountStatus.
  • Note: An alternative to this breaking change was considered (see PR Have consumedquantity be 0 for unused commitment #598 for details). We discussed the pros and cons and decided to proceed with the current version.

• PricingCategory column update:

  • According to the normative requirements specified in version 1.1, PricingCategory MUST be 'Committed' when the charge is subject to an existing commitment discount and is not related to the purchase of the commitment discount.
  • In contrast, version 1.0 specified that PricingCategory MUST be 'Committed' when the charge is subject to an existing commitment discount, even in cases where the charge pertains to the purchase of the commitment discount.

• CommitmentDiscountStatus / CommitmentDiscountCategory column update:

  • According to the normative requirements specified in version 1.1, CommitmentDiscountStatus MUST be one of the allowed values: 'Used' or 'Unused'. Due to an oversight in version 1.0, the same requirement for the CommitmentDiscountStatus column was incorrectly written as: "The CommitmentDiscountCategory MUST be one of the allowed values: 'Used' or 'Unused'."
  • This oversight might also be classified as an editorial change.

[ijurica]

Some Common Characteristics of Breaking Changes:

• Schema/Structure Changes:

  • Changes in data structure, such as renaming or removing columns or metadata elements, can break existing integrations or queries.
  • Example: Removing or renaming a column in a dataset would require consumers (practitioners, ETL systems, etc.) to adapt their queries or processing logic.

• Modifying Data Types:

  • Changing the data type of a column can cause failures if consumers were designed to handle the previous format or behavior.
  • Example: Changing a column’s data type from String to Decimal.

• Modifying Constraints That Result in a Change in Expected Behavior:

  • Any modification that alters column nullability (e.g., MUST be null vs. MUST NOT be null), the list of available values, or the expected value for a particular use case scenario can potentially cause the same queries to yield different outputs or results.
  • Example: If a query against a dataset returns different results due to changes in column nullability or available value definitions, this constitutes a breaking change.

• Etc.

Additional topics to discuss:

• Should we focus on the practitioner’s (consumer’s) perspective, or should we encompass both consumers and producers?
  • For example, introduction of stricter validation or constraints - such as a more limited value range, or changing from "MAY be null" or "MAY NOT be null" to "MUST", etc.

[shawnalpay]

Assigning this one to @AWS-ZachErdman and @ijurica, per our discussions this past week.

EDIT: per the Oct 10 Member call, adding @ahullah and @rileyjenk as assignees as well.

[jpradocueva]

Action Items during Members' call on Oct 11:

• [Members-#578] Zach, @AWS-ZachErdman , Alex, @ahullah, Riley, @rileyjenk : Create a formal document outlining the backwards compatibility process.
• [Members-#578] Zach, @AWS-ZachErdman , Alex, @ahullah, Riley, @rileyjenk : Develop a checklist of evaluation criteria to be used for assessing proposed changes.
• [Members-#578] Joaquin: Schedule follow-up meetings to review specific scenarios and refine the process.

[jpradocueva]

Summary Members' call on Oct 17:

Primary Issue: Backwards compatibility between versions 1.1 and 1.2 needs to be formally documented.
Core Problem: Some changes in version 1.1 were considered breaking changes, raising concerns about compatibility.
Divergent Views: Members disagreed on what constitutes a breaking change and how to document these criteria.
Final Agreement: The group will create formal documentation defining breaking changes and the backward compatibility process.
Action Items:

• Membes-[#578] Zach @AWS-ZachErdman , Alex @ahullah , Riley @rileyjenk : Collaborate to draft the formal document defining breaking changes and criteria for backwards compatibility.

[shawnalpay]

@AWS-ZachErdman (cc @ahullah and @rileyjenk)
Defining and aligning on this issue is quite important to moving forward on a few other issues currently up for 1.2 consideration, like #540 and #557. Have you had a chance to move this one forward yet? Let's discuss soon.

[jpradocueva]

Action Items from Maintainers' call on Oct 21:

[#578] Zach @AWS-ZachErdman to present a draft backward compatibility definition in the next meeting.

Comments from Members' call on Oct 31:

Analysis:
This topic involves establishing a formal definition and approach to backward compatibility for FOCUS releases. The team needs to outline principles around maintaining compatibility across versions to ensure that existing implementations aren’t disrupted by updates. This is particularly important as more organizations adopt FOCUS, requiring predictable upgrade paths and minimized disruption for legacy implementations.
Agreements:
Prioritize defining clear guidelines for backward compatibility to support stable versioning and adoption by practitioners.

[jpradocueva]

Notes from the Maintainers' call on November 4:

Context: Backward compatibility principles are essential for ensuring future spec updates do not disrupt existing implementations. This item aims to formalize compatibility guidelines to prevent potential disruptions.
Level of Effort Required: High — Defining backward compatibility requires thorough planning and agreement across the specification to account for all potential future changes.

[shawnalpay]

#643 looks good to me! @AWS-ZachErdman, is it alright if I close this issue in favor of that Work Item?

[AWS-ZachErdman]

Of course - I'll close it out!