[Work_Item] Define change management principles to determine backwards compatibility

[AWS-ZachErdman]

Problem Statement

FinOps practitioners who adopt FOCUS will need to upgrade to new versions over time as they are released in order to be able to receive the new features provided. However, upgrading to a new version from an old version becomes challenging if, between the new and previous version, there are changes requirement dictating how data must appear in a given column. This would mean the new specification is not "backwards compatible" and this change would be a "breaking change".

Breaking changes may require practitioners who upgrade FOCUS versions to either (1) update their existing SQL queries that they used to run on the previous version in order to generate the desired results in the new version or (2) modify a FOCUS export such that the data of an existing column in a new export version matches the requirements of the previous version so that they do not need to change their queries.

Objective

The FOCUS working group recognizes that breaking changes make it more difficult for FinOps practitioners to upgrade to new FOCUS versions and thus, should be avoided when possible. However, breaking changes may be required in certain circumstances when the benefit of the change outweighs the cost it presents to practitioners to adopt.

This work item should result in clear rules for how the FOCUS working group handles breaking changes when formulating new FOCUS versions to make sure that practitioners don't have to deal with breaking changes that don't add tangible value to their FinOps practices. In order to do this, we will define:

1. What is a "breaking change" and "backwards compatibility" according to FOCUS?
2. How do we evaluate the benefit and cost to a practitioner of a breaking change?
3. How can we change our specification development process to ensure we are properly evaluating breaking changes?

Supporting Documentation

• [DISCUSSION]: Formally document the backwards compatibility process for FOCUS #578

Proposed Solution / Approach

We will add guidelines in our FOCUS github repo that clearly define what a breaking change is and how they should be evaluated. We may also update some of our issue and work item templates to include references to breaking changes.

Epic or Theme Association

Epic: [Backwards Compatibility]

Stakeholders

Primary Stakeholder: Zach Erdman (AWS)

[jpradocueva]

Summary from the Maintainers' call on Nov 25:

Context:
The aim is to document and formalize principles that ensure backwards compatibility across FOCUS versions, balancing innovation with stability for users.
This WI needs to be defined as soon as possible as the rest of WIs may have a dependency with the finding from this work.
Maintainers Assigned:
Zach, Riley
Task Force Assigned:
Task Force 2 (TF2), which deals with long-term strategy and compatibility.

Next Steps from TF-2 call on November 27:

• [#643] Zach @AWS-ZachErdman & Riley @rileyjenk : Prepare a draft of the "Backwards Compatibility Principles" document for review, including categories of changes and initial criteria. (list of properties with the impact on practitioners and other stakeholders if changed)
• [#643] Michael @flanakin : Provide feedback on naming conventions and potential improvements to reduce ambiguity in data columns.
• [#643] All Members: Begin behaving as if principles are in place by carefully evaluating potential breaking changes during PR reviews.
• [#643] Task Force: Add the discussion to the agenda for the next meeting where Zach or Riley can participate.
• [#643] Joaquin @jpradocueva : Notify Zach and Riley about their action items and encourage preparation for the next meeting.

[jpradocueva]

Action Items from TF-2 call on December 4:

• [#643] Irena @ijurica : Prepare a Google Drive folder for this issue, create an initial draft of the Backwards Compatibility Notes document and notify TF-2 Members and FOCUS Maintainers. Move the List of Change Types Brainstormed During the Meeting from the TF-2 meeting minutes into the new document. Ensure the content is well-structured and formatted for further discussion and refinement.
• [#643] TF-2 Members, FOCUS Maintainers: Review the Backwards Compatibility Notes document and provide feedback, additional suggestions, concerns, comments, and discussion topics. This document will serve as the foundation for further discussions in upcoming meetings and as a basis for drafting the Backwards Compatibility Principles document.
• [#643] Zach @AWS-ZachErdman & Riley @rileyjenk : Prepare a draft of the "Backwards Compatibility Principles" document for review, including categories of changes and initial criteria. (list of properties with the impact on practitioners and other stakeholders if changed)
• [#643] All Members: Begin behaving as if principles are in place by carefully evaluating potential breaking changes during PR reviews. (Carried over from the previous meeting.)

Action Items from the Members' call on December 5:

• [#643] Riley @rileyjenk : Draft a document outlining proposed compatibility tiers and share for feedback.
• [#643] Volunteer: Map potential changes to compatibility tiers and identify high-risk scenarios.
• [#643] All members: Review the draft compatibility principles and provide feedback before the next TF meeting.

[jpradocueva]

Action Items from the Maintainers' call on December 9:

• [#643] Letian @letian-wang-92 to confirm with Mohit Gupta their ability to lead and participate in backward compatibility discussions.
• [#643] Riley @rileyjenk to share the current notes document with Letian and Mohit for review.
• [#643] Shawn @shawnalpay to set up a focused meeting with AWS representatives to discuss next steps if necessary.

[jpradocueva]

Action Items from the TF-2 call on December 11:

• [#643] Riley @rileyjenk : Refine the change classification framework and propose additional examples for each category.
• [#643] All Members: Review the use case library to determine whether it adequately defines the supported scenarios and identify gaps that may require documentation in the specification.

[jpradocueva]

Action Items from the Maintainers' call on December 16:

• [#643] Riley @rileyjenk : Review the use case library to filter out scenarios that are currently supported (by which version) and categorization of change types.
• [#643] Maintainers to decide: is the use case library our official source for scenario-based compatibility?

[jpradocueva]

Action Items from the TF-2 call on December 18:

• [#643] Shawn @shawnalpay : Provide a consolidated spreadsheet of use cases and queries to ensure all team members work from the same data source.
• [[Work_Item] Define change management principles to determine backwards compatibility #643] Shawn @shawnalpay /Irena @ijurica : Define a version-specific list of supported use cases to establish a baseline for compatibility classification.
• [[Work_Item] Define change management principles to determine backwards compatibility #643] Beau @beau-nelford-anglepoint : for each use case query, determine whether it is supported by FOCUS 1.0
• [[Work_Item] Define change management principles to determine backwards compatibility #643] Beau @beau-nelford-anglepoint : for each use case query, determine whether it falls into one of the following categories when comparing 1.0 and 1.1:
  1) No change
  2) Degraded / deprecated support
  3) Same support, but modification required
• [[Work_Item] Define change management principles to determine backwards compatibility #643] Riley @rileyjenk : Go through 10 “high-importance” columns and do similar classifications as the above from a column perspective alongside the use case perspective

[jpradocueva]

Action Items from the Maintainers' meeting on Jan 6:

• [[Work_Item] Define change management principles to determine backwards compatibility #643] Dave Hatcher - Categorize all use cases from FOCUS 1.0 and 1.1 into the three compatibility categories.
• [[Work_Item] Define change management principles to determine backwards compatibility #643] Riley - Categorize all columns in the specification based on the same compatibility principles.

[jpradocueva]

Action Items from the TF-2 call on Jan 8:

• [[Work_Item] Define change management principles to determine backwards compatibility #643] Beau @beau-nelford-anglepoint : create a brief service level pass with a sample set for 1.0, and 1.1 . Check use cases that are unchanged and flag the ones that have changed columns to see if they are no longer supported.

[jpradocueva]

Action Items from Members' call on Jan 9:

• [#643] Beau @beau-nelford-anglepoint : create a brief service level pass with a sample set for 1.0, and 1.1 .Check use cases that are unchanged and flag the ones that have changed columns to see if they are no longer supported.
• [#643] Beau @beau-nelford-anglepoint - Review all use cases to classify them into No Change, Supported with Change, or No Longer Supported categories.