[Work_Item] Introduce experimental columns or data elements into the FOCUS schema

[udam-f2]

1. Problem Statement *

Describe the problem, issue, use case, or opportunity that this work item addresses.
Include practitioner quotes illustrating real examples a) of questions being asked by practitioners and b) value unlocked by answering these questions, if available.

• What is the problem?: Explain the context and why it needs resolution.
• Impact: Describe how the problem affects users, systems, or the project.

As we burn through adding support for many of the obvious constructs into FOCUS, we will start approaching ones that we won't have a conviction on whether it should be added, or whether one or more columns are warranted. We need a way to introduce constructs that may be more experimental - without the WG spending extended periods arguing over something that they may not have all the supporting data for (but is being brought up as a key need from some consumers).

Use cases:

(Use cases would be applicable to what lands in this column and not the column itself.)

2. Objective *

State the objective of this work item. What outcome is expected?

• Success Criteria: Define how success will be measured (e.g. metrics and KPIs).

Streamline the process for adding new concepts to the specification by introducing a common column with "experimental" properties that will eventually be promoted to independent columns in a future release.

3. Supporting Documentation *

Include links to supporting documents such as:

• Data Examples: [Link to data or relevant files; DO NOT share proprietary information]
• Related Use Cases or Discussion Documents: [Link to discussion]
• PRs or Other References: [Link to relevant references]

N/A

4. Proposed Solution / Approach

Outline any proposed solutions, approaches, or potential paths forward. Do not submit detailed solutions; please keep suggestions high-level.

• Initial Ideas: Describe potential solution paths, tools, or technologies.
• Considerations: Include any constraints, dependencies, or risks.
• Feasibility: Include any information that helps quantify feasibility, such as perceived level of effort to augment the spec, or existing fields in current data generator exports.
• Benchmarks: Are there established best practices for solving this problem available to practitioners today (e.g. mappings from existing CSP exports that are widely used)?

If we have a construct like ChargeDetails or something even specific for experimental schema updates, we could more easily introduce changes and naturally graduate them into a more permanent location.

ODCR is a good example of something we didn't originally have a strong conviction on whether it should be introduced with many columns specifically for it or if it should be grouped together with other constructs etc.

5. Epic or Theme Association

This section will be completed by the Maintainers.

• Epic: [Epic Name]
• Theme: [Theme Name, if applicable]

TBD

6. Stakeholders *

List the main stakeholders for this issue.

• Primary Stakeholder: [Name/Role]
• Other Involved Parties: [Names/Roles]

TBD

Do you want to see this column in FOCUS?

Give it a 👍 below ↴

Comments are welcome and encouraged!

[flanakin]

I like this assuming it's an easier path to get semi-well-defined (whatever that means) concepts into the spec. We do need some principles around it, like:

1. Don't add P0 columns (e.g., UsageType, MeterId) that are already defined as extended columns by providers.

   Justification: If I have a highly used extended column today, I don't want to move it to JSON and make life harder for people. On the contrary, if I have something informational that is not critical to cost that provides value (e.g., BillingAccountType), then it would be nice to at least make that available in a standard way that we can decide to promote or even remove in the future based on feedback.

2. Keys in the JSON object MUST be from an allowed list.

   Justification: We don't want this to become a junk draw of forgotten hopes and dreams.

3. We should strive to promote what's in this JSON object with every release unless we determine it should not be an explicit column.

   Justification: Same as 2. We may not promote everything in the next release, but we should at least evaluate them to confirm that we're okay with keeping them in ChargeDetails so it doesn't turn into a junk drawer.

I would like to see a lightweight process for getting something into this that should not require a full column spec. Maybe just a few sentences to describe the key and convey what goes in it. Each key can have its own requirements, but I would expect most things would be optional here.

[flanakin]

Some things I would love to consider with this approach:

1. CapacityReservationId, CapacityReservationStatus, CapacityReservationTotal

   Justification: Assuming this only applies to one service, I'm not sure it warrants a dedicated column. This would be a way to define it and start to get feedback on it.

2. BillingAccountType, SubAccountType

   Justification: Practitioners have said they want these columns and the behavior isn't debated. We are just not adding them because there are higher priority things. I would love to define a formalized way to optionally get them in. When we discuss it further, we may decide to promote or remove them.

3. ResourceParentId, ResourceParentType

   Justification: This has come up several times about Azure resource groups, given how important they are for chargeback. I would love to see this as an optional column. My only hesitation is not knowing if practitioners view it as a P0 or not. That's the only question I'd have as it would make it harder if I remove x_ResourceGroup.

4. ServiceSubcategory

   Justification: This is not a P0 column and defining the values needs time. Adding it here allows us to get something in that people can utilize, if important, but doesn't lock us into anything for the dedicated column, which we can add in June.

5. ServiceCategoryV2

   Justification: I'm not sure about this, but just throwing it out. If we were to drastically change ServiceCategory values, we could hypothetically do that here. I'm not convinced. This is just a thought.

6. SkuCategory

   Justification: There's no debate in the name. There's a question about the values, but I would argue starting with ServiceCategory values is a no-brainer. This allows us to add that value without committing 100% to not changing. (Especially with the idea of a drastically different ServiceCategory breakdown coming.)

7. Provider-specific SKU categorization

   Justification: We know this is needed. The only reason this might not be feasible here is the names we choose will require a few hours to nail down, at best.

8. PricingSubcategory

   Justification: We know there's value in this column. We just haven't driven the conversation home around the precise values.

I could keep going... there's a lot of potential here to add incremental value. Love the idea!

[shawnalpay]

Discussed in Oct 22 TF1 call.

Pro: speed to market for new datapoints
Con: possible breaking change -- how to handle "promoting" datapoints from experimental to fully-fledged columns
(@ahullah made the point that this is experimental, so it's on the practitioner to own the possible movement of that column if they build against it)

Alternative: less friction around making a column optional -- though that would be at risk if we rename

[jpradocueva]

Notes from the Maintainers' call on November 4:

Context: This work item proposes allowing experimental data points in the spec, giving practitioners early access to features still in development. This process aims to gather feedback while maintaining flexibility for adjustments before formalizing new features.
Level of Effort Required: Medium — Implementing an experimental approach is feasible but requires careful planning to establish guidelines for experimentation without disrupting the core specification.
Level of Impact: Very High – Allowing experimental data points provides practitioners with early access to new features, promoting iterative improvements. This flexibility enables practitioners to adopt and test new capabilities before they are formalized, accelerating feedback and adaptation.