Strategy for Caseflow product data and metrics tooling

[alisan16]

The goal of this issue is to define the internal data and metrics needs for Caseflow products and evaluate whether the current state of data is meeting those needs, ultimately producing an analytics strategy. There are a number of disparate tools and in order to better organize the information and render it usable, we need an overview of the current state. The primary audience of these data and metrics is the Caseflow team (system administrators) and not Caseflow users.

Acceptance criteria

• Write-up that includes
  • current needs for data and metrics per product
  • existing tooling
  • recommendation for a path forward to v0, whether it be to enhance existing tools, migrate to new tools, deprecate certain tools, etc.

Out of scope

Actually implementing changes is outside the scope of this ticket. Future tickets should be created and added to a common epic based on the results of the investigation.

Background/context

Back in 2016, the data and metrics strategy is laid out in this ticket: https://github.com/department-of-veterans-affairs/appeals-pm/issues/673 and here: https://github.com/department-of-veterans-affairs/appeals-pm/blob/master/caseflow/data/data-reporting.md

We have previously set up the Caseflow Dashboard at /stats (for Dispatch, Certification and Intake)

• Dispatch Stats | Display stats charts #819
• Certification Dashboard #331
• Add stats page stats for certification v2 #1897

That were recently deprecated. We should consider reviving these if there is value to the Caseflow internal team as the intended audience (not Caseflow users).

• Dispatch Stats Page | Deprecate stats pages #12991
• Deprecate Stat pages #13095

Non-exhaustive list of historical approaches for product metrics:

• Caseflow Dashboard (aka /stats page): https://github.com/department-of-veterans-affairs/appeals-pm/blob/master/Project%20Administration/Government%20Inquiries/GAO/GAO%20100666.md#caseflow-dashboard-and-performance-measurement
• Google Analytics
• Datadog: https://github.com/department-of-veterans-affairs/appeals-pm/blob/master/caseflow/tech_specs/business_metrics.md

[yoomlam]

@alisan16 Here's my write-up so far: https://docs.google.com/document/d/1s-akZeibptmhb4HF0Z5B8GrYG4ukYI81gq0ms1d2BXQ/edit?usp=sharing
I'd like some feedback to make sure I'm providing what you are expecting before I spend time writing up the epic. Thanks!

[alisan16]

@yoomlam Thanks for putting that proposal together and gathering information from each team about what their needs are! You're on the right track, and I encourage you to focus on exploring the tooling that best meets the team's needs. I don't imagine that we would be able to streamline our analytics to use a single tool, but the team needs more guidance to understand how each of the existing tools should be used. You start addressing this in the "Categorized Data and Metrics Needs" document, and it would be helpful if you perhaps created a comparison matrix of the main categories and the tooling required for each and what you recommend.

To clarify the scoping of this ticket, the main goal is to answer the question of what the data needs on the team are to get a sense of current state and then ultimately recommend what tooling best meets those needs. The focus is on the technical tooling required. The data needs themselves will be refined over the coming weeks, and we are working as a project leads group to provide more guidance on team-wide KPIs.

One other thing that wasn't included in the report was our history with Looker and the coming Metabase replacement. This kind of tool should be considered, especially for the "can be done with a query" category of needs.

[yoomlam]

@alisan16 For your review, I documented tool guidance and tool applicability on a new wiki page: https://github.com/department-of-veterans-affairs/caseflow/wiki/Collecting-Data.

[alisan16]

Amazing! Can you move this to the appeals-team wiki? I am putting together updated guidelines for documentation, but since the purpose of this guide is for the Caseflow team to improve how we work and what we work on, it should go into the appeals-team wiki. The caseflow wiki is for technical documentation related to the Caseflow application.

[yoomlam]

Moved to https://github.com/department-of-veterans-affairs/appeals-team/wiki/Collecting-Data and added a link in the sidebar. Deleted old page.

@alisan16 What's left to do on this ticket? Do you have any feedback regarding the epic: https://docs.google.com/document/d/1s-akZeibptmhb4HF0Z5B8GrYG4ukYI81gq0ms1d2BXQ/edit#heading=h.u2ipswaytb2v?

[alisan16]

Re-read the proposal and added some comments, but in general let's pause the creation of the epic and implementation. I've updated this ticket to just focus on the data and metrics tooling, and once we make more progress on defining the specific data needs, that's when we can revive defining an analytics strategy.

The wiki you created above fulfills the AC on this issue and it can be closed.