Discovery: Comprehensive Test Plan

[gracekretschmer-metrostar]

Background

For CMS development, test plans and test scripts have historically been created by developers without QA engineering oversight and right now, flakey tests and testing gaps have created errors during the daily deploy process.

User Story or Problem Statement

As a developer working with CMS, I need to have a comprehensive test plan that will meaningfully catch all of the errors before my code gets pushed into CMS production.

Affected users and stakeholders

• CMS product teams
• VA Developers

Hypothesis

By conducting an audit of the existing test scripts within CMS, the QA engineer will find the gaps and opportunities to improve the quality of the existing tests to ultimately build a comprehensive test plan and in turn, a comprehensive test plan will help reduce errors during the daily deploy process.

Assumptions

• There a gaps in the current testing infrastructure within CMS.
• The existing test plans are not using testing best practices
• By creating a comprehensive test plan, there will be a reduction in failures during the daily deploy process.

Definition of Done

• An audit of the existing test scripts within CMS to identify any possible gaps.
• A review of the existing test scripts to understand areas for the most critical path to determine automated tests.
• A plan for how to fill the existing gaps for CMS testing (smoke test, most critical path tests) and improve the quality of existing test scripts.

Team

Please check the team(s) that will do this work.

• CMS Team
• Public Websites
• Facilities
• User support
• Accelerated Publishing

Tasks

Preview Give feedback

1. Investigate Flakey Cypress Tests from 3/7/2024 Daily Deploy Failure #17465
   CMS Team +1
   

[gracekretschmer-metrostar]

Notes from 3/26/2024 meeting:

• Opportunity to leverage synthetic monitoring for Data Dog
• For the comprehensive test plan, make sure to include a smoke test, identify the critical pathways to determine automated tests
• Need to do: determine how the comprehensive test plan will be included with product team's testing.
• Within the collab cycle, we should likely have QA pulled in before staging review. TBD when, maybe at mid-point review?
• Next steps: have Jon sit on a few design intent sessions to see if there's value to have QA represented during that point in the collab cycle.

[gracekretschmer-metrostar]

4/3/2024 Jon's feedback: discovery around automated test plan.

[gracekretschmer-metrostar]

https://vfs.atlassian.net/wiki/spaces/~712020eee99537517b4d9594003707b252e24b/pages/3079766076/CMS+Test+Plan+Discovery

[gracekretschmer-metrostar]

@keisterj-oddball do you consider this done? Looking at your confluence page, I would say so, but wanted to confirm with you first before closing.

[keisterj-oddball]

@gracekretschmer-metrostar I think we could call this done, then I'll spin up more tickets for next steps.

[gracekretschmer-metrostar]

Great! My headspace for next sprint, is to have you focused on the collab cycle and troubleshooting runbook. I moved our 1:1 to today, so that we can refine these tickets together.

#18110
#18171