[Question][DORA] Benchmarks appear to be outdated

[elijah-roberts]

Question

Should we update benchmarks to match 2023 report?

Current state it appears that several DORA benchmarks may be outdated.

The for example the 2023 DevOps report indicates that high performing teams deploy "Between once per day and once per week".

Currently dashboards indicate that a high performing team is "Between once per week and once per month"

These are the instances I found referencing the outdated benchmark for deployment frequency.

https://github.com/search?q=repo%3Aapache%2Fincubator-devlake+%22Between+once+per+week+and+once+per+month%22&type=code

However, I imagine that there may need to be some changes to calculations as well.

Screenshots

2020 Report: https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance

2023 Report: 2023_final_report_sodr.pdf

Additional context

Summary of changes:

Here's a comparison table highlighting the changes between the 2020 and 2023 DevOps reports:

Performance Level	Deployment Frequency (2020)	Deployment Frequency (2023)	Lead Time for Changes (2020)	Lead Time for Changes (2023)	Change Failure Rate (2020)	Change Failure Rate (2023)	Time to Restore Service (2020)	Time to Restore Service (2023)
Elite	On-demand	On demand	Less than one day	Less than one day	0%-15%	5%	Less than one hour	Less than one hour
High	Between once per week and per month	Between once per day and per week	Between one day and one week	Between one day and one week	16%-30%	10%	Less than one day	Less than one day
Medium	Between once per month and per 6 months	Between once per week and per month	Between one month and six months	Between one week and one month	16%-30%	15%	Between one day and one week	Between one day and one week
Low	Fewer than once per six months	Between once per week and per month	More than six months	Between one week and one month	16%-30%	64%	More than six months	Between one month and six months

Key Changes:

1. Deployment Frequency:

   • High: Shifted from "Between once per week and per month" to "Between once per day and per week."
   • Medium: Shifted from "Between once per month and per 6 months" to "Between once per week and per month."
   • Low: Shifted from "Fewer than once per six months" to "Between once per week and per month."

2. Lead Time for Changes:

   • Medium: Shifted from "Between one month and 6 months" to "Between one week and one month"
   • Low: Shifted from "More than six months" to "Between one week and one month"

3. Change Failure Rate:

   • Elite: Reduced from 0%-15% to 5%.
   • High: Reduced from 16%-30% to 10%.
   • Medium: Reduced from 16%-30% to 15%.
   • Low: Significant increase from 16%-30% to 64%.

4. Time to Restore Service:

   • Low: Decrease from "More than six months" to "Between one month and six months."

TODO LIST:

• update dora dashboard
• update dora team dashboard
• add 2023 dora benchmarks docs: https://devlake.apache.org/docs/Metrics/LeadTimeForChanges/#how-is-it-calculated

[Startrekzky]

Yes, you're correct. Will update it in the next version.