Epic: establish a centralized observability environment

[sduranc]

• Currently we have grafana per environment/(service)owner. This is not optimal when it comes to on-call or just having a good overview of all the environments. Therefore we need to gather observability signals in one place.
• The initial idea is to rely on managed solutions but we should evaluate the feasibility of this
• This epic will have as an outcome a proposed solution, along with an RFC and a PoC.

Tasks

Preview Give feedback

1. Create centralized grafana #908
   3 of 3
   PartOfEpic area/observability kind/user-story +3
   
2. Create centralized prometheus #909
   2 of 2
   PartOfEpic area/observability kind/user-story +3
   
3. Create centralized tracing solution #910
   area/observability +1
4. Define the agent that exports obs signals to the centralized obs env #911
   4 of 4
   area/observability +1
   
5. Dashboards should be reconciled from one source of truth to all grafanas #912
   PartOfEpic area/observability +2
   
6. Define an alert manager solution for predefined alerts (slos etc) #913
   area/observability +1
   
7. Define external/public dashboard #915
   area/observability +1
8. Filter out metrics #917
   area/observability +1
9. Decide on Status page #918
   area/observability +1
   
10. Enable traces from CI/CD workflows for all teams/services #919
    area/observability +1
    
11. Deploy Pyrra in the apps clusters #995
    PartOfEpic area/observability +2
    
12. Deploy Traefik SLO similar to the one defined in the platform clusters #996
    PartOfEpic area/observability +2
    
13. Reconcile the Pyrra generated PrometheusRule CRs with Azure PrometheusRuleGroups resources #997
    PartOfEpic area/observability +2
    
14. Create a Dashboard to visualize the Traefik SLO from the various environments #998
    PartOfEpic area/observability +2
15. Send selected metrics from platform clusters to the centralized azure monitor #1000
    PartOfEpic area/observability +2
    
16. Refactor the Monitoring module so it can be more easily re-used #1025
    PartOfEpic area/observability +2
    
17. Add annotations in grafana for each deployment #1083
    +0
    
18. Find and alternative way to generate metrics from nginx logs #1084
    +0
    
19. Deploy centralized monitoring solution #1115
    PartOfEpic area/observability +2
    
20. Re-create the DCR and DCRA in the TT02 environments #1120
    PartOfEpic area/observability +2