Skip to content
This repository has been archived by the owner on Jan 12, 2024. It is now read-only.

docs: Improve troubleshooting content for evaluation failed #734

Merged
merged 2 commits into from
Mar 9, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file not shown.
Binary file removed documentation/images/dashboard-not-found.png
Binary file not shown.
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
78 changes: 52 additions & 26 deletions documentation/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,46 +39,72 @@ Likely cause:
- SLIs and SLOs should be defined using YAML files but no SLOs have been defined i.e. no `slo.yaml` file can be found

Suggested solution:
- If SLIs and SLOs should be defined using YAML files, define SLOs by creating a `slo.yaml` file, or
- If SLIs and SLOs should be sourced from a dashboard, add a [`dashboard` entry to the `dynatrace/dynatrace.conf.yaml` configuration file](dynatrace-conf-yaml-file.md#dashboard-sli-mode-configuration-dashboard)

## Evaluation fails: Dynatrace Metrics API returned zero data points

## Evaluation fails: No evaluation performed by Lighthouse because SLI failed

Evaluation fails and the shipyard-controller reports:
> `no evaluation performed by lighthouse because SLI failed with message Dynatrace Metrics API returned zero data points...`
> `no evaluation performed by lighthouse because SLI failed....`

![Dynatrace Metrics API returned zero data points](images/dynatrace-metrics-api-returned-zero-data-points.png)
![No evaluation performed by Lighthouse](images/no-evaluation-performed-by-lighthouse.png)

Likely cause:
- No data is available during the evaluation timeframe
In this case the Lighthouse performed no evaluation because the received `sh.keptn.event.get-sli.finished` event had a `result` of `fail`. The root cause can be identified from the `message` field of this event.

Suggested solution:
- Check the availability of data for the evaluation timeframe, e.g. using the Data Explorer in the Dynatrace tenant
In instances where this is related to a specific SLI, check the `message` field of the individual SLI result in the within `indicatorValues`. This can be viewed in Keptn UI by expanding the `sh.keptn.event.get-sli.finished` event and viewing its payload:

## Evaluation fails: Could not query Dynatrace dashboard for SLIs: ...could not find a matching dashboard name
![View `sh.keptn.event.get-sli.finished` payload](images/get-sli-finished-event-payload.png)

Evaluation fails and the shipyard-controller reports:
> `could not query Dynatrace dashboard for SLIs: error while processing dashboard config '' - could not find a matching dashboard name - e.g. KQG;project=<project>;service=<service>;stage=<stage>`
The following subsections detail some common messages as well as likely causes and solutions:

![Could not find a matching dashboard](images/could-not-find-a-matching-dashboard.png)
### Message: `Could not retrieve any SLI results`

Likely cause:
- Dynatrace-service is configured to query a dashboard for SLIs (i.e. `dynatrace/dynatrace.conf.yaml` contains `dashboard: query`) but no dashboard named with the pattern `KQG;project=<project>;service=<service>;stage=<stage>` is available
**Likely cause**

Suggested solution:
- If SLIs and SLOs should be sourced from a dashboard, create a dashboard named with the pattern `KQG;project=<project>;service=<service>;stage=<stage>` in the Dynatrace tenant, or
- If SLIs and SLOs should be defined using YAML files, remove the [`dashboard: query` entry from `dynatrace/dynatrace.conf.yaml`](dynatrace-conf-yaml-file.md#dashboard-sli-mode-configuration-dashboard)
- SLIs and SLOs should be defined using YAML files but no SLOs have been defined, i.e. no `slo.yaml` file can be found

## Evaluation fails: Could not query Dynatrace dashboard for SLIs: ...Dashboard "..." not found
**Suggested solution**

Evaluation fails and the shipyard-controller reports:
> `could not query Dynatrace dashboard for SLIs: error while processing dashboard config '12345678-1234-1234-1234-12345678abcd' - Dynatrace API error (404): Dashboard 12345678-1234-1234-1234-12345678abcd not found`
- If SLIs and SLOs should be defined using YAML files, define SLOs by creating a `slo.yaml` file, or
- If SLIs and SLOs should be sourced from a dashboard, add a [`dashboard` entry to the `dynatrace/dynatrace.conf.yaml` configuration file](dynatrace-conf-yaml-file.md#dashboard-sli-mode-configuration-dashboard)

![Dashboard not found](images/dashboard-not-found.png)
### Message: `Metrics API v2 returned zero data points`

**Likely cause**

- No data is available for an SLI during the evaluation timeframe

**Suggested solution**

- Check the availability of data for the associated SLIs during the evaluation timeframe, e.g. using the Data Explorer in the Dynatrace tenant

### Message: `Metrics API v2 returned more than one data point`

**Likely cause**

- Multiple data points are returned for an SLI during the evaluation timeframe where only one is expected

**Suggested solution**

- Ensure the SLI only returns a single data point by e.g. including a `merge` or `splitBy` transformation in the query

### Message: `could not query Dynatrace dashboard for SLIs: error while processing dashboard config '12345678-1234-1234-1234-12345678abcd' - Dynatrace API error (404): Dashboard 12345678-1234-1234-1234-12345678abcd not found`

**Likely cause**

Likely cause:
- Dynatrace-service is configured to query a specific dashboard for SLIs (i.e. [`dynatrace/dynatrace.conf.yaml` contains `dashboard: <dashboard-id>`](dynatrace-conf-yaml-file.md#dashboard-sli-mode-configuration-dashboard)) but no dashboard with the specified ID is available

Suggested solution:
- If SLIs and SLOs should be sourced from a dashboard, ensure `dynatrace/dynatrace.conf.yaml` contains a `dashboard: <dashboard-id>` entry with the correct dashboard ID, or
- If SLIs and SLOs should be defined using YAML files, remove the `dashboard: <dashboard-id>` entry from `dynatrace/dynatrace.conf.yaml`
**Suggested solution**

- If SLIs and SLOs should be sourced from a dashboard, ensure `dynatrace/dynatrace.conf.yaml` contains a `dashboard: <dashboard-id>` entry with the correct dashboard ID, or
- If SLIs and SLOs should be defined using YAML files, remove the `dashboard: <dashboard-id>` entry from `dynatrace/dynatrace.conf.yaml`

### Message: `could not query Dynatrace dashboard for SLIs: error while processing dashboard config '' - could not find a matching dashboard name - e.g. KQG;project=<project>;service=<service>;stage=<stage>`

**Likely cause**

- Dynatrace-service is configured to query a dashboard for SLIs (i.e. `dynatrace/dynatrace.conf.yaml` contains `dashboard: query`) but no dashboard named with the pattern `KQG;project=<project>;service=<service>;stage=<stage>` is available

**Suggested solution**

- If SLIs and SLOs should be sourced from a dashboard, create a dashboard named with the pattern `KQG;project=<project>;service=<service>;stage=<stage>` in the Dynatrace tenant, or
- If SLIs and SLOs should be defined using YAML files, remove the [`dashboard: query` entry from `dynatrace/dynatrace.conf.yaml`](dynatrace-conf-yaml-file.md#dashboard-sli-mode-configuration-dashboard)