Skip to content
This repository has been archived by the owner on Jan 19, 2024. It is now read-only.

fix: Deployment Type in remediation.triggered incompatible with Keptn cloud events #366

Merged
merged 1 commit into from
Aug 30, 2022

Conversation

christian-kreuzberger-dtx
Copy link
Contributor

@christian-kreuzberger-dtx christian-kreuzberger-dtx commented Aug 29, 2022

During testing I noticed that our remediation workflow never gets into an evaluation.triggered, unfortunately.

We end up getting an error message in Keptn's lighthouse-service like this:

ime="2022-08-29T12:15:49Z" level=error msg="Could not parse event payload: [json] found bytes \"{\"action\":{\"action\":\"scaling\",\"description\":\"Scale up\",\"name\":\"scaling\",\"value\":\"1\"},\"deployment\":\"primary\",\"evaluation\":{\"timeframe\":\"15m\"},\"get-action\":{\"actionIndex\":1},\"labels\":{\"Problem URL\":\"http://prometheus-server-57f4946cdb-fghkq:9090/graph?g0.expr=histogram_quantile%280.9%2C+sum+by%28le%29+%28rate%28http_response_time_milliseconds_bucket%7Bhandler%3D%22ItemsController.addToCart%22%2Cjob%3D%22carts-sockshop-production-primary%22%7D%5B3m%5D%29%29%29+%3E+1000\\u0026g0.tab=1\"},\"message\":\"\",\"problem\":{\"ImpactedEntity\":\"carts-primary\",\"PID\":\"\",\"ProblemDetails\":{\"problemDetails\":\"Pod name \"},\"ProblemID\":\"\",\"ProblemTitle\":\"response_time_p90\",\"ProblemURL\":\"http://prometheus-server-57f4946cdb-fghkq:9090/graph?g0.expr=histogram_quantile%280.9%2C+sum+by%28le%29+%28rate%28http_response_time_milliseconds_bucket%7Bhandler%3D%22ItemsController.addToCart%22%2Cjob%3D%22carts-sockshop-production-primary%22%7D%5B3m%5D%29%29%29+%3E+1000\\u0026g0.tab=1\",\"State\":\"OPEN\",\"labels\":{\"deployment\":\"primary\"},\"project\":\"sockshop\",\"service\":\"carts\",\"stage\":\"production\"},\"project\":\"sockshop\",\"result\":\"pass\",\"service\":\"carts\",\"stage\":\"production\",\"status\":\"succeeded\",\"temporaryData\":{\"distributor\":{\"subscriptionID\":\"d5596f2b-0f33-4fd2-a4c2-bb658f5ebe7b\"}}}\", but failed to unmarshal: json: cannot unmarshal string into Go struct field EvaluationTriggeredEventData.deployment of type v0_2_0.Deployment"

The root-cause of this is the remediation.triggered event which contains "deployment": "primary", which is not compatible with lighthouse-service.

{
  "data": {
    "deployment": "primary", <------------ HERE
    "labels": {
      "Problem URL": "http://prometheus-server-6776d8bb7d-pqkln:9090/graph?g0.expr=histogram_quantile%280.9%2C+sum+by%28le%29+%28rate%28http_response_time_milliseconds_bucket%7Bhandler%3D%22ItemsController.addToCart%22%2Cjob%3D%22carts-sockshop-production-primary%22%7D%5B3m%5D%29%29%29+%3E+1000\\u0026g0.tab=1"
    },
    "problem": { ...
    }
  },
  "id": "8933f061-ff7e-4434-b4b4-ca2c0866825a",
  "shkeptncontext": "bdb4d1f9-cc6f-47a6-957b-620f28a2a873",
  "source": "prometheus",
  "specversion": "1.0",
  "time": "2022-08-29T14:12:41.783119566Z",
  "type": "sh.keptn.event.production.remediation.triggered"
}

This PR fixes it, and adapts the end-to-end test to detect this kind of thing.

Proof that it works

Integration Test Run:https://github.com/keptn-contrib/prometheus-service/actions/runs/2949712155

Manual Run:
image

@github-actions
Copy link
Contributor

github-actions bot commented Aug 29, 2022

Unit Test Results

12 tests   11 ✔️  0s ⏱️
  6 suites    1 💤
  1 files      0

Results for commit c899ac4.

♻️ This comment has been updated with latest results.

@christian-kreuzberger-dtx christian-kreuzberger-dtx force-pushed the patch/fix-remediation-triggered branch from 12adcfd to 6b985b1 Compare August 29, 2022 14:48
@christian-kreuzberger-dtx christian-kreuzberger-dtx force-pushed the patch/fix-remediation-triggered branch from 6b985b1 to c899ac4 Compare August 29, 2022 15:18
@github-actions
Copy link
Contributor

The following Docker Images have been built:
keptncontrib/prometheus-service:0.9.1-dev-PR-366,keptncontrib/prometheus-service:0.9.1-dev-PR-366.202208291519

Copy link
Member

@Raffy23 Raffy23 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@christian-kreuzberger-dtx christian-kreuzberger-dtx merged commit af1db26 into master Aug 30, 2022
@christian-kreuzberger-dtx christian-kreuzberger-dtx deleted the patch/fix-remediation-triggered branch August 30, 2022 06:27
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants