Alertmanager/Opsgenie: AM dosen't update alerts fileds (message and description) with the new informations

[HamidEDD]

What did you do?
I have created a group of alerts based on the __name__ with the rule bellow :

groups:
- name: Monitoring Probes 
  rules:
  - alert: "Test"
    description: "Test Probes"
    expr: "label_replace({__name__=~'test_.+',job='node-exporter'} >= 290,'metric', '$1', '__name__', '(.*)')"
    labels:
      alert_type: "alerting"
      severity: 'P1'
    annotations:
      summary: "{{ $labels.metric }} = {{ $value }}"

In the Alertmanager side, I selected to group_by (instance,alertname,severity) with this configuration:

global:
 opsgenie_api_url: 'https://api.eu.opsgenie.com/'
route:
 group_by: ['alertname','instance','severity']
 group_wait: 1m
 group_interval: 2m
 repeat_interval: 5m
 routes:
  - match:
      alert_type: 'alerting'
    receiver: opsgenie
receivers:
- name: opsgenie
  opsgenie_configs:
  - send_resolved: true
    source: 'Alertmanager'
    message: '{{ .GroupLabels.alertname }} {{ .GroupLabels.instance }} {{if eq .GroupLabels.alertname "Test"}}{{ range .Alerts }}{{ .Labels.metric }},{{ end }}{{ end }}'
    description: "Alerts:\n{{ range .Alerts }}{{ .Annotations.summary }}\n{{ end }}"
    priority: '{{ .GroupLabels.severity }}'
    responders:
    - name: 'alertmanager-responder'
      type: 'team'

What did you expect to see?
When Test rings (for example: test_service1, test_service2), the alerts arrive within 1m (goup_wait) and everything works fine.
Alertmanager notifies opsgenie, the message and description fields contain the correct information

message : Test host1.localhost.local test_service1,test_service2

What did you see instead? Under which circumstances?
But when another Test alert (test_service3) arrives later (after 3m), the message/description in opsgenie seem to always have the first value:
message : Test host1.localhost.local test_service1,test_service2
instead of :
message : Test host1.localhost.local test_service1,test_service2,test_service3

it seems that Alertmanager does not update the message and description fields on the opsgenie side, when an alert is updated.
I reproduce the same issue with several different alert rules and with a group_by (instance, severity)

Environment

• System information:
  Linux 4.18.0-25-generic x86_64

• Alertmanager version:

alertmanager, version 0.21.0 (branch: HEAD, revision: 4c6c03ebfe21009c546e4d1e9b92c371d67c021d)
  build user:       root@dee35927357f
  build date:       20200617-08:54:02
  go version:       go1.14.4

• Prometheus version:

prometheus, version 2.19.0 (branch: HEAD, revision: 5d7e3e970602c755855340cb190a972cebdd2ebf)
  build user:       root@d4cf5c7e268d
  build date:       20200609-10:29:59
  go version:       go1.14.4

[HamidEDD]

Hi @ricoberger, sorry for the noise :( I see that you're the last one who has contributed to opsgenie integration, do you have any idea about this issue please ?

[ricoberger]

Hi @HamidEDD, we had the same problem some months ago. We tried also to modify an existing alert directly via the OpsGenie API, without success.

We finally decided to rework our grouping of alerts 😞

[HamidEDD]

Thanks for your feedback @ricoberger :)

Have you tried this :
https://docs.opsgenie.com/docs/alert-api-continued#update-alert-message
https://docs.opsgenie.com/docs/alert-api-continued#update-alert-description

it seems that opsgenie support the update of message and description now.

[ricoberger]

No, I haven't tried this yet, but nice finding.

I'm not sure if this can be implemented easily in the Opsgenie provider, because when I'm right we don't know if it is a new alert or an updated alert.

Only thing I can imagine right now, is to just try the update and if this fails then create a new alert, but this wouldn't be a nice implementation.

[tomasfreund]

Hi, we are facing the same issue.

I have contacted OpsGenie support and they told me that there is a ticket for updating alert content on deduplication but did not give me any further info regarding whether is is planned or when it would be completed.

Until something changes, I think the only option is to try to update the alert first as @ricoberger mentioned.
It's not a very nice solution but I don't see any other ones. Perhaps it could be disabled by default?

[tomasfreund]

Pull request is ready in case someone would like to review it #2519

[HamidEDD]

Hi @roidelapluie,

can you take a look to the PR submitted by @tomasfreund please ?
I tested it on our preprod infra, it absolutely fixed my issue :)
Many thanks.

[crepetl]

Hello,
We would be super happy to have this feature as well, as we are grouping alerts. We tried notes, but as the whole group of alerts is pushed each time (and I found no way to get something incremental), this does not bring a lot. For now, we would go with more generic alert message and description, including with the URL pointing to AlertManager with pre-filled filters to show the group of alerts, and then notes for all alert sources and so on.
This may be also painful as OpsGenie may have default settings to notify the alert responder when note is added to the alert.

So, definitively waiting to see PR #2519 being merged.