POST to /api/alerts causes invalid memory address or nil pointer dereference

[discordianfish]

The current prometheus is crashing alertmanager when sending alerts:

2014/04/03 11:42:04 /go/src/code.google.com/p/gorest/gorest.go:194 (0x572cf3)
  google.com/p/gorest.func.001: log.Printf("%s", debug.Stack())
/usr/local/go/src/pkg/runtime/panic.c:248 (0x415766)
  panic: runtime·newstackcall(d->fn, (byte*)d->args, d->siz);
/usr/local/go/src/pkg/runtime/panic.c:482 (0x41600d)
  panicstring: runtime·panic(err);
/usr/local/go/src/pkg/runtime/os_linux.c:234 (0x414eaa)
  sigpanic: runtime·panicstring("invalid memory address or nil pointer dereference");
/go/src/github.com/prometheus/alertmanager/web/api/alert.go:40 (0x46d70c)
  com/prometheus/alertmanager/web/api.AlertManagerService.AddAlerts: s.Manager.Receive(as)
/go/src/github.com/prometheus/alertmanager/web/api/alert.go:1 (0x46e6d4)
  com/prometheus/alertmanager/web/api.(*AlertManagerService).AddAlerts: // Copyright 2013 Prometheus Team
/usr/local/go/src/pkg/runtime/asm_amd64.s:339 (0x426b32)
  call32: CALLFN(call32, 32)
/usr/local/go/src/pkg/reflect/value.go:474 (0x490a7b)
  Value.call: call(fn, ptr, uint32(size))
/usr/local/go/src/pkg/reflect/value.go:345 (0x48fb6d)
  Value.Call: return v.call("Call", in)
/go/src/code.google.com/p/gorest/reflect.go:391 (0x5708c0)
  google.com/p/gorest.prepareServe: ret = servVal.Method(ep.methodNumberInParent).Call(arrArgs)
/go/src/code.google.com/p/gorest/gorest.go:215 (0x56771b)
  google.com/p/gorest.manager.ServeHTTP: data, state := prepareServe(ctx, ep)
/go/src/code.google.com/p/gorest/api.go:1 (0x5739ae)
  google.com/p/gorest.(*manager).ServeHTTP: //Copyright 2011 Siyabonga Dlamini ([email protected]). All rights reserved.
/go/src/github.com/prometheus/alertmanager/web/compression.go:90 (0x46b2b2)
  com/prometheus/alertmanager/web.compressionHandler.ServeHTTP: c.handler.ServeHTTP(compWriter, req)
/go/src/github.com/prometheus/alertmanager/web/alerts.go:1 (0x46d3ee)
  com/prometheus/alertmanager/web.(*compressionHandler).ServeHTTP: // Copyright 2013 Prometheus Team
/go/src/github.com/prometheus/client_golang/prometheus/exp/coarsemux.go:65 (0x573e0a)
  com/prometheus/client_golang/prometheus/exp.handlerDelegator.ServeHTTP: h.delegate.ServeHTTP(rwd, r)
/go/src/github.com/prometheus/client_golang/prometheus/exp/coarsemux.go:1 (0x5755b4)
  com/prometheus/client_golang/prometheus/exp.(*handlerDelegator).ServeHTTP: // Copyright (c) 2013, Prometheus Team
/usr/local/go/src/pkg/net/http/server.go:1496 (0x537f03)
  (*ServeMux).ServeHTTP: h.ServeHTTP(w, r)
/usr/local/go/src/pkg/net/http/server.go:1597 (0x53870e)
  serverHandler.ServeHTTP: handler.ServeHTTP(rw, req)
/usr/local/go/src/pkg/net/http/server.go:1167 (0x5365a7)
  (*conn).serve: serverHandler{c.server}.ServeHTTP(w, w.req)
/usr/local/go/src/pkg/runtime/proc.c:1394 (0x419a60)
  goexit: runtime·goexit(void)

I've did a tcpdump and reproduced what prometheus posts:

 curl -L -d '[{"Description":"http://192.168.100.66:9080/metrics has been down for more than 5 minutes.","Labels":{"alertname":"NodeDown","instance":"http://192.168.100.66:9080/metrics","job":"node_exporter","severity":"page"},"Payload":{"ActiveSince":"2014-04-03T11:20:49Z","AlertingRule":"ALERT NodeDown IF (up == 0) FOR 2m WITH {severity=\"page\"}","GeneratorUrl":"http://19275cb8ad55:9090/graph#%5B%7B%22expr%22%3A%22%28up%20%3D%3D%200%29%22%2C%22tab%22%3A1%7D%5D","Value":"0.000000"},"Summary":"Node http://192.168.100.66:9080/metrics down"}]' -X POST localhost:9090/api/alerts

It seems that in web/api/alert.go:40 s.Manager is nil.

[juliusv]

This curl line doesn't crash the alertmanager for me. Instead, the alert gets shown on the AM's status page. Does this happen for you every time or only intermittently?

[discordianfish]

@juliusv Odd, for me it happens all the time. I just updated dependencies and did a fresh build and still the same. Maybe you have outdated dependencies?

[juliusv]

Yeah, breaks for me after updating dependencies. I'd guess probably it's another gorest upstream breakage. Can't guarantee I'll look at it soon though, since alertmanager is currently on ice.

[discordianfish]

@juliusv I can fix it, but a few hints would be great.

[juliusv]

@discordianfish I have only hazy memories, but we once had an issue where gorest was changed in such a way that state didn't get tracked properly anymore. Maybe the variables set in this service: https://github.com/prometheus/alertmanager/blob/master/web/api/api.go#L22 weren't accessible anymore because every handler got a fresh (empty? no idea) copy of the struct. I think we had a similar problem in Prometheus. You might want to check the history of web files there!

[juliusv]

@discordianfish Ah, from Prometheus: prometheus/prometheus@eba01d1

[discordianfish]

Ha :) Phew, okay the job is to remove gorest. Will look into that.

[matttproud]

+10 to removing gorest. It was great for prototyping but little else
beyond that.

On Thu, Apr 3, 2014 at 3:14 PM, discordianfish [email protected]:

Ha :) Phew, okay the job is to remove gorest. Will look into that.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/15#issuecomment-39448964
.

Key: 0xC42234343FDE43A8
Fingerprint: 6945 9248 4362 68CB EA83 F1EA C422 3434 3FDE 43A8

[juliusv]

This has been fixed a while ago.