Intermittent crash on token request

[matthewmrichter]

Hi, we have experienced this issue intermittently where VGM will throw a memory exception when a token is requested (see below).

[31m2016/10/18 19:25:03 [Recovery] panic recovered:
POST /token HTTP/1.1
Host: vgm.foo.com
Accept: */*
Content-Length: 48
Content-Type: application/json
User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.21 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2


runtime error: invalid memory address or nil pointer dereference
/usr/local/go/src/runtime/panic.go:458 (0x43f0f3)
    gopanic: reflectcall(nil, unsafe.Pointer(d.fn), deferArgs(d), uint32(d.siz), uint32(d.siz))
/usr/local/go/src/runtime/panic.go:62 (0x43dc4d)
    panicmem: panic(memoryError)
/usr/local/go/src/runtime/sigpanic_unix.go:24 (0x453af4)
    sigpanic: panicmem()
/go/src/github.com/channelmeter/vault-gatekeeper-mesos/provider.go:63 (0x408295)
    createWrappedToken: defer r.Body.Close()
/go/src/github.com/channelmeter/vault-gatekeeper-mesos/provider.go:107 (0x408b09)
    createTokenPair: return createWrappedToken(token, permTokenOpts, 10*time.Minute)
/go/src/github.com/channelmeter/vault-gatekeeper-mesos/provider.go:202 (0x409924)
    Provide: if tempToken, err := createTokenPair(token, policy); err == nil {
/go/src/github.com/gin-gonic/gin/context.go:97 (0x4c44ba)
    (*Context).Next: c.handlers[c.index](c)
/go/src/github.com/gin-gonic/gin/recovery.go:45 (0x4d3efa)
    RecoveryWithWriter.func1: c.Next()
/go/src/github.com/gin-gonic/gin/context.go:97 (0x4c44ba)
    (*Context).Next: c.handlers[c.index](c)
/go/src/github.com/gin-gonic/gin/logger.go:63 (0x4d3067)
    LoggerWithWriter.func1: c.Next()
/go/src/github.com/gin-gonic/gin/context.go:97 (0x4c44ba)
    (*Context).Next: c.handlers[c.index](c)
/go/src/github.com/gin-gonic/gin/gin.go:284 (0x4ca5be)
    (*Engine).handleHTTPRequest: context.Next()
/go/src/github.com/gin-gonic/gin/gin.go:265 (0x4c9ea0)
    (*Engine).ServeHTTP: engine.handleHTTPRequest(c)
/usr/local/go/src/net/http/server.go:2202 (0x51f2cd)
    serverHandler.ServeHTTP: handler.ServeHTTP(rw, req)
/usr/local/go/src/net/http/server.go:1579 (0x51bc37)
    (*conn).serve: serverHandler{c.server}.ServeHTTP(w, w.req)
/usr/local/go/src/runtime/asm_amd64.s:2086 (0x46e8d1)
    goexit: BYTE    $0x90    // NOP
�[0m

The chronos task failed, then re-ran. When it re-ran, the token worked fine. Any idea what would cause this? Could we need to allocate more memory to the VGM process?

Thanks
Matt

[nemosupremo]

This is caused by a bug in the code. There was likely an error communicating with Vault, but the code in provider.go, tries to close a nil body (because an error occurred).

There is likely another communication error occurring, that is being swallowed by the panic. The patch is pretty simple.

[nemosupremo]

I'll cut a new release by EOD tomorrow.