vault crashed with : panic: runtime error: invalid memory address or nil pointer dereference

[wwalker]

This is the most likely pertinent info:

2015/12/06 13:29:51 [INFO] core: pre-seal teardown starting
2015/12/06 13:29:51 [INFO] rollback: stopping rollback manager
2015/12/06 13:29:51 [INFO] core: pre-seal teardown complete
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x94b267]

goroutine 194700 [running]:
github.com/hashicorp/vault/vault.(*ExpirationManager).emitMetrics(0x0)
  /gopath/src/github.com/hashicorp/vault/vault/expiration.go:613 +0xf7
github.com/hashicorp/vault/vault.(*Core).emitMetrics(0xc82007cdc0, 0xc8204c3080)
  /gopath/src/github.com/hashicorp/vault/vault/core.go:1624 +0xa8
created by github.com/hashicorp/vault/vault.(*Core).postUnseal
  /gopath/src/github.com/hashicorp/vault/vault/core.go:1380 +0x961

Attached log file excerpt: vault.log-excerpt.txt

[jefferai]

Hi Wayne,

What version of Vault is this?

Thanks,
Jeff

[jefferai]

@wwalker FYI, those line numbers seem to correspond to an older version of Vault (earlier in dev, or 0.3.1). I think that this was fixed by #694. Any chance you can confirm/deny?

[wwalker]

Definitely running Vault v0.3.1

Let's just close this and we'll see if it comes back after we upgrade ( which will be a while.

[samira-barouti]

We are facing the same issue with v0.6.2. Unable to unseal the vault. Any idea?

{VAULT_HOME}/vault --version
Vault v0.6.2

journalctl --unit vault
-- Logs begin at Wed 2018-03-07 02:07:40 GMT, end at Wed 2018-03-07 09:21:10 GMT. --
Starting Vault service...
Starting vault with conf /opt/vault/etc/vault.hcl: [ OK ]
Started Vault service.
http: panic serving 127.0.0.1:51094: runtime error: invalid memory address or nil pointer dereference
goroutine 10417 [running]:
net/http.(*conn).serve.func1(0xc42056ee80)
/goroot/src/net/http/server.go:1491 +0x12a
panic(0x12f6bc0, 0xc42000e030)
/goroot/src/runtime/panic.go:458 +0x243
github.com/hashicorp/vault/vault.(*Core).teardownAudits(0xc420052a80, 0x0, 0x0)
/gopath/src/github.com/hashicorp/vault/vault/audit.go:259 +0x85
github.com/hashicorp/vault/vault.(*Core).preSeal(0xc420052a80, 0x0, 0x0)
/gopath/src/github.com/hashicorp/vault/vault/core.go:1181 +0x1e5
github.com/hashicorp/vault/vault.(*Core).postUnseal.func1(0xc4200fd998, 0xc420052a80)
/gopath/src/github.com/hashicorp/vault/vault/core.go:1100 +0x42
github.com/hashicorp/vault/vault.(*Core).postUnseal(0xc420052a80, 0x1dac7c0, 0xc420234060)
/gopath/src/github.com/hashicorp/vault/vault/core.go:1141 +0x32b
github.com/hashicorp/vault/vault.(*Core).Unseal(0xc420052a80, 0xc42056d590, 0x21, 0x21, 0x0, 0x0, 0x0)
/gopath/src/github.com/hashicorp/vault/vault/core.go:810 +0xa7f
github.com/hashicorp/vault/http.handleSysUnseal.func1(0x1db8a40, 0xc42052b040, 0xc420529d10)
/gopath/src/github.com/hashicorp/vault/http/sys_seal.go:118 +0x379
net/http.HandlerFunc.ServeHTTP(0xc420234df0, 0x1db8a40, 0xc42052b040, 0xc420529d10)
/goroot/src/net/http/server.go:1726 +0x44
net/http.(*ServeMux).ServeHTTP(0xc42000bef0, 0x1db8a40, 0xc42052b040, 0xc420529d10)
/goroot/src/net/http/server.go:2022 +0x7f
github.com/hashicorp/vault/http.handleHelpHandler.func1(0x1db8a40, 0xc42052b040, 0xc420529d10)
/gopath/src/github.com/hashicorp/vault/http/help.go:18 +0xd8
net/http.HandlerFunc.ServeHTTP(0xc420161ca0, 0x1db8a40, 0xc42052b040, 0xc420529d10)
/goroot/src/net/http/server.go:1726 +0x44
net/http.serverHandler.ServeHTTP(0xc420240380, 0x1db8a40, 0xc42052b040, 0xc420529d10)
/goroot/src/net/http/server.go:2202 +0x7d
net/http.(*conn).serve(0xc42056ee80, 0x1db9c00, 0xc420575080)
/goroot/src/net/http/server.go:1579 +0x4b7
created by net/http.(*Server).Serve
/goroot/src/net/http/server.go:2293 +0x44d

[jefferai]

That panic is a bug that has been fixed for more than a year and a half.

[smitjainsj]

@jefferai It seems this bug was fixed earlier, but I am still getting this error again on the new version. Any help is appreciated.
$ vault server -config=config.hcl -log-level=debug
==> Vault server configuration:

                 Cgo: disabled
          Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", tls: "disabled")
           Log Level: debug
               Mlock: supported: false, enabled: false
             Storage: s3
             Version: Vault v0.9.5
         Version Sha: 36edb4d42380d89a897e7f633046423240b710d9

==> Vault server started! Log data will stream in below:

2018/03/29 15:53:05.789456 [INFO ] core: security barrier not initialized
2018/03/29 15:53:06.876811 [INFO ] core: security barrier initialized: shares=1 threshold=1
2018-03-29 15:53:07.126404 I | http: panic serving 127.0.0.1:53339: runtime error: invalid memory address or nil pointer dereference
goroutine 60 [running]:
net/http.(*conn).serve.func1(0xc420426640)
/goroot/src/net/http/server.go:1697 +0xd0
panic(0x2b83920, 0x4129f90)
/goroot/src/runtime/panic.go:491 +0x283
github.com/hashicorp/vault/physical/s3.(*S3Backend).Get(0xc420584c90, 0x40c0bc0, 0xc4200140b0, 0x2f59e20, 0xc, 0x0, 0x0, 0x0)
/gopath/src/github.com/hashicorp/vault/physical/s3/s3.go:192 +0x2cd
github.com/hashicorp/vault/vault.(*sealUnwrapper).Get(0xc420583500, 0x40c0bc0, 0xc4200140b0, 0x2f59e20, 0xc, 0x0, 0x0, 0x0)
/gopath/src/github.com/hashicorp/vault/vault/sealunwrapper.go:66 +0x95
github.com/hashicorp/vault/physical.(*Cache).Get(0xc420185e00, 0x40c0bc0, 0xc4200140b0, 0x2f59e20, 0xc, 0x0, 0x0, 0x0)
/gopath/src/github.com/hashicorp/vault/physical/cache.go:110 +0x37f
github.com/hashicorp/vault/vault.(*AESGCMBarrier).Unseal(0xc4205716e0, 0x40c0bc0, 0xc4200140b0, 0xc42067f720, 0x20, 0x20, 0x0, 0x0)
/gopath/src/github.com/hashicorp/vault/vault/barrier_aes_gcm.go:336 +0x14f
github.com/hashicorp/vault/vault.(*Core).Initialize(0xc420564d80, 0x40c0bc0, 0xc4200140b0, 0xc4204f5a38, 0x0, 0x0, 0x0)
/gopath/src/github.com/hashicorp/vault/vault/init.go:153 +0xc1d
github.com/hashicorp/vault/http.handleSysInitPut(0xc420564d80, 0x40bf3c0, 0xc42019e0e0, 0xc420176200)
/gopath/src/github.com/hashicorp/vault/http/sys_init.go:105 +0x346
github.com/hashicorp/vault/http.handleSysInit.func1(0x40bf3c0, 0xc42019e0e0, 0xc420176200)
/gopath/src/github.com/hashicorp/vault/http/sys_init.go:20 +0x8f
net/http.HandlerFunc.ServeHTTP(0xc42062a3e0, 0x40bf3c0, 0xc42019e0e0, 0xc420176200)
/goroot/src/net/http/server.go:1918 +0x44
net/http.(*ServeMux).ServeHTTP(0xc4205850b0, 0x40bf3c0, 0xc42019e0e0, 0xc420176200)
/goroot/src/net/http/server.go:2254 +0x130
github.com/hashicorp/vault/http.wrapHelpHandler.func1(0x40bf3c0, 0xc42019e0e0, 0xc420176200)
/gopath/src/github.com/hashicorp/vault/http/help.go:22 +0x166
net/http.HandlerFunc.ServeHTTP(0xc4205792a0, 0x40bf3c0, 0xc42019e0e0, 0xc420176200)
/goroot/src/net/http/server.go:1918 +0x44
github.com/hashicorp/vault/http.wrapCORSHandler.func1(0x40bf3c0, 0xc42019e0e0, 0xc420176200)
/gopath/src/github.com/hashicorp/vault/http/cors.go:32 +0x10a
net/http.HandlerFunc.ServeHTTP(0xc4205792c0, 0x40bf3c0, 0xc42019e0e0, 0xc420176200)
/goroot/src/net/http/server.go:1918 +0x44
github.com/hashicorp/vault/http.wrapGenericHandler.func1(0x40bf3c0, 0xc42019e0e0, 0xc420176200)
/gopath/src/github.com/hashicorp/vault/http/handler.go:109 +0xb1
net/http.HandlerFunc.ServeHTTP(0xc4205792e0, 0x40bf3c0, 0xc42019e0e0, 0xc420176200)
/goroot/src/net/http/server.go:1918 +0x44
github.com/hashicorp/vault/vendor/github.com/hashicorp/go-cleanhttp.PrintablePathCheckHandler.func1(0x40bf3c0, 0xc42019e0e0, 0xc420176200)
/gopath/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/go-cleanhttp/handlers.go:40 +0xcf
net/http.HandlerFunc.ServeHTTP(0xc420579300, 0x40bf3c0, 0xc42019e0e0, 0xc420176200)
/goroot/src/net/http/server.go:1918 +0x44
net/http.serverHandler.ServeHTTP(0xc42057d380, 0x40bf3c0, 0xc42019e0e0, 0xc420176200)
/goroot/src/net/http/server.go:2619 +0xb4
net/http.(*conn).serve(0xc420426640, 0x40c0b80, 0xc4205c7880)
/goroot/src/net/http/server.go:1801 +0x71d
created by net/http.(*Server).Serve
/goroot/src/net/http/server.go:2720 +0x288

[jefferai]

@smitjainsj File a new ticket, please.

[smitjainsj]

@jefferai Here is the link for the new issue .
#4222