Vault panic

[mdsimoneau]

Environment:

• Vault Version: Vault v0.11.4
• Operating System/Architecture: Kubernetes v1.10.11 (w/Docker)

Vault Config File:

storage "s3" {
      bucket = "our-bucketname"
      region = "us-east-1"
    }

    listener "tcp" {
      address = "0.0.0.0:8200"
      tls_cert_file = "/etc/secretvault-cert/cert"
      tls_key_file = "/etc/secretvault-cert/key"
    }

    telemetry {
      statsd_address = "statsd-exporter:9125"
    }

    default_lease_ttl = "168h"
    max_lease_ttl = "720h"
    ui = true

Startup Log Output:

Forgive the log ordering as it may be slightly off here and there due to the way our logging service displays things.
December 10th 2018, 00:01:07.000	==> Vault server configuration:
December 10th 2018, 00:01:07.000	
December 10th 2018, 00:01:07.000	               Log Level: (not set)
December 10th 2018, 00:01:07.000	             Version Sha: 612120e76de651ef669c9af5e77b27a749b0dba3
December 10th 2018, 00:01:07.000	
December 10th 2018, 00:01:07.000	==> Vault server started! Log data will stream in below:
December 10th 2018, 00:01:07.000	                     Cgo: disabled
December 10th 2018, 00:01:07.000	                   Mlock: supported: true, enabled: true
December 10th 2018, 00:01:07.000	                 Storage: s3
December 10th 2018, 00:01:07.000	                 Version: Vault v0.11.4
December 10th 2018, 00:01:09.000	2018-12-10T05:01:09.721Z [INFO]  core: successfully mounted backend: type=generic path=secret/
December 10th 2018, 00:01:09.000	2018-12-10T05:01:09.722Z [INFO]  core: successfully mounted backend: type=pki path=root-ca/
December 10th 2018, 00:01:09.000	2018-12-10T05:01:09.722Z [INFO]  core: successfully mounted backend: type=transit path=transit/
December 10th 2018, 00:01:09.000	2018-12-10T05:01:09.722Z [INFO]  core: successfully mounted backend: type=cubbyhole path=cubbyhole/
December 10th 2018, 00:01:09.000	2018-12-10T05:01:09.722Z [INFO]  core: successfully mounted backend: type=system path=sys/
December 10th 2018, 00:01:09.000	2018-12-10T05:01:09.658Z [INFO]  core: loaded wrapping token key
December 10th 2018, 00:01:09.000	2018-12-10T05:01:09.658Z [INFO]  core: successfully setup plugin catalog: plugin-directory=
December 10th 2018, 00:01:09.000	2018-12-10T05:01:09.722Z [INFO]  core: successfully mounted backend: type=pki path=dev-ca/
December 10th 2018, 00:01:09.000	2018-12-10T05:01:09.605Z [INFO]  core: vault is unsealed
December 10th 2018, 00:01:09.000	2018-12-10T05:01:09.629Z [INFO]  core: post-unseal setup starting
December 10th 2018, 00:01:09.000	2018-12-10T05:01:09.722Z [INFO]  core: successfully mounted backend: type=pki path=intermediate-ca/
December 10th 2018, 00:01:09.000	2018-12-10T05:01:09.722Z [INFO]  core: successfully mounted backend: type=database path=database/
December 10th 2018, 00:01:09.000	2018-12-10T05:01:09.722Z [INFO]  core: successfully mounted backend: type=aws path=aws/
December 10th 2018, 00:01:09.000	2018-12-10T05:01:09.722Z [INFO]  core: successfully mounted backend: type=identity path=identity/
December 10th 2018, 00:01:10.000	2018-12-10T05:01:10.016Z [INFO]  core: successfully enabled credential backend: type=aws-ec2 path=aws-ec2/
December 10th 2018, 00:01:10.000	2018-12-10T05:01:10.535Z [INFO]  core: post-unseal setup complete
December 10th 2018, 00:01:10.000	2018-12-10T05:01:10.016Z [INFO]  core: successfully enabled credential backend: type=approle path=approle/
December 10th 2018, 00:01:10.000	2018-12-10T05:01:10.016Z [INFO]  core: successfully enabled credential backend: type=ldap path=ldap/
December 10th 2018, 00:01:10.000	2018-12-10T05:01:10.016Z [INFO]  core: restoring leases
December 10th 2018, 00:01:10.000	2018-12-10T05:01:10.497Z [INFO]  identity: entities restored
December 10th 2018, 00:01:10.000	2018-12-10T05:01:10.507Z [INFO]  identity: groups restored
December 10th 2018, 00:01:10.000	2018-12-10T05:01:10.016Z [INFO]  core: successfully enabled credential backend: type=token path=token/
December 10th 2018, 00:01:10.000	2018-12-10T05:01:10.016Z [INFO]  core: successfully enabled credential backend: type=kubernetes path=kubernetes/
December 10th 2018, 00:01:10.000	2018-12-10T05:01:10.016Z [INFO]  rollback: starting rollback manager
December 10th 2018, 00:01:10.000	2018-12-10T05:01:10.535Z [INFO]  core: starting listener: listener_address=0.0.0.0:8201
December 10th 2018, 00:01:10.000	2018-12-10T05:01:10.535Z [INFO]  core: serving cluster requests: cluster_listen_address=[::]:8201
December 10th 2018, 00:03:22.000	2018-12-10T05:03:22.894Z [INFO]  expiration: revoked lease: lease_id=auth/kubernetes/login/BLAH
December 10th 2018, 00:03:23.000	2018-12-10T05:03:23.316Z [INFO]  expiration: revoked lease: lease_id=auth/kubernetes/login/BLAH
December 10th 2018, 00:03:24.000	2018-12-10T05:03:24.054Z [INFO]  expiration: lease restore complete
December 10th 2018, 00:04:28.000	
December 10th 2018, 00:04:28.000	github.com/hashicorp/vault/builtin/logical/pki.(*backend).pathTidyWrite.func1.1(0x0, 0x0)
December 10th 2018, 00:04:28.000		/gopath/src/github.com/hashicorp/vault/builtin/logical/pki/path_tidy.go:116 +0x1370
December 10th 2018, 00:04:28.000	github.com/hashicorp/vault/builtin/logical/pki.(*backend).pathTidyWrite.func1(0xc000d6ad40, 0xc013094150, 0x4843001, 0xc0531e9cc0, 0x34630b8a000, 0x100)
December 10th 2018, 00:04:28.000	[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x14fe8b0]
December 10th 2018, 00:04:28.000		/gopath/src/github.com/hashicorp/vault/builtin/logical/pki/path_tidy.go:199 +0x170
December 10th 2018, 00:04:28.000	2018-12-10T05:04:28.375Z [WARN]  secrets.pki.pki_b02161ee.tidy: certificate entry is nil; tidying up since it is no longer useful for any server operations: serial=BLAH
December 10th 2018, 00:04:28.000	panic: runtime error: invalid memory address or nil pointer dereference
December 10th 2018, 00:04:28.000	goroutine 93248 [running]:
December 10th 2018, 00:04:28.000	created by github.com/hashicorp/vault/builtin/logical/pki.(*backend).pathTidyWrite
December 10th 2018, 00:04:28.000		/gopath/src/github.com/hashicorp/vault/builtin/logical/pki/path_tidy.go:88 +0x296

Expected Behavior:

This should start up and work, in fact, it does it most of the time.

Actual Behavior:

Looks like a panic occurred multiple times on startup this morning (~20x). It came up and stabilized eventually and then later died again and then ran fine the rest of the day with no human interaction.

Steps to Reproduce:

Not sure how to reproduce this. It's only happened on this one occasion.

Important Factoids:

Single pod with an S3 backend.

References: