stale db connection due to "broken pipe"

[schinns]

Stacktrace:

Nov 04 12:51:13 sd-int-postgres201 pgreba[28741]: goroutine 33 [running]:
Nov 04 12:51:13 sd-int-postgres201 pgreba[28741]: net/http.(*conn).serve.func1(0xc00028c000)
Nov 04 12:51:13 sd-int-postgres201 pgreba[28741]:         /usr/local/go/src/net/http/server.go:1801 +0x147
Nov 04 12:51:13 sd-int-postgres201 pgreba[28741]: panic(0x73a2e0, 0xc00018f6e0)
Nov 04 12:51:13 sd-int-postgres201 pgreba[28741]:         /usr/local/go/src/runtime/panic.go:975 +0x3e9
Nov 04 12:51:13 sd-int-postgres201 pgreba[28741]: main.(*HealthCheckWebService).apiGetIsReplica(0xc0001b4080, 0x7f22c28cbdf8, 0xc000288270, 0xc00029a200)
Nov 04 12:51:13 sd-int-postgres201 pgreba[28741]:         /Users/benschinn/code/pgreba/main.go:40 +0x1c9
Nov 04 12:51:13 sd-int-postgres201 pgreba[28741]: net/http.HandlerFunc.ServeHTTP(0xc00018f5d0, 0x7f22c28cbdf8, 0xc000288270, 0xc00029a200)
Nov 04 12:51:13 sd-int-postgres201 pgreba[28741]:         /usr/local/go/src/net/http/server.go:2042 +0x44
Nov 04 12:51:13 sd-int-postgres201 pgreba[28741]: github.com/gorilla/handlers.loggingHandler.ServeHTTP(0x7e8580, 0xc0001b4010, 0x7e8ae0, 0xc00018f5d0, 0x79c6b8, 0x7ed920, 0xc0002a2000, 0xc00029a200)
Nov 04 12:51:13 sd-int-postgres201 pgreba[28741]:         /Users/benschinn/go/pkg/mod/github.com/gorilla/[email protected]/logging.go:45 +0x248
Nov 04 12:51:13 sd-int-postgres201 pgreba[28741]: github.com/gorilla/mux.(*Router).ServeHTTP(0xc00025e000, 0x7ed920, 0xc0002a2000, 0xc00029a000)
Nov 04 12:51:13 sd-int-postgres201 pgreba[28741]:         /Users/benschinn/go/pkg/mod/github.com/gorilla/[email protected]/mux.go:210 +0xd3
Nov 04 12:51:13 sd-int-postgres201 pgreba[28741]: net/http.serverHandler.ServeHTTP(0xc000262000, 0x7ed920, 0xc0002a2000, 0xc00029a000)
Nov 04 12:51:13 sd-int-postgres201 pgreba[28741]:         /usr/local/go/src/net/http/server.go:2843 +0xa3
Nov 04 12:51:13 sd-int-postgres201 pgreba[28741]: net/http.(*conn).serve(0xc00028c000, 0x7ee160, 0xc000294000)
Nov 04 12:51:13 sd-int-postgres201 pgreba[28741]:         /usr/local/go/src/net/http/server.go:1925 +0x8ad
Nov 04 12:51:13 sd-int-postgres201 pgreba[28741]: created by net/http.(*Server).Serve
Nov 04 12:51:13 sd-int-postgres201 pgreba[28741]:         /usr/local/go/src/net/http/server.go:2969 +0x36c
Nov 04 12:51:14 sd-int-postgres201 pgreba[28741]: 2020/11/04 12:51:14 http: panic serving 10.65.33.194:59026: write tcp 10.65.33.131:52758->10.65.33.131:5432: write: broken pipe
Nov 04 12:51:14 sd-int-postgres201 pgreba[28741]: goroutine 35 [running]:
Nov 04 12:51:14 sd-int-postgres201 pgreba[28741]: net/http.(*conn).serve.func1(0xc00028c0a0)
Nov 04 12:51:14 sd-int-postgres201 pgreba[28741]:         /usr/local/go/src/net/http/server.go:1801 +0x147
Nov 04 12:51:14 sd-int-postgres201 pgreba[28741]: panic(0x73a2e0, 0xc00018f6e0)
Nov 04 12:51:14 sd-int-postgres201 pgreba[28741]:         /usr/local/go/src/runtime/panic.go:975 +0x3e9
Nov 04 12:51:14 sd-int-postgres201 pgreba[28741]: main.(*HealthCheckWebService).apiGetIsPrimary(0xc0001b4080, 0x7f22c28cbdf8, 0xc0002884e0, 0xc00029a500)
Nov 04 12:51:14 sd-int-postgres201 pgreba[28741]:         /Users/benschinn/code/pgreba/main.go:26 +0x177
Nov 04 12:51:14 sd-int-postgres201 pgreba[28741]: net/http.HandlerFunc.ServeHTTP(0xc00018f560, 0x7f22c28cbdf8, 0xc0002884e0, 0xc00029a500)
Nov 04 12:51:14 sd-int-postgres201 pgreba[28741]:         /usr/local/go/src/net/http/server.go:2042 +0x44
Nov 04 12:51:14 sd-int-postgres201 pgreba[28741]: github.com/gorilla/handlers.loggingHandler.ServeHTTP(0x7e8580, 0xc0001b4010, 0x7e8ae0, 0xc00018f560, 0x79c6b8, 0x7ed920, 0xc0002a20e0, 0xc00029a500)
Nov 04 12:51:14 sd-int-postgres201 pgreba[28741]:         /Users/benschinn/go/pkg/mod/github.com/gorilla/[email protected]/logging.go:45 +0x248
Nov 04 12:51:14 sd-int-postgres201 pgreba[28741]: github.com/gorilla/mux.(*Router).ServeHTTP(0xc00025e000, 0x7ed920, 0xc0002a20e0, 0xc00029a300)
Nov 04 12:51:14 sd-int-postgres201 pgreba[28741]:         /Users/benschinn/go/pkg/mod/github.com/gorilla/[email protected]/mux.go:210 +0xd3
Nov 04 12:51:14 sd-int-postgres201 pgreba[28741]: net/http.serverHandler.ServeHTTP(0xc000262000, 0x7ed920, 0xc0002a20e0, 0xc00029a300)
Nov 04 12:51:14 sd-int-postgres201 pgreba[28741]:         /usr/local/go/src/net/http/server.go:2843 +0xa3
Nov 04 12:51:14 sd-int-postgres201 pgreba[28741]: net/http.(*conn).serve(0xc00028c0a0, 0x7ee160, 0xc000294140)
Nov 04 12:51:14 sd-int-postgres201 pgreba[28741]:         /usr/local/go/src/net/http/server.go:1925 +0x8ad
Nov 04 12:51:14 sd-int-postgres201 pgreba[28741]: created by net/http.(*Server).Serve
Nov 04 12:51:14 sd-int-postgres201 pgreba[28741]:         /usr/local/go/src/net/http/server.go:2969 +0x36c
Nov 04 12:51:15 sd-int-postgres201 pgreba[28741]: 2020/11/04 12:51:15 http: panic serving 10.65.33.194:59036: write tcp 10.65.33.131:52758->10.65.33.131:5432: write: broken pipe

Related issues:
lib/pq#870

Possible solution:
go-sql-driver/mysql#446 (comment)

[schinns]

#14 addresses this issue