Memcached port drop appears to flush database rather than move to an offline state

[AndrewLipscomb]

Sync Gateway version

Version 3.0.3 - Docker image

Operating system

Docker on Ubuntu 20.04

Expected behavior

A network disconnection in the memcached/DCP connection stream causes an offline state (or partial offline) state to come up

Actual behavior

What appears to be a database removal/readdition and full DCP resync, as evidenced by

Config: Database "params" was running on this node, but config was not found on the server - removing database

And then

Jun 28 09:33:01 : 2022-06-28T09:33:01.624Z [INF] DCP: Attempting to start import DCP feed...
Jun 28 09:33:01 : 2022-06-28T09:33:01.624Z [WRN] destKey swarmfarm_geospatial_import already exists in cbgtDestFactories - new value will replace the existing dest -- base.StoreDestFactory() at dcp_sharded.go:573
Jun 28 09:33:01 : 2022-06-28T09:33:01.624Z [INF] DCP: Starting DCP import feed for bucket: "<ud>params</ud>"
Jun 28 09:33:01 : 2022-06-28T09:33:01.673Z [INF] DCP: Previously persisted backfill sequences found - will resume
Jun 28 09:33:01 : 2022-06-28T09:33:01.673Z [INF] DCP: c:params-SGI Restarting vb 0 using metadata sequence 0  (backfill 0 not in [0-0])
Jun 28 09:33:01 : 2022-06-28T09:33:01.674Z [INF] DCP: c:params-SGI Restarting vb 1 using metadata sequence 0  (backfill 0 not in [0-0])
Jun 28 09:33:01 : 2022-06-28T09:33:01.681Z [INF] DCP: c:params-SGI Restarting vb 2 using metadata sequence 4  (backfill 0 not in [4-4])
Jun 28 09:33:01 : 2022-06-28T09:33:01.681Z [INF] DCP: c:params-SGI Restarting vb 3 using metadata sequence 0  (backfill 0 not in [0-0])
Jun 28 09:33:01 : 2022-06-28T09:33:01.682Z [INF] DCP: c:params-SGI Restarting vb 4 using metadata sequence 0  (backfill 0 not in [0-0])

Steps to reproduce

Unsure yet - still investigating any networking issues. However the drop itself could be replicated simply using tc or iptables to drop a connection in a "real" way

Log output

Jun 28 09:32:52 : 2022-06-28T09:32:52.242Z [WRN] gocb: CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- base.GoCBCoreLogger.Log() at logger_external.go:79
Jun 28 09:32:52 : 2022-06-28T09:32:52.242Z [WRN] gocb: CCCPPOLL: Failed to retrieve config from any node. -- base.GoCBCoreLogger.Log() at logger_external.go:79
Jun 28 09:32:57 : 2022-06-28T09:32:57.744Z [WRN] gocb: CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- base.GoCBCoreLogger.Log() at logger_external.go:79
Jun 28 09:32:57 : 2022-06-28T09:32:57.744Z [WRN] gocb: CCCPPOLL: Failed to retrieve config from any node. -- base.GoCBCoreLogger.Log() at logger_external.go:79
Jun 28 09:32:59 : Err read tcp 172.17.0.2:54244->172.31.48.157:11210: use of closed network connection
Jun 28 09:32:59 : Err read tcp 172.17.0.2:48128->172.31.49.134:11210: use of closed network connection
Jun 28 09:32:59 : Err read tcp 172.17.0.2:48108->172.31.49.134:11210: use of closed network connection
Jun 28 09:32:59 : 2022-06-28T09:32:59.243Z [WRN] gocb: Pipeline Client 0xc00ebcd2d0 failed to bootstrap: check server ports and cluster encryption setting: dial tcp 172.31.48.229:11210: i/o timeout -- base.GoCBCoreLogger.Log() at logger_external.go:79
Jun 28 09:32:59 : 2022-06-28T09:32:59.243Z [INF] Config: Database "params" was running on this node, but config was not found on the server - removing database
Jun 28 09:32:59 : 2022-06-28T09:32:59.243Z [INF] Closing db /params (bucket "params")
Jun 28 09:32:59 : 2022-06-28T09:32:59.243Z [WRN] c:params-SG Error processing DCP stream - will attempt to restart/reconnect if appropriate: pkt.Receive, err: read tcp 172.17.0.2:54244->172.31.48.157:11210: use of closed network connection. -- base.(*DCPReceiver).OnError() at dcp_receiver.go:60
Jun 28 09:32:59 : 2022-06-28T09:32:59.244Z [WRN] c:params-SGI Error processing DCP stream - will attempt to restart/reconnect if appropriate: pkt.Receive, err: read tcp 172.17.0.2:48128->172.31.49.134:11210: use of closed network connection. -- base.(*DCPReceiver).OnError() at dcp_receiver.go:60
Jun 28 09:32:59 : 2022-06-28T09:32:59.244Z [WRN] c:params-SG Error processing DCP stream - will attempt to restart/reconnect if appropriate: pkt.Receive, err: read tcp 172.17.0.2:48108->172.31.49.134:11210: use of closed network connection. -- base.(*DCPReceiver).OnError() at dcp_receiver.go:60
Jun 28 09:32:59 : Err read tcp 172.17.0.2:54258->172.31.48.157:11210: use of closed network connection
Jun 28 09:32:59 : Err read tcp 172.17.0.2:46378->172.31.48.47:11210: use of closed network connection
Jun 28 09:32:59 : Err read tcp 172.17.0.2:46630->172.31.48.157:11210: use of closed network connection
Jun 28 09:32:59 : Err read tcp 172.17.0.2:38750->172.31.48.47:11210: use of closed network connection
Jun 28 09:32:59 : Err read tcp 172.17.0.2:46364->172.31.48.47:11210: use of closed network connection
Jun 28 09:32:59 : 2022-06-28T09:32:59.245Z [WRN] c:params-SGI Error processing DCP stream - will attempt to restart/reconnect if appropriate: pkt.Receive, err: read tcp 172.17.0.2:54258->172.31.48.157:11210: use of closed network connection. -- base.(*DCPReceiver).OnError() at dcp_receiver.go:60
Jun 28 09:32:59 : 2022-06-28T09:32:59.247Z [WRN] c:params-SGI Error processing DCP stream - will attempt to restart/reconnect if appropriate: pkt.Receive, err: read tcp 172.17.0.2:46378->172.31.48.47:11210: use of closed network connection. -- base.(*DCPReceiver).OnError() at dcp_receiver.go:60