It takes 2 hours for tikv failover

[zyguan]

When a tikv is down, its state in pd firstly turns to Disconnected, then becomes Down after 1 hour. According to failover logic here, it will take 2 hours for failover. Does this behave as expected? It's little misleading.

[weekface]

Now, it will failover after becoming Down for 1 hours. Did not consider the state of Disconnected.

@tennix @xiaojingchen PTAL

[tennix]

How will PD handle this when a TiKV fails especially when will PD begin to schedule data on the failed store? @nolouch

[nolouch]

@zyguan Can you show the config of the PD? pd-ctl>> config show all

[zyguan]

@nolouch here it is.

{
  "client-urls": "http://0.0.0.0:2379",
  "peer-urls": "http://0.0.0.0:2380",
  "advertise-client-urls": "http://demo-pd-1.demo-pd-peer.test-calico-ipip.svc:2379",
  "advertise-peer-urls": "http://demo-pd-1.demo-pd-peer.test-calico-ipip.svc:2380",
  "name": "demo-pd-1",
  "data-dir": "/var/lib/pd",
  "initial-cluster": "demo-pd-1=http://demo-pd-1.demo-pd-peer.test-calico-ipip.svc:2380",
  "initial-cluster-state": "new",
  "join": "",
  "lease": 3,
  "log": {
    "level": "info",
    "format": "text",
    "disable-timestamp": false,
    "file": {
      "filename": "",
      "log-rotate": true,
      "max-size": 0,
      "max-days": 0,
      "max-backups": 0
    }
  },
  "log-file": "",
  "log-level": "",
  "tso-save-interval": "3s",
  "metric": {
    "job": "demo-pd-1",
    "address": "",
    "interval": "15s"
  },
  "schedule": {
    "max-snapshot-count": 3,
    "max-pending-peer-count": 16,
    "max-merge-region-size": 0,
    "max-merge-region-keys": 0,
    "split-merge-interval": "1h0m0s",
    "patrol-region-interval": "100ms",
    "max-store-down-time": "1h0m0s",
    "leader-schedule-limit": 4,
    "region-schedule-limit": 4,
    "replica-schedule-limit": 8,
    "merge-schedule-limit": 8,
    "tolerant-size-ratio": 5,
    "low-space-ratio": 0.8,
    "high-space-ratio": 0.6,
    "disable-raft-learner": "false",
    "disable-remove-down-replica": "false",
    "disable-replace-offline-replica": "false",
    "disable-make-up-replica": "false",
    "disable-remove-extra-replica": "false",
    "disable-location-replacement": "false",
    "disable-namespace-relocation": "false",
    "schedulers-v2": [
      {
        "type": "balance-region",
        "args": null,
        "disable": false
      },
      {
        "type": "balance-leader",
        "args": null,
        "disable": false
      },
      {
        "type": "hot-region",
        "args": null,
        "disable": false
      },
      {
        "type": "label",
        "args": null,
        "disable": false
      }
    ]
  },
  "replication": {
    "max-replicas": 3,
    "location-labels": "zone,rack,host"
  },
  "namespace": {},
  "cluster-version": "2.1.3",
  "quota-backend-bytes": "0 B",
  "auto-compaction-mode": "periodic",
  "auto-compaction-retention-v2": "1h",
  "TickInterval": "500ms",
  "ElectionInterval": "3s",
  "PreVote": true,
  "security": {
    "cacert-path": "",
    "cert-path": "",
    "key-path": ""
  },
  "label-property": {},
  "WarningMsgs": null,
  "namespace-classifier": "table"
}

[weekface]

@nolouch PTAL

[weekface]

We should failover when the TiKV instance becomes Down. Do not need to wait another 1 hour. @zyguan

[zyguan]

So, the failover should be triggered in a short time after pd.maxStoreDownTime, rather than 2*pd.maxStoreDownTime.

[weekface]

Yes