fix(go-client): update config once replica server failed and forward to primary meta server if it was changed

[lengyuexuexuan]

What problem does this PR solve?

#1880
#1856

What is changed and how does it work?

As for #1856.
when go client is writing to one partition and the replica node core dump, go client will finish after timeout without updating the configuration. In this case, the go client only restart to solve the problem.

In this pr, the client would update conconfiguration of table automatically when someone replica core dump.
After testing, we found that the the replicaerror is "context.DeadlineExceeded" when the replica core dump.

incubator-pegasus/go-client/pegasus/table_connector.go

Lines 705 to 706 in 41141c1

	case context.DeadlineExceeded:
	confUpdate = true

Therefore, when client meets the errror, the go client will update configuration automatically.
Besides, this request will not retry. Because only in the case of timeout, the configuration will be automatically updated. If you try again before then, it will still fail. There is also the risk of infinite retries.
Therefore, it is better to directly return the request error to the user and let the user try again.

As for #1880
When the client sends an RPC message "RPC_CM_QUERY_PARTITION_CONFIG_BY_INDEX" to the meta server, if the meta server isn't primary, the response that forward to the primary meta server will return.

According to the above description, assuming that the client does not have a primary meta server configured, we can connect to the primary meta server in this way.

In this PR, we implement this function through the following steps.

1. First parse the response, determine whether its errno is ERR_FORWARD_TO_OTHERS, and then parse it to get the primary meta server address.
   

   incubator-pegasus/go-client/session/meta_call.go

   Lines 166 to 177 in 41141c1

   	func (c *metaCall) getMetaServiceForwardAddress(resp metaResponse) *base.RPCAddress {
   	rep, ok := resp.(*replication.QueryCfgResponse)
   	if !ok || rep.GetErr().Errno != base.ERR_FORWARD_TO_OTHERS.String() {
   	return nil
   	} else if rep.GetPartitions() == nil || len(rep.GetPartitions()) == 0 {
   	return nil
   	} else {
   	return rep.Partitions[0].Primary
   	}
   	}

2. Secondly, determine whether the address is already in the client configuration. If it is already there, skip it directly. Otherwise, establish a connection and pull the configuration directly from the primary meta server.
   

   incubator-pegasus/go-client/session/meta_call.go

   Lines 118 to 138 in 41141c1

   	if forwardAddr == nil {
   	return false
   	}
   	addr := forwardAddr.GetAddress()
   	found := false
   	for i := range c.metaIPAddrs {
   	if addr == c.metaIPAddrs[i] {
   	found = true
   	break
   	}
   	}
   	if !found {
   	c.metaIPAddrs = append(c.metaIPAddrs, addr)
   	c.metas = append(c.metas, &metaSession{
   	NodeSession: newNodeSession(addr, NodeTypeMeta),
   	logger: pegalog.GetLogger(),
   	})
   	curLeader = len(c.metas) - 1
   	c.metas[curLeader].logger.Printf("add forward address %s as meta server", addr)
   	}
   	resp, err = c.callFunc(ctx, c.metas[curLeader])

It should be noted that the IP address and session do not have a one-to-one correspondence, because there may be situations where the IP address is unavailable.
This is why there is a priamry meta server configuration in the client, but the curllead cannot be used as the index of the metaIPAddrs array.

incubator-pegasus/go-client/session/meta_call.go

Lines 123 to 128 in 41141c1

	for i := range c.metaIPAddrs {
	if addr == c.metaIPAddrs[i] {
	found = true
	break
	}
	}

Tests

• Unit test
• Manual test (add detailed scripts or steps below)

1. Start onebox, and the primary meta server is not added to the go client configuration.
2. The go client writes data to a certain partition and then kills the replica process.

[acelyc111]

Thanks for the contribution!

[acelyc111]

The meta servers in the test are 0.0.0.0:3460{1..3}, which one is "not in the client metalist" ?

[lengyuexuexuan]

When onebox starts, the primary meta server is randomized. Therefore, a loop is used, and only one meta server is passed to the go client each time. This ensures that redirection is required twice in the loop.